Merge branch '1.10' into sphinx-256

.gitignore

@@ -7,3 +7,4 @@ test.log
 docs/
 !sites/docs
 _build
+.coverage

.travis.yml

@@ -8,7 +8,15 @@ install:
   # Dev (doc/test running) requirements
   - pip install coveralls # For coveralls.io specifically
   - pip install -r dev-requirements.txt
-script: coverage run --source=paramiko test.py --verbose
+script:
+  # Main tests, with coverage!
+  - invoke coverage
+  # Ensure documentation & invoke pipeline run OK.
+  # Run 'docs' first since its objects.inv is referred to by 'www'.
+  # Also force warnings to be errors since most of them tend to be actual
+  # problems.
+  - invoke docs -o -W
+  - invoke www -o -W
 notifications:
   irc:
     channels: "irc.freenode.org#paramiko"

NEWS

@@ -9,102 +9,14 @@ Issues noted as "'ssh' #NN" can be found at https://github.com/bitprophet/ssh/.
 
 Issues noted as "Fabric #NN" can be found at https://github.com/fabric/fabric/.
 
+
+**PLEASE NOTE:** For changes in 1.10.x and newer releases, please see
+www.paramiko.org's changelog page, or the source file, sites/www/changelog.rst
+
+
 Releases
 ========
 
-v1.10.6 (21st Jan 2014)
------------------------
-
-* #193 (and its attentant PRs #230 & #253): Fix SSH agent problems present on
-  Windows. Thanks to David Hobbs for initial report and to Aarni Koskela & Olle
-  Lundberg for the patches.
-
-v1.10.5 (8th Jan 2014)
-----------------------
-
-* #176: Fix AttributeError bugs in known_hosts file (re)loading. Thanks to
-  Nathan Scowcroft for the patch & Martin Blumenstingl for the initial test
-  case.
-
-v1.10.4 (27th Sep 2013)
------------------------
-
-* #179: Fix a missing variable causing errors when an ssh_config file has a
-  non-default AddressFamily set. Thanks to Ed Marshall & Tomaz Muraus for catch
-  & patch.
-* #200: Fix an exception-causing typo in `demo_simple.py`. Thanks to Alex
-  Buchanan for catch & Dave Foster for patch.
-* #199: Typo fix in the license header cross-project. Thanks to Armin Ronacher
-  for catch & patch.
-
-v1.10.3 (20th Sep 2013)
------------------------
-
-* #162: Clean up HMAC module import to avoid deadlocks in certain uses of
-  SSHClient. Thanks to Gernot Hillier for the catch & suggested
-  fix.
-* #36: Fix the port-forwarding demo to avoid file descriptor errors. Thanks to
-  Jonathan Halcrow for catch & patch.
-* #168: Update config handling to properly handle multiple 'localforward' and
-  'remoteforward' keys. Thanks to Emre Yılmaz for the patch.
-
-v1.10.2 (26th Jul 2013)
------------------------
-
-* #153, #67: Warn on parse failure when reading known_hosts file. Thanks to
-  `@glasserc` for patch.
-* #146: Indentation fixes for readability. Thanks to Abhinav Upadhyay for catch
-  & patch.
-
-v1.10.1 (5th Apr 2013)
-----------------------
-
-* #142: (Fabric #811) SFTP put of empty file will still return the attributes
-  of the put file. Thanks to Jason R. Coombs for the patch.
-* #154: (Fabric #876) Forwarded SSH agent connections left stale local pipes
-  lying around, which could cause local (and sometimes remote or network)
-  resource starvation when running many agent-using remote commands. Thanks to
-  Kevin Tegtmeier for catch & patch.
-
-v1.10.0 (1st Mar 2013)
---------------------
-
-* #66: Batch SFTP writes to help speed up file transfers. Thanks to Olle
-  Lundberg for the patch.
-* #133: Fix handling of window-change events to be on-spec and not
-  attempt to wait for a response from the remote sshd; this fixes problems with
-  less common targets such as some Cisco devices. Thanks to Phillip Heller for
-  catch & patch.
-* #93: Overhaul SSH config parsing to be in line with `man ssh_config` (& the
-  behavior of `ssh` itself), including addition of parameter expansion within
-  config values. Thanks to Olle Lundberg for the patch.
-* #110: Honor SSH config `AddressFamily` setting when looking up local
-  host's FQDN. Thanks to John Hensley for the patch.
-* #128: Defer FQDN resolution until needed, when parsing SSH config files.
-  Thanks to Parantapa Bhattacharya for catch & patch.
-* #102: Forego random padding for packets when running under `*-ctr` ciphers.
-  This corrects some slowdowns on platforms where random byte generation is
-  inefficient (e.g. Windows). Thanks to  `@warthog618` for catch & patch, and
-  Michael van der Kolff for code/technique review.
-* #127: Turn `SFTPFile` into a context manager. Thanks to Michael Williamson
-  for the patch.
-* #116: Limit `Message.get_bytes` to an upper bound of 1MB to protect against
-  potential DoS vectors. Thanks to `@mvschaik` for catch & patch.
-* #115: Add convenience `get_pty` kwarg to `Client.exec_command` so users not
-  manually controlling a channel object can still toggle PTY creation. Thanks
-  to Michael van der Kolff for the patch.
-* #71: Add `SFTPClient.putfo` and `.getfo` methods to allow direct
-  uploading/downloading of file-like objects. Thanks to Eric Buehl for the
-  patch.
-* #113: Add `timeout` parameter to `SSHClient.exec_command` for easier setting
-  of the command's internal channel object's timeout. Thanks to Cernov Vladimir
-  for the patch.
-* #94: Remove duplication of SSH port constant. Thanks to Olle Lundberg for the
-  catch.
-* #80: Expose the internal "is closed" property of the file transfer class
-  `BufferedFile` as `.closed`, better conforming to Python's file interface.
-  Thanks to `@smunaut` and James Hiscock for catch & patch.
-
 v1.9.0 (6th Nov 2012)
 ---------------------
 

dev-requirements.txt

@@ -1,7 +1,7 @@
 # For newer tasks like building Sphinx docs.
 # NOTE: Requires Python >=2.6
-invoke>=0.6.1
+invoke>=0.7.0
 invocations>=0.4.4
 sphinx>=1.1.3
-alabaster>=0.1.0
+alabaster>=0.3.0
-releases>=0.2.4
+releases>=0.5.1

paramiko/__init__.py

@@ -55,7 +55,7 @@ if sys.version_info < (2, 5):
 
 
 __author__ = "Jeff Forcier <jeff@bitprophet.org>"
-__version__ = "1.10.5"
+__version__ = "1.10.6"
 __version_info__ = tuple([ int(d) for d in __version__.split(".") ])
 __license__ = "GNU Lesser General Public License (LGPL)"
 

paramiko/hostkeys.py

@@ -28,7 +28,7 @@ import UserDict
 from paramiko.common import *
 from paramiko.dsskey import DSSKey
 from paramiko.rsakey import RSAKey
-from paramiko.util import get_logger
+from paramiko.util import get_logger, constant_time_bytes_eq
 
 
 class InvalidHostKey(Exception):
@@ -243,7 +243,7 @@ class HostKeys (UserDict.DictMixin):
         entries = []
         for e in self._entries:
             for h in e.hostnames:
-                if (h.startswith('|1|') and (self.hash_host(hostname, h) == h)) or (h == hostname):
+                if h.startswith('|1|') and constant_time_bytes_eq(self.hash_host(hostname, h), h) or h == hostname:
                     entries.append(e)
         if len(entries) == 0:
             return None

paramiko/packet.py

@@ -358,7 +358,7 @@ class Packetizer (object):
             mac = post_packet[:self.__mac_size_in]
             mac_payload = struct.pack('>II', self.__sequence_number_in, packet_size) + packet
             my_mac = compute_hmac(self.__mac_key_in, mac_payload, self.__mac_engine_in)[:self.__mac_size_in]
-            if my_mac != mac:
+            if not util.constant_time_bytes_eq(my_mac, mac):
                 raise SSHException('Mismatched MAC')
         padding = ord(packet[0])
         payload = packet[1:packet_size - padding]

paramiko/sftp_client.py

@@ -736,7 +736,7 @@ class SFTPClient (BaseSFTP):
                     self._convert_status(msg)
                 return t, msg
             if fileobj is not type(None):
-                fileobj._async_response(t, msg)
+                fileobj._async_response(t, msg, num)
             if waitfor is None:
                 # just doing a single check
                 break

paramiko/sftp_file.py

@@ -20,6 +20,8 @@
 :class:`SFTPFile`
 """
 
+from __future__ import with_statement
+
 from binascii import hexlify
 from collections import deque
 import socket
@@ -53,7 +55,8 @@ class SFTPFile (BufferedFile):
         self._prefetching = False
         self._prefetch_done = False
         self._prefetch_data = {}
-        self._prefetch_reads = []
+        self._prefetch_extents = {}
+        self._prefetch_lock = threading.Lock()
         self._saved_exception = None
         self._reqs = deque()
 
@@ -91,7 +94,7 @@ class SFTPFile (BufferedFile):
             pass
 
     def _data_in_prefetch_requests(self, offset, size):
-        k = [i for i in self._prefetch_reads if i[0] <= offset]
+        k = [x for x in self._prefetch_extents.values() if x[0] <= offset]
         if len(k) == 0:
             return False
         k.sort(lambda x, y: cmp(x[0], y[0]))
@@ -447,7 +450,6 @@ class SFTPFile (BufferedFile):
     def _start_prefetch(self, chunks):
         self._prefetching = True
         self._prefetch_done = False
-        self._prefetch_reads.extend(chunks)
 
         t = threading.Thread(target=self._prefetch_thread, args=(chunks,))
         t.setDaemon(True)
@@ -457,9 +459,11 @@ class SFTPFile (BufferedFile):
         # do these read requests in a temporary thread because there may be
         # a lot of them, so it may block.
         for offset, length in chunks:
-            self.sftp._async_request(self, CMD_READ, self.handle, long(offset), int(length))
+            with self._prefetch_lock:
+                num = self.sftp._async_request(self, CMD_READ, self.handle, long(offset), int(length))
+                self._prefetch_extents[num] = (offset, length)
 
-    def _async_response(self, t, msg):
+    def _async_response(self, t, msg, num):
         if t == CMD_STATUS:
             # save exception and re-raise it on next file operation
             try:
@@ -470,10 +474,12 @@ class SFTPFile (BufferedFile):
         if t != CMD_DATA:
             raise SFTPError('Expected data')
         data = msg.get_string()
-        offset, length = self._prefetch_reads.pop(0)
+        with self._prefetch_lock:
-        self._prefetch_data[offset] = data
+            offset, length = self._prefetch_extents[num]
-        if len(self._prefetch_reads) == 0:
+            self._prefetch_data[offset] = data
-            self._prefetch_done = True
+            del self._prefetch_extents[num]
+            if len(self._prefetch_extents) == 0:
+                self._prefetch_done = True
     
     def _check_exception(self):
         "if there's a saved exception, raise & clear it"

paramiko/util.py

@@ -309,3 +309,12 @@ class Counter (object):
     def new(cls, nbits, initial_value=1L, overflow=0L):
         return cls(nbits, initial_value=initial_value, overflow=overflow)
     new = classmethod(new)
+
+
+def constant_time_bytes_eq(a, b):
+    if len(a) != len(b):
+        return False
+    res = 0
+    for i in xrange(len(a)):
+        res |= ord(a[i]) ^ ord(b[i])
+    return res == 0

setup.py

@@ -52,7 +52,7 @@ if sys.platform == 'darwin':
 
 
 setup(name = "paramiko",
-      version = "1.10.5",
+      version = "1.10.6",
       description = "SSH2 protocol library",
       author = "Jeff Forcier",
       author_email = "jeff@bitprophet.org",

sites/shared_conf.py

@@ -5,14 +5,13 @@ import sys
 import alabaster
 
 
-# Alabaster theme
+# Alabaster theme + mini-extension
 html_theme_path = [alabaster.get_path()]
+extensions = ['alabaster']
 # Paths relative to invoking conf.py - not this shared file
 html_static_path = ['../_shared_static']
 html_theme = 'alabaster'
 html_theme_options = {
-    'logo': 'logo.png',
-    'logo_name': 'true',
     'description': "A Python implementation of SSHv2.",
     'github_user': 'paramiko',
     'github_repo': 'paramiko',
@@ -21,19 +20,11 @@ html_theme_options = {
 
     'link': '#3782BE',
     'link_hover': '#3782BE',
-
 }
 html_sidebars = {
-    # Landing page (no ToC)
-    'index': [
-        'about.html',
-        'searchbox.html',
-        'donate.html',
-    ],
-    # Inner pages get a ToC
     '**': [
         'about.html',
-        'localtoc.html',
+        'navigation.html',
         'searchbox.html',
         'donate.html',
     ]
@@ -42,7 +33,7 @@ html_sidebars = {
 # Regular settings
 project = u'Paramiko'
 year = datetime.now().year
-copyright = u'%d Jeff Forcier, 2003-2012 Robey Pointer' % year
+copyright = u'%d Jeff Forcier' % year
 master_doc = 'index'
 templates_path = ['_templates']
 exclude_trees = ['_build']

sites/www/changelog.rst

@@ -2,8 +2,15 @@
 Changelog
 =========
 
-* :release:`1.10.6 <2014-01-21>`
+* :bug:`-` Use constant-time hash comparison operations where possible, to
-* :bug:`193` (and its attentant PRs :issue:`230` & :issue:`253`): Fix SSH agent
+  protect against `timing-based attacks
+  <http://codahale.com/a-lesson-in-timing-attacks/>`_. Thanks to Alex Gaynor
+  for the patch.
+* :release:`1.10.6 <2014-02-14>`
+* :bug:`34` (PR :issue:`35`) Fix SFTP prefetching incompatibility with some
+  SFTP servers regarding request/response ordering. Thanks to Richard
+  Kettlewell for catch & patch.
+* :bug:`193` (and its attentant PRs :issue:`230` & :issue:`253`) Fix SSH agent
   problems present on Windows. Thanks to David Hobbs for initial report and to
   Aarni Koskela & Olle Lundberg for the patches.
 * :release:`1.10.5 <2014-01-08>`
@@ -14,7 +21,7 @@ Changelog
 * :bug:`179` Fix a missing variable causing errors when an ssh_config file has
   a non-default AddressFamily set. Thanks to Ed Marshall & Tomaz Muraus for
   catch & patch.
-* :bug:`200` Fix an exception-causing typo in `demo_simple.py`. Thanks to Alex
+* :bug:`200` Fix an exception-causing typo in ``demo_simple.py``. Thanks to Alex
   Buchanan for catch & Dave Foster for patch.
 * :bug:`199` Typo fix in the license header cross-project. Thanks to Armin
   Ronacher for catch & patch.
@@ -27,7 +34,7 @@ Changelog
   and 'remoteforward' keys. Thanks to Emre Yılmaz for the patch.
 * :release:`1.10.2 <2013-07-26>`
 * :bug:`153` (also :issue:`67`) Warn on parse failure when reading known_hosts
-  file.  Thanks to `@glasserc` for patch.
+  file.  Thanks to ``@glasserc`` for patch.
 * :bug:`146` Indentation fixes for readability. Thanks to Abhinav Upadhyay for
   catch & patch.
 * :release:`1.10.1 <2013-04-05>`
@@ -46,32 +53,32 @@ Changelog
   attempt to wait for a response from the remote sshd; this fixes problems with
   less common targets such as some Cisco devices. Thanks to Phillip Heller for
   catch & patch.
-* :feature:`93` Overhaul SSH config parsing to be in line with `man
+* :feature:`93` Overhaul SSH config parsing to be in line with ``man
-  ssh_config` (& the behavior of `ssh` itself), including addition of parameter
+  ssh_config`` (& the behavior of ``ssh`` itself), including addition of parameter
   expansion within config values. Thanks to Olle Lundberg for the patch.
-* :feature:`110` Honor SSH config `AddressFamily` setting when looking up
+* :feature:`110` Honor SSH config ``AddressFamily`` setting when looking up
   local host's FQDN. Thanks to John Hensley for the patch.
 * :feature:`128` Defer FQDN resolution until needed, when parsing SSH config
   files.  Thanks to Parantapa Bhattacharya for catch & patch.
 * :bug:`102 major` Forego random padding for packets when running under
-  `*-ctr` ciphers.  This corrects some slowdowns on platforms where random byte
+  ``*-ctr`` ciphers.  This corrects some slowdowns on platforms where random
-  generation is inefficient (e.g. Windows). Thanks to  `@warthog618` for catch
+  byte generation is inefficient (e.g. Windows). Thanks to  ``@warthog618`` for
-  & patch, and Michael van der Kolff for code/technique review.
+  catch & patch, and Michael van der Kolff for code/technique review.
-* :feature:`127` Turn `SFTPFile` into a context manager. Thanks to Michael
+* :feature:`127` Turn ``SFTPFile`` into a context manager. Thanks to Michael
   Williamson for the patch.
-* :feature:`116` Limit `Message.get_bytes` to an upper bound of 1MB to protect
+* :feature:`116` Limit ``Message.get_bytes`` to an upper bound of 1MB to protect
-  against potential DoS vectors. Thanks to `@mvschaik` for catch & patch.
+  against potential DoS vectors. Thanks to ``@mvschaik`` for catch & patch.
-* :feature:`115` Add convenience `get_pty` kwarg to `Client.exec_command` so
+* :feature:`115` Add convenience ``get_pty`` kwarg to ``Client.exec_command`` so
   users not manually controlling a channel object can still toggle PTY
   creation. Thanks to Michael van der Kolff for the patch.
-* :feature:`71` Add `SFTPClient.putfo` and `.getfo` methods to allow direct
+* :feature:`71` Add ``SFTPClient.putfo`` and ``.getfo`` methods to allow direct
   uploading/downloading of file-like objects. Thanks to Eric Buehl for the
   patch.
-* :feature:`113` Add `timeout` parameter to `SSHClient.exec_command` for
+* :feature:`113` Add ``timeout`` parameter to ``SSHClient.exec_command`` for
   easier setting of the command's internal channel object's timeout. Thanks to
   Cernov Vladimir for the patch.
 * :support:`94` Remove duplication of SSH port constant. Thanks to Olle
   Lundberg for the catch.
 * :feature:`80` Expose the internal "is closed" property of the file transfer
-  class `BufferedFile` as `.closed`, better conforming to Python's file
+  class ``BufferedFile`` as ``.closed``, better conforming to Python's file
-  interface.  Thanks to `@smunaut` and James Hiscock for catch & patch.
+  interface.  Thanks to ``@smunaut`` and James Hiscock for catch & patch.

sites/www/conf.py

@@ -1,15 +1,35 @@
 # Obtain shared config values
-import os, sys
+import sys
-sys.path.append(os.path.abspath('..'))
+import os
+from os.path import abspath, join, dirname
+
+sys.path.append(abspath(join(dirname(__file__), '..')))
 from shared_conf import *
 
-# Add local blog extension
+# Local blog extension
-sys.path.append(os.path.abspath('.'))
+sys.path.append(abspath('.'))
-extensions = ['blog']
+extensions.append('blog')
 rss_link = 'http://paramiko.org'
 rss_description = 'Paramiko project news'
 
-# Add Releases changelog extension
+# Releases changelog extension
 extensions.append('releases')
 releases_release_uri = "https://github.com/paramiko/paramiko/tree/%s"
 releases_issue_uri = "https://github.com/paramiko/paramiko/issues/%s"
+
+# Intersphinx for referencing API/usage docs
+extensions.append('sphinx.ext.intersphinx')
+# Default is 'local' building, but reference the public docs site when building
+# under RTD.
+target = join(dirname(__file__), '..', 'docs', '_build')
+if os.environ.get('READTHEDOCS') == 'True':
+    # TODO: switch to docs.paramiko.org post go-live of sphinx API docs
+    target = 'http://paramiko-docs.readthedocs.org/en/latest/'
+#intersphinx_mapping = {
+#    'docs': (target, None),
+#}
+
+# Sister-site links to API docs
+html_theme_options['extra_nav_links'] = {
+    "API Docs": 'http://docs.paramiko.org',
+}

sites/www/contact.rst

@@ -8,4 +8,4 @@ following ways:
 * IRC: ``#paramiko`` on Freenode
 * Mailing list: ``paramiko@librelist.com`` (see `the LibreList homepage
   <http://librelist.com>`_ for usage details).
-* This website's :doc:`blog </blog>`.
+* This website - a blog section is forthcoming.

sites/www/contributing.rst

@@ -6,7 +6,7 @@ How to get the code
 ===================
 
 Our primary Git repository is on Github at `paramiko/paramiko
-<https://github.com/paramiko/paramiko>`; please follow their instruction for
+<https://github.com/paramiko/paramiko>`_; please follow their instructions for
 cloning to your local system. (If you intend to submit patches/pull requests,
 we recommend forking first, then cloning your fork. Github has excellent
 documentation for all this.)

sites/www/index.rst

@@ -12,12 +12,18 @@ usage and API documentation can be found at our code documentation site,
 `docs.paramiko.org <http://docs.paramiko.org>`_.
 
 .. toctree::
-    blog
     changelog
     installing
     contributing
     contact
 
+.. Hide blog in hidden toctree for now (to avoid warnings.)
+
+.. toctree::
+    :hidden:
+
+    blog
+
 
 .. rubric:: Footnotes
 

tasks.py

@@ -1,7 +1,7 @@
 from os.path import join
 
-from invoke import Collection
+from invoke import Collection, ctask as task
-from invocations import docs as _docs, testing
+from invocations import docs as _docs
 
 
 d = 'sites'
@@ -20,4 +20,15 @@ www = Collection.from_module(_docs, name='www', config={
     'sphinx.target': join(path, '_build'),
 })
 
-ns = Collection(testing.test, docs=docs, www=www)
+
+# Until we move to spec-based testing
+@task
+def test(ctx):
+    ctx.run("python test.py --verbose")
+
+@task
+def coverage(ctx):
+    ctx.run("coverage run --source=paramiko test.py --verbose")
+
+
+ns = Collection(test, coverage, docs=docs, www=www)