as john arbash meinel points out, this assert wasn't validating internal
consistency, because a readv() request may ask for more data than is left
in the file. so remove the assert.
add a ResourceManager to replace __del__ methods, and use it in SSHClient
to automatically close any open transport when the SSHClient is collected.
this won't work on Transport itself (to close the attached packetizer)
because Transport starts up its own thread, and the threading library
keeps a Transport object alive to run that thread. i think that's okay;
the SSHClient interface is meant to be the easier one, so that's the one
where it's important that some auto-cleanup is attempted.
bug 75370: notice garbage sftp packets
since sftp packets shouldn't be larger than about 32k, if the first length
byte is non-zero (ie, the packet size > 16M), raise an exception.
sometimes the sftp module is used with raw sockets, not just paramiko
Channels. in this case, calling recv() will never return. so notice
this and use select() to give python a chance to notice a closed socket.
this kind of thing is especially useful for unit tests.
bug 70398:
allow constructions like:
hostkeys['hostname'] = {}
to create an empty host entry object, so that future attempts to set
keys will at least not throw an exception. (they'll still silently do
nothing, though.)
(bug 69222)
on some recent linux kernels, a socket can return "readable" from select,
but a subsequent read() will return EAGAIN. this is against the contract
of select(), so python's socketmodule doesn't catch it or handle it.
therefore, we need to. EAGAIN should now be treated the same as a
socket timeout.
(bug 69330)
in SFTPClient._auth, check for the existence of the rsa/dsa keys before
trying to open them, so that an I/O exception doesn't mask an earlier
one.
variant of a patch from warren young to preserve the order of host entries from the 'known_hosts' file and preserve knowlege of which lines had multiple hostnames on them
allow prefetch + readv to occur at the same time (even though it will be really inefficient). instead of a moving pointer, use the prefetched buffers as an indication of what we've downloaded so far. break up large readv requests into the max packet size. add 2 more unit tests to test this stuff.
if open_channel fails, it now raises ChannelException. added a unit test for that too. renegotiate_keys will also raise an exception now instead of returning a bool.
some fixes for win32: potential fix for lockup during failed tests; don't try chmod/chown or symlinks; fix canonicalize to convert dos-style path separators to '/'; open local files in binary mode; close a file before erasing it
revise the prefetch machinery so that it doesn't assert if you seek to earlier than the prefetched buffers. also, keep around any prefetched data that has been seeked around, so that jumping around in the file will still get the benefit of prefetch, though only the first time any chunk is read.
reads were being rounded up to an 8k buffer size, even with no buffering specified when the file was opened. i traced this back to the old ChannelFile where it worked correctly. fix to only buffer when asked to.
doing a close() on Transport.__del__ is useless: because a Transport can't go away until all its Channels are collected, it will never make a difference. so don't do it, cuz __del__ is pretty hairy.
some performance improvements: be a LOT less aggressive about stirring the randpool; use buffering when reading the banner; add a hook for using a native-compiled hmac (which gives the biggest boost, but should probably be done in pycrypto)
don't call _send_user_message() while holding the channel lock. the call may
block waiting for CTS (during rekey), and the feeder thread may be waiting on
the channel lock to clear out channel traffic before it gets to the kex-init.
instead, the 2 methods that wanted to send from inside the lock now just return
messages to send. slightly hacky but functional. :)
remember once a transport is authenticated, so that after rekeying when using openssh-style delayed zlib (zlib@openssh.com), we know it's okay to immediately start new compression
windows users who switch between cygwin and native mode may sometimes have an SSH_AUTH_SOCK environ var set from cygwin, even when in native mode there's no such thing as an AF_LOCAL socket -- check for native windows mode and avoid trying unix ssh agents in that case
when closing an sftp file because of __del__, don't wait for a response, just shoot off a request and leave (on linux, the GC is run from a devoted thread)
nail down select() on EOF: occasionally a channel would be closed remotely but select() wouldn't trigger. when a channel gets EOF or is closed, set the pipe FOREVER.
don't attempt to start a rekey negotiation from within send_message -- always do it from the feeder thread. this prevents a situation where more than one thread may decide spontaneously to rekey, sending multiple kexinit messages, which confuses the hell out of the remote host :) also, do some locking around the clear-to-send event, to avoid a race when we first go into rekeying. add some tests for these things too
the window-adjust can be sent outside of the lock, as long as the window size tracking is done within the lock (ie: allocate window space within the lock, then send the ack later) -- helps avoid deadlocks
add auth_none and auth_interactive to support 'no auth' and 'keyboard-interactive auth'; for password auth, add a mode where it will try to fallback to keyboard-interactive in a pinch
smooth BaseTransport and Transport together, and move the auth stuff into AuthHandler -- an improvement i made in jaramiko and decided deserved to be backported
add SFTPFile.check and server support (and test) -- it's an sftp extension that allows a client to retrieve the hash of part or all of a file without downloading it. we're probably the only ones who implement it yet
yeah! figured out the last things that were causing GC cycles and got rid of them: Channels, Transports, SFTPClients, and SFTPFiles can all have __del__ methods now, which auto-close themselves :)
try to handle utf8 paths in a sane way in SFTPClient: assume incoming strings are already utf8, and encode incoming unicodes -- for return values, use str if an ascii encoding will work, otherwise return a unicode
pull in tiny changes from ralph
Patches applied:
* robey@lag.net--2005/paramiko--dev--1--patch-2
merge laptop work
* robey@lag.net--2005/paramiko--dev--1--patch-3
update from laptop
* robey@lag.net--2005/paramiko--dev--1--patch-4
integrate tons more laptop work
* robey@lag.net--2005/paramiko--dev--1--patch-5
thew has a good point: this is stable software now, not alpha
* robey@lag.net--2005/paramiko--dev--1--patch-6
roll up laptop changes
fix a comment claiming that channels are closed automatically when GC'd (they aren't and can't be); don't close the pipe until the app explicitly calls close(); signal EOF via the pipe
the previous windows pipe fix still didn't work. replace it with a new pipe.py abstraction of pipes (one for posix, one for windows) which appears to finally work on windows. for real this time. also add some more documentation to Channel to explain that after exec_command, invoke_shell, or invoke_subsystem, a Channel can't be reused.
oooh maybe i'll test things before checking them in next time: rekeying was a little bit overzealous. now it's careful to only rekey once and reset the counters in sync
split out Packetizer, fix banner detection bug, new unit test
split out a chunk of BaseTransport into a Packetizer class, which handles
the in/out packet data, ciphers, etc. it didn't make the code any smaller
(transport.py is still close to 1500 lines, which is awful) but it did split
out a coherent chunk of functionality into a discrete unit.
in the process, fixed a bug that alain spineux pointed out: the banner
check was too forgiving and would block forever waiting for an SSH banner.
now it waits 5 seconds for the first line, and 2 seconds for each subsequent
line, before giving up.
added a unit test to test keepalive, since i wasn't sure that was still
working after pulling out Packetizer.
make SFTPClient.file an alias for SFTPClient.open.
clean up docs a little, and make 'file' an alias for 'open'.
this is how python is heading in general.
integrated laptop work (test commit)
Patches applied:
* robey@lag.net--2003-public-master-shake/secsh--dev--1.0--base-0
tag of robey@lag.net--2003-public/secsh--dev--1.0--patch-160
* robey@lag.net--2003-public-master-shake/secsh--dev--1.0--patch-1
test commit
* robey@lag.net--2003-public/secsh--dev--1.0--base-0
initial import
* robey@lag.net--2003-public/secsh--dev--1.0--patch-1
no changes
change SubsystemHandler/SFTPServerInterface API
change the API of SubsystemHandler to accept a reference to the
ServerInstance object during construction. this will break all code
that currently creates subsystem handlers (like sftp servers) -- sorry!
lots of little doc fixups (mostly indenting).
rewrite channel pipes to work on windows
the pipe system i was using for simulating an os-level FD (for select) was
retarded. i realized this week that i could just use a single byte in the
pipe to signal "data is ready" and not try to feed all incoming data thru
the pipe -- and then i don't have to try to make the pipe non-blocking (which
should make it work on windows). a lot of duplicate code got removed and now
it's all going thru the same code-path on read.
there's still a slight penalty on incoming feeds and calling 'recv' when a
pipe has been opened (by calling 'fileno'), but it's tiny.
removed a bunch of documentation and comments about things not working on
windows, since i think they probably do now.
fix sending of large sftp packet sizes
fix a bug where packets larger than about 12KB would cause the session to
die on platforms other than osx. turns out that on most platforms, setting a
socket timeout also causes timeouts to occur on writes (but not on osx). so
on a huge write, once the os buffers were full, paramiko would get a
socket.timeout exception when writing, and bail.
since the timeout is primarily so we can periodically poll to see if the
session has been killed from elsewhere, do that on a timeout but otherwise
continue trying to write. large packet sizes (in sftp) should now work.
tweak sftp_file write behavior on large blocks of data
BufferedFile.write() wasn't correctly dealing with the possibility that the
underlying write might not write the entire data block at once (even though
the docs said it would). now that it's working, make sftp_file take
advantage of it in order to chop up blocks larger than 32kB (the max allowed
on sftp) and add a unit test for it.
fix race in transport thread startup
set active=True from the methods that start the main transport thread, right
before actually starting the thread. this avoids a race where the main
thread could be started, but the original thread could wake up from the
event.wait(0.1) before the new thread actually set the transport active.
impossible, you say? no machines so slow exist? au contraire, my sad
little linux box faced this problem earlier today.
when combining stderr with stdout on a channel, merge the buffers too
when turning on combine-stderr mode on a channel, grab the channel lock and
feed any existing stderr buffer into the normal buffer. this should help
applications (and my unit tests) avoid races between data coming in over
stderr and setting combine-stderr.
_send_eof is now slightly safer too, although i don't think that really fixed
anything. it just makes me feel better.
add thread ids to logs
add a logging filter that reports the thread-id of the logger, and use
that for all paramiko logging. since thread-local stuff didn't appear
until python 2.4, i hacked up my own little version to assign incrementing
numbers to threads as they log.
add methods for sending/receiving a channel's exit status
track a channel's exit status and provide a method (recv_exit_status) to
block waiting for it to arrive. also provide a convenience method for
servers to send it (send_exit_status). add shutdown_read and shutdown_write.
fix a bug in sending window change requests.
misc logging fixes
change the level of some log messages so interesting stuff gets logged at
info instead of debug. fix an oops where channels defaulted to being in
ultra debug mode, and make this mode depend on a new Transport method:
"set_hexdump".
more flexible logging
some tweaks to make channels etc follow the logger setting of their parent
transport, so that setting the log channel for a paramiko transport will
cause all sub-logging to branch out from that channel.
also, close all open file handles when the sftp server ends.
added listdir_attr()
add SFTPClient.listdir_attr() to fetch a list of files & their attributes,
instead of just their filenames. artur piwko would find this useful.
cleanup & docs in sftp
add some more docs to SFTPHandle, and give a default implementation for
close() that's usually right. add a flush() to the default implementation
of write(). document that symlink's args in the sftp protocol are out of
order (the spec is wrong).
server support for stderr & exec_command
for the server side of my stderr blunder, add send_stderr & sendall_stderr,
and make the sending side of makefile_stderr work correctly.
also, call check_channel_exec_request on a server object for exec requests
on a channel.
add client-side multi-part auth support
added support for multi-part authentication (even though nobody supports it
that i've seen). on a successful "partial" auth, the auth_* method will
return a list of acceptable means to continue authenticating.
clean up authentication
add new exception "BadAuthenticationType", which is raised when auth fails
because your auth type (password or public-key) isn't valid on the server.
used this as an excuse to clean up auth_password and auth_publickey so their
'event' arg is optional, and if missing, they block until auth is finished,
raising an exception on error.
also, don't close the session on failed auth -- the server may let you try
again.
added some test cases for failed auth.
symlink, readlink
add support for symlink command, and finish support for readlink. (i guess
i started readlink a while ago but forgot to add the right method to the
SFTPServerInterface class.)
add stderr support methods
big embarrassment: i didn't read the ssh2 docs close enough, and all this
time paramiko wasn't handling "extended_data" packets, which contain stderr
output.
so now, several new functions: recv_stderr_ready() and recv_stderr() to
mirror recv_ready() and recv(), and set_combined_stderr() to force stderr
to be combined into stdout. also, makefile_stderr() to create a fake file
object to represent stderr.
fix SFTPFile gettimeout/settimeout
i don't think the gettimeout/settimeout calls on SFTPFile ever worked.
also, simplify the implementation of _get_size() since it's nearly
identical to stat().
sftp server support!
finally check in sftp_handle (file handle abstraction), sftp_si (server
interface), and sftp_server (server implementation) -- all of which make
a roughly 90% implementation of server-side sftp.
add finish_subsystem()
when a SubsystemHandler is being decomissioned (the client has closed the
channel or transport, or the socket went away), make a callback to let the
handler do any shutdown it needs to.
fix extremely unlikely channel counter wrapping
Transport's channel counter can overflow after 4 billion some channels are
created. make it wrap back around after 16 million instead. also allow the
logging channel to be set manually. fix some comments elsewhere.
fix Transport.get_username() to work in server mode too
whenever i split the 'username' field into username and auth_username,
i guess that made get_username() stop working for server mode (because the
username was stored in a different field). this should fix it.
don't unlink a Channel until the server closes it too
when close()'ing a Channel, don't immediately unlink it from the Transport.
instead, wait for the server to send a close message.
this should fix a bug where doing close() on an EOF'd channel would cause
the entire transport to be killed, because the server would send an
'exit-status' and 'close' message for a channel that we no longer had a
record of.
better debugging, improve subsytem handler
add a list of ssh packet names for debugging. improve the server-mode
subsystem handler so it can take extra parameters (list or keyword) and
pass them to the subsystem constructor. remove a misleading comment
about rekeying (which was already implemented).
switch Transport.connect() to using a Pkey object for the host key
i suddenly realized that passing "hostkeytype" and "hostkey" as strings to
Transport.connect() was pretty silly since i went to all the effort of making
a class specifically for holding keys. so Transport.connect() now just takes
host-key argument: "hostkey" as a PKey object.
updated the demos to use PKey objects when reading the host key file, and to
use the new "hostkey" argument.
fix __init__
fix __init__ to export BufferedFile and randpool, and to catch up with the
changes from a week or 2 ago where sftp_attr & friends were split off.
add Message.rewind()
add rewind() method to Message, which just resets the pointer so you can
start reading from the beginning again. this is useful for some tests.
clean up pkey interface
change the pkey interface so that it's no longer possible to have a pkey
that doesn't represent a valid key. (ie: no more "blank" key objects.)
also add "get_bits" and "can_sign" methods to determine the key bit length
and whether it can sign things (contains the "private parts") respectively.
add sftp.normalize
kevin c. dorff pointed out that it would be nice to expose a way to
determine the server's "current working directory", so this new method
(normalize) directly maps to REALPATH.
move SubsystemHandler to server.py
move SubsystemHandler into server.py where it makes more sense (it's part of
the server interface).
also fix up paramiko's "version string" used in ssh2 negotiation to stop
saying "pyssh" and start saying "paramiko". :)
Message.add() can take many args
a bit of cleanup to Message: add() can now take any number of params, and
will add them all in order (using type guessing).
fix rbuffer -> _rbuffer in 3 places i missed
fix 3 places where "rbuffer" hadn't been converted to "_rbuffer". thanks to
kevin c. dorff for the bug report.
add sftp_client.py
i retardedly forgot to import this file a few days ago: it's the split-out
client mode for sftp. it now also has some changes to adapt it to the
improved SFTPAttributes object API.
clean up SFTPAttributes
add english descriptions to the FX_* error codes of sftp. clean up (and
document) SFTPAttributes since it's exported now, and make it simple to
generate one from a python os.stat object. make "_pythonize" the default --
that is, just use the same field names as python does for os.stat. (i'm not
sure why i didn't do it that way in the first place; probably ignorance.)
also add str() method that converts the SFTPAttributes into a string suitable
for use in ls (used in an obscure way in sftp servers).
split sftp into sftp, sftp_client; renamed SFTP -> SFTPClient
add sftp_client file, and split out the common code (sftp) from stuff specific
to client mode (sftp_client). renamed SFTP class to SFTPClient, but left an
alias so old code will still work.
renamed a bunch of sftp constants now that they're better hidden from epydoc.
some framework for adding subsystem handlers in server mode
you can now register a subsystem with a Transport by passing in the name
(like "sftp") and a class (like a hypothetical SFTPServer). the default
ServerInterface.check_channel_request_subsystem now checks this table in
Transport, and if it finds a match, it creates a new thread for the handler
and calls into it. a new class SubsystemHandler is added for this purpose
(to be subclassed).
remove redundant 'auth_complete' member
remove the redundant 'auth_complete' field and just use 'authenticated' for
both client and server mode. this makes the repr() string look correct in
server mode instead of always claiming that the transport is un-auth'd.
clean up server interface; no longer need to subclass Channel
- export AUTH_*, OPEN_FAILED_*, and the new OPEN_SUCCEEDED into the paramiko
namespace instead of making people dig into paramiko.Transport.AUTH_* etc.
- move all of the check_* methods from Channel to ServerInterface so apps
don't need to subclass Channel anymore just to run an ssh server
- ServerInterface.check_channel_request() returns an error code now, not a
new Channel object
- fix demo_server.py to follow all these changes
- fix a bunch of places where i used "string" in docstrings but meant "str"
- added Channel.get_id()
clean up SecurityOptions
the preferences are now tuples in Transport, and passed as tuples out of
SecurityOptions, so that the options can't be modified without setting them
back to the options field again. the algorithm lists in Transport are used
to validate the fields.
added Transport.get_security_options()
just something i wanted to play with:
added Transport.get_security_options() which returns a SecurityOptions object.
this object is a kind of proxy for the 4 "preferred_*" fields in Transport,
and lets me avoid exposing those fields directly in case i change my mind
later about how they should be stored.
added some docs to Channel explaining that the request methods now return
True/False, and fixed up docs in a few other places.
replay patch 63 (missing channel changes)
i'm still getting the hang of tla/arch, obviously.
replay patch 63, which was meant to be part of the later mega-patch, but
apparently when i reversed it, i lost it entirely.
new ServerInterface class, outbound rekey works, etc.
a bunch of changes that i'm too lazy to split out into individual patches:
* all the server overrides from transport.py have been moved into a separate
class ServerInterface, so server code doesn't have to subclass the whole
paramiko library
* updated demo_server to subclass ServerInterface
* when re-keying during a session, block other messages until the new keys
are activated (openssh doensn't like any other traffic during a rekey)
* re-key when outbound limits are tripped too (was only counting inbound
traffic)
* don't log scary things on EOF
add settimeout/gettimeout/setblocking, some bugfixes.
hide the command and response codes in sftp so they aren't exported.
add settimeout/gettimeout/setblocking that just wrap calls to the underlying
socket or channel. fix _read_all to not catch timeout exceptions.
add settimeout/gettimeout/setblocking, some bugfixes.
hide the command and response codes in sftp so they aren't exported.
add settimeout/gettimeout/setblocking that just wrap calls to the underlying
socket or channel. fix _read_all to not catch timeout exceptions.
limit read/write requests to 32KB, advertise 32KB max packet size
one of the unit tests was failing because the openssh sftp server was dropping
the connection without any error. turns out they have a maximum allowed write
size (possibly around 64KB). the sftp rfcs have a small hint that some servers
may drop read/write requests of greater than 32KB.
so, all reads are limited to 32KB, and all writes > 32KB are now chopped up
and sent in 32KB chunks. this seems to keep openssh happy.
also, we now advertise 32KB max packet size instead of 8KB (the speed
improves a lot), and log when we read/write a packet. and sftp files are
flushed on seek.
speed up parts of BufferedFile
BufferedFile uses cStringIO for the write buffer now (i don't actually notice
any speed difference so this might revert later) and the default buffer size
has been upped from 1KB to 8KB.
when scanning for linefeeds (when writing to a line-buffered file), only scan
the newly-written bytes, since we know all the previously buffered data is
linefeed-free. this was the #1 slowdown on the 1MB-file unit test.
also, limit the buffering on line-buffered files to whatever the default
buffer size is. there's no reason to buffer 1MB waiting for a linefeed.
some Channel fixes for max packet size & blocking on zero window
some clean-ups and fixes to channels:
* when send() is blocked on a zero-width window, check that the channel is
still open. this was causing some lockups.
* set a lower bound to the "maximum packet size" we accept from the remote
host. if they tell us anything less than 1KB, assume they meant 1KB. (it's
not reasonable to fragment below that.)
* leave a little padding instead of cutting right up to the maximum packet
size: some space will be taken up by protocol overhead.
* turn off some of the debug log lines unless "ultra_debug" is on (nobody
cares about the feed info)
add forward.py demo script; bump to gyarados
add a demo script to show how to do local port forwarding.
add gyarados to all the docs and bump the version number everywhere.
add direct-tcpip ability to open_channel
open_channel can now be given a dest_addr and src_addr, which are filled in
if the channel type is "forwarded-tcpip" or "direct-tcpip". these channel
types are used in remote & local port forwarding, respectively. i've only
tested "direct-tcpip" but i think if one works, they both should work.
also fixed a bug in connect where it was still assuming the old meaning for
get_remove_server_key() (oops!) and changed the sense of a send() failure
from <= 0 to < 0 since it may be possible for send() to return 0 and it not
be an EOF error.
fix deadlock in closing a channel
closing a channel would enter an odd codepath where the lock was grabbed,
some stuff was done, then another function was called where the lock was
grabbed again. unfortunately python locks aren't monitors so this would
deadlock. instead, make the smaller function lock-free with an explicit
notice that you must be holding the lock before calling.
fix utf8, raise packet size, log exceptions, be more lax with sfp servers
explicitly import utf8 encodings for "freezing" (and also because not all
platforms come with utf8, apparently). raise the max acceptable packet size
to 8kB, cuz 2kB was too low. log exceptions at error level instead of debug
level. and don't reject older sftp servers.
fearow date and last-minute fixes
update release date of fearow to 23apr. fix channel._set_closed() to grab
the lock before notifying the in/out buffers that the channel is closed.
try roger's trick for finding the home folder on windows.
Robey Pointer
Robey Pointer
Robey Pointer
Robey Pointer