the previous windows pipe fix still didn't work. replace it with a new pipe.py abstraction of pipes (one for posix, one for windows) which appears to finally work on windows. for real this time. also add some more documentation to Channel to explain that after exec_command, invoke_shell, or invoke_subsystem, a Channel can't be reused.
oooh maybe i'll test things before checking them in next time: rekeying was a little bit overzealous. now it's careful to only rekey once and reset the counters in sync
split out Packetizer, fix banner detection bug, new unit test
split out a chunk of BaseTransport into a Packetizer class, which handles
the in/out packet data, ciphers, etc. it didn't make the code any smaller
(transport.py is still close to 1500 lines, which is awful) but it did split
out a coherent chunk of functionality into a discrete unit.
in the process, fixed a bug that alain spineux pointed out: the banner
check was too forgiving and would block forever waiting for an SSH banner.
now it waits 5 seconds for the first line, and 2 seconds for each subsequent
line, before giving up.
added a unit test to test keepalive, since i wasn't sure that was still
working after pulling out Packetizer.
make SFTPClient.file an alias for SFTPClient.open.
clean up docs a little, and make 'file' an alias for 'open'.
this is how python is heading in general.
integrated laptop work (test commit)
Patches applied:
* robey@lag.net--2003-public-master-shake/secsh--dev--1.0--base-0
tag of robey@lag.net--2003-public/secsh--dev--1.0--patch-160
* robey@lag.net--2003-public-master-shake/secsh--dev--1.0--patch-1
test commit
* robey@lag.net--2003-public/secsh--dev--1.0--base-0
initial import
* robey@lag.net--2003-public/secsh--dev--1.0--patch-1
no changes
change SubsystemHandler/SFTPServerInterface API
change the API of SubsystemHandler to accept a reference to the
ServerInstance object during construction. this will break all code
that currently creates subsystem handlers (like sftp servers) -- sorry!
lots of little doc fixups (mostly indenting).
rewrite channel pipes to work on windows
the pipe system i was using for simulating an os-level FD (for select) was
retarded. i realized this week that i could just use a single byte in the
pipe to signal "data is ready" and not try to feed all incoming data thru
the pipe -- and then i don't have to try to make the pipe non-blocking (which
should make it work on windows). a lot of duplicate code got removed and now
it's all going thru the same code-path on read.
there's still a slight penalty on incoming feeds and calling 'recv' when a
pipe has been opened (by calling 'fileno'), but it's tiny.
removed a bunch of documentation and comments about things not working on
windows, since i think they probably do now.
fix sending of large sftp packet sizes
fix a bug where packets larger than about 12KB would cause the session to
die on platforms other than osx. turns out that on most platforms, setting a
socket timeout also causes timeouts to occur on writes (but not on osx). so
on a huge write, once the os buffers were full, paramiko would get a
socket.timeout exception when writing, and bail.
since the timeout is primarily so we can periodically poll to see if the
session has been killed from elsewhere, do that on a timeout but otherwise
continue trying to write. large packet sizes (in sftp) should now work.
tweak sftp_file write behavior on large blocks of data
BufferedFile.write() wasn't correctly dealing with the possibility that the
underlying write might not write the entire data block at once (even though
the docs said it would). now that it's working, make sftp_file take
advantage of it in order to chop up blocks larger than 32kB (the max allowed
on sftp) and add a unit test for it.
fix race in transport thread startup
set active=True from the methods that start the main transport thread, right
before actually starting the thread. this avoids a race where the main
thread could be started, but the original thread could wake up from the
event.wait(0.1) before the new thread actually set the transport active.
impossible, you say? no machines so slow exist? au contraire, my sad
little linux box faced this problem earlier today.
when combining stderr with stdout on a channel, merge the buffers too
when turning on combine-stderr mode on a channel, grab the channel lock and
feed any existing stderr buffer into the normal buffer. this should help
applications (and my unit tests) avoid races between data coming in over
stderr and setting combine-stderr.
_send_eof is now slightly safer too, although i don't think that really fixed
anything. it just makes me feel better.
add thread ids to logs
add a logging filter that reports the thread-id of the logger, and use
that for all paramiko logging. since thread-local stuff didn't appear
until python 2.4, i hacked up my own little version to assign incrementing
numbers to threads as they log.
add methods for sending/receiving a channel's exit status
track a channel's exit status and provide a method (recv_exit_status) to
block waiting for it to arrive. also provide a convenience method for
servers to send it (send_exit_status). add shutdown_read and shutdown_write.
fix a bug in sending window change requests.
misc logging fixes
change the level of some log messages so interesting stuff gets logged at
info instead of debug. fix an oops where channels defaulted to being in
ultra debug mode, and make this mode depend on a new Transport method:
"set_hexdump".
more flexible logging
some tweaks to make channels etc follow the logger setting of their parent
transport, so that setting the log channel for a paramiko transport will
cause all sub-logging to branch out from that channel.
also, close all open file handles when the sftp server ends.
added listdir_attr()
add SFTPClient.listdir_attr() to fetch a list of files & their attributes,
instead of just their filenames. artur piwko would find this useful.
cleanup & docs in sftp
add some more docs to SFTPHandle, and give a default implementation for
close() that's usually right. add a flush() to the default implementation
of write(). document that symlink's args in the sftp protocol are out of
order (the spec is wrong).
server support for stderr & exec_command
for the server side of my stderr blunder, add send_stderr & sendall_stderr,
and make the sending side of makefile_stderr work correctly.
also, call check_channel_exec_request on a server object for exec requests
on a channel.
add client-side multi-part auth support
added support for multi-part authentication (even though nobody supports it
that i've seen). on a successful "partial" auth, the auth_* method will
return a list of acceptable means to continue authenticating.
clean up authentication
add new exception "BadAuthenticationType", which is raised when auth fails
because your auth type (password or public-key) isn't valid on the server.
used this as an excuse to clean up auth_password and auth_publickey so their
'event' arg is optional, and if missing, they block until auth is finished,
raising an exception on error.
also, don't close the session on failed auth -- the server may let you try
again.
added some test cases for failed auth.
symlink, readlink
add support for symlink command, and finish support for readlink. (i guess
i started readlink a while ago but forgot to add the right method to the
SFTPServerInterface class.)
add stderr support methods
big embarrassment: i didn't read the ssh2 docs close enough, and all this
time paramiko wasn't handling "extended_data" packets, which contain stderr
output.
so now, several new functions: recv_stderr_ready() and recv_stderr() to
mirror recv_ready() and recv(), and set_combined_stderr() to force stderr
to be combined into stdout. also, makefile_stderr() to create a fake file
object to represent stderr.
fix SFTPFile gettimeout/settimeout
i don't think the gettimeout/settimeout calls on SFTPFile ever worked.
also, simplify the implementation of _get_size() since it's nearly
identical to stat().
sftp server support!
finally check in sftp_handle (file handle abstraction), sftp_si (server
interface), and sftp_server (server implementation) -- all of which make
a roughly 90% implementation of server-side sftp.
add finish_subsystem()
when a SubsystemHandler is being decomissioned (the client has closed the
channel or transport, or the socket went away), make a callback to let the
handler do any shutdown it needs to.
fix extremely unlikely channel counter wrapping
Transport's channel counter can overflow after 4 billion some channels are
created. make it wrap back around after 16 million instead. also allow the
logging channel to be set manually. fix some comments elsewhere.
fix Transport.get_username() to work in server mode too
whenever i split the 'username' field into username and auth_username,
i guess that made get_username() stop working for server mode (because the
username was stored in a different field). this should fix it.
don't unlink a Channel until the server closes it too
when close()'ing a Channel, don't immediately unlink it from the Transport.
instead, wait for the server to send a close message.
this should fix a bug where doing close() on an EOF'd channel would cause
the entire transport to be killed, because the server would send an
'exit-status' and 'close' message for a channel that we no longer had a
record of.
better debugging, improve subsytem handler
add a list of ssh packet names for debugging. improve the server-mode
subsystem handler so it can take extra parameters (list or keyword) and
pass them to the subsystem constructor. remove a misleading comment
about rekeying (which was already implemented).
switch Transport.connect() to using a Pkey object for the host key
i suddenly realized that passing "hostkeytype" and "hostkey" as strings to
Transport.connect() was pretty silly since i went to all the effort of making
a class specifically for holding keys. so Transport.connect() now just takes
host-key argument: "hostkey" as a PKey object.
updated the demos to use PKey objects when reading the host key file, and to
use the new "hostkey" argument.
fix __init__
fix __init__ to export BufferedFile and randpool, and to catch up with the
changes from a week or 2 ago where sftp_attr & friends were split off.
add Message.rewind()
add rewind() method to Message, which just resets the pointer so you can
start reading from the beginning again. this is useful for some tests.
clean up pkey interface
change the pkey interface so that it's no longer possible to have a pkey
that doesn't represent a valid key. (ie: no more "blank" key objects.)
also add "get_bits" and "can_sign" methods to determine the key bit length
and whether it can sign things (contains the "private parts") respectively.
add sftp.normalize
kevin c. dorff pointed out that it would be nice to expose a way to
determine the server's "current working directory", so this new method
(normalize) directly maps to REALPATH.
move SubsystemHandler to server.py
move SubsystemHandler into server.py where it makes more sense (it's part of
the server interface).
also fix up paramiko's "version string" used in ssh2 negotiation to stop
saying "pyssh" and start saying "paramiko". :)
Message.add() can take many args
a bit of cleanup to Message: add() can now take any number of params, and
will add them all in order (using type guessing).
fix rbuffer -> _rbuffer in 3 places i missed
fix 3 places where "rbuffer" hadn't been converted to "_rbuffer". thanks to
kevin c. dorff for the bug report.
add sftp_client.py
i retardedly forgot to import this file a few days ago: it's the split-out
client mode for sftp. it now also has some changes to adapt it to the
improved SFTPAttributes object API.
clean up SFTPAttributes
add english descriptions to the FX_* error codes of sftp. clean up (and
document) SFTPAttributes since it's exported now, and make it simple to
generate one from a python os.stat object. make "_pythonize" the default --
that is, just use the same field names as python does for os.stat. (i'm not
sure why i didn't do it that way in the first place; probably ignorance.)
also add str() method that converts the SFTPAttributes into a string suitable
for use in ls (used in an obscure way in sftp servers).
split sftp into sftp, sftp_client; renamed SFTP -> SFTPClient
add sftp_client file, and split out the common code (sftp) from stuff specific
to client mode (sftp_client). renamed SFTP class to SFTPClient, but left an
alias so old code will still work.
renamed a bunch of sftp constants now that they're better hidden from epydoc.
some framework for adding subsystem handlers in server mode
you can now register a subsystem with a Transport by passing in the name
(like "sftp") and a class (like a hypothetical SFTPServer). the default
ServerInterface.check_channel_request_subsystem now checks this table in
Transport, and if it finds a match, it creates a new thread for the handler
and calls into it. a new class SubsystemHandler is added for this purpose
(to be subclassed).
remove redundant 'auth_complete' member
remove the redundant 'auth_complete' field and just use 'authenticated' for
both client and server mode. this makes the repr() string look correct in
server mode instead of always claiming that the transport is un-auth'd.
clean up server interface; no longer need to subclass Channel
- export AUTH_*, OPEN_FAILED_*, and the new OPEN_SUCCEEDED into the paramiko
namespace instead of making people dig into paramiko.Transport.AUTH_* etc.
- move all of the check_* methods from Channel to ServerInterface so apps
don't need to subclass Channel anymore just to run an ssh server
- ServerInterface.check_channel_request() returns an error code now, not a
new Channel object
- fix demo_server.py to follow all these changes
- fix a bunch of places where i used "string" in docstrings but meant "str"
- added Channel.get_id()
clean up SecurityOptions
the preferences are now tuples in Transport, and passed as tuples out of
SecurityOptions, so that the options can't be modified without setting them
back to the options field again. the algorithm lists in Transport are used
to validate the fields.
added Transport.get_security_options()
just something i wanted to play with:
added Transport.get_security_options() which returns a SecurityOptions object.
this object is a kind of proxy for the 4 "preferred_*" fields in Transport,
and lets me avoid exposing those fields directly in case i change my mind
later about how they should be stored.
added some docs to Channel explaining that the request methods now return
True/False, and fixed up docs in a few other places.
replay patch 63 (missing channel changes)
i'm still getting the hang of tla/arch, obviously.
replay patch 63, which was meant to be part of the later mega-patch, but
apparently when i reversed it, i lost it entirely.
new ServerInterface class, outbound rekey works, etc.
a bunch of changes that i'm too lazy to split out into individual patches:
* all the server overrides from transport.py have been moved into a separate
class ServerInterface, so server code doesn't have to subclass the whole
paramiko library
* updated demo_server to subclass ServerInterface
* when re-keying during a session, block other messages until the new keys
are activated (openssh doensn't like any other traffic during a rekey)
* re-key when outbound limits are tripped too (was only counting inbound
traffic)
* don't log scary things on EOF
add settimeout/gettimeout/setblocking, some bugfixes.
hide the command and response codes in sftp so they aren't exported.
add settimeout/gettimeout/setblocking that just wrap calls to the underlying
socket or channel. fix _read_all to not catch timeout exceptions.
add settimeout/gettimeout/setblocking, some bugfixes.
hide the command and response codes in sftp so they aren't exported.
add settimeout/gettimeout/setblocking that just wrap calls to the underlying
socket or channel. fix _read_all to not catch timeout exceptions.
limit read/write requests to 32KB, advertise 32KB max packet size
one of the unit tests was failing because the openssh sftp server was dropping
the connection without any error. turns out they have a maximum allowed write
size (possibly around 64KB). the sftp rfcs have a small hint that some servers
may drop read/write requests of greater than 32KB.
so, all reads are limited to 32KB, and all writes > 32KB are now chopped up
and sent in 32KB chunks. this seems to keep openssh happy.
also, we now advertise 32KB max packet size instead of 8KB (the speed
improves a lot), and log when we read/write a packet. and sftp files are
flushed on seek.
speed up parts of BufferedFile
BufferedFile uses cStringIO for the write buffer now (i don't actually notice
any speed difference so this might revert later) and the default buffer size
has been upped from 1KB to 8KB.
when scanning for linefeeds (when writing to a line-buffered file), only scan
the newly-written bytes, since we know all the previously buffered data is
linefeed-free. this was the #1 slowdown on the 1MB-file unit test.
also, limit the buffering on line-buffered files to whatever the default
buffer size is. there's no reason to buffer 1MB waiting for a linefeed.
some Channel fixes for max packet size & blocking on zero window
some clean-ups and fixes to channels:
* when send() is blocked on a zero-width window, check that the channel is
still open. this was causing some lockups.
* set a lower bound to the "maximum packet size" we accept from the remote
host. if they tell us anything less than 1KB, assume they meant 1KB. (it's
not reasonable to fragment below that.)
* leave a little padding instead of cutting right up to the maximum packet
size: some space will be taken up by protocol overhead.
* turn off some of the debug log lines unless "ultra_debug" is on (nobody
cares about the feed info)
add forward.py demo script; bump to gyarados
add a demo script to show how to do local port forwarding.
add gyarados to all the docs and bump the version number everywhere.
add direct-tcpip ability to open_channel
open_channel can now be given a dest_addr and src_addr, which are filled in
if the channel type is "forwarded-tcpip" or "direct-tcpip". these channel
types are used in remote & local port forwarding, respectively. i've only
tested "direct-tcpip" but i think if one works, they both should work.
also fixed a bug in connect where it was still assuming the old meaning for
get_remove_server_key() (oops!) and changed the sense of a send() failure
from <= 0 to < 0 since it may be possible for send() to return 0 and it not
be an EOF error.
fix deadlock in closing a channel
closing a channel would enter an odd codepath where the lock was grabbed,
some stuff was done, then another function was called where the lock was
grabbed again. unfortunately python locks aren't monitors so this would
deadlock. instead, make the smaller function lock-free with an explicit
notice that you must be holding the lock before calling.
fix utf8, raise packet size, log exceptions, be more lax with sfp servers
explicitly import utf8 encodings for "freezing" (and also because not all
platforms come with utf8, apparently). raise the max acceptable packet size
to 8kB, cuz 2kB was too low. log exceptions at error level instead of debug
level. and don't reject older sftp servers.
fearow date and last-minute fixes
update release date of fearow to 23apr. fix channel._set_closed() to grab
the lock before notifying the in/out buffers that the channel is closed.
try roger's trick for finding the home folder on windows.
add set_keepalive()
add set_keepalive() to set an automatic keepalive mechanism. (while waiting
for a packet on a connection, we periodically check if it's time to send a
keepalive packet.)
add get_username() method for remembering who you auth'd as
add get_username() method for remembering who you auth'd as. also, fix these
bugs:
* "continue" auth response counted as a failure (in server mode).
* try to import 'logging' in py22 before falling back to the fake logger,
in case they have a backported version of 'logger'
* raise the right exception when told to read a private key from a file that
isn't a private key file
* tell channels to close when the transport dies
fix encrypted private key files
the random byte padding on private key files' BER data was confusing openssh,
so switch to null-byte padding, which is slightly less secure but works with
crappy old openssh. also, enforce the mode when writing the private key
file. we really really want it to be 0600. (python seems to ignore the
mode normally.)
support py22, more or less
add roger binns' patches for supporting python 2.2. i hedged a bit on the
logging stuff and just added some trickery to let logging be stubbed out for
python 2.2. this changed a lot of import statements but i managed to avoid
hacking at any of the existing logging.
socket timeouts are required for the threads to notice when they've been
deactivated. worked around it by using the 'select' module on py22.
also fixed the sftp unit tests to cope with a password-protected private key.
make get_remote_server_key() return a PKey object
a good suggestion from roger binns: make get_remote_server_key() just return
a pkey object instead of a tuple of strings. all the strings can be extracted
from the pkey object, as well as other potentially useful things.
add dss key generation too, and fix some bugs
added the ability to generate dss keys and write private dss key files,
similar to rsa. in the process, fixed a couple of bugs with ber encoding
and writing password-encrypted key files. the key has to be padded to the
iblock size of the cipher -- it's very difficult to determine how the others
do this, so i just add random bytes to the end.
fixed the simple demo to use Transport's (host, port) constructor for
simplicity, and fixed a bug where the standard demo's DSS login wouldn't
work.
also, move the common logfile setup crap into util so all the demos can just
call that one.
add global request mechanism
add transport.global_request() to make a global-style request (usually an
extension to the protocol -- like keepalives) and handle requests from the
remote host. incoming requests are now handled and responded to correctly,
which should make openssh-style keepalives work. (before, we would silently
ignore them, which was wrong.)
can now generate rsa keys (not dss yet)
added functionality to ber to create ber streams. added some common methods
to PKey to allow dumping the key to base64 (the format used by openssh for
public key files and host key lists), and a factory for creating a key from
a private key file, and a common way to save private keys. RSAKey luckily
didn't have to change that much.
also added a factory method to RSAKey to generate a new key.
fix some arcana in unpacking private keys
"!= type([])" is a pretty obscure way to say it. let's try "is not list"
which is a lot more readable.
(mostly this is a test to make sure tla is working okay on my laptop.)
finish up client sftp support
added 'stat' to SFTPFile and SFTP, documented 'open' and 'listdir', and added
'rmdir', 'lstat', 'symlink', 'chmod', 'chown', 'utime', 'readlink'.
turned off ultra debugging now that the unit tests are all working.
fix some docs and BufferedFile.readline
fix some documentation and fix readline()'s universal newline support to
always return strings ending with '\n', regardless of how they were in the
original file. (this is an obvious feature of python's universal newline
support that i somehow missed before.)
fix lingering thread bug
this bug has been in there forever and i could never figure out a workaround
till now.
when the python interpreter exits, it doesn't necessarily destroy the
remaining objects or call __del__ on anything, and it will lock up until all
threads finish running. how the threads are supposed to notice the exiting
interpreter has always been sort of a mystery to me.
tonight i figured out how to use the 'atexit' module to register a handler
that runs when the interpreter exits. now we keep a list of active threads
and ask them all to exit on shutdown. no more going to another shell to
kill -9 python! yeah!!
add BufferedFile abstraction
SFTP client mode is mostly functional. there are probably still some bugs
but most of the operations on "file" objects have survived my simple tests.
BufferedFile wraps a simpler stream in something that looks like a python
file (and can even handle seeking if the stream underneath supports it).
it's meant to be subclassed. most of it is ripped out of what used to be
ChannelFile so i can reuse it for sftp -- ChannelFile is now tiny.
SFTP and Message are now exported.
fixed util.format_binary_line to not quote spaces.
Transport constructor can take hostname or address tuple
part of an ongoing attempt to make "simple" versions of some of the API calls,
so you can do common-case operations with just a few calls:
Transport's constructor will now let you pass in a string or tuple instead
of a socket-like object. if you pass in a string, it assumes the string is
a hostname (with optional ":port" segment) and turns that into an address
tuple. if you pass in a tuple, it assumes it's an address tuple. in both
cases, it then creates a socket, connects to the given address, and then
continues as if that was the socket passed in.
the idea being that you can call Transport('example.com') and it will do
the right thing.
pkey no longer raises binascii.Error
catch binascii.Error in the private key decoder and convert it into an
SSHException. there's no reason people should have to care that it was a
decoding error vs. any of the other million things that could be wrong in
a corrupt key file.
document more of Message; add get_int64
all of the get_* methods are now documented, but there's a bit more to do.
get_int64 added for eventual sftp support.
fix MANIFEST.in, change version numbers to 0.9-doduo, fix LPGL notices
fixed MANIFEST.in to include the demo scripts, LICENSE, and ChangeLog.
upped everything to version 0.9-doduo.
fixed the copyright notice, and added the LGPL banner to the top of every
python file.
more docs, and password-protected key files can now be read
lots more documentation, some of it moved out of the README file, which is
now much smaller and less rambling.
repr(Transport) now reports the number of bits used in the cipher.
cleaned up BER to use util functions, and throw a proper exception (the new
BERException) on error. it doesn't ever have to be a full BER decoder, but
it can at least comb its hair and tuck in its shirt.
lots of stuff added to PKey.read_private_key_file so it can try to decode
password-protected key files. right now it only understands "DES-EDE3-CBC"
format, but this is the only format i've seen openssh make so far. if the
key is password-protected, but no password was given, a new exception
(PasswordRequiredException) is raised so an outer layer can ask for a password
and try again.
added public-key support to server mode, more docs
added public-key support to server mode (it can now verify a client signature)
and added a demo of that to the demo_server.py script (user_rsa_key). in the
process, cleaned up the API of PKey so that now it only has to know about
signing and verifying ssh2 blobs, and can be hashed and compared with other
keys (comparing & hashing only the public parts of the key). keys can also
be created from strings now too.
some more documentation and hiding private methods.
lots more documentation, and added Transport.connect()
renamed demo_host_key to demo_rsa_key. moved changelog to a separate file,
and indicated that future changelog entries should be fetched from tla.
tried to clean up "__all__" in a way that makes epydoc still work.
added lots more documentation, and renamed many methods and vars to hide
them as private non-exported API.
Transport's ModulusPack is now a static member, so it only has to be loaded
once, and can then be used by any future Transport object.
added Transport.connect(), which tries to wrap all the SSH2 negotiation and
authentication into one method. you should be able to create a Transport,
call connect(), and then create channels.
hook up server-side kex-gex; add more documentation
group-exchange kex should work now on the server side. it will only be
advertised if a "moduli" file has been loaded (see the -gasp- docs) so we
don't spend hours (literally. hours.) computing primes. some of the logic
was previously wrong, too, since it had never been tested.
fixed repr() string for Transport/BaseTransport. moved is_authenticated to
Transport where it belongs.
added lots of documentation (but still only about 10% documented). lots of
methods were made private finally.
fix up new paramiko/ folder.
moved SSHException to a new file (ssh_exception.py) and turned paramiko.py
into an __init__.py file. i'm still not entirely sure how this normally
works, so i may have done something wrong, but it's supposed to work the
same as before.
Robey Pointer