2010-03-25 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.5.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2010-03-25 Brian Warner <warner@lothar.com> * foolscap/banana.py, broker.py, copyable.py, pb.py * foolscap/appserver/cli.py, foolscap/logging/gatherer.py * foolscap/test/test_call.py, test_logging.py, test_observer.py, * test_promise.py, test_reference.py, test_registration.py, * test_schema.py, test_tub.py: clean up lots of pyflakes warnings, revealed by the new more-strict version of pyflakes * foolscap/banana.py (Banana.dataReceived): apply zooko's patch to use stringchain on the inbound data path, to fix the O(n^2) performance in large tokens (e.g. a 10MB string token). Thanks Zooko! Closes #149. * foolscap/test/bench_banana.py: also zooko's performance tests * foolscap/stringchain.py: Zooko's utility class to efficiently handle large strings split into several pieces, such as the inbound socket buffers that Banana.dataReceived() winds up with. * foolscap/test/test_stringchain.py: unit tests for the same. Zooko has given me permission to distribute both of these under Foolscap's MIT license. 2010-03-14 Brian Warner <warner@lothar.com> * foolscap/constraint.py: remove maxSize/maxDepth methods, and the related UnboundedSchema exception. As described in ticket #127, I'm giving up on resource-exhaustion defenses, which allows for a lot of code simplification. * foolscap/{copyable.py|removeinterface.py|schema.py}: same * foolscap/slicers/*.py: same * foolscap/test/test_schema.py: remove tests * doc/jobs.txt: remove TODO items around maxSize * foolscap/_version.py: bump version while between releases * misc/*/debian/changelog: same 2010-01-18 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.5.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2010-01-18 Brian Warner <warner@lothar.com> * NEWS: update for upcoming release 2010-01-11 Brian Warner <warner@lothar.com> * foolscap/logging/web.py (WebViewer.start): add --open, which uses stdlib's webbrowser.open() function to automatically open a browser window with the "flogtool web-viewer" contents. * foolscap/pb.py (Tub.getReferenceForName): for unknown names, only include the first two letters of the swissnum in the KeyError we return, to avoid revealing the whole thing in the caller's error logs. Closes #133. * foolscap/test/test_pb.py (TestCallable.testWrongSwiss): test it * foolscap/test/test_gifts.py (Bad.test_swissnum): same (Bad.testReturn_swissnum): same 2009-12-28 Brian Warner <warner@lothar.com> * doc/flappserver.xhtml: general improvements * foolscap/appserver/cli.py (Create.run): chmod the basedir go-rwx to keep the private key hidden from other users * foolscap/test/test_appserver.py (CLI.test_create): test it * doc/flappserver.xhtml: mention the chmod * foolscap/appserver/cli.py (Start.run): use the handy run() function from twisted.scripts.twistd instead of os.execvp to give control to twistd. This removes a lot of grossness and should remove the need to modify $PATH when running "bin/flappserver start" from an uninstalled source tree. * foolscap/logging/dumper.py (LogDumper.run): don't eat IOErrors when you point it at a missing file. Only eat the EPIPE that occurs when you run "flogtool dump FILE |less" and quit the pager early. * foolscap/logging/dumper.py: add --timestamps, which can be "short-local", "long-local" (ISO-8601 with timezone), or "utc" (ISO-8601 with "Z" suffix). Closes #100. * foolscap/logging/web.py: add --timestamps, "local" or "utc". This controls the default format. Added tooltips over the time strings to show the event time in local, iso-8601, and iso-8601-utc. Added links to the all-events page to reload it with various timestamp formats and sort options. * foolscap/test/test_logging.py (Dumper): update tests to match (Web): same 2009-10-04 Brian Warner <warner@lothar.com> * foolscap/slicers/set.py: suppress the DeprecationWarning that occurs when you import 'sets' on py2.6: seems like the best way to preserve compatibility with py2.3-compatible application code while still avoiding the noise when running with 2.6. Closes #124. * foolscap/appserver/client.py (Uploader.run): expanduser() the filename, so you can use 'flappclient upload-file ~/foo.txt' even without the shell expanding the filename for you (ClientOptions.read_furlfile): same, closes #134. 2009-09-17 Brian Warner <warner@lothar.com> * setup.py: use entry_points= on windows, to create executable .bat files for flogtool/flappserver/flappclient . Stick to scripts= on non-windows (and remember to install with --single-version-externally-managed) to get non-opaque flogtool scripts that don't use magic pkg_resources functions to import foolscap. This should close #109. 2009-07-30 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.setLocationAutomatically): fix comment * foolscap/test/test_serialize.py: oops, fix pyflakes warnings * foolscap/test/test_reference.py: same * foolscap/test/test_gifts.py: same 2009-07-28 Brian Warner <warner@lothar.com> * foolscap/test/*: stop using old-style shouldFail (which would mask errors) 2009-06-24 Brian Warner <warner@lothar.com> * doc/listings/xfer-client.py, xfer-server.py, command-client.py, command-server.py: remove, obsoleted by flappserver * foolscap/pb.py (getRemoteURL_TCP): remove this function, I never liked it, and it wasn't very good anyway. * foolscap/api.py, foolscap/__init__/py, foolscap/deprecated.py: same * foolscap/test/test_pb.py: remove dead code * foolscap/deprecated.py: define wrappers that emit DeprecationWarning upon use, then wrap them around everything that used to be in foolscap/__init__.py . * foolscap/__init__.py: apply the wrappers. Please import from foolscap.api instead. See #122 for details. * foolscap/*: stop importing functions/classes directly from the top-level 'foolscap' module, instead import them from the 'foolscap.api' module (i.e. 'from foolscap.api import Tub' instead of 'from foolscap import Tub'). I'm planning to add deprecation wrappers around the things you can get from 'foolscap', and this is to make sure our own code doesn't run afoul of those warnings. 2009-06-22 Brian Warner <warner@lothar.com> * foolscap/ipb.py (DeadReferenceError): improve the string form by adding remote tubid, and information from the PendingRequest. Since DeadReferenceError frequently happens in an eventually() context, there's not usually a useful stack trace, so any piece of information will help. * foolscap/referenceable.py (RemoteReference._callRemote): give interfacename+methodname to PendingRequest, so it can give that information to DeadReferenceError.__str__ . Some day it'd be cool to include a stack trace here. * foolscap/call.py (PendingRequest.__init__): store it (PendingRequest.getMethodNameInfo): return it * foolscap/broker.py (Broker.abandonAllRequests): pass PendingRequest to DeadReferenceError * foolscap/test/test_call.py (TestCall.test_connection_lost_is_deadref): test it * foolscap/test/test_pb.py: update to match, remove dead code * doc/examples/git-remote-add-furl: fix shbang line * doc/examples/git-clone-furl: same 2009-06-20 Brian Warner <warner@lothar.com> * doc/examples/git-clone-furl: easier tool to set up FURL-based git repo * doc/examples/git-remote-add-furl: same * doc/examples/git-publish-with-furl: better name for git-setup-flappserver-repo * doc/examples/git-proxy-flappclient: mention those scripts * foolscap/appserver/client.py: improve --help strings * foolscap/appserver/cli.py (CreateOptions.opt_umask): capture the current umask at "flappserver create" time for use by the eventual server. This should reduce surprises. * doc/flappserver.xhtml: explain the new behavior * foolscap/appserver/server.py (AppServer): require the umask file to contain an octal string, don't use eval() * foolscap/test/test_appserver.py: update tests 2009-06-19 Brian Warner <warner@lothar.com> * doc/flappserver.xhtml: add --umask= to "flappserver create", since twisted forces it to 077 on startup, and the git-proxy-flappclient system was winding up with overly restrictive filemodes * foolscap/appserver/cli.py (CreateOptions): same * foolscap/appserver/server.py (AppServer): same * foolscap/test/test_appserver.py (CLI.test_create2): test it * doc/examples/git-setup-flappserver-repo: suggest --umask 2009-06-18 Brian Warner <warner@lothar.com> * doc/examples/git-proxy-flappclient: * doc/examples/git-setup-flappserver-repo: new pair of tools to run the Git remote-access protocol over a flappserver-based connection * MANIFEST.in: include them in the source tarball * foolscap/_version.py: bump version between releases * misc/*/debian/changelog: same 2009-06-16 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.4.2 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-06-16 Brian Warner <warner@lothar.com> * foolscap/test/test_gifts.py: remove all timeouts.. I think one buildslave took slightly too long, which flunked the test, and then screwed up every test after that point. * setup.py (packages): add foolscap/appserver * NEWS: update for upcoming release * setup.py (setup_args): include flappclient as a script * foolscap/negotiate.py (TubConnector.connect): if we have no location hints, set self.failureReason so we can error out cleanly. Closes #129. * foolscap/tokens.py (NoLocationHintsError): new exception for this purpose * foolscap/negotiate.py (TubConnector.failed): call tub.connectorFinished() too, so any pending Tub.stopService can be retired. * foolscap/pb.py (Tub.getReference): wrap in a maybeDeferred, so that unparseable FURLs give us errbacks instead of synchronous exceptions. This is the other half of #129. (Tub.connectorFinished): stop complaining (at log.WEIRD) if the connector wasn't in our table. I don't know what was causing this, and I don't really care. Closes #81. * foolscap/test/test_tub.py (BadLocationFURL): test #129 stuff * foolscap/pb.py (Tub.getReference): always return a Deferred, even if given an unparseable FURL. Addresses #129. * foolscap/test/test_tub.py (NoLocationFURL): test it * foolscap/appserver/client.py (UploadFile): accept multiple files * foolscap/test/test_appserver.py (Upload.test_run): test it * doc/flappserver.xhtml: update example to match * doc/flappserver.xhtml: rename tools: use "upload-file" and "run-command" instead of file-uploader and exe * doc/foolscap/appserver/client.py: same * doc/foolscap/appserver/services.py: same * foolscap/test/test_appserver.py: same * foolscap/test/test_appserver.py (RunCommand): improve coverage * foolscap/appserver/cli.py: move flappclient code out of cli.py .. * foolscap/appserver/client.py (run_flappclient): .. into client.py (Exec.dataReceived): don't spin up StandardIO until after the server tells us they want stdin. This also means we can stop buffering stdin. * foolscap/test/test_appserver.py: match client.run_flappclient change (RunCommand.test_run): first test of run-command/exec code * foolscap/broker.py (Broker.doNextCall): don't run any calls after we've been shut down. This fixes a racy bug in which two calls arrive and get queued in the same input hunk, the first one provokes a Tub.stopService, and the second one gets run after the Tub has been shutdown, leaving an "Unhandled Error in Deferred" lying around. This was occasionally flunking an appserver test. * foolscap/test/test_tub.py (CancelPendingDeliveries): test it * foolscap/appserver/cli.py (Restart): add 'flappserver restart' * foolscap/appserver/services.py (Exec): make run-command work, including optional stdin/stdout/stderr, logging of each, and return of exitcode. Still needs unit tests and a better name. * foolscap/appserver/client.py (Exec): same (parse_options): pass a 'stdio' object in separately, to let unit tests control stdin 2009-06-15 Brian Warner <warner@lothar.com> * foolscap/appserver/services.py (FileUploader.remote_putfile): use os.chmod, since Twisted-2.5 doesn't have FilePath.chmod * bin/flappclient: generic client for flappserver * foolscap/appserver/client.py: same * foolscap/test/test_appserver.py (Upload.test_run): test file uploading (Client.test_help): test --help and --version * foolscap/appserver/*: fixes to make file-upload work * foolscap/appserver/services.py (FileUploader): implement it for real, using twisted's FilePath. Not sure it actually works yet. * foolscap/test/test_appserver.py: begin unit tests for appserver * foolscap/appserver/*: clean up Create, add AppServer.when_ready() to handle setLocationAutomatically() better, display simple UsageError/BadServiceArguments exceptions better (no traceback) 2009-06-14 Brian Warner <warner@lothar.com> * foolscap/appserver/server.py (AppServer.__init__): improve startup message * foolscap/appserver/services.py (FileUploader.__init__): note that tub= might be None (such as in 'add' when we're merely validating the service arguments) * foolscap/appserver/cli.py: run Tub for 'create', but not for 'add' or 'list'. Stash the furl-prefix to support this. The idea will be to rewrite furl-prefix each time the server is started. This also allows 'add' and 'list' to be used while the server is already running. * doc/flappserver.xhtml: add example transcript * foolscap/pb.py (generateSwissnumber): pull this out to a seperate function, to be called externally (Tub.generateSwissnumber): same * doc/flappserver.xhtml: start to create an "application server": an easy tool to deploy pre-written foolscap-based services in a shared server process. This first step creates the bin/flappserver tool, and implements the "create", "add", "list", "start", and "stop" commands. The next step will be to write some basic tests, then implement some built-in services (file-upload and exec-command), then write some clients to drive those services. Eventually new services will be loaded with a plugin architecture. Tracked in ticket #82. * bin/flappserver: this tool is used to manipulate app servers * foolscap/appserver/*.py: the actual code * setup.py: bin/flappserver is a script too 2009-06-05 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (RemoteReferenceOnly.getLocationHints): new method (RemoteReferenceOnly.getSturdyRef): remove "this is not secure" comment: getSturdyRef now *does* check the tubid against the connection (SturdyRef): factor out encode_furl/decode_furl into new functions * foolscap/test/test_pb.py (TestReferenceable): test both getLocationHints() and isConnected() (TestCallable.testGetSturdyRef): same * foolscap/test/test_gifts.py: update to use encode_furl 2009-06-03 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (RemoteReferenceOnly.isConnected): new method, tests the same thing as notifyOnDisconnect but this one is immediate instead of callback-based * foolscap/test/test_call.py (TestCall.testNotifyOnDisconnect): test it * foolscap/test/test_call.py (ReferenceCounting.test_reference_counting): oops, turn off debugging noise which was accidentally left in back at 0.4.0 * foolscap/referenceable.py (TheirReferenceUnslicer): remove the 200-byte limit on FURLs which are passed as third-party "gifts". (ReferenceUnslicer): same, for inbound my-references * foolscap/_version.py: bump version between releases * misc/*/debian/changelog: same 2009-05-22 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.4.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-05-22 Brian Warner <warner@lothar.com> * NEWS: udpate for upcoming release * foolscap/tokens.py (RemoteException): override __str__, since we don't put arguments in .args . Otherwise, attempting to stringify a RemoteException will fail under python2.4 (but not 2.5). * foolscap/test/test_logging.py (Basic.testFailure): test it * foolscap/test/test_call.py (ExamineFailuresMixin._examine_raise): more thorough test * foolscap/_version.py: bump version between releases 2009-05-19 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.4.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2009-05-19 Brian Warner <warner@lothar.com> * NEWS: update for the next release * README: mention python2.6 compatibility * foolscap/test/test__versions.py (Versions.test_record): record versions of platform/python/twisted/foolscap in the test logs * foolscap/api.py: add some more symbols: ChoiceOf, IntegerConstraint * foolscap/banana.py (Banana.handleData): increment the inbound objectCounter as soon as we see the OPEN token, rather than doing it in handleOpen(), since the latter is skipped if we're dropping tokens due to a Violation. This avoids the loss-of-sync that occurred when the dropped tokens included more OPEN sequences (e.g. when a remote method call used a bad method name, and its arguments contained lists or tuples or other sequences). This fixes the bug in which subsequent method calls that used shared arguments saw their "reference" sequences point to the wrong value. Closes #104. * foolscap/test/test_call.py (ReferenceCounting): test it * foolscap/referenceable.py (RemoteReferenceTracker.__init__): require the proposed per-my-reference FURL to match the TubID that we're using for the parent Broker. This prevents the far end from sending a bogus FURL and thus breaking the security properties of rref.getSturdyRef(). (they couldn't confuse Tub.getReference, because the spoofed FURL would only be added to the per-Broker table, but they could confuse someone who relied upon the correctness of getSturdyRef). This closes #84. (RemoteReferenceTracker.__init__): oops, don't compare tubids when using an UnauthenticatedTub, since that fails the non-crypto tests * foolscap/test/test_reference.py (TubID): test it * foolscap/test/test_schema.py (Interfaces.test_remotereference): update to match * foolscap/hashutil.py: new file to smoothly use hashlib when available and fall back to sha1/md5 when not (i.e. python2.4). When we eventually drop py2.4 support, we'll get rid of this file. Thanks to Stephan Peijnik for the patch. Closes #118. * foolscap/sslverify.py: use hashutil instead of the md5 module * foolscap/vocab.py (hashVocabTable): use hashutil instead of the sha module * foolscap/schema.py (PolyConstraint.checkToken): remove unused code, thanks to Stephan Peijnik for the patch. * foolscap/test/test_banana.py (VocabTest1.test_table_hashes): make sure we don't change the pre-agreed vocab tables, or the hash algorithm that we use to confirm their contents. This is in preparation for the conditional switch to 'hashlib', from Stephan's patch in ticket #118 2009-05-18 Stephan Peijnik <stephan@peijnik.at> * foolscap/call.py, foolscap/constraint.py, foolscap/eventual.py, foolscap/observer.py, foolscap/promise.py, foolscap/reconnector.py, foolscap/remoteinterface.py, foolscap/slicer.py: Convert all old-style classes to new-style classes. Should close #96. * foolscap/reconnector.py, foolscap/referenceable.py, foolscap/remoteinterface.py: Import each module with a separate import statement. 2009-05-18 Brian Warner <warner@lothar.com> * foolscap/test/test_tub.py (TestCertFile.test_tubid): confirm that we get the TubID right when using a pre-recorded cert. If we were to accidentally change the TubID-computing hash function in the future, this test should catch it. (CERT_DATA): stash the contents of a pre-recorded cert * bin/flogtool: let 'base' default to abs("."), to help windows and systems where somebody has managed to copy it to /tmp/flogtool or /sbin/flogtool . Closes #108. 2009-05-13 Brian Warner <warner@lothar.com> * foolscap/test/test_call.py (ExamineFailuresMixin): rearrange code to try and fix a weird py2.4 failure * foolscap/api.py (RemoteException): hush pyflakes 2009-05-12 Brian Warner <warner@lothar.com> * foolscap/logging/dumper.py: emit PID and embedded versions, except when using --just-numbers or --verbose. Closes #97. * foolscap/logging/web.py (Welcome.render): same * foolscap/logging/log.py (LogFileObserver): store app_versions in the log file created by $FLOGFILE * foolscap/test/test_logging.py: test it all * foolscap/tokens.py (RemoteException): add a Tub option that causes remote exceptions to be reported with a special exception type, named foolscap.api.RemoteException, so that they won't be confused with locally-generated exceptions. Closes #105. * foolscap/api.py: make RemoteException available through api.py * doc/failures.xhtml: explain the new option * foolscap/pb.py (Tub.setOption): handle the new expose-remote-exception-types option, default remains True to retain the old behavior * foolscap/broker.py (Broker.setTub): copy the flag from the Tub into the Broker * foolscap/call.py (ErrorUnslicer.receiveClose): wrap exceptions if the mode says we should * foolscap/test/test_call.py (Failures): test RemoteException wrapping (TubFailures): test setOption() too * foolscap/test/common.py (ShouldFailMixin.shouldFail.done): return the Failure for further testing, but wrapped in an array to avoid triggering the errback 2009-05-11 Brian Warner <warner@lothar.com> * Makefile (LORE): parameterize "lore" binary * doc/stylesheet.css: update to new version from Lore * doc/schema.xhtml: fix mis-nested H3 header, to appease new Lore 2009-05-04 Brian Warner <warner@lothar.com> * foolscap/api.py: new preferred entry point. Application code should import symbols from "foolscap.api" instead of directly from foolscap/__init__.py . Importing from the __init__.py will be deprecated in the next major release, and removed in the subsequent one. Addresses #122 (but it won't be closed until we remove everything from __init__.py). * foolscap/__init__.py: add comment about preferring foolscap.api * doc/using-foolscap.xhtml: update examples to use e.g. "from foolscap.api import Tub" * doc/listings/*.py: update examples * foolscap/pb.py (Tub.setLogGathererFURL): allow users to call both Tub.setOption("log-gatherer-furl") and Tub.setOption("log-gatherer-furlfile") on the same Tub. Previously this was disallowed. Also avoid making multiple connections to the same gatherer. Closes #114. * foolscap/test/test_logging.py (Gatherer.test_log_gatherer_furlfile_multiple): test it * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup): don't use an absolute delay, since it causes spurious test failures on slow systems. * foolscap/logging/web.py (WebViewer.start): put the welcome page at the root URL, in addition to /welcome . This simplifies the starting URL. Closes #120. (Reload.render_POST): make the "Reload Logfile" button point at the new URL too. * foolscap/logging/web.py (Welcome.render): display PID of logfile, if available. This will be present in incident records that were written out by the original process or gathered by a log gatherer, in the files written by "flogtool tail --save-to=", and in the file written by creating an explicit FileLogObserver (such as done by setting the FLOGFILE environment variable). This should close #80. * foolscap/logging/log.py (LogFileObserver): record PID in header * foolscap/test/test_logging.py (Web.test_basic): test PID * foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal): use Decimal("Inf") instead of decimal.Inf, to unbreak python2.6.2 which privatized both Inf and Nan. Thanks to ivank for the patch. Closes #121. 2008-10-15 Brian Warner <warner@allmydata.com> * misc/*/debian/rules (install/python-foolscap): include misc/classify_foolscap.py in the debian package, uncompressed, so it goes into /usr/share/doc/python-foolscap/classify_foolscap.py , so you can make a symlink to it from an incident-gatherer. * misc/classify_foolscap.py (TUBCON_RE): update to match foolscap-tubconnector messages for both old and new versions of Foolscap, and for python2.4 and 2.5 . * foolscap/logging/incident.py (IncidentClassifier): add --verbose option to 'flogtool classify-incident' to show the trigger dictionary for any unclassifiable incidents. This is useful when developing classification functions. * foolscap/test/test_logging.py (Incidents.test_classify): test it * Makefile (test-figleaf): oops, fix the test-figleaf-poll target, by making the test-figleaf target use $(TRIAL), not hardcoded 'trial'. * foolscap/_version.py: bump revision to 0.3.2+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-10-14 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.3.2 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * MANIFEST.in: add misc/classify_foolscap.py to source distribution * NEWS: update for the next release * Makefile (test-poll): add a convenience target, for running tests on a system with the broken combination of pyopenssl and twisted that requires the pollreactor (test-figleaf-poll): same 2008-10-14 Brian Warner <warner@allmydata.com> * foolscap/logging/tail.py (LogTail): include the retrieved PID in the saved logfile. Part of #80. * foolscap/logging/incident.py (IncidentReporter.incident_declared): also include the PID in the incident report's header. * doc/specifications/logfiles.xhtml: document the ['versions'] and ['pid'] fields in incident reports and the 'flogtool tail' savefile. * foolscap/logging/web.py: mark any incident triggers in the logfile, and add links to them from the welcome page. Closes #79. * foolscap/logging/web.py (Reload): add a 'Reload Logfile' button to the web-viewer's Welcome page. Not automatic, but it means you don't have to get back to the shell and restart the viewer. Works well-enough to say Closes #103. * misc/classify_foolscap.py: plugin to classify some foolscap-internal incidents * foolscap/logging/cli.py: new "flogtool classify-incident" subcommand: given an incident, say what categories it falls into. Closes #102. * foolscap/logging/gatherer.py (IncidentGathererService): factor out the classification pieces into IncidentClassifierBase * foolscap/logging/incident.py (IncidentClassifierBase): same (IncidentClassifier.run): support for the new CLI command * foolscap/test/test_logging.py (Incidents.test_classify): test it * foolscap/logging/gatherer.py (IncidentGathererService.classify_incident): change the classifier function signature: now it just takes a single 'trigger' dict. * foolscap/test/test_logging.py: update to match * doc/logging.xhtml (gatherer): same * foolscap/logging/gatherer.py (IncidentGathererService.add_classify_files): make the incident gatherer look in its base directory for "classify_*.py" files, use them as plugins with classification functions. (INCIDENT_GATHERER_TACFILE): update example text to match * foolscap/test/test_logging.py (IncidentGatherer): test it (Gatherer.test_wrongdir): exercise another error case * doc/logging.xhtml (gatherer): document it 2008-10-13 Brian Warner <warner@allmydata.com> * foolscap/logging/publish.py (IncidentSubscription.catch_up): Don't tell the gatherer about incidents that don't have triggers: these are malformed logfiles, such as the zero-length files that result from a process being terminated before it manages to write anything. * foolscap/logging/log.py (Count): replace itertools.count with a version that doesn't overflow at 2**31-1 (thanks to Zooko for the patch). Closes #99. (FoolscapLogger.__init__): use it * foolscap/pb.py (Listener.startFactory): add log facility identifiers (Listener.stopFactory): same (Listener.buildProtocol): same (Tub.getReference): same 2008-09-20 Brian Warner <warner@lothar.com> * doc/listings/xfer-client.py (_failure): do sys.exit(1) upon failure, so the caller can detect it via the exit code 2008-09-10 Brian Warner <warner@allmydata.com> * foolscap/test/test__versions.py (Versions.test_required): check for bug #62 (openssl>0.7, twisted<=8.1.0, selectreactor) and print a warning if it is likely that the tests would fail, to remind the user to re-run with -r poll. (Versions.test_required): oops, guard import of OpenSSL on having crypto available, otherwise this test gets an error when OpenSSL is not installed. 2008-09-08 Brian Warner <warner@lothar.com> * doc/listings/xfer-client.py: use os.path.expanduser() on the filenames passed in by the user, to enable things like --furlfile ~/.upload.furl * foolscap/_version.py: bump revision to 0.3.1+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-09-03 Brian Warner <warner@lothar.com> * foolscap/_version.py: release Foolscap-0.3.1 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * NEWS: update for next release 2008-09-02 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (TubConnector.__repr__): add more information to the repr, to help track down #81. Made TubConnector a new-style class in the process, to make upcalling easier. (TubConnector.checkForIdle): improve the log message * foolscap/pb.py (Tub._log_gatherer_connected): use callRemoteOnly to pass the logport to the gatherer: we don't need to hear about any problems it has. * foolscap/logging/gatherer.py (IncidentGathererService.classify_stored_incidents): reclassify everything that isn't already present in one of the classified/* files. This makes it a lot easier to iterate over the [start gatherer; see what is unknown; update classifiers; remove classified/unknown; repeat] loop. Also log classification events better. Closes #94. * foolscap/test/test_logging.py (IncidentGatherer.test_emit): test it * foolscap/test/common.py (PollMixin): replace the use of chained Deferreds with a task.LoopingCall-based version, from Tahoe. This avoids the weird and annoying maximum-recursion-depth-exceeded error that occurs when the check function is called more than about 600 times. Closes #95. 2008-08-29 Brian Warner <warner@lothar.com> * foolscap/logging/app_versions.py (versions): move the application version dict (which gets reported to remote subscribers, and copied into logfiles) to a separate module. It was causing circular import problems when it lived in an attribute of LogPublisher. (add_version): provide a setter method * foolscap/logging/publish.py (LogPublisher.versions): same (LogPublisher.remote_get_versions): same * foolscap/logging/incident.py (IncidentReporter.incident_declared): same * foolscap/test/test_logging.py (IncidentGatherer.test_emit): improve test shutdown a little bit 2008-08-28 Brian Warner <warner@lothar.com> * foolscap/logging/incident.py (IncidentReporter.incident_declared): put versions in the incident header too, also for #80. * foolscap/test/test_logging.py (Incidents.test_basic): test it * foolscap/logging/tail.py (LogPrinter.got_versions): record remote application versions to the --save-to file, in the header. Part of #80. (LogPrinter.got_versions): oops, control where stdout goes * foolscap/logging/dumper.py (LogDumper.start): show those versions with --verbose * foolscap/test/test_logging.py (Dumper.test_dump): update to match (Tail.test_logprinter): same, make sure got_versions is called * foolscap/logging/gatherer.py (IncidentObserver): only fetch one incident at a time, to limit the size of the sender's outbound queue. This should help close #85. (IncidentGathererService.new_incident): include the classification results in the per-incident log messages * foolscap/test/test_logging.py (IncidentGatherer.test_emit): tests * foolscap/logging/log.py (FoolscapLogger.declare_incident): combine overlapping incidents, by passing new triggers to an existing reporter instead of creating a new one. Helps with #85. * foolscap/logging/incident.py (IncidentReporter.new_trigger): new method to add subsequent triggers to an existing reporter. This doesn't do anything yet. It should be improved to record the other triggers in a trailer (since it's too late to add it to the header). Every triggering event will make it into an incident somewhere, but the report-file analysis tools may not know how to pay attention to the subsequent triggers. * foolscap/logging/interfaces.py (IIncidentReporter.new_trigger): same * foolscap/test/test_logging.py (Incidents.test_overlapping): test it * doc/logging.xhtml: mention incident-coalescing * foolscap/logging/log.py: add bridge-foolscap-logs-to-twisted functionality, set up by either calling bridgeLogsToTwisted(), or by setting the FLOGTOTWISTED environment variable (to anything). The default filter will exclude events below the OPERATIONAL severity level, and those generated by foolscap internals (i.e. facility.startswith("foolscap") ). Closes #93. * foolscap/pb.py (Tub.setOption): update to match * foolscap/test/test_logging.py (Bridge): tests for it (Publish): update to use new APIs * doc/logging.xhtml: docs 2008-08-25 Brian Warner <warner@lothar.com> * foolscap/broker.py (Broker.abandonAllRequests): map both ConnectionLost and ConnectionDone into DeadReferenceError, so that application code only needs to check for one exception type. * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): when an existing connection is dropped in favor of a new one, drop it with DeadReferenceError instead of ConnectionDone. The mapping in Broker.abandonAllRequests doesn't seem to quite catch everything in unit tests. (Negotiation.acceptDecisionVersion1): same, when we're the slave * foolscap/test/test_call.py (TestCall.test_connection_lost_is_deadref): test it (TestCall.test_connection_done_is_deadref): same (TestCall.testChoiceOf): switch to use ShouldFailMixin * foolscap/test/test_gifts.py (Gifts): remove ignoreConnectionDone, just look for DeadReferenceError 2008-08-21 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): bump revision to 0.3.0+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-08-04 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): release Foolscap-0.3.0 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: 2008-08-04 Brian Warner <warner@lothar.com> * foolscap/logging/interfaces.py (RILogPublisher.subscribe_to_incidents): small edit, make it more clear that since= is used for catch_up=True * NEWS: update for the upcoming release * foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID): make this secure, by using the broker's .remote_tubref field, instead of the remote-side-controlled sturdyref. (RemoteReferenceOnly.getSturdyRef): add a note about the insecurity of this method * foolscap/test/test_pb.py (TestCallable.testGetSturdyRef): add a test for getRemoteTubID * doc/logging.xhtml: change filenames in the incident-gatherer to have fewer files starting with incident*, so tab-completion works better. * foolscap/logging/gatherer.py (IncidentGathererService): same (create_incident_gatherer): same * foolscap/test/test_logging.py (Filter): test 'flogtool filter' * foolscap/logging/filter.py: control stdout/stderr better * foolscap/test/test_logging.py (Dumper): test 'flogtool dump' * foolscap/logging/cli.py (dispatch): pass stdout/stderr through attributes on the subcommand's Options instance, rather than as separate function arguments.. makes testing easier. * foolscap/logging/gatherer.py (create_log_gatherer): same (create_incident_gatherer): same (CreateGatherOptions): same (CreateIncidentGatherOptions): same * foolscap/logging/dumper.py (DumpOptions): same 2008-08-01 Brian Warner <warner@lothar.com> * foolscap/logging/log.py (LogFileObserver): make these more useful, by not doing the addSystemEventTrigger in __init__ (LogFileObserver.stop_on_shutdown): do it here instead (FLOGFILE): when using $FLOGFILE, add call to stop_on_shutdown * foolscap/logging/gatherer.py (GathererService.do_rotate): oops, implement the precautions claimed by the comment in startService: test self._savefile before doing anything. (GathererService.__init__): set self._savefile to None * foolscap/test/test_logging.py (Gatherer.test_log_gatherer2): add a timed rotator, which caught the problem in do_rotate * foolscap/test/test_logging.py (IncidentGatherer.test_emit): add test coverage for re-classifying existing incidents (IncidentGatherer.test_emit._update_classifiers_again): and verify that leaving the classified/ directory in place properly inhibits reclassification at startup * foolscap/referenceable.py (RemoteReferenceOnly.getRemoteTubID): new method to extract the remote tubid from a RemoteReference. I'm not sure this is a good idea, but it fixes the immediate problem I'm dealing with. * foolscap/logging/gatherer.py: use "tubid_s" instead of "nodeid_s" (IncidentGathererService.remote_logport): use getRemoteTubID() instead of trying to sanitize the tubid we receive, since we use it as a directory name (IncidentGathererService.add_classifier): rename addClassifier to add_classifier, I'm more fond of the latter form these days. Also remove a pyflakes warning. * foolscap/test/test_logging.py (IncidentGatherer.test_emit): add test of incident generation, publish, recording, and default classification * foolscap/logging/gatherer.py (IncidentGathererService): get control over stdout, so we can exercise more code during tests (IncidentGathererService.startService): oops, this needs to be startService instead of start * foolscap/test/test_logging.py (IncidentGatherer): basic test of an incident gatherer, just setup and connection so far * foolscap/test/common.py (StallMixin): factor stall() out into a separate class (TargetMixin): same * foolscap/test/test_logging.py (LogfileReaderMixin): refactor (Gatherer.test_log_gatherer): turn on bzip=True, for more coverage * foolscap/test/test_logging.py (Gatherer._check_gatherer): ignore internal foolscap messages (like connection negotiation), since they occur at unpredictable times (specifically in test_log_gatherer_furlfile_multiple, which has establishes multiple connections) * foolscap/test/common.py (GoodEnoughTub): factor this and crypto_available out of all the other unit tests, make GoodEnoughTub(certFile=) work even if crypto is unavailable and we must therefore discard the certFile= argument. * foolscap/test/test_crypto.py: same * foolscap/test/test_gifts.py: same * foolscap/test/test_keepalive.py: same * foolscap/test/test_logging.py: same * foolscap/test/test_loopback.py: same * foolscap/test/test_negotiate.py: same * foolscap/test/test_pb.py: same * foolscap/test/test_serialize.py: same * foolscap/test/test_tub.py: same * foolscap/logging/gatherer.py (GatheringBase): rewrite Gatherers to make them easier to test: now they are a Service (with a subordinate Tub), meant to be run by a .tac file or manually. The intermediate class has been removed. All .tac files are unchanged: gatherers created by old versions of 'flogtool create-gatherer' will continue to work. (GathererService.do_rotate): return the name of the logfile that was just closed and/or compressed, so tests can know where to look. * foolscap/test/test_logging.py (Gatherer): update to match * foolscap/util.py (get_local_ip_for): update reactor comment 2008-07-30 Brian Warner <warner@lothar.com> * setup.py: update comment about bug #62 (pyopenssl problems) * foolscap/test/test_logging.py (CLI.test_create_gatherer): improve test coverage a little bit, by recording stdout (CLI.test_create_incident_gatherer): exercise the 'flogtool create-incident-gatherer' command * foolscap/logging/cli.py (run_flogtool): capture stdout+stderr (dispatch): same * foolscap/logging/gatherer.py (create_log_gatherer): same (create_incident_gatherer): same * trees/tahoe/Makefile (figleaf-output): exclude foolscap/test/ from the HTML results * doc/logging.xhtml (Running an Incident Gatherer): describe the Incident Gatherer, like the Log Gatherer but it only gathers incidents. It also does classification, and will eventually do reporting. No unit tests yet, but some manual system-level tests have been run. * foolscap/logging/gatherer.py (IncidentGatherer): implement it * foolscap/logging/cli.py (Options.subCommands): add the CLI command, named 'flogtool create-incident-gatherer' * foolscap/logging/incident.py (IncidentReporter.incident_declared): clean up incident naming * foolscap/logging/interfaces.py (RILogPublisher.list_incidents): same * foolscap/logging/log.py (FoolscapLogger.incident_recorded): same * foolscap/logging/publish.py (LogPublisher.list_incident_names): same * foolscap/test/test_logging.py (IncidentPublisher.test_list_incident_names): test the incident-naming cleanup 2008-07-29 Brian Warner <warner@lothar.com> * foolscap/logging/gatherer.py (GatheringBase): refactor the log gatherer to share code with the upcoming incident gatherer (LogGatherer.__init__): add a basedir= argument, rather than using os.getcwd * foolscap/test/test_logging.py (Gatherer): add basedir= argument * doc/logging.xhtml (Setting up the logport): New feature (well, it didn't work before, and now it does, so make it explicit): the log-gatherer FURL can be configured (but will not be connected) until after setLocation. This should resolve a crash I've seen in Tahoe (which runs a slow /sbin/ifconfig command to figure out the addresses to pass to setLocation) in which the app connects to the log gatherer before it figures out its own location, and then gets an exception during registerReference. Closes #55. * foolscap/pb.py (Tub._maybeConnectToGatherer): don't initiate the gatherer connection until locationHints is set (Tub.setLocation): call _maybeConnectToGatherer after the location is set. Also, don't let setLocation be called multiple times. * foolscap/test/test_logging.py (Publish.test_logport_furlfile2): test it (Gatherer.test_log_gatherer2): same (Gatherer.test_log_gatherer_furlfile2): same * doc/logging.xhtml (Setting up the logport): slight API restriction: the logport and its FURL are not available until after Tub.setLocation is called. This results in a better error message than the usual one inside registerReference. * foolscap/pb.py (Tub.getLogPort): enforce the API restriction by throwing an exception when it is violated (Tub.getLogPortFURL): same * foolscap/tokens.py (NoLocationError): new exception for it * foolscap/test/test_tub.py (SetLocation.test_set_location): test it * doc/logging.xhtml (Configuring a Log Gatherer): allow multiple log-gatherer furls in the log-gatherer-furlfile * foolscap/pb.py (Tub._maybeConnectToGatherer): same * foolscap/logging/gatherer.py (LogGatherer.remote_logport): return the subscribe_to_all Deferred, for testing * foolscap/test/test_logging.py (Gatherer.test_log_gatherer): refactor, to accomodate new test (Gatherer.test_log_gatherer_furlfile_multiple): test it 2008-07-28 Brian Warner <warner@lothar.com> * foolscap/logging/interfaces.py (RILogPublisher.list_incidents): change signature to remove tubid/incarnation from the response. (RILogPublisher.subscribe_to_incidents): add pubsub interface for incidents, including catch_up= and since= (RILogObserver.new_incident): same (RILogObserver.done_with_incident_catchup): same * foolscap/logging/publish.py (IncidentSubscription): same * foolscap/logging/incident.py (IncidentReporter.finished_recording): tell the logger about the incident name, so it can publish it. * foolscap/logging/log.py (FoolscapLogger.addImmediateIncidentObserver): same (FoolscapLogger.incident_recorded): same * foolscap/test/test_logging.py (IncidentPublisher._check_listed): same (IncidentPublisher.test_subscribe): test it * foolscap/logging/interfaces.py (RISubscription.unsubscribe): move the RILogPublisher.unsubscribe() method to the RISubscription object, since that's a better place for it. (RILogPublisher.unsubscribe): deprecate this one * foolscap/logging/publish.py (Subscription.remote_unsubscribe): same * foolscap/logging/publish.py (LogPublisher.remote_get_incident): reject invalid incident names * doc/logging.xhtml: add details about running a Log Gatherer (Python 'logging' module): comment out this section, it is wrong 2008-07-09 Brian Warner <warner@lothar.com> * doc/specifications/logfiles.xhtml: fix typos * doc/logging.xhtml (Remote log aggregation): same 2008-07-07 Brian Warner <warner@lothar.com> * foolscap/logging/dumper.py (LogDumper): when dumping an Incident Report, mark the triggering event with "[INCIDENT-TRIGGER]" * foolscap/_version.py (verstr): bump revision to 0.2.9+ while between releases * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same 2008-07-02 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): release Foolscap-0.2.9 * misc/{dapper|edgy|feisty|gutsy|hardy|sarge|etch|sid}/debian/changelog: same * NEWS: update for the upcoming release * MANIFEST.in: add gutsy/hardy directories 2008-07-02 Brian Warner <warner@allmydata.com> * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): change poller condition to avoid spurious failures on slow systems * misc/*/debian/watch: update to point at foolscap.lothar.com * Makefile (debian-gutsy, debian-hardy): add .deb targets for gutsy and hardy. The rules are the same as for feisty. Closes #76. * misc/gutsy/*, misc/hardy/*: same * foolscap/logging/publish.py (Subscription.send): change the discard policy to discard-new instead of discard-random: in tests with a busy Tahoe node, this seems to give good enough behavior (probably since the busyness is bursty), although I can't say we fully understand what's really going on. discard-new lets us use a faster and simpler deque instead of requiring random access to the list. * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): update to match 2008-07-02 Brian Warner <warner@lothar.com> * foolscap/logging/publish.py (Subscription.send): add a size-limited queue for sending messages to subscribers, with a default queue size of 2000 entries. This should probably keep the memory footprint bounded to perhaps 1MB. When the queue gets full, we randomly discard old messages, so recent messages are more likely to survive than earlier ones. We allow 10 messages to be outstanding on the wire at once, to pipeline them a bit and improve network utilization. Any errors during sending will cause the subscription to be dropped. This should close #72. * foolscap/logging/log.py (FoolscapLogger.addImmediateObserver): add a new kind of observer, so that the logport publisher can also avoid unboundedness in the eventual-send queue. If we can't throw away messages fast enough, callers to log.msg will block, slowing down the inlet rate. * foolscap/test/test_logging.py (Publish.test_logpublisher_overload): test it, by throwing 10k messages at log.msg and counting how many make it through. * foolscap/test/common.py (PollMixin): add a comment: this chained-Deferred pattern will run up against python's recursion limit if the check function is called more than about 300 times. 2008-07-01 Brian Warner <warner@lothar.com> * foolscap/logging/publish.py (Subscription): refactor a bit, in preparation for #72 limit log-publishing queue size 2008-07-01 Brian Warner <warner@allmydata.com> * doc/logging.xhtml ("That Was Weird" Buttons): provide a simple example of giving the user a way to trigger Incident logging. Closes #75. * foolscap/logging/interfaces.py (RILogPublisher.get_pid): new interface, to retrieve the process ID through the logport * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup._got_logport._check_pid): test it * foolscap/logging/publish.py (LogPublisher.remote_get_pid): implement it * foolscap/logging/tail.py (LogTail._got_logpublisher._announce): Use it, but tolerate old logports that don't offer it. Closes #71. * foolscap/slicers/decimal_slicer.py: handle Decimal objects, serializing them as a string. No constraints yet. Closes #50. * foolscap/slicers/allslicers.py: import decimal_slicer * foolscap/test/test_banana.py (ThereAndBackAgain.test_decimal): test it * foolscap/logging/tail.py (LogTail._got_logpublisher): exit if we get an error while connecting or subscribing. Closes #63. (LogTail._print_versions): print remote versions immediately after connecting to the logport. Closes #70. * foolscap/logging/log.py (FoolscapLogger.setLogDir): oops, allow the incident directory to be re-used: os.makedirs throws an exception if the directory already exists. * foolscap/test/test_logging.py (Incidents.test_basic): test it * foolscap/_version.py (verstr): bump revision to 0.2.8+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-06-04 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): release Foolscap-0.2.8 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for the upcoming release 2008-06-04 Brian Warner <warner@allmydata.com> * foolscap/pb.py (Tub._tubsAreNotRestartable): fix args so that these methods actually get run (and produce a useful error message, instead of TypeError). Thanks to Brian Granger for the patch. Closes #65. * foolscap/test/test_tub.py (Shutdown.test_doublestop): test it * foolscap/test/common.py (ShouldFailMixin.shouldFail): add some support code * foolscap/referenceable.py (RemoteReferenceOnly.getPeer): new method to get the IP address and port number of the other end of a connection. This returns a twisted.internet.interfaces.IAddress provider. Loopback connections give foolscap.broker.LoopbackAddress instances. Real remote connections give twisted.internet.address.IPv4Address instances, so you can use rref.getPeer().host and rref.getPeer().port on them. Closes #45. * setup.py: add an "extras_require" clause to the setuptools-specific setup args, to declare that our "secure_connections" feature requires pyOpenSSL. This helps other packages, which can declare a dependency on "Foolscap[secure_connections]", rather than claiming to require pyOpenSSL themselves. Addresses #66. * foolscap/pb.py (Tub.registerReference): fix an exception that occurs if you call this with both name= and furlFile= and the furlFile already exists. Also prohibit attempts to change the name to something other than what is in the furlFile. Thanks to Brian Granger for the patch. Closes #64. * foolscap/_version.py (verstr): bump revision to 0.2.7+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-05-13 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): release Foolscap-0.2.7 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for new "OMG flogtool is broken" release. 2008-05-12 Brian Warner <warner@allmydata.com> * foolscap/logging/cli.py (run_flogtool): fix use of sys.argv, the previous version was completely broken * foolscap/_version.py (verstr): bump revision to 0.2.6+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-05-06 Brian Warner <warner@lothar.com> * foolscap/_version.py (verstr): release Foolscap-0.2.6 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * NEWS: update for the upcoming release 2008-05-05 Brian Warner <warner@allmydata.com> * foolscap/test/test_logging.py (Web): add tests for 'flogtool web-viewer' * foolscap/logging/web.py (WebViewer.start): make this class more amenable to being tested * foolscap/logging/interfaces.py: misc cleanups (RILogPublisher.list_incidents): provide an interface to retrieve stored incident reports from a logport. (RILogPublisher.get_incident): same * foolscap/logging/publish.py (LogPublisher): same * foolscap/test/test_logging.py (IncidentPublisher): test it * foolscap/pb.py (Tub.setup): make the Tub's logger a bit easier to override, for tests * foolscap/logging/incident.py (IncidentReporter): make the compressed logfile use a temporary name until we've closed it, so later observers don't get tricked into using an incomplete compressed logfile (they should use the more-complete uncompressed logfile instead). (IncidentReporter.incident_declared): make it easier to turn off the gather-trailing-events behavior (NonTrailingIncidentReporter): convenience subclass that does that * doc/logging.xhtml: update to match current reality, remove some TODO warnings * foolscap/logging/log.py (FoolscapLogger): rename setIncidentReporterClass to setIncidentReporterFactory * foolscap/test/test_logging.py (Incidents.test_customize): same * foolscap/logging/interfaces.py (RISubscription): remove spurious 'pass', figleaf thought it was real code * bin/flogtool: split out the CLI dispatcher to.. * foolscap/logging/cli.py (run_flogtool): here, to help with #51 (run_flogtool): make it possible to run with a wrapper, closes #51. * foolscap/test/test_logging.py (CLI.test_wrapper): test a wrapper, provide an example of how to build one * foolscap/logging/gatherer.py (CreateGatherOptions.optFlags): add a --quiet flag to make the unit test less noisy * misc/testutils/trial_figleaf.py: make this compatible with twisted-8.0.x: the earlier version didn't write out any coverage data on newer twisteds * foolscap/logging/log.py (FoolscapLogger.setLogDir): wire up Incident handling. We still need incident publishing. This gets us most of the way to #61. (setIncidentQualifier): this customizes the qualification function (setIncidentReporterClass): this customizes the reporter/recorder * foolscap/logging/levels.py: move log.WEIRD and friends here to avoid circular import problems elsewhere. They are still available from log.py . * foolscap/logging/incident.py (IncidentReporter.incident_declared): flush the uncompressed logfile just before we switch into gather-trailing-events mode (IncidentReporter.stop_recording): oops, fix typo * foolscap/test/test_logging.py (Incidents): test most incident-handling functionality 2008-05-02 Brian Warner <warner@allmydata.com> * foolscap/logging/incident.py: start to implement Incident handling, for ticket #61. Not complete yet, might be completely broken, needs tests and to be wired up. * foolscap/logging/log.py (FoolscapLogger.setIncidentQualifier): start to add the new interfaces, not complete yetxo * foolscap/logging/interfaces.py (IIncidentReporter): same * doc/logging.xhtml (Incidents): document the new features 2008-05-01 Brian Warner <warner@allmydata.com> * doc/logging.xhtml: fix discussion of format=, since the instructions and examples were simply wrong * doc/specifications/logfiles.xhtml (Logfile Headers): define "headers", a separate dictionary at the start of the logfile that contains metadata. The specific use for this will be the "Triggering Event" that gets put into incident reports, once we implement those. This induces a backwards compatibility break: logfiles produced after this change will probably not be tolerated by tools like 'flogtool dump' from before this point. The other direction is ok: newer tools can handle either format. * foolscap/logging/gatherer.py (LogGatherer._open_savefile): emit a header, with type="gatherer", and a starting timestamp in "start". Refactor a bit to make this easier. * foolscap/logging/tail.py (LogPrinter): emit a header, with type="tail" * foolscap/logging/log.py (LogFileObserver): emit a header, with type="log-file-observer", and a "threshold" key * foolscap/logging/dumper.py (LogDumper.start): tolerate headers * foolscap/logging/filter.py (Filter.run): same * foolscap/logging/web.py (WebViewer.process_logfiles): same * foolscap/test/test_logging.py: update tests to match * doc/specifications/logfiles.xhtml: document the current saved logfile format (event dictionaries, pickled wrapper dicts) * foolscap/referenceable.py (SturdyRef): accept multiple connection hints in unauthenticated FURLs. This fixes a test failure induced by the #60 changes when pyOpenSSL is unavailable. * doc/using-foolscap.xhtml: document the new feature. * foolscap/referenceable.py (SturdyRef): tolerate extensions in tubid and location-hints fields, change FURL parsing, store structured hints in the SturdyRef instead of just strings. This should give us some wiggle room in the future to gracefully transition applications to using new features while retaining backwards compatibilty. Many thanks to Zooko for the suggestion. Closes #60. * foolscap/negotiate.py (TubConnector): same * foolscap/test/test_sturdyref.py (URL.testTubIDExtensions): new tests for it (URL.testLocationHintExtensions): same * foolscap/test/test_tub.py (SetLocation.test_set_location): update to match * foolscap/test/test_gifts.py (Bad): same * foolscap/test/test_negotiate.py (Versus.testVersusHTTPServerAuthenticated): same * foolscap/base32.py (is_base32): new utility function * foolscap/test/test_util.py (Base32.test_is_base32): test it 2008-04-22 Brian Warner <warner@lothar.com> * doc/listings/command-client.py: don't emit extra newlines, use /usr/bin/env on shbang line * doc/listings/command-server.py: use /usr/bin/env on shbang line 2008-04-22 Brian Warner <warner@allmydata.com> * foolscap/logging/filter.py: add "--above UNUSUAL" option (to discard events below the given level", and "--from [TUBID]" (to discard events that weren't recorded by the given tubid). * foolscap/logging/dumper.py (LogDumper.print_event): use --rx-time to show both event-generation time and event-receive time in the logs, useful if you suspect the application is getting bogged down and events are being delivered slowly. (LogDumper.print_event): Use a different format to display failure tracebacks. * foolscap/logging/web.py (WebViewer.get_events): catch ValueError, mention it as a possible truncated pickle file * foolscap/logging/log.py (FoolscapLogger.msg): if something goes wrong, print both str() and repr() in case it helps figure out the problem * foolscap/logging/dumper.py (LogDumper.print_event): include failure tracebacks in 'flogtool dump' output 2008-04-09 Brian Warner <warner@allmydata.com> * foolscap/constraint.py (Constraint.checkOpentype): always accept ('reference',) sequences. The per-token constraint checking system is a defense against resource-exhaustion attacks, and shared reference don't consume any more memory or stack frames than any other object. The check-all-args that CallUnslicer does just before delivering the arguments is responsible for making sure the final (resolved) types all match the constraint. By allowing shared references here, we fix a bug in which a schema violation was raised when python combined two equivalent tuples into a single object. * foolscap/test/test_call.py (TestCall.testMega3): test it * foolscap/test/common.py (MegaSchema3): same * setup.py: finally remove zip_safe=False * foolscap/test/test_interfaces.py (TestInterface.testStack): allow the test to pass even if the source code isn't available (i.e. it's locked away inside an egg). This ought to remove the need for zip_safe=False. * foolscap/test/test_logging.py (Tail.test_logprinter): make the test pass on python2.5: I think twisted now emits type(exception) instead of str(exception), or something: the printed form changed. * foolscap/schema.py (PolyConstraint): improve error messages * foolscap/test/test_logging.py (Tail.test_logprinter): improve error messages, to figure out why this fails on python2.5 * foolscap/__init__.py (_unused): add __version__, to hush pyflakes * foolscap/test/test_logging.py (Publish.test_logpublisher): make the test work with twisted-8.0.x, which puts more information in log.err Failures than earlier versions did. * foolscap/__init__.py (__version__): move version from here.. * foolscap/_version.py (verstr): .. to here * foolscap/setup.py: merge zooko's patch to get foolscap version by scanning foolscap/_version.py instead of importing foolscap. This makes setuptools a lot happier: it doesn't need to have Twisted installed (so foolscap can be imported) while it's building foolscap as part of automatic dependency satisfaction. Also change the shebang line to use /usr/bin/env, and import (but do not use) setuptools if it is available. We set 'zip_safe=False' in the extra setuptools arguments because two of the unit tests assert that their stack traces have source code lines in them, and that doesn't happen if foolscap is living inside an egg. We will probably change those tests soon and allow zipped eggs. 2008-03-31 Brian Warner <warner@allmydata.com> * foolscap/test/test_logging.py (Tail): basic tests for logging.tail * foolscap/logging/tail.py (LogPrinter.remote_msg): parameterize the output filehandle so we can test it * foolscap/logging/gatherer.py (LogGatherer): remove the "Z" suffix from the from- and to- timestamps that we put into filenames, since we're actually using localtime. If and when we switch to use UTC, we'll bring the Z back. Also remove the hyphens between the time portions: from-2008-03-31-161721 instead of from-2008-03-31-16-17-21. I'm still looking for a format that feels readable, scannable, and clear. * foolscap/logging/tail.py (LogPrinter.formatted_print): remove spurious "0" from timestamps: i.e. print "08:09:12.345" instead of "08:09:120.345" * foolscap/logging/web.py (LogEvent.to_html): same * foolscap/__init__.py: bump revision to 0.2.5+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-03-25 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.2.5 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-03-25 Brian Warner <warner@lothar.com> * MANIFEST.in: add LICENSE * NEWS: update for the upcoming release * bin/flogtool (Options.synopsis): update with all commands. There's still something broken with --help output, though. * doc/listings/command-server.py: * doc/listings/command-client.py: new sample programs, a client/server pair which lets the client trigger a specific command to be run on the server. Like xfer-server.py, but for running commands instead of transferring files. * foolscap/test/common.py (PollMixin): refactor a bit, to extract the poll() method for use by other tests * foolscap/test/test_logging.py (Publish.test_logpublisher): improve the timing a bit, by waiting until the observer has heard only silence for a full second, instead of starting the verify pass one second after subscribing. (Publish.test_logpublisher_catchup): same. It makes a bigger difference here, because catch_up=True means that we'll be seeing several hundred messages, which may take a non-trivial amount of time to receive. I was seeing intermittent test failures with the one-second-from-subscribe stall. 2008-03-24 Brian Warner <warner@lothar.com> * foolscap/logging/interfaces.py (RILogPublisher.subscribe_to_all): add a catch_up= argument, which causes the publisher to dump all its stored messages just after adding the subscriber. Closes #49. * foolscap/logging/log.py (FoolscapLogger.get_buffered_events): same (LogFileObserver.msg): catch+print exceptions while pickling events * foolscap/logging/publish.py (LogPublisher): add catch_up=, make sure that any log events arrive after subscribe_to_all() has returned, to make event sequencing easier on the subscriber. Also make sure that any catch-up events arrive before subsequent log events. * foolscap/logging/tail.py: add --catch-up option, make sure we remain compatible with <=0.2.4 publishers (as long as you don't use --catch-up) * foolscap/test/test_logging.py (Publish.test_logpublisher_catchup): test it (Publish.test_logpublisher_catchup): make test more reliable * LICENSE: make it clear that Foolscap ships under the MIT license, the same as Twisted uses. Closes #47. * README: same * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/copyright: same 2008-03-24 Brian Warner <warner@allmydata.com> * foolscap/logging/dumper.py (LogDumper.print_event): event dicts store the printable tubid, not a binary form * foolscap/logging/gatherer.py (Observer.__init__): same (LogGatherer.remote_logport): same * foolscap/logging/interfaces.py (TubID): same * foolscap/logging/web.py (LogEvent.__init__): same * foolscap/logging/gatherer.py (LogGatherer.__init__): make bzip= argument optional, for unit tests * foolscap/logging/tail.py: fix --save-to, also put exception information into the twistd.log when we can't pickle the event * foolscap/logging/gatherer.py: add --rotate and --bzip options to 'flogtool create-gatherer': rotate the logfile every N seconds, and optionally compress the results. Each logfile gets named like from-2008-03-24-20-31-46Z--to-2008-03-24-20-31-56Z.flog.bz2 , and the open one is named from-2008-03-24-20-32-16Z--to-present.flog . Sending SIGHUP to the gatherer will force a rotation. Closes #48. * foolscap/logging/tail.py: add --save-to option to 'flogtool tail', which saves the log events to a file (in addition to printing them to stdout). 'flogtool dump' can be used on the saved file later. Also refactor things a bit to let us grab the tubid from the target furl, since this gets recorded in the save file format. * foolscap/logging/gatherer.py (LogSaver): move to tail.py * foolscap/logging/tail.py (LogPrinter): print formatted event lines by default. Use the new --verbose option to dump raw event dictionaries. Closes #43. * bin/flogtool (dispatch): same 2008-02-17 Brian Warner <warner@lothar.com> * doc/using-foolscap.xhtml: fix href, thanks to Stephen Waterbury for the catch. Note that the .xhtml points to .xhtml, and lore converts the target of the link to .html in the .html output. * doc/copyable.xhtml: same * foolscap/logging/gatherer.py: make 'flogtool create-gatherer' to build a .tac file (which can be launched with twistd) rather than starting a gatherer right away. This is much more useable in practice. * bin/flogtool: same 2008-01-31 Brian Warner <warner@allmydata.com> * foolscap/test/test_tub.py: hush pyflakes, make it work without crypto too * doc/listings/xfer-server.py: * doc/listings/xfer-client.py: new sample programs, a client/server pair which allow the client to put files in the server's directory. Useful as a replacement for restricted-command passphraseless ssh key arrangements. * foolscap/pb.py (Tub.setLocationAutomatically): new method that guesses an externally-visible IP address and uses it to call setLocation(). * foolscap/test/test_tub.py (SetLocation.test_set_location): test for it * foolscap/util.py (get_local_ip_for): make get_local_ip_for() more generally available, moving out of gatherer.py * foolscap/logging/gatherer.py: same * foolscap/negotiate.py (TubConnector.connectToAll): fix log message * foolscap/logging/web.py (LogEvent.__init__): include the incarnation number in the anchor index, so that logfiles which contain events from multiple incarnations of the same Tub will not suffer from href collisions * foolscap/broker.py: switch to foolscap.logging (Broker.connectionLost): log this event (Broker.freeYourReference): same, at log.UNUSUAL (Broker._callFinished): same, at log.UNUSUAL * foolscap/__init__.py: bump revision to 0.2.4+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2008-01-28 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.2.4 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2008-01-28 Brian Warner <warner@lothar.com> * NEWS: update for the upcoming release * foolscap/logging/filter.py (FilterOptions.parseArgs): make it possible to filter the event file in place, by using 'flogtool filter filename' instead of 'flogtool filter infile outfile' * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): include tubid in error message * foolscap/call.py (CopiedFailure.__getstate__): don't use reflect.qual on a string. This might improve behavior when we copy failures around, particularly in logging. 2008-01-18 Brian Warner <warner@lothar.com> * foolscap/negotiate.py: put all negotiation flog messages in facility "foolscap.negotiation", to make it easier to strip them out with 'flogtool filter --strip-facility foolscap' * foolscap/call.py (PendingRequest.fail): switch remote-exception logging over to flogging (InboundDelivery.logFailure): switch local-exception logging to flogging too * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): match it (TestCallable.testLogRemoteFailure): same * foolscap/logging/log.py (FoolscapLogger.__init__): remove unused .buffer attribute (FoolscapLogger.msg): avoid pickling application-specific exception classes when using failure=; use CopiedFailure to avoid it. This allows the emitted log pickle to be loaded on systems that do not have the original source code around. * foolscap/logging/filter.py: add --strip-facility option, to remove events that pertain to a given facility or its children 2008-01-17 Brian Warner <warner@lothar.com> * foolscap/logging/web.py (EventView.render): add a sort= query argument. If set to 'nested' (the default), you get the default nested view, in which the root events (those without a parent) are shown in chronological order, and the child events of each node are shown in chronological order underneath it. If sort=time, then no nesting is used, and all events are shown in chronological order. If sort=number, then all events are shown in numerical order, which is nominally better for single-process logfiles and coarse timestamps, but practically speaking is no better than sort=time. Using a non-nested mode can make it easier to spot events that happen in different areas but at about the same time; the nested display puts these events further apart (EventView._emit_events): minor formatting change 2008-01-10 Brian Warner <warner@lothar.com> * bin/flogtool (Options.opt_version): make 'flogtool --version' show the foolscap version. * foolscap/logging/web.py (SummaryView): add summaries to the 'flogtool web-viewer' tool: show counts of events by severity level, show lists of those events, with each event hyperlinked to the correct line in the full display. The full display also has anchor tags which let you construct bookmarks to specific lines. * foolscap/logging/filter.py (Filter): add 'flogtool filter' subcommand, to take one large eventlog pickle file and produce a smaller one with just a subset of the events. Currently this only allows you to filter by timestamp, and requires timestamps be provided as seconds since epoch. * bin/flogtool (dispatch): same * foolscap/logging/log.py (FoolscapLogger.err): add log.err, which behaves just like twisted's log.err: it accepts an exception or Failure object, or it can be used inside an except: clause to log the current exception. * foolscap/test/test_logging.py (Publish): test it * foolscap/logging/gatherer.py (LogSaver.remote_msg): if we can't pickle something, complain to stdout, rather than causing an error message to be sent back to the sender's Tub. * foolscap/call.py (CopiedFailure.__getstate__): make CopiedFailures pickleable, and make sure they come back looking just like they started. The issue was that we play games with the .type attribute to make .trap/.check work. * foolscap/test/test_copyable.py (Copyable._testFailure1_1): test it (Copyable._testFailure2_1): same * Makefile (pyflakes): new pyflakes doesn't uniqueify its output * NEWS: fix misspelling * foolscap/__init__.py: bump revision to 0.2.3+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-24 Brian Warner <warner@lothar.com> * foolscap/__init__.py (__version__): release 0.2.3 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-24 Brian Warner <warner@lothar.com> * setup.py: point url= at the trac page, instead of the root, since I keep forgetting to update the tarball links on the root * NEWS: update for the upcoming release * foolscap/logging/log.py (LogFileObserver.__init__): open the flogfile with mode "wb" instead of "a", so that both compressed and uncompressed files work the same way. This truncates the file on each run instead of appending to it. * foolscap/broker.py (Broker.getRemoteInterfaceByName): remove duplicate definition of method, detected by pyflakes * foolscap/test/test_banana.py: change import of names from foolscap.tokens to appease the new (stricter) pyflakes * foolscap/logging/gatherer.py (get_local_ip_for): move imports up to module level, remove duplicates * foolscap/pb.py (Tub.setLogPortFURLFile): if the application has done setOption("logport-furlfile"), make sure the logport furlfile is created as soon as possible (i.e. when setLocation is called). Allow these two calls to occur in either order. The setOption call must still occur before doing getLogPortFURL(), or before the tub connects to the log gatherer; otherwise the furlfile will be ignored. Closes #38. * foolscap/test/test_logging.py (Publish.test_logport_furlfile1): test it * doc/logging.xhtml: update to match, fix a few typos * foolscap/referenceable.py (RemoteReferenceOnly.getSturdyRef): fix this, it had bitrotted. Closes #35. * foolscap/test/test_pb.py (TestCallable.testGetSturdyRef): test it * foolscap/logging/log.py (FoolscapLogger.msg): only record e['args'] if there were any, to avoid spurious formatting attempts * foolscap/negotiate.py: update log.msg calls to use format= * foolscap/logging/log.py (format_message): refactor event formatting into a separate function. Switch to using Twisted's style of format= argument to indicate that we want to use keyword-argument formatting. Closes #39. (FoolscapLogger.msg): same, use format_message() to test whether the event is stringifiable or not. (TwistedLogBridge._old_twisted_log_observer): copy dict directly if there's a format= kwarg, or stringify the message portion if there isn't. I *think* this ought to match what twisted does. * foolscap/logging/dumper.py (LogDumper.print_event): same * foolscap/logging/web.py (LogEvent.to_html): use format_message * foolscap/test/test_logging.py (Basic.testLog): improve tests 2007-12-23 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation.__init__): log everything under the 'foolscap.negotiation' facility, and use NOISY by default. (Negotiation.log): same * foolscap/logging/log.py (NOISY): log levels are now defined in terms of their stdlib 'logging' counterparts, so they're ints from a larger scale (10 to 40). This removes the need for 'levelmap', and allows log levels to be compared better. (FoolscapLogger.msg): all messages get a ['level'] key (which defaults to OPERATIONAL), so that observers and viewers don't have to define a default or handle a missing key. (FoolscapLogger.add_event): observers are now a simple callable, and they're always invoked with eventually(). log-to-file is now done with an observer, as are remote subscribers. (FoolscapLogger.logTo): remove this, log-to-file is now done with an observer (LogFileObserver): new class to implement log-to-file. This accepts a minimum level to pay attention to. Pickling is done at level '2'. It will compress the output if the filename ends in .bz2, and will use twisted's reactor.addSystemEventTrigger to try and close the file at shutdown. This is good enough for trial test cases, since trial helpfully fires this trigger for us after all test cases have been run. (FLOGFILE): if the $FLOGFILE environment variable is set, write all log events of $FLOGLEVEL or higher to the named file (opened at import time). Set FLOGLEVEL=1 to include NOISY debug messages. Set FLOGTWISTED=1 to get twisted.log events in the same file. Use all three when running trial tests to see what foolscap is saying as the tests run, but note that you can't do FLOGFILE=_trial_temp/flog.out.bz2 because foolscap is usually imported before trial creates (or clears) that directory, thus deleting the newly-created flogfile. Applications are expected to use a better API to control log-to-file, but it doesn't really exist yet. * foolscap/logging/dumper.py (LogDumper): accept .bz2 files * foolscap/logging/web.py (WebViewer.get_events): same * foolscap/logging/publish.py (LogPublisher): observers are now simple callables, so build a wrapper to do the callRemote for us. Track both the wrapper and the subscriber, since we need to use them both at unsubscribe time. * foolscap/test/test_logging.py (Basic.testLog): make sure we can tolerate '%' in log messages. (this needs to be changed, see #39). Also test the generation threshold, and using num= (Advanced.testObserver): test the new observer-as-callable scheme (Advanced.testPriorities): facility names are now dot-separated 2007-12-21 Brian Warner <warner@lothar.com> * doc/logging.xhtml: define facilities as dot-separated instead of slash-separated, to match stdlib logging package * foolscap/__init__.py: bump revision to 0.2.2+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-12 Brian Warner <warner@lothar.com> * foolscap/__init__.py (__version__): release 0.2.2 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-12 Brian Warner <warner@lothar.com> * NEWS: update for new release * foolscap/negotiate.py (Negotiation): fix duplicate-connection handling logic. Simplify by removing the attempt-id information, and thus reject any connections with old seqnums (to prevent connection flap for parallel (connection-hint) offers). Reject connections from master_IR=None. Send the 'my-incarnation' field with all offers (both client and server), so that it will be available to whomever the master happens to be. * foolscap/broker.py (Broker.__init__): remove attempt-id * foolscap/test/test_negotiate.py (Replacement): update to match * foolscap/negotiate.py: covert logging to flogging (i.e. call foolscap.logging.log.msg instead of twisted.log.msg). This gives us parent/child structure, more useful severity levels, improves display of Failure instances, and will give us more data to work with in the future as the foolscap.logging tools mature. * foolscap/pb.py: same * foolscap/test/test_negotiate.py: same * bin/flogtool: move all usage.Options into the file that contains the implementations: TailOptions are moved to foolscap.logging.tail, etc. * foolscap/logging/gatherer.py: same * foolscap/logging/tail.py (TailOptions): same * foolscap/logging/dumper.py: same (DumpOptions.optFlags): add --verbose (show all event keys), --just-numbers. (LogDumper.print_event): change event printing to handle funny messages * foolscap/logging/web.py (WebViewerOptions.optParameters): same (FLOG_CSS): colorize the background of unusual messages (LogEvent): improve stringification, add better timestamps, event numbers, escape HTML better, display Failure instances better, (WebViewer.process_logfiles): event numbers use 'num', not 'number' * foolscap/logging/log.py (FoolscapLogger.set_generation_threshold): implement generation thresholds: don't even record messages if they fall below this threshold. Still needs a lot of usability work. The default is level>=NOISY, which records everything. (FoolscapLogger.logTo): give this a filename, and all log messages will be pickled and written to the given file. Messages are written with the same dict wrapper as 'flogtool gather' uses, so they are displayable by 'flogtool dump' or 'flogtool web-view'. Still needs work, ideally this would be a file to which we dump buffered messages once a problem is detected. To enable this from, say, trial, set os.environ['FLOGFILE'] to a filename. Also, set FLOGTWISTED to enable a twisted.log-to-flog bridge. (FoolscapLogger.msg): implement generation_threshold, also change stringification to survive things like "100%" in the log message. This needs work too, the 'except ValueError' clause is icky. * foolscap/pb.py (Tub.brokerAttached): use an eventual-send when informing everyone in waitingForBrokers, to match the asynchronicity of disconnection notifications delivered in Broker.finish() 2007-12-11 Brian Warner <warner@allmydata.com> * foolscap/call.py (PendingRequest.fail): tolerate pass-then-fail, which is just as weird of an error case but shouldn't cause an exception. The fact that this is necessary indicates significant problems in the new connection-management code. Sigh. * foolscap/banana.py (Banana.handleError): loseConnection takes a Failure, not an exception. Addresses #36. * foolscap/negotiate.py (Negotiation.acceptDecisionVersion1): same * foolscap/test/test_call.py: same * foolscap/test/test_reconnector.py (Reconnector._got_ref): same * foolscap/broker.py (Broker.shutdown): assert that we get a Failure * foolscap/test/common.py (Loopback.loseConnection): same * foolscap/__init__.py: bump revision to 0.2.1+ while between releases * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-10 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release 0.2.1 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same * foolscap/broker.py (Broker.connectionTimedOut): sigh, brown paper bag bug: broker.shutdown() requires a Failure, not an exception. Unfortunately the unit tests didn't catch this. * foolscap/pb.py (Tub.stopService): same * foolscap/negotiate.py (Negotiation.evaluateNegotiationVersion1): same * foolscap/logging/interfaces.py: give fully-qualified __remote_name__ strings to all RemoteInterfaces, to avoid collision with other code that might use these names. 2007-12-10 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release 0.2.0 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-12-10 Brian Warner <warner@allmydata.com> * setup.py: add the foolscap/logging package * NEWS: update for upcoming release * foolscap/negotiate.py (Negotiation.handle_old): finally implement this: if enabled, "old" brokers (older than 60 seconds) will be replaced by new offers, but "new" ones are not. The default timeout is 60 seconds, but you can set it to something else by calling tub.setOption("handle-old-duplicate-connections", 120) instead of using 'True'. Closes #34. * foolscap/pb.py (Tub.setOption): same * foolscap/broker.py (Broker.__init__): record creation timestamp * foolscap/test/test_negotiate.py (Replacement.testAncientClientWorkaround): test it 2007-12-10 Brian Warner <warner@lothar.com> * doc/logging.xhtml: updates * foolscap/reconnector.py (Reconnector.reset): add a utility method to manually force a Reconnector to reconnect, most useful from a manhole or other in-process eval loop. Closes #30. (Reconnector.getDelayUntilNextAttempt): another to query the delay until the next connection attempt will be made. Returns None if no attempt is currently scheduled. (Reconnector.getLastFailure): and another to provide the last failure, useful if you want to find out why it keeps reconnecting. * foolscap/negotiate.py (Negotiation.compareOfferAndExisting): if the seqnum is too old, accept the offer anyways: this handles the lost-a-decision-message case better at the expense of worse handling of offers-delivered-out-of-order case (which is far less likely to occur than a lost decision message) * foolscap/test/test_negotiate.py (Replacement): more tests (Replacement.testBouncedClient_Reverse): same * foolscap/logging/log.py (FoolscapLogger.msg): handle two kinds of argument formatting: log.msg("%d>%d", a, b), and log.msg("%(foo)s!", foo=something). Also handle the twisted_log equivalent of the dict case: log.msg(format="%(foo)s", foo=blah). * foolscap/test/test_logging.py (Basic.testLog): test it (Publish.test_logpublisher._got_logport._check_observer): same 2007-12-07 Brian Warner <warner@allmydata.com> * foolscap/negotiate.py (Negotiation.compareOfferAndExisting): new duplicate-connection-handling, for ticket #28. Details in the ticket, and need to be put in the docs. Each end stores and sends more information to give the master a better chance of detecting race conditions correctly. The end result is that silently-lost TCP connections (due to NAT timeouts or laptops being yanked from one network to another) should not cause 35-minute reconnector delays any longer. Many many thanks to Rob Kinninmont and Zooko for their invaluable help in finding a good solution to this problem. Hopefully this closes #28. (Negotiation.handle_old): placeholder method for the code to handle <=0.1.7 clients, not yet written. (Negotiation.acceptDecisionVersion1): record information from the decision in the Broker, so we can compare against it later * foolscap/broker.py (Broker.__init__): same * foolscap/pb.py (Tub.setup): record an "Incarnation Record", just a unique (random) string, so that remote servers can tell if we ought to remember them or not. (Tub.setOption): add new 'handle-old-duplicate-connections' option, for the <=0.1.7-client-handler code (not yet written). This option name might change. * foolscap/test/test_negotiate.py (Replacement): test much of the new code, probably covers 60% of the code paths. Still need to test the opposite direction (client==master) to make sure I didn't break anything. * Makefile (.figleaf.el): copy more tools from Tahoe: convert figleaf coverage data into an emacs-lisp -parseable format * misc/testutils/figleaf.el: elisp code to highlight uncovered lines * misc/testutils/figleaf_htmlizer.py: emit lines *not* covered in a separate column, and sort by that: this makes it a lot easier to pay attention to the places that need work. Code copied over from Tahoe. 2007-12-06 Brian Warner <warner@allmydata.com> * foolscap/broker.py (Broker.shutdown): refactor shutdown code, to make it safer to replace one connection with another. (Broker.abandonAllRequests): use an eventual-send for each flunking, to make sure the connection is completely gone by the time application code gets to run in the errbacks. * foolscap/pb.py (Tub.stopService): give a reason for the shutdown. Keep it a ConnectionDone (so we don't need to update other code that wouldn't expect a new exception type), but mention Tub.stopService in the arguments. (Tub.brokerAttached): add assert self.running.. this should never be called on a Tub that's already been shutdown. (Tub.brokerDetached): fix last-broker-detached detection, since the sequencing changed a bit. Make sure we don't fire the observerlist twice, by only firing it when we actually remove a broker and it was the last one (i.e. we started with some brokers, and finished with none). (Tub.__repr__): add the TubID to the repr * foolscap/call.py (FailureSlicer.getStateToCopy): truncate .value, .type, and .parents to the lengths defined in FailureConstraint. Unfortunately we don't seem to impose these constraints during serialization, so if some exception happens to have a really long name or arguments, it's the recipient of the CopiedFailure who will complain, making it hard to figure out where the real problem lies. The fact that the Violation doesn't seem to specify which attribute was in violation only adds confusion: it said <RootUnslicer>.<error-2>.??? for a 'token too large: 1819>1000', in RemoteCopyUnslicer.checkToken, which was probably f.value since that's the only 1000byte constraint. 2007-11-29 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub._log_gatherer_connected): for UnauthenticatedTubs, send tubid="<unauth>" instead of tubid=None, since RILogGatherer.logport is expecting a string. Ideally I'd like this to be ChoiceOf(str,None), but ChoiceOf doesn't work. In the long run UnauthenticatedTubs will acquire normal (but distinct) tubids, so this issue will go away. * foolscap/logging/interfaces.py: minor cleanup * foolscap/test/test_logging.py (Publish._test_gatherer._check): update to match * foolscap/constraint.py: improve various Violation messages * foolscap/logging/log.py: get logport/gatherers working the way they're described in the docs * foolscap/logging/publish.py: merge into log.py * foolscap/logging/interfaces.py: improve imports * foolscap/pb.py: fix API for access to this stuff * foolscap/test/test_logging.py: fix up tests to match * doc/logging.xhtml: more cleanup 2007-11-28 Brian Warner <warner@lothar.com> * Makefile: misc new targets, parameterize $(TRIAL) * doc/logging.xhtml: document new logging scheme, some of this is still speculative, unimplemented, or just plain wrong * foolscap/logging/log.py: frontend for logging scheme * foolscap/pb.py: partial API support for logging * foolscap/test/test_logging.py: tests for new code, partially adapted from Petmail log tests 2007-11-27 Brian Warner <warner@lothar.com> * foolscap/logging/*: new logging support, including hierarchical logging, remote publishing, gathering tools * foolscap/test/test_logging.py: minimal tests for it * bin/flogtool: command-line tool to view/gather log events * setup.py (scripts=): include flogtool 2007-10-16 Brian Warner <warner@lothar.com> * foolscap/test/test_gifts.py (Bad.test_location._introduce): use 127.0.0.1:2 as the bogus port, instead of 127.0.0.47:1 . Connecting to .47 causes a long delay on OS-X (probably because of some sort of personal firewall), whereas .1 fails right away. Port 2 is not listed in my copy of /etc/services and seems unlikely to have a real service running on it. 2007-09-24 Brian Warner <warner@lothar.com> * doc/serializing.xhtml: oops, fix closing tags * foolscap/__init__.py: bump revision to 0.1.7+ while between releases * misc/*/debian/changelog: same 2007-09-24 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release 0.1.7 * misc/{dapper|edgy|etch|fesity|sarge|sid}/debian/changelog: same 2007-09-24 Brian Warner <warner@lothar.com> * setup.py: remove download_url, I think url= is sufficient. * NEWS: update for the upcoming release * all: remove default size limits on Constraints. Users who wish to enforce size limits should provide a maxLength= argument to their Constraint constructors. These limits turned out to be more surprising than helpful. Closes #26. * foolscap/constraint.py (everythingTaster): remove the default SIZE_LIMIT on STRING tokens. (ByteStringConstraint.__init__): and on StringConstraint * foolscap/slicers/unicode.py (UnicodeConstraint.__init__): and here * foolscap/slicers/list.py (ListConstraint.__init__): and here * foolscap/slicers/dict.py (DictConstraint.__init__): and here * foolscap/slicers/set.py (SetConstraint.__init__): and here * foolscap/test/test_schema.py (CreateTest.testMakeConstraint): update tests to match * foolscap/test/test_copyable.py (MyRemoteCopy4.stateSchema): same 2007-09-16 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.debug_listBrokers): add a debugging method that tells you about all the connected brokers, and what methods are outstanding (both inbound and outbound) for each. If you see any method is sitting in one of these lists for a long time, there might be a problem in the execution of that method. * foolscap/broker.py (Broker.doNextCall): don't allow slow remote_foo methods to stall subsequent calls. This fixes a major message-delivery bug. Closes #25. * foolscap/test/test_call.py (TestCall.testStallOrdering): test it * all: merge in serialization-refactoring branch, adding Tub.(un)serialize (which can handle Referenceables) and moving us slightly closer to Sealers/Unsealers. The regular storage.serialize can now handle Copyables and objects for which you've registered an ISlicer adapter. Eventually the default unserialization interface will be 'safe' (meaning it won't create instances of arbitrary classes), and you'll have to give it additional arguments to enable 'unsafe' behavior. * doc/serializing.xhtml: document it * foolscap/banana.py: create slicer/unslicer at connectionMade, not __init__. Also change the way that errors are handled, and remove the use of 'types'. * foolscap/broker.py (PBRootUnslicer.open): refactor, move code to slicers.root.RootUnslicer (Broker.use_remote_broker): new attribute, True for connected brokers, False for the non-connected one that Tub.serialize uses. This is used by ReferenceableSlicer to decide whether to emit a my-reference sequence or a their-reference sequence. (StorageBroker): new subclass of Broker, uses new StorageBrokerRoot(un)Slicer. This broker is specialized to accept exactly one object, and hand it off to a waiting Deferred. There's still some useful refactoring to do, to mix in the behavior of ScopedRootUnslicer better. * foolscap/debug.py: delete this, having it around made the refactoring too difficult * foolscap/ipb.py (IBroker): use this to distinguish between a regular Banana instance and a Broker. The PB-specific slicers use it to assert that their .protocol is really a Broker. * foolscap/referenceable.py (ReferenceableSlicer): if the Referenceable is being sliced for storage, emit a their-reference sequence instead of a (useless) my-reference. Use giftID=0 to indicate that we don't want to do reference counting. (TheirReferenceUnslicer.ackGift): giftID=0 means don't ack * foolscap/slicers/root.py: RootSlicer refactoring (ScopedRootSlicer): new class for refactoring (ScopedRootUnslicer): same * foolscap/storage.py: same. Rewrite serialize/unserialize, they now return Deferreds and let you override the banana and root (un)slicer class to use, as well as the IO stream. * foolscap/test/test_banana.py: massive hacking to make it work. The storage classes changed a lot, requiring this cleanup. Also all banana/broker objects must have a transport and their connectionMade() method needs to be called before you can use them. * foolscap/test/test_pb.py: same * foolscap/test/test_serialize.py: test all the new functionality, including how serialized data cannot keep a Referenceable alive, and how you you can only serialize Referenceables with Tub.serialize(), not with foolscap.serialize(). 2007-09-11 Brian Warner <warner@lothar.com> * foolscap/call.py (PendingRequest.fail): improve logRemoteFailures by adding a source+dest TubID to the log message. Closes #23. (InboundDelivery.logFailure): same for logLocalFailures (CallUnslicer.receiveClose): tell the InboundDelivery about its broker * foolscap/pb.py (Tub.getTubID, .getShortTubID): utility methods (UnauthenticatedTub.getTubID, .getShortTubID): same * foolscap/referenceable.py (TubRef.getShortTubID): same (NoAuthTubRef.getShortTubID): same (RemoteReference._callRemote): tell the PendingRequest about the interface name it is using, so it can log a fully-qualified remote method name * foolscap/broker.py (Broker): tell the Broker about the remote TubRef it is attached to (Broker.doNextCall): make sure to log.err any problems that occur during callFailed, rather than discarding them. This catches errors in InboundDelivery.logFailure * foolscap/negotiate.py (Negotiation.switchToBanana): pass the remote TubRef to the newly-created Broker * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): test it (TestCallable.testLogLocalFailure): reduce timeout (TestCallable.testLogRemoteFailure._check): test it (TestCallable.testLogRemoteFailure): reduce timeout * foolscap/test/test_call.py (TestCall.testFailStringException): same (TestCall.testCopiedFailure): same * foolscap/test/test_gifts.py (Gifts.testGift): same * foolscap/test/common.py (TargetMixin.setupBrokers): provide tubref 2007-09-08 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.registerReference): add furlFile= argument, to make it easy to persist unguessable FURLs in a file on disk. * foolscap/test/test_tub.py (FurlFile): test it * foolscap/tokens.py (WrongTubIdError): new exception for it * doc/using-foolscap.xhtml (Using a Persistent FURL): document it. Also replace all use of 'url' with 'furl'. * foolscap/__init__.py: bump revision to 0.1.6+ while between releases * misc/*/debian/changelog: same 2007-09-02 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release 0.1.6 * misc/{dapper|edgy|feisty|sarge|sid}/debian/changelog: same * misc/etch: copy sid packaging for etch * Makefile (debian-etch): new target for etch .debs * setup.py: update download_url for 0.1.6 * MANIFEST.in: include etch files in the source release * NEWS: update for the upcoming release * Makefile (api-docs): new target to run 'epydoc' and generate API documentation. Addresses #16. (docs): fix the URLs of API doc targets to match * foolscap/negotiate.py: fix some docstrings * foolscap/pb.py: same * foolscap/promise.py: same 2007-08-31 Brian Warner <warner@lothar.com> * foolscap/schema.py (PolyConstraint.checkToken): we need to override this method as well as checkObject, otherwise inbound tokens may get rejected by the token-checking phase. robk noticed this with a ChoiceOf(StringConstraint(maxLength=3000, None)), which refused to accept a 2000 byte string (since the default Constraint.checkToken uses the everythingTaster which only accepts 1000-byte strings). Closes #13. * foolscap/test/test_call.py (TestCall.testChoiceOf): test it 2007-08-21 Brian Warner <warner@lothar.com> * foolscap/call.py (FailureSlicer.getStateToCopy): if we can't fit the whole traceback, elide the middle rather than truncate the end, since the end is usually the most interesting part. * foolscap/pb.py (Tub.setupEncryptionFile): don't use os.path.exists to decide if the certFile exists or not, just try to open it and deal with the exception if it happens. This avoids a race condition, not a big deal here but a good pattern to get in the habit of using everywhere. * foolscap/slicers/unicode.py (UnicodeConstraint.checkObject): improve error message 2007-08-17 Brian Warner <warner@lothar.com> * doc/using-foolscap.xhtml: fix more typos, as reported by arch_o_median. Closes #15. * .hgignore: ignore generated .html files in doc/ 2007-08-11 Brian Warner <warner@lothar.com> * doc/using-foolscap.xhtml: replace the phrase "private-key certificate" with the more accurate and more widely-used "public-key certificate". Thanks to Zooko for the patch. 2007-08-09 Brian Warner <warner@lothar.com> * doc/using-foolscap.xhtml: fix typos, thanks to David Ripton for the catch. * setup.py: add classifiers= and platforms=, update metadata, so that I can use 'setup.py register' for the next release. Closes #7. * README: update references to the home page. Closes #9. 2007-08-08 Brian Warner <warner@lothar.com> * misc/{sid|sarge|dapper|edgy|feisty}/debian/rules: fix references to renamed docs/ files. Closes #8. * foolscap/__init__.py: bump revision to 0.1.5+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-08-07 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.5 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-08-07 Brian Warner <warner@lothar.com> * NEWS: update for the upcoming release * foolscap/pb.py (Tub.registerNameLookupHandler): new function to augment Tub.registerReference(). This allows names to be looked up at request time, rather than requiring all Referenceables be pre-registered with registerReference(). The chief use of this would be for FURLs which point at objects that live on disk in some persistent state until they are needed. Closes #6. (Tub.unregisterNameLookupHandler): allow handlers to be removed (Tub.getReferenceForName): use the handler during lookup * foolscap/test/test_tub.py (NameLookup): test it 2007-07-27 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (LocalReferenceable): implement an adapter that allows code to do IRemoteReference(t).callRemote(...) and have it work for both RemoteReferences and local Referenceables. You might want to do this if you're getting back introductions to a variety of remote Referenceables, some of which might actually be on your local system, and you want to treat all of the, the same way. Local Referenceables will be wrapped with a class that implements callRemote() and makes it behave like an actual remote callRemote() would. Closes ticket #1. * foolscap/test/test_reference.py (LocalReference): test it 2007-07-26 Brian Warner <warner@lothar.com> * foolscap/call.py (AnswerUnslicer.receiveChild): accept a ready_deferred, to accomodate Gifts in return values. Closes #5. (AnswerUnslicer.receiveClose): .. and don't fire the response until any such Gifts resolve * foolscap/test/test_gifts.py (Gifts.testReturn): test it (Gifts.testReturnInContainer): same (Bad.testReturn_swissnum): and test the failure case too * foolscap/test/test_pb.py (TestAnswer.testAccept1): fix a test which wasn't calling start() properly and was broken by that change (TestAnswer.testAccept2): same * foolscap/test/test_gifts.py (Bad.setUp): disable these tests when we don't have crypto, since TubIDs are not mangleable in the same way without crypto. * foolscap/slicer.py (BaseUnslicer.receiveChild): new convention: Unslicers should accumulate their children's ready_deferreds into an AsyncAND, and pass it to the parent. If something goes wrong, the ready_deferred should errback, which will abandon the method call that contains it. * foolscap/slicers/dict.py (DictUnslicer.receiveClose): same * foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): same (TupleUnslicer.complete): same * foolscap/slicers/set.py (SetUnslicer.receiveClose): same * foolscap/slicers/list.py (ListUnslicer.receiveClose): same * foolscap/call.py (CallUnslicer.receiveClose): same * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): use our ready_deferred to signal whether the gift resolves correctly or not. If it fails, errback ready_deferred (to prevent the message from being delivered without the resolved gift), but callback obj_deferred with a placeholder to avoid causing too much distress to the container. * foolscap/broker.py (PBRootUnslicer.receiveChild): accept ready_deferred in the InboundDelivery, stash both of them in the broker. (Broker.scheduleCall): rewrite inbound delivery handling: use a self._call_is_running flag to prevent concurrent deliveries, and wait for the ready_deferred before delivering the top-most message. If the ready_deferred errbacks, that gets routed to self.callFailed so the caller hears about the problem. This closes ticket #2. * foolscap/call.py (InboundDelivery): remove whenRunnable, relying upon the ready_deferred to let the Broker know when the message can be delivered. (ArgumentUnslicer): significant cleanup, using ready_deferred. Remove isReady and whenReady. * foolscap/test/test_gifts.py (Base): factor setup code out (Base.createCharacters): registerReference(tubname), for debugging (Bad): add a bunch of tests to make sure that gifts which fail to resolve (for various reasons) will inform the caller about the problem, via an errback on the original callRemote()'s Deferred. 2007-07-25 Brian Warner <warner@lothar.com> * foolscap/util.py (AsyncAND): new utility class, which is like DeferredList but is specifically for control flow rather than data flow. * foolscap/test/test_util.py: test it * foolscap/call.py (CopiedFailure.setCopyableState): set .type to a class that behaves (as least as far as reflect.qual() is concerned) just like the original exception class. This improves the behavior of derived Failure objects, as well as trial's handling of CopiedFailures that get handed to log.err(). CopiedFailures are now a bit more like actual Failures. See ticket #4 (http://foolscap.lothar.com/trac/ticket/4) for more details. (CopiedFailureSlicer): make sure that CopiedFailures can be serialized, so that A-calls-B-calls-C can return a failure all the way back. * foolscap/test/test_call.py (TestCall.testCopiedFailure): test it * foolscap/test/test_copyable.py: update to match, now we must compare reflect.qual(f.type) against some extension classname, rather than just f.type. * foolscap/test/test_pb.py: same * foolscap/test/common.py: same 2007-07-15 Brian Warner <warner@lothar.com> * foolscap/test/test_interfaces.py (TestInterface.testStack): don't look for a '/' in the stacktrace, since it won't be there under windows. Thanks to 'strank'. Closes Twisted#2731. 2007-06-29 Brian Warner <warner@lothar.com> * foolscap/__init__.py: bump revision to 0.1.4+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-14 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.4 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same, also remove a bunch of old between-release version numbers 2007-05-14 Brian Warner <warner@lothar.com> * NEWS: update for the upcoming release * doc/using-foolscap.xhtml: rename from doc/using-pb.xhtml * doc/using-pb.xhtml: replace all uses of 'PB URL' with 'FURL' * foolscap/pb.py (Tub.getReference): if getReference() is called before Tub.startService(), queue the request until startup. (Tub.connectTo): same for connectTo(). (Tub.startService): launch pending getReference() and connectTo() requests. There are all fired with eventual-sends. * foolscap/reconnector.py (Reconnector): don't automatically start the Reconnector in __init__, rather wait for the Tub to start it. * foolscap/test/test_tub.py (QueuedStartup): test it * doc/using-pb.xhtml: update docs to match * foolscap/test/test_call.py (TestCall.testCall1): replace an arbitrary delay with a polling loop, to make the test more reliable under load * foolscap/referenceable.py (SturdyRef.asLiveRef): remove a method that was never used, didn't work, and is of dubious utility anyways. (_AsLiveRef): remove this too * misc/testutils/figleaf.py (CodeTracer.start): remove leftover debug logging * foolscap/remoteinterface.py (RemoteInterfaceConstraint): accept gifts too: allow sending of RemoteReferences on the outbound side, and accept their-reference sequences on the inbound side. * foolscap/test/test_gifts.py (Gifts.test_constraint): test it * foolscap/test/test_schema.py (Interfaces.test_remotereference): update test, since now we allow RemoteReferences to be sent on the outbound side * foolscap/remoteinterface.py (getRemoteInterface): improve the error message reported when a Referenceable class implements multiple RemoteInterfaces * foolscap/remoteinterface.py (RemoteMethodSchema.initFromMethod): properly handle methods like 'def foo(nodefault)' that are missing *all* default values. Previously this resulted in an unhelpful exception (since typeList==None), now it gives a sensible InvalidRemoteInterface exception. * foolscap/test/test_schema.py (Arguments.test_bad_arguments): test it 2007-05-11 Brian Warner <warner@lothar.com> * foolscap/slicers/set.py (FrozenSetSlicer): finally acknowledge our dependence on python2.4 or newer, by using the built-in 'set' and 'frozenset' types by default. We'll serialize the old sets.Set and sets.ImmutableSet too, but they'll emerge as a set/frozenset. This will cause problems for code that was written to be compatible with python2.3 (by using sets.Set) and wasn't changed when moved to 2.4, if it tries to mingle sets.Set with the data coming out of Foolscap. Unfortunate, but doing it this way preserves both sanity and good behavior for modern 2.4-or-later apps. (SetUnslicer): fix handling of children that were unreferenceable during construction, fix handling of children that are not ready for use (i.e. gifts). (FrozenSetUnslicer): base this off of TupleUnslicer, since previously the cycle-handling logic was completely broken. I'm not entirely sure this is necessary, since I think the contents of sets must be transitively immutable (or at least transitively hashable), but it good to review and clean it up anyways. * foolscap/slicers/allslicers.py: match name change * foolscap/slicers/tuple.py (TupleUnslicer.receiveClose): fix handling of unready children (i.e. gifts), previously gifts inside containers were completely broken. * foolscap/slicers/list.py (ListUnslicer.receiveClose): same * foolscap/slicers/dict.py (DictUnslicer.receiveClose): same * foolscap/call.py: add debug log messages (disabled) * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): gifts must declare themselves 'unready' until the RemoteReference resolves, since we might be inside a container of some sort. Without this fix, methods would be invoked too early, before the RemoteReference was really available. * foolscap/test/test_banana.py (ThereAndBackAgain.test_set): match new set/sets.Set behavior (ThereAndBackAgain.test_cycles_1): test some of the cycles (ThereAndBackAgain.test_cycles_3): add (disabled) test for checking cycles that involve sets. I think these tests are non-sensical, since sets can't really participate in the sorts of cycles we worry about, but I left the (disabled) test code in place in case it becomes useful again. * foolscap/test/test_gifts.py (Gifts.testContainers): validate that gifts can appear in all sorts of containers successfully. 2007-05-11 Brian Warner <warner@lothar.com.com> * foolscap/__init__.py: bump revision to 0.1.3+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-02 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.3 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-05-02 Brian Warner <warner@lothar.com> * MANIFEST.in: include some recently-added files to the source tarball * NEWS: update for the upcoming release * foolscap/reconnector.py (Reconnector._failed): simplify log/no-log logic * foolscap/slicers/unicode.py (UnicodeConstraint): add a new constraint that only accepts unicode objects. It isn't complete: I've forgotten how the innards of Constraints work, and as a result this one is too permissive: it will probably accept too many tokens over the wire before raising a Violation (although the post-receive just-before-the-method-is-called check should still be enforced, so application code shouldn't notice the issue). * foolscap/test/test_schema.py (ConformTest.testUnicode): test it (CreateTest.testMakeConstraint): check the typemap too * foolscap/test/test_call.py (TestCall.testMegaSchema): test in a call * foolscap/test/common.py: same * foolscap/constraint.py (ByteStringConstraint): rename StringConstraint to ByteStringConstraint, to more accurately describe its function. This constraint will *not* accept unicode objects. * foolscap/call.py, foolscap/copyable.py, foolscap/referenceable.py: * foolscap/slicers/vocab.py: same * foolscap/schema.py (AnyStringConstraint): add a new constraint to accept either bytestrings or unicode objects. I don't think it actually works yet, particularly when used inside containers. (constraintMap): map 'str' to ByteStringConstraint for now. Maybe someday it should be mapped to AnyStringConstraint, but not today. Map 'unicode' to UnicodeConstraint. * foolscap/pb.py (Tub.getReference): assert that the Tub is already running, either because someone called Tub.startService(), or because we've been attached (with tub.setServiceParent) to a running service. This requirement appeared with the connector-tracking code, and I hope to relax it at some point (such that any pre-startService getReferences will be queued and serviced when the Tub is finally started), but for this release it is a requirement to start the service before trying to use it. (Tub.connectTo): same * doc/using-pb.xhtml: document it * doc/listings/pb1client.py: update example to match * doc/listings/pb2client.py: update example to match * doc/listings/pb3client.py: update example to match * foolscap/pb.py (Tub.connectorFinished): if, for some reason, we're removing the same connector twice, log and ignore rather than explode. I can't find a code path that would allow this, but I *have* seen it occur in practice, and the results aren't pretty. Since the whole connection-tracking thing is really for the benefit of unit tests anyways (who want to know when Tub.stopService is done), I think it's more important to keep application code running. * foolscap/negotiate.py (TubConnector.shutdown): clear out self.remainingLocations too, in case it helps to shut things down faster. Add some comments. * foolscap/negotiate.py (Negotiation): improve error-message delivery, by keeping track of what state the receiver is in (i.e. whether we should send them an HTTP error block, an rfc822-style error-block, or a banana ERROR token). (Negotiation.switchToBanana): empty self.buffer, to make sure that any extra data is passed entirely to the new Banana protocol and none of it gets passed back to ourselves (Negotiation.dataReceived): same, only recurse if there's something still in self.buffer. In other situtations we recurse here because we might have somehow received data for two separate phases in a single packet. * foolscap/banana.py (Banana.sendError): rather than explode when trying to send an overly-long error message, just truncate it. 2007-04-30 Brian Warner <warner@lothar.com> * foolscap/broker.py (Broker.notifyOnDisconnect): if the RemoteReference is already dead, notify the callback right away. Previously we would never notify them, which was a problem. (Broker.dontNotifyOnDisconnect): be tolerant of attempts to unregister callbacks that have already fired. I think this makes it easier to write correct code, but on the other hand it loses the assertion feedback if somebody tries to unregister something that was never registered in the first place. * foolscap/test/test_call.py (TestCall.testNotifyOnDisconnect): test this new tolerance (TestCall.testNotifyOnDisconnect_unregister): same (TestCall.testNotifyOnDisconnect_already): test that a handler fires when the reference was already broken * foolscap/call.py (InboundDelivery.logFailure): don't use f.getTraceback() on string exceptions: twisted explodes (FailureSlicer.getStateToCopy): same * foolscap/test/test_call.py (TestCall.testFailStringException): skip the test on python2.5, since string exceptions are deprecated anyways and I don't want the warning message to clutter the test logs * doc/using-pb.xhtml (RemoteInterfaces): document the fact that the default name is *not* fully-qualified, necessitating the use of __remote_name__ to distinguish between foo.RIBar and baz.RIBar * foolscap/remoteinterface.py: same * foolscap/call.py (FailureSlicer.getStateToCopy): handle string exceptions without exploding, annoying as they are. * foolscap/test/test_call.py (TestCall.testFail4): test them 2007-04-27 Brian Warner <warner@lothar.com> * foolscap/broker.py (Broker.freeYourReference._ignore_loss): change the way we ignore DeadReferenceError and friends, since f.trap is not suitable for direct use as an errback * foolscap/referenceable.py (SturdyRef.__init__): log the repr of the unparseable FURL, rather than just the str, in case there are weird control characters in it * foolscap/banana.py (Banana.handleData): rewrite the typebyte scanning loop, to remove the redundant pos<64 check. Also, if we get an overlong prefix, log it so we can figure out what's going wrong. * foolscap/test/test_banana.py: update to match * foolscap/negotiate.py (Negotiation.dataReceived): if a non-NegotiationError exception occurs, log it, since it indicates a foolscap coding failure rather than some disagreement with the remote end. Log it with 'log.msg' for now, since some of the unit tests seem to trigger startTLS errors that flunk tests which should normally pass. I suspect some problems with error handling in twisted's TLS implementation, but I'll have to investigate it later. Eventually this will turn into a log.err. * foolscap/pb.py (Tub.keepaliveTimeout): set the default keepalive timer to 4 minutes. This means that at most 8 minutes will go by without any traffic at all, which should be a reasonable value to keep NAT table entries alive. PINGs are only sent if no other traffic was received, and they are only one byte long, so the traffic overhead should be minimal. Note that we are not turning on disconnectTimeout by default: if you want quietly broken connections to be disconnected before TCP notices a problem you'll need to do tub.setOption("disconnectTimeout", 10*60) or something. * foolscap/pb.py: remove an unused import * foolscap/pb.py (Tub.generateSwissnumber): always use os.urandom to generate the unguessable identifier. Previously we used either PyCrypto or fell back to the stdlib 'random' module (which of course isn't very random at all). I did it this way originally to provide compatibility with python2.3 (which lacked os.urandom): now that we require python2.4 or newer, os.urandom is a far better source (it uses /dev/random or equivalent). * doc/using-pb.xhtml: don't mention PyCrypto now that we aren't using it at all. * foolscap/negotiate.py (Negotiation.minVersion): bump both min and max version to '3', since we've added PING and PONG tokens that weren't present before. It would be feasible to accomodate v2 peers (by adding a Banana flag that refrains from ever sending PINGs), but there aren't enough 0.1.2 installations present to make this seem like a good idea just now. (Negotiation.maxVersion): same (Negotiation.evaluateNegotiationVersion3): same (Negotiation.acceptDecisionVersion3): same * foolscap/test/test_negotiate.py (Future): same * foolscap/banana.py (Banana): add keepalives and idle-disconnect. The first timeout value says that if we haven't received any data for this long, poke the other side by sending them a PING message. The other end is obligated to respond with a PONG (both PING and PONG are otherwise ignored). If we still haven't heard anything from them by the time the second timeout expires, we drop the connection. (Banana.dataReceived): if we're using keepalives, update the dataLastReceivedAt timestamp on every inbound message. (Banana.sendPING, sendPONG): new messages and handlers. Both are ignored, and serve only to update dataLastReceivedAt. * foolscap/tokens.py: add PING and PONG tokens * doc/specifications/banana.xhtml: document PING and PONG * foolscap/broker.py (Broker.__init__): add keepaliveTimeout and disconnectTimeout arguments. Both default to 'None' to disable keepalives and disconnects. * foolscap/negotiate.py (Negotiation.switchToBanana): copy timeouts from the Tub into the new Banana/Broker instance * foolscap/pb.py (Tub.setOption): accept 'keepaliveTimeout' and 'disconnectTimeout' options to enable this stuff. * foolscap/test/test_keepalive.py: test it * foolscap/pb.py (Tub.brokerClass): parameterize the kind of Broker that this Tub will create, to make certain unit tests easier to write (allowing them to substitute a custom Broker subclass). * foolscap/negotiate.py (Negotiation.brokerClass): same (Negotiation.initClient): capture the brokerClass here for clients (Negotiation.handlePLAINTEXTServer): and here for listeners (Negotiation.switchToBanana): use it 2007-04-26 Brian Warner <warner@lothar.com> * README (DEPENDENCIES, INSTALLATION): add docs 2007-04-16 Brian Warner <warner@lothar.com> * foolscap/remoteinterface.py (RemoteInterfaceConstraint.checkObject): string-format the object inside a tuple, to avoid an annoying logging failure when the object in question is actually a tuple * foolscap/test/test_gifts.py (ignoreConnectionDone): trap both ConnectionDone and ConnectionLost, since it appears that windows signals ConnectionLost. Hopefully this will make the unit tests pass under windows. * foolscap/banana.py (Banana.handleData): when the token prefix is too long, log and emit the repr of the prefix string, so somebody can figure out where it came from. * foolscap/test/test_banana.py (InboundByteStream.testString): update to match 2007-04-13 Brian Warner <warner@lothar.com> * foolscap/copyable.py (CopyableSlicer.slice): set self.streamable before yielding any tokens, otherwise contained elements that use streaming will trigger an exception. Many thanks to Ricky (iacovou-AT-gmail.com) for trying out advanced features of Foolscap and discovering this problem, I would never have stumbled over this one on my own. TODO: we still need unit tests to exercise this sort of thing on a regular basis. (Copyable2): same thing * foolscap/schema.py (_tupleConstraintMaker): redefine what tuples mean in constraint specifications. They used to indicate an alternative: (int,str) meant accept either an int *or* a string. Now tuples indicate actual tuples, so (int,str) means a 2-element tuple in which the first element is an int, and the second is a string. I don't know what I was thinking back then. If you really want to use alternatives, use schema.ChoiceOf instead. * foolscap/test/test_schema.py (CreateTest.testMakeConstraint): test that tuples mean tuples * foolscap/reconnector.py (Reconnector._failed): the old f.trap() could sometimes cause the reconnector to stop trying forever. Remove that. Thanks to Rob Kinninmont for finding the problem. Add new code to log the failure if f.check() indicates that it is a NegotiationError, since that's the sort of weird thing that users will probably want to see. * foolscap/test/test_reconnector.py: add lots of new tests * misc/testutils: add tools to do figleaf-based code-coverage checks while running unit tests. We have 89.2% coverage! Use 'make test-figleaf figleaf-output' to see the results. * Makefile: new targets for figleaf (test): enable 'make test TEST=foolscap.test.test_call' to work (test-figleaf): same * foolscap/__init__.py: bump revision to 0.1.2+ while between releases * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same 2007-04-04 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.2 * misc/{sid|sarge|dapper|edgy|feisty}/debian/changelog: same * NEWS: update for new release 2007-04-04 Brian Warner <warner@lothar.com> * misc/feisty/debian/*: add debian packaging support for the Ubuntu 'feisty' distribution * Makefile: and a way to invoke it * misc/edgy/debian/*: same for the 'edgy' distribution * MANIFEST.in: include the edgy/feisty files in the source tarball * foolscap/test/test_call.py (TestCall.testMegaSchema): add a new test to exercise lots of constraint code * foolscap/test/common.py: support code for it * foolscap/slicers/set.py (SetUnslicer.setConstraint): fix bugs discovered as a result (SetConstraint.__init__): same * foolscap/__init__.py: bump revision to 0.1.1+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-04-03 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.1 * misc/{sid|sarge|dapper}/debian/changelog: same * NEWS: update for new release 2007-04-03 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation): bump both minVersion and maxVersion to 2, indicating that this release is not compatible with 0.1.0, since the reqID=0 change will cause the first method call in either direction (probably a getYourReferenceByName) to never receive a response. The handler functions were rearranged a bit too. * foolscap/test/test_negotiate.py (Future): update to match * NEWS: get ready for release * foolscap/test/test_pb.py (TestCallable.testLogLocalFailure): validate that Tub.setOption("logLocalFailures") actually works (TestCallable.testLogRemoteFailure): same * foolscap/remoteinterface.py (UnconstrainedMethod): add a "constraint" that can be used to mark a method as accepting anything and returning anything. This might be useful if you have RemoteInterface for most of your application, but there are still one or two methods which should not enforce a schema of any sort. This mostly defeats the purpose of schemas in the first place, but offering UnconstrainedMethod means developers can make the schema-or-not decision differently for individual methods, rather than for a whole class at a time. * foolscap/constraint.py (IRemoteMethodConstraint): document the requirements on IRemoteMethodConstraint-providing classes, now that there are two of them. * foolscap/test/test_call.py (TestCall.testUnconstrainedMethod): test it * foolscap/test/common.py: add some support code for the test * foolscap/referenceable.py (RemoteReferenceTracker._handleRefLost): refrain from sending decref messages with count=0 * foolscap/negotiate.py (TubConnectorClientFactory.__repr__): include both the origin and the target of the factory * foolscap/test/*.py (tearDown): insure that all tests use the now-standard stopService+flushEventualQueue teardown procedure, to avoid trial complaints about leftover timers and selectables. * foolscap/test/test_pb.py (GoodEnoughTub): when crypto is not available, skip some tests that really require it. Modify others to not really require it. * foolscap/test/test_crypto.py: same * foolscap/test/test_gifts.py: same * foolscap/test/test_loopback.py: same * foolscap/test/test_negotiate.py: same * foolscap/test/test_*.py (localhost): replace all use of "localhost" with "127.0.0.1" to avoid invoking the address resolver, which sometimes leaves a cleanup timer running. I think the root problem is there's no clean way to interrupt a connection attempt which still in the address resolution phase. You can stop it, but there's no way to wait for the resolver's cleanup timer to finish, which is what we'd need to make Trial happy. tcp.BaseClient.resolveAddress does not keep a handle to the resolver, so failIfNotConnected cannot halt its timer. * foolscap/test/test_zz_resolve.py: removed this test * foolscap/crypto.py (_ssl): import SSL goo in a different way to appease pyflakes * all: fix some pyflakes warnings by checking for the importability of foolscap.crypto in a different way * foolscap/pb.py (Tub.stopService): when shutting down the Tub, make sure all outstanding connections are shut down as well. By the time stopService's deferred fires, all of our TCP transports should have had their 'connectionLost' methods fired. This is specifically to help unit tests that use Trial, which insists upon having a clean reactor between tests. With this change, test suites should use a tearDown() method that looks like: 'd = tub.stopService(); d.addCallback(flushEventualQueue); return d', and trial shouldn't complain about foolscap selectables or timers being left over. (Tub.stopService): also, since Tubs are not currently restartable, modify some entry points at shutdown to make sure nobody gets confused about why their getReference() doesn't work anymore. Be aware that at some point soon, we'll start enforcing the rule that the Tub must be started before you can get any connections out of it, at which point getReference() will queue requests until startService() is called. The idea is that the Tub will not use the network at all unless it is running. * foolscap/broker.py: drop the connection when shutdown() is called * foolscap/negotiate.py (Negotiate): rearrange error reporting and connection shutdown. Now errors are stashed and loseConnection() is called, but the errors are not reported to negotiationFailed() until connectionLost() is fired (which will be after any remaining data gets sent out over the wire). (TubConnector): the TubConnector reports success once the first connection has passed negotiation, but now lives until all of the connections are finally closed. It then informs the Tub that it is done, so the Tub can forget about it (and possibly notify stopService that it can finally complete). * foolscap/observer.py (OneShotObserverList): eventual-send -using event distributor, following the pattern espoused by Mark Miller's "Concurrency Among Strangers" paper. Many thanks to AllMyData.com for contributing this class. * foolscap/test/test_observer.py: tests for it 2007-03-22 Brian Warner <warner@lothar.com> * foolscap/constraint.py (StringConstraint): add a regexp= argument * foolscap/test/test_schema.py (ConformTest.testString): test it * foolscap/test/test_banana.py (TestBananaMixin.shouldDropConnection): fix a pyflakes warning * foolscap/call.py: same, don't fall back to plain StringIO if cStringIO is unavailable * foolscap/debug.py: same * foolscap/storage.py: same * foolscap/slicers/list.py (ListConstraint): add a minLength= argument, fix maxLength=None * foolscap/test/test_schema.py (ConformTest.testList): test it * foolscap/constraint.py (StringConstraint): add a minLength= argument * foolscap/test/test_schema.py (ConformTest.testString): test it * foolscap/slicers/set.py (BuiltinFrozenSetSlicer): add slicer for the builtin 'frozenset' type that appeared in python2.4 (SetConstraint): provide a constraint for sets * foolscap/schema.py (SetOf): add an alias * foolscap/test/test_schema.py (ConformTest.testSet): test it 2007-03-20 Brian Warner <warner@lothar.com> * foolscap/banana.py (Banana.outgoingVocabTableWasReplaced): remove verbose debug message, not really needed anymore * foolscap/ipb.py (IRemoteReference.callRemoteOnly): new method to invoke a remote method without waiting for a response. Useful for certain messages where we really don't care whether the far end receives them or not. * foolscap/referenceable.py (RemoteReference.callRemoteOnly): implement it (TheirReferenceUnslicer.ackGift): use it * foolscap/broker.py (Broker.initBroker): use reqID=0 to mean "we don't want a response". Note that this is a compatibility barrier: older endpoints which use reqID=0 for the first message will not get a response. All subsequent messages will be ok, though. (Broker._callFinished): don't send a response if reqID=0 (Broker.callFailed): don't send an error if reqID=0 * foolscap/call.py (InboundDelivery.logFailure): fix arg logging (CallUnslicer.receiveChild): don't create an activeLocalCalls entry if reqID=0 * foolscap/test/test_call.py (TestCallOnly.testCallOnly): test it (TestCall._testFailWrongReturnLocal_1): update expectations, now that reqIDs start at 1 instead of 0 * foolscap/test/common.py (TargetMixin.poll): new support code * foolscap/referenceable.py: add Referenceable to schema.constraintMap, so that RemoteInterfaces can use 'return Referenceable' to indicate that they return a Referenceable of any sort. This is like using 'return RIFoo' to indicate that the method returns a Referenceable that implements RIFoo, but without the specific interface requirement. * foolscap/remoteinterface.py (RemoteInterfaceConstraint): support this by skipping the interface check if self.interface=None * foolscap/test/test_schema.py (CreateTest): test it * foolscap/test/test_interfaces.py (Types): update test to match, since the error messages changed * foolscap/test/common.py: more test support changes 2007-03-19 Brian Warner <warner@lothar.com> * foolscap/ipb.py (IRemoteReference): new interface .. * foolscap/referenceable.py (RemoteReferenceOnly): .. implemented here * foolscap/remoteinterface.py (RemoteInterfaceConstraint.checkObject): remove a circular import by using IRemoteReference to detect RemoteReference instances, rather than using isinstance(). * foolscap/test/test_schema.py (Interfaces): test it * everything: massive Constraint refactoring. Primitive constraints (StringConstraint, IntegerConstraint, etc) are now in foolscap/constraint.py, while opentype-specific constraints like ListConstraint and BooleanConstraint are in the same module that defines the associated Slicer. Remote method constraints are in remoteinterface.py and copyable.py, FailureConstraint is in call.py . A new foolscap/constraint.py module contains the base classes but is careful not to import much else. foolscap/schema.py contains a reference to all constraints, so that user code can get at them conveniently. Tests were updated to import from the new places. Some circular imports were resolved. zope.interface adaptation has been used to assist with the conversion from the "shorthand" forms of constraint specification into the full form (i.e. converting x=str into x=StringConstraint()), specifically IConstraint(shorthand) will return a Constraint instance. * foolscap/__init__.py: bump revision to 0.1.0+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-03-15 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.1.0 * misc/{sid|sarge|dapper}/debian/changelog: same * README: update for new release * NEWS: update for new release 2007-02-16 Brian Warner <warner@lothar.com> * foolscap/eventual.py (_SimpleCallQueue._turn): retire all pending eventual-send messages before returning control to the reactor, rather than doing exactly one event per reactor turn. This seems likely to help avoid starvation, as we now finish as much work as possible before accepting IO (which might cause more work to be added to our queue), and probably makes the interaction between eventual-send and DelayedCalls a bit more consistent. Thanks to Rob Kinninmont for the suggestion. 2007-02-08 Brian Warner <warner@lothar.com> * foolscap/test/test_pb.py: move TestCall out to.. * foolscap/test/test_call.py: .. a new test file * foolscap/test/test_negotiate.py (BaseMixin.tearDown): add a 100ms stall between shutting down all the Tubs and actually finishing the test. This seems to be enough to stop the occasional test failures that probably occur because TCP connections that we've dropped haven't finished signalling the other end (also in our process) that they've been closed. 2007-01-30 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub): add certFile= argument, to allow the Tub to manage its own certificates. This argument provides a filename where the Tub should read or write its certificate. If the file exists, the Tub will read the certificate data from there. If not, the Tub will generate a new certificate and write it to the file. * foolscap/test/test_tub.py: test it * doc/using-pb.xhtml: document certFile= * doc/listings/pb2server.py: use certFile= in the example 2007-01-24 Brian Warner <warner@lothar.com> * foolscap/crypto.py (MyOptions._makeContext.alwaysValidate): add code to ignore two additional OpenSSL certificate validation errors: X509_V_ERR_CERT_NOT_YET_VALID (9) and X509_V_ERR_CERT_HAS_EXPIRED (10). Foolscap uses certificates very differently than web sites, and it is exceedingly common to start using a cert mere seconds after creating it. If there is any significant clock skew between the two systems, then insisting that the cert's "valid after X" time is actually in the past will cause a lot of false errors. 2007-01-22 Brian Warner <warner@lothar.com> * .darcs-boringfile: ignore files that are generated by distutils when we make a source release (dist/*) and when making a debian package (build/* and the debian install directory). * foolscap/__init__.py: bump revision to 0.0.7+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2007-01-16 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.0.7 * misc/{sid|sarge|dapper}/debian/changelog: same * NEWS: update for 0.0.7 2007-01-16 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.getBrokerForTubRef): special-case an attempt to connect to a tub with an ID equal to our own, by attaching a Broker to a special LoopbackTransport that delivers serialized data directly to a peer without going through a socket. * foolscap/broker.py (LoopbackTransport): same (Broker.setTub): refactor some code out of negotiate.py * foolscap/negotiate.py (Negotiation.switchToBanana): same (Negotiation.loopbackDecision): new method to determine params for a loopback connection * foolscap/test/test_loopback.py: enable all tests, add a check to make sure we can connect to ourselves twice * foolscap/referenceable.py (RemoteReferenceTracker.getRef): the weakref this holds may have become stale, so check that we both have self.ref *and* that self.ref() is not None to decide whether we must re-create the RemoteReference. This fixes a bug in which two calls to Tub.getReference() for the same URL would result in the second call getting None. (RemoteReferenceTracker._handleRefLost): only send a decref message if we haven't already re-created the RemoteReference * foolscap/test/test_pb.py (TestService.testConnect3): modify this test to validate the 'call Tub.getReference() twice' bug is fixed 2007-01-15 Brian Warner <warner@lothar.com> * foolscap/test/test_loopback.py (ConnectToSelf): Add tests to validate that we can connect to our own Tub properly. This test does not yet pass for authenticated Tubs: the negotiation hangs until the 30 second timeout is reached. To fix this requires special-casing such connections to use a different kind of Broker, one that wires transport.write to eventual(rcvr.dataReceived) and skips negotiation completely. 2007-01-10 Brian Warner <warner@lothar.com> * doc/using-pb.xhtml: fix some "pb" references to mention "Foolscap" instead * doc/schema.xhtml: same 2007-01-09 Brian Warner <warner@lothar.com> * foolscap/pb.py (Listener.removeTub): disownServiceParent is not guaranteed to return a Deferred, so don't try to make removeTub do so either. I saw a failure related to this, but was unable to analzye it well enough to reproduce it or write a test case. (Tub.stopListeningOn): tolerate removeTub returning synchronously (Tub.stopService): same 2007-01-04 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation.dataReceived): when sending an error message to the far end inside the decision block, make sure the error text itself has no newlines, since that would break the format of the block, and probably cause all sorts of confusion. * foolscap/ipb.py (IRemotelyCallable.doRemoteCall): remote calls now accept positional args 2007-01-04 Brian Warner <warner@lothar.com> * foolscap/__init__.py: bump revision to 0.0.6+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2006-12-18 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.0.6 * misc/{sid|sarge|dapper}/debian/changelog: same 2006-12-18 Brian Warner <warner@lothar.com> * misc/{sid|sarge|dapper}/debian/rules: include copyable.xhtml * NEWS: update for 0.0.6 * foolscap/negotiate.py (Negotiation): Send less data. When sending a range (both for the fundamental banana negotiation version and for the initial vocab table index), send it in a single line with "0 1" rather than two separate min and max lines. This brings the hello message down to about 105 bytes and improves the benefit of using a negotiated initial-vocab-table-range rather than an early (but post-negotiation) SET-VOCAB banana sequence. * foolscap/schema.py: add RemoteInterfaceConstraints. This works by declaring an argument as, e.g., RIFoo, which means that this argument must be passed a RemoteReference that is connected to a remote Referenceable which implements RIFoo. This works as a return value constraint too. (Constraint.checkObject): add inbound= argument to this method, so RemoteInterfaceConstraint can work properly (InterfaceConstraint): split this into local and remote forms (LocalInterfaceConstraint): only check real local objects, not RemoteReferences. This isn't really useful yet, but eventually schemas will turn into proper local Guards and then it will be. (RemoteInterfaceConstraint): only check RemoteReferences. The check performed must be different on inbound and outbound (since we'll see a RemoteReference when inbound=True, and a Referenceable when inbound=False). (makeConstraint): distinguish between Interfaces and RemoteInterfaces, so we can figure out whether to use LocalInterfaceConstraint or RemoteInterfaceConstraint (callable): get rid of this, the functionality has been absorbed into RemoteMethodSchema.initFromMethod * foolscap/broker.py (Broker._doCall): use inbound= argument (Broker._callFinished): same * foolscap/referenceable.py (RemoteReference._callRemote): same * foolscap/slicer.py (ReferenceUnslicer.receiveChild): same * foolscap/test/test_schema.py: same * foolscap/test/test_interfaces.py: rearrange, add tests for RemoteInterfaces. (LocalTypes): there are tests for Interfaces too, but without local Guards they're disabled for now. * foolscap/test/common.py: refactoring * foolscap/schema.py (makeConstraint): map None to Nothing(), which only accepts None. This is pretty handy for methods which are always supposed to return None. * foolscap/schema.py (RemoteMethodSchema.checkResults): don't annotate any Violations here.. leave that up to the caller * foolscap/broker.py (Broker._callFinished): update the annotation * foolscap/test/test_pb.py: update to match * foolscap/tokens.py (Violation.prependLocation): add new methods to Violations for easier annotation of where they occurred (Violation.appendLocation): same (Violation.__str__): remove the "at" from the location text * foolscap/test/test_pb.py: update to match * foolscap/broker.py (Broker._callFinished): if the outbound return value violates the schema, annotate the Violation to indicate the object and method that was responsible. 2006-12-15 Brian Warner <warner@lothar.com> * foolscap/call.py (CopiedFailure): clean up a bit, make it match the current Failure class better * doc/using-pb.xhtml: document positional arguments * foolscap/call.py: pass both positional and keyword arguments to remote methods. Previously only keyword arguments were accepted. This is a pretty far-reaching change, and introduces a compatibility barrier. (ArgumentSlicer): send both positional args and kwargs in a separate container (CallSlicer): move arg-sending out of CallSlicer (InboundDelivery.isRunnable): make InboundDelivery itself responsible for determining when it is runnable, instead of leaving that up to the CallUnslicer. The InboundDelivery is always referenceable, making the resulting object delivery simpler. (ArgumentUnslicer): move arg-receiving out of CallUnslicer. All of the schema-checking takes place here. Simplify the are-we-ready tests. (CallUnslicer): most of the code has moved out. The (call) sequence is now ('call', reqID, objID, methname, (args)). * foolscap/broker.py (Broker.scheduleCall): simplify, allow posargs * foolscap/referenceable.py (Referenceable.doRemoteCall): deliver posargs to the target method as well as kwargs (RemoteReference): same, stop trying to pre-map posargs into kwargs, no longer require a RemoteInterface to use posargs * foolscap/vocab.py (vocab_v1): add 'arguments' to the v1 vocab list. This is a compatibilty barriers, and changes like this are only allowed between releases. Once 0.0.6 is out we should leave the v1 list alone and make any additions to v2 instead. * foolscap/schema.py (RemoteMethodSchema): allow posargs, deal correctly with a mixture of posargs and kwargs * foolscap/test/test_schema.py (Arguments): test the RemoteMethodSchema class * foolscap/test/test_pb.py (TestCall.testCall1a): new tests of posargs and mixed posargs/kwargs (TestService.tearDown): use flushEventualQueue for cleanup * foolscap/test/test_interfaces.py (TestInterface): change a few things to match RemoteMethodSchema's new interfaces * foolscap/eventual.py (flushEventualQueue): allow this method to accept a single argument, which it ignores. This enables it to be used easily as a Deferred callback/errback, such as in a Trial tearDown method. The recommended usage is: d = clean_stuff(); d.addBoth(flushEventualQueue); return d 2006-12-11 Brian Warner <warner@lothar.com> * foolscap/vocab.py: add code to negotiate an initial set of words with which to pre-fill the VOCAB token list. Each side proposes a range and they use the highest common index (and they exchange a short hash of the list itself to guard against disagreements). This serve to compress the protocol traffic by maybe 50% over the longer run. * foolscap/negotiate.py: send the 'initial-vocab-table-min' and '-max' keys in the offer, and 'initial-vocab-table-index' in the decision (and in the Banana params) * foolscap/broker.py (Broker.__init__): populate the table * foolscap/banana.py (Banana.populateVocabTable): new method * foolscap/test/test_banana.py (Sliceable.testAdapter): todo items * foolscap/referenceable.py (RemoteReferenceOnly.notifyOnDisconnect): document this method. * foolscap/broker.py (Broker.shutdown): cancel all disconnect watchers upon shutdown * foolscap/pb.py (Tub.stopService): same * foolscap/negotiate.py (Negotiation.evaluateHello): if we spot an <=0.0.5 peer, mention that fact in our error message, to distinguish this case from some completely non-Foolscapish protocol trying to talk to us. 2006-12-10 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (TubConnectorClientFactory.__repr__): annotate the string form to include which Tub we're connecting to. This makes the default factory's "BlahFactory Starting" log messages more interesting to look at. * foolscap/referenceable.py (TubRef.getTubID): support method (NoAuthTubRef.getTubID): same 2006-12-01 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (RemoteReference.callRemote): use defer.maybeDeferred to rearrange and simplify. Clarify the comments about the various phases of commitment. * foolscap/call.py (AnswerUnslicer.checkToken): when re-raising an exception, use bareword 'raise' rather than explicitly re-raising the same exception instance with 'raise v'. Both forms get the right instance, but the latter loses the earlier stack trace. * foolscap/schema.py (RemoteMethodSchema.checkResults): same (RemoteMethodSchema.checkAllArgs): same * foolscap/referenceable.py (RemoteReference.callRemote): same * foolscap/test/test_interfaces.py (TestInterface.testStack): new test to verify that the Failure you get when you violate outbound method arguments actually includes the call to callRemote. * foolscap/schema.py (StringConstraint.checkObject): make the Violation message more useful (InterfaceConstraint.checkObject): same, by printing the repr() of the object that didn't meet the constraint. I'm not sure if this could be considered to leak sensitive information or not. (ClassConstraint.checkObject): same (RemoteMethodSchema.checkAllArgs): record which argument caused the problem in the Violation * foolscap/referenceable.py (RemoteReference.callRemote): add RemoteInterface and method name to the Violation when a caller violates their outbound constraint * foolscap/tokens.py (Violation.setLocation,getLocation): make it easier to modify an existing location value * foolscap/test/test_interfaces.py (TestInterface.testFail): verify that RemoteFailures pass a StringConstraint schema * foolscap/test/test_copyable.py: remove unused imports, from pyflakes * foolscap/test/test_pb.py: same * foolscap/test/test_reconnector.py: same * foolscap/test/test_registration.py: same * foolscap/test/test_interfaces.py: split the RemoteInterface tests out to a separate file * foolscap/test/test_pb.py: split them from here * foolscap/test/common.py: factor out some common utility classes 2006-11-30 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation.dataReceived): when sending an error block, set banana-decision-version to '1' so the recipient knows that it's safe to interpret the 'error' key. Thanks to Rob Kinninmont for the catch. 2006-11-27 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation._evaluateNegotiationVersion1): ignore extra keys in the offer, since a real v2 (and beyond) offer will have all sorts of extra keys. * foolscap/test/test_negotiate.py (NegotiationV2): test it by putting extra keys in the offer 2006-11-26 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation): change negotiation protocol: now each end sends a minVersion/maxVersion pair, using banana-negotiation-min-version and banana-negotiation-max-version, indicating that they can handle all versions between those numbers, inclusive. The deciding end finds the highest version number that fits in the ranges of both ends, and includes it in the banana-decision-version key of the decision block. This is an incompatible protocol change, but should make it easier (i.e. possible) to have compatible protocol changes in the future. Thanks to Zooko for suggesting this approach. (Negotiation.evaluateNegotiationVersion1): each negotiation version gets is own methods (Negotiation.acceptDecisionVersion1): same (TubConnectorClientFactory.buildProtocol): allow the Tub to make us use other Negotiation classes, for testing * foolscap/pb.py (Listener.__init__): same, use the class from the Tub that first caused the Listener to be created * foolscap/broker.py (Broker.__init__): record the banana-decision-version value, so tests can check it * foolscap/test/test_negotiate.py (Future): test it 2006-11-17 Brian Warner <warner@lothar.com> * foolscap/pb.py: remove unused and dodgy urlparse stuff * doc/using-pb.xhtml: move and expand the section on Copyable and other pass-by-copy things into a new file * doc/copyable.xhtml: new document. Thanks to Ricky Iacovou for the registerCopier examples. * doc/listings/copyable-{receive|send}.py: new examples * doc/stylesheet.css, doc/stylesheet-unprocessed.css * doc/template.tpl: docs utilities * Makefile: add 'make docs' target * foolscap/__init__.py: export registerCopier and registerRemoteCopyFactory * foolscap/copyable.py (Copyable): The new preferred Copyable usage is to have a class-level attribute named "typeToCopy" which holds the unique string. This must match the class-level "copytype" attribute of the corresponding RemoteCopy class. Copyable subclasses (or ICopyable adapters) may still implement getTypeToCopy(), but the default just returns self.typeToCopy . Most significantly, we no longer automatically use the fully-qualified classname: instead we *require* that the class definition include "typeToCopy". Feel free to use any stable and globally-unique string here. (RemoteCopyClass): Require that RemoteCopy subclasses set their "copytype" attribute, and use it for auto-registration. These subclasses can still use "copytype=None" to inhibit auto-registration. They no longer auto-register with the fully-qualified classname. * foolscap/referenceable.py (SturdyRef): match this change * foolscap/test/test_copyable.py: same 2006-11-16 Brian Warner <warner@lothar.com> * foolscap/negotiate.py (Negotiation.dataReceived): include the error message in the '500 Internal Server Error' string. (Negotiation.handlePLAINTEXTClient): include the full '500 Internal Server Error' string in the reported exception. These two changes make it easier to spot mismatched TubIDs. Thanks to Rob Kinninmont for the suggestion. 2006-11-14 Brian Warner <warner@lothar.com> * foolscap/__init__.py: bump revision to 0.0.5+ while between releases * misc/{sid|sarge|dapper}/debian/changelog: same 2006-11-04 Brian Warner <warner@lothar.com> * NEWS: update for 0.0.5 * foolscap/__init__.py: release Foolscap-0.0.5 * misc/{sid|sarge|dapper}/debian/changelog: same * MANIFEST.in: add debian packaging files to source tarball 2006-11-01 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.setOption): new API to set options. Added logRemoteFailures and logLocalFailures, which cause failed callRemotes to be sent to the twisted log via log.msg . The defaults are False, which means that failures are only reported through the caller's Deferred.errback . Setting logRemoteFailures to True means that the client's log will contain a record of every callRemote that it sent to someone else that failed on the far side. This can be implemented on a per-callRemote basis by just doing d.addErrback(log.err) everywhere, but often there are reasons (like debugging) for logging failures that are completely independent of the desired error-handling path. These log messages have a REMOTE: prefix to make it very clear that the stack trace being shown is *not* occurring on the local system, but rather on some remote one. Setting logLocalFailures to True means that the server's log will contain a record of every callRemote that someone sent to it which failed on that server. This cannot be implemented with addErrbacks, since normally the server does not care about the methods it is running for other people's benefit. This option is purely for debugging purposes. These log messages have a LOCAL: prefix to make it clear that the stack trace is happening locally, but on behalf of some remote caller. * foolscap/call.py (PendingRequest.fail): improve the logging, make it conditional on logRemoteFailures, add the REMOTE: prefix (InboundDelivery): put more information into the InboundDelivery, move logLocalFailures logging into it (CallUnslicer.receiveClose): put the .runnable flag on the InboundDelivery object instead of on the CallUnslicer * foolscap/broker.py (Broker): pass the InboundDelivery around instead of the CallUnslicer that it points to. (Broker.callFailed): Add logLocalFailures checking here. * foolscap/reconnector.py: oops, add missing import that would break any actual reconnection attempts 2006-10-31 Brian Warner <warner@lothar.com> * misc/sarge/debian/control: add sarge packaging * misc/dapper/debian/control: update dependencies, add Recommends on pyopenssl * misc/sid/debian/control: same * Makefile: add 'debian-sarge' target * misc/dapper/debian: move debian packaging up a level * misc/sid/debian: same * Makefile: same * foolscap/__init__.py (__version__): bump to 0.0.4+ while between releases * misc/debs/sid/debian/changelog: same * misc/debs/dapper/debian/changelog: same 2006-10-26 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.0.4 * misc/debs/sid/debian/changelog: same * misc/debs/dapper/debian/changelog: same 2006-10-26 Brian Warner <warner@lothar.com> * setup.py: fix project URL * MANIFEST.in: include misc/debs/* in the source tarball * NEWS: update for 0.0.4 * foolscap/test/test_reconnector.py: verify that the Reconnector's callbacks are properly interleaved with any notifyOnDisconnect callbacks the user might have registered. A Reconnector cb that uses notifyOnDisconnect should see a strictly-ordered sequence of connect, disconnect, connect, disconnect. 2006-10-25 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (RemoteReferenceOnly.notifyOnDisconnect): accept args/kwargs to pass to the callback. Return a marker that can be passed to dontNotifyOnDisconnect() to de-register the callback. * foolscap/broker.py (Broker.notifyOnDisconnect): same (Broker.connectionLost): fire notifyOnDisconnect callbacks in a separate turn, using eventually(), so that problems or side-effects in one call cannot affect other calls or the connectionLost process * foolscap/test/test_pb.py (TestCall.testDisconnect4): test it * foolscap/pb.py (Tub.registerReference): undo that, make registerReference *always* create a strongref to the target, but split some of the work out to an internal function which makes the weakrefs. Tub.registerReference() is the API that application code uses to publish an object (make it reachable) *and* have the Tub keep it alive for you. I'm not sure I can think of a use case for making it reachable but *not* wanting the Tub to keep it alive. If you want to make it reachable but still ephemeral, just pass it over the wire. (Tub._assignName): new method to make weakrefs and assign names. (Tub.getOrCreateURLForReference): renamed from getURLForReference. Changed to assign a name if possible and one didn't already exist. BEHAVIOR CHANGE: This causes *all* objects passed over the wire, whether explicitly registered or just implicitly passed along, to be shareable as gifts (assuming the Tub is reachable and has a location, of course). * foolscap/referenceable.py (ReferenceableTracker.getURL): update * foolscap/test/test_registration.py (Registration.testWeak): use _assignName instead of registerReference * foolscap/test/test_gifts.py (Gifts.testOrdering): test it 2006-10-25 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub.registerReference): add a strong= argument which means the Tub should keep the registered object alive. If strong=False, the tub uses a weakref, so that when the application and all remote peers forget about the object, the Tub will too. strong= defaults to True to match the previous behavior, but this might change in the future, and/or it might become a property to be set on the Tub. * foolscap/test/test_registration.py: new tests for it * foolscap/test/test_pb.py (TestService.testStatic): disable this test, since static data (like tuples) are not weakreffable. The registration of static data is an outstanding issue. * foolscap/pb.py (Tub.connectTo): provide a new method, sets up a repeating connection to a given url (with randomized exponential backoff) that will keep firing a callback each time a new connection is made. This is the foolscap equivalent of ReconnectingClientFactory, and is the repeating form of getReference(). Thanks to AllMyData.com for sponsoring this work. * foolscap/reconnector.py (Reconnector): implement it here * foolscap/test/test_reconnector.py: test it * doc/using-pb.xhtml: update to reflect that we now have secure PBURLs and TubIDs, and that methods are delivered in-order (at least within a Tub-to-Tub connection) even in the face of gifts. * misc/debs/dapper/debian/rules (binary-indep): remove obsolete reference to the old python-twisted-pb2 package * foolscap/referenceable.py (YourReferenceSlicer.slice): assert that we actually have a URL to give out, since otherwise the error will confusingly show up on the far end (as a Violation). This occurs when we (as Alice) try to introduce Carol to a Bob that was not explicitly registered in Bob's Tub, such that Bob does not have a URL to give out. * foolscap/pb.py (Tub): tubID is no longer a parameter to Tub, since it is always computed from the certificate (UnauthenticatedTub): but it *is* a parameter here, since there is no certificate * foolscap/broker.py (Broker.getMyReferenceByCLID): relax the assertion to (int,long), since eventually clids will overrun a 31-bit integer. Thanks to Rob Kinninmont for the catch. (Broker.remote_decref): same 2006-10-10 Brian Warner <warner@lothar.com> * misc/debs: add some debian packaging, separate directories for sid and dapper because sid has pycentral and dapper is still in the versioned-python-package era * Makefile: simple Makefile to remind me how to create .debs 2006-10-05 Brian Warner <warner@lothar.com> * foolscap/__init__.py: bump to 0.0.3+ while between releases 2006-10-05 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.0.3 * NEWS: update for 0.0.3 release 2006-10-05 Brian Warner <warner@lothar.com> * foolscap/test/test_gifts.py (Gifts): split out the Introduction tests from test_pb.py (Gifts.testOrdering): test the ordering of messages around a gift. Doing [send(1), send(2, carol), send(3)] should result in Bob seeing [1, (2,carol), 3] in that order. Before the recent ordering fix, the presence of the gift would delay message delivery, resulting in something like [1, 3, (2,carol)] * foolscap/test/test_pb.py: same * foolscap/call.py (CallUnslicer): fix ordering of message delivery in the face of Gifts. Each inbound method call gets unserialized into an InboundDelivery/CallUnslicer pair, which gets put on a queue. Messages get pulled off the queue in order, but only when the head of the queue is ready (i.e. all of its arguments are available, which means any pending Gifts have been retrieved). (InboundDelivery): same (CallUnslicer.describe): stop losing useful information * foolscap/broker.py (Broker.doNextCall): add inboundDeliveryQueue to implement all this * foolscap/test/test_pb.py (TestCall.testFailWrongArgsRemote1): match the change to CallUnslicer.describe * foolscap/referenceable.py (TheirReferenceUnslicer.receiveClose): don't bother returning ready_deferred, since we're returning an unreferenceable Deferred anyway. * foolscap/test/test_pb.py (Test3Way): put off the check that Alice's gift table is empty until we're sure she's received the 'decgift' message. Add a note about a race condition that we have to work around in a weird way to avoid spurious test failures until I implement sendOnly (aka callRemoteOnly). 2006-10-04 Brian Warner <warner@lothar.com> * foolscap/test/test_banana.py (ThereAndBackAgain.testIdentity): use an actual tuple. Obviously I wasn't thinking when I first wrote this and tried to use "(x)" to construct a one-item tuple. 2006-10-02 Brian Warner <warner@lothar.com> * everything: fix most of the pyflakes warnings. Some of the remaining ones are actual bugs where I need to finish implementing something. * foolscap/slicers/*.py: move most Slicers/Unslicers out to separate files, since slicer.py was way too big * foolscap/slicers/allslicers.py: new module to pull them all in. banana.py imports this to make sure all the auto-registration hooks get triggered. * everything: rearrange imports to match * setup.py: add new sub-package 2006-10-01 Brian Warner <warner@lothar.com> * foolscap/slicer.py: rearrange the internals, putting the corresponding Slicer and Unslicer for each type next to each other * foolscap/slicer.py: move all "unsafe" Slicers and Unslicers out to storage.py where it belongs * foolscap/storage.py: same * foolscap/test/test_banana.py: fix some imports to match * foolscap/test/test_pb.py: same * foolscap/slicer.py (ReplaceVocabSlicer): clean up VOCAB handling: add the ('add-vocab') sequence to incrementally add to the receiving end's incomingVocabulary table, fix the race condition that would have caused problems for strings that were serialized after the setOutgoingVocabulary() call was made but before the ('set-vocab') sequence was actually emitted. Lay the groundwork for adaptive tokenization and negotiated vocab table presets. Other classes involved are AddVocabSlicer, AddVocabUnslicer, and ReplaceVocabUnslicer. (BananaUnslicerRegistry): handle the add-vocab and set-vocab sequences with a registry rather than special-casing them. * foolscap/storage.py (UnsafeRootUnslicer): same, add the BananaUnslicerRegistry * foolscap/banana.py (setOutgoingVocabulary): make it safe to call this function at any time, as it merely schedules an update. Change the signature to accept a list of strings that should be tokenized rather than expecting the caller to choose the index values as well. (addToOutgoingVocabulary): new function to tokenize a single string, also safe to call at any time (outgoingVocabTableWasReplaced): (allocateEntryInOutgoingVocabTable): (outgoingVocabTableWasAmended): new functions for use by the Slicers that are sending the 'set-vocab' and 'add-vocab' sequences (Banana.maybeVocabizeString): reserve a place for adaptize tokenizing * foolscap/test/test_banana.py: match the changes * foolscap/broker.py: s/topRegistry/topRegistries/, since it is actually a list of Registries. Same for openRegistry and openRegistries * foolscap/slicer.py: same * foolscap/storage.py: same * foolscap/test/test_banana.py: same * foolscap/slicer.py (BuiltinSetSlicer): use a different test to look for python < 2.4, one which doesn't make pyflakes complain about using __builtins__ 2006-09-30 Brian Warner <warner@lothar.com> * foolscap/promise.py (Promise): implement a simpler syntax, at the encouragement of Zooko and others: now p.foo(args) does an eventual-send. This is a simpler form of send(p).foo(args) . Added _then and _except methods to do simple callback/errback handling. You can still do send() and sendOnly() on either immediate values or Promises: this shortcut only helps with send() on a Promise. You can still do when() on a Promise, which is more flexible because it returns a Deferred. The new syntax gives you a more dataflow-ish style of coding, which might be confusing in some ways but can also make the overall code much much easier to read. * foolscap/test/test_promise.py: update tests * foolscap/test/common.py (HelperTarget.remote_defer): replace callLater(0) with fireEventually() * foolscap/test/test_banana.py (ErrorfulSlicer.next): same (EncodeFailureTest.tearDown): use flushEventualQueue() for cleanup * foolscap/crypto.py (CertificateError): In Twisted >2.5, this exception is defined in twisted.internet.error, and it is sometimes raised by the SSL transport (in getPeerCertificate), and we need to catch it. In older versions, we define it ourselves even though it will never be raised, so that the code which catches it doesn't have to have weird conditionals. * foolscap/negotiate.py (Negotiation.handleENCRYPTED): catch the CertificateError exception (which indicates that we have an encrypted but unauthenticated connection: the other end did not supply a certificate). In older versions of twisted's SSL code, this was just indicated by having getPeerCertificate() return None. * foolscap/test/test_negotiate.py: re-enable all negotiation tests * foolscap/pb.py (UnauthenticatedTub): change the API and docs to refer to "Unauthenticated" tubs rather than "Unencrypted" ones, since that's really the choice you get to make. We use encrypted connections whenever possible; what you get to control is whether we use keys to provide secure identification and introduction. * foolscap/__init__.py: same, export UnauthenticatedTub instead of UnencryptedTub * foolscap/negotiate.py: same * foolscap/referenceable.py: same * foolscap/test/test_negotiate.py: same * doc/listings/pb1server.py: update examples to match * doc/using-pb.xhtml: same 2006-09-26 Brian Warner <warner@lothar.com> * foolscap/pb.py (Tub): rename PBService to Tub, make it always be encrypted (UnencryptedTub): new class for unencrypted tubs * all: fix everything else (code, docs, tests) to match * foolscap/ipb.py (ITub): new interface to mark a Tub 2006-09-24 Brian Warner <warner@lothar.com> * foolscap/referenceable.py (RemoteReferenceTracker._refLost): now that we have eventually(), use it to avoid the ugly bug-inducing indeterminacies that result from weakref callbacks being fired in the middle of other operations. * foolscap/promise.py (Promise._resolve): I think I figured out chained Promises. In the process, I made it illegal to call _break after the Promise has already been resolved. This also means that _resolve() can only be called once. We'll figure out breakable Far references later. * foolscap/test/test_promise.py (Chained): tests for them * foolscap/broker.py (Broker.getRemoteInterfaceByName): fix a bunch of typos caught by pyflakes. Left a couple of ones in there that I haven't figured out how to fix yet. * foolscap/slicer.py (InstanceUnslicer.receiveChild): same * foolscap/schema.py (RemoteMethodSchema.initFromMethod): same * foolscap/pb.py (Listener.addTub): same * foolscap/debug.py (TokenBanana.reportReceiveError): same * foolscap/copyable.py: same * foolscap/test/common.py: same * foolscap/test/test_pb.py (TestReferenceable.NOTtestRemoteRef1): same * foolscap/eventual.py: move eventual-send handling out to a separate file. This module now provides eventually(cb), d=fireEventually(), and d=flushEventualQueue() (for use by unit tests, not user code). * foolscap/negotiate.py: update to match * foolscap/test/common.py: same * foolscap/test/test_pb.py: same * foolscap/test/test_eventual.py: new tests for eventually() * foolscap/promise.py: rework Promise handling, now it behaves like I want it to (although chained Promises aren't working yet) * foolscap/test/test_promise.py: rework tests 2006-09-16 Brian Warner <warner@lothar.com> * foolscap/crypto.py: fall back to using our own sslverify.py if Twisted doesn't provide one (i.e. Twisted-2.4.x). * foolscap/sslverify.py: copy from the Divmod tree 2006-09-14 Brian Warner <warner@lothar.com> * foolscap/banana.py: remove #! line from non-script * foolscap/remoteinterface.py: same * foolscap/tokens.py: same * foolscap/test/test_schema.py: same * foolscap/__init__.py: bump to 0.0.2+ while between releases 2006-09-14 Brian Warner <warner@lothar.com> * foolscap/__init__.py: release Foolscap-0.0.2 2006-09-14 Brian Warner <warner@lothar.com> * doc/using-pb.xhtml: update pb3 example to match current usage, show an example of using encrypted Tubs * doc/listings/pb3calculator.py: same * doc/listings/pb3user.py: same * foolscap/__init__.py: rearrange the API: now 'import foolscap' is the preferred entry point, rather than 'from foolscap import pb'. * foolscap/pb.py: stop importing things just to make them available to people who import foolscap.pb * all: same, update docs, examples, tests * all: rename newpb to 'Foolscap' * setup.py: fix packages= to get tests too 2006-05-15 Brian Warner <warner@lothar.com> * test_zz_resolve.py: rename test file, I'd like to sit at the end of the tests rather than at the beginning. This is to investigate ticket #1390. * test_negotiate.py (Crossfire): oops, a cut-and-paste error resulted in two CrossfireReverse tests and zero Crossfire tests. Fixed this to enable the possibly-never-run real CrossfireReverse test case. (top): disable all negotiation tests unless NEWPB_TEST_NEGOTIATION is set in the environment, since they are sensitive to system load and the intermittent buildbot failures are annoying. 2006-05-05 Brian Warner <warner@lothar.com> * release-twisted: add 'pb' subproject * twisted/python/dist.py: same * twisted/pb/__init__.py: set version to 0.0.1 * twisted/pb/topfiles/setup.py: fix subproject name, set version 0.0.1 2006-04-29 Brian Warner <warner@lothar.com> * topfiles/README, topfiles/NEWS: prepare for 0.0.1 release * setup.py: fix up description, project name * test_ZZresolve.py: add some instrumentation to try and debug the occasional all-connection-related-tests-fail problem, which I suspect involves the threadpool being broken. 2006-02-28 Brian Warner <warner@lothar.com> * sslverify.py: update to latest version (r5075) from Vertex SVN, to fix a problem reported on OS-X with python2.4 . Removed the test-case-name tag to prevent troubles with buildbot on systems that don't also have vertex installed. I need to find a better solution for this in the long run: I don't want newpb to depend upon Vertex, but I also don't want to duplicate code. 2006-02-27 Brian Warner <warner@lothar.com> * debug.py (encodeTokens): return a Deferred rather than use deferredResult 2006-02-02 Brian Warner <warner@lothar.com> * test_negotiate.py: skip pb-vs-web tests when we don't have twisted.web, thanks to <marienz> for the patch. 2006-01-26 Brian Warner <warner@lothar.com> * test/test_banana.py (ErrorfulSlicer.next): don't use callLater() with non-zero timeout * test/test_promise.py (TestPromise.test2): same * test/common.py (HelperTarget.remote_defer): same 2006-01-25 Brian Warner <warner@lothar.com> * copyable.py: refactor ICopyable and IRemoteCopy to make it possible to register adapters for third-party classes. (RemoteCopy): allow RemoteCopy to auto-register with the fully-qualified classname. This is only useful if you inherit from both pb.Copyable and pb.RemoteCopy at the same time, otherwise the sender and receiver will be using different names so they won't match up. * broker.py (PBRootUnslicer.open): now that registerRemoteCopy is done purely in terms of Unslicers, remove all the special-case code that handled IRemoteCopy (PBRootSlicer.slicerForObject): since zope.interface won't do transitive adaptation, manually handle the ThirdPartyClass -> ICopyable -> ISlicer case * test/test_copyable.py: clean up, improve comments (MyRemoteCopy3Unslicer): update to match new RemoteCopyUnslicer behavior. This needs to be documented and made easier. Also switch from pb.registerRemoteCopy to registerRemoteCopyUnslicerFactory, which is a mouthful. (Registration): split this out, update to match new debug tables in copyable.py (Adaptation): test ICopyable adapters 2006-01-23 Brian Warner <warner@lothar.com> * common.py: remove missing test_gift from the test-case-name tag, not sure how that got in there * test/test_copyable.py: split Copyable tests out of test_pb.py * test/common.py: factor out some more common test utility pieces * copyable.py: add suitable test-case-name tag * base32.py: rename Base32.py to base32.py, to match Twisted naming conventions * crypto.py: same * pb.py: same 2006-01-02 Brian Warner <warner@lothar.com> * negotiate.py (eventually): add glyph's eventual-send operator, based upon a queue cranked by callLater(0). (flushEventualQueue): provide a way to flush that queue, so tests know when to finish. * test/test_pb.py: switch to negotiate.eventually * test/__init__.py: add test-case-name tag 2005-12-31 Brian Warner <warner@lothar.com> * test_gift.py (TestOrderedGifts.testGift): verify that the presence of a gift (a third-party reference) in the arguments of a method does not cause that method to be run out-of-order. Marked TODO because at the moment they *are* run out-of-order. * common.py (RIHelper.append): new method * referenceable.py (TheirReferenceUnslicer.ackGift): ignore errors that involve losing the connection, since if these happen, the giver will decref the gift reference anyway. This removes some spurious log.errs and makes the unit tests happier. 2005-12-30 Brian Warner <warner@lothar.com> * test_negotiate.py (Versus.testVersusHTTPServerEncrypted): stall for a second after the test completes, to give the HTTP server a moment to tear down its socket. Otherwise trial flunks the test because of the lingering socket. I don't care for the arbitrary 1.0-second delay, but twisted.web doesn't give me any convenient way to wait for it to shut down. (this test was only failing under the gtk2 reactor, but I think this was an unlucky timing thing). (Versus.testVersusHTTPServerUnencrypted): same * negotiate.py (eventually): add an eventual-send operator (Negotiation.negotiationFailed): fire connector.negotiationFailed through eventually(), to give us a chance to loseConnection beforehand. This helps the unit tests clean up better. * negotiation.py (eventually): change the eventual-send operator to (ab)use reactor.callFromThread instead of callLater(0). exarkun warned me, but I didn't listen: callLater(0) does not guarantee relative ordering of sequentially-scheduled calls, and the windows reactors in fact execute them in random order. Obviously I'd like the reactor to provide a clearly-defined method for this purpose. * test_pb.py (eventually): same (Loopback.write): same. It was the reordering of these _write calls that was breaking the unit tests on windows so badly. (Loopback.loseConnection): same 2005-12-29 Brian Warner <warner@lothar.com> * test_pb.py (Loopback): fix plan-coordination bug by deferring all write() and loseConnection() calls until the next reactor turn, using reactor.callLater(0) as an 'eventual send' operator. This avoids an infinite-mutual-recursion hang that confuses certain test failures. Tests which use this Loopback must call flush() and wait on the returned Deferred before finishing. (TargetMixin): do proper setup/teardown of Loopback (TestCall.testDisconnect2): use proper CONNECTION_LOST exception (TestCall.testDisconnect3): same (TestReferenceable.testArgs1): rename some tests (TestReferenceable.testArgs2): test sending shared objects in multiple arguments of a single method call (TestReferenceable.testAnswer1): test shared objects in the return value of a method call (TestReferenceable.testAnswer2): another test for return values * call.py (CallUnslicer): inherit from ScopedUnslicer, so arguments that reference shared objects will accurately reproduce the object graph (AnswerUnslicer): same, for answers that have shared objects (ErrorUnslicer): same, just in case serialized Failures do too * slicer.py (ImmutableSetSlicer): set trackReferences=False, since immutable objects are never shared, so don't require reference tracking * banana.py (Banana.sendError): do loseConnection() in sendError rather than inside dataReceived. 2005-12-26 Brian Warner <warner@lothar.com> * slicer.py (ScopedSlicer.registerReference): track references with a (obj,refid) pair instead of just refid. This insures that the object being tracked stays alive until the scope is retired, preventing some ugly bugs that result from dead object id() values being reused. These bugs would only happen if the object graph changes during serialization (which you aren't supposed to do), but this is a cheap fix that limits the damage that could happen. In particular, it should fix a test failure on the OS-X buildslave that results from a unit test that is violating this object-graph -shouldn't-change prohibition. * banana.py (StorageBanana): refactor storage-related things, moving them from banana.py and slicer.py into the new storage.py . This includes UnsafeRootSlicer, StorageRootSlicer, UnsafeRootUnslicer, and StorageRootUnslicer. Also provide a simple serialize()/unserialize() pair in twisted.pb.storage, which will be the primary interface for simple pickle.dumps()-like serialization. 2005-12-24 Brian Warner <warner@lothar.com> * slicer.py: remove #!, add test-case-name (SetSlicer): define this unconditionally, now that python2.2 is no longer supported. (BuiltinSetSlicer): just like SetSlicer, used when there is a builtin 'set' type (python2.4 and higher) (ImmutableSetSlicer): define this unconditionally (SetUnslicer): same (ImmutableSetUnslicer): same * test_banana.py (TestBananaMixin.looptest): make it easier to test roundtrip encode/decode pairs that don't *quite* re-create the original object (TestBananaMixin.loop): clear the token stream for each test (ThereAndBackAgain.test_set): verify that python2.4's builtin 'set' type is serialized as a sets.Set * all: drop python2.2 compatibility, now that Twisted no longer supports it 2005-12-22 Brian Warner <warner@lothar.com> * pb.py (Listener.getPortnum): more python2.2 fixes, str in str (PBService.__init__): same, bool issues * test/test_banana.py: same, use failUnlessSubstring * test/test_negotiate.py: same * test/test_pb.py: same * negotiate.py: same, str in str stuff * broker.py: don't import itertools, for python2.2 compatibility * sslverify.py: same 2005-12-20 Brian Warner <warner@lothar.com> * test/test_banana.py: remove all remaining uses of deferredResult/deferredError * test/test_pb.py: same 2005-12-09 Brian Warner <warner@lothar.com> * pb.py (PBService.__init__): switch to SHA-1 for TubID digests * negotiate.py (Negotiation.evaluateHello): same * crypto.py (digest32): same 2005-12-08 Brian Warner <warner@lothar.com> * pb.py (PBService): allow all Tubs to share the same RandomPool 2005-10-10 Brian Warner <warner@lothar.com> * lots: overhaul negotiation, add lots of new tests. Implement shared Listeners, correct handling of both encrypted and non-encrypted Tubs, follow multiple locationHints correctly. More docs, update encrypted-tub examples to match new usage. 2005-09-15 Brian Warner <warner@lothar.com> * test_pb.py: remove some uses of deferredResult/deferredError 2005-09-14 Brian Warner <warner@lothar.com> * pb.py (PBService.generateSwissnumber): use PyCrypto RNG if available, otherwise use the stdlib 'random' module. Create a 160-bit swissnumber by default, this can be changed by the NAMEBITS class attribute. (PBService.__init__): use a random 32-bit number as a TubID when we aren't using crypto and an SSL certificate * Base32.py: copy module from the waterken.org Web-Calculus python implementation * test/test_crypto.py (TestService.getRef): let it register a random swissnumber instead of a well-known name * crypto.py: Implement encrypted PB connections, so PB-URLs are closer to being secure capabilities. This file contains utility functions. * sslverify.py: some pyOpenSSL wrappers, copied from Divmod's Vertex/vertex/sslverify.py * test/test_crypto.py: test case for encrypted connections * pb.py (PBServerFactory.buildProtocol): accomodate missing tubID, this needs to be re-thought when I do the "what if we aren't using crypto" pass. (PBServerFactory.clientConnectionMade): get the remote_tubid from a .theirTubID attribute, not the negotiated connection parameters, which won't include tub IDs anyway) (PBClientFactory.buildProtocol): if we're using crypto, tell the other side we want an encrypted connection (PBService.__init__): add useCrypto= parameter, currently defaults to False. This should switch to =True soon. (PBService.createCertificate): if useCrypto=True, create an SSL certificate for the Tub. * ipb.py (DeadReferenceError): actually define it somewhere * broker.py (Broker.handleNegotiation_v1): cleanup, make the different negotiation-parameter dictionaries distinct, track the ['my-tub-id'] field of each end more carefully. Start a TLS session when both ends want it. (Broker.startTLS): method to actually start the TLS session. This is called on both sides (client and server), the t.i.ssl subclasses figure out which is which and inform SSL appropriately. (Broker.acceptNegotiation): Make a PB-specific form. Start TLS if the server tells us to. When the second (encrypted) negotiation block arrives, verify that the TubID we're looking for matches both what they claim and what their SSL certificate contains. (Broker.freeYourReference): ignore DeadReferenceErrors too * banana.py (Banana.__init__): each instance must have its own copy of self.negotiationOffer, rather than setting it at the class level (Banana.negotiationDataReceived): let both handleNegotiation() and acceptNegotiation() return a 'done' flag, if False then the negotiation is re-started (Banana.handleNegotiation): make handleNegotiation_v1 responsible for setting self.negotiationResults (Banana.handleNegotiation_v1): same (Banana.acceptNegotiation): same 2005-09-09 Brian Warner <warner@lothar.com> * broker.py: big sanity-cleanup of RemoteInterface usage. Only allow a single RemoteInterface on any given pb.Referenceable. Tub.getReference() now only takes a string-form method name, so the rr.callRemote(RIFoo['bar'], *args) form is gone, and the one RemoteInterface associated with the RemoteReference (is available) will be checked. Tub.getReference() no longer takes an interface name: you request an object, and then later find out what it implements (rather than specifying your expectations ahead of time). Gifts (i.e. 'their-reference' sequences) no longer have an interfacename.. that is left up to the actual owner of the reference, who will provide it in the 'my-reference' sequence. * call.py, pb.py, referenceable.py, remoteinterface.py: same * test/test_pb.py: update to match, still needs some cleanup 2005-09-08 Brian Warner <warner@lothar.com> * setup.py, twisted/pb/topfiles: add "PB" sub-project * banana.py (Banana.sendFailed): oops, loseConnection() doesn't take an argument * copyable.py (RemoteCopyClass): make it possible to disable auto-registration of RemoteCopy classes * test/test_pb.py (TestCopyable.testRegistration): test it * referenceable.py (CallableSlicer): make it possible to publish callables (bound methods in particular) as secure capabilities. They are handled very much like pb.Referenceable, but with a negative CLID number and a slightly different callRemote() codepath. * broker.py (Broker.getTrackerForMyCall): same (Broker.getTrackerForYourReference): same, use a RemoteMethodReferenceTracker for negative CLID values (Broker.doCall): callables are distinguished by having a methodname of 'None', and are dispatched differently * call.py (CallUnslicer.checkToken): accept INT/NEG for the object ID (the CLID), but not string (leftover from old scheme) (CallUnslicer.receiveChild): handle negative CLIDs specially * test/test_pb.py (TestCallable): tests for it all (TestService.getRef): refactor (TestService.testStatic): verify that we can register static data too, at least stuff that can be hashed. We need to decide whether it would be useful to publish non-hashable static data too. 2005-09-05 Brian Warner <warner@lothar.com> * pb.py (PBService): move to using tubIDs as the primary identity key for a Tub, replacing the baseURL with a .location attribute. Look up references by name instead of by URL, and start using SturdyRefs locally instead of URLs whenever possible. (PBService.getReference): accept either a SturdyRef or a URL (RemoteTub.__init__): take a list of locationHints instead of a single location. The try-all-of-them code is not yet written, nor is the optional redirect-following. (RemoteTub.getReference): change the inter-Tub protocol to pass a name over the wire instead of a full URL. The Broker is already connected to a specific Tub (multiple Tubs sharing the same port will require separate Brokers), and by this point the location hints have already served their purpose, so the name is the only appropriate thing left to send. * broker.py (RIBroker.getReferenceByName): match that change to the inter-Tub protocol: pass name over the wire, not URL (Broker.getYourReferenceByName): same (Broker.remote_getReferenceByName): same * referenceable.py (RemoteReferenceOnly): replace getURL with getSturdyRef, since the SturdyRef can be stringified into a URL if necessary (SturdyRef): new class. When these are sent over the wire, they appear at the far end as an identical SturdyRef; if you want them to appear as a live reference, send sr.asLiveRef() instead. * test/test_pb.py (TestService.testRegister): match changes (Test3Way.setUp): same (HelperTarget.__init__): add some debugging annotations * test/test_sturdyref.py: new test * doc/pb/using-pb.xhtml: update to match new usage, explain PB URLs and secure identifiers * doc/pb/listings/pb1server.py: same * doc/pb/listings/pb1client.py: same * doc/pb/listings/pb2calculator.py: same * doc/pb/listings/pb2user.py: same 2005-05-12 Brian Warner <warner@lothar.com> * doc/pb/using-pb.xhtml: document RemoteInterface, Constraints, most of Copyable (still need examples), Introductions (third-party references). * doc/pb/listings/pb2calculator.py, pb2user.py: demostrate bidirectional references, using service.Application 2005-05-10 Brian Warner <warner@lothar.com> * broker.py (Broker.freeYourReference): also ignore ConnectionLost errors * doc/pb/listings/pb1client.py, pb1server.py: use reactor.run() * doc/pb/using-pb.xhtml: add shell output for examples * doc/pb/using-pb.xhtml: started writing usage docs * banana.py (Banana.dataReceived): add .connectionAbandoned, don't accept inbound data if it has been set. I don't trust .loseConnection to work right away, and sending multiple negotiation error messages is bad. (Banana.negotiationDataReceived): split out negotiation stuff to a separate method. Improve failure-reporting code to make sure we either report a problem with a negotation block, or with an ERROR token, not both, and not with multiple ERROR tokens. Catch errors in the upper-level bananaVersionNegotiated() call. Make sure we only send a response if we're the server. Report negotiation errors with NegotiationError, not BananaError. (Banana.reportReceiveError): rearrange a bit, accept a Failure object. Don't do transport.loseConnection here, do it in whatever calls reportReceiveError * debug.py (TokenBanana.reportReceiveError): match signature change (TokenStorageBanana.reportReceiveError): same * test/test_banana.py: match changes * tokens.py (NegotiationError): new exception * broker.py (Broker.handleNegotiation_v1): use the negotiation block to exchange TubIDs. (Broker.connectionFailed): tell the factory if negotiation failed (Broker.freeYourReference): ignore lost-connection errors, call freeYourReferenceTracker even if the connection was lost, since in that case the reference has gone away anyway. (Broker.freeYourReferenceTracker): don't explode if the keys were already deleted, since .connectionLost will clear everything before the decref-ack mechanism gets a chance to delete them. * referenceable.py (RemoteReferenceTracker.__repr__): stringify these with more useful information. * pb.py (PBServerFactory.buildProtocol): copy .debugBanana flag into the new Broker (both .debugSend and .debugReceive) (PBServerFactory.clientConnectionMade): survive a missing TubID (PBClientFactory.negotiationFailed): notify all onConnect watchers 2005-05-08 Brian Warner <warner@lothar.com> * test_pb.py (TestService): test the use of PBService without RemoteInterfaces too 2005-05-04 Brian Warner <warner@lothar.com> * broker.py (Broker): add tables to track gifts (third-party references) (PBOpenRegistry): add their-reference entry (RIBroker.decgift): new method to release pending gifts * call.py (PendingRequest): add some debugging hints (CallUnslicer): accept deferred arguments, don't invoke the method until all arguments are available * pb.py (PBService.listenOn): return the Service, for testing (PBService.generateUnguessableName): at least make them unique, if not actually unguessable (top): remove old URL code, all is now PBService * referenceable.py (RemoteReferenceOnly.__repr__): include the URL, if available (RemoteReference.callRemote): set .methodName on the PendingRequest, to make debugging easier (YourReferenceSlicer.slice): handle third-party references (TheirReferenceUnslicer): accept third-party references * schema.py (Nothing): a constraint which only accepts None * test/test_pb.py (Test3Way): validate third-party reference gifts 2005-04-28 Brian Warner <warner@lothar.com> * tokens.py (IReferenceable): move to flavors.py * flavors.py (IReferenceable): add it, mark Referenceable as implementing it. * pb.py (PBServerFactory): make root= optional (PBService): new class. In the future, all PB uses will go through this service, rather than using factories and connectTCPs directly. The service uses urlparse to map PB URLs to target hosts. * test_pb.py (TestService): start adding tests for PBService 2005-04-26 Brian Warner <warner@lothar.com> * banana.py: add preliminary newpb connection negotiation * test_banana.py: start on tests for negotiation, at least verify that newpb-newpb works, and that newpb-http and http-newpb fail. 2005-04-16 Brian Warner <warner@lothar.com> * banana.py (Banana.handleData): handle -2**31 properly * test_banana.py (ThereAndBackAgain.test_bigint): test it properly * flavors.py: python2.2 compatibility: __future__.generators * pb.py: same * schema.py (TupleConstraint.maxSize): don't use sum() (AttributeDictConstraint.maxSize): same (makeConstraint): in 2.2, 'bool' is a function, not a type, and there is no types.BooleanType * slicer.py: __future__.generators, and the 'sets' module might not be available (SetSlicer): only define it if 'sets' is available (SetUnslicer): same * test_banana.py: __future__.generators, 'sets' might not exist, (EncodeFailureTest.failUnlessIn): 2.2 can't do 'str in str', only 'char in str', so use str.find() instead (InboundByteStream2.testConstrainedBool): skip bool constraints unless we have a real BooleanType (ThereAndBackAgain.test_set): skip sets unless they're supported * test_schema.py (ConformTest.testBool): skip on 2.2 (CreateTest.testMakeConstraint): same * test_pb.py: __future__.generators, use str.find() * test_banana.py (DecodeTest.test_ref2): accomodate python2.4, which doesn't try to be quite as clever as python2.3 when comparing complex object graphs with == (DecodeTest.test_ref5): same. Do the comparison by hand. (DecodeTest.test_ref6): same, big gnarly validation phase * test_pb.py (TestReferenceUnslicer.testNoInterfaces): update to new signature for receiveClose() (TestReferenceUnslicer.testInterfaces): same (TestCall.testFail1): deferredError doesn't seem to like CopiedFailure all that much. Use retrial's return-a-deferred support instead. (MyRemoteCopy3Unslicer.receiveClose): same (TestCall.testFail2): same (TestCall.testFail3): same (TestFactory): clean up both server and client sockets, to avoid the "unclean reactor" warning from trial (Test3Way.tearDown): clean up client sockets * tokens.py (receiveClose): fix documentation * pb.py (CopiedFailure): make CopiedFailure old-style, since you can't raise new-style instances as exceptions, and CopiedFailure may have its .trap() method invoked, which does 'raise self'. (CopiedFailure.__str__): make it clear that this is a CopiedFailure, not a normal Failure. (callRemoteURL_TCP): Add a _gotReferenceCallback argument, to allow test cases to clean up their client connections. * flavors.py (RemoteCopyOldStyle): add an old-style base class, so CopiedFailure can be old-style. Make RemoteCopy a new-style derivative. * test_banana.py (DecodeTest.test_instance): fix the manually-constructed class names to reflect their new location in the tree (test_banana to twisted.pb.test.test_banana) (EncodeFailureTest.test_instance_unsafe): same * twisted/pb/*: move newpb from Sandbox/warner into the 'newpb' branch, distributed out in twisted/pb/ and doc/pb/ * twisted/pb: add __init__.py files to make it a real module * twisted/pb/test/test_*.py: fix up import statements 2005-03-22 Brian Warner <warner@twistedmatrix.com> * flavors.py: implement new signature * pb.py: same * test_pb.py: same * test_banana.py (BrokenDictUnslicer.receiveClose): new signature (ErrorfulUnslicer.receiveChild): same (ErrorfulUnslicer.receiveClose): same (FailingUnslicer.receiveChild): same * slicer.py: implement new receiveChild/receiveClose signature. Require that ready_deferred == None for now. (ListUnslicer.receiveChild): put "placeholder" in the list instead of the Deferred (TupleUnslicer.start): change the way we keep track of not-yet-constructable tuples, using a counter of unreferenceable children instead of counting the Deferred placeholders in the list (TupleUnslicer.receiveChild): put "placeholder" in the list instead of the Deferred * banana.py (Banana.reportReceiveError): when debugging, log the exception in a way that doesn't cause trial to think the test failed. (Banana.handleToken): implement new receiveChild signature (Banana.handleClose): same * debug.py (LoggingBananaMixin.handleToken): same * tokens.py (IUnslicer.receiveChild): new signature for receiveClose and receiveChild, they now pass a pair of (obj, ready_deferred), where obj is still object-or-deferred, but ready_deferred is non-None when the object will not be ready to use until some other event takes place (like a "slow" global reference is established). # Local Variables: # add-log-time-format: add-log-iso8601-time-string # End: