<html xmlns="http://www.w3.org/1999/xhtml"> <head> <title>Using Pass-By-Copy in Foolscap</title> <style src="stylesheet-unprocessed.css"></style> </head> <body> <h1>Using Pass-By-Copy in Foolscap</h1> <p>Certain objects (including subclasses of <code class="API">foolscap.Copyable</code> and things for which an <code class="API" base="foolscap.copyable">ICopyable</code> adapter has been registered) are serialized using copy-by-value semantics. Each such object is serialized as a (copytype, state) pair of values. On the receiving end, the "copytype" is looked up in a table to find a suitable deserializer. The "state" information is passed to this deserializer to create a new instance that corresponds to the original. Note that the sending and receiving ends are under no obligation to use the same class on each side: it is fairly common for the remote form of an object to have different methods than the original instance.</p> <p>Copy-by-value (as opposed to copy-by-reference) means that the remote representation of an object leads an independent existence, unconnected to the original. Sending the same object multiple times will result in separate independent copies. Sending the result of a pass-by-copy operation back to the original sender will, at best, result in the sender holding two separate objects containing similar state (and at worst will not work at all: not all RemoteCopies are themselves Copyable).</p> <p>More complex copy semantics can be accomplished by writing custom Slicer code. For example, to get an object that is copied by value the first time it traverses the wire, and then copied by reference all later times, you will need to write a Slicer/Unslicer pair to implement this functionality. Likewise the oldpb <code>Cacheable</code> class would need to be implemented with a custom Slicer/Unslicer pair.</p> <h2>Copyable</h2> <p>The easiest way to send your own classes over the wire is to use <code>Copyable</code>. On the sending side, this requires two things: your class must inherit from <code class="API">foolscap.Copyable</code>, and it must define an attribute named <code>typeToCopy</code> with a unique string. This copytype string is shared between both sides, so it is a good idea to use a stable and globally unique value: perhaps a URL rooted in a namespace that you control, or a UUID, or perhaps the fully-qualified package+module+class name of the class being serialized. Any string will do, as long as it matches the one used on the receiving side.</p> <p>The object being sent is asked to provide a state dictionary by calling its <code class="API" base="foolscap.copyable.ICopyable">getStateToCopy</code> method. The default implementation of <code>getStateToCopy</code> will simply return <code>self.__dict__</code>. You can override <code>getStateToCopy</code> to control what pieces of the source object get copied to the target. In particular, you may want to override <code>getStateToCopy</code> if there is any portion of the object's state that should <b>not</b> be sent over the wire: references to objects that can not or should not be serialized, or things that are private to the application. It is common practice to create an empty dictionary in this method and then copy items into it.</p> <p>On the receiving side, you must register the copytype and provide a function to deserialize the state dictionary back into an instance. For each <code>Copyable</code> subclass you will create a corresponding <code class="API" base="foolscap">RemoteCopy</code> subclass. There are three requirements which must be fulfilled by this subclass:</p> <ol> <li><code>copytype</code>: Each <code>RemoteCopy</code> needs a <code>copytype</code> attribute which contains the same string as the corresponding <code>Copyable</code>'s <code>typeToCopy</code> attribute. (metaclass magic is used to auto-register the <code>RemoteCopy</code> class in the global copytype-to-RemoteCopy table when the class is defined. You can also use <code class="API">foolscap.registerRemoteCopy</code> to manually register a class).</li> <li><code>__init__</code>: The <code>RemoteCopy</code> subclass must have an __init__ method that takes no arguments. When the receiving side is creating the incoming object, it starts by creating a new instance of the correct <code>RemoteCopy</code> subclass, and at this point it has no arguments to work with. Later, once the instance is created, it will call <code>setCopyableState</code> to populate it.</li> <li><code>setCopyableState</code>: Your <code>RemoteCopy</code> subclass must define a method named <code class="API" base="foolscap.copyable.IRemoteCopy">setCopyableState</code>. This method will be called with the state dictionary that came out of <code>getStateToCopy</code> on the sending side, and is expected to set any necessary internal state.</li> </ol> <p>Note that <code>RemoteCopy</code> is a new-style class: if you want your copies to be old-style classes, inherit from <code>RemoteCopyOldStyle</code> and manually register the copytype-to-subclass mapping with <code>registerRemoteCopy</code>.</p> <a href="listings/copyable-send.py" class="py-listing" skipLines="2">copyable-send.py</a> <a href="listings/copyable-receive.py" class="py-listing" skipLines="2">copyable-receive.py</a> <h2>Registering Copiers to serialize third-party classes</h2> <p>If you wish to serialize instances of third-party classes that are out of your control (or you simply want to avoid subclassing), you can register a Copier to provide serialization mechanisms for those instances.</p> <p>There are plenty of cases where it is difficult to arrange for all of the data you send over the wire to be in the form of <code>Copyable</code> subclasses. For example, you might have a codebase that produces a deeply-nested data structure that contains instances of pre-existing classes. Those classes are written by other people, and do not happen to inherit from <code>Copyable</code>. Without Copiers, you would have to traverse the whole structure, locate all instances of these non-<code>Copyable</code> classes, and wrap them in some new <code>Copyable</code> subclass. Registering a Copier for the third-party class is much easier.</p> <p>The <code class="API" base="foolscap.copyable">registerCopier</code> function is used to provide a "copier" for any given class. This copier is a function that accepts an instance of the given class, and returns a (copytype, state) tuple. For example<span class="footnote">many thanks to Ricky Iacovou for the xmlrpclib.DateTime example</span>, the xmlrpclib module provides a <code>DateTime</code> class, and you might have a data structure that includes some instances of them:</p> <pre class="python"> import xmlrpclib from foolscap import registerCopier def copy_DateTime(xd): return ("_xmlrpclib_DateTime", {"value": xd.value}) registerCopier(xmlrpclib.DateTime, copy_DateTime) </pre> <p>This insures that any <code>xmlrpclib.DateTime</code> that is encountered while serializing arguments or return values will be serialized with a copytype of "_xmlrpclib_DateTime" and a state dictionary containing the single "value" key. Even <code>DateTime</code> instances that appear arbitrarily deep inside nested data structures will be serialized this way. For example, one a method argument might be dictionary, and one of its keys was a list, and that list could containe a <code>DateTime</code> instance.</p> <p>To deserialize this object, the receiving side needs to register a corresponding deserializer. <code class="API" base="foolscap.copyable">registerRemoteCopyFactory</code> is the receiving-side parallel to <code>registerCopier</code>. It associates a copytype with a function that will receive a state dictionary and is expected to return a fully-formed instance. For example:</p> <pre class="python"> import xmlrpclib from foolscap import registerRemoteCopyFactory def make_DateTime(state): return xmlrpclib.DateTime(state["value"]) registerRemoteCopyFactory("_xmlrpclib_DateTime", make_DateTime) </pre> <p>Note that the "_xmlrpclib_DateTime" copytype <b>must</b> be the same for both the copier and the RemoteCopyFactory, otherwise the receiving side will be unable to locate the correct deserializer.</p> <p>It is perfectly reasonable to include both of these function/registration pairs in the same module, and import it in the code on both sides of the wire. The examples describe the sending and receiving sides separately to emphasize the fact that the recipient may be running completely different code than the sender.</p> <h2>Registering ICopyable adapters</h2> <p>A slightly more generalized way to teach Foolscap about third-party classes is to register an <code class="API" base="foolscap.copyable">ICopyable</code> adapter for them, using the usual (i.e. zope.interface) adapter-registration mechanism. The object that provide <code>ICopyable</code> needs to implement two methods: <code>getTypeToCopy</code> (which returns the copytype), and <code>getStateToCopy</code>, which returns the state dictionary. Any object which can be adapted to <code>ICopyable</code> can be serialized this way.</p> <p>On the receiving side, the copytype is looked up in the <code>CopyableRegistry</code> to find a corresponding UnslicerFactory. The <code>registerRemoteCopyUnslicerFactory</code> function accepts two arguments: the copytype, and the unslicer factory to use. This unslicer factory is simply a function that takes no arguments and returns a new Unslicer. Each time an inbound message with the matching copytype is received, ths unslicer factory is invoked to create an Unslicer that will be responsible for the single instance described in the message. This Unslicer must implement an interface described in the <a href="specifications/pb.xhtml">Unslicer specifications</a>.</p> <h2>Registering ISlicer adapters</h2> <p>The most generalized way to serialize classes is to register a whole <code>ISlicer</code> adapter for them. The <code>ISlicer</code> gets complete control over serialization: it can stall the production of tokens by implementing a <code>slice</code> method that yields Deferreds instead of basic objects. It can also interact with other objects while the target is being serialized. As an extreme example, if you had a service that wanted to migrate an open HTTP connection from one process to another, the <code>ISlicer</code> could communication with a front-end load-balancing box to redirect the connection to the new host. In this case, the slicer could theoretically tell the load-balancer to pause the connection and assign it a rendezvous number, then serialize this rendezvous number as a form of "claim check" to the target process. The <code>IUnslicer</code> on the receiving end could open a new listening port, then use the claim check to tell the load-balancer to direct the connection to this new port. Likewise two services running on the same host could conspire to pass open file descriptors over a Foolscap connection (via an auxilliary unix-domain socket) through suitable magic in the <code>ISlicer</code> and <code>IUnslicer</code> on each end.</p> <p>The Slicers and Unslicers are described in more detail in the <a href="specifications/pb.xhtml">specifications</a>.</p> <p>Note that a <code>Copyable</code> with a copytype of "foo" is serialized as the following token stream: OPEN, "copyable", "foo", [state dictionary..], CLOSE. Any <code>ISlicer</code> adapter which wishes to match what <code>Copyable</code> does needs to include the extra "copyable" opentype string first.</p> <p>Also note that using a custom Slicer introduces an opportunity to violate serialization coherency. <code>Copyable</code> and Copiers transform the original object into a state dictionary in one swell foop, not allowing any other code to get control (and possibly mutate the object's state). If your custom Slicer allows other code to get control during serialization, then the object's state might be changed, and thus the serialized state dictionary could wind up looking pretty weird.</p> </body></html>