<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01//EN"> <html> <head> <title>Xapian: Remote Backend Protocol</title> </head> <body bgcolor="white" text="black"> <h1>Remote Backend Protocol</h1> <p> This document describes <em>version 30.5</em> of the protocol used by Xapian's remote backend. Clients and servers must support matching major protocol versions and the client's minor protocol version must be the same or lower. This means that for a minor protocol version change, you can upgrade first servers and then clients and everything should work during the upgrades. </p> <p> The protocol assumes a reliable two-way connection across which arbitrary data can be sent - this could be provided by a TCP socket for example (as it is with xapian-tcpsrv), but any such connection could be used. For example, you could used xapian-progsrv across an ssh connection, or even a custom server across a suitable serial connection. </p> <p> All messages start with a single byte identifying code. A message from client to server has a <code>MSG_XXX</code> identifying code, while a message from server to client has a <code>REPLY_XXX</code> identifying code (but note that a reply might not actually be in response to a message - REPLY_GREETING isn't - and some messages result in multiple replies). </p> <p> The identifying code is followed by the encoded length of the contents followed by the contents themselves. </p> <p> Inside the contents, strings are generally passed as an encoded length followed by the string data (this is indicated below by <code>L<...></code>) except when the string is the last or only thing in the contents in which case we know the length because we know the length of the contents so we don't need to explicitly specify it. </p> <p> Integers are encoded using the same encoding used for string lengths (indicated by <code>I<...></code> below). </p> <p> Floating pointing values are passed using a bit packed encoding of the sign and exponent and a base-256 encoding of the mantissa which avoids any rounding issues (assuming that both machines have FLT_RADIX equal to some power of 2). This is indicated by <code>F<...></code> below. </p> <p> Boolean values are passed as a single byte which is the ASCII character value for <code>0</code> or <code>1</code>. This is indicated by <code>B<...></code> below. </p> <h2>Server greeting and statistics</h2> <ul> <li> <code>REPLY_GREETING <protocol major version> <protocol minor version> I<db doc count> I<last docid> B<has positions?> F<db average length></code> </ul> <p> The protocol major and minor versions are passed as a single byte each (e.g. <code>'\x1e\x01'</code> for version 30.1). The server and client must understand the same protocol major version, and the server protocol minor version must be greater than or equal to that of the client (this means that the server understands newer MSG_<i>XXX</i>, but will only send newer REPLY_<i>YYY</i> in response to an appropriate client message. </p> <h2>Exception</h2> <ul> <li> <code>REPLY_EXCEPTION <serialised Xapian::Error object></code> </ul> <p> If an unknown exception is caught by the server, this message is sent but with empty contents. </p> <p> This message can be sent at any point - the serialised exception is unserialised by the client and thrown. The server and client both abort any current sequence of messages. </p> <h2>All Terms</h2> <ul> <li> <code>MSG_ALLTERMS</code> <li> <code>REPLY_ALLTERMS I<term freq> L<term name></code> <li> <code>...</code> <li> <code>REPLY_DONE</code> </ul> <h2>Term Exists</h2> <ul> <li> <code>MSG_TERMEXISTS <term name></code> <li> <code>REPLY_TERMEXISTS</code> or <code>REPLY_TERMDOESNTEXIST</code> </ul> <h2>Term Frequency</h2> <ul> <li> <code>MSG_TERMFREQ <term name></code> <li> <code>REPLY_TERMFREQ I<term freq></code> </ul> <h2>Collection Frequency</h2> <ul> <li> <code>MSG_COLLFREQ <term name></code> <li> <code>REPLY_COLLFREQ I<collection freq></code> </ul> <h2>Document</h2> <ul> <li> <code>MSG_DOCUMENT I<document id></code> <li> <code>REPLY_DOCDATA L<document data></code> <li> <code>REPLY_VALUE I<value no> <value></code> <li> <code>...</code> <li> <code>REPLY_DONE</code> </ul> <h2>Document Length</h2> <ul> <li> <code>MSG_DOCLENGTH I<document id></code> <li> <code>REPLY_DOCLENGTH F<document length></code> </ul> <h2>Keep Alive</h2> <ul> <li> <code>MSG_KEEPALIVE</code> <li> <code>REPLY_DONE</code> </ul> <h2>Reopen</h2> <ul> <li> <code>MSG_REOPEN</code> <li> <code>REPLY_UPDATE I<db doc count> I<last docid> B<has positions?> F<db average length></code> </ul> <p>The reply is the same as for <code>MSG_UPDATE</code>.</p> <h2>Query</h2> <ul> <li> <code>MSG_QUERY L<serialised Xapian::Query object> I<query length> I<collapse key number> <docid order> I<sort key number> <sort by> B<sort value forward> <percent cutoff> F<weight cutoff> <serialised Xapian::Weight object> <serialised Xapian::RSet object></code> <li> <code>REPLY_STATS <serialised Stats object></code> <li> <code>MSG_GETMSET I<first> I<max items> I<check at least> <serialised global Stats object></code> <li> <code>REPLY_RESULTS <serialised Xapian::MSet object> <percent_factor></code> </ul> <p> (Instead of MSG_GETMSET and REPLY_RESULTS, clients running protocol 30.3 or 30.4 have: </p> <ul> <li> <code>MSG_GETMSET_PRE_30_5 I<first> I<max items> I<check at least> <serialised global Stats object></code> <li> <code>REPLY_RESULTS_PRE_30_5 <serialised Xapian::MSet object></code> </li> </ul> <p> and clients running protocol 30.2 or earlier send: </p> <ul> <li> <code>MSG_GETMSET_PRE_30_3 I<first> I<max items> <serialised global Stats object></code> <li> <code>REPLY_RESULTS_PRE_30_5 <serialised Xapian::MSet object></code> </li> </ul> <p>).</p> <p>docid order is <code>'0'</code>, <code>'1'</code> or <code>'2'</code>.</p> <p>sort by is <code>'0'</code>, <code>'1'</code>, <code>'2'</code> or <code>'3'</code>.</p> <h2>Termlist</h2> <ul> <li> <code>MSG_TERMLIST I<document id></code> <li> <code>REPLY_DOCLENGTH F<document length></code> <li> <code>REPLY_TERMLIST I<wdf> I<term freq> L<term name></code> <li> <code>...</code> <li> <code>REPLY_DONE</code> </ul> <h2>Positionlist</h2> <ul> <li> <code>MSG_POSITIONLIST I<document id> <term name></code> <li> <code>REPLY_POSITIONLIST I<termpos delta - 1></code> <li> <code>...</code> <li> <code>REPLY_DONE</code> </ul> <p> Since positions must be strictly monotonically increasing, we encode <tt>(pos - lastpos - 1)</tt> so that small differences between large position values can still be encoded compactly. The first position is encoded as its true value. </p> <h2>Postlist</h2> <ul> <li> <code>MSG_POSTLIST <term name></code> <li> <code>REPLY_POSTLISTSTART I<termfreq> I<collfreq></code> <li> <code>REPLY_POSTLISTITEM I<docid delta - 1> I<wdf> F<document length></code> <li> <code>...</code> <li> <code>REPLY_DONE</code> </ul> <p> Since document IDs in postlists must be strictly monotonically increasing, we encode <tt>(docid - lastdocid - 1)</tt> so that small differences between large document IDs can still be encoded compactly. The first document ID is encoded as its true value - 1 (since document IDs are always > 0). </p> <h2>Shut Down</h2> <ul> <li> <code>MSG_SHUTDOWN</code> </ul> <p> No reply is sent - this message signals that the client has ended the session. </p> <h2>Update</h2> <ul> <li> <code>MSG_UPDATE</code> <li> <code>REPLY_UPDATE I<db doc count> I<last docid> B<has positions?> F<db average length></code> </ul> <p> Only useful for a <code>WritableDatabase</code> (since the same statistics are sent when the connection is initiated in the <code>REPLY_GREETING</code> and they don't change if the database can't change). </p> <h2>Add document</h2> <ul> <li> <code>MSG_ADDDOCUMENT <serialised Xapian::Document object></code> <li> <code>REPLY_ADDDOCUMENT I<document id></code> </ul> <h2>Delete document</h2> <ul> <li> <code>MSG_DELETEDOCUMENT I<document id></code> <li> <code>REPLY_DONE</code> </ul> <h2>Delete document (for compatibility with clients using protocols < 30.2)</h2> <ul> <li> <code>MSG_DELETEDOCUMENT_PRE_30_2 I<document id></code> </ul> <h2>Delete document by term</h2> <ul> <li> <code>MSG_DELETEDOCUMENTTERM <term name></code> </ul> <h2>Replace document</h2> <ul> <li> <code>MSG_REPLACEDOCUMENT I<document id> <serialised Xapian::Document object></code> </ul> <h2>Replace document by term</h2> <ul> <li> <code>MSG_REPLACEDOCUMENTTERM L<term name> <serialised Xapian::Document object></code> </ul> <h2>Cancel</h2> <ul> <li> <code>MSG_CANCEL</code> </ul> <h2>Flush</h2> <ul> <li> <code>MSG_FLUSH</code> <li> <code>REPLY_DONE</code> </ul> </body> </html>
