<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd"> <html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en"> <head> <meta http-equiv="Content-Type" content="text/html; charset=utf-8" /> <meta name="generator" content="Docutils 0.4.1: http://docutils.sourceforge.net/" /> <title>Benchmarks and Speed</title> <meta name="author" content="Stefan Behnel" /> <meta content="Performance evaluation of lxml and ElementTree" name="description" /> <meta content="lxml performance, lxml.etree, lxml.objectify, benchmarks, ElementTree" name="keywords" /> <link rel="stylesheet" href="style.css" type="text/css" /> </head> <body> <div class="document" id="benchmarks-and-speed"> <div class="sidemenu"><ul id="lxml"><li><span class="section title">lxml</span><ul class="menu foreign" id="index"><li class="menu title"><a href="index.html">lxml</a><ul class="submenu"><li class="menu item"><a href="index.html#introduction">Introduction</a></li><li class="menu item"><a href="index.html#documentation">Documentation</a></li><li class="menu item"><a href="index.html#download">Download</a></li><li class="menu item"><a href="index.html#mailing-list">Mailing list</a></li><li class="menu item"><a href="index.html#bug-tracker">Bug tracker</a></li><li class="menu item"><a href="index.html#license">License</a></li><li class="menu item"><a href="index.html#old-versions">Old Versions</a></li></ul></li></ul><ul class="menu foreign" id="intro"><li class="menu title"><a href="intro.html">Why lxml?</a><ul class="submenu"><li class="menu item"><a href="intro.html#motto">Motto</a></li><li class="menu item"><a href="intro.html#aims">Aims</a></li></ul></li></ul><ul class="menu foreign" id="installation"><li class="menu title"><a href="installation.html">Installing lxml</a><ul class="submenu"><li class="menu item"><a href="installation.html#requirements">Requirements</a></li><li class="menu item"><a href="installation.html#installation">Installation</a></li><li class="menu item"><a href="installation.html#building-lxml-from-sources">Building lxml from sources</a></li><li class="menu item"><a href="installation.html#ms-windows">MS Windows</a></li><li class="menu item"><a href="installation.html#macos-x">MacOS-X</a></li></ul></li></ul><ul class="menu foreign" id="lxml2"><li class="menu title"><a href="lxml2.html">What's new in lxml 2.0?</a><ul class="submenu"><li class="menu item"><a href="lxml2.html#changes-in-etree-and-objectify">Changes in etree and objectify</a></li><li class="menu item"><a href="lxml2.html#new-modules">New modules</a></li></ul></li></ul><ul class="menu current" id="performance"><li class="menu title"><a href="performance.html">Benchmarks and Speed</a><ul class="submenu"><li class="menu item"><a href="performance.html#general-notes">General notes</a></li><li class="menu item"><a href="performance.html#how-to-read-the-timings">How to read the timings</a></li><li class="menu item"><a href="performance.html#parsing-and-serialising">Parsing and Serialising</a></li><li class="menu item"><a href="performance.html#the-elementtree-api">The ElementTree API</a></li><li class="menu item"><a href="performance.html#xpath">XPath</a></li><li class="menu item"><a href="performance.html#a-longer-example">A longer example</a></li><li class="menu item"><a href="performance.html#lxml-objectify">lxml.objectify</a></li></ul></li></ul><ul class="menu foreign" id="compatibility"><li class="menu title"><a href="compatibility.html">ElementTree compatibility of lxml.etree</a></li></ul><ul class="menu foreign" id="FAQ"><li class="menu title"><a href="FAQ.html">lxml FAQ - Frequently Asked Questions</a><ul class="submenu"><li class="menu item"><a href="FAQ.html#general-questions">General Questions</a></li><li class="menu item"><a href="FAQ.html#installation">Installation</a></li><li class="menu item"><a href="FAQ.html#contributing">Contributing</a></li><li class="menu item"><a href="FAQ.html#bugs">Bugs</a></li><li class="menu item"><a href="FAQ.html#threading">Threading</a></li><li class="menu item"><a href="FAQ.html#parsing-and-serialisation">Parsing and Serialisation</a></li><li class="menu item"><a href="FAQ.html#xpath-and-document-traversal">XPath and Document Traversal</a></li></ul></li></ul></li></ul><ul id="Developing with lxml"><li><span class="section title">Developing with lxml</span><ul class="menu foreign" id="tutorial"><li class="menu title"><a href="tutorial.html">The lxml.etree Tutorial</a><ul class="submenu"><li class="menu item"><a href="tutorial.html#the-element-class">The Element class</a></li><li class="menu item"><a href="tutorial.html#the-elementtree-class">The ElementTree class</a></li><li class="menu item"><a href="tutorial.html#parsing-from-strings-and-files">Parsing from strings and files</a></li><li class="menu item"><a href="tutorial.html#namespaces">Namespaces</a></li><li class="menu item"><a href="tutorial.html#the-e-factory">The E-factory</a></li><li class="menu item"><a href="tutorial.html#elementpath">ElementPath</a></li></ul></li></ul><ul class="menu foreign" id="api index"><li class="menu title"><a href="api/index.html">API reference</a></li></ul><ul class="menu foreign" id="api"><li class="menu title"><a href="api.html">APIs specific to lxml.etree</a><ul class="submenu"><li class="menu item"><a href="api.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="api.html#other-element-apis">Other Element APIs</a></li><li class="menu item"><a href="api.html#trees-and-documents">Trees and Documents</a></li><li class="menu item"><a href="api.html#iteration">Iteration</a></li><li class="menu item"><a href="api.html#error-handling-on-exceptions">Error handling on exceptions</a></li><li class="menu item"><a href="api.html#error-logging">Error logging</a></li><li class="menu item"><a href="api.html#serialisation">Serialisation</a></li><li class="menu item"><a href="api.html#xinclude-and-elementinclude">XInclude and ElementInclude</a></li><li class="menu item"><a href="api.html#write-c14n-on-elementtree">write_c14n on ElementTree</a></li></ul></li></ul><ul class="menu foreign" id="parsing"><li class="menu title"><a href="parsing.html">Parsing XML and HTML with lxml</a><ul class="submenu"><li class="menu item"><a href="parsing.html#parsers">Parsers</a></li><li class="menu item"><a href="parsing.html#the-target-parser-interface">The target parser interface</a></li><li class="menu item"><a href="parsing.html#the-feed-parser-interface">The feed parser interface</a></li><li class="menu item"><a href="parsing.html#iterparse-and-iterwalk">iterparse and iterwalk</a></li><li class="menu item"><a href="parsing.html#python-unicode-strings">Python unicode strings</a></li></ul></li></ul><ul class="menu foreign" id="validation"><li class="menu title"><a href="validation.html">Validation with lxml</a><ul class="submenu"><li class="menu item"><a href="validation.html#validation-at-parse-time">Validation at parse time</a></li><li class="menu item"><a href="validation.html#dtd">DTD</a></li><li class="menu item"><a href="validation.html#relaxng">RelaxNG</a></li><li class="menu item"><a href="validation.html#xmlschema">XMLSchema</a></li><li class="menu item"><a href="validation.html#schematron">Schematron</a></li></ul></li></ul><ul class="menu foreign" id="xpathxslt"><li class="menu title"><a href="xpathxslt.html">XPath and XSLT with lxml</a><ul class="submenu"><li class="menu item"><a href="xpathxslt.html#xpath">XPath</a></li><li class="menu item"><a href="xpathxslt.html#xslt">XSLT</a></li></ul></li></ul><ul class="menu foreign" id="objectify"><li class="menu title"><a href="objectify.html">lxml.objectify</a><ul class="submenu"><li class="menu item"><a href="objectify.html#the-lxml-objectify-api">The lxml.objectify API</a></li><li class="menu item"><a href="objectify.html#asserting-a-schema">Asserting a Schema</a></li><li class="menu item"><a href="objectify.html#objectpath">ObjectPath</a></li><li class="menu item"><a href="objectify.html#python-data-types">Python data types</a></li><li class="menu item"><a href="objectify.html#how-data-types-are-matched">How data types are matched</a></li><li class="menu item"><a href="objectify.html#what-is-different-from-lxml-etree?">What is different from lxml.etree?</a></li></ul></li></ul><ul class="menu foreign" id="lxmlhtml"><li class="menu title"><a href="lxmlhtml.html">lxml.html</a><ul class="submenu"><li class="menu item"><a href="lxmlhtml.html#parsing-html">Parsing HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-element-methods">HTML Element Methods</a></li><li class="menu item"><a href="lxmlhtml.html#running-html-doctests">Running HTML doctests</a></li><li class="menu item"><a href="lxmlhtml.html#creating-html-with-the-e-factory">Creating HTML with the E-factory</a></li><li class="menu item"><a href="lxmlhtml.html#working-with-links">Working with links</a></li><li class="menu item"><a href="lxmlhtml.html#forms">Forms</a></li><li class="menu item"><a href="lxmlhtml.html#cleaning-up-html">Cleaning up HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-diff">HTML Diff</a></li><li class="menu item"><a href="lxmlhtml.html#examples">Examples</a></li></ul></li></ul><ul class="menu foreign" id="cssselect"><li class="menu title"><a href="cssselect.html">lxml.cssselect</a><ul class="submenu"><li class="menu item"><a href="cssselect.html#the-cssselector-class">The CSSSelector class</a></li><li class="menu item"><a href="cssselect.html#css-selectors">CSS Selectors</a></li><li class="menu item"><a href="cssselect.html#limitations">Limitations</a></li></ul></li></ul><ul class="menu foreign" id="elementsoup"><li class="menu title"><a href="elementsoup.html">BeautifulSoup Parser</a><ul class="submenu"><li class="menu item"><a href="elementsoup.html#entity-handling">Entity handling</a></li><li class="menu item"><a href="elementsoup.html#using-soupparser-as-a-fallback">Using soupparser as a fallback</a></li></ul></li></ul></li></ul><ul id="Extending lxml"><li><span class="section title">Extending lxml</span><ul class="menu foreign" id="resolvers"><li class="menu title"><a href="resolvers.html">Document loading and URL resolving</a><ul class="submenu"><li class="menu item"><a href="resolvers.html#resolvers">Resolvers</a></li><li class="menu item"><a href="resolvers.html#document-loading-in-context">Document loading in context</a></li><li class="menu item"><a href="resolvers.html#i-o-access-control-in-xslt">I/O access control in XSLT</a></li></ul></li></ul><ul class="menu foreign" id="extensions"><li class="menu title"><a href="extensions.html">Extension functions for XPath and XSLT</a><ul class="submenu"><li class="menu item"><a href="extensions.html#the-functionnamespace">The FunctionNamespace</a></li><li class="menu item"><a href="extensions.html#global-prefix-assignment">Global prefix assignment</a></li><li class="menu item"><a href="extensions.html#the-xpath-context">The XPath context</a></li><li class="menu item"><a href="extensions.html#evaluators-and-xslt">Evaluators and XSLT</a></li><li class="menu item"><a href="extensions.html#evaluator-local-extensions">Evaluator-local extensions</a></li><li class="menu item"><a href="extensions.html#what-to-return-from-a-function">What to return from a function</a></li></ul></li></ul><ul class="menu foreign" id="element classes"><li class="menu title"><a href="element_classes.html">Using custom Element classes in lxml</a><ul class="submenu"><li class="menu item"><a href="element_classes.html#element-initialization">Element initialization</a></li><li class="menu item"><a href="element_classes.html#setting-up-a-class-lookup-scheme">Setting up a class lookup scheme</a></li><li class="menu item"><a href="element_classes.html#implementing-namespaces">Implementing namespaces</a></li></ul></li></ul><ul class="menu foreign" id="sax"><li class="menu title"><a href="sax.html">Sax support</a><ul class="submenu"><li class="menu item"><a href="sax.html#building-a-tree-from-sax-events">Building a tree from SAX events</a></li><li class="menu item"><a href="sax.html#producing-sax-events-from-an-elementtree-or-element">Producing SAX events from an ElementTree or Element</a></li><li class="menu item"><a href="sax.html#interfacing-with-pulldom-minidom">Interfacing with pulldom/minidom</a></li></ul></li></ul><ul class="menu foreign" id="capi"><li class="menu title"><a href="capi.html">The public C-API of lxml.etree</a><ul class="submenu"><li class="menu item"><a href="capi.html#writing-external-modules-in-cython">Writing external modules in Cython</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-c">Writing external modules in C</a></li></ul></li></ul></li></ul><ul id="Developing lxml"><li><span class="section title">Developing lxml</span><ul class="menu foreign" id="build"><li class="menu title"><a href="build.html">How to build lxml from source</a><ul class="submenu"><li class="menu item"><a href="build.html#cython">Cython</a></li><li class="menu item"><a href="build.html#subversion">Subversion</a></li><li class="menu item"><a href="build.html#setuptools">Setuptools</a></li><li class="menu item"><a href="build.html#running-the-tests-and-reporting-errors">Running the tests and reporting errors</a></li><li class="menu item"><a href="build.html#contributing-an-egg">Contributing an egg</a></li><li class="menu item"><a href="build.html#providing-newer-library-versions-on-mac-os-x">Providing newer library versions on Mac-OS X</a></li><li class="menu item"><a href="build.html#static-linking-on-windows">Static linking on Windows</a></li><li class="menu item"><a href="build.html#building-debian-packages-from-svn-sources">Building Debian packages from SVN sources</a></li></ul></li></ul><ul class="menu foreign" id="lxml source howto"><li class="menu title"><a href="lxml-source-howto.html">How to read the source of lxml</a><ul class="submenu"><li class="menu item"><a href="lxml-source-howto.html#what-is-cython?">What is Cython?</a></li><li class="menu item"><a href="lxml-source-howto.html#where-to-start?">Where to start?</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="lxml-source-howto.html#python-modules">Python modules</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-objectify">lxml.objectify</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-pyclasslookup">lxml.pyclasslookup</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-html">lxml.html</a></li></ul></li></ul><ul class="menu foreign" id="changes 2 0 11"><li class="menu title"><a href="changes-2.0.11.html">Release Changelog</a></li></ul><ul class="menu foreign" id="credits"><li class="menu title"><a href="credits.html">Credits</a><ul class="submenu"><li class="menu item"><a href="credits.html#special-thanks-goes-to:">Special thanks goes to:</a></li></ul></li></ul></li></ul></div><h1 class="title">Benchmarks and Speed</h1> <table class="docinfo" frame="void" rules="none"> <col class="docinfo-name" /> <col class="docinfo-content" /> <tbody valign="top"> <tr><th class="docinfo-name">Author:</th> <td>Stefan Behnel</td></tr> </tbody> </table> <p>As an XML library, lxml.etree is very fast. It is also slow. As with all software, it depends on what you do with it. Rest assured that lxml is fast enough for most applications, so lxml is probably somewhere between 'fast enough' and 'the best choice' for yours. Read some <a class="reference" href="http://permalink.gmane.org/gmane.comp.python.lxml.devel/3250">messages</a> from <a class="reference" href="http://article.gmane.org/gmane.comp.python.lxml.devel/3246">happy</a> <a class="reference" href="http://thread.gmane.org/gmane.comp.python.lxml.devel/3244/focus=3244">users</a> to see what we mean.</p> <p>This text describes where lxml.etree (abbreviated to 'lxe') excels, gives hints on some performance traps and compares the overall performance to the original <a class="reference" href="http://effbot.org/zone/element-index.htm">ElementTree</a> (ET) and <a class="reference" href="http://effbot.org/zone/celementtree.htm">cElementTree</a> (cET) libraries by Fredrik Lundh. The cElementTree library is a fast C-implementation of the original ElementTree.</p> <div class="contents topic"> <p class="topic-title first"><a id="contents" name="contents">Contents</a></p> <ul class="simple"> <li><a class="reference" href="#general-notes" id="id1" name="id1">General notes</a></li> <li><a class="reference" href="#how-to-read-the-timings" id="id2" name="id2">How to read the timings</a></li> <li><a class="reference" href="#parsing-and-serialising" id="id3" name="id3">Parsing and Serialising</a></li> <li><a class="reference" href="#the-elementtree-api" id="id4" name="id4">The ElementTree API</a><ul> <li><a class="reference" href="#child-access" id="id5" name="id5">Child access</a></li> <li><a class="reference" href="#element-creation" id="id6" name="id6">Element creation</a></li> <li><a class="reference" href="#merging-different-sources" id="id7" name="id7">Merging different sources</a></li> <li><a class="reference" href="#deepcopy" id="id8" name="id8">deepcopy</a></li> <li><a class="reference" href="#tree-traversal" id="id9" name="id9">Tree traversal</a></li> </ul> </li> <li><a class="reference" href="#xpath" id="id10" name="id10">XPath</a></li> <li><a class="reference" href="#a-longer-example" id="id11" name="id11">A longer example</a></li> <li><a class="reference" href="#lxml-objectify" id="id12" name="id12">lxml.objectify</a><ul> <li><a class="reference" href="#objectpath" id="id13" name="id13">ObjectPath</a></li> <li><a class="reference" href="#caching-elements" id="id14" name="id14">Caching Elements</a></li> <li><a class="reference" href="#further-optimisations" id="id15" name="id15">Further optimisations</a></li> </ul> </li> </ul> </div> <div class="section"> <h1><a id="general-notes" name="general-notes">General notes</a></h1> <p>First thing to say: there <em>is</em> an overhead involved in having a DOM-like C library mimic the ElementTree API. As opposed to ElementTree, lxml has to generate Python representations of tree nodes on the fly when asked for them, and the internal tree structure of libxml2 results in a higher maintenance overhead than the simpler top-down structure of ElementTree. What this means is: the more of your code runs in Python, the less you can benefit from the speed of lxml and libxml2. Note, however, that this is true for most performance critical Python applications. No one would implement fourier transformations in pure Python when you can use NumPy.</p> <p>The up side then is that lxml provides powerful tools like tree iterators, XPath and XSLT, that can handle complex operations at the speed of C. Their pythonic API in lxml makes them so flexible that most applications can easily benefit from them.</p> </div> <div class="section"> <h1><a id="how-to-read-the-timings" name="how-to-read-the-timings">How to read the timings</a></h1> <p>The statements made here are backed by the (micro-)benchmark scripts <a class="reference" href="http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_etree.py">bench_etree.py</a>, <a class="reference" href="http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_xpath.py">bench_xpath.py</a> and <a class="reference" href="http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_objectify.py">bench_objectify.py</a> that come with the lxml source distribution. They are distributed under the same BSD license as lxml itself, and the lxml project would like to promote them as a general benchmarking suite for all ElementTree implementations. New benchmarks are very easy to add as tiny test methods, so if you write a performance test for a specific part of the API yourself, please consider sending it to the lxml mailing list.</p> <p>The timings cited below compare lxml 2.0 final (with libxml2 2.6.31) to the January 2008 SVN trunk versions of ElementTree (1.3alpha) and cElementTree (1.2.7). They were run single-threaded on a 1.8GHz Intel Core Duo machine under Ubuntu Linux 7.10 (Gutsy). The C libraries were compiled with the same platform specific optimisation flags. The Python interpreter (2.5.1) was used as provided by the distribution.</p> <p>The scripts run a number of simple tests on the different libraries, using different XML tree configurations: different tree sizes (T1-4), with or without attributes (-/A), with or without ASCII string or unicode text (-/S/U), and either against a tree or its serialised XML form (T/X). In the result extracts cited below, T1 refers to a 3-level tree with many children at the third level, T2 is swapped around to have many children below the root element, T3 is a deep tree with few children at each level and T4 is a small tree, slightly broader than deep. If repetition is involved, this usually means running the benchmark in a loop over all children of the tree root, otherwise, the operation is run on the root node (C/R).</p> <p>As an example, the character code <tt class="docutils literal"><span class="pre">(SATR</span> <span class="pre">T1)</span></tt> states that the benchmark was running for tree T1, with plain string text (S) and attributes (A). It was run against the root element (R) in the tree structure of the data (T).</p> <p>Note that very small operations are repeated in integer loops to make them measurable. It is therefore not always possible to compare the absolute timings of, say, a single access benchmark (which usually loops) and a 'get all in one step' benchmark, which already takes enough time to be measurable and is therefore measured as is. An example is the index access to a single child, which cannot be compared to the timings for <tt class="docutils literal"><span class="pre">getchildren()</span></tt>. Take a look at the concrete benchmarks in the scripts to understand how the numbers compare.</p> </div> <div class="section"> <h1><a id="parsing-and-serialising" name="parsing-and-serialising">Parsing and Serialising</a></h1> <p>Serialisation is an area where lxml excels. The reason is that it executes entirely at the C level, without any interaction with Python code. The results are rather impressive, especially for UTF-8, which is native to libxml2. While 20 to 40 times faster than (c)ElementTree 1.2, lxml is still more than 5 times as fast as the much improved ElementTree 1.3:</p> <pre class="literal-block"> lxe: tostring_utf16 (SATR T1) 19.0921 msec/pass cET: tostring_utf16 (SATR T1) 129.8430 msec/pass ET : tostring_utf16 (SATR T1) 136.1301 msec/pass lxe: tostring_utf16 (UATR T1) 20.4630 msec/pass cET: tostring_utf16 (UATR T1) 130.1570 msec/pass ET : tostring_utf16 (UATR T1) 136.3101 msec/pass lxe: tostring_utf16 (S-TR T2) 18.8632 msec/pass cET: tostring_utf16 (S-TR T2) 136.9388 msec/pass ET : tostring_utf16 (S-TR T2) 143.9550 msec/pass lxe: tostring_utf8 (S-TR T2) 14.4310 msec/pass cET: tostring_utf8 (S-TR T2) 137.0859 msec/pass ET : tostring_utf8 (S-TR T2) 144.3110 msec/pass lxe: tostring_utf8 (U-TR T3) 2.6381 msec/pass cET: tostring_utf8 (U-TR T3) 52.1040 msec/pass ET : tostring_utf8 (U-TR T3) 53.1070 msec/pass </pre> <p>For parsing, on the other hand, the advantage is clearly with cElementTree. The (c)ET libraries use a very thin layer on top of the expat parser, which is known to be extremely fast:</p> <pre class="literal-block"> lxe: parse_stringIO (SAXR T1) 144.1851 msec/pass cET: parse_stringIO (SAXR T1) 14.4269 msec/pass ET : parse_stringIO (SAXR T1) 245.9190 msec/pass lxe: parse_stringIO (S-XR T3) 5.6100 msec/pass cET: parse_stringIO (S-XR T3) 5.3229 msec/pass ET : parse_stringIO (S-XR T3) 82.4831 msec/pass lxe: parse_stringIO (UAXR T3) 23.4420 msec/pass cET: parse_stringIO (UAXR T3) 30.2689 msec/pass ET : parse_stringIO (UAXR T3) 165.7169 msec/pass </pre> <p>While about as fast for smaller documents, the expat parser allows cET to be up to 10 times faster than lxml on plain parser performance for large input documents. Similar timings can be observed for the <tt class="docutils literal"><span class="pre">iterparse()</span></tt> function:</p> <pre class="literal-block"> lxe: iterparse_stringIO (SAXR T1) 160.3689 msec/pass cET: iterparse_stringIO (SAXR T1) 19.1891 msec/pass ET : iterparse_stringIO (SAXR T1) 274.8971 msec/pass lxe: iterparse_stringIO (UAXR T3) 24.9629 msec/pass cET: iterparse_stringIO (UAXR T3) 31.7740 msec/pass ET : iterparse_stringIO (UAXR T3) 173.8000 msec/pass </pre> <p>However, if you benchmark the complete round-trip of a serialise-parse cycle, the numbers will look similar to these:</p> <pre class="literal-block"> lxe: write_utf8_parse_stringIO (S-TR T1) 160.0718 msec/pass cET: write_utf8_parse_stringIO (S-TR T1) 207.6778 msec/pass ET : write_utf8_parse_stringIO (S-TR T1) 450.2120 msec/pass lxe: write_utf8_parse_stringIO (UATR T2) 173.5830 msec/pass cET: write_utf8_parse_stringIO (UATR T2) 253.0849 msec/pass ET : write_utf8_parse_stringIO (UATR T2) 519.2261 msec/pass lxe: write_utf8_parse_stringIO (S-TR T3) 8.4269 msec/pass cET: write_utf8_parse_stringIO (S-TR T3) 75.7639 msec/pass ET : write_utf8_parse_stringIO (S-TR T3) 156.1930 msec/pass lxe: write_utf8_parse_stringIO (SATR T4) 1.2100 msec/pass cET: write_utf8_parse_stringIO (SATR T4) 6.4859 msec/pass ET : write_utf8_parse_stringIO (SATR T4) 9.9051 msec/pass </pre> <p>For applications that require a high parser throughput and do little serialization, cET is the best choice. Also for iterparse applications that extract small amounts of data from large XML data sets. If it comes to round-trip performance, however, lxml tends to be between 30% and multiple times faster in total. So, whenever the input documents are not considerably bigger than the output, lxml is the clear winner.</p> </div> <div class="section"> <h1><a id="the-elementtree-api" name="the-elementtree-api">The ElementTree API</a></h1> <p>Since all three libraries implement the same API, their performance is easy to compare in this area. A major disadvantage for lxml's performance is the different tree model that underlies libxml2. It allows lxml to provide parent pointers for elements, but also increases the overhead of tree building and restructuring. This can be seen from the tree setup times of the benchmark (given in seconds):</p> <pre class="literal-block"> lxe: -- S- U- -A SA UA T1: 0.0783 0.0777 0.0774 0.0787 0.0781 0.0783 T2: 0.0799 0.0796 0.0799 0.0879 0.0882 0.0886 T3: 0.0245 0.0216 0.0217 0.0577 0.0575 0.0572 T4: 0.0003 0.0003 0.0003 0.0011 0.0011 0.0011 cET: -- S- U- -A SA UA T1: 0.0272 0.0264 0.0267 0.0268 0.0261 0.0265 T2: 0.0280 0.0274 0.0273 0.0273 0.0276 0.0275 T3: 0.0065 0.0066 0.0065 0.0111 0.0088 0.0088 T4: 0.0001 0.0001 0.0001 0.0001 0.0001 0.0001 ET : -- S- U- -A SA UA T1: 0.1302 0.1903 0.2208 0.1265 0.2542 0.1267 T2: 0.2994 0.1301 0.3402 0.3746 0.1326 0.4170 T3: 0.0301 0.0310 0.0302 0.0348 0.3654 0.0349 T4: 0.0006 0.0005 0.0008 0.0006 0.0007 0.0006 </pre> <p>While lxml is still faster than ET in most cases (10-70%), cET can be up to three times faster than lxml here. One of the reasons is that lxml must additionally discard the created Python elements after their use, when they are no longer referenced. ET and cET represent the tree itself through these objects, which reduces the overhead in creating them.</p> <div class="section"> <h2><a id="child-access" name="child-access">Child access</a></h2> <p>The same reason makes operations like collecting children as in <tt class="docutils literal"><span class="pre">list(element)</span></tt> more costly in lxml. Where ET and cET can quickly create a shallow copy of their list of children, lxml has to create a Python object for each child and collect them in a list:</p> <pre class="literal-block"> lxe: root_list_children (--TR T1) 0.0160 msec/pass cET: root_list_children (--TR T1) 0.0081 msec/pass ET : root_list_children (--TR T1) 0.0541 msec/pass lxe: root_list_children (--TR T2) 0.2100 msec/pass cET: root_list_children (--TR T2) 0.0319 msec/pass ET : root_list_children (--TR T2) 0.4420 msec/pass </pre> <p>This handicap is also visible when accessing single children:</p> <pre class="literal-block"> lxe: first_child (--TR T2) 0.2429 msec/pass cET: first_child (--TR T2) 0.2170 msec/pass ET : first_child (--TR T2) 0.9968 msec/pass lxe: last_child (--TR T1) 0.2470 msec/pass cET: last_child (--TR T1) 0.2291 msec/pass ET : last_child (--TR T1) 0.9830 msec/pass </pre> <p>... unless you also add the time to find a child index in a bigger list. ET and cET use Python lists here, which are based on arrays. The data structure used by libxml2 is a linked tree, and thus, a linked list of children:</p> <pre class="literal-block"> lxe: middle_child (--TR T1) 0.2759 msec/pass cET: middle_child (--TR T1) 0.2229 msec/pass ET : middle_child (--TR T1) 1.0030 msec/pass lxe: middle_child (--TR T2) 1.7071 msec/pass cET: middle_child (--TR T2) 0.2229 msec/pass ET : middle_child (--TR T2) 0.9930 msec/pass </pre> </div> <div class="section"> <h2><a id="element-creation" name="element-creation">Element creation</a></h2> <p>As opposed to ET, libxml2 has a notion of documents that each element must be in. This results in a major performance difference for creating independent Elements that end up in independently created documents:</p> <pre class="literal-block"> lxe: create_elements (--TC T2) 2.8961 msec/pass cET: create_elements (--TC T2) 0.1929 msec/pass ET : create_elements (--TC T2) 1.3590 msec/pass </pre> <p>Therefore, it is always preferable to create Elements for the document they are supposed to end up in, either as SubElements of an Element or using the explicit <tt class="docutils literal"><span class="pre">Element.makeelement()</span></tt> call:</p> <pre class="literal-block"> lxe: makeelement (--TC T2) 1.9000 msec/pass cET: makeelement (--TC T2) 0.3211 msec/pass ET : makeelement (--TC T2) 1.6358 msec/pass lxe: create_subelements (--TC T2) 1.7891 msec/pass cET: create_subelements (--TC T2) 0.2351 msec/pass ET : create_subelements (--TC T2) 3.2270 msec/pass </pre> <p>So, if the main performance bottleneck of an application is creating large XML trees in memory through calls to Element and SubElement, cET is the best choice. Note, however, that the serialisation performance may even out this advantage, especially for smaller trees and trees with many attributes.</p> </div> <div class="section"> <h2><a id="merging-different-sources" name="merging-different-sources">Merging different sources</a></h2> <p>A critical action for lxml is moving elements between document contexts. It requires lxml to do recursive adaptations throughout the moved tree structure.</p> <p>The following benchmark appends all root children of the second tree to the root of the first tree:</p> <pre class="literal-block"> lxe: append_from_document (--TR T1,T2) 3.3841 msec/pass cET: append_from_document (--TR T1,T2) 0.2699 msec/pass ET : append_from_document (--TR T1,T2) 1.2650 msec/pass lxe: append_from_document (--TR T3,T4) 0.0441 msec/pass cET: append_from_document (--TR T3,T4) 0.0169 msec/pass ET : append_from_document (--TR T3,T4) 0.0820 msec/pass </pre> <p>Although these are fairly small numbers compared to parsing, this easily shows the different performance classes for lxml and (c)ET. Where the latter do not have to care about parent pointers and tree structures, lxml has to deep traverse the appended tree. The performance difference therefore increases with the size of the tree that is moved.</p> <p>This difference is not always as visible, but applies to most parts of the API, like inserting newly created elements:</p> <pre class="literal-block"> lxe: insert_from_document (--TR T1,T2) 5.7020 msec/pass cET: insert_from_document (--TR T1,T2) 0.4041 msec/pass ET : insert_from_document (--TR T1,T2) 1.4789 msec/pass </pre> <p>or replacing the child slice by a newly created element:</p> <pre class="literal-block"> lxe: replace_children_element (--TC T1) 0.2210 msec/pass cET: replace_children_element (--TC T1) 0.0238 msec/pass ET : replace_children_element (--TC T1) 0.1600 msec/pass </pre> <p>as opposed to replacing the slice with an existing element from the same document:</p> <pre class="literal-block"> lxe: replace_children (--TC T1) 0.0179 msec/pass cET: replace_children (--TC T1) 0.0119 msec/pass ET : replace_children (--TC T1) 0.0739 msec/pass </pre> <p>You should keep this difference in mind when you merge very large trees.</p> </div> <div class="section"> <h2><a id="deepcopy" name="deepcopy">deepcopy</a></h2> <p>Deep copying a tree is fast in lxml:</p> <pre class="literal-block"> lxe: deepcopy_all (--TR T1) 9.7558 msec/pass cET: deepcopy_all (--TR T1) 120.6188 msec/pass ET : deepcopy_all (--TR T1) 902.6880 msec/pass lxe: deepcopy_all (-ATR T2) 12.3210 msec/pass cET: deepcopy_all (-ATR T2) 136.9810 msec/pass ET : deepcopy_all (-ATR T2) 944.2801 msec/pass lxe: deepcopy_all (S-TR T3) 8.3981 msec/pass cET: deepcopy_all (S-TR T3) 35.6541 msec/pass ET : deepcopy_all (S-TR T3) 221.6041 msec/pass </pre> <p>So, for example, if you have a database-like scenario where you parse in a large tree and then search and copy independent subtrees from it for further processing, lxml is by far the best choice here.</p> </div> <div class="section"> <h2><a id="tree-traversal" name="tree-traversal">Tree traversal</a></h2> <p>Another area where lxml is very fast is iteration for tree traversal. If your algorithms can benefit from step-by-step traversal of the XML tree and especially if few elements are of interest or the target element tag name is known, lxml is a good choice:</p> <pre class="literal-block"> lxe: getiterator_all (--TR T1) 5.7251 msec/pass cET: getiterator_all (--TR T1) 39.9489 msec/pass ET : getiterator_all (--TR T1) 23.0000 msec/pass lxe: getiterator_islice (--TR T2) 0.0830 msec/pass cET: getiterator_islice (--TR T2) 0.3440 msec/pass ET : getiterator_islice (--TR T2) 0.2429 msec/pass lxe: getiterator_tag (--TR T2) 0.3011 msec/pass cET: getiterator_tag (--TR T2) 14.1001 msec/pass ET : getiterator_tag (--TR T2) 7.4241 msec/pass lxe: getiterator_tag_all (--TR T2) 0.6340 msec/pass cET: getiterator_tag_all (--TR T2) 40.7901 msec/pass ET : getiterator_tag_all (--TR T2) 21.0390 msec/pass </pre> <p>This translates directly into similar timings for <tt class="docutils literal"><span class="pre">Element.findall()</span></tt>:</p> <pre class="literal-block"> lxe: findall (--TR T2) 7.8950 msec/pass cET: findall (--TR T2) 44.5340 msec/pass ET : findall (--TR T2) 27.1149 msec/pass lxe: findall (--TR T3) 1.7281 msec/pass cET: findall (--TR T3) 12.9611 msec/pass ET : findall (--TR T3) 8.6131 msec/pass lxe: findall_tag (--TR T2) 0.7720 msec/pass cET: findall_tag (--TR T2) 40.6358 msec/pass ET : findall_tag (--TR T2) 21.4581 msec/pass lxe: findall_tag (--TR T3) 0.2050 msec/pass cET: findall_tag (--TR T3) 9.6831 msec/pass ET : findall_tag (--TR T3) 5.2109 msec/pass </pre> <p>Note that all three libraries currently use the same Python implementation for <tt class="docutils literal"><span class="pre">findall()</span></tt>, except for their native tree iterator.</p> </div> </div> <div class="section"> <h1><a id="xpath" name="xpath">XPath</a></h1> <p>The following timings are based on the benchmark script <a class="reference" href="http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_xpath.py">bench_xpath.py</a>.</p> <p>This part of lxml does not have an equivalent in ElementTree. However, lxml provides more than one way of accessing it and you should take care which part of the lxml API you use. The most straight forward way is to call the <tt class="docutils literal"><span class="pre">xpath()</span></tt> method on an Element or ElementTree:</p> <pre class="literal-block"> lxe: xpath_method (--TC T1) 1.7459 msec/pass lxe: xpath_method (--TC T2) 22.0850 msec/pass lxe: xpath_method (--TC T3) 0.1309 msec/pass lxe: xpath_method (--TC T4) 1.0772 msec/pass </pre> <p>This is well suited for testing and when the XPath expressions are as diverse as the trees they are called on. However, if you have a single XPath expression that you want to apply to a larger number of different elements, the <tt class="docutils literal"><span class="pre">XPath</span></tt> class is the most efficient way to do it:</p> <pre class="literal-block"> lxe: xpath_class (--TC T1) 0.6740 msec/pass lxe: xpath_class (--TC T2) 3.1760 msec/pass lxe: xpath_class (--TC T3) 0.0548 msec/pass lxe: xpath_class (--TC T4) 0.1700 msec/pass </pre> <p>Note that this still allows you to use variables in the expression, so you can parse it once and then adapt it through variables at call time. In other cases, where you have a fixed Element or ElementTree and want to run different expressions on it, you should consider the <tt class="docutils literal"><span class="pre">XPathEvaluator</span></tt>:</p> <pre class="literal-block"> lxe: xpath_element (--TR T1) 0.4151 msec/pass lxe: xpath_element (--TR T2) 11.6129 msec/pass lxe: xpath_element (--TR T3) 0.1299 msec/pass lxe: xpath_element (--TR T4) 0.3409 msec/pass </pre> <p>While it looks slightly slower, creating an XPath object for each of the expressions generates a much higher overhead here:</p> <pre class="literal-block"> lxe: xpath_class_repeat (--TC T1) 1.6699 msec/pass lxe: xpath_class_repeat (--TC T2) 20.4420 msec/pass lxe: xpath_class_repeat (--TC T3) 0.1230 msec/pass lxe: xpath_class_repeat (--TC T4) 0.9859 msec/pass </pre> </div> <div class="section"> <h1><a id="a-longer-example" name="a-longer-example">A longer example</a></h1> <p>... based on lxml 1.3.</p> <p>A while ago, Uche Ogbuji posted a <a class="reference" href="http://www.onlamp.com/pub/wlg/6291">benchmark proposal</a> that would read in a 3MB XML version of the <a class="reference" href="http://www.ibiblio.org/bosak/xml/eg/religion.2.00.xml.zip">Old Testament</a> of the Bible and look for the word <em>begat</em> in all verses. Apparently, it is contained in 120 out of almost 24000 verses. This is easy to implement in ElementTree using <tt class="docutils literal"><span class="pre">findall()</span></tt>. However, the fastest and most memory friendly way to do this is obviously <tt class="docutils literal"><span class="pre">iterparse()</span></tt>, as most of the data is not of any interest.</p> <p>Now, Uche's original proposal was more or less the following:</p> <pre class="literal-block"> def bench_ET(): tree = ElementTree.parse("ot.xml") result = [] for v in tree.findall("//v"): text = v.text if 'begat' in text: result.append(text) return len(result) </pre> <p>which takes about one second on my machine today. The faster <tt class="docutils literal"><span class="pre">iterparse()</span></tt> variant looks like this:</p> <pre class="literal-block"> def bench_ET_iterparse(): result = [] for event, v in ElementTree.iterparse("ot.xml"): if v.tag == 'v': text = v.text if 'begat' in text: result.append(text) v.clear() return len(result) </pre> <p>The improvement is about 10%. At the time I first tried (early 2006), lxml didn't have <tt class="docutils literal"><span class="pre">iterparse()</span></tt> support, but the <tt class="docutils literal"><span class="pre">findall()</span></tt> variant was already faster than ElementTree. This changes immediately when you switch to cElementTree. The latter only needs 0.17 seconds to do the trick today and only some impressive 0.10 seconds when running the iterparse version. And even back then, it was quite a bit faster than what lxml could achieve.</p> <p>Since then, lxml has matured a lot and has gotten much faster. The iterparse variant now runs in 0.14 seconds, and if you remove the <tt class="docutils literal"><span class="pre">v.clear()</span></tt>, it is even a little faster (which isn't the case for cElementTree).</p> <p>One of the many great tools in lxml is XPath, a swiss army knife for finding things in XML documents. It is possible to move the whole thing to a pure XPath implementation, which looks like this:</p> <pre class="literal-block"> def bench_lxml_xpath_all(): tree = etree.parse("ot.xml") result = tree.xpath("//v[contains(., 'begat')]/text()") return len(result) </pre> <p>This runs in about 0.13 seconds and is about the shortest possible implementation (in lines of Python code) that I could come up with. Now, this is already a rather complex XPath expression compared to the simple "//v" ElementPath expression we started with. Since this is also valid XPath, let's try this instead:</p> <pre class="literal-block"> def bench_lxml_xpath(): tree = etree.parse("ot.xml") result = [] for v in tree.xpath("//v"): text = v.text if 'begat' in text: result.append(text) return len(result) </pre> <p>This gets us down to 0.12 seconds, thus showing that a generic XPath evaluation engine cannot always compete with a simpler, tailored solution. However, since this is not much different from the original findall variant, we can remove the complexity of the XPath call completely and just go with what we had in the beginning. Under lxml, this runs in the same 0.12 seconds.</p> <p>But there is one thing left to try. We can replace the simple ElementPath expression with a native tree iterator:</p> <pre class="literal-block"> def bench_lxml_getiterator(): tree = etree.parse("ot.xml") result = [] for v in tree.getiterator("v"): text = v.text if 'begat' in text: result.append(text) return len(result) </pre> <p>This implements the same thing, just without the overhead of parsing and evaluating a path expression. And this makes it another bit faster, down to 0.11 seconds. For comparison, cElementTree runs this version in 0.17 seconds.</p> <p>So, what have we learned?</p> <ul class="simple"> <li>Python code is not slow. The pure XPath solution was not even as fast as the first shot Python implementation. In general, a few more lines in Python make things more readable, which is much more important than the last 5% of performance.</li> <li>It's important to know the available options - and it's worth starting with the most simple one. In this case, a programmer would then probably have started with <tt class="docutils literal"><span class="pre">getiterator("v")</span></tt> or <tt class="docutils literal"><span class="pre">iterparse()</span></tt>. Either of them would already have been the most efficient, depending on which library is used.</li> <li>It's important to know your tool. lxml and cElementTree are both very fast libraries, but they do not have the same performance characteristics. The fastest solution in one library can be comparatively slow in the other. If you optimise, optimise for the specific target platform.</li> <li>It's not always worth optimising. After all that hassle we got from 0.12 seconds for the initial implementation to 0.11 seconds. Switching over to cElementTree and writing an <tt class="docutils literal"><span class="pre">iterparse()</span></tt> based version would have given us 0.10 seconds - not a big difference for 3MB of XML.</li> <li>Take care what operation is really dominating in your use case. If we split up the operations, we can see that lxml is slightly slower than cElementTree on <tt class="docutils literal"><span class="pre">parse()</span></tt> (both about 0.06 seconds), but more visibly slower on <tt class="docutils literal"><span class="pre">iterparse()</span></tt>: 0.07 versus 0.10 seconds. However, tree iteration in lxml is increadibly fast, so it can be better to parse the whole tree and then iterate over it rather than using <tt class="docutils literal"><span class="pre">iterparse()</span></tt> to do both in one step. Or, you can just wait for the lxml authors to optimise iterparse in one of the next releases...</li> </ul> </div> <div class="section"> <h1><a id="lxml-objectify" name="lxml-objectify">lxml.objectify</a></h1> <p>The following timings are based on the benchmark script <a class="reference" href="http://codespeak.net/svn/lxml/branch/lxml-1.3/benchmark/bench_objectify.py">bench_objectify.py</a>.</p> <p>Objectify is a data-binding API for XML based on lxml.etree, that was added in version 1.1. It uses standard Python attribute access to traverse the XML tree. It also features ObjectPath, a fast path language based on the same meme.</p> <p>Just like lxml.etree, lxml.objectify creates Python representations of elements on the fly. To save memory, the normal Python garbage collection mechanisms will discard them when their last reference is gone. In cases where deeply nested elements are frequently accessed through the objectify API, the create-discard cycles can become a bottleneck, as elements have to be instantiated over and over again.</p> <div class="section"> <h2><a id="objectpath" name="objectpath">ObjectPath</a></h2> <p>ObjectPath can be used to speed up the access to elements that are deep in the tree. It avoids step-by-step Python element instantiations along the path, which can substantially improve the access time:</p> <pre class="literal-block"> lxe: attribute (--TR T1) 9.4581 msec/pass lxe: attribute (--TR T2) 52.5560 msec/pass lxe: attribute (--TR T4) 9.1729 msec/pass lxe: objectpath (--TR T1) 4.8690 msec/pass lxe: objectpath (--TR T2) 47.8780 msec/pass lxe: objectpath (--TR T4) 4.7870 msec/pass lxe: attributes_deep (--TR T1) 54.7471 msec/pass lxe: attributes_deep (--TR T2) 62.7451 msec/pass lxe: attributes_deep (--TR T4) 15.1050 msec/pass lxe: objectpath_deep (--TR T1) 48.2810 msec/pass lxe: objectpath_deep (--TR T2) 51.3949 msec/pass lxe: objectpath_deep (--TR T4) 6.1419 msec/pass </pre> <p>Note, however, that parsing ObjectPath expressions is not for free either, so this is most effective for frequently accessing the same element.</p> </div> <div class="section"> <h2><a id="caching-elements" name="caching-elements">Caching Elements</a></h2> <p>A way to improve the normal attribute access time is static instantiation of the Python objects, thus trading memory for speed. Just create a cache dictionary and run:</p> <pre class="literal-block"> cache[root] = list(root.iter()) </pre> <p>after parsing and:</p> <pre class="literal-block"> del cache[root] </pre> <p>when you are done with the tree. This will keep the Python element representations of all elements alive and thus avoid the overhead of repeated Python object creation. You can also consider using filters or generator expressions to be more selective. By choosing the right trees (or even subtrees and elements) to cache, you can trade memory usage against access speed:</p> <pre class="literal-block"> lxe: attribute_cached (--TR T1) 7.5061 msec/pass lxe: attribute_cached (--TR T2) 50.1881 msec/pass lxe: attribute_cached (--TR T4) 7.4170 msec/pass lxe: attributes_deep_cached (--TR T1) 48.7239 msec/pass lxe: attributes_deep_cached (--TR T2) 55.2199 msec/pass lxe: attributes_deep_cached (--TR T4) 9.9740 msec/pass lxe: objectpath_deep_cached (--TR T1) 43.4160 msec/pass lxe: objectpath_deep_cached (--TR T2) 47.6480 msec/pass lxe: objectpath_deep_cached (--TR T4) 3.4680 msec/pass </pre> <p>Things to note: you cannot currently use <tt class="docutils literal"><span class="pre">weakref.WeakKeyDictionary</span></tt> objects for this as lxml's element objects do not support weak references (which are costly in terms of memory). Also note that new element objects that you add to these trees will not turn up in the cache automatically and will therefore still be garbage collected when all their Python references are gone, so this is most effective for largely immutable trees. You should consider using a set instead of a list in this case and add new elements by hand.</p> </div> <div class="section"> <h2><a id="further-optimisations" name="further-optimisations">Further optimisations</a></h2> <p>Here are some more things to try if optimisation is required:</p> <ul class="simple"> <li>A lot of time is usually spent in tree traversal to find the addressed elements in the tree. If you often work in subtrees, do what you would also do with deep Python objects: assign the parent of the subtree to a variable or pass it into functions instead of starting at the root. This allows accessing its descendents more directly.</li> <li>Try assigning data values directly to attributes instead of passing them through DataElement.</li> <li>If you use custom data types that are costly to parse, try running <tt class="docutils literal"><span class="pre">objectify.annotate()</span></tt> over read-only trees to speed up the attribute type inference on read access.</li> </ul> <p>Note that none of these measures is guaranteed to speed up your application. As usual, you should prefer readable code over premature optimisations and profile your expected use cases before bothering to apply optimisations at random.</p> </div> </div> </div> <div class="footer"> <hr class="footer" /> Generated on: 2008-12-12. </div> </body> </html>