<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
<meta name="generator" content="Docutils 0.4.1: http://docutils.sourceforge.net/" />
<title>BeautifulSoup Parser</title>
<link rel="stylesheet" href="style.css" type="text/css" />
</head>
<body>
<div class="document" id="beautifulsoup-parser">
<div class="sidemenu"><ul id="lxml"><li><span class="section title">lxml</span><ul class="menu foreign" id="index"><li class="menu title"><a href="index.html">lxml</a><ul class="submenu"><li class="menu item"><a href="index.html#introduction">Introduction</a></li><li class="menu item"><a href="index.html#documentation">Documentation</a></li><li class="menu item"><a href="index.html#download">Download</a></li><li class="menu item"><a href="index.html#mailing-list">Mailing list</a></li><li class="menu item"><a href="index.html#bug-tracker">Bug tracker</a></li><li class="menu item"><a href="index.html#license">License</a></li><li class="menu item"><a href="index.html#old-versions">Old Versions</a></li></ul></li></ul><ul class="menu foreign" id="intro"><li class="menu title"><a href="intro.html">Why lxml?</a><ul class="submenu"><li class="menu item"><a href="intro.html#motto">Motto</a></li><li class="menu item"><a href="intro.html#aims">Aims</a></li></ul></li></ul><ul class="menu foreign" id="installation"><li class="menu title"><a href="installation.html">Installing lxml</a><ul class="submenu"><li class="menu item"><a href="installation.html#requirements">Requirements</a></li><li class="menu item"><a href="installation.html#installation">Installation</a></li><li class="menu item"><a href="installation.html#building-lxml-from-sources">Building lxml from sources</a></li><li class="menu item"><a href="installation.html#ms-windows">MS Windows</a></li><li class="menu item"><a href="installation.html#macos-x">MacOS-X</a></li></ul></li></ul><ul class="menu foreign" id="lxml2"><li class="menu title"><a href="lxml2.html">What's new in lxml 2.0?</a><ul class="submenu"><li class="menu item"><a href="lxml2.html#changes-in-etree-and-objectify">Changes in etree and objectify</a></li><li class="menu item"><a href="lxml2.html#new-modules">New modules</a></li></ul></li></ul><ul class="menu foreign" id="performance"><li class="menu title"><a href="performance.html">Benchmarks and Speed</a><ul class="submenu"><li class="menu item"><a href="performance.html#general-notes">General notes</a></li><li class="menu item"><a href="performance.html#how-to-read-the-timings">How to read the timings</a></li><li class="menu item"><a href="performance.html#parsing-and-serialising">Parsing and Serialising</a></li><li class="menu item"><a href="performance.html#the-elementtree-api">The ElementTree API</a></li><li class="menu item"><a href="performance.html#xpath">XPath</a></li><li class="menu item"><a href="performance.html#a-longer-example">A longer example</a></li><li class="menu item"><a href="performance.html#lxml-objectify">lxml.objectify</a></li></ul></li></ul><ul class="menu foreign" id="compatibility"><li class="menu title"><a href="compatibility.html">ElementTree compatibility of lxml.etree</a></li></ul><ul class="menu foreign" id="FAQ"><li class="menu title"><a href="FAQ.html">lxml FAQ - Frequently Asked Questions</a><ul class="submenu"><li class="menu item"><a href="FAQ.html#general-questions">General Questions</a></li><li class="menu item"><a href="FAQ.html#installation">Installation</a></li><li class="menu item"><a href="FAQ.html#contributing">Contributing</a></li><li class="menu item"><a href="FAQ.html#bugs">Bugs</a></li><li class="menu item"><a href="FAQ.html#threading">Threading</a></li><li class="menu item"><a href="FAQ.html#parsing-and-serialisation">Parsing and Serialisation</a></li><li class="menu item"><a href="FAQ.html#xpath-and-document-traversal">XPath and Document Traversal</a></li></ul></li></ul></li></ul><ul id="Developing with lxml"><li><span class="section title">Developing with lxml</span><ul class="menu foreign" id="tutorial"><li class="menu title"><a href="tutorial.html">The lxml.etree Tutorial</a><ul class="submenu"><li class="menu item"><a href="tutorial.html#the-element-class">The Element class</a></li><li class="menu item"><a href="tutorial.html#the-elementtree-class">The ElementTree class</a></li><li class="menu item"><a href="tutorial.html#parsing-from-strings-and-files">Parsing from strings and files</a></li><li class="menu item"><a href="tutorial.html#namespaces">Namespaces</a></li><li class="menu item"><a href="tutorial.html#the-e-factory">The E-factory</a></li><li class="menu item"><a href="tutorial.html#elementpath">ElementPath</a></li></ul></li></ul><ul class="menu foreign" id="api index"><li class="menu title"><a href="api/index.html">API reference</a></li></ul><ul class="menu foreign" id="api"><li class="menu title"><a href="api.html">APIs specific to lxml.etree</a><ul class="submenu"><li class="menu item"><a href="api.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="api.html#other-element-apis">Other Element APIs</a></li><li class="menu item"><a href="api.html#trees-and-documents">Trees and Documents</a></li><li class="menu item"><a href="api.html#iteration">Iteration</a></li><li class="menu item"><a href="api.html#error-handling-on-exceptions">Error handling on exceptions</a></li><li class="menu item"><a href="api.html#error-logging">Error logging</a></li><li class="menu item"><a href="api.html#serialisation">Serialisation</a></li><li class="menu item"><a href="api.html#xinclude-and-elementinclude">XInclude and ElementInclude</a></li><li class="menu item"><a href="api.html#write-c14n-on-elementtree">write_c14n on ElementTree</a></li></ul></li></ul><ul class="menu foreign" id="parsing"><li class="menu title"><a href="parsing.html">Parsing XML and HTML with lxml</a><ul class="submenu"><li class="menu item"><a href="parsing.html#parsers">Parsers</a></li><li class="menu item"><a href="parsing.html#the-target-parser-interface">The target parser interface</a></li><li class="menu item"><a href="parsing.html#the-feed-parser-interface">The feed parser interface</a></li><li class="menu item"><a href="parsing.html#iterparse-and-iterwalk">iterparse and iterwalk</a></li><li class="menu item"><a href="parsing.html#python-unicode-strings">Python unicode strings</a></li></ul></li></ul><ul class="menu foreign" id="validation"><li class="menu title"><a href="validation.html">Validation with lxml</a><ul class="submenu"><li class="menu item"><a href="validation.html#validation-at-parse-time">Validation at parse time</a></li><li class="menu item"><a href="validation.html#dtd">DTD</a></li><li class="menu item"><a href="validation.html#relaxng">RelaxNG</a></li><li class="menu item"><a href="validation.html#xmlschema">XMLSchema</a></li><li class="menu item"><a href="validation.html#schematron">Schematron</a></li></ul></li></ul><ul class="menu foreign" id="xpathxslt"><li class="menu title"><a href="xpathxslt.html">XPath and XSLT with lxml</a><ul class="submenu"><li class="menu item"><a href="xpathxslt.html#xpath">XPath</a></li><li class="menu item"><a href="xpathxslt.html#xslt">XSLT</a></li></ul></li></ul><ul class="menu foreign" id="objectify"><li class="menu title"><a href="objectify.html">lxml.objectify</a><ul class="submenu"><li class="menu item"><a href="objectify.html#the-lxml-objectify-api">The lxml.objectify API</a></li><li class="menu item"><a href="objectify.html#asserting-a-schema">Asserting a Schema</a></li><li class="menu item"><a href="objectify.html#objectpath">ObjectPath</a></li><li class="menu item"><a href="objectify.html#python-data-types">Python data types</a></li><li class="menu item"><a href="objectify.html#how-data-types-are-matched">How data types are matched</a></li><li class="menu item"><a href="objectify.html#what-is-different-from-lxml-etree?">What is different from lxml.etree?</a></li></ul></li></ul><ul class="menu foreign" id="lxmlhtml"><li class="menu title"><a href="lxmlhtml.html">lxml.html</a><ul class="submenu"><li class="menu item"><a href="lxmlhtml.html#parsing-html">Parsing HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-element-methods">HTML Element Methods</a></li><li class="menu item"><a href="lxmlhtml.html#running-html-doctests">Running HTML doctests</a></li><li class="menu item"><a href="lxmlhtml.html#creating-html-with-the-e-factory">Creating HTML with the E-factory</a></li><li class="menu item"><a href="lxmlhtml.html#working-with-links">Working with links</a></li><li class="menu item"><a href="lxmlhtml.html#forms">Forms</a></li><li class="menu item"><a href="lxmlhtml.html#cleaning-up-html">Cleaning up HTML</a></li><li class="menu item"><a href="lxmlhtml.html#html-diff">HTML Diff</a></li><li class="menu item"><a href="lxmlhtml.html#examples">Examples</a></li></ul></li></ul><ul class="menu foreign" id="cssselect"><li class="menu title"><a href="cssselect.html">lxml.cssselect</a><ul class="submenu"><li class="menu item"><a href="cssselect.html#the-cssselector-class">The CSSSelector class</a></li><li class="menu item"><a href="cssselect.html#css-selectors">CSS Selectors</a></li><li class="menu item"><a href="cssselect.html#limitations">Limitations</a></li></ul></li></ul><ul class="menu current" id="elementsoup"><li class="menu title"><a href="elementsoup.html">BeautifulSoup Parser</a><ul class="submenu"><li class="menu item"><a href="elementsoup.html#entity-handling">Entity handling</a></li><li class="menu item"><a href="elementsoup.html#using-soupparser-as-a-fallback">Using soupparser as a fallback</a></li></ul></li></ul></li></ul><ul id="Extending lxml"><li><span class="section title">Extending lxml</span><ul class="menu foreign" id="resolvers"><li class="menu title"><a href="resolvers.html">Document loading and URL resolving</a><ul class="submenu"><li class="menu item"><a href="resolvers.html#resolvers">Resolvers</a></li><li class="menu item"><a href="resolvers.html#document-loading-in-context">Document loading in context</a></li><li class="menu item"><a href="resolvers.html#i-o-access-control-in-xslt">I/O access control in XSLT</a></li></ul></li></ul><ul class="menu foreign" id="extensions"><li class="menu title"><a href="extensions.html">Extension functions for XPath and XSLT</a><ul class="submenu"><li class="menu item"><a href="extensions.html#the-functionnamespace">The FunctionNamespace</a></li><li class="menu item"><a href="extensions.html#global-prefix-assignment">Global prefix assignment</a></li><li class="menu item"><a href="extensions.html#the-xpath-context">The XPath context</a></li><li class="menu item"><a href="extensions.html#evaluators-and-xslt">Evaluators and XSLT</a></li><li class="menu item"><a href="extensions.html#evaluator-local-extensions">Evaluator-local extensions</a></li><li class="menu item"><a href="extensions.html#what-to-return-from-a-function">What to return from a function</a></li></ul></li></ul><ul class="menu foreign" id="element classes"><li class="menu title"><a href="element_classes.html">Using custom Element classes in lxml</a><ul class="submenu"><li class="menu item"><a href="element_classes.html#element-initialization">Element initialization</a></li><li class="menu item"><a href="element_classes.html#setting-up-a-class-lookup-scheme">Setting up a class lookup scheme</a></li><li class="menu item"><a href="element_classes.html#implementing-namespaces">Implementing namespaces</a></li></ul></li></ul><ul class="menu foreign" id="sax"><li class="menu title"><a href="sax.html">Sax support</a><ul class="submenu"><li class="menu item"><a href="sax.html#building-a-tree-from-sax-events">Building a tree from SAX events</a></li><li class="menu item"><a href="sax.html#producing-sax-events-from-an-elementtree-or-element">Producing SAX events from an ElementTree or Element</a></li><li class="menu item"><a href="sax.html#interfacing-with-pulldom-minidom">Interfacing with pulldom/minidom</a></li></ul></li></ul><ul class="menu foreign" id="capi"><li class="menu title"><a href="capi.html">The public C-API of lxml.etree</a><ul class="submenu"><li class="menu item"><a href="capi.html#writing-external-modules-in-cython">Writing external modules in Cython</a></li><li class="menu item"><a href="capi.html#writing-external-modules-in-c">Writing external modules in C</a></li></ul></li></ul></li></ul><ul id="Developing lxml"><li><span class="section title">Developing lxml</span><ul class="menu foreign" id="build"><li class="menu title"><a href="build.html">How to build lxml from source</a><ul class="submenu"><li class="menu item"><a href="build.html#cython">Cython</a></li><li class="menu item"><a href="build.html#subversion">Subversion</a></li><li class="menu item"><a href="build.html#setuptools">Setuptools</a></li><li class="menu item"><a href="build.html#running-the-tests-and-reporting-errors">Running the tests and reporting errors</a></li><li class="menu item"><a href="build.html#contributing-an-egg">Contributing an egg</a></li><li class="menu item"><a href="build.html#providing-newer-library-versions-on-mac-os-x">Providing newer library versions on Mac-OS X</a></li><li class="menu item"><a href="build.html#static-linking-on-windows">Static linking on Windows</a></li><li class="menu item"><a href="build.html#building-debian-packages-from-svn-sources">Building Debian packages from SVN sources</a></li></ul></li></ul><ul class="menu foreign" id="lxml source howto"><li class="menu title"><a href="lxml-source-howto.html">How to read the source of lxml</a><ul class="submenu"><li class="menu item"><a href="lxml-source-howto.html#what-is-cython?">What is Cython?</a></li><li class="menu item"><a href="lxml-source-howto.html#where-to-start?">Where to start?</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-etree">lxml.etree</a></li><li class="menu item"><a href="lxml-source-howto.html#python-modules">Python modules</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-objectify">lxml.objectify</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-pyclasslookup">lxml.pyclasslookup</a></li><li class="menu item"><a href="lxml-source-howto.html#lxml-html">lxml.html</a></li></ul></li></ul><ul class="menu foreign" id="changes 2 0 11"><li class="menu title"><a href="changes-2.0.11.html">Release Changelog</a></li></ul><ul class="menu foreign" id="credits"><li class="menu title"><a href="credits.html">Credits</a><ul class="submenu"><li class="menu item"><a href="credits.html#special-thanks-goes-to:">Special thanks goes to:</a></li></ul></li></ul></li></ul></div><h1 class="title">BeautifulSoup Parser</h1>
<p><a class="reference" href="http://www.crummy.com/software/BeautifulSoup/">BeautifulSoup</a> is a Python package that parses broken HTML. While libxml2
(and thus lxml) can also parse broken HTML, BeautifulSoup is a bit more
forgiving and has superiour <a class="reference" href="http://www.crummy.com/software/BeautifulSoup/documentation.html#Beautiful%20Soup%20Gives%20You%20Unicode,%20Dammit">support for encoding detection</a>.</p>
<p>lxml can benefit from the parsing capabilities of BeautifulSoup
through the <tt class="docutils literal"><span class="pre">lxml.html.soupparser</span></tt> module. It provides three main
functions: <tt class="docutils literal"><span class="pre">fromstring()</span></tt> and <tt class="docutils literal"><span class="pre">parse()</span></tt> to parse a string or file
using BeautifulSoup, and <tt class="docutils literal"><span class="pre">convert_tree()</span></tt> to convert an existing
BeautifulSoup tree into a list of top-level Elements.</p>
<p>The functions <tt class="docutils literal"><span class="pre">fromstring()</span></tt> and <tt class="docutils literal"><span class="pre">parse()</span></tt> behave as known from
ElementTree. The first returns a root Element, the latter returns an
ElementTree.</p>
<p>There is also a legacy module called <tt class="docutils literal"><span class="pre">lxml.html.ElementSoup</span></tt>, which
mimics the interface provided by ElementTree's own <a class="reference" href="http://effbot.org/zone/element-soup.htm">ElementSoup</a>
module. Note that the <tt class="docutils literal"><span class="pre">soupparser</span></tt> module was added in lxml 2.0.3.
Previous versions of lxml 2.0.x only have the <tt class="docutils literal"><span class="pre">ElementSoup</span></tt> module.</p>
<p>Here is a document full of tag soup, similar to, but not quite like, HTML:</p>
<pre class="literal-block">
>>> tag_soup = '<meta><head><title>Hello</head<body onload=crash()>Hi all<p>'
</pre>
<p>all you need to do is pass it to the <tt class="docutils literal"><span class="pre">fromstring()</span></tt> function:</p>
<pre class="literal-block">
>>> from lxml.html.soupparser import fromstring
>>> root = fromstring(tag_soup)
</pre>
<p>To see what we have here, you can serialise it:</p>
<pre class="literal-block">
>>> from lxml.etree import tostring
>>> print tostring(root, pretty_print=True),
<html>
<meta/>
<head>
<title>Hello</title>
</head>
<body onload="crash()">Hi all<p/></body>
</html>
</pre>
<p>Not quite what you'd expect from an HTML page, but, well, it was broken
already, right? BeautifulSoup did its best, and so now it's a tree.</p>
<p>To control which Element implementation is used, you can pass a
<tt class="docutils literal"><span class="pre">makeelement</span></tt> factory function to <tt class="docutils literal"><span class="pre">parse()</span></tt> and <tt class="docutils literal"><span class="pre">fromstring()</span></tt>.
By default, this is based on the HTML parser defined in <tt class="docutils literal"><span class="pre">lxml.html</span></tt>.</p>
<div class="section">
<h1><a id="entity-handling" name="entity-handling">Entity handling</a></h1>
<p>By default, the BeautifulSoup parser also replaces the entities it
finds by their character equivalent:</p>
<pre class="literal-block">
>>> tag_soup = '<body>&copy;&euro;&#45;&#245;&#445;<p>'
>>> body = fromstring(tag_soup).find('.//body')
>>> body.text
u'\xa9\u20ac-\xf5\u01bd'
</pre>
<p>If you want them back on the way out, you can serialise with the
'html' method, which will always use escaping for safety reasons:</p>
<pre class="literal-block">
>>> tostring(body, method="html")
'<body>&#xA9;&#x20AC;-&#xF5;&#x1BD;<p></p></body>'
>>> tostring(body, method="html", encoding="utf-8")
'<body>&#xA9;&#x20AC;-&#xF5;&#x1BD;<p></p></body>'
>>> tostring(body, method="html", encoding=unicode)
u'<body>&#xA9;&#x20AC;-&#xF5;&#x1BD;<p></p></body>'
</pre>
<p>Otherwise, when serialising to XML, only the plain ASCII encoding will
escape non-ASCII characters:</p>
<pre class="literal-block">
>>> tostring(body)
'<body>&#169;&#8364;-&#245;&#445;<p/></body>'
>>> tostring(body, encoding="utf-8")
'<body>\xc2\xa9\xe2\x82\xac-\xc3\xb5\xc6\xbd<p/></body>'
>>> tostring(body, encoding=unicode)
u'<body>\xa9\u20ac-\xf5\u01bd<p/></body>'
</pre>
<p>There is also a legacy module called <tt class="docutils literal"><span class="pre">lxml.html.ElementSoup</span></tt>, which
mimics the interface provided by ElementTree's own <a class="reference" href="http://effbot.org/zone/element-soup.htm">ElementSoup</a>
module.</p>
</div>
<div class="section">
<h1><a id="using-soupparser-as-a-fallback" name="using-soupparser-as-a-fallback">Using soupparser as a fallback</a></h1>
<p>The downside of using this parser is that it is <a class="reference" href="http://blog.ianbicking.org/2008/03/30/python-html-parser-performance/">much slower</a> than
the HTML parser of lxml. So if performance matters, you might want to
consider using <tt class="docutils literal"><span class="pre">soupparser</span></tt> only as a fallback for certain cases.</p>
<p>One common problem of lxml's parser is that it might not get the
encoding right in cases where the document contains a <tt class="docutils literal"><span class="pre"><meta></span></tt> tag
at the wrong place. In this case, you can exploit the fact that lxml
serialises much faster than most other HTML libraries for Python.
Just serialise the document to unicode and if that gives you an
exception, re-parse it with BeautifulSoup to see if that works
better:</p>
<pre class="literal-block">
>>> tag_soup = '''\
... <meta http-equiv="Content-Type"
... content="text/html;charset=utf-8" />
... <html>
... <head>
... <title>Hello W\xc3\xb6rld!</title>
... </head>
... <body>Hi all</body>
... </html>'''
>>> import lxml.html
>>> import lxml.html.soupparser
>>> root = lxml.html.fromstring(tag_soup)
>>> try:
... ignore = tostring(root, encoding=unicode)
... except UnicodeDecodeError:
... root = lxml.html.soupparser.fromstring(tag_soup)
</pre>
</div>
</div>
<div class="footer">
<hr class="footer" />
Generated on: 2008-12-12.
</div>
</body>
</html>
