<html> <head> <meta http-equiv="Content-Type" content="text/html; charset=US-ASCII"> <title>About Tokens and Token Values</title> <link rel="stylesheet" href="../../../../../../../../doc/html/boostbook.css" type="text/css"> <meta name="generator" content="DocBook XSL Stylesheets V1.75.0"> <link rel="home" href="../../../../index.html" title="Spirit 2.2"> <link rel="up" href="../lexer_primitives.html" title="Lexer Primitives"> <link rel="prev" href="../lexer_primitives.html" title="Lexer Primitives"> <link rel="next" href="../lexer_tokenizing.html" title="Tokenizing Input Data"> </head> <body bgcolor="white" text="black" link="#0000FF" vlink="#840084" alink="#0000FF"> <table cellpadding="2" width="100%"><tr> <td valign="top"><img alt="Boost C++ Libraries" width="277" height="86" src="../../../../../../../../boost.png"></td> <td align="center"><a href="../../../../../../../../index.html">Home</a></td> <td align="center"><a href="../../../../../../../libraries.htm">Libraries</a></td> <td align="center"><a href="http://www.boost.org/users/people.html">People</a></td> <td align="center"><a href="http://www.boost.org/users/faq.html">FAQ</a></td> <td align="center"><a href="../../../../../../../../more/index.htm">More</a></td> </tr></table> <hr> <div class="spirit-nav"> <a accesskey="p" href="../lexer_primitives.html"><img src="../../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lexer_primitives.html"><img src="../../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../index.html"><img src="../../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../lexer_tokenizing.html"><img src="../../../../../../../../doc/html/images/next.png" alt="Next"></a> </div> <div class="section" title="About Tokens and Token Values"> <div class="titlepage"><div><div><h5 class="title"> <a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values"></a><a class="link" href="lexer_token_values.html" title="About Tokens and Token Values"> About Tokens and Token Values</a> </h5></div></div></div> <p> As already discussed, lexical scanning is the process of analyzing the stream of input characters and separating it into strings called tokens, most of the time separated by whitespace. The different token types recognized by a lexical analyzer often get assigned unique integer token identifiers (token ids). These token ids are normally used by the parser to identifiy the current token without having to look at the matched string again. The <span class="emphasis"><em>Spirit.Lex</em></span> library is not different with respect to this, as it uses the token ids as the main means of identification of the different token types defined for a particular lexical analyzer. However, it is different from commonly used lexical analyzers in the sense that it returns (references to) instances of a (user defined) token class to the user. The only limitation of this token class is that it must carry at least the token id of the token it represents. For more information about the interface a user defined token type has to expose please look at the Token Class reference. The library provides a default token type based on the <a href="http://www.benhanson.net/lexertl.html" target="_top">Lexertl</a> library which should be sufficient in most cases: the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><></span></code> type. This section focusses on the description of general features a token class may implement and how this integrates with the other parts of the <span class="emphasis"><em>Spirit.Lex</em></span> library. </p> <a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_anatomy_of_a_token"></a><h6> <a name="id943998"></a> <a class="link" href="lexer_token_values.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_anatomy_of_a_token">The Anatomy of a Token</a> </h6> <p> It is very important to understand the difference between a token definition (represented by the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><></span></code> template) and a token itself (for instance represented by the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><></span></code> template). </p> <p> The token definition is used to describe the main features of a particular token type, especially: </p> <div class="itemizedlist"><ul class="itemizedlist" type="disc"> <li class="listitem"> to simplify the definition of a token type using a regular expression pattern applied while matching this token type, </li> <li class="listitem"> to associate a token type with a particular lexer state, </li> <li class="listitem"> to optionally assign a token id to a token type, </li> <li class="listitem"> to optionally associate some code to execute whenever an instance of this token type has been matched, </li> <li class="listitem"> and to optionally specify the attribute type of the token value. </li> </ul></div> <p> The token itself is a data structure returned by the lexer iterators. Dereferencing a lexer iterator returns a reference to the last matched token instance. It encapsulates the part of the underlying input sequence matched by the regular expression used during the definiton of this token type. Incrementing the lexer iterator invokes the lexical analyzer to match the next token by advancing the underlying input stream. The token data structure contains at least the token id of the matched token type, allowing to identify the matched character sequence. Optionally, the token instance may contain a token value and/or the lexer state this token instance was matched in. The following <a class="link" href="lexer_token_values.html#spirit.lex.tokenstructure" title="Figure 8. The structure of a token">figure</a> shows the schematic structure of a token. </p> <p> </p> <div class="figure"> <a name="spirit.lex.tokenstructure"></a><p class="title"><b>Figure 8. The structure of a token</b></p> <div class="figure-contents"><span class="inlinemediaobject"><img src="../../../.././images/tokenstructure.png" alt="The structure of a token"></span></div> </div> <p><br class="figure-break"> </p> <p> The token value and the lexer state the token has been recognized in may be omitted for optimization reasons, thus avoiding the need for the token to carry more data than actually required. This configuration can be achieved by supplying appropriate template parameters for the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><></span></code> template while defining the token type. </p> <p> The lexer iterator returns the same token type for each of the different matched token definitions. To accomodate for the possible different token <span class="emphasis"><em>value</em></span> types exposed by the various token types (token definitions), the general type of the token value is a <a href="http://www.boost.org/doc/html/variant.html" target="_top">Boost.Variant</a>. At a minimum (for the default configuration) this token value variant will be configured to always hold a <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a> containing the pair of iterators pointing to the matched input sequence for this token instance. </p> <div class="note" title="Note"><table border="0" summary="Note"> <tr> <td rowspan="2" align="center" valign="top" width="25"><img alt="[Note]" src="../../../../images/note.png"></td> <th align="left">Note</th> </tr> <tr><td align="left" valign="top"><p> If the lexical analyzer is used in conjunction with a <span class="emphasis"><em>Spirit.Qi</em></span> parser, the stored <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a> token value will be converted to the requested token type (parser attribute) exactly once. This happens at the time of the first access to the token value requiring the corresponding type conversion. The converted token value will be stored in the <a href="http://www.boost.org/doc/html/variant.html" target="_top">Boost.Variant</a> replacing the initially stored iterator range. This avoids having to convert the input sequence to the token value more than once, thus optimizing the integration of the lexer with <span class="emphasis"><em>Spirit.Qi</em></span>, even during parser backtracking. </p></td></tr> </table></div> <p> Here is the template prototype of the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><></span></code> template: </p> <pre class="programlisting"><span class="keyword">template</span> <span class="special"><</span> <span class="keyword">typename</span> <span class="identifier">Iterator</span> <span class="special">=</span> <span class="keyword">char</span> <span class="keyword">const</span><span class="special">*,</span> <span class="keyword">typename</span> <span class="identifier">AttributeTypes</span> <span class="special">=</span> <span class="identifier">mpl</span><span class="special">::</span><span class="identifier">vector0</span><span class="special"><>,</span> <span class="keyword">typename</span> <span class="identifier">HasState</span> <span class="special">=</span> <span class="identifier">mpl</span><span class="special">::</span><span class="identifier">true_</span> <span class="special">></span> <span class="keyword">struct</span> <span class="identifier">lexertl_token</span><span class="special">;</span> </pre> <div class="variablelist" title="where:"> <p class="title"><b>where:</b></p> <dl> <dt><span class="term">Iterator</span></dt> <dd><p> This is the type of the iterator used to access the underlying input stream. It defaults to a plain <code class="computeroutput"><span class="keyword">char</span> <span class="keyword">const</span><span class="special">*</span></code>. </p></dd> <dt><span class="term">AttributeTypes</span></dt> <dd><p> This is either a mpl sequence containing all attribute types used for the token definitions or the type <code class="computeroutput"><span class="identifier">omit</span></code>. If the mpl sequence is empty (which is the default), all token instances will store a <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a><code class="computeroutput"><span class="special"><</span><span class="identifier">Iterator</span><span class="special">></span></code> pointing to the start and the end of the matched section in the input stream. If the type is <code class="computeroutput"><span class="identifier">omit</span></code>, the generated tokens will contain no token value (attribute) at all. </p></dd> <dt><span class="term">HasState</span></dt> <dd><p> This is either <code class="computeroutput"><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">true_</span></code> or <code class="computeroutput"><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">false_</span></code>, allowing control as to whether the generated token instances will contain the lexer state they were generated in. The default is mpl::true_, so all token instances will contain the lexer state. </p></dd> </dl> </div> <p> Normally, during construction, a token instance always holds the <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a> as its token value, unless it has been defined using the <code class="computeroutput"><span class="identifier">omit</span></code> token value type. This iterator range then is converted in place to the requested token value type (attribute) when it is requested for the first time. </p> <a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_physiognomy_of_a_token_definition"></a><h6> <a name="id944532"></a> <a class="link" href="lexer_token_values.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.the_physiognomy_of_a_token_definition">The Physiognomy of a Token Definition</a> </h6> <p> The token definitions (represented by the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><></span></code> template) are normally used as part of the definition of the lexical analyzer. At the same time a token definition instance may be used as a parser component in <span class="emphasis"><em>Spirit.Qi</em></span>. </p> <p> The template prototype of this class is shown here: </p> <pre class="programlisting"><span class="keyword">template</span><span class="special"><</span> <span class="keyword">typename</span> <span class="identifier">Attribute</span> <span class="special">=</span> <span class="identifier">unused_type</span><span class="special">,</span> <span class="keyword">typename</span> <span class="identifier">Char</span> <span class="special">=</span> <span class="keyword">char</span> <span class="special">></span> <span class="keyword">class</span> <span class="identifier">token_def</span><span class="special">;</span> </pre> <div class="variablelist" title="where:"> <p class="title"><b>where:</b></p> <dl> <dt><span class="term">Attribute</span></dt> <dd><p> This is the type of the token value (attribute) supported by token instances representing this token type. This attribute type is exposed to the <span class="emphasis"><em>Spirit.Qi</em></span> library, whenever this token definition is used as a parser component. The default attribute type is <code class="computeroutput"><span class="identifier">unused_type</span></code>, which means the token instance holds a <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a> pointing to the start and the end of the matched section in the input stream. If the attribute is <code class="computeroutput"><span class="identifier">omit</span></code> the token instance will expose no token type at all. Any other type will be used directly as the token value type. </p></dd> <dt><span class="term">Char</span></dt> <dd><p> This is the value type of the iterator for the underlying input sequence. It defaults to <code class="computeroutput"><span class="keyword">char</span></code>. </p></dd> </dl> </div> <p> The semantics of the template parameters for the token type and the token definition type are very similar and interdependent. As a rule of thumb you can think of the token definition type as the means of specifying everything related to a single specific token type (such as <code class="computeroutput"><span class="identifier">identifier</span></code> or <code class="computeroutput"><span class="identifier">integer</span></code>). On the other hand the token type is used to define the general properties of all token instances generated by the <span class="emphasis"><em>Spirit.Lex</em></span> library. </p> <div class="important" title="Important"><table border="0" summary="Important"> <tr> <td rowspan="2" align="center" valign="top" width="25"><img alt="[Important]" src="../../../../images/important.png"></td> <th align="left">Important</th> </tr> <tr><td align="left" valign="top"> <p> If you don't list any token value types in the token type definition declaration (resulting in the usage of the default <a href="../../../../../../../../libs/range/doc/utility_class.html#iter_range" target="_top"><code class="computeroutput"><span class="identifier">boost</span><span class="special">::</span><span class="identifier">iterator_range</span></code></a> token type) everything will compile and work just fine, just a bit less efficient. This is because the token value will be converted from the matched input sequence every time it is requested. </p> <p> But as soon as you specify at least one token value type while defining the token type you'll have to list all value types used for <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><></span></code> declarations in the token definition class, otherwise compilation errors will occur. </p> </td></tr> </table></div> <a name="spirit.lex.abstracts.lexer_primitives.lexer_token_values.examples_of_using__code__phrase_role__identifier__lex__phrase__phrase_role__special______phrase__phrase_role__identifier__lexertl__phrase__phrase_role__special______phrase__phrase_role__identifier__token__phrase__phrase_role__special___lt__gt___phrase___code_"></a><h6> <a name="id944793"></a> <a class="link" href="lexer_token_values.html#spirit.lex.abstracts.lexer_primitives.lexer_token_values.examples_of_using__code__phrase_role__identifier__lex__phrase__phrase_role__special______phrase__phrase_role__identifier__lexertl__phrase__phrase_role__special______phrase__phrase_role__identifier__token__phrase__phrase_role__special___lt__gt___phrase___code_">Examples of using <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><></span></code></a> </h6> <p> Let's start with some examples. We refer to one of the <span class="emphasis"><em>Spirit.Lex</em></span> examples (for the full source code of this example please see <a href="../../../../../../example/lex/example4.cpp" target="_top">example4.cpp</a>). </p> <p> The first code snippet shows an excerpt of the token definition class, the definition of a couple of token types. Some of the token types do not expose a special token value (<code class="computeroutput"><span class="identifier">if_</span></code>, <code class="computeroutput"><span class="identifier">else_</span></code>, and <code class="computeroutput"><span class="identifier">while_</span></code>). Their token value will always hold the iterator range of the matched input sequence. The token definitions for the <code class="computeroutput"><span class="identifier">identifier</span></code> and the integer <code class="computeroutput"><span class="identifier">constant</span></code> are specialized to expose an explicit token type each: <code class="computeroutput"><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span></code> and <code class="computeroutput"><span class="keyword">unsigned</span> <span class="keyword">int</span></code>. </p> <p> </p> <p> </p> <pre class="programlisting"><span class="comment">// these tokens expose the iterator_range of the matched input sequence </span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><></span> <span class="identifier">if_</span><span class="special">,</span> <span class="identifier">else_</span><span class="special">,</span> <span class="identifier">while_</span><span class="special">;</span> <span class="comment">// The following two tokens have an associated attribute type, 'identifier' </span><span class="comment">// carries a string (the identifier name) and 'constant' carries the </span><span class="comment">// matched integer value. </span><span class="comment">// </span><span class="comment">// Note: any token attribute type explicitly specified in a token_def<> </span><span class="comment">// declaration needs to be listed during token type definition as </span><span class="comment">// well (see the typedef for the token_type below). </span><span class="comment">// </span><span class="comment">// The conversion of the matched input to an instance of this type occurs </span><span class="comment">// once (on first access), which makes token attributes as efficient as </span><span class="comment">// possible. Moreover, token instances are constructed once by the lexer </span><span class="comment">// library. From this point on tokens are passed by reference only, </span><span class="comment">// avoiding them being copied around. </span><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><</span><span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> <span class="identifier">identifier</span><span class="special">;</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><</span><span class="keyword">unsigned</span> <span class="keyword">int</span><span class="special">></span> <span class="identifier">constant</span><span class="special">;</span> </pre> <p> </p> <p> </p> <p> As the parsers generated by <span class="emphasis"><em>Spirit.Qi</em></span> are fully attributed, any <span class="emphasis"><em>Spirit.Qi</em></span> parser component needs to expose a certain type as its parser attribute. Naturally, the <code class="computeroutput"><span class="identifier">lex</span><span class="special">::</span><span class="identifier">token_def</span><span class="special"><></span></code> exposes the token value type as its parser attribute, enabling a smooth integration with <span class="emphasis"><em>Spirit.Qi</em></span>. </p> <p> The next code snippet demonstrates how the required token value types are specified while defining the token type to use. All of the token value types used for at least one of the token definitions have to be re-iterated for the token definition as well. </p> <p> </p> <p> </p> <pre class="programlisting"><span class="comment">// This is the lexer token type to use. The second template parameter lists </span><span class="comment">// all attribute types used for token_def's during token definition (see </span><span class="comment">// calculator_tokens<> above). Here we use the predefined lexertl token </span><span class="comment">// type, but any compatible token type may be used instead. </span><span class="comment">// </span><span class="comment">// If you don't list any token attribute types in the following declaration </span><span class="comment">// (or just use the default token type: lexertl_token<base_iterator_type>) </span><span class="comment">// it will compile and work just fine, just a bit less efficient. This is </span><span class="comment">// because the token attribute will be generated from the matched input </span><span class="comment">// sequence every time it is requested. But as soon as you specify at </span><span class="comment">// least one token attribute type you'll have to list all attribute types </span><span class="comment">// used for token_def<> declarations in the token definition class above, </span><span class="comment">// otherwise compilation errors will occur. </span><span class="keyword">typedef</span> <span class="identifier">lex</span><span class="special">::</span><span class="identifier">lexertl</span><span class="special">::</span><span class="identifier">token</span><span class="special"><</span> <span class="identifier">base_iterator_type</span><span class="special">,</span> <span class="identifier">boost</span><span class="special">::</span><span class="identifier">mpl</span><span class="special">::</span><span class="identifier">vector</span><span class="special"><</span><span class="keyword">unsigned</span> <span class="keyword">int</span><span class="special">,</span> <span class="identifier">std</span><span class="special">::</span><span class="identifier">string</span><span class="special">></span> <span class="special">></span> <span class="identifier">token_type</span><span class="special">;</span> </pre> <p> </p> <p> </p> <p> To avoid the token to have a token value at all, the special tag <code class="computeroutput"><span class="identifier">omit</span></code> can be used: <code class="computeroutput"><span class="identifier">token_def</span><span class="special"><</span><span class="identifier">omit</span><span class="special">></span></code> and <code class="computeroutput"><span class="identifier">lexertl_token</span><span class="special"><</span><span class="identifier">base_iterator_type</span><span class="special">,</span> <span class="identifier">omit</span><span class="special">></span></code>. </p> </div> <table xmlns:rev="http://www.cs.rpi.edu/~gregod/boost/tools/doc/revision" width="100%"><tr> <td align="left"></td> <td align="right"><div class="copyright-footer">Copyright © 2001-2010 Joel de Guzman, Hartmut Kaiser<p> Distributed under the Boost Software License, Version 1.0. (See accompanying file LICENSE_1_0.txt or copy at <a href="http://www.boost.org/LICENSE_1_0.txt" target="_top">http://www.boost.org/LICENSE_1_0.txt</a>) </p> </div></td> </tr></table> <hr> <div class="spirit-nav"> <a accesskey="p" href="../lexer_primitives.html"><img src="../../../../../../../../doc/html/images/prev.png" alt="Prev"></a><a accesskey="u" href="../lexer_primitives.html"><img src="../../../../../../../../doc/html/images/up.png" alt="Up"></a><a accesskey="h" href="../../../../index.html"><img src="../../../../../../../../doc/html/images/home.png" alt="Home"></a><a accesskey="n" href="../lexer_tokenizing.html"><img src="../../../../../../../../doc/html/images/next.png" alt="Next"></a> </div> </body> </html>