Spirit.Lex Tutorials Overview

| Home | Libraries | People | FAQ | More

#### Spirit.Lex Tutorials Overview

The Spirit.Lex library implements several components on top of possibly different lexer generator libraries. It exposes a pair of iterators, which, when dereferenced, return a stream of tokens generated from the underlying character stream. The generated tokens are based on the token definitions supplied by the user.

Currently, Spirit.Lex is built on top of Ben Hansons excellent Lexertl library (which is a proposed Boost library). Lexertl provides the necessary functionality to build state machines based on a set of supplied regular expressions. But Spirit.Lex is not restricted to be used with Lexertl. We expect it to be usable in conjunction with any other lexical scanner generator library, all what needs to be implemented is a set of wrapper objects exposing a well defined interface as described in this documentation.

| Note
For the sake of clarity all examples in this documentation assume Spirit.Lex to be used on top of Lexertl.

Building a lexer using Spirit.Lex is highly configurable, where most of this configuration is done at compile time. Almost all of the configurable parameters have generally useful default values, allowing project startup to be a easy and straightforward task. Here is a (non-complete) list of features you can tweak to adjust the generated lexer instance to the actual needs:

- Select and customize the token type to be generated by the lexer instance.
- Select and customize the token value types the generated token instances will be able to hold.
- Select the iterator type of the underlying input stream, which will be used as the source for the character stream to tokenize.
- Customize the iterator type returned by the lexer to enable debug support, special handling of certain input sequences, etc.
- Select the dynamic or the static runtime model for the lexical analyzer.

Special care has been taken during the development of the library that optimal code will be generated regardless of the configuration options selected.

The series of tutorial examples of this section will guide you through some common use cases helping to understand the big picture. The first two quick start examples (Lex Quickstart 1 - A word counter using Spirit.Lex and Lex Quickstart 2 - A better word counter using Spirit.Lex) introduce the Spirit.Lex library while building two standalone applications, not being connected to or depending on any other part of Spirit. The section Lex Quickstart 3 - Counting Words Using a Parser demonstrates how to use a lexer in conjunction with a parser (where obviously the parser is built using Spirit.Qi).