Editing Lexical analysis (section)

== Lexical grammar ==
The specification of a [[programming language]] often includes a set of rules, the [[lexical grammar]], which defines the lexical syntax. The lexical syntax is usually a [[regular language]], with the grammar rules consisting of [[regular expression]]s; they define the set of possible character sequences (lexemes) of a token. A lexer recognizes strings, and for each kind of string found, the lexical program takes an action, most simply producing a token.

Two important common lexical categories are [[Whitespace character|white space]] and [[Comment (computer programming)|comments]]. These are also defined in the grammar and processed by the lexer but may be discarded (not producing any tokens) and considered ''non-significant'', at most separating two tokens (as in <code>if&nbsp;x</code> instead of <code>ifx</code>). There are two important exceptions to this. First, in [[off-side rule]] languages that delimit [[Block (programming)|blocks]] with indenting, initial whitespace is significant, as it determines block structure, and is generally handled at the lexer level; see [[#Phrase structure|phrase structure]], below. Secondly, in some uses of lexers, comments and whitespace must be preserved – for examples, a [[prettyprint]]er also needs to output the comments and some debugging tools may provide messages to the programmer showing the original source code. In the 1960s, notably for [[ALGOL]], whitespace and comments were eliminated as part of the [[line reconstruction]] phase (the initial phase of the [[compiler frontend]]), but this separate phase has been eliminated and these are now handled by the lexer.