Editing Lexical analysis (section)

=== Semicolon insertion ===
Many languages use the semicolon as a statement terminator. Most often this is mandatory, but in some languages the semicolon is optional in many contexts. This is mainly done at the lexer level, where the lexer outputs a semicolon into the token stream, despite one not being present in the input character stream, and is termed ''semicolon insertion'' or ''automatic semicolon insertion''. In these cases, semicolons are part of the formal phrase grammar of the language, but may not be found in input text, as they can be inserted by the lexer. Optional semicolons or other terminators or separators are also sometimes handled at the parser level, notably in the case of [[trailing comma]]s or semicolons.

Semicolon insertion is a feature of [[BCPL]] and its distant descendant [[Go (programming language)|Go]],<ref>''[http://golang.org/doc/effective_go.html Effective Go]'', "[http://golang.org/doc/effective_go.html#semicolons Semicolons]"</ref> though it is absent in B or C.<ref>"[https://groups.google.com/forum/#!topic/golang-nuts/XuMrWI0Q8uk Semicolons in Go]", golang-nuts, Rob 'Commander' Pike, 12/10/09</ref> Semicolon insertion is present in [[JavaScript]], though the rules are somewhat complex and much-criticized; to avoid bugs, some recommend always using semicolons, while others use initial semicolons, termed [[defensive semicolon]]s, at the start of potentially ambiguous statements.

Semicolon insertion (in languages with semicolon-terminated statements) and line continuation (in languages with newline-terminated statements) can be seen as complementary: Semicolon insertion adds a token even though newlines generally do ''not'' generate tokens, while line continuation prevents a token from being generated even though newlines generally ''do'' generate tokens.