Editing Atari BASIC (section)

===Tokenizer===
When the user presses {{keypress|RETURN}} while editing, the current line is copied into the BASIC Input Line Buffer in memory between $0580 and $05FF.{{sfn|Crawford|1982|p=10.3}} Atari BASIC's [[token (parser)|tokenizer]] scans the text, converting each keyword to a single-byte token (for example, {{code|PRINT}} is $20),{{sfn|Crawford|1982|p=10.6}} each number to a six-byte floating point value, each variable name to an index into a table, and so on, until the line is fully turned into an easy to interpret format. The result is stored in an output buffer located at the first 256 bytes of the lowest available free memory, pointed to by the LOMEM pointer stored at $80, $81.{{sfn|Crawford|1982|p=10.3}} The output from the tokenizer is then relocated. The program is stored as a [[parse tree]].{{efn|Although the parse tree is implemented as a set of tables, which is really an implementation detail.}}

Shepardson referred to this complete-tokenizing concept as a "pre-compiling interpreter".{{sfn|Wilkinson|O'Brien|Laughton|1983|p=5}} The resulting tokenized code eliminates any parsing during runtime, making it run faster. It has the disadvantage that small constants, like 0 or 1, are six bytes each, longer than the original text.

A set of pointers (addresses) indicates various data: variable names are stored in the ''variable name table'' (VNTP &ndash; $82, $83) and their values are stored in the ''variable value table'' (pointed to at VVTP &ndash; $86, $87). By [[indirection|indirecting]] the variable names in this way, a reference to a variable needs only one byte to address its entry into the appropriate table. String variables have their own area (pointed to at STARP &ndash; $8C, $8D) as does the runtime stack (pointed to at RUNSTK &ndash; $8E, $8F) used to store the line numbers of looping statements (<code>FOR...NEXT</code>) and subroutines (<code>GOSUB...RETURN</code>). Finally, the end of BASIC memory usage is indicated by an address stored at MEMTOP &ndash; $90, $91) pointer.