Difference between revisions of "Scanner/Tokenizer"

From Lazarus wiki
(Added link to Macro internal)
Line 8: Line 8:
 
variables of the compiler, and that all illegal characters found in the
 
variables of the compiler, and that all illegal characters found in the
 
input stream cause an error.
 
input stream cause an error.
 +
 +
Info about how macros work: [[Macro internals]]
  
 
== Architecture ==
 
== Architecture ==

Revision as of 23:04, 9 January 2005

back to contents FPC internals

Scanner/Tokenizer

The scanner and tokenizer is used to construct an input stream of tokens which will be fed to the parser. It is in this stage that the preprocessing is done, that all read compiler directives change the internal state variables of the compiler, and that all illegal characters found in the input stream cause an error.

Info about how macros work: Macro internals

Architecture

The general architecture of the scanner is shown in the following figure:

Todo: Add Image

Several types can be read from the input stream, a string, handled by readstring, a numeric value, handled by readnumeric, comments , compiler and preprocessor directives.

Next chapter: The parse tree