Definition

The lexical analyzer (scanner) reads raw source characters and groups them into tokens — the smallest meaningful units (keywords, identifiers, operators, literals).

Key Points

• Input Buffering: source is read in large blocks (not char-by-char) to reduce I/O overhead. Double-buffering scheme uses two buffers with a sentinel character (EOF marker) to detect buffer boundaries efficiently without extra checks per character.
• Token, Lexeme, Pattern:
  • Token: category name, e.g. ID, NUM
  • Lexeme: actual character sequence, e.g. total, 42
  • Pattern: rule describing the lexeme, usually a regular expression
• Specification of tokens: done using regular expressions/regular definitions.
• Recognition of tokens: implemented via finite automata (transition diagrams) built from the regex patterns.
• Handles removal of whitespace, comments; tracks line numbers for error reporting.

Example

For total = rate + 12:

📌 CSS Frequency: High (explicitly listed as 2016 repeated topic) — often paired with "define token, lexeme, pattern" and buffering diagrams.

Model Answer (short)

"Lexical analysis converts a character stream into tokens using patterns specified as regular expressions and recognized via finite automata. Input buffering (commonly double-buffering with sentinels) minimizes the overhead of re-reading characters. A token is the category (e.g. ID), a lexeme is the actual matched text (e.g. 'total'), and a pattern is the rule (regex) that defines the token."