A Lexical Analysis Tool with Ambiguity Support
This addresses the challenge of handling lexical ambiguities in language processing for developers and researchers, though it appears incremental as it builds on existing lexical analysis concepts.
The authors tackled the problem of lexical ambiguities in languages by introducing Lamb, a lexical analyzer that generates a lexical analysis graph representing all possible token sequences, enabling parsers to perform context-sensitive lexical analysis and discard invalid sequences.
Lexical ambiguities naturally arise in languages. We present Lamb, a lexical analyzer that produces a lexical analysis graph describing all the possible sequences of tokens that can be found within the input string. Parsers can process such lexical analysis graphs and discard any sequence of tokens that does not produce a valid syntactic sentence, therefore performing, together with Lamb, a context-sensitive lexical analysis in lexically-ambiguous language specifications.