How to make custom token builder trigger only in certain cases

[dhrubomoy]

I am new to langium so this might be a silly question that has straight forward answer. I have some rules:

LoadStatement: 'load' OtherVeryComplexRules 'from' FILE_PATH ';' ;
// ... other rules
DropFieldStatement: 'drop' SomeComplexRules 'from'  table=ID (',' table+=ID)* ;
// ... other rules

// terminals
terminal ID: /[_a-zA-Z][\w_]*/;

Here FILE_PATH matches any string (special chars, '/', '\' etc). I added FILE_PATH a terminal with regex pattern , but it creates conflicts with the ID terminal. After many failed attempts, the only solution I found by reading the documentation is to use synthetic token concept used in indent/detent detection here. Added FILE_PATH as a terminal terminal FILE_PATH: 'synthetic:filepath';, then created a custom token builder with a function to match.

Basically in the pattern function in the custom token builder, I am checking if the last token is 'from' keyword, only then match anything until it reaches a semicolon.

But the problem arises when I add other Rules that also have a 'from' keyword but not followed by a FILE_PATH, e.g. the DropFieldStatement rule above.

At this point, using synthetic token option is also not working. Is there a way to trigger the pattern/matcher function for FILE_PATH only when it is mentioned in the rule, e.g. in LoadStatement after 'from' keyword. and not in DropFieldStatement after 'from' keyword

Is there any other solution that will help me achieve my goal? Any help would be appreciated. Thanks in advance.

Here is the full token builder class in case it helps.

import { createToken, TokenType, TokenVocabulary } from "chevrotain";
import {
    DefaultTokenBuilder,
    Grammar,
    GrammarAST,
    TokenBuilderOptions,
} from "langium";

// charcodes
const SEMICOLON = 59;
const NEWLINE = 10;
const DOUBLE_QUOTE = 34;
const SINGLE_QUOTE = 39;

export class MyScriptTokenBuilder extends DefaultTokenBuilder {
    override buildTokens(
        grammar: Grammar,
        options?: TokenBuilderOptions | undefined
    ): TokenVocabulary {
        const out = super.buildTokens(grammar, options);
        return out;
    }

    private processQuotedString(
        text: string,
        charCode: number,
        endOffset: number,
        startQuote: number,
        endQuote: number
    ) {
        while (!isNaN(charCode)) {
            endOffset++;
            charCode = text.charCodeAt(endOffset);
            if (charCode === startQuote) {
                endOffset++;
                charCode = text.charCodeAt(endOffset);
                if (charCode === endQuote) {
                    // allow for escaped quote e.g. ]] or ""
                    endOffset++;
                } else {
                    break;
                }
            }
        }
        return endOffset;
    }

    protected override buildTerminalToken(
        terminal: GrammarAST.TerminalRule
    ): TokenType {
        const tokenType = super.buildTerminalToken(terminal);
        if (tokenType.name === "FILE_PATH") {
            return createToken({
                name: "FILE_PATH",
                pattern: (text, startOffset, tokens): any => {
                    if (tokens.length) {
                        const tokenName =
                            tokens[tokens.length - 1]?.tokenType?.name;
                        if (
                            tokenName === "from"
                        ) {
                            let endOffset = startOffset;
                            let charCode = text.charCodeAt(endOffset);
                            if (charCode === DOUBLE_QUOTE) {
                                endOffset = this.processQuotedString(
                                    text,
                                    charCode,
                                    endOffset,
                                    DOUBLE_QUOTE,
                                    DOUBLE_QUOTE
                                );
                            } else if (charCode === SINGLE_QUOTE) {
                                endOffset = this.processQuotedString(
                                    text,
                                    charCode,
                                    endOffset,
                                    SINGLE_QUOTE,
                                    SINGLE_QUOTE
                                );
                            } else {
                                while (!isNaN(charCode)) {
                                    endOffset++;
                                    charCode = text.charCodeAt(endOffset);
                                    if (
                                        charCode === NEWLINE ||
                                        charCode === SEMICOLON
                                    ) {
                                        break;
                                    }
                                }
                            }

                            if (endOffset === startOffset) {
                                return null;
                            } else {
                                const matchedString = text.substring(
                                    startOffset,
                                    endOffset
                                );

                                return [matchedString];
                            }
                            return null;
                        }
                    }
                    return null;
                },
                line_breaks: false,
            });
        }
        return tokenType;
    }
}

[msujew]

Hey @dhrubomoy,

unfortunately, that's an inherent limitation of how LL language parsers work. During the tokenization phase, you have rather limited information about what the tokens in the input actually represent. Due to the other from keyword in your language, even features such as lexer modes or positive lookbehind in the regex will not work as expected. Other systems such as PEG can take the context of the current rule into account during the parsing/tokenization of the input (PEG has no separate lexing/parsing phase), but PEG parsers are rather slow as a consequence.

My recommendation would be to use a system similar to TypeScript, which uses STRING terminals for imports. See also our own import grammar here:

langium/packages/langium/src/grammar/langium-grammar.langium

Lines 65 to 66 in 1a93a22

	GrammarImport returns GrammarImport:
	'import' path=STRING ';'?;

Generally, when referencing paths, I would always recommend to use STRING, since except for a very specific list of characters, a file path can contain essentially anything, which makes it rather annoying to tokenize correctly (as you already found out).

[dhrubomoy]

Thank you so much for the reply. Appreciate it. :)