Getting this error: Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element out of range

[agiovacchini]

I'm using last version pulled out from git (4.5.1) and built under OSx Yosemite, tryed both with jdk6 and jdk7 but with my file, which is: https://gist.github.com/agiovacchini/52cc6dc9210a65dfc659#file-example-g4 I get this error:

Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element out of range
at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:384)
at org.antlr.v4.runtime.atn.ATNSerializer.getSerialized(ATNSerializer.java:563)
at org.antlr.v4.codegen.model.SerializedATN.(SerializedATN.java:46)
at org.antlr.v4.codegen.model.Recognizer.(Recognizer.java:87)
at org.antlr.v4.codegen.model.Lexer.(Lexer.java:51)
at org.antlr.v4.codegen.OutputModelController.lexer(OutputModelController.java:176)
at org.antlr.v4.codegen.OutputModelController.buildLexerOutputModel(OutputModelController.java:129)
at org.antlr.v4.codegen.CodeGenerator.generateLexer(CodeGenerator.java:142)
at org.antlr.v4.codegen.CodeGenPipeline.process(CodeGenPipeline.java:73)
at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:428)
at org.antlr.v4.Tool.process(Tool.java:378)
at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:345)
at org.antlr.v4.Tool.main(Tool.java:192)

The launch script is like this:

!/bin/zsh

export CLASSPATH=".:./antlr4/dist/antlr4-4.5.1-complete.jar"
alias antlr4='java -jar ./antlr4/dist/antlr4-4.5.1-complete.jar'
alias grun='java org.antlr.v4.runtime.misc.TestRig'
antlr4 example.g4

[parrt]

likely classpath issue. remove any prior from class path
On Mar 19, 2015, at 4:11 PM, agiovacchini notifications@github.com wrote:

I'm using last version pulled out from git (4.5.1) and built under OSx Yosemite, tryed both with jdk6 and jdk7 but with my file, which is: https://gist.github.com/agiovacchini/52cc6dc9210a65dfc659#file-example-g4 I get this error:

Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element out of range
at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:384)
at org.antlr.v4.runtime.atn.ATNSerializer.getSerialized(ATNSerializer.java:563)
at org.antlr.v4.codegen.model.SerializedATN.(SerializedATN.java:46)
at org.antlr.v4.codegen.model.Recognizer.(Recognizer.java:87)
at org.antlr.v4.codegen.model.Lexer.(Lexer.java:51)
at org.antlr.v4.codegen.OutputModelController.lexer(OutputModelController.java:176)
at org.antlr.v4.codegen.OutputModelController.buildLexerOutputModel(OutputModelController.java:129)
at org.antlr.v4.codegen.CodeGenerator.generateLexer(CodeGenerator.java:142)
at org.antlr.v4.codegen.CodeGenPipeline.process(CodeGenPipeline.java:73)
at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:428)
at org.antlr.v4.Tool.process(Tool.java:378)
at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:345)
at org.antlr.v4.Tool.main(Tool.java:192)

—
Reply to this email directly or view it on GitHub.

[agiovacchini]

I've already tryed with this, but same result:

#!/bin/zsh
export CLASSPATH="./antlr4/dist/antlr4-4.5.1-complete.jar"
alias antlr4='/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java -jar ./antlr4/dist/antlr4-4.5.1-complete.jar'
alias grun='/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home/bin/java org.antlr.v4.runtime.misc.TestRig'

And I'm sure I'm using the right antlr jar, if I add my debugs and rebuild the jar I see them appearing:

nsets: 29
atn.lexerActions.length: 0
data.size: 743398
<<i:1 ><data[i]:1070>>
<<i:2 ><data[i]:54991>>
<<i:3 ><data[i]:33284>>
<<i:4 ><data[i]:44331>>
<<i:5 ><data[i]:17429>>
<<i:6 ><data[i]:44783>>
<<i:7 ><data[i]:36222>>
<<i:8 ><data[i]:43739>>
<<i:9 ><data[i]:0>>
<<i:10 ><data[i]:4408>>
<<i:11 ><data[i]:91799>>
Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element out of range
at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:384)
at org.antlr.v4.runtime.atn.ATNSerializer.getSerialized(ATNSerializer.java:563)
at org.antlr.v4.codegen.model.SerializedATN.(SerializedATN.java:46)
at org.antlr.v4.codegen.model.Recognizer.(Recognizer.java:87)
at org.antlr.v4.codegen.model.Lexer.(Lexer.java:51)
at org.antlr.v4.codegen.OutputModelController.lexer(OutputModelController.java:176)
at org.antlr.v4.codegen.OutputModelController.buildLexerOutputModel(OutputModelController.java:129)
at org.antlr.v4.codegen.CodeGenerator.generateLexer(CodeGenerator.java:142)
at org.antlr.v4.codegen.CodeGenPipeline.process(CodeGenPipeline.java:73)
at org.antlr.v4.Tool.processNonCombinedGrammar(Tool.java:428)
at org.antlr.v4.Tool.process(Tool.java:378)
at org.antlr.v4.Tool.processGrammarsOnCommandLine(Tool.java:345)
at org.antlr.v4.Tool.main(Tool.java:192)

These are the debugs I've added to ATNSerializer.java:

    ...
    //
    // LEXER ACTIONS
    //
    if (atn.grammarType == ATNType.LEXER) {
        data.add(atn.lexerActions.length);
        System.out.println("atn.lexerActions.length: " + atn.lexerActions.length);
        for (LexerAction action : atn.lexerActions) {
            data.add(action.getActionType().ordinal());
            System.out.println("action.getActionType().ordinal(): " + action.getActionType().ordinal());
            System.out.println("action.getActionType(): " + action.getActionType());
            switch (action.getActionType()) {
            case CHANNEL:
                int channel = ((LexerChannelAction)action).getChannel();
                data.add(channel != -1 ? channel : 0xFFFF);

                System.out.println("channel != -1 ? channel : 0xFFFF: " + (channel != -1 ? channel : 0xFFFF));
                data.add(0);
                break;

            case CUSTOM:
                int ruleIndex = ((LexerCustomAction)action).getRuleIndex();
                int actionIndex = ((LexerCustomAction)action).getActionIndex();
                data.add(ruleIndex != -1 ? ruleIndex : 0xFFFF);
                System.out.println("ruleIndex != -1 ? ruleIndex : 0xFFFF: " + (ruleIndex != -1 ? ruleIndex : 0xFFFF));
                data.add(actionIndex != -1 ? actionIndex : 0xFFFF);
                System.out.println("actionIndex != -1 ? actionIndex : 0xFFFF: " + (actionIndex != -1 ? actionIndex : 0xFFFF));
                break;

            case MODE:
                int mode = ((LexerModeAction)action).getMode();
                data.add(mode != -1 ? mode : 0xFFFF);
                System.out.println("mode != -1 ? mode : 0xFFFF: " + (mode != -1 ? mode : 0xFFFF));
                data.add(0);
                break;

            case MORE:
                data.add(0);
                data.add(0);
                break;

            case POP_MODE:
                data.add(0);
                data.add(0);
                break;

            case PUSH_MODE:
                mode = ((LexerPushModeAction)action).getMode();
                data.add(mode != -1 ? mode : 0xFFFF);
                System.out.println("mode != -1 ? mode : 0xFFFF: " + (mode != -1 ? mode : 0xFFFF));
                data.add(0);
                break;

            case SKIP:
                data.add(0);
                data.add(0);
                break;

            case TYPE:
                int type = ((LexerTypeAction)action).getType();
                data.add(type != -1 ? type : 0xFFFF);
                System.out.println("type != -1 ? type : 0xFFFF: " + (type != -1 ? type : 0xFFFF));
                data.add(0);
                break;

            default:
                String message = String.format(Locale.getDefault(), "The specified lexer action type %s is not valid.", action.getActionType());
                throw new IllegalArgumentException(message);
            }
        }
    }
    System.out.println("data.size: " + data.size());
    // don't adjust the first value since that's the version number
    for (int i = 1; i < data.size(); i++) {

        System.out.println("<<i:" + i + " ><data[i]:" + data.get(i) + ">>");
        if (data.get(i) < Character.MIN_VALUE || data.get(i) > Character.MAX_VALUE) {
            throw new UnsupportedOperationException("Serialized ATN data element out of range");
            //System.out.println("Serialized ATN data element out of range");
        }


        int value = (data.get(i) + 2) & 0xFFFF;
        data.set(i, value);
    }
    ...

I've seen #76 that seem very similar to my problem but is now fixed; the file attacched for issue 76 (https://gist.github.com/sharwell/3955844#file-example-g4) succeeds in my env; so seems like really I don't have any classpath issue.

Can you please try the file attached to this issue (https://gist.github.com/agiovacchini/52cc6dc9210a65dfc659#file-example-g4) in your env and see if it succeeds? If I make it smaller by cutting part of the "ingr:" entryes it succeeds in my env but not with the full list, maybe I've hitten some other limitation in data size in JDK/ANTLR?

[parrt]

have you completely regenerated all your grammars? are you sure 4.5.1 is built correctly?

[agiovacchini]

Yes, I have regenerated everything, before running again I did this:
rm *.class
rm *.java
rm *.tokens

This is the content of bild.log:

[03/20/15 09:39:20 ./bild.py:42 bilder.py:124] platform=darwin
[03/20/15 09:39:20 ./bild.py:42 bilder.py:125] jdk={'1.6': '/System/Library/Java/JavaVirtualMachines/1.6.0.jdk/Contents/Home', '1.7': '/Library/Java/JavaVirtualMachines/jdk1.7.0_21.jdk/Contents/Home', '1.8': '/Library/Java/JavaVirtualMachines/jdk1.8.0_31.jdk/Contents/Home'}
[03/20/15 09:39:20] require compile
[03/20/15 09:39:20 require ./bild.py:87 bilder.py:422] require compile
[03/20/15 09:39:20] require parsers
[03/20/15 09:39:20 require ./bild.py:73 bilder.py:422] require parsers
[03/20/15 09:39:20] build compile
[03/20/15 09:39:20 require ./bild.py:73 bilder.py:430] build compile
[03/20/15 09:39:20] build mkjar_complete
[03/20/15 09:39:20 require ./bild.py:87 bilder.py:430] build mkjar_complete
[03/20/15 09:39:21 print_and_log ./bild.py:114 bilder.py:761] Generated dist/antlr4-4.5.1-complete.jar

But I've tryed building myself after using with the same error the prebuilded jar downloaded from the ANTLR site.

Does my grammar work in your environment?

[parrt]

with 4.5 i get error and with 4.5.1. weird!

terence:~/tmp $ java -jar ~/antlr/code/antlr4/dist/antlr4-4.5.1-complete.jar example.g4 
Exception in thread "main" java.lang.UnsupportedOperationException: Serialized ATN data element out of range.
    at org.antlr.v4.runtime.atn.ATNSerializer.serialize(ATNSerializer.java:370)

I'll check at work.

[sharwell]

The ATN for the grammar (most likely the number of states and/or the number of transitions) is larger than the current representation is capable of encoding.

[agiovacchini]

Thank you for support. Is there some kind of workaround to this problem?

[sharwell]

@agiovacchini There's not that I know of right now. The serialization logic could be rewritten to use compressed integers like the ones used in ECMA-335 (bytecode for .NET), but it wouldn't be a small undertaking. It's arguably a good idea in the long run though.

[agiovacchini]

Ok, so the only option is to have a smaller grammar, right?

[joson]

I've fixed the same issue caused such a grammer in my .g4 file:
fragment
YearDigit
: '1900'..'3000'
;

I've modified it as below and then compiled successfully:
fragment
YearDigit
: '19' ('0'..'9') ('0'..'9')
| ('2'|'3') ('0'..'9') ('0'..'9') ('0'..'9')
;

[agiovacchini]

In my case I have many different literal strings to use as key in grammar but maybe I've found a workaround done this way:

1. Make a list of all possible keys and assign them an id
2. Encode the id in some base (eg: base64) to reduce the chars used by the id; take care the base has no digits used in grammar
3. Use the encoded ids in the .g4 grammar
4. Preprocess the data to be interpreted with the grammar by replacing the string literals with the base encoded ids
5. Feed the grammar with the file

It takes a bit of work to start using but then allows for many more possible keys given the less space used by one.

[parrt]

Hiya. I guess I'll close and I'll likely not fix this. thanks for documenting though!!