Protobuf3.g4 cannot distinguish between a field of type message and enum

[dsyer]

Example:

syntax = "proto3";
enum TestEnum {
        UNKNOWN = 0;
        FOO = 1;
        BAR = 2;
}
message TestMessage {
        TestEnum value = 1;
}

TestMessage.value is of type TestEnum (which is an enum by construction). The grammar can't tell the difference between a message type field and an enum type, so it matches as a message type (I assume because it is listed first in the grammar). The grammar looks like this:

...
messageType
    : (DOT)? (ident DOT)* messageName
    ;

enumType
    : (DOT)? (ident DOT)* enumName
    ;
...
messageName
    : ident
    ;

enumName
    : ident
    ;
...

Maybe there is no way for Antlr to express this difference? it's definitely a problem for anyone who wanted to use the generated parser.

Example command line to test:

$ grun Protobuf3 proto -tree
syntax = "proto3";
enum TestEnum {
        UNKNOWN = 0;
        FOO = 1;
        BAR = 2;
}
message TestMessage {
        TestEnum value = 1;
}
(proto (syntax syntax = "proto3" ;) (topLevelDef (enumDef enum (enumName (ident TestEnum)) (enumBody { (enumElement (enumField (ident UNKNOWN) = (intLit 0) ;)) (enumElement (enumField (ident FOO) = (intLit 1) ;)) (enumElement (enumField (ident BAR) = (intLit 2) ;)) }))) (topLevelDef (messageDef message (messageName (ident TestMessage)) (messageBody { (messageElement (field (type (messageType (messageName (ident TestEnum)))) (fieldName (ident value)) = (fieldNumber (intLit 1)) ;)) }))) <EOF>)

[kaby76]

It means that the grammar is not context-free, that it is ambiguous. ANTLR can handle this, but a symbol table has to be added. We'll need to know how it resolves types. Examples will be help.

[dsyer]

Interesting. Is a "symbol table" a feature of Antlr, or of the thing that you write with the generated parser? I didn't find any obvious links in the docs. (Antlr noob here, sorry.)

I gave an example above. It's fairly simple, so in real life other corner cases would arise, but it's representative.

[kaby76]

Interesting. Is a "symbol table" a feature of Antlr, or of the thing that you write with the generated parser? I didn't find any obvious links in the docs. (Antlr noob here, sorry.)

I gave an example above. It's fairly simple, so in real life other corner cases would arise, but it's representative.

The symbol table is hand-written code within the base classes for a grammar. There aren't any examples of it in grammars-v4 as of yet, but I am adding one for golang. Essentially, one needs to add semantic predicates at key points in the grammar to make sure the id matches the type expected. If applied uses can occur before defining occurrences of the id, then the parse has to be two passes.

[dsyer]

Thanks. An example would be very helpful, especially with the 2 passes. I'm not sure I see how that would work otherwise.

[kaby76]

It looks like one pass should be fine because enum TestEnum { (the defining occurrence for TestEnum) occurs before the use TestEnum value = 1;. At least for this example, two passes would not be needed because the use never occurs before the definition. So, what would happen in the parse, it would enter TestEnum as an enum identifier while recognizing the definition. Parsing the use, there would be semantic predicates elimating uses for anything but enumName, e.g., as a messageName. Is there a concept of "include" files or libraries in Protobuf as in C?

[dsyer]

Yes, include files are a thing (it's called "import"), so you need to parse all the imports as well before you can necessarily say that you are ready to identify all the enums. And definitions don't have to precede usage.

[kaby76]

A preliminary fix for the ambiguity is via PR #4538. It is currently a one-pass parse with no import, and no predefined types. Empty non-terminals are added to perform symbol table construction within actions. Semantic predicates check the use is of the correct type. Many parse errors because if a field type is not declared, the parse fails. The .trq file verifies that TestEnum is actually found as an enum type (declared prior to use).

[kaby76]

@dsyer Is there a particular target you want supported? I have Antlr4ng, CSharp, Java, and TypeScript. I may stop with just these for now.

[dsyer]

I'm using Java, so I'd be delighted to see that, but I can see what you did already with the CSharp.

[kaby76]

@dsyer Please give #4538 a try.