Skip to content

Commit 4a30b9a

Browse files
Jim Idlesharknoonjosufraericvergnaud
authored
Feature/fixembedding (#4176)
* feat: Createa n Init routine for BaseATNConfig so we can embed sructs rather than allocate to pointers Signed-off-by: Jim.Idle <[email protected]> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <[email protected]> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <[email protected]> * feat: Switch the use of pointers to embedded ATN states to true embeddding Saves many allocations and grbage collections Signed-off-by: Jim.Idle <[email protected]> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <[email protected]> * Feature/docclean Greatly improve the godoc comments in the runtime (#4169) * doc: Updates to some of the Go doc comments to start a ful ldocumentation cleanup Signed-off-by: Jim.Idle <[email protected]> * doc: More documentation fixes. Using this as a method of forcing myself to read every line of code in the runtime, and therefore discover mistakes in the original implementation. And, of course, actually working docs for the Go runtime, can only be a good thing. Signed-off-by: Jim.Idle <[email protected]> * doc: More documentation fixes Also changes the exporet level of a some variables and funcs that were not correct, even though no user has currently needed them it would seem. Signed-off-by: Jim.Idle <[email protected]> * doc: Many updates to document exported fuctions correctly and reformat the ingerited Java code It looks like a massive amount of changes, but it is almost all doc or changing exports or renaming unused paramters etc to make the Go linter happy. No actual code changes yet. Signed-off-by: Jim.Idle <[email protected]> * doc: More additions and corrections to the Go documentation for the runtime Signed-off-by: Jim.Idle <[email protected]> * doc: Final clean of exported func and type documentation There will be more to do here as there are a lot of things that are hidden internal to the antlr package that probably should not be. There are also a lot of exported funcs and types without any documentation, that will eventually need to be cleaned up. Signed-off-by: Jim.Idle <[email protected]> * Changed Parser typings (#4149) Signed-off-by: Josua Frank <[email protected]> Co-authored-by: Josua Frank <[email protected]> Signed-off-by: Jim.Idle <[email protected]> * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file (#4163) Arrrgh! Signed-off-by: Jim.Idle <[email protected]> * present antlr before versioning (#4156) Signed-off-by: Jim.Idle <[email protected]> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (#4161) Signed-off-by: Jim.Idle <[email protected]> * Feature/gotestfix (#4168) * fix: Fixes the failing go runtime test suite which was missing the /v4 off the replace option on the go.mod file Arrrgh! Signed-off-by: Jim.Idle <[email protected]> * present antlr before versioning (#4156) Signed-off-by: Jim.Idle <[email protected]> * fix: Prevent use of labels such as start= from generating code that clashes with builtin funcs (#4161) Signed-off-by: Jim.Idle <[email protected]> * fix: Cater for the fact that some test rules use start as a label or rule name As a fix for other cvode gen errors when start, end, or exception are used as label names, they are now translated to have a suffix of `_` at code gen time. However, the runtime tests sometimes use start as a rule name and so we must now cater for this in the tests. Signed-off-by: Jim.Idle <[email protected]> --------- Signed-off-by: Jim.Idle <[email protected]> Co-authored-by: ericvergnaud <[email protected]> Signed-off-by: Jim.Idle <[email protected]> --------- Signed-off-by: Jim.Idle <[email protected]> Signed-off-by: Josua Frank <[email protected]> Co-authored-by: Josua Frank <[email protected]> Co-authored-by: Josua Frank <[email protected]> Co-authored-by: ericvergnaud <[email protected]> * feat: Change BaseATNConfig to be properly embedded in other structs such as LexerATNConfig instead of by pointer This is the first of many changes that switches the embedded class structure that was copying Java class hieracrchy from allocations/new to proper embedding such that any struct is allocated with one allocation not two or more. Main PR will cover what this means. Signed-off-by: Jim.Idle <[email protected]> * feat: Change embedding for ATNBaseSimulator to true embedding instaed of pointer Saves an extra allocation and helps the GC Signed-off-by: Jim.Idle <[email protected]> * fix: Correct the way that PredictionContext is compared for merge Should reduce allocation count by tons. Signed-off-by: Jim.Idle <[email protected]> * doc: Merge documentation updates Signed-off-by: Jim.Idle <[email protected]> * feat: Rework predictions tructs to use emedding instead of pointers Signed-off-by: Jim.Idle <[email protected]> * feat: more reworking of PredictionContext for embedding Signed-off-by: Jim.Idle <[email protected]> * feat: Ensure that EmptyPredictionContext is correctly initialized Rework of the variaous PredictionContexts has reduced memory allocations to between 30% and 50% of previous version. Signed-off-by: Jim.Idle <[email protected]> * feat: Change from use of type casting to using stored type Signed-off-by: Jim.Idle <[email protected]> * feat: Convert CommonToken to true emedding rather than pointers Signed-off-by: Jim.Idle <[email protected]> --------- Signed-off-by: Jim.Idle <[email protected]> Signed-off-by: Josua Frank <[email protected]> Co-authored-by: Josua Frank <[email protected]> Co-authored-by: Josua Frank <[email protected]> Co-authored-by: ericvergnaud <[email protected]>
1 parent 95fd266 commit 4a30b9a

22 files changed

+1279
-1109
lines changed
Lines changed: 115 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,115 @@
1+
package antlr
2+
3+
import (
4+
"golang.org/x/exp/slices"
5+
"strconv"
6+
)
7+
8+
type ArrayPredictionContext struct {
9+
BasePredictionContext
10+
parents []PredictionContext
11+
returnStates []int
12+
}
13+
14+
func NewArrayPredictionContext(parents []PredictionContext, returnStates []int) *ArrayPredictionContext {
15+
// Parent can be nil only if full ctx mode and we make an array
16+
// from {@link //EMPTY} and non-empty. We merge {@link //EMPTY} by using
17+
// nil parent and
18+
// returnState == {@link //EmptyReturnState}.
19+
hash := murmurInit(1)
20+
for _, parent := range parents {
21+
hash = murmurUpdate(hash, parent.Hash())
22+
}
23+
for _, returnState := range returnStates {
24+
hash = murmurUpdate(hash, returnState)
25+
}
26+
hash = murmurFinish(hash, len(parents)<<1)
27+
28+
return &ArrayPredictionContext{
29+
BasePredictionContext: BasePredictionContext{
30+
cachedHash: hash,
31+
pcType: PredictionContextArray,
32+
},
33+
parents: parents,
34+
returnStates: returnStates,
35+
}
36+
}
37+
38+
func (a *ArrayPredictionContext) GetReturnStates() []int {
39+
return a.returnStates
40+
}
41+
42+
func (a *ArrayPredictionContext) hasEmptyPath() bool {
43+
return a.getReturnState(a.length()-1) == BasePredictionContextEmptyReturnState
44+
}
45+
46+
func (a *ArrayPredictionContext) isEmpty() bool {
47+
// since EmptyReturnState can only appear in the last position, we
48+
// don't need to verify that size==1
49+
return a.returnStates[0] == BasePredictionContextEmptyReturnState
50+
}
51+
52+
func (a *ArrayPredictionContext) length() int {
53+
return len(a.returnStates)
54+
}
55+
56+
func (a *ArrayPredictionContext) GetParent(index int) PredictionContext {
57+
return a.parents[index]
58+
}
59+
60+
func (a *ArrayPredictionContext) getReturnState(index int) int {
61+
return a.returnStates[index]
62+
}
63+
64+
// Equals is the default comparison function for ArrayPredictionContext when no specialized
65+
// implementation is needed for a collection
66+
func (a *ArrayPredictionContext) Equals(o interface{}) bool {
67+
if a == o {
68+
return true
69+
}
70+
other, ok := o.(*ArrayPredictionContext)
71+
if !ok {
72+
return false
73+
}
74+
if a.cachedHash != other.Hash() {
75+
return false // can't be same if hash is different
76+
}
77+
78+
// Must compare the actual array elements and not just the array address
79+
//
80+
return slices.Equal(a.returnStates, other.returnStates) &&
81+
slices.EqualFunc(a.parents, other.parents, func(x, y PredictionContext) bool {
82+
return x.Equals(y)
83+
})
84+
}
85+
86+
// Hash is the default hash function for ArrayPredictionContext when no specialized
87+
// implementation is needed for a collection
88+
func (a *ArrayPredictionContext) Hash() int {
89+
return a.BasePredictionContext.cachedHash
90+
}
91+
92+
func (a *ArrayPredictionContext) String() string {
93+
if a.isEmpty() {
94+
return "[]"
95+
}
96+
97+
s := "["
98+
for i := 0; i < len(a.returnStates); i++ {
99+
if i > 0 {
100+
s = s + ", "
101+
}
102+
if a.returnStates[i] == BasePredictionContextEmptyReturnState {
103+
s = s + "$"
104+
continue
105+
}
106+
s = s + strconv.Itoa(a.returnStates[i])
107+
if a.parents[i] != nil {
108+
s = s + " " + a.parents[i].String()
109+
} else {
110+
s = s + "nil"
111+
}
112+
}
113+
114+
return s + "]"
115+
}

runtime/Go/antlr/v4/atn_config.go

Lines changed: 51 additions & 20 deletions
Original file line numberDiff line numberDiff line change
@@ -106,17 +106,23 @@ func NewBaseATNConfig1(c ATNConfig, state ATNState, context PredictionContext) *
106106
// are just wrappers around this one.
107107
func NewBaseATNConfig(c ATNConfig, state ATNState, context PredictionContext, semanticContext SemanticContext) *BaseATNConfig {
108108
if semanticContext == nil {
109-
panic("semanticContext cannot be nil")
109+
panic("semanticContext cannot be nil") // TODO: Remove this - probably put here for some bug that is now fixed
110110
}
111111

112-
return &BaseATNConfig{
113-
state: state,
114-
alt: c.GetAlt(),
115-
context: context,
116-
semanticContext: semanticContext,
117-
reachesIntoOuterContext: c.GetReachesIntoOuterContext(),
118-
precedenceFilterSuppressed: c.getPrecedenceFilterSuppressed(),
119-
}
112+
b := &BaseATNConfig{}
113+
b.InitBaseATNConfig(c, state, c.GetAlt(), context, semanticContext)
114+
115+
return b
116+
}
117+
118+
func (b *BaseATNConfig) InitBaseATNConfig(c ATNConfig, state ATNState, alt int, context PredictionContext, semanticContext SemanticContext) {
119+
120+
b.state = state
121+
b.alt = alt
122+
b.context = context
123+
b.semanticContext = semanticContext
124+
b.reachesIntoOuterContext = c.GetReachesIntoOuterContext()
125+
b.precedenceFilterSuppressed = c.getPrecedenceFilterSuppressed()
120126
}
121127

122128
func (b *BaseATNConfig) getPrecedenceFilterSuppressed() bool {
@@ -237,49 +243,74 @@ func (b *BaseATNConfig) String() string {
237243
// BaseATNConfig struct.
238244
// TODO: Stop using a pointer and embed the struct instead as this saves allocations. Same for the LexerATNConfig "constructors"
239245
type LexerATNConfig struct {
240-
*BaseATNConfig
246+
BaseATNConfig
241247
lexerActionExecutor *LexerActionExecutor
242248
passedThroughNonGreedyDecision bool
243249
}
244250

245251
func NewLexerATNConfig6(state ATNState, alt int, context PredictionContext) *LexerATNConfig {
246-
return &LexerATNConfig{BaseATNConfig: NewBaseATNConfig5(state, alt, context, SemanticContextNone)}
252+
253+
return &LexerATNConfig{
254+
BaseATNConfig: BaseATNConfig{
255+
state: state,
256+
alt: alt,
257+
context: context,
258+
semanticContext: SemanticContextNone,
259+
},
260+
}
247261
}
248262

249263
func NewLexerATNConfig5(state ATNState, alt int, context PredictionContext, lexerActionExecutor *LexerActionExecutor) *LexerATNConfig {
250264
return &LexerATNConfig{
251-
BaseATNConfig: NewBaseATNConfig5(state, alt, context, SemanticContextNone),
265+
BaseATNConfig: BaseATNConfig{
266+
state: state,
267+
alt: alt,
268+
context: context,
269+
semanticContext: SemanticContextNone,
270+
},
252271
lexerActionExecutor: lexerActionExecutor,
253272
}
254273
}
255274

256275
func NewLexerATNConfig4(c *LexerATNConfig, state ATNState) *LexerATNConfig {
257-
return &LexerATNConfig{
258-
BaseATNConfig: NewBaseATNConfig(c, state, c.GetContext(), c.GetSemanticContext()),
276+
lac := &LexerATNConfig{
277+
259278
lexerActionExecutor: c.lexerActionExecutor,
260279
passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
261280
}
281+
lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), c.GetContext(), c.GetSemanticContext())
282+
return lac
262283
}
263284

264285
func NewLexerATNConfig3(c *LexerATNConfig, state ATNState, lexerActionExecutor *LexerActionExecutor) *LexerATNConfig {
265-
return &LexerATNConfig{
266-
BaseATNConfig: NewBaseATNConfig(c, state, c.GetContext(), c.GetSemanticContext()),
286+
lac := &LexerATNConfig{
267287
lexerActionExecutor: lexerActionExecutor,
268288
passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
269289
}
290+
lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), c.GetContext(), c.GetSemanticContext())
291+
return lac
270292
}
271293

272294
func NewLexerATNConfig2(c *LexerATNConfig, state ATNState, context PredictionContext) *LexerATNConfig {
273-
return &LexerATNConfig{
274-
BaseATNConfig: NewBaseATNConfig(c, state, context, c.GetSemanticContext()),
295+
lac := &LexerATNConfig{
275296
lexerActionExecutor: c.lexerActionExecutor,
276297
passedThroughNonGreedyDecision: checkNonGreedyDecision(c, state),
277298
}
299+
lac.BaseATNConfig.InitBaseATNConfig(c, state, c.GetAlt(), context, c.GetSemanticContext())
300+
return lac
278301
}
279302

280303
//goland:noinspection GoUnusedExportedFunction
281304
func NewLexerATNConfig1(state ATNState, alt int, context PredictionContext) *LexerATNConfig {
282-
return &LexerATNConfig{BaseATNConfig: NewBaseATNConfig5(state, alt, context, SemanticContextNone)}
305+
lac := &LexerATNConfig{
306+
BaseATNConfig: BaseATNConfig{
307+
state: state,
308+
alt: alt,
309+
context: context,
310+
semanticContext: SemanticContextNone,
311+
},
312+
}
313+
return lac
283314
}
284315

285316
// Hash is the default hash function for LexerATNConfig objects, it can be used directly or via
@@ -330,7 +361,7 @@ func (l *LexerATNConfig) Equals(other Collectable[ATNConfig]) bool {
330361
return false
331362
}
332363

333-
return l.BaseATNConfig.Equals(otherT.BaseATNConfig)
364+
return l.BaseATNConfig.Equals(&otherT.BaseATNConfig)
334365
}
335366

336367
func checkNonGreedyDecision(source *LexerATNConfig, target ATNState) bool {

0 commit comments

Comments
 (0)