[Go] SBE implementation in Go using flyweights (#951)

* [Go] SBE implementation in Go using flyweights #765

To generate flyweights instead of structs, set the following flag:

```
sbe.go.generate.generate.flyweights=true
```

* [Go] Checkstyle fix

* [Go] Fix null string return value

* [Go] Remove spurious type definition for choices

* [Go] Fix formatting in WrapForDecode

* [Go] Add returning an array by []byte

* [Go] Fix array not present in version

* [Go] Add ComputeLength methods

* [Go] Fix some compiler warnings and remove unused code

* [Go] Draft of OTF decoder

* [Go] Add OTF Json Printer

* [Go] Fix SbeBlockAndHeaderLength and add constants for composite sizes

* [Go] Add IrDecoder for runtime types

* [Go] Add enum get method

* [Go] Fixes for updating component counts in irdecoder

* [Go] Run goimports

* [Go] Update with latest changes

* [Go] Change the file structure for GOPATH

* [Go] Fix copywrite year

* [Go] Fix tests for otf

This generates flyweights in the sbe-tool build for use in the otf library.

* [Go] Add helpers for otf PrimitiveValue

Author: ethanf

.gitignore

@@ -65,31 +65,18 @@ cppbuild/Win32
 
 # golang
 gocode/pkg
-gocode/src/baseline/*.go
-!gocode/src/baseline/*_test.go
-gocode/src/baseline-bigendian/*.go
-!gocode/src/baseline-bigendian/*_test.go
-gocode/src/composite/*.go
-!gocode/src/composite/*_test.go
-gocode/src/composite_elements/*.go
-!gocode/src/composite_elements/*_test.go
-gocode/src/since-deprecated/*.go
-!gocode/src/since-deprecated/*_test.go
-gocode/src/group_with_data*/*.go
-!gocode/src/group_with_data*/*_test.go
-gocode/src/mktdata/*.go
-!gocode/src/mktdata/*_test.go
-gocode/src/simple/*.go
-!gocode/src/simple/*_test.go
-gocode/src/issue*/*.go
-!gocode/src/issue*/*_test.go
-gocode/src/*/*/*.go
-!gocode/src/*/*/*_test.go
-gocode/src/example-schema/example-schema*
-gocode/src/example-schema/cpu.out
-gocode/src/example-socket-clientserver/example-socket-clientserver
-gocode/src/extension
-gocode/src/extension2
+gocode/*/pkg
+gocode/struct/src/*/*.go
+!gocode/struct/src/*/*_test.go
+gocode/struct/src/*/*/*.go
+!gocode/struct/src/*/*/*_test.go
+gocode/struct/src/example-schema/example-schema*
+gocode/struct/src/example-socket-clientserver/example-socket-clientserver
+gocode/flyweight/src/*/*.go
+!gocode/flyweight/src/*/*_test.go
+gocode/flyweight/src/*/*/*.go
+!gocode/flyweight/src/*/*/*_test.go
+gocode/flyweight/src/otf/code-generation-schema.sbeir
 
 # csharp
 csharp/*/bin

README.md

@@ -97,6 +97,8 @@ For convenience on Linux, a gnu Makefile is provided that runs some tests and co
     $ cd gocode
     # make # test, examples, bench
 
+Go supports both generating Go structs with encode / decode methods, and flyweights like the other languages. Structs are generated by default for compatibility. Set `sbe.go.generate.generate.flyweights=true` to generate flyweights.
+
 Users of golang generated code should see the [user
 documentation](https://github.com/real-logic/simple-binary-encoding/wiki/Golang-User-Guide).
 

build.gradle

@@ -700,7 +700,7 @@ tasks.register('generateGolangCodecTestComposite', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src',
+            'sbe.output.dir': 'gocode/struct/src',
             'sbe.target.language': 'golang')
     args = ['sbe-tool/src/test/resources/composite-elements-schema-rc4.xml']
 }
@@ -709,7 +709,7 @@ tasks.register('generateGolangCodecTestBasic', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src/basic',
+            'sbe.output.dir': 'gocode/struct/src/basic',
             'sbe.target.language': 'golang')
     args = ['sbe-tool/src/test/resources/basic-types-schema.xml']
 }
@@ -718,7 +718,7 @@ tasks.register('generateGolangCodecTestGroup', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src/group',
+            'sbe.output.dir': 'gocode/struct/src/group',
             'sbe.target.language': 'golang')
     args = ['sbe-tool/src/test/resources/basic-group-schema.xml']
 }
@@ -727,7 +727,7 @@ tasks.register('generateGolangCodecTestVarData', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src/vardata',
+            'sbe.output.dir': 'gocode/struct/src/vardata',
             'sbe.target.language': 'golang')
     args = ['sbe-tool/src/test/resources/basic-variable-length-schema.xml']
 }
@@ -736,7 +736,7 @@ tasks.register('generateGolangCodecsWithXIncludes', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src',
+            'sbe.output.dir': 'gocode/struct/src',
             'sbe.target.language': 'golang',
             'sbe.xinclude.aware': 'true',
             'sbe.validation.xsd': validationXsdPath)
@@ -748,7 +748,87 @@ tasks.register('generateGolangCodecsWithXSD', JavaExec) {
     mainClass.set('uk.co.real_logic.sbe.SbeTool')
     classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
     systemProperties(
-            'sbe.output.dir': 'gocode/src',
+            'sbe.output.dir': 'gocode/struct/src',
+            'sbe.target.language': 'golang',
+            'sbe.xinclude.aware': 'true',
+            'sbe.validation.xsd': validationXsdPath)
+    args = ['sbe-tool/src/test/resources/group-with-data-schema.xml',
+            'sbe-tool/src/test/resources/FixBinary.xml',
+            'sbe-tool/src/test/resources/issue435.xml',
+            'sbe-tool/src/test/resources/issue472.xml',
+            'sbe-tool/src/test/resources/issue483.xml',
+            'sbe-tool/src/test/resources/issue488.xml',
+            'sbe-tool/src/test/resources/issue560.xml',
+            'sbe-tool/src/test/resources/issue661.xml',
+            'sbe-tool/src/test/resources/issue847.xml',
+            'sbe-tool/src/test/resources/issue848.xml',
+            'sbe-tool/src/test/resources/issue849.xml',
+            'sbe-tool/src/test/resources/since-deprecated-test-schema.xml',
+            'sbe-tool/src/test/resources/example-bigendian-test-schema.xml',
+            'gocode/resources/example-composite.xml',
+            'gocode/resources/group-with-data-extension-schema.xml',
+            'gocode/resources/simple.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecTestComposite', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src',
+            'sbe.go.generate.generate.flyweights': 'true',
+            'sbe.target.language': 'golang')
+    args = ['sbe-tool/src/test/resources/composite-elements-schema-rc4.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecTestBasic', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src/basic',
+            'sbe.go.generate.generate.flyweights': 'true',
+            'sbe.target.language': 'golang')
+    args = ['sbe-tool/src/test/resources/basic-types-schema.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecTestGroup', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src/group',
+            'sbe.go.generate.generate.flyweights': 'true',
+            'sbe.target.language': 'golang')
+    args = ['sbe-tool/src/test/resources/basic-group-schema.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecTestVarData', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src/vardata',
+            'sbe.go.generate.generate.flyweights': 'true',
+            'sbe.target.language': 'golang')
+    args = ['sbe-tool/src/test/resources/basic-variable-length-schema.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecsWithXIncludes', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src',
+            'sbe.go.generate.generate.flyweights': 'true',
+            'sbe.target.language': 'golang',
+            'sbe.xinclude.aware': 'true',
+            'sbe.validation.xsd': validationXsdPath)
+    args = ['sbe-samples/src/main/resources/example-schema.xml',
+            'sbe-samples/src/main/resources/example-extension-schema.xml']
+}
+
+tasks.register('generateGolangFlyweightCodecsWithXSD', JavaExec) {
+    mainClass.set('uk.co.real_logic.sbe.SbeTool')
+    classpath = project(':sbe-tool').sourceSets.main.runtimeClasspath
+    systemProperties(
+            'sbe.output.dir': 'gocode/flyweight/src',
+            'sbe.go.generate.generate.flyweights': 'true',
             'sbe.target.language': 'golang',
             'sbe.xinclude.aware': 'true',
             'sbe.validation.xsd': validationXsdPath)
@@ -780,6 +860,13 @@ tasks.register('generateGolangCodecs') {
             'generateGolangCodecTestComposite',
             'generateGolangCodecsWithXIncludes',
             'generateGolangCodecsWithXSD',
+            'generateGolangFlyweightCodecTestVarData',
+            'generateGolangFlyweightCodecTestGroup',
+            'generateGolangFlyweightCodecTestBasic',
+            'generateGolangFlyweightCodecTestComposite',
+            'generateGolangFlyweightCodecsWithXIncludes',
+            'generateGolangFlyweightCodecsWithXSD',
+	    'generateIrCodecs',
             ':sbe-all:jar'
 }
 
@@ -868,6 +955,7 @@ tasks.register('generateGolangIrCodecs', JavaExec) {
     systemProperties(
             'sbe.output.dir': 'sbe-tool/src/main/golang',
             'sbe.target.language': 'golang',
+	    'sbe.go.generate.generate.flyweights': true,
             'sbe.validation.xsd': validationXsdPath)
     args = ['sbe-tool/src/main/resources/sbe-ir.xml']
 }

gocode/Makefile

@@ -19,69 +19,99 @@ all: example test # bench
 # This target is used to build golang files using parent gradle
 # script's invocation of sbe-tool in case it needs to be run again. We
 # use this one output file to check that dependency as it's simple
-DEP=src/simple/SbeMarshalling.go
+DEP=struct/src/simple/SbeMarshalling.go
 $(DEP): $(SBE_JAR)
 	(cd ..; ./gradlew generateGolangCodecs)
-	(export GOPATH=$(GOPATH) && \
-	cd src/simple && \
+	(export GOPATH=$(GOPATH)/struct && \
+	cd struct/src/simple && \
 	go build && \
 	$(SAVE_FORMAT) \
 	go fmt && \
 	go test)
 
 # Will regenerate codecs by removing dependencies
 clean:
-	rm -f src/*/SbeMarshalling.go src/*/*/SbeMarshalling.go
+	rm -f struct/src/*/SbeMarshalling.go struct/src/*/*/SbeMarshalling.go
 
-# Example code and benchmarking 
-example: src/example-schema/example-schema
-src/example-schema/example-schema: $(DEP)
-	(export GOPATH=$(GOPATH) && \
-	cd src/example-schema && \
+# Example code and benchmarking
+example: struct/src/example-schema/example-schema
+struct/src/example-schema/example-schema: $(DEP)
+	(export GOPATH=$(GOPATH)/struct && \
+	cd struct/src/example-schema && \
 	go build && \
 	go fmt && \
 	./example-schema)
 
 bench: $(DEP)
-	(export GOPATH=$(GOPATH) && \
-        cd src/example-schema && \
+	(export GOPATH=$(GOPATH)/struct && \
+        cd struct/src/example-schema && \
 	go test --bench . -cpuprofile=cpu.out && \
 	go tool pprof -text example-schema.test cpu.out)
 
-src/baseline/SbeMarshalling.go: $(DEP)
+struct/src/baseline/SbeMarshalling.go: $(DEP)
 	$(GOPATH)/sbe-tool -d src -s $(EXAMPLES_SCHEMA)
-	(export GOPATH=$(GOPATH) && \
-        cd src/baseline && \
+	(export GOPATH=$(GOPATH)/struct && \
+        cd struct/src/baseline && \
 	go build && \
 	go fmt && \
 	go test && \
 	go install)
 
 # The first set does a make install as there is a test that uses
 # multiple packages and needs them in GOPATH The second set work in
-# src/foo, and the third need a GOPATH addition as for Java and C++
+# struct/src/foo, and the third need a GOPATH addition as for Java and C++
 # they generate into the same directory but golang doesn't allow that
 test: $(DEP)
-	(export GOPATH=$(GOPATH) && \
+	(export GOPATH=$(GOPATH)/struct && \
 		(for t in baseline extension; do \
-		export GOPATH=$(GOPATH) && \
-	        cd $(GOPATH)/src/$$t && \
+		export GOPATH=$(GOPATH)/struct && \
+	        cd $(GOPATH)/struct/src/$$t && \
 		go build && \
 		go fmt && \
 		go test && \
 		go install \
 		;done))
-	(export GOPATH=$(GOPATH) && \
+	(export GOPATH=$(GOPATH)/struct && \
 		(for t in baseline-bigendian mktdata group_with_data group_with_data_extension composite_elements composite since-deprecated simple issue435 issue472 issue483 issue488 issue560 issue847 issue848 issue849 issue505 test973; do \
-		cd $(GOPATH)/src/$$t && \
+		cd $(GOPATH)/struct/src/$$t && \
 		go build && \
 		go fmt && \
 		go test \
 		;done))
 	(for t in vardata group basic; do \
-		export GOPATH=$(GOPATH)/$$t && \
-		cd $(GOPATH)/src/$$t/'SBE tests' && \
+		export GOPATH=$(GOPATH)/struct/$$t && \
+		cd $(GOPATH)/struct/src/$$t/'SBE tests' && \
 		go build && \
 		go fmt && \
 		go test \
 		;done)
+	(export GOPATH=$(GOPATH)/flyweight && \
+		(for t in baseline extension; do \
+		export GOPATH=$(GOPATH)/flyweight && \
+	        cd $(GOPATH)/flyweight/src/$$t && \
+		go build && \
+		go fmt && \
+		go test && \
+		go install \
+		;done))
+	(export GOPATH=$(GOPATH)/flyweight && \
+		(for t in baseline-bigendian mktdata group_with_data group_with_data_extension composite_elements composite since-deprecated simple issue435 issue472 issue483 issue488 issue560 issue847 issue848 issue849; do \
+		cd $(GOPATH)/flyweight/src/$$t && \
+		go build && \
+		go fmt && \
+		go test \
+		;done))
+	(for t in vardata group basic; do \
+		export GOPATH=$(GOPATH)/flyweight/$$t && \
+		cd $(GOPATH)/flyweight/src/$$t/'SBE tests' && \
+		go build && \
+		go fmt && \
+		go test \
+		;done)
+	(export SBE_JAR=$(SBE_JAR) && \
+	 export GO111MODULE=on && \
+		(cd $(GOPATH)/flyweight/src/otf && \
+			go generate && \
+			go build && \
+			go fmt && \
+			go test))

gocode/README.md

@@ -5,10 +5,13 @@ There is now a
 [user guide](https://github.com/real-logic/simple-binary-encoding/wiki/Golang-User-Guide)
 and this document is for development of the SBE golang generator.
 
+Go supports both generating Go structs with encode / decode methods, and flyweights like the other languages.
+Structs are generated by default for compatibility. Set `sbe.go.generate.generate.flyweights=true` to generate flyweights.
+
 Code Layout
 -----------
 The Java code that performs the generation of golang code is
-[here](https://github.com/real-logic/simple-binary-encoding/tree/master/sbe-tool/src/main/java/uk/co/real_logic/sbe/generation/golang).
+[here](https://github.com/real-logic/simple-binary-encoding/tree/master/sbe-tool/src/main/java/uk/co/real_logic/sbe/generation/golang). There is both a struct and a flyweight generator.
 
 Golang code used for testing resides in the top-level
 [gocode directory](https://github.com/real-logic/simple-binary-encoding/tree/master/gocode).
@@ -24,11 +27,12 @@ the build environment, example and test code are structured into a top
 level gocode directory hierarchy.
 
 Code is generated into this structure with pre-existing test code in place.
+There are tests for both the struct and flyweight styles of generated code.
 
 For the example, the code is generated into a library and the matching
 example code lives in it's own directory at the same level. For
 example, the example-schema generates the baseline library code into
-`gocode/src/baseline` and example code lives in `gocode/src/example-schema`.
+`gocode/src/struct/baseline` and example code lives in `gocode/src/example-schema`.
 
 To use this layout you should `set GOPATH=/path/to/gocode` or use the
 supplied Makefile which does this for you. For the tests you will need
@@ -60,5 +64,3 @@ generator processes.
 Roadmap
 =======
  * Windows developer support (currently tested on Linux/MacOS)
- * Further Unicode support
- * Testing/Bug fixes