Skip to content

Commit efb33c6

Browse files
protolambdamds1
protolambda
and
mds1
authored
protocol: forge scripts deploy in Go (#61)
* protocol: design forge scripts deploy in Go * Update protocol/forge-scripts-deploy-in-go.md Co-authored-by: Matt Solomon <[email protected]> --------- Co-authored-by: Matt Solomon <[email protected]>
1 parent bad336b commit efb33c6

File tree

1 file changed

+222
-0
lines changed

1 file changed

+222
-0
lines changed

‎protocol/forge-scripts-deploy-in-go.md‎

Lines changed: 222 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,222 @@
1+
# Forge Scripts Deploy in Go
2+
3+
# Purpose
4+
5+
The purpose of this doc is to improve the integration between the `forge script` based chain deployment/genesis flow
6+
and the Go tooling and testing.
7+
8+
# Summary
9+
10+
By running the chain-deployment/genesis solidity scripts in an instrumented Go EVM we can reduce the roundtrip time,
11+
and greatly improve integration with inputs (configs, dependencies) and outputs (state, artifacts).
12+
13+
This unlocks runtime-customization of deployments in Go tests,
14+
to increase coverage and create new multi-L2 chain deployments for Interop tests.
15+
16+
# Problem Statement + Context
17+
18+
From [Design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52):
19+
20+
> The current L2 chain deployment approach originates from a time with Hardhat,
21+
> single L1 target, and a single monolithic set of features.
22+
>
23+
> Since then the system has migrated to Foundry and extended for more features,
24+
> but remains centered around a single monolithic deploy-config for all its features.
25+
>
26+
> **With Interop we need to configure a new multi-L2 deployment**:
27+
> the number of ways to compose L2s in tests grows past what a single legacy config template can support.
28+
>
29+
> Outside of interop, deployment also seems increasingly complex and opaque, while it does not have to be,
30+
> due to the same configuration and composability troubles.
31+
32+
See the design-doc for further in-depth context on the problem.
33+
34+
The integration between (1) Go testing/tooling and (2) Forge deployment/genesis needs to improve
35+
to reduce the complexity of deploying and testing.
36+
37+
Specifically, Go tools need reliable and organized inputs for deployment/genesis,
38+
and interface with the forge scripts artifacts or scripts themselves
39+
in a way that is configurable but not error-prone.
40+
41+
# Proposed Solution
42+
43+
In [deployment-chains design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52) and
44+
[OPStackManager design doc 60](https://github.com/ethereum-optimism/design-docs/pull/60) the importance
45+
of Modular configs and incremental deploy steps is outlined.
46+
47+
To utilize those modular configs and deploy steps, we need a way to either
48+
run and cache the individual Forge script functions (as outlined in 52),
49+
or integrate it closer such that no caching complexity is required.
50+
51+
This document proposes the latter: run the Forge scripts within Go,
52+
such that we do not need multiple steps of Forge script sub-process to build a state for testing / genesis tooling.
53+
54+
## Running Forge scripts in Go
55+
56+
Forge scripts are really just solidity smart-contracts, with the addition of Forge cheat-codes.
57+
Running a contract in Go is relatively trivial: e.g. Cannon tests run the Cannon contracts in an instrumented Go EVM.
58+
59+
To support the Forge cheatcodes, we need to emulate the cheatcode behavior, as the Geth EVM does not natively have support for it.
60+
Cheatcodes are essentially responses to `STATICCALL`s to a special system address, acting like a precompile,
61+
with functions that interact with the EVM environment.
62+
63+
We do not have to support all Forge cheatcodes; just the subset used by the OP-Stack scripts would be sufficient.
64+
65+
The most important cheat-codes are:
66+
- `vm.chainId`: change chain ID
67+
- `vm.load`, `vm.store`: storage getter/setter
68+
- `vm.etch`: write contract code
69+
- `vm.deal`: set balance
70+
- `vm.prank`: change sender
71+
- `vm.getNonce`/`vm.setNonce`: account nonce getter/setter
72+
- `vm.broadcast`,`vm.startBroadcast`, `vm.stopBroadcast`: capture calls as transaction candidates. (Can be no-op, if we do not want to go through Go for production deployments)
73+
- `vm.getCode`/`vm.getDeployedCode`: artifact inspection
74+
- `vm.env{Bool/Uint/Int/etc.}`: env var getters
75+
- `vm.keyExists{...}/parse{...}/writeJson/etc.`: encoding utils
76+
- `vm.addr`: priv key to addr
77+
- `vm.label`: name an address
78+
- `vm.dumpState`: export EVM state
79+
- `vm.loadAllocs`: import EVM state
80+
81+
Note that with many of these, Go instrumentation can really improve integration:
82+
- `dumpState`/`loadAllocs`: no need to encode/decode state or read/write to disk -> faster Go tests
83+
- `getCode`/`getDeployedCode`: attach directly to artifacts, and *track which artifacts were used for a deployment*
84+
- `broadcast`: we can extend the deploy-tool functionality with transactions. Out of scope for now, but can be very useful.
85+
86+
## Forge-artifacts as Go FS
87+
88+
A Go FS is a simple filesystem abstraction: it provides read-only access to some source of files.
89+
A local directory can be wrapped into such FS, but also tarballs, or even data embedded in the Go binary, can be represented as Go FS.
90+
By using this FS abstraction, we can make the access to artifacts very simple, and "mount" the relevant FS into it.
91+
92+
For Go tests, this would be the local FS.
93+
94+
For Go genesis tooling, this might be a bundled FS, or one from a versioned release tarball.
95+
96+
Using an FS helps simplify the way we interact with forge-artifacts.
97+
98+
## Semver the scripts
99+
100+
We have an existing `Semver` pattern for production contracts, but don't apply the same to deploy scripts, yet.
101+
If we introduce this, then the version of the scripts (part of the artifacts) can be inspected
102+
by the Go genesis / test tooling, and usage of the script can then be adapted.
103+
Or at the very least, a warning can be thrown when the Go tool does not support the script.
104+
105+
This improves on the current situation, since the Go tool cannot tell anything about the compatibility
106+
of the deployment output of the forge scripts with the genesis-generation it does.
107+
108+
## Fast input/output
109+
110+
In addition to the Go forge cheatcodes, we could also substitute known contracts in the Forge script setup.
111+
In particular, deploy configs are registered at fixed global addresses.
112+
113+
By mocking these contracts, we can couple config-reads directly to the actual config,
114+
rather than having to load a JSON into a long list of EVM MPT storage leafs,
115+
only then to read it construct it back into a memory JSON string many times over right after.
116+
117+
Instrumentation of the config inputs can also provide a clear trace of when and how configuration affects a deployment.
118+
119+
Similar to config inputs, we can capture outputs more efficiently:
120+
upon `vm.dumpState` we do not have to encode the state; we can simply copy it in-process.
121+
This is great for the execution speed of the Go tests: we do not have to use intermediate state JSON files on disk.
122+
123+
## Usage by op-chain-ops
124+
125+
The op-e2e tests and `op-node genesis` tool both rely on `op-chain-ops/genesis` to prepare a chain state,
126+
given some deploy-configuration.
127+
128+
The `op-chain-ops` package is then responsible of applying the configuration, to generate the correct state.
129+
130+
With this solution it means that it:
131+
- Opens the forge-artifacts FS
132+
- Takes any configuration, and sets up the necessary ENV vars for cheat-code usage.
133+
- Instantiates an instrumented EVM, with the initial script entry-point loaded into it.
134+
- Including cheat-codes hooked up to the forge-artifacts for ENV data
135+
- Includes cheat-codes hooked up to state loading / writing ability.
136+
- Includes any mocked config contracts, where we basically map the bytes4 calldata back to a config attribute.
137+
- Calls the entry-point script, with an ABI argument, to run the particular deploy function of interest.
138+
- E.g. `deploySuperchain`, or smaller deploy functions like `deployProxyAdmin`
139+
- Capture any output state
140+
- Capture any deployed contract addresses (calls to `Artifacts.s.sol` interface),
141+
or alternatively simplify the script to just use simple `vm.label` functionality.
142+
- Capture any `vm.label` for later debugging.
143+
144+
## Resource Usage
145+
146+
While this solution does not include caching like
147+
[deployment-chains design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52),
148+
it does improve the performance of genesis generation a lot by bringing the forge execution closer, into the Go process.
149+
In this new solution there is less cost in disk-IO, as there is no writing of cache files,
150+
and configuration and state data can all stay in-process and skip JSON encoding/decoding.
151+
152+
In the past we have generated L2 genesis state through manual artifact inspection and Go based state surgery,
153+
which was moved away from due to complexity of the surgery code, but was sufficiently fast for Go scripts.
154+
155+
This keeps the test setup simple (we don't duplicate any special deploy function work into Go)
156+
and fast (avoid lots of IO / encoding / sub-process overhead).
157+
158+
## Close integration, but no contract logic leaks
159+
160+
By loading the forge scripts, the deployment logic all stays native to the smart-contracts,
161+
to avoid manual surgery steps in Go.
162+
163+
When adding a deployment config variable, the only Go change needed is to add it to the Go config definition,
164+
and write any tests to exercise that deployment, as should be the default for every protocol feature.
165+
166+
The deployment implementation details stay encapsulated in the Forge scripting,
167+
which is unified with the Forge testing, and thus unifying the production and test code paths.
168+
169+
## Potential future extension: deployment integration
170+
171+
The `vm.broadcast` cheat-code is an opportunity for future deployment improvements:
172+
rather than running through Forge script when preparing the production deployment transactions,
173+
we could run through the Go genesis tool.
174+
175+
This would allow us to script more advanced post-processing of the transactions:
176+
- cross-validation against the superchain-registry
177+
- simulation of the transaction with custom tracing
178+
179+
And all bundled in Go, so the end-user does not need to ensure a specific Forge version,
180+
does not need to as many manual `forge script` invocations,
181+
and all environment settings (that might otherwise be set with forge script flags)
182+
can be controlled by defining the exact CLI interface.
183+
184+
This is out-of-scope for now, but may help unify the deploy process that production chains use,
185+
and the deploy process that devnets / op-e2e use.
186+
187+
# Alternatives Considered
188+
189+
See proposed solution [in design doc 52](https://github.com/ethereum-optimism/design-docs/pull/52),
190+
where `forge script` is used as is, and performance concerns are mitigated with caching.
191+
192+
An [experimental draft of the caching](https://github.com/ethereum-optimism/optimism/pull/11297) was implemented,
193+
but arguably the caching introduced too much complexity and fragility.
194+
195+
Other solutions / ideas are discussed in design-doc 52 as well, but were not viable.
196+
197+
# Risks & Uncertainties
198+
199+
## Geth Go EVM
200+
201+
The Go EVM instrumentation might be difficult, as the geth EVM is not as widely used in tooling.
202+
However, the tracing functionality is excellent, and Go is quite flexible.
203+
If we need to we can make very minor tweaks to `op-geth`,
204+
to expose any inaccessible EVM internals needed to implement the forge cheatcodes.
205+
206+
## Eng time
207+
208+
This is a mini-project: the scope of implementing the cheat-codes is not that large (a few days at most),
209+
but the integration into tooling and op-e2e may be more involved (can be a week, maybe two).
210+
211+
In the past the `L2Genesis.s.sol` and `allocs` work, that moved us away from the manual and error-prone op-chain-ops surgery,
212+
was completed successfully in the form of an interop side-project to remove tech-debt. This project scope looks quite similar.
213+
214+
This functionality does block Interop op-e2e testing: without it,
215+
we are not able to customize deployments sufficiently and cleanly (avoiding many more `allocs` special cases),
216+
to get multi-L2 deployments into the op-e2e.
217+
218+
## Devrel feedback
219+
220+
Historically devrel has not been included sufficiently in the deployment-flow design.
221+
Known pain-points like inconsistency between the Go and forge genesis generation,
222+
unclear allocs, and monolithic deploy-config are being addressed, but more feedback may still improve the design.

0 commit comments

Comments
 (0)