v1.12.1

Add changelog entries for v1.12.1 release

v1.12.0

Substreams v1.17.0

New sf.substreams.rpc.v3.Stream/Blocks endpoint added

• This new endpoint removes the need for complex "mangling" of the package on the client side.
• Instead of expecting sf.substreams.v1.Modules (with the client having to apply parameters, network, etc.), the sf.substreams.rpc.v3.Request now expects:
  • a sf.substreams.v1.Package.
  • a map<string, string> of params
  • the network string
    which will all be applied to the package server-side.
• It returns the same object as the v2 endpoint, i.e. a stream of sf.substreams.rpc.v2.Response
• It is added on top of the existing 'v2' endpoint, both being active at the same time.
• To enable it, operators will simply need to ensure that their routing allows the /sf.substreams.rpc.v3.Stream/* path.
• Cached spkg on the server will now contain protobuf definitions, simplifying debugging of user requests.
• Emitted metrics for requests can now be sf.substreams.rpc.v3/Blocks instead of always sf.substreams.rpc.v2/Blocks, make sure that your metering endpoint can support it.

Note: recent substreams clients will support both endpoints, first trying the v3 and automatically falling back to v2 if they hit a "404 Not Found" or "Not Implemented" error.

Bug fixes

• Fixed a bug with BlockFilter: a skipped module would send BlockScopedData (in dev or near HEAD, to follow progress) with an empty module name, breaking some sinks. Module name was present if requesting a module dependent on that skipped module. Now the module name is always included.

v1.11.3

• Improved panic message when reader node encounter a block whose finality is bigger than the block itself to include lib_num, block_num, distance, and max_distance for easier debugging.

• Updated firehose-networks dependency to v0.2.2 (latest).

• Fixed common-one-block-store-url flag not expanding environment variables in all apps.

v1.11.2

Substreams v1.16.6

• Updated Wasmtime runtime from v30.0.0 to v36.0.0, bringing performance improvements, inlining support, Component Model async implementation, and enhanced security features.
• Added WASM bindgen shims support for Wasmtime runtime to handle WASM modules with WASM bindgen imports (when Substreams Module binary is defined as type wasm/rust-v1+wasm-bindgen-shims).
• Added support for foundational-store (in wasmtime and wazero).
• Added foundational-store grpc client to substreams engine.
• Fixed module caching to properly handle modules with different runtime extensions.

v1.11.1

Substreams

Metering

• 'paymentgateway' metering plugin renamed to tgm, now supports the indexer-api-key parameter.

Session (stream + workers management)

• Concurrent streams and workers limits are now handled under the new session plugin, available under common-session-plugin argument.

• The following flags were removed, now handled by that session plugin

  • substreams-tier1-global-worker-pool-address
  • substreams-tier1-global-request-pool-address
  • substreams-tier1-global-worker-pool-keep-alive-delay
  • substreams-tier1-global-request-pool-keep-alive-delay
  • substreams-tier1-default-max-request-per-use
  • substreams-tier1-default-minimal-request-life-time-second

• To use thegraph.market as a session plugin, use:
  --common-session-plugin=tgm://session.thegraph.market:443?indexer-api-key={your-api-key} (requires specific indexer API key)
  see https://github.com/streamingfast/tgm-gateway/tree/develop/session for details on the various flags

• To use simple local session management, use:
  --common-session-plugin=local://?max_sessions=30&max_sessions_per_user=3&max_workers_per_user=10&max_workers_per_session=10
  see https://github.com/streamingfast/dsession/tree/main/local for details on those flags

• Note: The 'max_sessions' parameter from the common-session-plugin is now also used to limit the number of firehose streams.

• If you were using a custom GRPC implementation for --substreams-tier1-global-worker-pool-address and --substreams-tier1-global-request-pool-address (ex: localhost:9010),
  simply use this value for the session plugin: --common-session-plugin=tgm://localhost:9010?plaintext=true, it is compatible.

Stability

• Fix a slow memory leak around metering plugin on tier2
• Add a maximum execution time for a full tier2 segment. By default, this is 60 minutes. It will fail with rpc error: code = DeadlineExceeded desc = request active for too long.
  It can be configured from the --substreams-tier2-segment-execution-timeout flag
• Fix subscription channel at max capacity error: when the LIVE channel is full (ex: slow module execution or slow client reader), the request will be continued from merged files instead of failing, and gracefully recover if performance is restored.
• Improve log message for 'request active for a long time', adding stats.

v1.11.0

CLI

• Improved how firecore tools --output=protojson and firecore tools --output=json renders pbbstream.Block type now printing the underlying chain's specific block.

Substreams (v1.16.4)

Tier1 thread / memory leak

• Fix thread leak on filereader.

• If --advertise-chain-name is sey, substreams-tier1 app will now infer default --substreams-tier1-block-type value by using chain's name and extracting chain's block type Protobuf package id, which will fix some cases where substreams-tier1 waits for 100 blocks before starting up.

Authentication changes

People using their own authentication layer will need to consider these changes before upgrading!

• Renamed config headers that come from authentication layer:
  • x-sf-user-id renamed to x-user-id (from dauth module)
  • x-sf-api-key-id renamed to x-api-key-id (from dauth module)
  • x-sf-meta renamed to x-meta (from dauth module)
  • x-sf-substreams-parallel-jobs renamed to x-substreams-parallel-workers
• Allow decreasing x-substreams-parallel-workers through an HTTP headers (auth layer determines higher bound)
• Detect value for the 'stage layer parallel executor max count' based on the x-plan-tier header (removed x-sf-substreams-stage-layer-parallel-executor-max-count handling)

New authentication plugin

• Added tgm://auth.thegraph.market?indexer-api-key=<API_KEY>&reissue-jwt-max-age-secs=600 plugin that allows an indexer to use The Graph Market as the authentication source.
  An API key with special "indexer" feature is needed to allow repeated calls to the API without rate limiting (for Key-based authentication and reissuance of "untrusted long-lived JWTs").

v1.10.2

Substreams (v1.6.2)

• Added mechanism to immediately cancel pending requests that are doing an 'external call' (ex: eth_call) on a given block when it gets forked out (UNDO because of a reorg).
• Fixed handling of invalid module kind: prevent heavy logging from recovered panic
• Error considered deterministic which will cache the error forever are now suffixed with <original message> (deterministic error).

v1.10.1

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.10.0

DO NOT USE in production

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.9.12

• This release is a hotfix for a thread leak leading to a slow memory leak.