v1.11.2

Substreams v1.16.6

• Updated Wasmtime runtime from v30.0.0 to v36.0.0, bringing performance improvements, inlining support, Component Model async implementation, and enhanced security features.
• Added WASM bindgen shims support for Wasmtime runtime to handle WASM modules with WASM bindgen imports (when Substreams Module binary is defined as type wasm/rust-v1+wasm-bindgen-shims).
• Added support for foundational-store (in wasmtime and wazero).
• Added foundational-store grpc client to substreams engine.
• Fixed module caching to properly handle modules with different runtime extensions.

v1.11.1

Substreams

Metering

• 'paymentgateway' metering plugin renamed to tgm, now supports the indexer-api-key parameter.

Session (stream + workers management)

• Concurrent streams and workers limits are now handled under the new session plugin, available under common-session-plugin argument.

• The following flags were removed, now handled by that session plugin

  • substreams-tier1-global-worker-pool-address
  • substreams-tier1-global-request-pool-address
  • substreams-tier1-global-worker-pool-keep-alive-delay
  • substreams-tier1-global-request-pool-keep-alive-delay
  • substreams-tier1-default-max-request-per-use
  • substreams-tier1-default-minimal-request-life-time-second

• To use thegraph.market as a session plugin, use:
  --common-session-plugin=tgm://session.thegraph.market:443?indexer-api-key={your-api-key} (requires specific indexer API key)
  see https://github.com/streamingfast/tgm-gateway/tree/develop/session for details on the various flags

• To use simple local session management, use:
  --common-session-plugin=local://?max_sessions=30&max_sessions_per_user=3&max_workers_per_user=10&max_workers_per_session=10
  see https://github.com/streamingfast/dsession/tree/main/local for details on those flags

• Note: The 'max_sessions' parameter from the common-session-plugin is now also used to limit the number of firehose streams.

• If you were using a custom GRPC implementation for --substreams-tier1-global-worker-pool-address and --substreams-tier1-global-request-pool-address (ex: localhost:9010),
  simply use this value for the session plugin: --common-session-plugin=tgm://localhost:9010?plaintext=true, it is compatible.

Stability

• Fix a slow memory leak around metering plugin on tier2
• Add a maximum execution time for a full tier2 segment. By default, this is 60 minutes. It will fail with rpc error: code = DeadlineExceeded desc = request active for too long.
  It can be configured from the --substreams-tier2-segment-execution-timeout flag
• Fix subscription channel at max capacity error: when the LIVE channel is full (ex: slow module execution or slow client reader), the request will be continued from merged files instead of failing, and gracefully recover if performance is restored.
• Improve log message for 'request active for a long time', adding stats.

v1.11.0

CLI

• Improved how firecore tools --output=protojson and firecore tools --output=json renders pbbstream.Block type now printing the underlying chain's specific block.

Substreams (v1.16.4)

Tier1 thread / memory leak

• Fix thread leak on filereader.

• If --advertise-chain-name is sey, substreams-tier1 app will now infer default --substreams-tier1-block-type value by using chain's name and extracting chain's block type Protobuf package id, which will fix some cases where substreams-tier1 waits for 100 blocks before starting up.

Authentication changes

People using their own authentication layer will need to consider these changes before upgrading!

• Renamed config headers that come from authentication layer:
  • x-sf-user-id renamed to x-user-id (from dauth module)
  • x-sf-api-key-id renamed to x-api-key-id (from dauth module)
  • x-sf-meta renamed to x-meta (from dauth module)
  • x-sf-substreams-parallel-jobs renamed to x-substreams-parallel-workers
• Allow decreasing x-substreams-parallel-workers through an HTTP headers (auth layer determines higher bound)
• Detect value for the 'stage layer parallel executor max count' based on the x-plan-tier header (removed x-sf-substreams-stage-layer-parallel-executor-max-count handling)

New authentication plugin

• Added tgm://auth.thegraph.market?indexer-api-key=<API_KEY>&reissue-jwt-max-age-secs=600 plugin that allows an indexer to use The Graph Market as the authentication source.
  An API key with special "indexer" feature is needed to allow repeated calls to the API without rate limiting (for Key-based authentication and reissuance of "untrusted long-lived JWTs").

v1.10.2

Substreams (v1.6.2)

• Added mechanism to immediately cancel pending requests that are doing an 'external call' (ex: eth_call) on a given block when it gets forked out (UNDO because of a reorg).
• Fixed handling of invalid module kind: prevent heavy logging from recovered panic
• Error considered deterministic which will cache the error forever are now suffixed with <original message> (deterministic error).

v1.10.1

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.10.0

DO NOT USE in production

Substreams

• [OPERATORS] Tier2 servers must be upgraded BEFORE tier1 servers
• tier2 servers will now stream outputs for the 'first segment', to speed up time to first block
• Progress notifications will only be sent every 500ms for the first minute, then reduce rate up to every 5 seconds (can be overridden per request)
• Return 'processed blocks' counter to client at the end of the request
• Added dev_output_modules to protobuf request (if present, in dev mode, only send the output of the modules listed)
• Added progress_messages_interval_ms to protobuf request (if present, overrides the rate of progress messages to that many milliseconds)

v1.9.12

• This release is a hotfix for a thread leak leading to a slow memory leak.

v1.9.11

[!WARNING] This version contains a thread leak, leading to eventual higher memory usage, bump to v1.9.12

Substreams improvements v1.15.8

Rework the execout File read/write to improve memory efficiency:

• This reduces the RAM usage necessary to read and stream data to the user on tier1,
  as well as to read the existing execouts on tier2 jobs (in multi-stage scenario)

• The cached execouts need to be rewritten to take advantage of this, since their data is currently not ordered:
  the system will automatically load and rewrite existing execout when they are used.

• Code changes include:

  • new FileReader / FileWriter that "read as you go" or "write as you go"
  • No more 'KV' map attached to the File
  • Split the IndexWriter away from its dependencies on execoutMappers.
  • Clock distributor now also reads "as you go", using a small "one-block-cache"

• Removed SUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT env var (since this is not a RAM issue anymore)

• Add uncompressed_egress_bytes field to substreams request stats log message

Various

• (dstore) Add storageClass query parameter for s3:// urls on stores (@fschoell)
• Update the firehose-beacon proto to include the new Electra spec in the 'well-known' protobuf definitions (@fschoell)
• Use The Graph's Network Registry to recognize chains by genesis blocks and fill the 'advertise' server on substreams/firehose

v1.9.10

Substreams improvements v1.15.7

• Tier2 jobs now write mapper outputs "as they progress", preventing memory usage spikes when saving them to disk.
• Tier2 jobs now limit writing and loading mapper output files to a maximum size of 8GiB by default.
• Tier2 jobs now release existingExecOuts memory as blocks progress
• Speed up DeleteByPrefix operations on all tiers (5x perf improvement on some heavy substreams)
• AddedSUBSTREAMS_OUTPUT_SIZE_LIMIT_PER_SEGMENT (int) environment variable to control this new limit.
• Added SUBSTREAMS_STORE_SIZE_LIMIT (uint64) env var to allow overwriting the default 1GiB value
• Added SUBSTREAMS_PRINT_STACK (bool) env var to enable printing full stack traces when caught panic occurs
• Added SUBSTREAMS_DEBUG_API_ADDR (string) environment variable to expose a "debug API" HTTP interface that allows blocking connections, running GC, listing or canceling active requests.
• Prevent a deterministic failure on a module definition (mode, valueType, updatePolicy) from persisting when the issue is fixed in the substreams.yaml streamingfast/substreams#621
• Metering events on tier2 now bundled at the end of the job (prevents sending metering events for failing jobs)
• Added metering for: "processed_blocks" (block * number of stages where execution happened) and "egress_bytes"

v1.9.9

Substreams performance improvements v1.15.4

• (RAM+CPU) dedupe execution of modules with same hash but different name when computing dependency graph. (#619)
• (RAM) prevent memory usage burst on tier2 when writing mapper by streaming protobuf items to writer
• Tier1 requests will no longer error out with "service currently overloaded" because tier2 servers are ramping up

New 'firehose' reader

• Add reader-node-firehose which creates one-blocks by consuming blocks from an already existing Firehose endpoint. This can be used to set up an indexer stack without having to run an instrumented blockchain node, or getting redundancy from another firehose provider.

Other

• Bumped grpc-go lib to 1.72.0
• Now building amd64 and arm64 Docker images on push & release.