[Geneva Exporter] HTTP/2 Multiplexing Bottleneck in Geneva Uploader

[lalitb]

The Geneva uploader uses a single reqwest::Client instance that establishes one HTTP/2 connection per host. While multiple uploads are processed concurrently (default: 4), they all multiplex over this single HTTP/2 connection. The Geneva server prevents establishing additional connections until n uploads complete on the existing connection, creating a bottleneck despite application-level concurrency.

The Issue:

• Geneva backend server has a policy that prevents establishing additional connections until n uploads complete on the existing connection.
• Upload requests are slow and take significant time to complete.
• Subsequent upload requests get queued behind existing multiplexed requests, creating a bottleneck.
• This results in poor throughput despite HTTP/2's multiplexing capabilities.

Current Implementation
In uploader.rs:

  let http_client = Client::builder()
      .timeout(Duration::from_secs(30))
      .default_headers(headers)
      .build()?;

This creates a single client that multiplexes all requests on one HTTP/2 connection, leading to head-of-line blocking at the application level.

Proposed Solution

Use multiple reqwest::Client instances in a pool to establish independent HTTP/2 connections. Each client maintains its own connection pool, allowing uploads to be distributed across multiple connections and bypassing the server's per-connection limitations.

Implementation Plan

1. Add pool size configuration to GenevaUploaderConfig
2. Replace single client with client pool in GenevaUploader
3. Implement round-robin client selection for load distribution
4. Maintain backward compatibility with existing API

Benefits

• Circumvents server connection policy - each client establishes independent connections
• Improved throughput - uploads proceed in parallel across multiple connections
• Configurable pool size - can be tuned based on server characteristics
• Simple implementation - minimal changes to existing codebase

Resource Overhead

• Each reqwest::Client maintains its own internal state (connection pool, DNS cache etc).
• Each client establishes separate TCP connections, which means more file descriptors used.
• Separate TLS handshake for each connection, and multiple TLS sessions maintained simultaneously
• Certificate validation performed per connection.

Alternative approaches

• HTTP/1.1 with connection pooling: Force HTTP/1.1 to avoid multiplexing entirely (Need to check if server supports it)
• Request queuing: Implement smarter request queuing instead of more connections
• Server side fix: Work with Geneva team to address the multiplexing limitation