Enabling/disabling tracing cannot be done after redirect

[sjberman]

Describe the bug

When redirecting a request from one location to another, I cannot turn tracing on or off in the "child" location. It can only be turned on/off in the root location. I can set span attributes in the child location, which properly sets the attributes in the span.

It seems that the decision to sample or not is decided before a redirect happens. The problem here is that I could redirect to different child location blocks depending on the request, and I only want tracing for a specific child, not all children.

Note: this also affects otel_trace_context.

To reproduce

When sending requests to /:

The following config will not enable tracing:

      server {
        listen 80;

        location / {
          try_files /dev/null /child;
        }

        location /child {
          otel_trace on;
          otel_span_attr "testkey" "val";
          return 200 "ok";
        }
      }

The following config will enable tracing and set the span attribute:

      server {
        listen 80;

        location / {
          otel_trace on;
          try_files /dev/null /child;
        }

        location /child {
          otel_span_attr "testkey" "val";
          return 200 "ok";
        }
      }

The following config will not disable tracing:

      server {
        listen 80;

        location / {
          otel_trace on;
          try_files /dev/null /child;
        }

        location /child {
          otel_trace off;
          return 200 "ok";
        }
      }

May be somewhat related to #47?

Expected behavior

I want to be able to make a sampling decision on a trace after a redirect happens. In simpler terms, I want to be able to set otel_trace and otel_trace_context in a redirected location. The following config should work.

      server {
        listen 80;

        location / {
          try_files /dev/null /child;
        }

        location /child {
          otel_trace on;
          otel_trace_context propagate;
          return 200 "ok";
        }
      }

The question becomes, is this possible with nginx? Can the trace headers be sent through the parent location to the child location if otel_trace_context is only set on the child?

Your environment

• nginx 1.27.0
• NGINX Gateway Fabric 1.3.0

Workaround for NGF specifically

Attempted to dynamically set the tracing settings in the parent location by using an NJS module or map. One problem with this is that otel_trace_context does not accept variable input. The NGINX Gateway Fabric API currently allows users to set this field, so we wouldn't be able to support it going forward using this workaround (we would have to remove it from our API and hardcode a value).

The workaround has also proven a bit messy trying to use maps or modules to properly get the right tracing settings for a particular redirected location, based on the current architecture of the NGF control plane. It still may be possible to find a proper workaround that fits our code design, but it would be much easier if the module worked as desired, allowing NGF to use it directly in a redirected location instead of fumbling around with numerous maps and variables that could add a much larger risk for bugs.