HTTP Health check spam in telemetry when chaining health checks

[MermaidIsla]

Is there an existing issue for this?

• I have searched the existing issues

Describe the bug

This bug is related to #8580. Creating a custom health check class that implements IHealthCheck interface with an HTTP call to another service causes health check spam in the Dashboard's Traces tab. Even after applying a fix from #8643 the issue still persists.

Expected Behavior

No health check spam.

Steps To Reproduce

GitHub repo: https://github.com/MermaidIsla/aspire-health-check-bug/

Steps to reproduce this bug

1. Create a new project using the template .NET Aspire Starter App
2. In the Web project create a new class called ApiServiceHealthCheck
3. Copy the following code into ApiServiceHealthCheck.cs file:

public class ApiServiceHealthCheck(HttpClient httpClient) : IHealthCheck
{
    public async Task<HealthCheckResult> CheckHealthAsync(HealthCheckContext context, CancellationToken cancellationToken = default)
    {
        var response = await httpClient.GetAsync("http://apiservice/health", cancellationToken).ConfigureAwait(false);

        return response.IsSuccessStatusCode ? HealthCheckResult.Healthy() : HealthCheckResult.Unhealthy();
    }
}

4. In the Web project open Program.cs file and add the following code builder.Services.AddHealthChecks().AddCheck<ApiServiceHealthCheck>("apiservice"); just before var app = builder.Build();
5. In the AppHost project ensure that both projects (apiservice and webfrontend) have .WithHttpHealthCheck("/health", 200, "http")
6. Run the AppHost project, open the dashboard and click on Traces tab
7. You should now see a health check spam

Exceptions (if any)

No response

.NET Version info

SADA .NET SDK:
Version: 9.0.203
Commit: dc7acfa194
Workload version: 9.0.200-manifests.12d79ccf
MSBuild version: 17.13.20+a4ef1e90f

Běhové prostředí:
OS Name: Windows
OS Version: 10.0.19045
OS Platform: Windows
RID: win-x64
Base Path: C:\Program Files\dotnet\sdk\9.0.203\

Nainstalované úlohy .NET:
[aspire]
Zdroj instalace: VS 17.13.35931.197
Verze manifestu: 8.2.2/8.0.100
Cesta manifestu: C:\Program Files\dotnet\sdk-manifests\8.0.100\microsoft.net.sdk.aspire\8.2.2\WorkloadManifest.json
Typ instalace: Msi

Nakonfigurováno pro použití loose manifests při instalaci nových manifestů

Host:
Version: 9.0.4
Architecture: x64
Commit: f57e6dc747

.NET SDKs installed:
2.1.700 [C:\Program Files\dotnet\sdk]
2.1.701 [C:\Program Files\dotnet\sdk]
9.0.203 [C:\Program Files\dotnet\sdk]

.NET runtimes installed:
Microsoft.AspNetCore.All 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.All 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 8.0.15 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 9.0.4 [C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.11 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 2.1.12 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 8.0.15 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.NETCore.App 9.0.4 [C:\Program Files\dotnet\shared\Microsoft.NETCore.App]
Microsoft.WindowsDesktop.App 8.0.15 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]
Microsoft.WindowsDesktop.App 9.0.4 [C:\Program Files\dotnet\shared\Microsoft.WindowsDesktop.App]

Other architectures found:
x86 [C:\Program Files (x86)\dotnet]
registered at [HKLM\SOFTWARE\dotnet\Setup\InstalledVersions\x86\InstallLocation]

Environment variables:
Not set

global.json file:
Not found

Learn more:
https://aka.ms/dotnet/info

Download .NET:
https://aka.ms/dotnet/download

Anything else?

.NET Aspire version: 9.2
Visual Studio Community 2022, version: 17.13.6

Related issues and pull requests: #8580, #8643

A possible way to fix this bug: I found out that by editing AddHttpClientInstrumentation in Extensions.cs file in ServiceDefaults project, I can filter out the health check. Exact steps for this can be found in a README file the repo.

Image of a health check spam:

[DamianEdwards]

OK so this is the client-side part causing the trace now, i.e. a health check registered in an app is registered and it calls out to an HTTP endpoint of another app/resource and the HttpClient traces the outgoing request, which ends up in the trace of the app issuing the request.

Rather than filtering based on outgoing path (which could suppress traces for non-health-check related calls), I think it'd be better if we could configure the HttpClient instance used by the health check to disable tracing. Not sure if that's possible but I'll take a look.

[DamianEdwards]

BTW this isn't a regression from the 9.2 template change IIUC. This has always been the case if you add a health check from one app to another via HTTP.