Skip to content

Fix LLM Observability Data Submission #9476

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 8 commits into from
Sep 11, 2025
Merged

Fix LLM Observability Data Submission #9476

merged 8 commits into from
Sep 11, 2025

Conversation

@sabrenner
Copy link
Contributor

@sabrenner sabrenner commented Sep 5, 2025 •
edited
Loading

What Does This Do

Fixes the LLM Observability writers to correctly submit spans to APM and LLM Observability. Previously, LLM Observability would instantiate a multi-writer with an intake and agent writer. However, it looks like a bad copy-pasta meant we would override the agent writer with an additional intake writer. This meant that we wouldn't submit the APM span, and sometimes, we would strip the span kind tag, and log a "missing span kind" when processing the span again.

Shared tests, plus manual testing, confirm this resolution

$ TEST_LIBS=java pytest test_sdk.py::TestTracing::test_submits_llmobs_span_events_and_apm_spans
========================================================================================================================================================================= test session starts ==========================================================================================================================================================================
platform darwin -- Python 3.10.15, pytest-8.3.3, pluggy-1.6.0 -- **
cachedir: .pytest_cache
sdks: java
rootdir: **
configfile: .pytest.ini
collected 1 item                                                                                                                                                                                                                                                                                                                                                       

test_sdk.py::TestTracing::test_submits_llmobs_span_events_and_apm_spans[java-test-ml-app-tcp] PASSED

Motivation

Allows LLM Observability to submit both APM and LLM Observability spans.

Additional Notes

There's still a lingering issue of not completely stripping _ml_obs tags from the APM span. We can address this in a follow-up PR, as it might involve storing the tags in a different struct than the span tags (which are processed by different workers, so it's possible for the APM agent writer to write them first before the LLMObsSpanMapper gets a chance to strip them.

Contributor Checklist

Jira ticket: MLOB-3866

@sabrenner sabrenner added type: bug Bug report and fix comp: mlobs ML Observability (LLMObs) labels Sep 5, 2025
@datadog-official
Copy link

datadog-official bot commented Sep 5, 2025 •
edited by datadog-datadog-prod-us1 bot
Loading

🎯 Code Coverage
• Patch Coverage: 100.00%
• Total Coverage: 62.89% (+2.76%)

View detailed report

This comment will be updated automatically if new data arrives.
🔗 Commit SHA: 85c7b8b | Docs | Was this helpful? Give us feedback!

@pr-commenter
Copy link

pr-commenter bot commented Sep 5, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sabrenner/fix-llmobs-writers
git_commit_date 1757602568 1757616972
git_commit_sha fa49f3f 85c7b8b
release_version 1.54.0-SNAPSHOT~fa49f3fb37 1.54.0-SNAPSHOT~85c7b8b778
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1757618743 1757618743
ci_job_id 1125410335 1125410335
ci_pipeline_id 76237503 76237503
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-5vgwuaih 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-5vgwuaih 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 49 metrics, 10 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.06 s) : 0, 1059875
Total [baseline] (10.73 s) : 0, 10729968
Agent [candidate] (1.061 s) : 0, 1061120
Total [candidate] (10.746 s) : 0, 10746419
section appsec
Agent [baseline] (1.237 s) : 0, 1236759
Total [baseline] (10.951 s) : 0, 10950987
Agent [candidate] (1.232 s) : 0, 1232437
Total [candidate] (11.027 s) : 0, 11026566
section iast
Agent [baseline] (1.194 s) : 0, 1193822
Total [baseline] (11.009 s) : 0, 11008592
Agent [candidate] (1.189 s) : 0, 1189257
Total [candidate] (11.006 s) : 0, 11006394
section profiling
Agent [baseline] (1.222 s) : 0, 1221720
Total [baseline] (11.007 s) : 0, 11006778
Agent [candidate] (1.208 s) : 0, 1207842
Total [candidate] (10.841 s) : 0, 10840557
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.06 s -
Agent appsec 1.237 s 176.884 ms (16.7%)
Agent iast 1.194 s 133.946 ms (12.6%)
Agent profiling 1.222 s 161.845 ms (15.3%)
Total tracing 10.73 s -
Total appsec 10.951 s 221.019 ms (2.1%)
Total iast 11.009 s 278.624 ms (2.6%)
Total profiling 11.007 s 276.81 ms (2.6%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent appsec 1.232 s 171.317 ms (16.1%)
Agent iast 1.189 s 128.137 ms (12.1%)
Agent profiling 1.208 s 146.722 ms (13.8%)
Total tracing 10.746 s -
Total appsec 11.027 s 280.147 ms (2.6%)
Total iast 11.006 s 259.975 ms (2.4%)
Total profiling 10.841 s 94.138 ms (0.9%) 
gantt
    title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.461 ms) : 0, 1461
crashtracking [candidate] (1.451 ms) : 0, 1451
BytebuddyAgent [baseline] (731.905 ms) : 0, 731905
BytebuddyAgent [candidate] (732.344 ms) : 0, 732344
GlobalTracer [baseline] (252.013 ms) : 0, 252013
GlobalTracer [candidate] (252.48 ms) : 0, 252480
AppSec [baseline] (30.438 ms) : 0, 30438
AppSec [candidate] (30.491 ms) : 0, 30491
Debugger [baseline] (6.381 ms) : 0, 6381
Debugger [candidate] (6.392 ms) : 0, 6392
Remote Config [baseline] (685.897 µs) : 0, 686
Remote Config [candidate] (679.344 µs) : 0, 679
Telemetry [baseline] (15.775 ms) : 0, 15775
Telemetry [candidate] (16.131 ms) : 0, 16131
section appsec
crashtracking [baseline] (1.455 ms) : 0, 1455
crashtracking [candidate] (1.484 ms) : 0, 1484
BytebuddyAgent [baseline] (757.538 ms) : 0, 757538
BytebuddyAgent [candidate] (754.093 ms) : 0, 754093
GlobalTracer [baseline] (246.038 ms) : 0, 246038
GlobalTracer [candidate] (244.854 ms) : 0, 244854
IAST [baseline] (23.817 ms) : 0, 23817
IAST [candidate] (23.7 ms) : 0, 23700
AppSec [baseline] (171.479 ms) : 0, 171479
AppSec [candidate] (171.93 ms) : 0, 171930
Debugger [baseline] (6.067 ms) : 0, 6067
Debugger [candidate] (6.033 ms) : 0, 6033
Remote Config [baseline] (640.524 µs) : 0, 641
Remote Config [candidate] (630.81 µs) : 0, 631
Telemetry [baseline] (8.435 ms) : 0, 8435
Telemetry [candidate] (8.48 ms) : 0, 8480
section iast
crashtracking [baseline] (1.465 ms) : 0, 1465
crashtracking [candidate] (1.449 ms) : 0, 1449
BytebuddyAgent [baseline] (855.03 ms) : 0, 855030
BytebuddyAgent [candidate] (851.971 ms) : 0, 851971
GlobalTracer [baseline] (246.27 ms) : 0, 246270
GlobalTracer [candidate] (246.137 ms) : 0, 246137
IAST [baseline] (29.699 ms) : 0, 29699
IAST [candidate] (29.491 ms) : 0, 29491
AppSec [baseline] (25.382 ms) : 0, 25382
AppSec [candidate] (24.461 ms) : 0, 24461
Debugger [baseline] (6.064 ms) : 0, 6064
Debugger [candidate] (6.003 ms) : 0, 6003
Remote Config [baseline] (605.486 µs) : 0, 605
Remote Config [candidate] (593.626 µs) : 0, 594
Telemetry [baseline] (8.157 ms) : 0, 8157
Telemetry [candidate] (8.058 ms) : 0, 8058
section profiling
crashtracking [baseline] (1.447 ms) : 0, 1447
crashtracking [candidate] (1.42 ms) : 0, 1420
BytebuddyAgent [baseline] (770.179 ms) : 0, 770179
BytebuddyAgent [candidate] (760.569 ms) : 0, 760569
GlobalTracer [baseline] (235.494 ms) : 0, 235494
GlobalTracer [candidate] (232.753 ms) : 0, 232753
AppSec [baseline] (30.931 ms) : 0, 30931
AppSec [candidate] (31.071 ms) : 0, 31071
Debugger [baseline] (8.333 ms) : 0, 8333
Debugger [candidate] (8.888 ms) : 0, 8888
Remote Config [baseline] (711.939 µs) : 0, 712
Remote Config [candidate] (707.024 µs) : 0, 707
Telemetry [baseline] (14.914 ms) : 0, 14914
Telemetry [candidate] (13.326 ms) : 0, 13326
ProfilingAgent [baseline] (108.352 ms) : 0, 108352
ProfilingAgent [candidate] (108.02 ms) : 0, 108020
Profiling [baseline] (108.978 ms) : 0, 108978
Profiling [candidate] (108.681 ms) : 0, 108681
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.061 s) : 0, 1061283
Total [baseline] (8.613 s) : 0, 8613467
Agent [candidate] (1.062 s) : 0, 1062472
Total [candidate] (8.626 s) : 0, 8625545
section iast
Agent [baseline] (1.189 s) : 0, 1188745
Total [baseline] (9.303 s) : 0, 9302554
Agent [candidate] (1.198 s) : 0, 1198021
Total [candidate] (9.339 s) : 0, 9338984
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.061 s -
Agent iast 1.189 s 127.462 ms (12.0%)
Total tracing 8.613 s -
Total iast 9.303 s 689.086 ms (8.0%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.062 s -
Agent iast 1.198 s 135.549 ms (12.8%)
Total tracing 8.626 s -
Total iast 9.339 s 713.439 ms (8.3%) 
gantt
    title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37

    dateFormat X
    axisFormat %s
section tracing
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.452 ms) : 0, 1452
BytebuddyAgent [baseline] (732.648 ms) : 0, 732648
BytebuddyAgent [candidate] (733.613 ms) : 0, 733613
GlobalTracer [baseline] (252.017 ms) : 0, 252017
GlobalTracer [candidate] (253.07 ms) : 0, 253070
AppSec [baseline] (30.551 ms) : 0, 30551
AppSec [candidate] (30.531 ms) : 0, 30531
Debugger [baseline] (6.386 ms) : 0, 6386
Debugger [candidate] (6.36 ms) : 0, 6360
Remote Config [baseline] (689.236 µs) : 0, 689
Remote Config [candidate] (697.63 µs) : 0, 698
Telemetry [baseline] (16.406 ms) : 0, 16406
Telemetry [candidate] (15.619 ms) : 0, 15619
section iast
crashtracking [baseline] (1.456 ms) : 0, 1456
crashtracking [candidate] (1.47 ms) : 0, 1470
BytebuddyAgent [baseline] (851.65 ms) : 0, 851650
BytebuddyAgent [candidate] (858.509 ms) : 0, 858509
GlobalTracer [baseline] (247.498 ms) : 0, 247498
GlobalTracer [candidate] (247.325 ms) : 0, 247325
IAST [baseline] (27.973 ms) : 0, 27973
IAST [candidate] (29.836 ms) : 0, 29836
AppSec [baseline] (24.5 ms) : 0, 24500
AppSec [candidate] (24.849 ms) : 0, 24849
Debugger [baseline] (5.965 ms) : 0, 5965
Debugger [candidate] (6.121 ms) : 0, 6121
Remote Config [baseline] (594.124 µs) : 0, 594
Remote Config [candidate] (658.7 µs) : 0, 659
Telemetry [baseline] (8.098 ms) : 0, 8098
Telemetry [candidate] (8.149 ms) : 0, 8149
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sabrenner/fix-llmobs-writers
git_commit_date 1757602568 1757616972
git_commit_sha fa49f3f 85c7b8b
release_version 1.54.0-SNAPSHOT~fa49f3fb37 1.54.0-SNAPSHOT~85c7b8b778
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1757618320 1757618320
ci_job_id 1125410336 1125410336
ci_pipeline_id 76237503 76237503
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-m86wmv6r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-m86wmv6r 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 3 performance improvements and 2 performance regressions! Performance is the same for 7 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:insecure-bank:no_agent:high_load worse
[+93.195µs; +211.606µs] or [+2.164%; +4.913%]		 unstable
[-139.354op/s; +67.729op/s] or [-13.084%; +6.359%]		 4.460ms 1029.281op/s 4.307ms 1065.094op/s
scenario:load:insecure-bank:profiling:high_load worse
[+526.761µs; +871.264µs] or [+6.055%; +10.015%]		 unstable
[-102.032op/s; +23.469op/s] or [-19.137%; +4.402%]		 9.399ms 493.875op/s 8.700ms 533.156op/s
scenario:load:insecure-bank:iast_GLOBAL:high_load better
[-845.928µs; -426.223µs] or [-7.708%; -3.884%]		 unstable
[-19.320op/s; +71.070op/s] or [-4.561%; +16.779%]		 10.339ms 449.438op/s 10.975ms 423.562op/s
scenario:load:petclinic:profiling:high_load better
[-4.327ms; -3.406ms] or [-8.668%; -6.822%]		 unstable
[-0.409op/s; +13.736op/s] or [-0.431%; +14.468%]		 46.054ms 101.600op/s 49.920ms 94.937op/s
scenario:load:petclinic:tracing:high_load better
[-2.454ms; -1.652ms] or [-5.414%; -3.645%]		 unstable
[-3.009op/s; +12.734op/s] or [-2.915%; +12.336%]		 43.269ms 108.088op/s 45.322ms 103.225op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.307 ms) : 4253, 4362
.   : milestone, 4307,
iast (9.656 ms) : 9490, 9822
.   : milestone, 9656,
iast_FULL (13.995 ms) : 13719, 14270
.   : milestone, 13995,
iast_GLOBAL (10.975 ms) : 10782, 11168
.   : milestone, 10975,
profiling (8.7 ms) : 8555, 8844
.   : milestone, 8700,
tracing (7.898 ms) : 7786, 8010
.   : milestone, 7898,
section candidate
no_agent (4.46 ms) : 4404, 4515
.   : milestone, 4460,
iast (9.432 ms) : 9266, 9597
.   : milestone, 9432,
iast_FULL (14.023 ms) : 13745, 14301
.   : milestone, 14023,
iast_GLOBAL (10.339 ms) : 10142, 10536
.   : milestone, 10339,
profiling (9.399 ms) : 9224, 9573
.   : milestone, 9399,
tracing (7.848 ms) : 7731, 7965
.   : milestone, 7848,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.307 ms [4.253 ms, 4.362 ms] -
iast 9.656 ms [9.49 ms, 9.822 ms] 5.349 ms (124.2%)
iast_FULL 13.995 ms [13.719 ms, 14.27 ms] 9.688 ms (224.9%)
iast_GLOBAL 10.975 ms [10.782 ms, 11.168 ms] 6.668 ms (154.8%)
profiling 8.7 ms [8.555 ms, 8.844 ms] 4.392 ms (102.0%)
tracing 7.898 ms [7.786 ms, 8.01 ms] 3.591 ms (83.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.46 ms [4.404 ms, 4.515 ms] -
iast 9.432 ms [9.266 ms, 9.597 ms] 4.972 ms (111.5%)
iast_FULL 14.023 ms [13.745 ms, 14.301 ms] 9.564 ms (214.5%)
iast_GLOBAL 10.339 ms [10.142 ms, 10.536 ms] 5.879 ms (131.8%)
profiling 9.399 ms [9.224 ms, 9.573 ms] 4.939 ms (110.8%)
tracing 7.848 ms [7.731 ms, 7.965 ms] 3.388 ms (76.0%)
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.271 ms) : 37967, 38575
.   : milestone, 38271,
appsec (48.8 ms) : 48371, 49228
.   : milestone, 48800,
code_origins (44.843 ms) : 44467, 45220
.   : milestone, 44843,
iast (45.068 ms) : 44675, 45461
.   : milestone, 45068,
profiling (49.92 ms) : 49479, 50361
.   : milestone, 49920,
tracing (45.322 ms) : 44928, 45716
.   : milestone, 45322,
section candidate
no_agent (37.222 ms) : 36917, 37527
.   : milestone, 37222,
appsec (48.002 ms) : 47576, 48428
.   : milestone, 48002,
code_origins (45.965 ms) : 45557, 46373
.   : milestone, 45965,
iast (45.551 ms) : 45151, 45951
.   : milestone, 45551,
profiling (46.054 ms) : 45639, 46469
.   : milestone, 46054,
tracing (43.269 ms) : 42919, 43619
.   : milestone, 43269,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 38.271 ms [37.967 ms, 38.575 ms] -
appsec 48.8 ms [48.371 ms, 49.228 ms] 10.528 ms (27.5%)
code_origins 44.843 ms [44.467 ms, 45.22 ms] 6.572 ms (17.2%)
iast 45.068 ms [44.675 ms, 45.461 ms] 6.797 ms (17.8%)
profiling 49.92 ms [49.479 ms, 50.361 ms] 11.649 ms (30.4%)
tracing 45.322 ms [44.928 ms, 45.716 ms] 7.051 ms (18.4%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 37.222 ms [36.917 ms, 37.527 ms] -
appsec 48.002 ms [47.576 ms, 48.428 ms] 10.78 ms (29.0%)
code_origins 45.965 ms [45.557 ms, 46.373 ms] 8.743 ms (23.5%)
iast 45.551 ms [45.151 ms, 45.951 ms] 8.329 ms (22.4%)
profiling 46.054 ms [45.639 ms, 46.469 ms] 8.831 ms (23.7%)
tracing 43.269 ms [42.919 ms, 43.619 ms] 6.047 ms (16.2%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sabrenner/fix-llmobs-writers
git_commit_date 1757602568 1757616972
git_commit_sha fa49f3f 85c7b8b
release_version 1.54.0-SNAPSHOT~fa49f3fb37 1.54.0-SNAPSHOT~85c7b8b778
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1757618955 1757618955
ci_job_id 1125410337 1125410337
ci_pipeline_id 76237503 76237503
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-zfyrx7zua-project-304-concurrent-0-rfr8dcyd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-zfyrx7zua-project-304-concurrent-0-rfr8dcyd 6.8.0-1031-aws #33~22.04.1-Ubuntu SMP Thu Jun 26 14:22:30 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.48 ms) : 1468, 1491
.   : milestone, 1480,
appsec (2.46 ms) : 2410, 2510
.   : milestone, 2460,
iast (2.198 ms) : 2135, 2261
.   : milestone, 2198,
iast_GLOBAL (2.246 ms) : 2183, 2309
.   : milestone, 2246,
profiling (2.062 ms) : 2011, 2114
.   : milestone, 2062,
tracing (2.029 ms) : 1980, 2078
.   : milestone, 2029,
section candidate
no_agent (1.481 ms) : 1469, 1492
.   : milestone, 1481,
appsec (2.466 ms) : 2415, 2516
.   : milestone, 2466,
iast (2.2 ms) : 2137, 2263
.   : milestone, 2200,
iast_GLOBAL (2.252 ms) : 2188, 2315
.   : milestone, 2252,
profiling (2.072 ms) : 2020, 2124
.   : milestone, 2072,
tracing (2.029 ms) : 1980, 2078
.   : milestone, 2029,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.48 ms [1.468 ms, 1.491 ms] -
appsec 2.46 ms [2.41 ms, 2.51 ms] 980.531 µs (66.3%)
iast 2.198 ms [2.135 ms, 2.261 ms] 718.211 µs (48.5%)
iast_GLOBAL 2.246 ms [2.183 ms, 2.309 ms] 766.338 µs (51.8%)
profiling 2.062 ms [2.011 ms, 2.114 ms] 582.601 µs (39.4%)
tracing 2.029 ms [1.98 ms, 2.078 ms] 549.483 µs (37.1%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.481 ms [1.469 ms, 1.492 ms] -
appsec 2.466 ms [2.415 ms, 2.516 ms] 984.844 µs (66.5%)
iast 2.2 ms [2.137 ms, 2.263 ms] 719.522 µs (48.6%)
iast_GLOBAL 2.252 ms [2.188 ms, 2.315 ms] 770.792 µs (52.1%)
profiling 2.072 ms [2.02 ms, 2.124 ms] 591.078 µs (39.9%)
tracing 2.029 ms [1.98 ms, 2.078 ms] 548.105 µs (37.0%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~85c7b8b778, baseline=1.54.0-SNAPSHOT~fa49f3fb37
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.478 s) : 15478000, 15478000
.   : milestone, 15478000,
appsec (14.831 s) : 14831000, 14831000
.   : milestone, 14831000,
iast (18.36 s) : 18360000, 18360000
.   : milestone, 18360000,
iast_GLOBAL (17.945 s) : 17945000, 17945000
.   : milestone, 17945000,
profiling (15.244 s) : 15244000, 15244000
.   : milestone, 15244000,
tracing (15.121 s) : 15121000, 15121000
.   : milestone, 15121000,
section candidate
no_agent (15.358 s) : 15358000, 15358000
.   : milestone, 15358000,
appsec (14.841 s) : 14841000, 14841000
.   : milestone, 14841000,
iast (18.164 s) : 18164000, 18164000
.   : milestone, 18164000,
iast_GLOBAL (18.035 s) : 18035000, 18035000
.   : milestone, 18035000,
profiling (15.584 s) : 15584000, 15584000
.   : milestone, 15584000,
tracing (15.193 s) : 15193000, 15193000
.   : milestone, 15193000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.478 s [15.478 s, 15.478 s] -
appsec 14.831 s [14.831 s, 14.831 s] -647.0 ms (-4.2%)
iast 18.36 s [18.36 s, 18.36 s] 2.882 s (18.6%)
iast_GLOBAL 17.945 s [17.945 s, 17.945 s] 2.467 s (15.9%)
profiling 15.244 s [15.244 s, 15.244 s] -234.0 ms (-1.5%)
tracing 15.121 s [15.121 s, 15.121 s] -357.0 ms (-2.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.358 s [15.358 s, 15.358 s] -
appsec 14.841 s [14.841 s, 14.841 s] -517.0 ms (-3.4%)
iast 18.164 s [18.164 s, 18.164 s] 2.806 s (18.3%)
iast_GLOBAL 18.035 s [18.035 s, 18.035 s] 2.677 s (17.4%)
profiling 15.584 s [15.584 s, 15.584 s] 226.0 ms (1.5%)
tracing 15.193 s [15.193 s, 15.193 s] -165.0 ms (-1.1%)

@sabrenner sabrenner marked this pull request as ready for review September 11, 2025 15:03
@sabrenner sabrenner requested a review from a team as a code owner September 11, 2025 15:03
@sabrenner sabrenner requested a review from dougqh September 11, 2025 15:03
gary-huang
gary-huang approved these changes Sep 11, 2025
View reviewed changes
Copy link
Contributor

@gary-huang gary-huang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🔥 awesome stuff! Thank you so much Sam!!

@sabrenner sabrenner enabled auto-merge (squash) September 11, 2025 18:55
@sabrenner sabrenner merged commit 12c02b6 into master Sep 11, 2025
503 of 504 checks passed
@sabrenner sabrenner deleted the sabrenner/fix-llmobs-writers branch September 11, 2025 19:53
@github-actions github-actions bot added this to the 1.54.0 milestone Sep 11, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@gary-huang gary-huang gary-huang approved these changes

@AlexeyKuznetsov-DD AlexeyKuznetsov-DD AlexeyKuznetsov-DD approved these changes

@dougqh dougqh Awaiting requested review from dougqh dougqh is a code owner automatically assigned from DataDog/apm-java

Assignees

No one assigned

Labels

comp: mlobs ML Observability (LLMObs) type: bug Bug report and fix

Projects

None yet

Milestone

1.54.0

Development

Successfully merging this pull request may close these issues.

4 participants

@sabrenner @gary-huang @AlexeyKuznetsov-DD