Defer metrics aggregator classloading to save startup time #9590
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
amarziali
added
type: enhancement
amarziali
commented
Sep 23, 2025
| sb.append((char) c); | ||
| } | ||
| } catch (IOException e) { | ||
| } catch (Throwable ignored) { |
There was a problem hiding this comment.
I don't know why but I had some NPE time by time on my local build. I widen this to Throwable. While is not related to this PR, it's a good thing to do
There was a problem hiding this comment.
NPE likely means the dd-java-agent.version file did not end up in the jar being tested, so it wasn't found
This could point to an issue when doing incremental builds - but proactively widening the catch here makes sense
|
🎯 Code Coverage 🔗 Commit SHA: 7352724 | Docs | Was this helpful? Give us feedback! |
mcculls
approved these changes
Sep 23, 2025
There was a problem hiding this comment.
Thanks for tracking this down!
Excessive class-loading on the primordial thread during premain has caused issues in the past, especially on Java 8 . It's a careful balance of doing just enough to observe and capture key events in premain, while moving reporting/aggregation to another thread.
BenchmarksStartupParameters
See matching parameters
SummaryFound 5 performance improvements and 0 performance regressions! Performance is the same for 45 metrics, 9 unstable metrics.
Startup time reports for insecure-bankgantt
title insecure-bank - global startup overhead: candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.022 s) : 0, 1022222
Total [baseline] (8.646 s) : 0, 8645577
Agent [candidate] (1.015 s) : 0, 1014551
Total [candidate] (8.661 s) : 0, 8661391
section iast
Agent [baseline] (1.152 s) : 0, 1152471
Total [baseline] (9.293 s) : 0, 9292547
Agent [candidate] (1.151 s) : 0, 1151177
Total [candidate] (9.282 s) : 0, 9281840
gantt
title insecure-bank - break down per module: candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.46 ms) : 0, 1460
crashtracking [candidate] (1.467 ms) : 0, 1467
BytebuddyAgent [baseline] (689.208 ms) : 0, 689208
BytebuddyAgent [candidate] (693.938 ms) : 0, 693938
GlobalTracer [baseline] (258.033 ms) : 0, 258033
GlobalTracer [candidate] (250.091 ms) : 0, 250091
AppSec [baseline] (31.818 ms) : 0, 31818
AppSec [candidate] (31.573 ms) : 0, 31573
Debugger [baseline] (6.353 ms) : 0, 6353
Debugger [candidate] (6.432 ms) : 0, 6432
Remote Config [baseline] (701.936 µs) : 0, 702
Remote Config [candidate] (703.681 µs) : 0, 704
Telemetry [baseline] (13.592 ms) : 0, 13592
Telemetry [candidate] (9.076 ms) : 0, 9076
section iast
crashtracking [baseline] (1.454 ms) : 0, 1454
crashtracking [candidate] (1.466 ms) : 0, 1466
BytebuddyAgent [baseline] (808.061 ms) : 0, 808061
BytebuddyAgent [candidate] (814.237 ms) : 0, 814237
GlobalTracer [baseline] (247.787 ms) : 0, 247787
GlobalTracer [candidate] (238.767 ms) : 0, 238767
IAST [baseline] (29.334 ms) : 0, 29334
IAST [candidate] (26.483 ms) : 0, 26483
AppSec [baseline] (29.785 ms) : 0, 29785
AppSec [candidate] (33.825 ms) : 0, 33825
Debugger [baseline] (6.145 ms) : 0, 6145
Debugger [candidate] (6.116 ms) : 0, 6116
Remote Config [baseline] (602.827 µs) : 0, 603
Remote Config [candidate] (599.832 µs) : 0, 600
Telemetry [baseline] (8.351 ms) : 0, 8351
Telemetry [candidate] (8.338 ms) : 0, 8338
Startup time reports for petclinicgantt
title petclinic - global startup overhead: candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section tracing
Agent [baseline] (1.019 s) : 0, 1019453
Total [baseline] (10.659 s) : 0, 10658736
Agent [candidate] (1.009 s) : 0, 1009384
Total [candidate] (10.699 s) : 0, 10699338
section appsec
Agent [baseline] (1.199 s) : 0, 1198648
Total [baseline] (10.908 s) : 0, 10907765
Agent [candidate] (1.188 s) : 0, 1187716
Total [candidate] (11.034 s) : 0, 11033671
section iast
Agent [baseline] (1.152 s) : 0, 1152205
Total [baseline] (11.052 s) : 0, 11051960
Agent [candidate] (1.147 s) : 0, 1146790
Total [candidate] (10.895 s) : 0, 10894781
section profiling
Agent [baseline] (1.164 s) : 0, 1164052
Total [baseline] (11.042 s) : 0, 11041710
Agent [candidate] (1.151 s) : 0, 1150829
Total [candidate] (11.0 s) : 0, 11000208
gantt
title petclinic - break down per module: candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section tracing
crashtracking [baseline] (1.472 ms) : 0, 1472
crashtracking [candidate] (1.476 ms) : 0, 1476
BytebuddyAgent [baseline] (688.455 ms) : 0, 688455
BytebuddyAgent [candidate] (691.302 ms) : 0, 691302
GlobalTracer [baseline] (257.725 ms) : 0, 257725
GlobalTracer [candidate] (248.369 ms) : 0, 248369
AppSec [baseline] (31.63 ms) : 0, 31630
AppSec [candidate] (31.027 ms) : 0, 31027
Debugger [baseline] (6.361 ms) : 0, 6361
Debugger [candidate] (6.315 ms) : 0, 6315
Remote Config [baseline] (681.018 µs) : 0, 681
Remote Config [candidate] (683.405 µs) : 0, 683
Telemetry [baseline] (12.046 ms) : 0, 12046
Telemetry [candidate] (8.945 ms) : 0, 8945
section appsec
crashtracking [baseline] (1.468 ms) : 0, 1468
crashtracking [candidate] (1.464 ms) : 0, 1464
BytebuddyAgent [baseline] (712.267 ms) : 0, 712267
BytebuddyAgent [candidate] (713.1 ms) : 0, 713100
GlobalTracer [baseline] (250.157 ms) : 0, 250157
GlobalTracer [candidate] (240.254 ms) : 0, 240254
IAST [baseline] (25.154 ms) : 0, 25154
IAST [candidate] (24.721 ms) : 0, 24721
AppSec [baseline] (171.479 ms) : 0, 171479
AppSec [candidate] (171.106 ms) : 0, 171106
Debugger [baseline] (6.074 ms) : 0, 6074
Debugger [candidate] (5.972 ms) : 0, 5972
Remote Config [baseline] (630.459 µs) : 0, 630
Remote Config [candidate] (627.569 µs) : 0, 628
Telemetry [baseline] (10.134 ms) : 0, 10134
Telemetry [candidate] (9.206 ms) : 0, 9206
section iast
crashtracking [baseline] (1.467 ms) : 0, 1467
crashtracking [candidate] (1.447 ms) : 0, 1447
BytebuddyAgent [baseline] (807.631 ms) : 0, 807631
BytebuddyAgent [candidate] (810.46 ms) : 0, 810460
GlobalTracer [baseline] (247.967 ms) : 0, 247967
GlobalTracer [candidate] (238.615 ms) : 0, 238615
IAST [baseline] (30.987 ms) : 0, 30987
IAST [candidate] (26.246 ms) : 0, 26246
AppSec [baseline] (27.277 ms) : 0, 27277
AppSec [candidate] (33.748 ms) : 0, 33748
Debugger [baseline] (6.208 ms) : 0, 6208
Debugger [candidate] (6.084 ms) : 0, 6084
Remote Config [baseline] (597.502 µs) : 0, 598
Remote Config [candidate] (593.902 µs) : 0, 594
Telemetry [baseline] (9.012 ms) : 0, 9012
Telemetry [candidate] (8.308 ms) : 0, 8308
section profiling
crashtracking [baseline] (1.449 ms) : 0, 1449
crashtracking [candidate] (1.437 ms) : 0, 1437
BytebuddyAgent [baseline] (719.928 ms) : 0, 719928
BytebuddyAgent [candidate] (718.185 ms) : 0, 718185
GlobalTracer [baseline] (234.836 ms) : 0, 234836
GlobalTracer [candidate] (224.016 ms) : 0, 224016
AppSec [baseline] (31.068 ms) : 0, 31068
AppSec [candidate] (31.127 ms) : 0, 31127
Debugger [baseline] (6.476 ms) : 0, 6476
Debugger [candidate] (6.445 ms) : 0, 6445
Remote Config [baseline] (725.643 µs) : 0, 726
Remote Config [candidate] (693.611 µs) : 0, 694
Telemetry [baseline] (16.037 ms) : 0, 16037
Telemetry [candidate] (16.404 ms) : 0, 16404
ProfilingAgent [baseline] (102.481 ms) : 0, 102481
ProfilingAgent [candidate] (101.474 ms) : 0, 101474
Profiling [baseline] (103.063 ms) : 0, 103063
Profiling [candidate] (102.086 ms) : 0, 102086
LoadParameters
See matching parameters
SummaryFound 4 performance improvements and 3 performance regressions! Performance is the same for 5 metrics, 12 unstable metrics.
Request duration reports for insecure-bankgantt
title insecure-bank - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section baseline
no_agent (4.189 ms) : 4142, 4237
. : milestone, 4189,
iast (9.891 ms) : 9724, 10059
. : milestone, 9891,
iast_FULL (14.663 ms) : 14369, 14956
. : milestone, 14663,
iast_GLOBAL (10.585 ms) : 10399, 10772
. : milestone, 10585,
profiling (9.494 ms) : 9340, 9649
. : milestone, 9494,
tracing (7.905 ms) : 7784, 8026
. : milestone, 7905,
section candidate
no_agent (4.342 ms) : 4292, 4393
. : milestone, 4342,
iast (10.333 ms) : 10149, 10516
. : milestone, 10333,
iast_FULL (13.71 ms) : 13444, 13976
. : milestone, 13710,
iast_GLOBAL (10.814 ms) : 10620, 11009
. : milestone, 10814,
profiling (8.909 ms) : 8764, 9054
. : milestone, 8909,
tracing (7.809 ms) : 7689, 7928
. : milestone, 7809,
Request duration reports for petclinicgantt
title petclinic - request duration [CI 0.99] : candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section baseline
no_agent (37.961 ms) : 37658, 38264
. : milestone, 37961,
appsec (47.589 ms) : 47168, 48011
. : milestone, 47589,
code_origins (44.426 ms) : 44047, 44804
. : milestone, 44426,
iast (45.86 ms) : 45462, 46258
. : milestone, 45860,
profiling (48.433 ms) : 48010, 48856
. : milestone, 48433,
tracing (46.835 ms) : 46441, 47229
. : milestone, 46835,
section candidate
no_agent (36.684 ms) : 36385, 36982
. : milestone, 36684,
appsec (48.678 ms) : 48244, 49112
. : milestone, 48678,
code_origins (43.665 ms) : 43304, 44027
. : milestone, 43665,
iast (44.782 ms) : 44396, 45168
. : milestone, 44782,
profiling (50.071 ms) : 49588, 50553
. : milestone, 50071,
tracing (43.784 ms) : 43419, 44149
. : milestone, 43784,
DacapoParameters
See matching parameters
SummaryFound 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics. Execution time for biojavagantt
title biojava - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section baseline
no_agent (14.996 s) : 14996000, 14996000
. : milestone, 14996000,
appsec (14.856 s) : 14856000, 14856000
. : milestone, 14856000,
iast (18.949 s) : 18949000, 18949000
. : milestone, 18949000,
iast_GLOBAL (18.067 s) : 18067000, 18067000
. : milestone, 18067000,
profiling (15.346 s) : 15346000, 15346000
. : milestone, 15346000,
tracing (15.26 s) : 15260000, 15260000
. : milestone, 15260000,
section candidate
no_agent (15.137 s) : 15137000, 15137000
. : milestone, 15137000,
appsec (14.945 s) : 14945000, 14945000
. : milestone, 14945000,
iast (18.433 s) : 18433000, 18433000
. : milestone, 18433000,
iast_GLOBAL (18.271 s) : 18271000, 18271000
. : milestone, 18271000,
profiling (15.101 s) : 15101000, 15101000
. : milestone, 15101000,
tracing (15.263 s) : 15263000, 15263000
. : milestone, 15263000,
Execution time for tomcatgantt
title tomcat - execution time [CI 0.99] : candidate=1.54.0-SNAPSHOT~7352724498, baseline=1.54.0-SNAPSHOT~a519ee2013
dateFormat X
axisFormat %s
section baseline
no_agent (1.489 ms) : 1477, 1501
. : milestone, 1489,
appsec (3.733 ms) : 3515, 3952
. : milestone, 3733,
iast (2.21 ms) : 2148, 2272
. : milestone, 2210,
iast_GLOBAL (2.252 ms) : 2189, 2315
. : milestone, 2252,
profiling (2.063 ms) : 2013, 2114
. : milestone, 2063,
tracing (2.049 ms) : 2000, 2098
. : milestone, 2049,
section candidate
no_agent (1.491 ms) : 1479, 1502
. : milestone, 1491,
appsec (3.694 ms) : 3480, 3908
. : milestone, 3694,
iast (2.219 ms) : 2156, 2282
. : milestone, 2219,
iast_GLOBAL (2.253 ms) : 2190, 2316
. : milestone, 2253,
profiling (2.049 ms) : 1999, 2099
. : milestone, 2049,
tracing (2.032 ms) : 1983, 2080
. : milestone, 2032,
|
There was a problem hiding this comment.
👏 praise: Thanks for the tedious investigation!
While there is no magic to reduce this time
Cough Leyden cough 😇
mhlidd
pushed a commit
that referenced
this pull request
Sep 24, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What Does This Do
This PR firstly reverts #9548 since the feature discovery was not the origin of the increased startup time.
NB: while the agent discovery causes I/O
SharedCommunicationObject.featureDiscoveryis already called before to initialize the writer. Hence, at this point, the I/O has already been produced.As a future improvement, we can defer some I/O outside the tracer initialisation but this is another refactoring.
Coming back to this PR, the increased startup time was mainly due to the classloading. The metrics aggregator, in fact, pulls in lots of jctools and ddsketches classes that requires additional time to be loaded for the first time.
While there is no magic to reduce this time, this PR moves the initialisation of the metrics aggregator outside the tracer initialisation block so that the tracer startup time stays almost untouched.
Motivation
Additional Notes
Contributor Checklist
type:and (comp:orinst:) labels in addition to any usefull labelsclose,fixor any linking keywords when referencing an issue.Use
solvesinstead, and assign the PR milestone to the issueJira ticket: [PROJ-IDENT]