Skip to content

Benchmark code changes from 1.49.0 to 1.50.0 #8968

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 18 commits into
base: master
Choose a base branch
from
Draft

Benchmark code changes from 1.49.0 to 1.50.0 #8968

wants to merge 18 commits into from

Conversation

sarahchen6
Copy link
Contributor

What Does This Do

This is a branch with 1.49.0 code and 1.50.0 build/CI.

Motivation

Compare benchmarks between this branch and master to see if code changes from 1.49.0 to 1.50.0 introduce any serious regressions.

Additional Notes

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@sarahchen6 sarahchen6 added the tag: do not merge Do not merge changes label Jun 11, 2025
@pr-commenter
Copy link

pr-commenter bot commented Jun 11, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/1.49.0-branch
git_commit_date 1750259064 1750274142
git_commit_sha 712d5c3 d1c08f0
release_version 1.50.0-SNAPSHOT~712d5c320e 1.50.0-SNAPSHOT~d1c08f02be
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1750277247 1750277247
ci_job_id 988983429 988983429
ci_pipeline_id 68164219 68164219
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-clrnfvyp-project-304-concurrent-0-s5b0bmsw 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-clrnfvyp-project-304-concurrent-0-s5b0bmsw 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None

Summary

Found 3 performance improvements and 2 performance regressions! Performance is the same for 56 metrics, 10 unstable metrics.

scenario Δ mean execution_time candidate mean execution_time baseline mean execution_time
scenario:startup:insecure-bank:iast_TELEMETRY_OFF:AppSec worse
[+4.183ms; +9.412ms] or [+7.952%; +17.894%]		 59.395ms 52.598ms
scenario:startup:insecure-bank:tracing:Remote Config better
[-68.325µs; -18.507µs] or [-9.072%; -2.457%]		 709.745µs 753.161µs
scenario:startup:petclinic:profiling:GlobalTracer worse
[+15.033ms; +24.654ms] or [+4.140%; +6.789%]		 382.963ms 363.120ms
scenario:startup:petclinic:profiling:AppSec better
[-9.609ms; -7.674ms] or [-14.720%; -11.756%]		 56.635ms 65.276ms
scenario:startup:petclinic:tracing:Remote Config better
[-73.788µs; -17.528µs] or [-9.700%; -2.304%]		 715.058µs 760.716µs
Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.029 s) : 0, 1028997
Total [baseline] (10.651 s) : 0, 10651010
Agent [candidate] (1.015 s) : 0, 1015254
Total [candidate] (10.578 s) : 0, 10577847
section appsec
Agent [baseline] (1.189 s) : 0, 1188907
Total [baseline] (10.696 s) : 0, 10695723
Agent [candidate] (1.167 s) : 0, 1166686
Total [candidate] (10.751 s) : 0, 10751085
section iast
Agent [baseline] (1.165 s) : 0, 1164863
Total [baseline] (10.878 s) : 0, 10878156
Agent [candidate] (1.143 s) : 0, 1143353
Total [candidate] (10.781 s) : 0, 10780534
section profiling
Agent [baseline] (1.285 s) : 0, 1285458
Total [baseline] (10.979 s) : 0, 10979367
Agent [candidate] (1.278 s) : 0, 1278004
Total [candidate] (10.877 s) : 0, 10877169
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.029 s -
Agent appsec 1.189 s 159.91 ms (15.5%)
Agent iast 1.165 s 135.867 ms (13.2%)
Agent profiling 1.285 s 256.461 ms (24.9%)
Total tracing 10.651 s -
Total appsec 10.696 s 44.713 ms (0.4%)
Total iast 10.878 s 227.146 ms (2.1%)
Total profiling 10.979 s 328.357 ms (3.1%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.015 s -
Agent appsec 1.167 s 151.432 ms (14.9%)
Agent iast 1.143 s 128.099 ms (12.6%)
Agent profiling 1.278 s 262.75 ms (25.9%)
Total tracing 10.578 s -
Total appsec 10.751 s 173.238 ms (1.6%)
Total iast 10.781 s 202.687 ms (1.9%)
Total profiling 10.877 s 299.322 ms (2.8%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.484 ms) : 0, 684484
BytebuddyAgent [candidate] (672.58 ms) : 0, 672580
GlobalTracer [baseline] (242.631 ms) : 0, 242631
GlobalTracer [candidate] (240.942 ms) : 0, 240942
AppSec [baseline] (58.954 ms) : 0, 58954
AppSec [candidate] (57.798 ms) : 0, 57798
Debugger [baseline] (6.278 ms) : 0, 6278
Debugger [candidate] (6.242 ms) : 0, 6242
Remote Config [baseline] (760.716 µs) : 0, 761
Remote Config [candidate] (715.058 µs) : 0, 715
Telemetry [baseline] (12.299 ms) : 0, 12299
Telemetry [candidate] (13.547 ms) : 0, 13547
section appsec
BytebuddyAgent [baseline] (712.545 ms) : 0, 712545
BytebuddyAgent [candidate] (694.464 ms) : 0, 694464
GlobalTracer [baseline] (236.861 ms) : 0, 236861
GlobalTracer [candidate] (234.015 ms) : 0, 234015
IAST [baseline] (21.96 ms) : 0, 21960
IAST [candidate] (21.811 ms) : 0, 21811
AppSec [baseline] (179.916 ms) : 0, 179916
AppSec [candidate] (179.207 ms) : 0, 179207
Debugger [baseline] (5.919 ms) : 0, 5919
Debugger [candidate] (5.832 ms) : 0, 5832
Remote Config [baseline] (616.202 µs) : 0, 616
Remote Config [candidate] (612.406 µs) : 0, 612
Telemetry [baseline] (7.372 ms) : 0, 7372
Telemetry [candidate] (7.391 ms) : 0, 7391
section iast
BytebuddyAgent [baseline] (811.048 ms) : 0, 811048
BytebuddyAgent [candidate] (790.821 ms) : 0, 790821
GlobalTracer [baseline] (233.095 ms) : 0, 233095
GlobalTracer [candidate] (231.728 ms) : 0, 231728
IAST [baseline] (27.146 ms) : 0, 27146
IAST [candidate] (22.994 ms) : 0, 22994
AppSec [baseline] (55.217 ms) : 0, 55217
AppSec [candidate] (59.673 ms) : 0, 59673
Debugger [baseline] (6.065 ms) : 0, 6065
Debugger [candidate] (6.068 ms) : 0, 6068
Remote Config [baseline] (610.222 µs) : 0, 610
Remote Config [candidate] (608.828 µs) : 0, 609
Telemetry [baseline] (7.973 ms) : 0, 7973
Telemetry [candidate] (8.01 ms) : 0, 8010
section profiling
BytebuddyAgent [baseline] (681.922 ms) : 0, 681922
BytebuddyAgent [candidate] (669.237 ms) : 0, 669237
GlobalTracer [baseline] (363.12 ms) : 0, 363120
GlobalTracer [candidate] (382.963 ms) : 0, 382963
AppSec [baseline] (65.276 ms) : 0, 65276
AppSec [candidate] (56.635 ms) : 0, 56635
Debugger [baseline] (6.261 ms) : 0, 6261
Debugger [candidate] (6.21 ms) : 0, 6210
Remote Config [baseline] (671.756 µs) : 0, 672
Remote Config [candidate] (657.122 µs) : 0, 657
Telemetry [baseline] (8.334 ms) : 0, 8334
Telemetry [candidate] (8.184 ms) : 0, 8184
ProfilingAgent [baseline] (108.653 ms) : 0, 108653
ProfilingAgent [candidate] (103.554 ms) : 0, 103554
Profiling [baseline] (108.679 ms) : 0, 108679
Profiling [candidate] (103.58 ms) : 0, 103580
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.038 s) : 0, 1037677
Total [baseline] (8.57 s) : 0, 8570036
Agent [candidate] (1.011 s) : 0, 1010663
Total [candidate] (8.548 s) : 0, 8547576
section iast
Agent [baseline] (1.156 s) : 0, 1156041
Total [baseline] (9.215 s) : 0, 9214706
Agent [candidate] (1.152 s) : 0, 1152038
Total [candidate] (9.266 s) : 0, 9266399
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.155 s) : 0, 1154719
Total [baseline] (9.172 s) : 0, 9171820
Agent [candidate] (1.148 s) : 0, 1147510
Total [candidate] (9.213 s) : 0, 9212678
section iast_TELEMETRY_OFF
Agent [baseline] (1.161 s) : 0, 1160802
Total [baseline] (9.236 s) : 0, 9235622
Agent [candidate] (1.142 s) : 0, 1141686
Total [candidate] (9.228 s) : 0, 9227587
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.038 s -
Agent iast 1.156 s 118.364 ms (11.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.155 s 117.042 ms (11.3%)
Agent iast_TELEMETRY_OFF 1.161 s 123.124 ms (11.9%)
Total tracing 8.57 s -
Total iast 9.215 s 644.67 ms (7.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.172 s 601.784 ms (7.0%)
Total iast_TELEMETRY_OFF 9.236 s 665.586 ms (7.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.011 s -
Agent iast 1.152 s 141.375 ms (14.0%)
Agent iast_HARDCODED_SECRET_DISABLED 1.148 s 136.847 ms (13.5%)
Agent iast_TELEMETRY_OFF 1.142 s 131.023 ms (13.0%)
Total tracing 8.548 s -
Total iast 9.266 s 718.822 ms (8.4%)
Total iast_HARDCODED_SECRET_DISABLED 9.213 s 665.102 ms (7.8%)
Total iast_TELEMETRY_OFF 9.228 s 680.011 ms (8.0%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (690.71 ms) : 0, 690710
BytebuddyAgent [candidate] (671.164 ms) : 0, 671164
GlobalTracer [baseline] (243.185 ms) : 0, 243185
GlobalTracer [candidate] (240.424 ms) : 0, 240424
AppSec [baseline] (60.286 ms) : 0, 60286
AppSec [candidate] (57.538 ms) : 0, 57538
Debugger [baseline] (7.779 ms) : 0, 7779
Debugger [candidate] (6.167 ms) : 0, 6167
Remote Config [baseline] (753.161 µs) : 0, 753
Remote Config [candidate] (709.745 µs) : 0, 710
Telemetry [baseline] (11.263 ms) : 0, 11263
Telemetry [candidate] (11.223 ms) : 0, 11223
section iast
BytebuddyAgent [baseline] (804.508 ms) : 0, 804508
BytebuddyAgent [candidate] (798.536 ms) : 0, 798536
GlobalTracer [baseline] (231.651 ms) : 0, 231651
GlobalTracer [candidate] (232.429 ms) : 0, 232429
IAST [baseline] (25.319 ms) : 0, 25319
IAST [candidate] (23.085 ms) : 0, 23085
AppSec [baseline] (56.498 ms) : 0, 56498
AppSec [candidate] (59.795 ms) : 0, 59795
Debugger [baseline] (6.031 ms) : 0, 6031
Debugger [candidate] (6.014 ms) : 0, 6014
Remote Config [baseline] (596.455 µs) : 0, 596
Remote Config [candidate] (606.608 µs) : 0, 607
Telemetry [baseline] (7.903 ms) : 0, 7903
Telemetry [candidate] (7.986 ms) : 0, 7986
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (804.316 ms) : 0, 804316
BytebuddyAgent [candidate] (793.845 ms) : 0, 793845
GlobalTracer [baseline] (231.386 ms) : 0, 231386
GlobalTracer [candidate] (232.598 ms) : 0, 232598
IAST [baseline] (26.033 ms) : 0, 26033
IAST [candidate] (23.201 ms) : 0, 23201
AppSec [baseline] (55.135 ms) : 0, 55135
AppSec [candidate] (59.8 ms) : 0, 59800
Debugger [baseline] (5.924 ms) : 0, 5924
Debugger [candidate] (6.078 ms) : 0, 6078
Remote Config [baseline] (581.986 µs) : 0, 582
Remote Config [candidate] (591.634 µs) : 0, 592
Telemetry [baseline] (7.861 ms) : 0, 7861
Telemetry [candidate] (7.855 ms) : 0, 7855
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (808.05 ms) : 0, 808050
BytebuddyAgent [candidate] (790.144 ms) : 0, 790144
GlobalTracer [baseline] (232.921 ms) : 0, 232921
GlobalTracer [candidate] (231.696 ms) : 0, 231696
IAST [baseline] (29.136 ms) : 0, 29136
IAST [candidate] (22.593 ms) : 0, 22593
AppSec [baseline] (52.598 ms) : 0, 52598
AppSec [candidate] (59.395 ms) : 0, 59395
Debugger [baseline] (6.041 ms) : 0, 6041
Debugger [candidate] (6.027 ms) : 0, 6027
Remote Config [baseline] (608.361 µs) : 0, 608
Remote Config [candidate] (621.749 µs) : 0, 622
Telemetry [baseline] (7.867 ms) : 0, 7867
Telemetry [candidate] (7.82 ms) : 0, 7820
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/1.49.0-branch
git_commit_date 1750259064 1750274142
git_commit_sha 712d5c3 d1c08f0
release_version 1.50.0-SNAPSHOT~712d5c320e 1.50.0-SNAPSHOT~d1c08f02be
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1750276536 1750276536
ci_job_id 988983430 988983430
ci_pipeline_id 68164219 68164219
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-bziamzy-project-304-concurrent-0-vrgzmbkr 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-bziamzy-project-304-concurrent-0-vrgzmbkr 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 1 performance improvements and 1 performance regressions! Performance is the same for 10 metrics, 12 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:appsec:high_load better
[-3.806ms; -2.879ms] or [-7.698%; -5.823%]		 unstable
[+0.047op/s; +13.653op/s] or [+0.049%; +14.423%]		 46.097ms 101.513op/s 49.440ms 94.662op/s
scenario:load:petclinic:iast:high_load worse
[+1.807ms; +2.631ms] or [+4.221%; +6.148%]		 unstable
[-12.631op/s; +1.881op/s] or [-11.558%; +1.722%]		 45.023ms 103.912op/s 42.804ms 109.287op/s
Request duration reports for petclinic 
gantt
    title petclinic - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
    dateFormat X
    axisFormat %s
section baseline
no_agent (38.696 ms) : 38392, 39001
.   : milestone, 38696,
appsec (49.44 ms) : 48986, 49893
.   : milestone, 49440,
code_origins (45.687 ms) : 45297, 46077
.   : milestone, 45687,
iast (42.804 ms) : 42436, 43173
.   : milestone, 42804,
profiling (49.532 ms) : 49091, 49973
.   : milestone, 49532,
tracing (44.836 ms) : 44478, 45194
.   : milestone, 44836,
section candidate
no_agent (38.609 ms) : 38302, 38917
.   : milestone, 38609,
appsec (46.097 ms) : 45691, 46503
.   : milestone, 46097,
code_origins (46.513 ms) : 46130, 46897
.   : milestone, 46513,
iast (45.023 ms) : 44626, 45421
.   : milestone, 45023,
profiling (49.102 ms) : 48674, 49530
.   : milestone, 49102,
tracing (44.005 ms) : 43647, 44362
.   : milestone, 44005,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 38.696 ms [38.392 ms, 39.001 ms] -
appsec 49.44 ms [48.986 ms, 49.893 ms] 10.743 ms (27.8%)
code_origins 45.687 ms [45.297 ms, 46.077 ms] 6.991 ms (18.1%)
iast 42.804 ms [42.436 ms, 43.173 ms] 4.108 ms (10.6%)
profiling 49.532 ms [49.091 ms, 49.973 ms] 10.836 ms (28.0%)
tracing 44.836 ms [44.478 ms, 45.194 ms] 6.14 ms (15.9%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 38.609 ms [38.302 ms, 38.917 ms] -
appsec 46.097 ms [45.691 ms, 46.503 ms] 7.488 ms (19.4%)
code_origins 46.513 ms [46.13 ms, 46.897 ms] 7.904 ms (20.5%)
iast 45.023 ms [44.626 ms, 45.421 ms] 6.414 ms (16.6%)
profiling 49.102 ms [48.674 ms, 49.53 ms] 10.493 ms (27.2%)
tracing 44.005 ms [43.647 ms, 44.362 ms] 5.395 ms (14.0%)
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
    dateFormat X
    axisFormat %s
section baseline
no_agent (4.35 ms) : 4298, 4402
.   : milestone, 4350,
iast (9.361 ms) : 9201, 9522
.   : milestone, 9361,
iast_FULL (14.046 ms) : 13767, 14326
.   : milestone, 14046,
iast_GLOBAL (9.992 ms) : 9818, 10166
.   : milestone, 9992,
profiling (8.794 ms) : 8659, 8928
.   : milestone, 8794,
tracing (7.708 ms) : 7592, 7823
.   : milestone, 7708,
section candidate
no_agent (4.389 ms) : 4337, 4441
.   : milestone, 4389,
iast (9.084 ms) : 8939, 9230
.   : milestone, 9084,
iast_FULL (13.85 ms) : 13572, 14128
.   : milestone, 13850,
iast_GLOBAL (10.191 ms) : 10002, 10381
.   : milestone, 10191,
profiling (9.068 ms) : 8927, 9209
.   : milestone, 9068,
tracing (7.611 ms) : 7504, 7719
.   : milestone, 7611,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.35 ms [4.298 ms, 4.402 ms] -
iast 9.361 ms [9.201 ms, 9.522 ms] 5.011 ms (115.2%)
iast_FULL 14.046 ms [13.767 ms, 14.326 ms] 9.696 ms (222.9%)
iast_GLOBAL 9.992 ms [9.818 ms, 10.166 ms] 5.641 ms (129.7%)
profiling 8.794 ms [8.659 ms, 8.928 ms] 4.443 ms (102.1%)
tracing 7.708 ms [7.592 ms, 7.823 ms] 3.357 ms (77.2%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 4.389 ms [4.337 ms, 4.441 ms] -
iast 9.084 ms [8.939 ms, 9.23 ms] 4.695 ms (107.0%)
iast_FULL 13.85 ms [13.572 ms, 14.128 ms] 9.46 ms (215.5%)
iast_GLOBAL 10.191 ms [10.002 ms, 10.381 ms] 5.802 ms (132.2%)
profiling 9.068 ms [8.927 ms, 9.209 ms] 4.679 ms (106.6%)
tracing 7.611 ms [7.504 ms, 7.719 ms] 3.222 ms (73.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master sarahchen6/1.49.0-branch
git_commit_date 1750259064 1750274142
git_commit_sha 712d5c3 d1c08f0
release_version 1.50.0-SNAPSHOT~712d5c320e 1.50.0-SNAPSHOT~d1c08f02be
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1750277113 1750277113
ci_job_id 988983431 988983431
ci_pipeline_id 68164219 68164219
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-bziamzy-project-304-concurrent-1-hk30q2b4 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-bziamzy-project-304-concurrent-1-hk30q2b4 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 12 metrics, 0 unstable metrics.

Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.481 ms) : 1469, 1493
.   : milestone, 1481,
appsec (2.413 ms) : 2364, 2462
.   : milestone, 2413,
iast (2.203 ms) : 2141, 2264
.   : milestone, 2203,
iast_GLOBAL (2.253 ms) : 2190, 2315
.   : milestone, 2253,
profiling (2.058 ms) : 2009, 2108
.   : milestone, 2058,
tracing (2.014 ms) : 1966, 2061
.   : milestone, 2014,
section candidate
no_agent (1.487 ms) : 1475, 1498
.   : milestone, 1487,
appsec (2.406 ms) : 2357, 2454
.   : milestone, 2406,
iast (2.191 ms) : 2130, 2252
.   : milestone, 2191,
iast_GLOBAL (2.237 ms) : 2175, 2299
.   : milestone, 2237,
profiling (2.027 ms) : 1978, 2075
.   : milestone, 2027,
tracing (2.009 ms) : 1962, 2057
.   : milestone, 2009,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.481 ms [1.469 ms, 1.493 ms] -
appsec 2.413 ms [2.364 ms, 2.462 ms] 931.671 µs (62.9%)
iast 2.203 ms [2.141 ms, 2.264 ms] 721.944 µs (48.7%)
iast_GLOBAL 2.253 ms [2.19 ms, 2.315 ms] 771.559 µs (52.1%)
profiling 2.058 ms [2.009 ms, 2.108 ms] 577.36 µs (39.0%)
tracing 2.014 ms [1.966 ms, 2.061 ms] 532.848 µs (36.0%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.487 ms [1.475 ms, 1.498 ms] -
appsec 2.406 ms [2.357 ms, 2.454 ms] 918.677 µs (61.8%)
iast 2.191 ms [2.13 ms, 2.252 ms] 704.387 µs (47.4%)
iast_GLOBAL 2.237 ms [2.175 ms, 2.299 ms] 750.44 µs (50.5%)
profiling 2.027 ms [1.978 ms, 2.075 ms] 539.794 µs (36.3%)
tracing 2.009 ms [1.962 ms, 2.057 ms] 522.245 µs (35.1%)
Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~d1c08f02be, baseline=1.50.0-SNAPSHOT~712d5c320e
    dateFormat X
    axisFormat %s
section baseline
no_agent (14.728 s) : 14728000, 14728000
.   : milestone, 14728000,
appsec (14.825 s) : 14825000, 14825000
.   : milestone, 14825000,
iast (18.552 s) : 18552000, 18552000
.   : milestone, 18552000,
iast_GLOBAL (17.869 s) : 17869000, 17869000
.   : milestone, 17869000,
profiling (15.075 s) : 15075000, 15075000
.   : milestone, 15075000,
tracing (14.906 s) : 14906000, 14906000
.   : milestone, 14906000,
section candidate
no_agent (15.321 s) : 15321000, 15321000
.   : milestone, 15321000,
appsec (15.047 s) : 15047000, 15047000
.   : milestone, 15047000,
iast (18.99 s) : 18990000, 18990000
.   : milestone, 18990000,
iast_GLOBAL (18.174 s) : 18174000, 18174000
.   : milestone, 18174000,
profiling (15.423 s) : 15423000, 15423000
.   : milestone, 15423000,
tracing (14.77 s) : 14770000, 14770000
.   : milestone, 14770000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 14.728 s [14.728 s, 14.728 s] -
appsec 14.825 s [14.825 s, 14.825 s] 97.0 ms (0.7%)
iast 18.552 s [18.552 s, 18.552 s] 3.824 s (26.0%)
iast_GLOBAL 17.869 s [17.869 s, 17.869 s] 3.141 s (21.3%)
profiling 15.075 s [15.075 s, 15.075 s] 347.0 ms (2.4%)
tracing 14.906 s [14.906 s, 14.906 s] 178.0 ms (1.2%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.321 s [15.321 s, 15.321 s] -
appsec 15.047 s [15.047 s, 15.047 s] -274.0 ms (-1.8%)
iast 18.99 s [18.99 s, 18.99 s] 3.669 s (23.9%)
iast_GLOBAL 18.174 s [18.174 s, 18.174 s] 2.853 s (18.6%)
profiling 15.423 s [15.423 s, 15.423 s] 102.0 ms (0.7%)
tracing 14.77 s [14.77 s, 14.77 s] -551.0 ms (-3.6%)

amarziali and others added 8 commits June 12, 2025 11:57
@amarziali @PerfectSlayer
align opensearch rest muzzle to latestdep coverage (#8778)
32f011d
@daniel-mohedano @PerfectSlayer
Update muzzle directive for spark-2.13 (#8872)
8c1fa12
@amarziali @PerfectSlayer
Limit latestdep for spark 3.x (#8895)
e22b775
@amarziali @PerfectSlayer
adjust muzzle for vertx-mysql-client (#8828)
4cd55d3
@mhlidd @amarziali @PerfectSlayer
Adjust muzzle limit for vertx instrumentations (#8846)
3e38785 
* removing limits for version 5.0.0 on muzzle

* add vertx-pg-client-4.4.2

* workaround

* use java 11 for vertx mysql client muzzle

* use java 11 for all the vertx 3.9 checking 5+

* fix muzzle for vertx-web

* modifying vertx-web-4.0

* skip assertInverse for muzzle vertx web 4

---------

Co-authored-by: Andrea Marziali <[email protected]>
@randomanderson @sarahchen6
Various Gitlab tweaks and test fixes (#8847)
8cd5fa8 
* skip should be for entire class

* bump memory to combat oom issues

* bump smoketests and forked tests memory and defaults
@randomanderson @smola @sarahchen6
Run tests in Gitlab (#8656)
d76348c 
* first pass checks

* add some debug

* Revert "add some debug"

This reverts commit 26a7adb.

* Normalize node index and node total

* fix normalization

* actually use the normalized values

* first pass: migrate tests to Gitlab

* less scenarios for now

* bump memory and workers

* actually use task partitions

* Add section start/end

* Allow individual tests to control gradle memory

* remove a little of the reportspam

* use concrete docker image tags

* configure test containers to use mirror

* fix limit

* fix missing container versions

* Better gitlab naming with smarter split

* update caches with test dep cache, save test results

* Cleanup repositories {} blocks

* update proxy url to use internal routing

* no "s"

* allow insecure everywhere

* include apache with confluence

* remove intentional fail

* use gradlePluginCache, mavenCentral after pluginportal

* proxy mule as well

* change BUILD_CACHE_TYPE to CACHE_TYPE since its used for both

* split cache like with circleci. Cache became too big

* Some caches are too big to be saved

* test fixes

* more test fixes

* disable some tests for now

* more fixes

* Use maven proxy with openliberty smoketests

* missed one more mongodb case

* enable some more of the matrix

* Fix lettuce tests

* Fix lettuce tests

* cleanup imports

* create asyncCommands

* codenarc

* Cleanup aggregator tests to try to reduce flakiness

* debug maven logs for now

* try non-ssh url

* Revert "debug maven logs for now"

This reverts commit c5c158f.

* add some debug logging

* codenarc

* try fixed value of parallelism

* enable debug logs for smoketest building only

* limit number of reserved threads

* some debug for why the smoketest is not running

* more debug for gradle launcher failure

* try fixing graal test

* try to force the logging

* codenarc

* set available processors for jetty

* codenarc

* bluntly bump max threads

* some more debug

* remove verbose logging

* try memcached test with new location

* opt out of split by parallel

* Try with 3 gradle workers

* try decreasing reporting interval

* Refractor CrashTrackingSmokeTest to use OutputThreads

* Simple forked memory test

* better ergonomics

* try increasing timeout

* extend agent test runner

* don't override forkedMinHeapSize and forkedMaxHeapSize

* print out requests

* remove logging statement

* Better crash tracking

* remove intentionally failing test

* use a test UDP server to record oom events

* Make crash tracking smoke test more resiliant

* loop through messages until the correct event

* some debug

* configurable dogstatsd port

* enable everything

* use sleep instead of park nanos

* save body data ahead of time

* retry once for system failures

* spotless

* cleanup

* spotless/merge issues

* Skip ExceptionHistogramTest on J9

* add a print statement around the body for debugging

* comment out all but 8, 17, and 21 for now

* fix spring 6 forwarding headers

* revert logging statements

* needs to be set in 3.1 for latestDepTest

* faster OOM

* property was set in incorrect location

* ssi_smoke, flaky, and esoteric jvms on master only

* use normal runner when TestContainers isn't needed

* skip forwarding header filter here too

* tweaks

* limit executor size and fix IPV4 assumptions

* try more workers with less memory

* set max memory for all crashtracking tests

* double retry

* scale up memory, less jobs

* revert memory bump, increase forked tests

* tweaks

* add flag for nondefault jvms and flaky tests

* correct description

* Externalize runtime processors override

* ExceptionHistogramTest exclusions already handled by master

* increase spring native build memory

* bump gradle build timeout

* bump graalvm compile memory

---------

Co-authored-by: Santiago Mola <[email protected]>
@sarahchen6
Address remaining merge conflicts
3023be2
@pr-commenter
Copy link

pr-commenter bot commented Jun 12, 2025

Debugger benchmarks

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
ci_job_date 1749744816 1749745200
end_time 2025-06-12T16:14:56 2025-06-12T16:21:19
git_branch master sarahchen6/1.49.0-branch
git_commit_sha 1a9af7b a616391
start_time 2025-06-12T16:13:37 2025-06-12T16:20:01
See matching parameters
Baseline Candidate
ci_job_id 979184465 979184465
ci_pipeline_id 67563981 67563981
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
git_commit_date 1749743744 1749743744

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 10 metrics, 5 unstable metrics.

See unchanged results
scenario Δ mean agg_http_req_duration_min Δ mean agg_http_req_duration_p50 Δ mean agg_http_req_duration_p75 Δ mean agg_http_req_duration_p99 Δ mean throughput
scenario:noprobe unstable
[-41.158µs; +13.562µs] or [-14.605%; +4.812%]		 unstable
[-58.197µs; +19.863µs] or [-17.943%; +6.124%]		 unstable
[-68.454µs; +30.873µs] or [-20.202%; +9.111%]		 unstable
[-56.897µs; +186.769µs] or [-5.908%; +19.394%]		 same
scenario:basic same same same unstable
[-33.720µs; +138.638µs] or [-4.022%; +16.536%]		 same
scenario:loop unsure
[-50.753µs; -32.812µs] or [-0.474%; -0.307%]		 unsure
[-67.090µs; -6.616µs] or [-0.617%; -0.061%]		 same same same
Request duration reports for reports 
gantt
    title reports - request duration [CI 0.99] : candidate=None, baseline=None
    dateFormat X
    axisFormat %s
section baseline
noprobe (324.337 µs) : 276, 372
.   : milestone, 324,
basic (314.767 µs) : 304, 325
.   : milestone, 315,
loop (10.877 ms) : 10855, 10900
.   : milestone, 10877,
section candidate
noprobe (305.17 µs) : 284, 327
.   : milestone, 305,
basic (308.677 µs) : 301, 316
.   : milestone, 309,
loop (10.84 ms) : 10807, 10873
.   : milestone, 10840,
Loading
  • baseline results
Scenario Request median duration [CI 0.99]
noprobe 324.337 µs [276.412 µs, 372.262 µs]
basic 314.767 µs [304.194 µs, 325.34 µs]
loop 10.877 ms [10.855 ms, 10.9 ms]
  • candidate results
Scenario Request median duration [CI 0.99]
noprobe 305.17 µs [283.577 µs, 326.763 µs]
basic 308.677 µs [300.905 µs, 316.449 µs]
loop 10.84 ms [10.807 ms, 10.873 ms]

@sarahchen6 sarahchen6 force-pushed the sarahchen6/1.49.0-branch branch from a616391 to 3023be2 Compare June 12, 2025 19:10
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
tag: do not merge Do not merge changes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
7 participants
@sarahchen6 @amarziali @daniel-mohedano @mhlidd @randomanderson @smola @ddyurchenko