Skip to content

Auto upgrade system-tests #8922

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

Auto upgrade system-tests #8922

wants to merge 2 commits into from

Conversation

TonyCTHsu
Copy link
Contributor

@TonyCTHsu TonyCTHsu commented Jun 4, 2025 •
edited
Loading

Motivation

Test with HEAD of main from system tests is inconsistent.

What Does This Do

Setup automation to upgrade the version with commit SHA periodically. It would create an automated pull requests to be review and merged.

The script is validated with

$ export TARGET=".github/workflows/run-system-tests.yaml"
$ export PATTERN='(\s*system-tests.yml@)(\S+)(\s+# Automated:.*)'
$ export REF=007f43b50f720be299f0dae27f4fcbabdd623591
$ .github/scripts/update_reference.sh

Target: .github/workflows/run-system-tests.yaml
Ref: 007f43b50f720be299f0dae27f4fcbabdd623591
Pattern: (\s*system-tests.yml@)(\S+)(\s+# Automated:.*)
✓ Updated references in .github/workflows/run-system-tests.yaml

$ git diff
diff --git a/.github/workflows/run-system-tests.yaml b/.github/workflows/run-system-tests.yaml
index bb320a46dd..feceb83dbb 100644
--- a/.github/workflows/run-system-tests.yaml
+++ b/.github/workflows/run-system-tests.yaml
@@ -50,7 +50,7 @@ jobs:
   main:
     needs:
     - build
-    uses: DataDog/system-tests/.github/workflows/system-tests.yml@main # Automated: This reference is automatically updated.
+    uses: DataDog/system-tests/.github/workflows/system-tests.yml@007f43b50f720be299f0dae27f4fcbabdd623591 # Automated: This reference is automatically updated.
     secrets: inherit
     permissions:
       contents: read

Additional Notes

The workflow requires to allow peter-evans/create-pull-request@* in the repository settings.

Contributor Checklist

Jira ticket: [PROJ-IDENT]

@pr-commenter
Copy link

pr-commenter bot commented Jun 4, 2025 •
edited
Loading

Benchmarks

Startup

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749714344 1749729087
git_commit_sha 58aca3d a6253fc
release_version 1.50.0-SNAPSHOT~58aca3d365 1.50.0-SNAPSHOT~a6253fcf42
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749731395 1749731395
ci_job_id 978379745 978379745
ci_pipeline_id 67515222 67515222
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-63wmvl1-project-304-concurrent-1-4wd99zot 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-63wmvl1-project-304-concurrent-1-4wd99zot 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
module Agent Agent
parent None None
variant iast iast

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 58 metrics, 13 unstable metrics.

Startup time reports for petclinic 
gantt
    title petclinic - global startup overhead: candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.028 s) : 0, 1027550
Total [baseline] (10.47 s) : 0, 10470295
Agent [candidate] (1.021 s) : 0, 1021276
Total [candidate] (10.511 s) : 0, 10510996
section appsec
Agent [baseline] (1.161 s) : 0, 1161153
Total [baseline] (10.673 s) : 0, 10673486
Agent [candidate] (1.162 s) : 0, 1161921
Total [candidate] (10.637 s) : 0, 10637299
section iast
Agent [baseline] (1.149 s) : 0, 1148954
Total [baseline] (10.764 s) : 0, 10764300
Agent [candidate] (1.158 s) : 0, 1158106
Total [candidate] (10.827 s) : 0, 10827261
section profiling
Agent [baseline] (1.265 s) : 0, 1265268
Total [baseline] (10.816 s) : 0, 10816165
Agent [candidate] (1.272 s) : 0, 1271631
Total [candidate] (10.847 s) : 0, 10847124
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.028 s -
Agent appsec 1.161 s 133.603 ms (13.0%)
Agent iast 1.149 s 121.404 ms (11.8%)
Agent profiling 1.265 s 237.718 ms (23.1%)
Total tracing 10.47 s -
Total appsec 10.673 s 203.191 ms (1.9%)
Total iast 10.764 s 294.005 ms (2.8%)
Total profiling 10.816 s 345.87 ms (3.3%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.021 s -
Agent appsec 1.162 s 140.645 ms (13.8%)
Agent iast 1.158 s 136.83 ms (13.4%)
Agent profiling 1.272 s 250.356 ms (24.5%)
Total tracing 10.511 s -
Total appsec 10.637 s 126.303 ms (1.2%)
Total iast 10.827 s 316.265 ms (3.0%)
Total profiling 10.847 s 336.128 ms (3.2%) 
gantt
    title petclinic - break down per module: candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (686.771 ms) : 0, 686771
BytebuddyAgent [candidate] (682.153 ms) : 0, 682153
GlobalTracer [baseline] (241.371 ms) : 0, 241371
GlobalTracer [candidate] (239.51 ms) : 0, 239510
AppSec [baseline] (59.175 ms) : 0, 59175
AppSec [candidate] (56.334 ms) : 0, 56334
Debugger [baseline] (6.123 ms) : 0, 6123
Debugger [candidate] (6.184 ms) : 0, 6184
Remote Config [baseline] (716.903 µs) : 0, 717
Remote Config [candidate] (735.974 µs) : 0, 736
Telemetry [baseline] (9.854 ms) : 0, 9854
Telemetry [candidate] (12.773 ms) : 0, 12773
section appsec
BytebuddyAgent [baseline] (699.141 ms) : 0, 699141
BytebuddyAgent [candidate] (700.182 ms) : 0, 700182
GlobalTracer [baseline] (237.136 ms) : 0, 237136
GlobalTracer [candidate] (237.264 ms) : 0, 237264
IAST [baseline] (21.792 ms) : 0, 21792
IAST [candidate] (21.646 ms) : 0, 21646
AppSec [baseline] (176.472 ms) : 0, 176472
AppSec [candidate] (175.817 ms) : 0, 175817
Debugger [baseline] (5.983 ms) : 0, 5983
Debugger [candidate] (6.344 ms) : 0, 6344
Remote Config [baseline] (627.226 µs) : 0, 627
Remote Config [candidate] (630.442 µs) : 0, 630
Telemetry [baseline] (7.377 ms) : 0, 7377
Telemetry [candidate] (7.313 ms) : 0, 7313
section iast
BytebuddyAgent [baseline] (800.866 ms) : 0, 800866
BytebuddyAgent [candidate] (807.815 ms) : 0, 807815
GlobalTracer [baseline] (230.508 ms) : 0, 230508
GlobalTracer [candidate] (231.47 ms) : 0, 231470
IAST [baseline] (28.767 ms) : 0, 28767
IAST [candidate] (26.469 ms) : 0, 26469
AppSec [baseline] (50.886 ms) : 0, 50886
AppSec [candidate] (54.044 ms) : 0, 54044
Debugger [baseline] (5.948 ms) : 0, 5948
Debugger [candidate] (6.0 ms) : 0, 6000
Remote Config [baseline] (596.197 µs) : 0, 596
Remote Config [candidate] (601.591 µs) : 0, 602
Telemetry [baseline] (7.931 ms) : 0, 7931
Telemetry [candidate] (7.99 ms) : 0, 7990
section profiling
BytebuddyAgent [baseline] (675.151 ms) : 0, 675151
BytebuddyAgent [candidate] (679.641 ms) : 0, 679641
GlobalTracer [baseline] (360.255 ms) : 0, 360255
GlobalTracer [candidate] (361.887 ms) : 0, 361887
AppSec [baseline] (61.474 ms) : 0, 61474
AppSec [candidate] (61.567 ms) : 0, 61567
Debugger [baseline] (6.11 ms) : 0, 6110
Debugger [candidate] (6.034 ms) : 0, 6034
Remote Config [baseline] (669.532 µs) : 0, 670
Remote Config [candidate] (658.721 µs) : 0, 659
Telemetry [baseline] (8.177 ms) : 0, 8177
Telemetry [candidate] (8.092 ms) : 0, 8092
ProfilingAgent [baseline] (102.745 ms) : 0, 102745
ProfilingAgent [candidate] (102.585 ms) : 0, 102585
Profiling [baseline] (102.769 ms) : 0, 102769
Profiling [candidate] (102.609 ms) : 0, 102609
Loading
Startup time reports for insecure-bank 
gantt
    title insecure-bank - global startup overhead: candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365

    dateFormat X
    axisFormat %s
section tracing
Agent [baseline] (1.024 s) : 0, 1024272
Total [baseline] (8.535 s) : 0, 8534636
Agent [candidate] (1.039 s) : 0, 1039210
Total [candidate] (8.542 s) : 0, 8542493
section iast
Agent [baseline] (1.15 s) : 0, 1150265
Total [baseline] (9.194 s) : 0, 9194462
Agent [candidate] (1.147 s) : 0, 1147031
Total [candidate] (9.187 s) : 0, 9186508
section iast_HARDCODED_SECRET_DISABLED
Agent [baseline] (1.152 s) : 0, 1151613
Total [baseline] (9.167 s) : 0, 9167257
Agent [candidate] (1.149 s) : 0, 1148819
Total [candidate] (9.184 s) : 0, 9183618
section iast_TELEMETRY_OFF
Agent [baseline] (1.146 s) : 0, 1145617
Total [baseline] (9.288 s) : 0, 9288417
Agent [candidate] (1.145 s) : 0, 1145047
Total [candidate] (9.241 s) : 0, 9241004
Loading
  • baseline results
Module Variant Duration Δ tracing
Agent tracing 1.024 s -
Agent iast 1.15 s 125.994 ms (12.3%)
Agent iast_HARDCODED_SECRET_DISABLED 1.152 s 127.341 ms (12.4%)
Agent iast_TELEMETRY_OFF 1.146 s 121.346 ms (11.8%)
Total tracing 8.535 s -
Total iast 9.194 s 659.826 ms (7.7%)
Total iast_HARDCODED_SECRET_DISABLED 9.167 s 632.622 ms (7.4%)
Total iast_TELEMETRY_OFF 9.288 s 753.781 ms (8.8%)
  • candidate results
Module Variant Duration Δ tracing
Agent tracing 1.039 s -
Agent iast 1.147 s 107.821 ms (10.4%)
Agent iast_HARDCODED_SECRET_DISABLED 1.149 s 109.609 ms (10.5%)
Agent iast_TELEMETRY_OFF 1.145 s 105.837 ms (10.2%)
Total tracing 8.542 s -
Total iast 9.187 s 644.015 ms (7.5%)
Total iast_HARDCODED_SECRET_DISABLED 9.184 s 641.125 ms (7.5%)
Total iast_TELEMETRY_OFF 9.241 s 698.511 ms (8.2%) 
gantt
    title insecure-bank - break down per module: candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365

    dateFormat X
    axisFormat %s
section tracing
BytebuddyAgent [baseline] (684.689 ms) : 0, 684689
BytebuddyAgent [candidate] (696.165 ms) : 0, 696165
GlobalTracer [baseline] (240.183 ms) : 0, 240183
GlobalTracer [candidate] (241.557 ms) : 0, 241557
AppSec [baseline] (57.596 ms) : 0, 57596
AppSec [candidate] (56.015 ms) : 0, 56015
Debugger [baseline] (6.178 ms) : 0, 6178
Debugger [candidate] (6.244 ms) : 0, 6244
Remote Config [baseline] (733.054 µs) : 0, 733
Remote Config [candidate] (736.461 µs) : 0, 736
Telemetry [baseline] (11.365 ms) : 0, 11365
Telemetry [candidate] (14.515 ms) : 0, 14515
section iast
BytebuddyAgent [baseline] (802.546 ms) : 0, 802546
BytebuddyAgent [candidate] (799.95 ms) : 0, 799950
GlobalTracer [baseline] (230.632 ms) : 0, 230632
GlobalTracer [candidate] (230.06 ms) : 0, 230060
IAST [baseline] (26.127 ms) : 0, 26127
IAST [candidate] (24.528 ms) : 0, 24528
AppSec [baseline] (53.012 ms) : 0, 53012
AppSec [candidate] (54.661 ms) : 0, 54661
Debugger [baseline] (5.983 ms) : 0, 5983
Debugger [candidate] (5.911 ms) : 0, 5911
Remote Config [baseline] (591.743 µs) : 0, 592
Remote Config [candidate] (589.105 µs) : 0, 589
Telemetry [baseline] (7.923 ms) : 0, 7923
Telemetry [candidate] (7.832 ms) : 0, 7832
section iast_HARDCODED_SECRET_DISABLED
BytebuddyAgent [baseline] (803.868 ms) : 0, 803868
BytebuddyAgent [candidate] (801.206 ms) : 0, 801206
GlobalTracer [baseline] (230.095 ms) : 0, 230095
GlobalTracer [candidate] (230.349 ms) : 0, 230349
IAST [baseline] (28.632 ms) : 0, 28632
IAST [candidate] (27.735 ms) : 0, 27735
AppSec [baseline] (50.966 ms) : 0, 50966
AppSec [candidate] (51.603 ms) : 0, 51603
Debugger [baseline] (5.969 ms) : 0, 5969
Debugger [candidate] (5.94 ms) : 0, 5940
Remote Config [baseline] (600.92 µs) : 0, 601
Remote Config [candidate] (592.495 µs) : 0, 592
Telemetry [baseline] (7.96 ms) : 0, 7960
Telemetry [candidate] (7.94 ms) : 0, 7940
section iast_TELEMETRY_OFF
BytebuddyAgent [baseline] (797.957 ms) : 0, 797957
BytebuddyAgent [candidate] (797.424 ms) : 0, 797424
GlobalTracer [baseline] (230.603 ms) : 0, 230603
GlobalTracer [candidate] (230.502 ms) : 0, 230502
IAST [baseline] (27.138 ms) : 0, 27138
IAST [candidate] (26.555 ms) : 0, 26555
AppSec [baseline] (51.876 ms) : 0, 51876
AppSec [candidate] (52.058 ms) : 0, 52058
Debugger [baseline] (6.0 ms) : 0, 6000
Debugger [candidate] (5.971 ms) : 0, 5971
Remote Config [baseline] (610.017 µs) : 0, 610
Remote Config [candidate] (597.886 µs) : 0, 598
Telemetry [baseline] (7.896 ms) : 0, 7896
Telemetry [candidate] (7.778 ms) : 0, 7778
Loading

Load

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
end_time 2025-06-12T12:00:25 2025-06-12T12:09:23
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749714344 1749729087
git_commit_sha 58aca3d a6253fc
release_version 1.50.0-SNAPSHOT~58aca3d365 1.50.0-SNAPSHOT~a6253fcf42
start_time 2025-06-12T12:00:10 2025-06-12T12:09:09
See matching parameters
Baseline Candidate
application insecure-bank insecure-bank
ci_job_date 1749730771 1749730771
ci_job_id 978379746 978379746
ci_pipeline_id 67515222 67515222
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-28kegq-project-304-concurrent-1-dpx6gvqt 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-28kegq-project-304-concurrent-1-dpx6gvqt 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
thresholds_or_results results results
variant iast iast

Summary

Found 3 performance improvements and 0 performance regressions! Performance is the same for 7 metrics, 16 unstable metrics.

scenario Δ mean http_req_duration Δ mean throughput candidate mean http_req_duration candidate mean throughput baseline mean http_req_duration baseline mean throughput
scenario:load:petclinic:appsec better
[-1.440ms; -1.317ms] or [-101.289%; -92.672%]		 unstable
[+4903.360op/s; +21579.398op/s] or [+177.747%; +782.253%]		 0.043ms 16000.000op/s 1.421ms 2758.621op/s
scenario:load:petclinic:profiling better
[-1.453ms; -1.301ms] or [-69.472%; -62.246%]		 unstable
[-1587.369op/s; +7587.369op/s] or [-79.368%; +379.368%]		 0.714ms 5000.000op/s 2.091ms 2000.000op/s
scenario:load:petclinic:tracing better
[-1.451ms; -1.319ms] or [-13.531%; -12.294%]		 unstable
[+36.654op/s; +103.408op/s] or [+7.972%; +22.491%]		 9.340ms 529.801op/s 10.725ms 459.770op/s
Request duration reports for insecure-bank 
gantt
    title insecure-bank - request duration [CI 0.99] : candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365
    dateFormat X
    axisFormat %s
section baseline
no_agent (380.84 µs) : 361, 401
.   : milestone, 381,
iast (518.406 µs) : 495, 542
.   : milestone, 518,
iast_FULL (745.787 µs) : 722, 769
.   : milestone, 746,
iast_GLOBAL (582.078 µs) : 558, 606
.   : milestone, 582,
iast_HARDCODED_SECRET_DISABLED (531.412 µs) : 510, 553
.   : milestone, 531,
iast_INACTIVE (472.321 µs) : 449, 495
.   : milestone, 472,
iast_TELEMETRY_OFF (510.433 µs) : 487, 534
.   : milestone, 510,
tracing (463.58 µs) : 441, 486
.   : milestone, 464,
section candidate
no_agent (383.507 µs) : 363, 404
.   : milestone, 384,
iast (529.472 µs) : 508, 551
.   : milestone, 529,
iast_FULL (747.423 µs) : 724, 771
.   : milestone, 747,
iast_GLOBAL (583.056 µs) : 559, 607
.   : milestone, 583,
iast_HARDCODED_SECRET_DISABLED (524.533 µs) : 503, 546
.   : milestone, 525,
iast_INACTIVE (471.951 µs) : 449, 495
.   : milestone, 472,
iast_TELEMETRY_OFF (524.173 µs) : 501, 548
.   : milestone, 524,
tracing (465.639 µs) : 443, 488
.   : milestone, 466,
Loading
  • baseline results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 380.84 µs [360.669 µs, 401.011 µs] -
iast 518.406 µs [494.917 µs, 541.896 µs] 137.566 µs (36.1%)
iast_FULL 745.787 µs [722.165 µs, 769.409 µs] 364.947 µs (95.8%)
iast_GLOBAL 582.078 µs [558.5 µs, 605.656 µs] 201.238 µs (52.8%)
iast_HARDCODED_SECRET_DISABLED 531.412 µs [509.736 µs, 553.088 µs] 150.572 µs (39.5%)
iast_INACTIVE 472.321 µs [449.498 µs, 495.144 µs] 91.481 µs (24.0%)
iast_TELEMETRY_OFF 510.433 µs [487.015 µs, 533.85 µs] 129.592 µs (34.0%)
tracing 463.58 µs [441.286 µs, 485.874 µs] 82.74 µs (21.7%)
  • candidate results
Variant Request duration [CI 0.99] Δ no_agent
no_agent 383.507 µs [362.945 µs, 404.069 µs] -
iast 529.472 µs [507.501 µs, 551.443 µs] 145.965 µs (38.1%)
iast_FULL 747.423 µs [723.686 µs, 771.16 µs] 363.916 µs (94.9%)
iast_GLOBAL 583.056 µs [559.431 µs, 606.681 µs] 199.548 µs (52.0%)
iast_HARDCODED_SECRET_DISABLED 524.533 µs [502.915 µs, 546.151 µs] 141.026 µs (36.8%)
iast_INACTIVE 471.951 µs [449.353 µs, 494.55 µs] 88.444 µs (23.1%)
iast_TELEMETRY_OFF 524.173 µs [500.84 µs, 547.506 µs] 140.666 µs (36.7%)
tracing 465.639 µs [443.428 µs, 487.849 µs] 82.131 µs (21.4%)

Dacapo

Parameters

Baseline Candidate
baseline_or_candidate baseline candidate
git_branch master tonycthsu/pin-system-tests
git_commit_date 1749714344 1749729087
git_commit_sha 58aca3d a6253fc
release_version 1.50.0-SNAPSHOT~58aca3d365 1.50.0-SNAPSHOT~a6253fcf42
See matching parameters
Baseline Candidate
application biojava biojava
ci_job_date 1749730992 1749730992
ci_job_id 978379747 978379747
ci_pipeline_id 67515222 67515222
cpu_model Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz Intel(R) Xeon(R) Platinum 8259CL CPU @ 2.50GHz
kernel_version Linux runner-63wmvl1-project-304-concurrent-3-2hrh7muu 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux Linux runner-63wmvl1-project-304-concurrent-3-2hrh7muu 6.8.0-1029-aws #31~22.04.1-Ubuntu SMP Thu Apr 24 21:16:18 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
variant appsec appsec

Summary

Found 0 performance improvements and 0 performance regressions! Performance is the same for 11 metrics, 1 unstable metrics.

Execution time for biojava 
gantt
    title biojava - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365
    dateFormat X
    axisFormat %s
section baseline
no_agent (15.404 s) : 15404000, 15404000
.   : milestone, 15404000,
appsec (14.973 s) : 14973000, 14973000
.   : milestone, 14973000,
iast (18.789 s) : 18789000, 18789000
.   : milestone, 18789000,
iast_GLOBAL (18.025 s) : 18025000, 18025000
.   : milestone, 18025000,
profiling (15.312 s) : 15312000, 15312000
.   : milestone, 15312000,
tracing (14.963 s) : 14963000, 14963000
.   : milestone, 14963000,
section candidate
no_agent (15.081 s) : 15081000, 15081000
.   : milestone, 15081000,
appsec (14.586 s) : 14586000, 14586000
.   : milestone, 14586000,
iast (18.499 s) : 18499000, 18499000
.   : milestone, 18499000,
iast_GLOBAL (18.021 s) : 18021000, 18021000
.   : milestone, 18021000,
profiling (15.31 s) : 15310000, 15310000
.   : milestone, 15310000,
tracing (14.944 s) : 14944000, 14944000
.   : milestone, 14944000,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.404 s [15.404 s, 15.404 s] -
appsec 14.973 s [14.973 s, 14.973 s] -431.0 ms (-2.8%)
iast 18.789 s [18.789 s, 18.789 s] 3.385 s (22.0%)
iast_GLOBAL 18.025 s [18.025 s, 18.025 s] 2.621 s (17.0%)
profiling 15.312 s [15.312 s, 15.312 s] -92.0 ms (-0.6%)
tracing 14.963 s [14.963 s, 14.963 s] -441.0 ms (-2.9%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 15.081 s [15.081 s, 15.081 s] -
appsec 14.586 s [14.586 s, 14.586 s] -495.0 ms (-3.3%)
iast 18.499 s [18.499 s, 18.499 s] 3.418 s (22.7%)
iast_GLOBAL 18.021 s [18.021 s, 18.021 s] 2.94 s (19.5%)
profiling 15.31 s [15.31 s, 15.31 s] 229.0 ms (1.5%)
tracing 14.944 s [14.944 s, 14.944 s] -137.0 ms (-0.9%)
Execution time for tomcat 
gantt
    title tomcat - execution time [CI 0.99] : candidate=1.50.0-SNAPSHOT~a6253fcf42, baseline=1.50.0-SNAPSHOT~58aca3d365
    dateFormat X
    axisFormat %s
section baseline
no_agent (1.489 ms) : 1478, 1501
.   : milestone, 1489,
appsec (2.424 ms) : 2375, 2473
.   : milestone, 2424,
iast (2.201 ms) : 2139, 2262
.   : milestone, 2201,
iast_GLOBAL (2.237 ms) : 2175, 2300
.   : milestone, 2237,
profiling (2.492 ms) : 2320, 2665
.   : milestone, 2492,
tracing (2.015 ms) : 1967, 2062
.   : milestone, 2015,
section candidate
no_agent (1.477 ms) : 1465, 1488
.   : milestone, 1477,
appsec (2.411 ms) : 2362, 2459
.   : milestone, 2411,
iast (2.2 ms) : 2139, 2261
.   : milestone, 2200,
iast_GLOBAL (2.247 ms) : 2185, 2309
.   : milestone, 2247,
profiling (2.03 ms) : 1981, 2079
.   : milestone, 2030,
tracing (2.015 ms) : 1968, 2062
.   : milestone, 2015,
Loading
  • baseline results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.489 ms [1.478 ms, 1.501 ms] -
appsec 2.424 ms [2.375 ms, 2.473 ms] 934.665 µs (62.8%)
iast 2.201 ms [2.139 ms, 2.262 ms] 711.48 µs (47.8%)
iast_GLOBAL 2.237 ms [2.175 ms, 2.3 ms] 748.172 µs (50.2%)
profiling 2.492 ms [2.32 ms, 2.665 ms] 1.003 ms (67.4%)
tracing 2.015 ms [1.967 ms, 2.062 ms] 525.262 µs (35.3%)
  • candidate results
Variant Execution Time [CI 0.99] Δ no_agent
no_agent 1.477 ms [1.465 ms, 1.488 ms] -
appsec 2.411 ms [2.362 ms, 2.459 ms] 933.934 µs (63.2%)
iast 2.2 ms [2.139 ms, 2.261 ms] 723.441 µs (49.0%)
iast_GLOBAL 2.247 ms [2.185 ms, 2.309 ms] 770.309 µs (52.2%)
profiling 2.03 ms [1.981 ms, 2.079 ms] 553.15 µs (37.5%)
tracing 2.015 ms [1.968 ms, 2.062 ms] 538.261 µs (36.4%)

@TonyCTHsu TonyCTHsu force-pushed the tonycthsu/pin-system-tests branch from 2fdd417 to 3532e4a Compare June 5, 2025 13:33
@TonyCTHsu TonyCTHsu marked this pull request as ready for review June 5, 2025 13:43
@TonyCTHsu TonyCTHsu requested a review from a team as a code owner June 5, 2025 13:43
@TonyCTHsu TonyCTHsu requested review from sarahchen6 and removed request for a team June 5, 2025 13:43
@github-actions GitHub Actions
Copy link
Contributor

github-actions bot commented Jun 5, 2025

Hi! 👋 Thanks for your pull request! 🎉

To help us review it, please make sure to:

  • Add at least one type, and one component or instrumentation label to the pull request

If you need help, please check our contributing guidelines.

@TonyCTHsu TonyCTHsu added tag: no release notes Changes to exclude from release notes comp: tooling Build & Tooling dependencies labels Jun 5, 2025
@TonyCTHsu TonyCTHsu changed the title Setup auto update system-tests Auto upgrade system-tests Jun 12, 2025
sarahchen6
sarahchen6 approved these changes Jun 13, 2025
View reviewed changes
Copy link
Contributor

@sarahchen6 sarahchen6 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice to automate this!

@PerfectSlayer PerfectSlayer added tag: dependencies Dependencies related changes and removed dependencies labels Jun 13, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sarahchen6 sarahchen6 sarahchen6 approved these changes

Assignees
No one assigned
Labels
comp: tooling Build & Tooling tag: dependencies Dependencies related changes tag: no release notes Changes to exclude from release notes
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@TonyCTHsu @sarahchen6 @PerfectSlayer