2025 benchmark update

[Kobzol]

In 2025, it will be time to refresh and update our primary benchmarks again, based on our benchmark update policy.

The update policy, and a guide of how to actually update the benchmarks is described here. In short:

1. Download the new benchmark

$ cargo run --bin collector download -c Primary -a <library/binary> crate <name> <version>

2. Add Cargo.lock in the downloaded crate to git, even if it is ignored!
3. Commit
4. Copy perf-config.json and .patch files from the old version
5. Add [workspace] to Cargo.toml if it's not already there
6. Check that

$ cargo build -p collector; cargo run --bin collector bench_local `rustup +stable which rustc` --include <benchmark-name>

works

• If it fails because "patch does not apply", re-create the old patch files based on the updated benchmark source code. For each patch file, perform the same change as was in the original patch file, and then run git diff --patch --relative from the directory of the benchmark and store the output into the patch filename.

7. Modify collector/compile-benchmarks/README.md
8. Modify collector/compile-benchmarks/REUSE.toml
9. Commit
10. Run bench_local for both the old and the new benchmark version, then run the website locally (cargo run --bin site), click on "Display raw data" and "Show non-relevant results" in the "Filters" section and post a screenshot of the results. Before running the benchmark, make sure to clear any stale benchmark data (rm results.db*).

Progress

• bitmaps (@Kobzol)
  • Updated to 3.2.1 (Update bitmaps to 3.2.1 #2040, 14. 4.)
  • Removed old version (Remove bitmaps-3.1.0 benchmark #2075)
• cargo (@Kobzol)
  • Updated to 0.87.1 (Update cargo to 0.87.1 #2071, 16. 4.)
  • Removed old version (Remove cargo 0.60.0 benchmark #2079)
• clap (@Kobzol)
  • Replaced with clap_derive 4.5.32 (Add clap_derive 4.5.32 benchmark #2073, 28. 4.)
  • Removed old version
• cranelift-codegen (@Kobzol)
  • Updated to 0.119.0 (Update cranelift codegen to 0.119.0 #2077, 23. 4.)
  • Removed old version (Remove cranelift-codegen-0.82.1 benchmark #2089)
• diesel (@Kobzol)
  • Updated to 2.2.10 (Update diesel to 2.2.10 #2082, 25. 4.)
  • Removed old version (Remove diesel-1.4.8 benchmark #2090)
• exa (@Kobzol)
  • Replaced with eza 0.21.2 (Add eza 0.21.2 benchmark #2084, 29. 4.)
  • Removed old version
• html5ever (@Kobzol)
  • Updated to 0.31.0 (Update html5ever to 0.31.0 #2069, 15. 4.)
  • Removed old version (Remove html5ever-0.26.0 benchmark #2076)
• hyper (@Kobzol)
  • Updated to 1.6.0 (Update hyper to 1.6.0 #2091, 30. 4.)
  • Removed old version
• image (@Kobzol)
  • Updated to X.Y.Z
  • Removed old version
• libc (@Kobzol)
  • Updated to X.Y.Z
  • Removed old version
• nalgebra
  • Updated to X.Y.Z
  • Removed old version
• regex
  • Updated to X.Y.Z
  • Removed old version
• ripgrep
  • Updated to X.Y.Z
  • Removed old version
• ripgrep-tiny
  • Updated to X.Y.Z
  • Removed old version
• serde
  • Updated to X.Y.Z
  • Removed old version
• serde_derive
  • Updated to X.Y.Z
  • Removed old version
• stm32f4
  • Updated to X.Y.Z
  • Removed old version
• syn
  • Updated to X.Y.Z
  • Removed old version
• typenum (@nnethercote)
  • Updated to 0.18.0 (Add typenum-1.18.0 benchmark #2088, 30. 4.)
  • Removed old version
• unicode-normalization (@nnethercote)
  • Updated to 0.1.24 (Add unicode-normalization-0.1.24 #2087, 30. 4.)
  • Removed old version
• webrender (@nnethercote)
  • No new version to be added
  • Removed old version (Remove webrender-2022 #2085)

[nnethercote]

As per the discussion on Zulip, we're going to wait a little bit and get some ducks lined up in a row before starting on this.

[Kobzol]

We have started the benchmark update process this week.