Skip to content

GH-5520 WAL for ValueStore #5522

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 2 commits into from
Nov 19, 2025
Merged

GH-5520 WAL for ValueStore #5522

merged 2 commits into from
Nov 19, 2025

Conversation

@hmottestad
Copy link
Contributor

@hmottestad hmottestad commented Oct 10, 2025

GitHub issue resolved: #5520

Briefly describe the changes proposed in this PR:

PR Author Checklist (see the contributor guidelines for more details):

  • my pull request is self-contained
  • I've added tests for the changes I made
  • I've applied code formatting (you can use mvn process-resources to format from the command line)
  • I've squashed my commits where necessary
  • every commit message starts with the issue number (GH-xxxx) followed by a meaningful description of the change

@hmottestad hmottestad force-pushed the GH-5520-WAL-for-ValueStore branch from 294d018 to 3d451b3 Compare November 13, 2025 10:38
@hmottestad
Copy link
Contributor Author

hmottestad commented Nov 13, 2025 •
edited
Loading

Main branch

Benchmark                                                       Mode  Cnt     Score      Error  Units
TransactionsPerSecondBenchmark.largerTransaction               thrpt    3    11.905 ±    3.017  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone      thrpt    3    12.950 ±    3.077  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone     thrpt    3  3788.433 ± 1288.196  ops/s
TransactionsPerSecondBenchmark.transactions                    thrpt    3  6036.651 ± 1945.931  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone           thrpt    3  6380.981 ± 1222.201  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone  thrpt    3     0.127 ±    0.062  ops/s

Benchmark                                                     Mode  Cnt     Score   Error  Units
QueryBenchmark.complexQuery                                   avgt    2    11.014          ms/op
QueryBenchmark.different_datasets_with_similar_distributions  avgt    2     4.019          ms/op
QueryBenchmark.groupByQuery                                   avgt    2     7.598          ms/op
QueryBenchmark.long_chain                                     avgt    2  1925.489          ms/op
QueryBenchmark.lots_of_optional                               avgt    2   486.006          ms/op
QueryBenchmark.minus                                          avgt    2    18.028          ms/op
QueryBenchmark.nested_optionals                               avgt    2   413.050          ms/op
QueryBenchmark.pathExpressionQuery1                           avgt    2    49.646          ms/op
QueryBenchmark.pathExpressionQuery2                           avgt    2     8.634          ms/op
QueryBenchmark.query_distinct_predicates                      avgt    2   504.651          ms/op
QueryBenchmark.simple_filter_not                              avgt    2    20.957          ms/op
QueryBenchmark.zeroOrMore                                     avgt    2     2.139          ms/op

This branch

Benchmark                                                       Mode  Cnt     Score      Error  Units
TransactionsPerSecondBenchmark.largerTransaction               thrpt    3    10.982 ±    3.003  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone      thrpt    3    11.651 ±    1.063  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone     thrpt    3  3721.740 ± 1193.675  ops/s
TransactionsPerSecondBenchmark.transactions                    thrpt    3  6224.654 ± 2793.112  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone           thrpt    3  6213.495 ± 1905.099  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone  thrpt    3     0.104 ±    0.082  ops/s

Benchmark                                                     Mode  Cnt     Score   Error  Units
QueryBenchmark.complexQuery                                   avgt    2     9.565          ms/op
QueryBenchmark.different_datasets_with_similar_distributions  avgt    2     3.161          ms/op
QueryBenchmark.groupByQuery                                   avgt    2     7.738          ms/op
QueryBenchmark.long_chain                                     avgt    2  1750.708          ms/op
QueryBenchmark.lots_of_optional                               avgt    2   413.775          ms/op
QueryBenchmark.minus                                          avgt    2    17.382          ms/op
QueryBenchmark.nested_optionals                               avgt    2   369.031          ms/op
QueryBenchmark.pathExpressionQuery1                           avgt    2    43.984          ms/op
QueryBenchmark.pathExpressionQuery2                           avgt    2     8.141          ms/op
QueryBenchmark.query_distinct_predicates                      avgt    2   483.067          ms/op
QueryBenchmark.simple_filter_not                              avgt    2    17.645          ms/op
QueryBenchmark.zeroOrMore                                     avgt    2     1.970          ms/op

@hmottestad hmottestad changed the base branch from main to develop November 13, 2025 14:25
@hmottestad
Copy link
Contributor Author

hmottestad commented Nov 14, 2025 •
edited
Loading

This branch (with the new memory mapped transaction status file code)

Benchmark                                                            Mode  Cnt      Score      Error  Units
TransactionsPerSecondBenchmark.largerTransaction                    thrpt    3     10.563 ±    8.107  ops/s
TransactionsPerSecondBenchmark.largerTransactionLevelNone           thrpt    3     11.259 ±    4.726  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelNone          thrpt    3   8363.387 ± 2265.402  ops/s
TransactionsPerSecondBenchmark.mediumTransactionsLevelSnapshotRead  thrpt    3   8928.995 ± 1839.064  ops/s
TransactionsPerSecondBenchmark.transactions                         thrpt    3  38936.210 ± 5782.759  ops/s
TransactionsPerSecondBenchmark.transactionsLevelNone                thrpt    3  42461.130 ± 5753.787  ops/s
TransactionsPerSecondBenchmark.veryLargerTransactionLevelNone       thrpt    3      0.102 ±    0.072  ops/s

@hmottestad hmottestad force-pushed the GH-5520-WAL-for-ValueStore branch 3 times, most recently from 26ee78a to 79fee1d Compare November 15, 2025 08:35
@hmottestad hmottestad marked this pull request as ready for review November 15, 2025 08:35
@hmottestad hmottestad force-pushed the GH-5520-WAL-for-ValueStore branch from 79fee1d to 6a046fe Compare November 16, 2025 19:03
hmottestad
hmottestad commented Nov 18, 2025
View reviewed changes
.github/workflows/pr-verify.yml
- name: Build
run: mvn --quiet clean && mvn -B --quiet -T 2C install -Pquick
- name: Test
run: mvn -B test -DskipITs -P-formatting -Dmaven.javadoc.skip -Djapicmp.skip -Denforcer.skip -Danimal.sniffer.skip
Copy link
Contributor Author

@hmottestad hmottestad Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this and the other changes here can probably be reverted

hmottestad
hmottestad commented Nov 18, 2025
View reviewed changes
core/common/iterator/src/main/java/org/eclipse/rdf4j/common/iteration/QueueIteration.java
@Override
public void handleClose() {
done.set(true);
do {
Copy link
Contributor Author

@hmottestad hmottestad Nov 18, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

revert this, since the arrayblockingqueue signals when clear() is called

@hmottestad hmottestad force-pushed the GH-5520-WAL-for-ValueStore branch from 6a046fe to 2c233cb Compare November 19, 2025 14:06
@hmottestad hmottestad force-pushed the GH-5520-WAL-for-ValueStore branch from 2c233cb to 2e5717f Compare November 19, 2025 14:14
@hmottestad hmottestad merged commit 3a22f57 into develop Nov 19, 2025
18 of 20 checks passed
@hmottestad hmottestad deleted the GH-5520-WAL-for-ValueStore branch November 19, 2025 16:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Add a WAL to the ValueStore to allow us to recover values that are corrupt

2 participants

@hmottestad