Custom da merge customda daprovider interface #3712
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This is part of a series of changes for the Custom DA project. Summary: - Added a new daprovider.Validator interface with methods for generating proofs for Custom DA systems. - Added a reference implementation of a Custom DA provider. - Added a daprovider factory that supports anytrust and referenceda modes. In a follow-up PR the nitro node startup sequence will be modified to use this. Currently only the separate daprovider server uses this. - Replaced the AnyTrust-specific (aka das) provider server with a unified provider server that works with anytrust or referenceda modes. - Extended the DA Client with new RPC methods for generating proofs. Notes: The separate provider server executable is a thin RPC server wrapper around the anytrust and referenceda implementations. The idea is that people wanting to integrate their own DA system can use this as a guide for how to implement their own RPC service that lives outside the nitro codebase; we won't be including support for any additional DA implementations in provider server executable that we distribute. For legacy AnyTrust deployments we will most likely continue to have nitro spawn the daprovider server in-process to avoid needing to run an extra service, but by channeling everything through the JSON-RPC interface it reduces surface area of what we have to support. The Reference DA (referenceda) implementation is a minimal working example of how one could implement a daprovider, including support for validating the certificate against trusted signers. It uses an in-memory data storage backend. In various places there is commented out code related to the osp contract bindings that haven't yet been committed to a nitro-contracts branch that we want to use in nitro master. This PR shouldn't change any existing functionality or behavior, except the daprovider executable (which isn't used in production) has some new configuration options: ``` --mode must be "anytrust" or "referenceda" --provider-server.* server config eg address, port, etc --anytrust.* was previosly called das-server --referenceda.* referenceda specific options ``` As much as possible we will try to rename references to "das" to "anytrust". When we launched Anytrust, we only had one offchain data availability mode so we just called it "das" at the time. This PR doesn't include new test code, but testing was done with the end-to-end block validator and challenge system tests on the custom-da branch.
We will add DACertificateMessageHeaderFlag back to KnownHeaderBits in a future PR; removing it for now avoids changing the replay binary unnecessarily when merging in this PR.
daprovider.Readers are now registered with their respective header bytes at initialization. For external DA providers where the RPC client is used this becomes single call at init to daprovider_getSupportedHeaderBytes. Where previously in the inbox and block validator components we had loops over each provider, checking each provider if they handle that type of message, we now just check if we have a provider registered for that message. This means less RPC calls and makes the interface that providers need to implement a bit simpler, with room for taking it out completely for CustomDA if we can detect from other config that the external provider is expected to be a CustomDA provider.
We moved the contents of dasserver to provider_server on the custom-da branch and I accidentally left dasserver in place on this branch when moving things across. I don't want to bring over the changes to the daprovider construction logic in node.go yet so I just made the das_migration.go file which we'll delete later.
In normal execution we only care about the payload, and in validation we only care about the preimages (we will reconstruct the payload using the preimages). This change separates these concerns and simplifies the signatures of these methods.
For DAS and ReferenceDA we almost always want to validate the certificate. The only time we don't is in the replay binary when we want to panic instead if it's invalid on the first read of the message, otherwise we want to ignore it and assume it's already valid. So this means that it doesn't make sense for validateSeqMsg to be part of the reader API at all, which is great because it means we can simplify the API. We can pass into the reader at construction time how to handle this because in replay it always constructs a new (fake, non-network calling) reader for each invocation.
The DbStorageService has been deprecated for over a year, with a warning printed for any remaining users during that time informing them of how to migrate and that if they didn't, continuing to try to use it would be a fatal error in future. The future is here. This also lets us remove badgerdb is a dependency.
Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
…block snapshot (#3619) Co-authored-by: Raul Jordan <[email protected]> Co-authored-by: Joshua Colvin <[email protected]> Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
…tools (#3702) * fix inconsistent setup-go versions across workflows * update golangci-lint to v2.5.0 (latest stable release) * Update _fast.yml * upgrade Go version from 1.24.5 to 1.25 * Update Dockerfile * Update Dockerfile --------- Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
* fix(events): use stable subscription ids and remove by id * Update producer.go * Update producer.go * Update producer_test.go * Update producer.go * Update producer_test.go --------- Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
…tomda-daprovider-interface Resolved conflicts from changes on customda-daprovider-interface branch, mostly keep changes from that branch except where code had been temporarily commented out.
Also update the evil_da_provider to use the new separate RecoverPayload and CollectPreimages.
…server with trusting verifier (it's usually behind jwt anyway)
Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
The RUN_URL needs to be interpolated by the github actions framework. This doesn't happen if without the double braces. Also, the action expects valid JSON input, so even more braces are applied to the payload argument. Fixes: NIT-3924
* Update sequencer.go * Update restful_server.go * Update dasRpcServer.go * Update dasRpcClient.go * Update validation_client.go * Update dasRpcClient.go * Update dasRpcServer.go * Update restful_server.go * Update validation_client.go * lint * ci fix --------- Co-authored-by: Pepper Lebeck-Jobe <[email protected]> Co-authored-by: Tristan-Wilson <[email protected]>
Co-authored-by: Tristan-Wilson <[email protected]>
…tomda-daprovider-interface
…der-interface' into pmikolajczyk/nit-3515 # Conflicts: # daprovider/server/das_migration.go
Co-authored-by: Ganesh Vanahalli <[email protected]>
Co-authored-by: Pepper Lebeck-Jobe <[email protected]> Co-authored-by: Raul Jordan <[email protected]>
* Add multi-dimensional gas metrics * Add total multigas metrics * Change multigas metrics from histograms to counters --------- Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
Use streaming protocol between DA client and DA provider server
* Instrument multi-gas in precompiles Close NIT-1557 * Bump go-ethereum * Change balance multi-gas dimension * Test for context burn, burned and gas left * Fix lint warning
* Reduce CI
* Run basic test results action
* Parse junitfile argument for gotestsum
* Add permissions: pull-requests: write
* ... in a right place. Also fail if tests fail
* Fail test intentionally
* Inspect the problem
* parse single file
* fail another test
* print why 22 tests failed
* single rerun
* run for defaults
* fix l3challenge test mode
* revert geth update
* shortening script
* run script in CI
* keep last 20 lines instead of first 2048 characters
* move codecov steps just after testing
* revert changes to ci.yml
* Try nextest
* Upload Rust results
* Upload go test results
* revert changes to docker
* merge results from different workflows
* needs
* line breaks
* line breaks
* rm icon
* minor fixes
* restore CI
* needs fix
* permissions:
pull-requests: write
* revert
* whitechars
* Refactor caller context creationi and remove redundant argument * fir merge conflicts
* Instrument multi-gas in precompiles Close NIT-1557 * Bump go-ethereum * Change balance multi-gas dimension * Test for context burn, burned and gas left * Fix lint warning * Merge v1.16.4 --------- Co-authored-by: Gabriel de Quadros Ligneul <[email protected]> Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
… and persistent checks (#3735) Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
* s/github.ref/github.head_ref * actually... both * add fallback value
* remove: delete unused HashPlusInt function from arbos/util * Update util.go --------- Co-authored-by: Pepper Lebeck-Jobe <[email protected]>
Co-authored-by: Joshua Colvin <[email protected]>
* Add DA proof support to daprovider interface This is part of a series of changes for the Custom DA project. Summary: - Added a new daprovider.Validator interface with methods for generating proofs for Custom DA systems. - Added a reference implementation of a Custom DA provider. - Added a daprovider factory that supports anytrust and referenceda modes. In a follow-up PR the nitro node startup sequence will be modified to use this. Currently only the separate daprovider server uses this. - Replaced the AnyTrust-specific (aka das) provider server with a unified provider server that works with anytrust or referenceda modes. - Extended the DA Client with new RPC methods for generating proofs. Notes: The separate provider server executable is a thin RPC server wrapper around the anytrust and referenceda implementations. The idea is that people wanting to integrate their own DA system can use this as a guide for how to implement their own RPC service that lives outside the nitro codebase; we won't be including support for any additional DA implementations in provider server executable that we distribute. For legacy AnyTrust deployments we will most likely continue to have nitro spawn the daprovider server in-process to avoid needing to run an extra service, but by channeling everything through the JSON-RPC interface it reduces surface area of what we have to support. The Reference DA (referenceda) implementation is a minimal working example of how one could implement a daprovider, including support for validating the certificate against trusted signers. It uses an in-memory data storage backend. In various places there is commented out code related to the osp contract bindings that haven't yet been committed to a nitro-contracts branch that we want to use in nitro master. This PR shouldn't change any existing functionality or behavior, except the daprovider executable (which isn't used in production) has some new configuration options: ``` --mode must be "anytrust" or "referenceda" --provider-server.* server config eg address, port, etc --anytrust.* was previosly called das-server --referenceda.* referenceda specific options ``` As much as possible we will try to rename references to "das" to "anytrust". When we launched Anytrust, we only had one offchain data availability mode so we just called it "das" at the time. This PR doesn't include new test code, but testing was done with the end-to-end block validator and challenge system tests on the custom-da branch. * Reduce code duplication in factory, more cert validation * Fix for interface change on master * Remove DACert flag from KnownHeaderBits for now We will add DACertificateMessageHeaderFlag back to KnownHeaderBits in a future PR; removing it for now avoids changing the replay binary unnecessarily when merging in this PR. * Remove preimageType from daprovider.Validator * Remove IsValidHeaderByte from RPC API daprovider.Readers are now registered with their respective header bytes at initialization. For external DA providers where the RPC client is used this becomes single call at init to daprovider_getSupportedHeaderBytes. Where previously in the inbox and block validator components we had loops over each provider, checking each provider if they handle that type of message, we now just check if we have a provider registered for that message. This means less RPC calls and makes the interface that providers need to implement a bit simpler, with room for taking it out completely for CustomDA if we can detect from other config that the external provider is expected to be a CustomDA provider. * Move daprovider api types to own go pkg * Use Promises with daprovider.Reader iface * dasserver migration to provider_server We moved the contents of dasserver to provider_server on the custom-da branch and I accidentally left dasserver in place on this branch when moving things across. I don't want to bring over the changes to the daprovider construction logic in node.go yet so I just made the das_migration.go file which we'll delete later. * Split rpc payload and preimage methods In normal execution we only care about the payload, and in validation we only care about the preimages (we will reconstruct the payload using the preimages). This change separates these concerns and simplifies the signatures of these methods. * Removed the validateSeqMsg param from Reader For DAS and ReferenceDA we almost always want to validate the certificate. The only time we don't is in the replay binary when we want to panic instead if it's invalid on the first read of the message, otherwise we want to ignore it and assume it's already valid. So this means that it doesn't make sense for validateSeqMsg to be part of the reader API at all, which is great because it means we can simplify the API. We can pass into the reader at construction time how to handle this because in replay it always constructs a new (fake, non-network calling) reader for each invocation. * Use Promises with Validator interface * Add missing import * Resgister DAS reader with both flag variants * Use cancelable promises * Fix reader registration for batch correctness checking * Use promise for GetSupportedHeaderBytes too
…tomda-daprovider-interface Fix conflicts related to Promises, streaming store, commented out usages of contracts.
Tristan-Wilson
deleted the
custom-da-merge-customda-daprovider-interface
branch
❌ 11 Tests Failed:
View the top 3 failed tests by shortest run time
📣 Thoughts on this report? Let Codecov know! | Powered by Codecov |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR is a merge of
customda-provider-interface(#3600) back intocustom-da(#3237), which it was originally split off from, but has changed as it's been updated to address PR comments. There were significant conflicts that have now been resolved on this branch. Assuming #3600 doesn't change too much, then we won't have to rework the merge conflict resolutions that we've already made on this branch and can just merge it tocustom-daafter first merging inmaster. Basically I'm hoping that I've been able to frontload the merge conflict resolutions before #3600 is "done".In the meantime we can target this branch instead of
custom-dafor further Custom DA project development.