feat(upload-client): run commp/piece calculation while blob is uploading #168
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
instead of waiting for blob upload to finish, run commp calculation during the upload process, which will likely take just as long, meaning there's less extra time spent waiting for commp to calculate
volmedo
approved these changes
Feb 20, 2025
|
Yeah...hmm, I seem to remember the reason we didn't do this was because piece hashing blocks the main thread (unless you're using a custom piece hasher that doesn't) and that would cause requests to fail/timeout because nothing was sending the data. |
|
Valid point @alanshaw I will modify to passthrough an option on this one that we can enable for the CLI. |
travis
added a commit
that referenced
this pull request
Apr 16, 2025
this will give us nice stack traces and other conveniences in Sentry
travis
pushed a commit
that referenced
this pull request
Apr 16, 2025
🤖 I have created a release *beep* *boop* --- ## [1.17.1](storacha/console@w3console-v1.17.0...w3console-v1.17.1) (2024-12-19) ### Bug Fixes * add sentry auth token to enable sourcemap upload ([#168](storacha/console#168)) ([a48c8c4](storacha/console@a48c8c4)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please). Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
travis
added a commit
that referenced
this pull request
Apr 16, 2025
🤖 I have created a release *beep* *boop* --- ## [1.18.0](console-v1.17.6...console-v1.18.0) (2025-04-16) ### Features * Actually put error in console, as promised ([#131](#131)) ([2f438d5](2f438d5)) * add a PNG version of our logo ([#153](#153)) ([52a7854](52a7854)) * add old web3 storage migrator ([#129](#129)) ([8cbd2bb](8cbd2bb)) * add Sentry error tracking ([#165](#165)) ([5188e07](5188e07)) * change email ([b624f1d](b624f1d)) * **copy:** Add missing comma ([#115](#115)) ([6e231eb](6e231eb)) * Improved layout styling, and other tales ([#114](#114)) ([5484d36](5484d36)) * more styles ([a9f0172](a9f0172)) * point staging env referral link at staging marketing site ([0073963](0073963)) * prototype of referral system in console ([#142](#142)) ([d2b284b](d2b284b)) * referrals tweaks ([#155](#155)) ([32cc78f](32cc78f)) * **repo:** merge main from https://github.com/storacha/console ([152954e](152954e)) * share spaces via email ([#123](#123)) ([b0dd61a](b0dd61a)) * styles for upload page ([9cdac7c](9cdac7c)) * trigger deploy ([a5ae681](a5ae681)) * update TOS links to point to the Storacha docs ([#148](#148)) ([7d5ad13](7d5ad13)) * Use Blob protocol ([#132](#132)) ([0fd47ba](0fd47ba)) * Use latest `w3ui` to support reloading spaces ([#138](#138)) ([5a886bf](5a886bf)) * wording ([032218e](032218e)) ### Fixes * a little package.json cleanup ([13859fc](13859fc)) * add missing receiptsEndpoint to W3UIProvider ([#144](#144)) ([e8ca3d3](e8ca3d3)) * add more deps to the catalog and pin eslint typescript plugin ([e00af97](e00af97)) * add sentry auth token to enable sourcemap upload ([#168](#168)) ([bec52c9](bec52c9)) * apostrophe ([ee51ff9](ee51ff9)) * capture some errors we were logging to console ([#188](#188)) ([673043a](673043a)) * catalog more dependencies ([cc52f81](cc52f81)) * check ref ([b1c985f](b1c985f)) * **console:** pnpm run dev + nx configs(sentry,tailwind) ([ad5db7e](ad5db7e)) * cors preflight issue in production ([#178](#178)) ([583abb3](583abb3)) * create space ([#173](#173)) ([7ac4478](7ac4478)) * create space using the latest w3up-client version ([#171](#171)) ([5c67bdb](5c67bdb)) * **createSpace:** using the latest w3ui with w3up-client ([#176](#176)) ([fdd5895](fdd5895)) * deploy-storacha.yml needs to use the new main branch ([#122](#122)) ([3240385](3240385)) * deploy.yml needs to use the old-main branch ([#120](#120)) ([6252d89](6252d89)) * fix linting ([0c5f06b](0c5f06b)) * gateway authorization based on env ([#174](#174)) ([f446ab9](f446ab9)) * Humanode config tweaks ([b3debc4](b3debc4)) * import space preview ([#127](#127)) ([d935f5f](d935f5f)) * init page loading right after button click + stop on errors ([220df2d](220df2d)) * is this enough? ([ac552ac](ac552ac)) * lock down next and react versions ([4d316af](4d316af)) * make it possible to skip the plan gate after Stripe checkout ([#162](#162)) ([e50d81f](e50d81f)) * **prod:** gateway host ([#180](#180)) ([14c8c38](14c8c38)) * quotes ([9fb1aba](9fb1aba)) * remove console github configs ([b5a2307](b5a2307)) * remove console github configs ([#227](#227)) ([71db228](71db228)) * remove referrals feature flag ([#161](#161)) ([9c774dd](9c774dd)) * remove sentry test page ([b442306](b442306)) * remove sentry test page ([#231](#231)) ([f4071dc](f4071dc)) * Revert "feat: Use Blob protocol" ([#134](#134)) ([9611be0](9611be0)) * Revert "fix: Revert "feat: Use Blob protocol"" (😩) ([#136](#136)) ([2ec041b](2ec041b)) * set ipfs gateway url based on env var ([#177](#177)) ([49727f7](49727f7)) * share space via ucan file ([#126](#126)) ([2b7b623](2b7b623)) * **space-creator:** check provision result before proceeding ([#119](#119)) ([92e087a](92e087a)) * **staging:** gateway host ([794a856](794a856)) * trigger a console release ([b9862a1](b9862a1)) * trigger a console release ([#228](#228)) ([d9677d8](d9677d8)) * tweak plan gate query param ([#163](#163)) ([2346d15](2346d15)) * typo on space page ([#187](#187)) ([12e2b02](12e2b02)) * update packages to use catalog dependencies ([9191594](9191594)) * use correct branch name ([374fb0c](374fb0c)) * use main branch ([ea0a170](ea0a170)) * use the correct referral URL ([#149](#149)) ([728ae7e](728ae7e)) * use the same Sentry project for different envs ([#167](#167)) ([1bf2d13](1bf2d13)) * Use Ucanto 10 across the board ([#140](#140)) ([b38477a](b38477a)) * using the correct gateway host ([#183](#183)) ([83e0582](83e0582)) ### Other Changes * add storacha deploy action ([617b096](617b096)) * import console ([#139](#139)) ([59ebc4e](59ebc4e)) * **main:** release client 1.0.6 ([27cb383](27cb383)) * **main:** release client 1.0.6 ([#104](#104)) ([07f27a2](07f27a2)) * **main:** release upload-api 1.1.8 ([aec53e7](aec53e7)) * **main:** release upload-api 1.1.8 ([#103](#103)) ([e71494a](e71494a)) * **main:** release w3console 1.10.0 ([#125](#125)) ([d7484f7](d7484f7)) * **main:** release w3console 1.11.0 ([#128](#128)) ([4a9cd39](4a9cd39)) * **main:** release w3console 1.12.0 ([#130](#130)) ([c1b894d](c1b894d)) * **main:** release w3console 1.13.0 ([#133](#133)) ([a5cdbf5](a5cdbf5)) * **main:** release w3console 1.13.1 ([#135](#135)) ([0cf946d](0cf946d)) * **main:** release w3console 1.14.0 ([#137](#137)) ([928cd1e](928cd1e)) * **main:** release w3console 1.15.0 ([#139](#139)) ([18c33d3](18c33d3)) * **main:** release w3console 1.15.1 ([#141](#141)) ([d82d22e](d82d22e)) * **main:** release w3console 1.15.2 ([#145](#145)) ([810224b](810224b)) * **main:** release w3console 1.16.0 ([#146](#146)) ([2a7dcaa](2a7dcaa)) * **main:** release w3console 1.16.1 ([#160](#160)) ([145b288](145b288)) * **main:** release w3console 1.16.2 ([#164](#164)) ([c3ea3be](c3ea3be)) * **main:** release w3console 1.17.0 ([#166](#166)) ([0ea9a86](0ea9a86)) * **main:** release w3console 1.17.1 ([#169](#169)) ([7a77c70](7a77c70)) * **main:** release w3console 1.17.2 ([#172](#172)) ([a118e11](a118e11)) * **main:** release w3console 1.17.3 ([#179](#179)) ([7c1948e](7c1948e)) * **main:** release w3console 1.17.4 ([#181](#181)) ([217cd1e](217cd1e)) * **main:** release w3console 1.17.5 ([#182](#182)) ([9db003e](9db003e)) * **main:** release w3console 1.17.6 ([#185](#185)) ([152e230](152e230)) * nx sync ([5933677](5933677)) * Set `packageManager` ([#117](#117)) ([a8c7c93](a8c7c93)) * trigger deployment ([8051d81](8051d81)) * update copy ([#159](#159)) ([60250f1](60250f1)) * update deploy name ([184fe57](184fe57)) --- This PR was generated with [Release Please](https://github.com/googleapis/release-please). See [documentation](https://github.com/googleapis/release-please#release-please).
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What
CommP calculation on a desktop machine seems to take non-trivial time for 128mb, at least with current code. It makes sense to run it while we're waiting for the upload data transfer to finish, since those are likely to be limited by different factors (network bandwidth vs CPU)