Skip to content

Conversation

forsaken628
Copy link
Collaborator

@forsaken628 forsaken628 commented Sep 4, 2025

I hereby agree to the terms of the CLA available at: https://docs.databend.com/dev/policies/cla/

Summary

  • Spilling is supported on the channel between the pipeline and the page manager, allowing the pipeline to continue executing while result set consumption is blocked.
  • Introduced enable_result_set_spilling, which is turned off by default and is fully compatible with the previous behavior.
  • Introduced result_set_spilling_to_disk_bytes_limit to support spilling result set data to local disk.

Tests

  • Unit Test
  • Logic Test
  • Benchmark Test
  • No Test - Explain why

Type of change

  • Bug Fix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that could cause existing functionality not to work as expected)
  • Documentation Update
  • Refactoring
  • Performance Improvement
  • Other (please describe):

This change is Reviewable

@github-actions github-actions bot added the pr-feature this PR introduces a new feature to the codebase label Sep 4, 2025
@KKould
Copy link
Member

KKould commented Sep 10, 2025

please update the code synchronization #18709 This PR solves the CI test_private_tasks error

@forsaken628
Copy link
Collaborator Author

In this PR, the sender writes the spilled page into a file, which the receiver then restores.

However, this logic is not very reasonable because the size of the page depends on the transmission layer, which does not necessarily correspond to the spilling mode.

In the follow-up, a new spill unit called a "chunk" can be introduced. The sender writes the chunks, which then generates a page from it.
We also need to consider how to be compatible with the old PaginationConf.

@forsaken628 forsaken628 changed the title feat(query): result spill feat(query): support result set spilling Sep 11, 2025
@forsaken628 forsaken628 marked this pull request as ready for review September 11, 2025 09:05
@BohuTANG BohuTANG added the ci-cloud Build docker image for cloud test label Sep 12, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18679-0be4047-1757663905

note: this image tag is only available for internal use.

@BohuTANG
Copy link
Member

Will do some test on prod this weekend.

@BohuTANG BohuTANG added ci-cloud Build docker image for cloud test and removed ci-cloud Build docker image for cloud test labels Sep 13, 2025
Copy link
Contributor

Docker Image for PR

  • tag: pr-18679-ddaae69-1757775034

note: this image tag is only available for internal use.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
ci-cloud Build docker image for cloud test pr-feature this PR introduces a new feature to the codebase
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants