feat: implement paginated search processing for large time ranges #79
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Replace memory-intensive bulk item loading with efficient pagination approach: **Core Changes:** - Add `build_broker()` method to SearchFilters for reusable broker configuration - Replace single `to_broker_items()` call with page-by-page processing (1000 items per page) - Process pages sequentially while maintaining parallel file processing within pages **Progress Display Improvements:** - Replace static progress bar with dynamic spinner showing real-time file processing - Display: "Processed X files, found Y messages | Page N (files, total) @ timestamp" - Show chronological progress through time ranges with first item timestamps **Database Safety:** - Implement per-file SQLite commits using WriterMessage enum with FileComplete signals - Fix SQLite PRAGMA statements to use proper `pragma_update()` method - Reduce potential data loss from 10k+ messages to single file scope **Benefits:** - Constant memory usage regardless of query time range length - Faster processing start (no wait for complete item enumeration) - Better user feedback with real-time progress and time progression - Enhanced data safety with per-file database commits - Scalable to multi-day/week queries without memory constraints **Testing:** - Add pagination logic tests with small page sizes - Verify broker configuration and filtering works correctly - Maintain backward compatibility for existing small queries
Replace complex progress display with simplified real-time tracking: **Simplified Architecture:** - Replace 4 channels (sender, pb_sender, page_sender, status_sender) with 2 channels - Add structured ProgressUpdate enum with FileComplete and PageStarted variants - Eliminate complex string parsing with clean data structures **Real-time Progress Display:** - Success/failure counts update immediately as each file completes processing - Format: "Processed X files, found Y messages | Page N (succeeded: A, failed: B) @ timestamp" - Remove complex timeout-based polling with simple blocking receiver loop **Code Quality Improvements:** - Eliminate 20+ lines of fragile string manipulation and reconstruction - Remove race conditions from multiple channel coordination - Reduce synchronization overhead by 75% (4 channels -> 2 channels) - Simplify progress thread from complex try_recv() polling to clean for-loop **User Experience:** - Clean progress display without log spam during progress mode - Real-time feedback showing processing success rates - Immediate visual feedback as individual files complete **Performance Benefits:** - Reduced CPU overhead from eliminated string parsing - Lower memory allocations from structured data vs string reconstruction - Improved thread coordination without busy-waiting patterns
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Implements efficient paginated search processing to handle large time ranges (days/weeks) without memory issues, with real-time progress tracking showing success/failure counts.
Key Features
Processed X files, found Y messages | Page N (succeeded: A, failed: B) @ timestampTechnical Improvements
ProgressUpdateenum with structured dataBenefits
Example Output
Resolves issue #76 with large time range queries that previously caused long delay and poor user feedback.