feat: gap detection & rewind #288

sergeytimoshin · 2025-07-30T23:49:27Z

Gap Detection & Rewind: A comprehensive system has been implemented to detect sequence gaps across all state update types (V1/V2 trees, nullifications, batch events, etc.). When a gap is detected, the indexer now automatically triggers a "rewind" command to the block fetcher, which repositions the stream to re-fetch the missing blocks, ensuring no data is missed.
Tree Filtering: The indexer can now be started with a --tree argument. When provided, it will filter and process only the blocks and transactions relevant to the specified Merkle tree. This speeds up indexing, when backfilling a specific tree.
Snapshot Analysis Tool: A new photon-analyze-snapshot tool added to provide info about snapshot files.

ananas-block · 2025-07-31T00:04:43Z

tests/integration_tests/mock_tests.rs

                    seq: *seq as u64,
+                    signature: Default::default(),
                },
            );


why not unwrap ?

src/ingester/detect_gaps.rs

ananas-block · 2025-07-31T01:10:25Z

src/ingester/detect_gaps.rs

+                                crate::ingester::parser::tree_info::StateV2SeqWithContext::default()
+                            }
+                        } else {
+                            crate::ingester::parser::tree_info::StateV2SeqWithContext::default()


this looks suspicous

ananas-block · 2025-07-31T01:11:04Z

src/ingester/detect_gaps.rs

+                } else {
+                    SequenceEntry {
+                        sequence: 0,
+                        slot: 0,
+                        signature: String::new(),
+                    }
+                }
+            } else {
+                SequenceEntry {
+                    sequence: 0,
+                    slot: 0,
+                    signature: String::new(),
+                }
+            };


this looks suspicous

src/ingester/detect_gaps.rs

ananas-block · 2025-07-31T01:16:25Z

src/ingester/detect_gaps.rs

+                if let Some(entry) = &seq_context.output_queue_entry {
+                    (entry.sequence, Some(entry.clone()))
+                } else {
+                    (0, None)


is this correct?

ananas-block · 2025-07-31T01:16:29Z

src/ingester/detect_gaps.rs

+                if let Some(entry) = &seq_context.batch_event_entry {
+                    (entry.sequence, Some(entry.clone()))
+                } else {
+                    (0, None)


is this correct?

ananas-block · 2025-07-31T01:16:32Z

src/ingester/detect_gaps.rs

+                if let Some(entry) = &seq_context.batch_event_entry {
+                    (entry.sequence, Some(entry.clone()))
+                } else {
+                    (0, None)


is this correct?

ananas-block · 2025-07-31T01:17:36Z

src/ingester/detect_gaps.rs

+    if unpacked_start_seq > 0
+        && unpacked_start_seq != u64::MAX


hm we handle both 0 and u64::MAX here, but it's not clean we should refactor it

added more comments + moved to ingester/gap module

src/ingester/mod.rs

src/ingester/persist/persisted_indexed_merkle_tree.rs

src/ingester/gap/sequences.rs

ananas-block · 2025-07-31T17:10:31Z

apart from those nitpicks lgtm

Co-authored-by: Copilot <[email protected]>

refactor: add a dedicated error variant for gap detection

The new `--disable-gap-detection` flag allows users to opt out of the sequence gap detection and rewinding behavior, simplifying the indexing process in cases where it's not needed.

- Adds sequential index validation for both operations - Checks for already processed batches during re-indexing - Improves handling of address batch indices and queue state - Retains nullifier queue indices for re-indexing support - Makes temporary file creation more robust with fallback