Forward sync columns by root #7946

pawanjay176 · 2025-08-27T02:09:19Z

Issue Addressed

N/A

The Problem

Our current strategy of syncing blocks + columns by range works roughly as follows for each batch:

Find a peer from the current SyncingChain to fetch blocks from and send a BlocksByRange request
Find peer(s) from the global peer pool that should have columns for the same batch based on their status message and the columns they are supposed to custody , and send them a by DataColumnsByRange request at the same time
Once we get responses for all blocks + columns components, try to couple them by checking that the block_root and the kzg_commitment matches. If the coupling failed, try to re-request the failed columns.
Send them for processing and try to progress the chain

This strategy works decently well when the chain is finalizing as most of our peers are on the same chain. However, in times of non-finality we need to potentially sync multiple head chains.
This leads to issues with our current approach because the block peer and the data column peers might have a different view of the canonical chain due to multiple heads. So when we use the above approach, it is possible that the block peer returns us a batch of blocks for chain A while some or all data column peers send us the batch of data columns for a different chain B. Different data column peers might also be following different chains.

We initially tried to get around this problem by selecting column peers only from within the current SyncingChain. Each SyncingChain represents a head_root that we are trying to sync to and we group peers based on same head_root. That way, we know for sure that the block and column peers are on the same chain. This works in theory, but in practice, during long periods of non-finality, we tend to create multiple head chains based on the head_root and split the global peerset. Pre-fulu, this isn't a big deal since all peers are supposed to have all the blob data.
But splitting peers with peerdas is a big challenge due to not all peers having the full data available. There are supernodes, but during bad network conditions, supernodes would be getting way too many requests and not even have any incoming peer slots. As we saw on fusaka devnets, this strategy leads to sync getting stalled and not progressing.

Proposed Changes

1. Use `DataColumnsByRoot` instead of `DataColumnsByRange` to fetch columns for forward sync

This is the main change. The new strategy would go as follows:

Find a peer from the current SyncingChain to fetch blocks from and send a BlocksByRange request
Hold off on requesting for columns until we receive the response for the above blocks request
Once we get the blocks response, extract all the block_roots and trigger a DataColumnsByRoot request for every block in the response that has any blobs based on the expected_kzg_commitments field.
Since we request by root, we know what we are expecting in the response. The column peer's view of the canonical chain might be chain A, but if we ask for chain B and they have chain B in their fork choice, they can still serve us what we need.
We couple the block + column responses and send them for processing as before.

(4) kinda assumes that most synced/advanced peers would have different chains in their fork choice to be able to serve specific by root requests. My hunch is that this is true, but we should validate this in a devnet 4 like chain split scenario.

Note: that we currently use this by root strategy only for forward sync, not for backfill. Backfill has to deal with only a single canonical chain so byrange requests should work well there.

2. ResponsiblePeers to attribute peer fault correctly

Adds the ResponsiblePeers struct which stores the block and the column peers that we made the download requests to.
For most of our peer attributable errors, the processing error indicates whether the block peer was at fault or if the column peer was at fault.

We now communicate this information back to sync and downscore specific peers based on the fault type. This imo, is an improvement over current unstable where most of the time, we attribute fault to the peer that "completed" the request by being the last peer to respond.
Due to this ambiguity in fault attribution, we weren't downscoring pretty serious processing errors like InvalidKzgProofs, InvalidExecutionPayload etc. I think this PR attributes the errors to the right peers. Reviewers please check that this claim is actually true.

3. Make `AwaitingDownload` an allowable in-between state

Note: This has been extracted to its own PR here and merged #7984
Prior to peerdas, a batch should never have been in AwaitingDownload state because we immediataly try to move from AwaitingDownload to Downloading state by sending batches. This was always possible as long as we had peers in the SyncingChain in the pre-peerdas world.

However, this is no longer the case as a batch can be stuck waiting in AwaitingDownload state if we have no peers to request the columns from. This PR makes AwaitingDownload to be an allowable in between state. If a batch is found to be in this state, then we attempt to send the batch instead of erroring like before.
Note to reviewer: We need to make sure that this doesn't lead to a bunch of batches stuck in AwaitingDownload state if the chain can be progressed.

N/A Extracts (3) from #7946. Prior to peerdas, a batch should never have been in `AwaitingDownload` state because we immediataly try to move from `AwaitingDownload` to `Downloading` state by sending batches. This was always possible as long as we had peers in the `SyncingChain` in the pre-peerdas world. However, this is no longer the case as a batch can be stuck waiting in `AwaitingDownload` state if we have no peers to request the columns from. This PR makes `AwaitingDownload` to be an allowable in between state. If a batch is found to be in this state, then we attempt to send the batch instead of erroring like before. Note to reviewer: We need to make sure that this doesn't lead to a bunch of batches stuck in `AwaitingDownload` state if the chain can be progressed. Backfill already retries all batches in AwaitingDownload state so we just need to make `AwaitingDownload` a valid state during processing and validation. This PR explicitly adds the same logic for forward sync to download batches stuck in `AwaitingDownload`. Apart from that, we also force download of the `processing_target` when sync stops progressing. This is required in cases where `self.batches` has > `BATCH_BUFFER_SIZE` batches that are waiting to get processed but the `processing_batch` has repeatedly failed at download/processing stage. This leads to sync getting stuck and never recovering.

mergify · 2025-09-05T02:02:54Z

Some required checks have failed. Could you please take a look @pawanjay176? 🙏

beacon_node/network/src/sync/backfill_sync/mod.rs

beacon_node/network/src/sync/range_sync/chain.rs

beacon_node/network/src/sync/block_sidecar_coupling.rs

jimmygchen

Hi @pawanjay176
I haven't managed to go through all the logic, but I've added comments for the findings I have so far. Let me know what you think. I'll continue next week.

jimmygchen · 2025-09-05T06:20:10Z

beacon_node/network/src/sync/network_context.rs

+                .peers
+                .read()
+                .good_custody_subnet_peer_range_sync(data_column, batch_epoch)
+                .next()


This always picks the first matching peer, would it be possible for the same peer to keep getting selected for the same column across different batches?

Its a hashmap so every iteration in the peerdb would return a different order with new peers getting added and old peers getting removed I think.

This program prints the same result on every iteration BUT different across runs - it uses a randomised hasher but the iteration order is deterministic, so I'm not sure if we can't rely on this:

#[test] fn main() { let mut a = HashMap::new(); a.insert("a", 1); a.insert("z", 2); a.insert("b", 3); a.insert("y", 4); a.insert("c", 5); a.insert("x", 6); let get = || a.iter().filter(|(key,&val)| val % 2 == 0).map(|(key,val)| key); for i in 0..100 { println!("{:?}", get().next()); } }

Even if it's really random, i think callers should not rely on the implementation details of peer_db , and either (1) perform the random logic on the caller site (2) encapsulate the logic in a get_next_good_custody_subnet_peer

This way callers of the API don't make assumptions about the internal data structure of the DB, and prevents us from creating unexpected bugs, i.e. we don't break things if we change the internal data structure

Good call, i'll choose randomly at the call site too

Fixed in 04398ad using .choose()

beacon_node/network/src/sync/block_sidecar_coupling.rs

jimmygchen · 2025-09-05T06:31:11Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+                Some(resp)
+            }
+            // Reuse same logic that we use for coupling data columns for now.
+            // todo(pawan): we should never get a coupling error here, so simplify this


What do you mean here?

I meant that since we are requesting by root and also verifying the inclusion proof when adding a data column, by the time we reach here we should have caught all errors.
If we get a valid response for the data columns (no VerifyError), then there shouldn't be any issues with coupling

jimmygchen · 2025-09-05T06:33:58Z

beacon_node/network/src/sync/range_sync/batch.rs

+/// This is used for penalizing in case of invalid batches.
+#[derive(Debug, Clone)]
+pub struct ResponsiblePeers {
+    pub block_blob: PeerId,


do you mean block_peer

maybe block_blob_peer? the peer we request the block and blobs from are the same

sounds good

What about block_and_blobs? Or just block_peer and a doc explaining that it serves both.

beacon_node/network/src/sync/range_sync/batch.rs

beacon_node/network/src/sync/network_context.rs

dapplion

Why not use the same strategy for backfill sync too? Would simplify the code. Do you believe it would make backfill sync much slower?

dapplion · 2025-09-09T21:48:51Z

beacon_node/network/src/sync/manager.rs

+                }
+            }
+            DataColumnsByRootRequester::RangeSync { parent } => {
+                if let Some(resp) = self.network.on_data_columns_by_root_range_response(


Is there a reason to repeat the on_data_columns_by_root_range_response call inside the match? You can do:

if let Some(_) = on_data_columns_by_root_range_response { match req_id.requester { .. } }

dapplion · 2025-09-09T21:49:45Z

beacon_node/network/src/sync/manager.rs

                                range_request_id.id,
                                blocks,
+                                responsible_peers,


Should blocks_by_range_response have the same order of arguments as on_block_response?

dapplion · 2025-09-09T22:01:17Z

beacon_node/network/src/sync/network_context/requests/data_columns_by_root.rs

+    }
+}
+
+impl<E: EthSpec> ActiveRequestItems for DataColumnsByRootRangeRequestItems<E> {


You can de-duplicate this code by making DataColumnsByRootRequestItems take a vec of block roots

dapplion · 2025-09-09T22:02:09Z

beacon_node/network/src/sync/range_sync/batch.rs

+/// This is used for penalizing in case of invalid batches.
+#[derive(Debug, Clone)]
+pub struct ResponsiblePeers {
+    pub block_blob: PeerId,


What about block_and_blobs? Or just block_peer and a doc explaining that it serves both.

dapplion · 2025-09-09T22:04:50Z

beacon_node/network/src/sync/range_sync/batch.rs

+///
+/// This is used for penalizing in case of invalid batches.
+#[derive(Debug, Clone)]
+pub struct ResponsiblePeers {


They could be very irresponsible if they serve bad data :)

What about SourcePeers or ProviderPeers or ServingPeers? Or just BatchPeers since in the context peers only do one thing and it's serving data

dapplion · 2025-09-09T22:12:54Z

beacon_node/network/src/sync/range_sync/chain.rs

@@ -1163,6 +1190,28 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
            self.send_batch(network, batch_id)?;
        }

+        // Force requesting the `processing_batch` to progress sync if required
+        if !self.batches.contains_key(&self.processing_target) {
+            debug!(?self.processing_target,"Forcing requesting processing_target to progress sync");


Suggested change

debug!(?self.processing_target,"Forcing requesting processing_target to progress sync");

debug!(?self.processing_target, "Forcing requesting processing_target to progress sync");

dapplion · 2025-09-09T22:15:04Z

beacon_node/network/src/sync/range_sync/chain.rs

@@ -1051,6 +1077,8 @@ impl<T: BeaconChainTypes> SyncingChain<T> {
                    }
                },
            }
+        } else {
+            debug!(?self.to_be_downloaded, ?self.processing_target,"Did not get batch");


Suggested change

debug!(?self.to_be_downloaded, ?self.processing_target,"Did not get batch");

debug!(?self.to_be_downloaded, ?self.processing_target, "Did not get batch");

dapplion · 2025-09-09T22:19:08Z

beacon_node/network/src/sync/network_context.rs

+    ///
+    /// This function is used when we want to request data columns by root instead of range.
+    /// Pre-fulu, it works similar to `Self::block_components_by_range_request`.
+    pub fn block_components_by_range_request_without_components(


Duplicated code from block_components_by_range_request you can add a new ByRangeRequestType and re-use the function above

dapplion · 2025-09-09T22:29:48Z

beacon_node/network/src/sync/network_context.rs

+    ///
+    /// This function must be manually invoked at regular intervals or when a new peer
+    /// gets added.
+    pub fn retry_pending_root_range_requests(&mut self) -> Result<(), String> {


The logic here is good. To sum up, the new functionality added in this PR is:

New by_range request type to:

Request blocks first

Then with those roots request data columns by root

If any of those requests fail or mismatch retry

All this logic can be grouped and moved into its own file (the coupling service) and make network_context.rs less chaotic. I did so in my tree-sync WIP, and subjectively feels nicer https://github.com/dapplion/lighthouse/blob/47c93578c418bdac5c3beb3064ab5f675c3c177d/beacon_node/network/src/sync/network_context/block_components_by_range.rs

dapplion · 2025-09-09T22:31:24Z

beacon_node/network/src/sync/network_context.rs

+
+        // Re-insert entries that still need to be retried
+        self.pending_column_by_root_range_requests
+            .extend(entries_to_keep);


This retries should have either a retry counter or an expiry time. We should have metrics also to track the count of retries

dapplion · 2025-09-09T22:39:38Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+        column_requests: Vec<(DataColumnsByRootRequestId, Vec<ColumnIndex>)>,
+    ) -> Result<(), String> {
+        // Nothing to insert, do not initialize
+        if column_requests.is_empty() {


Should this ever happen?

dapplion · 2025-09-09T22:40:18Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+
+                Ok(())
+            }
+            _ => Err("Invalid initialization".to_string()),


Suggested change

_ => Err("Invalid initialization".to_string()),

_ => Err("Invalid state: expected DataColumnsFromRoot".to_string()),

dapplion · 2025-09-09T22:42:11Z

beacon_node/network/src/sync/block_sidecar_coupling.rs

+    /// Note: this variant starts out in an uninitialized state because we typically make
+    /// the column requests by root only **after** we have fetched the corresponding blocks.
+    /// We can initialize this variant only after the columns requests have been made.
+    DataColumnsFromRoot {


Do we really need a new variant for it?

pawanjay176 added 20 commits August 14, 2025 09:07

Penalize if invalid EL block

490b627

Priorotize status v2

836f9c6

Increase columns_by_root quota

156449c

Reduce backfill buffer size

6bd8944

Without retries

9455153

Add a function to retry column requests that could not be made

5337e46

Small fixes

ca9cfd5

Try to avoid chains failing for rpc errors

68cce37

Fix bug in initialization code

6da924b

Also penalize all batch peers for availability check errors

1a0df30

Avoid root requests for backfill sync

17c4e34

Implement responsible peer tracking

fdce537

Request columns from global peer pool

4540195

Random logs

521778b

Merge branch 'unstable' into blocks-then-columns

da27441

Handle 0 blobs per epoch case

52762b9

Merge branch 'unstable' into blocks-then-columns

7c214f5

Merge branch 'unstable' into blocks-then-columns

90d319f

Remove debug statements

27d0b36

Add docs

a97cf88

jimmygchen added syncing v8.0.0-rc.0 Q3 2025 release for Fusaka on Holesky labels Aug 27, 2025

pawanjay176 added 7 commits August 27, 2025 14:26

Fix bug with partial column responses before all column requests sent

05adb71

Remove more debug logs

b4bc7fe

Merge branch 'unstable' into blocks-then-columns

8386bd9

AwaitingValidation state only needs block peer

7331323

Revise error tolerance

da1aaba

Merge branch 'unstable' into blocks-then-columns

8e1337d

Merge branch 'unstable' into blocks-then-columns

19b0a5c

jimmygchen self-requested a review August 29, 2025 08:35

jimmygchen requested a review from dapplion August 29, 2025 08:35

pawanjay176 added 5 commits August 29, 2025 16:18

Force requests if batch buffer is full under certain conditions

b07bc6d

Add logs to debug stuck range sync

4f60e86

Force processing_target request

7a6d0d9

Attempt sending awaitingDownload batches when restarting sync

8458df6

Cleanup SyncingChain

29c2f83

pawanjay176 mentioned this pull request Sep 3, 2025

Allow AwaitingDownload to be a valid in-between state #7984

Merged

pawanjay176 added 2 commits September 4, 2025 18:08

Merge branch 'unstable' into blocks-then-columns

7e91eeb

Tests compile

e0d8f04

pawanjay176 force-pushed the blocks-then-columns branch from c81ce28 to e0d8f04 Compare September 5, 2025 01:13

pawanjay176 marked this pull request as ready for review September 5, 2025 01:13

pawanjay176 requested a review from jxs as a code owner September 5, 2025 01:13

pawanjay176 added the ready-for-review The code is ready for review label Sep 5, 2025

mergify bot added waiting-on-author The reviewer has suggested changes and awaits thier implementation. and removed ready-for-review The code is ready for review labels Sep 5, 2025

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/backfill_sync/mod.rs Outdated Show resolved Hide resolved

jimmygchen mentioned this pull request Sep 5, 2025

Sync fixes for fusaka-devnet-4 #7876

Closed

5 tasks

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/range_sync/chain.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Outdated Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Show resolved Hide resolved

jimmygchen reviewed Sep 5, 2025

View reviewed changes

beacon_node/network/src/sync/block_sidecar_coupling.rs Outdated Show resolved Hide resolved

jimmygchen requested changes Sep 5, 2025

View reviewed changes

pawanjay176 added 4 commits September 5, 2025 12:48

Fix some issues from review

6a2a33d

More renamings

e259ecd

Merge branch 'unstable' into blocks-then-columns

4f62a9c

Fix some more issues from review

04398ad

dapplion reviewed Sep 9, 2025

View reviewed changes

	debug!(?self.processing_target,"Forcing requesting processing_target to progress sync");
	debug!(?self.processing_target, "Forcing requesting processing_target to progress sync");

	debug!(?self.to_be_downloaded, ?self.processing_target,"Did not get batch");
	debug!(?self.to_be_downloaded, ?self.processing_target, "Did not get batch");

	_ => Err("Invalid initialization".to_string()),
	_ => Err("Invalid state: expected DataColumnsFromRoot".to_string()),

Forward sync columns by root #7946

Are you sure you want to change the base?

Forward sync columns by root #7946

Conversation

pawanjay176 commented Aug 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Issue Addressed

The Problem

Proposed Changes

1. Use DataColumnsByRoot instead of DataColumnsByRange to fetch columns for forward sync

2. ResponsiblePeers to attribute peer fault correctly

3. Make AwaitingDownload an allowable in-between state

Uh oh!

mergify bot commented Sep 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmygchen left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jimmygchen Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimmygchen Sep 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dapplion left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pawanjay176 commented Aug 27, 2025 •

edited

Loading

1. Use `DataColumnsByRoot` instead of `DataColumnsByRange` to fetch columns for forward sync

3. Make `AwaitingDownload` an allowable in-between state

jimmygchen Sep 8, 2025 •

edited

Loading

jimmygchen Sep 5, 2025 •

edited

Loading