Platform blocks are racing despite being empty

[kxcd]

The expected time to generate a new block in the abscence of any transaction is about 3 minutes, recently, we've noticed that blocks are coming within 2 seconds of each other despite there being no transactions made, please investigate.

https://platform-explorer.com/blocks

[shumkov]

Thank you for the report. We will look at this.

[lklimek]

From Tenderdash perspective, this is expected behavior.

We have 100 validators, and each of them have a mempool. If some invalid transaction gets into the mempool, it will signal to Tenderdash that there is a pending transaction. Note that transaction can become invalid over time, for example if the user executes another, conflicting transaction.

When a node that should be a proposer has at least 1 transaction in mempool, it will not wait ~3 minutes, but try to execute immediately.

On a proposer (1 node out of 100), Drive will receive this "invalid" transaction and will decide it cannot be added to the block. This one node will remove it from the mempool, but it will stay on other nodes. So when next validator becomes a proposer, it will see the tx in mempool, decide tx cannot be included in the block, and generate new block without waiting.

So worst case scenario is that each validator will propose one block without waiting for response.

To fix this issue, we have a mechanism called "re-check tx", which continously goes through the mempool and checks each pending tx (sends CheckTx request to Drive). It means this tx passed the CheckTx check, but finally (when proposing block) Drive decided it's not correct. To debug that issue, we need to catch this invalid tx and see what really happened there.

Long story short, looks like a task for Drive to improve Check TX logic. To debug, we need to dump this problematic tx.