-
-
Notifications
You must be signed in to change notification settings - Fork 46
feat: mud plugin #1728
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
base: 05-21-init-mud-example
Are you sure you want to change the base?
feat: mud plugin #1728
Conversation
…-monorepo into 05-13-mud-plugin
…-monorepo into 05-13-mud-plugin
…-monorepo into 05-13-mud-plugin
…arzero/tevm-monorepo into 05-13-mud-plugin
## Description The MUD plugin before that has several flaws, and is super slow to execute optimistic updates, which makes the whole plugin not really relevant. This PR introduces multiple fixes to highly reduce the occurrence of sync inconsistencies/race conditions, some layer of caching and optimizations, and attempts to fix remaining issues. Fixes: - Before, when getting the optimistic state, for each tx ran, we would apply the logs and notify subscribers, which would cause consumers to request the state, hence run that tx again. This would be problematic as multiple txs need to be processed, since each tx update would trigger another run with all the txs so far again. The fix here is simply to run txs and apply the logs to build up the state, and notify subscribers only after we ran all the txs. - Much better separation between view and mutable state. This causes the main optimistic state getters to become sync, as we now "cache" the state. Two biggest pain points that make the whole plugin broken and unoptimized respectively, are the following: - [x] There is an architectural sync issue: we have two separate, asynchronous event streams that should be synchronized but aren't: - canonical sync stream: stash updates when canonical transactions are indexed - optimistic pool stream: txremoved events when optimistic transactions are removed These streams are inherently out of sync because: - The canonical sync happens when the indexer processes blocks - The optimistic tx removal happens when we detect the canonical tx exists - We don't have the canonical tx hash available when the sync event fires **-> For now, this one has been fixed by appending a small hex string to the end of both the canonical and optimistic txs data so we can link them back together later** - [ ] Everytime we run a call locally basically, we're clearing the fork cache for the store address, as if we didn't, it would directly retrieve storage values from that cache and never hit our getStorageAt override (that fetches from the synced MUD store instead). This is super bad design, and also causes a bunch of other storage slots to be invalidated, when these actually should stay. - [ ] We'll probably want our fork clients to rebase frequently so it considers the latest fork state when executing txs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Didn't get through entire pr but left a bunch of comments.
@@ -0,0 +1,119 @@ | |||
{ | |||
"name": "@tevm/mud", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why is this in bundler-packages instead of packages?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I thought everything inside packages
was related to the evm. Saw whatsabi
in bundler-packages
so just copied that. Maybe we can move right before merging to not break comments?
type QueuedUpdate = { type: 'optimistic'; fn: () => Promise<void> } | { type: 'canonical'; fn: () => Promise<void> } | ||
|
||
/** | ||
* Coordinates state updates to prevent race conditions between optimistic and canonical updates. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We shouldn't need this. Needing this is a smell that we are not letting MUD stay source of truth for all data and when state transitions.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes this is one of the atrocities I was mentioning, I just don't know how to avoid such compromise with the current design.
MUD sync is the source of truth, but applying the optimistic state & removing from the pending pool being async we eventually end up with race conditions when multiple MUD sync -> stash update -> remove tx from pending pool fire rapidly and both read and alter the state at the same time.
@@ -0,0 +1,17 @@ | |||
// React | |||
export const deepEqual = (obj1: any, obj2: any): boolean => { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
nit it isn't detecting cycles so this can run forever and freeze program. Generally we should avoid needing this it's a smell
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This deepEqual
is used in our react hooks. Since getting the optimistic state means internally copying the original stash state and adding optimistic records on top, we have to perform deepEqual as the state reference is never the same.
We use it in the memoize
commented below to avoid getting the records again for every render if the stash state (the optimistic one in our case) didn't actually change.
This shouldn't be in bundler-packages since it's not a bundler. extensions/* is better spot. I think I'm gonna just leave this on the branch and cherry pick stuff off of it when I circle around to looking at MUD so don' tworry about rebasing the pr or anything |
@claude Please analyze this PR to provide some insight structured as follows:
|
Claude encountered an error —— View job I'll analyze this and get back to you. |
Claude finished @0xpolarzero's task —— View job PR Analysis: MUD Plugin with Optimistic UpdatesTodo List:
1. Comprehensive SummaryThis pull request introduces Problem Solved:
Core Mechanism:
2. Highlight: Practical Integration ExamplesVanilla JS Exampleimport { createOptimisticHandler } from '@tevm/mud';
import { createMemoryClient } from '@tevm/memory-client';
import { createStash } from '@latticexyz/stash/internal';
import { http } from 'viem';
// Setup your MUD stash and client
const stash = createStash(mudConfig);
const client = createPublicClient({ transport: http('https://...') });
// Create the optimistic handler
const handler = createOptimisticHandler({
client,
storeAddress: '0x...', // Your MUD world address
stash,
loggingLevel: 'debug'
});
// Core API methods:
// Get current optimistic state (canonical + pending optimistic changes)
const optimisticState = handler.getOptimisticState();
// Get specific optimistic record with type safety
const playerRecord = handler.getOptimisticRecord({
table: PlayerTable,
key: { playerId: '0x123...' }
});
// Get multiple optimistic records
const allPlayers = handler.getOptimisticRecords({
table: PlayerTable
});
// Subscribe to optimistic state changes (fires on both optimistic and canonical updates)
const unsubscribe = handler.subscribeOptimisticState({
subscriber: (updates) => {
console.log('State updated:', updates);
// Update your UI here
}
});
// Subscribe to transaction status changes
const unsubscribeTx = handler.subscribeTx({
subscriber: (status) => {
console.log('Transaction status:', status);
// status.status can be: 'simulating' | 'optimistic' | 'confirmed' | 'reverted'
// Update loading states, show toasts, etc.
}
});
// Use the sync adapter with MUD's store-sync
const syncAdapter = handler.syncAdapter; // Pass to MUD's SyncProvider React ExampleBased on the pattern in import React from 'react';
import { OptimisticWrapperProvider, useOptimisticState } from '@tevm/mud/react';
import { useSessionClient } from '@latticexyz/entrykit/internal';
import { stash } from './mud/stash';
// Provider setup
function OptimisticProvider({ children }) {
const { data: sessionClient } = useSessionClient();
return (
<OptimisticWrapperProvider
stash={stash}
storeAddress="0x..." // Your MUD world address
client={sessionClient}
sync={{ startBlock: 0n }}
loggingLevel="debug"
>
{children}
</OptimisticWrapperProvider>
);
}
// Component using optimistic state
function PlayerComponent({ playerId }) {
// Get optimistic player data with automatic re-renders
const player = useOptimisticState(
state => getRecord({ state, table: PlayerTable, key: { playerId } }),
{ isEqual: deepEqual } // Prevents unnecessary re-renders
);
// Transaction status monitoring
const [txStatus, setTxStatus] = useState(null);
const wrapper = useOptimisticWrapper();
useEffect(() => {
if (!wrapper) return;
return wrapper.subscribeTx({
subscriber: setTxStatus
});
}, [wrapper]);
const movePlayer = async (x, y) => {
// This writeContract call is automatically intercepted
// UI updates instantly with optimistic state
await sessionClient.writeContract({
address: worldAddress,
abi: worldAbi,
functionName: 'movePlayer',
args: [playerId, x, y]
});
};
return (
<div>
<div>Player Position: {player?.x}, {player?.y}</div>
{txStatus?.status === 'simulating' && <div>Simulating...</div>}
{txStatus?.status === 'optimistic' && <div>Pending confirmation...</div>}
<button onClick={() => movePlayer(10, 20)}>
Move Player
</button>
</div>
);
} 3. Detailed Optimistic Update FlowHere's the precise step-by-step trace of the entire optimistic update process: Phase 1: Transaction Initiation
Phase 2: Optimistic Simulation
Phase 3: Optimistic State Calculation
Phase 4: UI Update Propagation
Phase 5: Storage Request Interception
Phase 6: Network Confirmation
Error Handling & Edge Cases
4. LimitationsClient Type RestrictionThe current implementation is specifically designed for MUD projects using an EntryKit SessionClient (bundler client) and will not work with other client types. The interception logic in Implementation Architecture IssuesThe implementation is a 'clunky' first version with several non-ideal practices:
Race Conditions and State ManagementThe example demonstrates that running multiple calls quickly causes the UI to flash incorrect states, highlighting probable race conditions or similar issues. Despite attempts to mitigate this with a
|
…po into 05-13-mud-plugin
Claude finished @0xpolarzero's task —— View job PR Review: MUD Plugin with Optimistic UpdatesTodo List:
Code Review SummaryI've conducted a thorough review of the MUD plugin implementation. This is an innovative and sophisticated approach to solving optimistic updates in blockchain gaming, but it has several architectural concerns and areas for improvement. 🎯 Overall AssessmentStrengths:
Concerns:
🔍 Detailed Code Quality Analysis1. Architecture & Design IssuesDirect Method Mutation ( client.writeContract = async function interceptedWriteContract(args) { This is a dangerous pattern that mutates the client object directly. This can:
Recommendation: Use a Proxy pattern instead to intercept calls without mutation. Dual Client Architecture 2. State Management & Race ConditionsState Update Coordinator ( // Race condition: when canonical sync removes optimistic txs from the pool...
// but before applying the canonical state to stash, a pending optimistic
// update could run and see empty pool + old canonical state Memory Management Issues ( // This is awful and obviously needs to be fixed as soon as there is a better solution
const handlerRegistry = new WeakMap<Client | SessionClient, CreateOptimisticHandlerResult<StoreConfig>>() The WeakMap registry to prevent multiple wrappers indicates fundamental architectural problems. 3. Performance ConcernsFork Cache Clearing ( // TODO: this should definitely be avoided
// clear the fork cache so it doesn't try to read `getStorageAt` from it
vmCopy.stateManager._baseState.forkCache.storage.clearStorage(createAddressFromString(storeAddress)) This aggressive cache clearing on every transaction will hurt performance significantly and accessing private Deep Cloning on Every State Access ( records: structuredClone(stash.get().records),
4. Security & RobustnessError Handling in Storage Override ( Type Safety Issues // @ts-expect-error - Type 'unknown' is not assignable to type '_returnType'.
return async function interceptedRequest(requestArgs: any, options: any) The 5. Code Quality IssuesRecursive Function Without Protection ( for (const key of keys1) {
if (!keys2.includes(key) || !deepEqual(obj1[key], obj2[key])) return false
} The recursive
Inefficient Array Operations ( 🔒 Security AnalysisLow Risk Issues:
Architectural Security:
🚀 Performance AnalysisMajor Performance Impacts:
Memory Usage:
🧪 Test Coverage AssessmentStrengths:
Gaps:
📋 RecommendationsHigh Priority (Architectural)
Medium Priority (Performance)
Low Priority (Quality)
💭 Final ThoughtsThis is an impressive technical achievement that solves a real problem in blockchain UX. However, the current implementation relies on several workarounds that indicate architectural issues. The core concept is sound, but the execution needs refinement to be production-ready. The most critical issue is the architectural complexity requiring multiple workarounds (state coordinator, WeakMap registry, dual clients). A redesign focusing on a cleaner integration pattern would significantly improve maintainability and reliability. The implementation shows deep understanding of both MUD and Tevm, but prioritizing architectural simplicity over feature completeness would yield better long-term results. |
Claude finished @0xpolarzero's task —— View job PR Review: MUD Plugin with Optimistic UpdatesTodo List:
Code Review SummaryI've conducted a thorough review of the MUD plugin implementation. This is an innovative and sophisticated approach to solving optimistic updates in blockchain gaming, but it has several architectural concerns and areas for improvement. 🎯 Overall AssessmentStrengths:
Concerns:
🔍 Detailed Code Quality Analysis1. Architecture & Design IssuesDirect Method Mutation ( client.writeContract = async function interceptedWriteContract(args) { This is a dangerous pattern that mutates the client object directly. This can:
Recommendation: Use a Proxy pattern instead to intercept calls without mutation. Dual Client Architecture 2. State Management & Race ConditionsState Update Coordinator ( // Race condition: when canonical sync removes optimistic txs from the pool...
// but before applying the canonical state to stash, a pending optimistic
// update could run and see empty pool + old canonical state Memory Management Issues ( // This is awful and obviously needs to be fixed as soon as there is a better solution
const handlerRegistry = new WeakMap<Client | SessionClient, CreateOptimisticHandlerResult<StoreConfig>>() The WeakMap registry to prevent multiple wrappers indicates fundamental architectural problems. 3. Performance ConcernsFork Cache Clearing ( // TODO: this should definitely be avoided
// clear the fork cache so it doesn't try to read `getStorageAt` from it
vmCopy.stateManager._baseState.forkCache.storage.clearStorage(createAddressFromString(storeAddress)) This aggressive cache clearing on every transaction will hurt performance significantly and accessing private Deep Cloning on Every State Access ( records: structuredClone(stash.get().records),
4. Security & RobustnessError Handling in Storage Override ( Type Safety Issues // @ts-expect-error - Type 'unknown' is not assignable to type '_returnType'.
return async function interceptedRequest(requestArgs: any, options: any) The 5. Code Quality IssuesRecursive Function Without Protection ( for (const key of keys1) {
if (!keys2.includes(key) || !deepEqual(obj1[key], obj2[key])) return false
} The recursive
Inefficient Array Operations ( 🔒 Security AnalysisLow Risk Issues:
Architectural Security:
🚀 Performance AnalysisMajor Performance Impacts:
Memory Usage:
🧪 Test Coverage AssessmentStrengths:
Gaps:
📋 RecommendationsHigh Priority (Architectural)
Medium Priority (Performance)
Low Priority (Quality)
💭 Final ThoughtsThis is an impressive technical achievement that solves a real problem in blockchain UX. However, the current implementation relies on several workarounds that indicate architectural issues. The core concept is sound, but the execution needs refinement to be production-ready. The most critical issue is the architectural complexity requiring multiple workarounds (state coordinator, WeakMap registry, dual clients). A redesign focusing on a cleaner integration pattern would significantly improve maintainability and reliability. The implementation shows deep understanding of both MUD and Tevm, but prioritizing architectural simplicity over feature completeness would yield better long-term results. |
const tableState = stash._.state.records[table.namespaceLabel]?.[table.label] | ||
const encodedKey = encodeKey({ table, key }) | ||
const prevRecord = tableState?.[encodedKey] | ||
const currentRecord = value == null ? undefined : tableState?.[encodedKey] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
might be mistaken, but wondering why this is emitting the previous value rather than the new update that came in?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
good catch, it should absolutely not do that... this might be a reason for some of the issues
Claude finished @0xpolarzero's task —— View job PR Code Review: MUD Plugin Implementation
Todo List:
I've conducted a thorough code review of this MUD plugin implementation. This is an innovative and sophisticated approach to solving optimistic updates in blockchain gaming, but it has several architectural concerns and areas for improvement. 🎯 Overall AssessmentStrengths:
Concerns:
🔍 Detailed Code Quality Analysis1. Architecture & Design IssuesDirect Method Mutation ( client.writeContract = async function interceptedWriteContract(args) { This is a dangerous pattern that mutates the client object directly. This can:
Recommendation: Use a Proxy pattern instead to intercept calls without mutation. Dual Client Architecture 2. State Management & Race ConditionsState Update Coordinator ( // Race condition: when canonical sync removes optimistic txs from the pool...
// but before applying the canonical state to stash, a pending optimistic
// update could run and see empty pool + old canonical state Memory Management Issues ( // This is awful and obviously needs to be fixed as soon as there is a better solution
const handlerRegistry = new WeakMap<Client | SessionClient, CreateOptimisticHandlerResult<StoreConfig>>() The WeakMap registry to prevent multiple wrappers indicates fundamental architectural problems. 3. Performance ConcernsFork Cache Clearing ( // TODO: this should definitely be avoided
// clear the fork cache so it doesn't try to read `getStorageAt` from it
vmCopy.stateManager._baseState.forkCache.storage.clearStorage(createAddressFromString(storeAddress)) This aggressive cache clearing on every transaction will hurt performance significantly and accessing private Deep Cloning on Every State Access ( records: structuredClone(stash.get().records),
4. Security & RobustnessError Handling in Storage Override ( Type Safety Issues // @ts-expect-error - Type 'unknown' is not assignable to type '_returnType'.
return async function interceptedRequest(requestArgs: any, options: any) The 5. Code Quality IssuesRecursive Function Without Protection ( for (const key of keys1) {
if (!keys2.includes(key) || !deepEqual(obj1[key], obj2[key])) return false
} The recursive
Inefficient Array Operations ( 🔒 Security AnalysisLow Risk Issues:
Architectural Security:
🚀 Performance AnalysisMajor Performance Impacts:
Memory Usage:
🧪 Test Coverage AssessmentStrengths:
Gaps:
📋 RecommendationsHigh Priority (Architectural)
Medium Priority (Performance)
Low Priority (Quality)
💭 Final ThoughtsThis is an impressive technical achievement that solves a real problem in blockchain UX. However, the current implementation relies on several workarounds that indicate architectural issues. The core concept is sound, but the execution needs refinement to be production-ready. The most critical issue is the architectural complexity requiring multiple workarounds (state coordinator, WeakMap registry, dual clients). A redesign focusing on a cleaner integration pattern would significantly improve maintainability and reliability. The implementation shows deep understanding of both MUD and Tevm, but prioritizing architectural simplicity over feature completeness would yield better long-term results. |
1. Summary
This pull request introduces
@tevm/mud
, an extension that implements optimistic updates for MUD applications.Problem Solved:
Core Mechanism:
The plugin intercepts
writeContract
calls andgetStorageAt
requests to create a parallel optimistic execution environment:Transaction Interception: When a user calls
writeContract
, the plugin:State Management: Maintains two memory clients:
Storage Override: Intercepts
eth_getStorageAt
calls to the MUD store contract, finding the right record inside thestash
to encode it into a storage value instead of canonical blockchain data (or falls back to the canonicaleth_getStorageAt
)Reconciliation: When transactions confirm on-chain, the plugin removes them from the pending pool, which "automatically" reconciles the optimistic state with canonical state (the optimistic tx won't need to be replayed on top of the canonical state anymore)
2. Highlight: Practical Integration Examples
Vanilla JS Example
React Example
Based on the pattern in
examples/mud/packages/client/src/Providers.tsx
:3. Detailed Optimistic Update Flow
Here's the precise step-by-step trace of the entire optimistic update process:
Phase 1: Transaction Initiation
client.writeContract()
(e.g., moving a player in a game)mudStoreWriteRequestOverride
intercepts the call inbundler-packages/mud/src/internal/decorators/mudStoreWriteRequestOverride.ts:36
generateTxIdentifier()
for trackingoriginalWriteContract()
Phase 2: Optimistic Simulation
'simulating'
, subscribers notifiedgetStorageAt
requests through interceptor (this is an ugly workaround to the request bypassing thegetStorageAt
procedure, which we intercept, and trying to get from the state's manager cache directly)memoryClient.tevmCall()
executes transaction with:addToMempool: true
- adds to pending transaction poolblockTag: 'pending'
- executes on pending block'txadded'
eventPhase 3: Optimistic State Calculation
processTransactionsAndUpdateLogs()
triggered by'txadded'
eventtxPool.txsByPriceAndNonce()
orders pending transactionsvmCopy.runTx()
internalLogs
getStateView()
creates optimistic state:Phase 4: UI Update Propagation
notifyStashSubscribers()
notifies all optimistic state subscribersuseOptimisticState
automatically re-render with optimistic data'optimistic'
Phase 5: Storage Request Interception
getStorageAt
for store contract:mudStoreGetStorageAtOverride
intercepts the requestPhase 6: Network Confirmation
waitForTransactionReceipt()
detects confirmation (this one is in case it wasn't synced by thestore-sync
, e.g. if it reverted/failed, so we remove it from the optimistic pool; otherwise, thestore-sync
will hit first)'confirmed'
or'reverted'
store-sync
processes incoming transaction:syncAdapter.onTx()
called with transaction hashmatchOptimisticTxCounterpart()
finds corresponding optimistic transaction (using the data suffix we added to both)'txremoved'
event triggers optimistic state recalculation (we reapply all remaining optimistic txs on top of the newly updated canonical state, to cache the optimistic logs)Error Handling & Edge Cases
stateUpdateCoordinator
serializes optimistic updates to prevent conflicts (one of the ugly workarounds)4. Limitations
Client Type Restriction
The current implementation is specifically designed for MUD projects using an EntryKit SessionClient (bundler client) and will not work with other client types. The interception logic in
mudStoreWriteRequestOverride.ts:18
explicitly checks forclient.type === 'bundlerClient'
and only overrides thewriteContract
method on SessionClient instances. This should not be hard to implement for other client types, but requires additional development.Implementation Architecture Issues
The implementation is a 'clunky' first version with several non-ideal practices:
Direct Method Mutation: The plugin directly overwrites/mutates the
writeContract
method on the provided client instance (client.writeContract = async function...
inmudStoreWriteRequestOverride.ts:36
), which is dangerous and can cause unexpected behaviors if other extensions depend on the original reference.Dual Client Architecture: The system requires two separate MemoryClients (
internalClient
andoptimisticClient
) with different storage interceptors, which adds complexity and resource overhead.Race Conditions and State Management
The example demonstrates that running multiple calls quickly causes the UI to flash incorrect states, highlighting probable race conditions or similar issues. Despite attempts to mitigate this with a
stateUpdateCoordinator
class to manage state inside the plugin, it's not sufficient:Asynchronous State Updates: The optimistic state calculation and removal from pending pool being async creates windows where multiple MUD sync → stash update → remove tx from pending pool operations can fire rapidly and interfere with each other. This is what the
stateUpdateCoordinator
fixes, but needing it indicates the architecture has fundamental synchronization issues.Memory Management: The plugin requires a
WeakMap
registry (handlerRegistry
inuseOptimisticWrapper.tsx:23
) to prevent multiple writeContract wrappers and txPool subscriptions, which is described as "awful and obviously needs to be fixed."