Skip to content

Conversation

@abelanger5
Copy link

Hey folks,

This PR makes some essential updates to the Hatchet benchmark to make it more accurately reflect Hatchet's performance.

The crux of the issue: while the Temporal benchmarks are written to spawn each fibonacci method as an individual Activity, the Hatchet benchmarks have been declared as a predefined DAG. For 400 iterations, this results in a 400-step DAG, which cannot be parallelized (the benchmarks across 10 workers seem to reflect this problem).

This PR updates the Go and Python implementation to use a parent-child task pattern, where the parent task's analogue is the Temporal Workflow, while the child tasks is like the Temporal Activity.

Based on my local testing, this brings all Hatchet benchmarks completely in line with the performance of Temporal. This is also the method that Hatchet recommends using for this type of dynamic orchestration pattern for workflows that are more than a few steps, so it's a much fairer benchmark.

Other notes

  • The latest stable Hatchet release is v0.71.0.
  • I am not sure how the CSV files are generated - the script didn't work for me.
  • I've removed the bulk workflow implementation - there is no similar behavior in Temporal and it shouldn't be necessary to get things to process in parallel.

Local Benchmarks

I don't expect these to be the final results, but they are meant to illustrate the discrepancy, particularly for Go. I've also included commands for easy repro:

# 400 lightweight tasks, 1 worker (n=10) -- 15.15001875s
export ITERATIONS=400
unset PARALLEL
sh run-workers.sh 1
go run cmd/run/main.go --n=10

# 100 long-running tasks, 1 worker (n=38) -- 16.024349875s
export ITERATIONS=100
unset PARALLEL
sh run-workers.sh 1
go run cmd/run/main.go --n=38

# 400 lightweight tasks, 10 workers (n=10) -- 2.305947375s
export ITERATIONS=400
export PARALLEL=true
sh run-workers.sh 10
go run cmd/run/main.go --n=10

# 100 long-running tasks, 10 workers (n=38) -- 1.978141375s
export ITERATIONS=100
export PARALLEL=true
sh run-workers.sh 10
go run cmd/run/main.go --n=38

We'd appreciate you updating the benchmarks to reflect this more fair comparison. Let me know if there's anything else needed for this PR.

@rubenfiszel
Copy link
Contributor

@alpetric could you take a look please

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants