Add AGENTS.md #114
Add AGENTS.md #114
Conversation
WalkthroughAdded a new documentation file, AGENTS.md, containing development and testing guidelines for Chainbench (Locust-based), covering environment setup, dependency management, formatting/linting/type checks, testing strategies, profile development, test data practices, workflow commands, and safety reminders. No code or public API changes. Changes
Estimated code review effort🎯 1 (Trivial) | ⏱️ ~2 minutes Poem
Tip 🔌 Remote MCP (Model Context Protocol) integration is now available!Pro plan users can now connect to remote MCP servers from the Integrations page. Connect with popular remote MCPs such as Notion and Linear to add more context to your reviews and chats. ✨ Finishing Touches🧪 Generate unit tests
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. CodeRabbit Commands (Invoked using PR/Issue comments)Type Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (10)
AGENTS.md (10)
3-3: Neutralize vendor-specific reference and fix punctuation.Avoid potentially outdated brand names and add the missing comma after “e.g.”.
-This repository contains Chainbench, a blockchain infrastructure benchmarking tool built on Locust. When working on the project interactively with an agent (e.g. the Codex CLI) please follow the guidelines below for efficient development and testing. +This repository contains Chainbench, a blockchain infrastructure benchmarking tool built on Locust. When working on the project interactively with an agent (e.g., a CLI assistant), please follow the guidelines below for efficient development and testing.
21-25: Tighten Poetry workflow; avoid redundant lock step afterpoetry add.
poetry addupdates the lock automatically. Reservepoetry lockfor when you edit pyproject.toml manually or need to refresh the lock without changing constraints.-1. Use Poetry to manage dependencies: `poetry add <package>` or `poetry add --group dev <package>`. -2. Run `poetry lock --no-update` to update the lock file. -3. Install updated dependencies with `poetry install`. -4. Verify compatibility with Python 3.10+ as specified in the project. +1. Use Poetry to manage dependencies: `poetry add <package>` or `poetry add --group dev <package>`. +2. Install updated dependencies with `poetry install`. +3. If you edited pyproject.toml manually, run `poetry lock` to regenerate the lock file. +4. Verify compatibility with Python 3.10+ as specified in the project.
28-33: Minor style and linter alignment tweaks.Add the hyphen in “120‑character” and note W503 alongside E203 for Black compatibility (common pairing).
-* Follow Black formatting (120 character line length). +* Follow Black formatting (120-character line length). * Use isort for import sorting (Black-compatible profile). -* Follow Flake8 linting rules (E203 ignored for Black compatibility). +* Follow Flake8 linting rules (ignore E203 and W503 for Black compatibility). * Use type hints where appropriate. * Keep MyPy checks passing.
34-44: Keep wording consistent in the table.Very minor: “Run type checks” reads more idiomatically.
-| `poetry run mypy .` | Run type checking | +| `poetry run mypy .` | Run type checks |
46-48: Addpre-commit installto make hooks effective locally.Without installation, hooks won’t run on commit.
Or use pre-commit hooks: ```bash +poetry run pre-commit install poetry run pre-commit run --all-files--- `77-81`: **Add a quick note on monitoring method.** Consider mentioning how to watch memory/CPU (e.g., `htop`, Docker stats, or built-in metrics) to make the advice immediately actionable. ```diff -* Monitor memory usage with larger data sizes. +* Monitor memory/CPU of both Chainbench workers and the target node (e.g., via `htop`, `docker stats`, or your cloud metrics) with larger data sizes.
91-101: Addstart --helpand--versionto the recap for quick discovery.| `poetry install` | Install all dependencies | | `poetry run chainbench --help` | Show all available commands | +| `poetry run chainbench start --help` | Show options for running a benchmark | +| `poetry run chainbench --version` | Show CLI version | | `poetry run chainbench list methods` | List supported RPC methods | | `poetry run chainbench list profiles` | List available profiles | | `poetry run chainbench list shapes` | List load pattern shapes | | `poetry run chainbench discover <url>` | Discover available methods |
102-109: Add a short privacy/ToS reminder.Helps prevent accidental leakage and policy violations during testing/reporting.
* **Use appropriate rate limits** to avoid overwhelming nodes. * **Start with light profiles** before heavy ones. * **Keep test durations short** during development. +* **Do not include private endpoints or credentials** in commands, logs, or bug reports. +* **Respect your provider’s Terms of Service** and published rate limits.
69-74: Specify the canonical profiles directoryPlease update AGENTS.md to point readers at the exact location where profiles live:
• File: AGENTS.md
Lines: 69–74Short explanation: instead of “Place custom profiles in appropriate subdirectories,” name the root path explicitly so users don’t have to guess.
Suggested diff:
-* Place custom profiles in appropriate subdirectories. +* Place custom profiles in the canonical directory: `chainbench/profile/<network>/…`Optionally, you can list the top-level folders under
chainbench/profile/(e.g.evm,solana,starknet, etc.) to make it even clearer.
1-112: Add a link to AGENTS.md in README and consider moving it under docs/I verified that:
- README.md exists but does not reference AGENTS.md.
- There is no CONTRIBUTING.md in the repo.
- A docs/ directory is already present.
Recommendations:
- In README.md, add a “Development” or “Contributing” section with a link to AGENTS.md.
- Optionally, relocate AGENTS.md into docs/ for better organization and update the README link to point to docs/AGENTS.md.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
AGENTS.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
AGENTS.md
[grammar] ~38-~38: There might be a mistake here.
Context: ...ose | | ------------------------ | -----------...
(QB_NEW_EN)
[grammar] ~39-~39: There might be a mistake here.
Context: ...-------------------------------------- | | poetry run black . | Format code...
(QB_NEW_EN)
[grammar] ~40-~40: There might be a mistake here.
Context: ...mat code to project standards | | poetry run isort . | Sort import...
(QB_NEW_EN)
[grammar] ~41-~41: There might be a mistake here.
Context: ... imports | | poetry run flake8 | Run linting...
(QB_NEW_EN)
[grammar] ~42-~42: There might be a mistake here.
Context: ... linting checks | | poetry run mypy . | Run type ch...
(QB_NEW_EN)
[grammar] ~77-~77: There might be a mistake here.
Context: ... ## 7. Working with Test Data * Start with smallest data size (--size XS) for de...
(QB_NEW_EN)
[grammar] ~77-~77: There might be a mistake here.
Context: ...data size (--size XS) for development. * Use --use-latest-blocks for nodes with...
(QB_NEW_EN)
🔇 Additional comments (3)
AGENTS.md (3)
1-112: Solid, actionable doc — nice addition.Clear, practical guidelines with concrete commands. This will help keep local/dev testing safe and fast. No blocking issues from me; see a few targeted refinements below.
14-15: Align AGENTS.md example with the actualstartsubcommand flags and use a<NODE_URL>placeholder
- Replace the hard-coded
https://test-nodewith<NODE_URL>to avoid accidental copy-paste against a real node.- Verify that the flags in your example match those exposed by the
startcommand inchainbench/main.py:
- The current code defines at least:
•--profile
•--users
•--headless
•--autoquit
•--target
•--run-id- If
--workersand--test-timeare not supported in your PR branch, either add those options to the CLI or update the example to use the correct flags for concurrency and duration.-poetry run chainbench start --profile evm.light --users 5 --workers 1 --test-time 30s --target https://test-node --headless --autoquit +poetry run chainbench start --profile evm.light --users 5 --workers 1 --test-time 30s --target <NODE_URL> --headless --autoquit
54-64: Clarify valid--clientsvalues in CLI docsIt’s not immediately obvious which client identifiers are supported by
chainbench discover --clients. To ensure users pick valid values, please update the snippet (and surrounding text) to:
- Show at least two real examples (e.g.
--clients geth,erigonor--clients nethermind,openethereum)- Or link to the definitive source of truth (
chainbench/tools/discovery/clients.jsonor call outRpcDiscovery.get_client_names())For example:
- chainbench discover https://test-node --clients geth + chainbench discover https://test-node --clients geth,erigon # (Valid values: geth, erigon, nethermind, openethereum, etc.; see chainbench/tools/discovery/clients.json)See the
validate_clientscallback andRpcDiscovery.get_client_names()for how client names are loaded.
erwin-wee
force-pushed
the
add-agents-md
branch
from
a865823 to
23e5ce6
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
♻️ Duplicate comments (1)
AGENTS.md (1)
28-33: Align Flake8 guidance with Black: add W503 to ignores.To be fully Black-compatible, recommend ignoring both E203 and W503 (the latter for line breaks around binary operators). Update the bullet to reflect this.
-* Follow Flake8 linting rules (E203 ignored for Black compatibility). +* Follow Flake8 linting rules (ignore E203 and W503 for Black compatibility).
🧹 Nitpick comments (6)
AGENTS.md (6)
21-25: Tighten Poetry workflow phrasing.poetry add already updates the lock; poetry lock --no-update is typically for metadata-only changes. Suggest clarifying when to use each.
-1. Use Poetry to manage dependencies: `poetry add <package>` or `poetry add --group dev <package>`. -2. Run `poetry lock --no-update` to update the lock file. +1. Use Poetry to manage dependencies: `poetry add <package>` (or `--group dev` for dev deps) — this updates the lock file. +2. If you only change metadata (no version bumps), run `poetry lock --no-update` to refresh the lock deterministically.
45-49: Pre-commit: add install step so hooks run on commit.Running hooks manually is fine, but most folks expect automatic checks on commit.
Or use pre-commit hooks: ```bash +poetry run pre-commit install poetry run pre-commit run --all-files--- `77-81`: **Grammar and repeatability nits; clarify when to use latest blocks.** ```diff -* Start with smallest data size (`--size XS`) for development. +* Start with the smallest data size (`--size XS`) for development. * Use `--use-latest-blocks` for nodes with limited history. -* Consider using `--ref-url` for test data generation from a reference node. +* Consider using `--ref-url` for test data generation from a reference node. +* For repeatable benchmarks across runs, prefer pinning a block range over `--use-latest-blocks`.
83-90: Add a quick test step to the workflow.If the repo has tests, include a minimal test invocation so contributors don’t skip it.
2. Run formatting: `poetry run black . && poetry run isort .` 3. Run linting: `poetry run flake8` 4. Run type checking: `poetry run mypy .` -5. Test with minimal profile first +5. Run tests: `poetry run pytest -q` # or project-specific test runner +6. Test with minimal profile first -6. Gradually increase complexity and load +7. Gradually increase complexity and loadIf the project doesn’t use pytest, I can tailor this to your test runner.
26-33: Optional: link to the authoritative configs so the doc stays in sync.A small “See also” tying to pyproject.toml sections for Black/isort/Flake8/MyPy helps readers map guidelines to enforcement.
* Use type hints where appropriate. * Keep MyPy checks passing. + +See also: the `[tool.black]`, `[tool.isort]`, `[tool.flake8]`, and `[tool.mypy]` sections in `pyproject.toml` (and `mypy.ini` if present).
12-15: Docs: Confirm CLI flags and add spawn‐rate in exampleVerified that the example flags in AGENTS.md match the current Chainbench CLI (all options below exist in chainbench/main.py, with
--test-timeas the correct flag and no--run-timealias). Since the--spawn-rateoption is supported and helps ensure a predictable ramp-up, please update the example command to include it:• File: AGENTS.md (around lines 12–15)
Update the example invocation to:- poetry run chainbench start --profile evm.light --users 5 --workers 1 --test-time 30s --target https://test-node --headless --autoquit + poetry run chainbench start --profile evm.light --users 5 --spawn-rate 1 --workers 1 --test-time 30s --target https://test-node --headless --autoquit
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
💡 Knowledge Base configuration:
- MCP integration is disabled by default for public repositories
- Jira integration is disabled by default for public repositories
- Linear integration is disabled by default for public repositories
You can enable these sources in your CodeRabbit configuration.
📒 Files selected for processing (1)
AGENTS.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
AGENTS.md
[grammar] ~38-~38: There might be a mistake here.
Context: ...ose | | ------------------------ | -----------...
(QB_NEW_EN)
[grammar] ~39-~39: There might be a mistake here.
Context: ...-------------------------------------- | | poetry run black . | Format code...
(QB_NEW_EN)
[grammar] ~40-~40: There might be a mistake here.
Context: ...mat code to project standards | | poetry run isort . | Sort import...
(QB_NEW_EN)
[grammar] ~41-~41: There might be a mistake here.
Context: ... imports | | poetry run flake8 | Run linting...
(QB_NEW_EN)
[grammar] ~42-~42: There might be a mistake here.
Context: ... linting checks | | poetry run mypy . | Run type ch...
(QB_NEW_EN)
[grammar] ~77-~77: There might be a mistake here.
Context: ... ## 7. Working with Test Data * Start with smallest data size (--size XS) for de...
(QB_NEW_EN)
[grammar] ~77-~77: There might be a mistake here.
Context: ...data size (--size XS) for development. * Use --use-latest-blocks for nodes with...
(QB_NEW_EN)
🔇 Additional comments (2)
AGENTS.md (2)
1-112: Overall: solid, practical doc.Well-structured and actionable. Clear guidance for safe, short, headless development runs and a tidy QA loop (format → lint → typecheck → smoke). Nice.
91-101: Useful commands list verifiedI’ve confirmed that all of the commands in the “Useful Commands Recap” section map directly to registered click subcommands in
chainbench/main.py:
chainbench list methodschainbench list profileschainbench list shapeschainbench discover <endpoint>(The
listgroup and its four subcommands—methods, profiles, shapes, and the client‐options listing—are all defined via@_list.commanddecorators.) No drift detected, so no updates are required here.
| Test changes progressively: | ||
|
|
||
| 1. **Unit testing**: Test individual user classes and methods | ||
| 2. **Profile validation**: Verify profiles load correctly | ||
| ```bash | ||
| poetry run chainbench list profiles | ||
| ``` | ||
| 3. **Short headless tests**: Run brief tests with minimal load | ||
| 4. **Method discovery**: Test endpoint compatibility | ||
| ```bash | ||
| poetry run chainbench discover https://test-node --clients geth | ||
| ``` | ||
|
|
There was a problem hiding this comment.
💡 Verification agent
🧩 Analysis chain
Confirm discover/list subcommands and client flag names.
Please verify that list profiles|methods|shapes and discover (with --clients geth) are the current subcommands and option names.
🏁 Script executed:
#!/bin/bash
# Verify subcommands referenced by the doc
set -euo pipefail
echo "Look for 'list' subcommands:"
rg -nP -C2 --type=py '\blist\s+(profiles|methods|shapes)\b' || true
echo -e "\nLook for 'discover' command and client flag names:"
rg -nP -C2 --type=py '\bdiscover\b' || true
rg -nP -C2 --type=py '\b(--clients|--client|--node|--engine)s?\b' || trueLength of output: 1730
Verify CLI subcommands and flags
- Confirmed that
chainbench list profiles(with optional-d <profile_dir>) is supported. - Confirmed that
chainbench list methodsexists (used in the error message: “Use ‘chainbench list methods’ to list all available methods.”). - There is no
chainbench list shapessubcommand in the code; please remove any references tolist shapesfrom the documentation (or implement it in the CLI). - Confirmed that
chainbench discover <endpoint>accepts a--clients <client1,client2…>flag for specifying one or more clients.
🤖 Prompt for AI Agents
In AGENTS.md around lines 52 to 64, the docs reference a non-existent
`chainbench list shapes` subcommand and include commands that should reflect
actual CLI behavior; remove any mention of `list shapes` (or implement the
subcommand if intended), and update the examples to match the real flags: note
that `chainbench list profiles` supports an optional `-d <profile_dir>` and that
`chainbench discover <endpoint>` accepts `--clients <client1,client2…>`; also
keep the `chainbench list methods` reference as-is since it exists.
Summary by CodeRabbit