Docs: Updates Benchmark Guide #789
Merged
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Daneyon Hansen <[email protected]>
k8s-ci-robot
added
the
cncf-cla: yes
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: danehans The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
✅ Deploy Preview for gateway-api-inference-extension ready!
To edit notification comments on pull requests, go to your Netlify site configuration. |
k8s-ci-robot
added
the
size/M
|
/lgtm |
k8s-ci-robot
added
the
lgtm
liu-cong
reviewed
May 7, 2025
| ```bash | ||
| git clone https://github.com/kubernetes-sigs/gateway-api-inference-extension | ||
| cd gateway-api-inference-extension | ||
| ``` | ||
|
|
||
| 1. Get the target IP. Examples below show how to get the IP of a gateway or a LoadBalancer k8s service. | ||
| 1. Get the target IP. The examples below shows how to get the IP of a gateway or a k8s service. |
There was a problem hiding this comment.
Suggested change
| 1. Get the target IP. The examples below shows how to get the IP of a gateway or a k8s service. | |
| 1. Get the target IP. The example below shows how to get the IP of a gateway or a k8s service. |
|
|
||
| ```bash | ||
| kubectl scale --replicas=8 -f https://github.com/kubernetes-sigs/gateway-api-inference-extension/raw/main/config/manifests/vllm/gpu-deployment.yaml | ||
| kubectl scale deployment vllm-llama3-8b-instruct --replicas=8 |
There was a problem hiding this comment.
nit: I suggest changing replicas to 6 as the example in the end uses 6 replicas, and the new regression test PR also uses 6
| @@ -37,7 +37,7 @@ spec: | |||
| - name: BACKEND | |||
| value: vllm | |||
| - name: PORT | |||
| value: "8081" | |||
| value: "80" | |||
There was a problem hiding this comment.
Great! 8081 was the port back then when we had envoy patches.
k8s-ci-robot
added
the
do-not-merge/hold
|
/lgtm |
rlakhtakia
pushed a commit
to rlakhtakia/gateway-api-inference-extension
that referenced
this pull request
May 13, 2025
Signed-off-by: Daneyon Hansen <[email protected]>
nayihz
pushed a commit
to nayihz/gateway-api-inference-extension
that referenced
this pull request
May 14, 2025
Signed-off-by: Daneyon Hansen <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
config/manifests/benchmark/benchmark.yaml: Updates the target port to80since the guide only sets thetarget-ipand most gateways use port 80 by default.site-src/performance/benchmark/index.md: Provides additional explanation in steps to help guide users. Adds a note to use the GPU-based vLLM deployment for benchmarking. Updates thebenchmark_idvalue to match the labels intools/benchmark/benchmark.ipynb.tools/benchmark/benchmark.ipynb: Sets the default run id and removes undefinedINTERACTIVE_PLOTvariable.