[CI/Build] Add basic multimodal lm eval for CI testing #19959
Conversation
Signed-off-by: Ye (Charlotte) Qi <[email protected]>
Signed-off-by: Ye (Charlotte) Qi <[email protected]>
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Summary of Changes
Hello @yeqcharlotte, I'm Gemini Code Assist1! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request significantly enhances the CI pipeline by introducing initial end-to-end evaluation correctness testing for multimodal models. It specifically adds a test for the chartqa task using the vllm-vlm backend, filling a critical gap in current CI coverage. The changes also include standardizing existing model configurations and making the evaluation framework more flexible to support different model backends.
Highlights
- CI Coverage for Multimodal Models: Introduced a new CI test to evaluate the end-to-end correctness of multimodal models, specifically using the
chartqatask with thevllm-vlmbackend. This addresses a current gap in CI coverage for multimodal capabilities. - New Multimodal Evaluation Configuration: Added a new YAML configuration file (
Qwen2.5-VL-7B-Instruct.yaml) for theQwen2.5-VL-7B-Instructmultimodal model, specifyingvllm-vlmas its evaluation backend andchartqaas the task. - Flexible LM Eval Backend Selection: Modified the
test_lm_eval_correctness.pyscript to dynamically select thelm_evalmodel backend based on the evaluation configuration, allowing forvllm-vlmalongside the existingvllmbackend. - Conditional Chat Template Application: Implemented logic within the
test_lm_eval_correctness.pyscript to apply chat templates only forvllm-vlmbackends during evaluation. This avoids regressions observed when applying chat templates to existing text models. - Dedicated Multimodal Evaluation Script: Added a new bash script (
run-lm-eval-chartqa-vllm-vlm-baseline.sh) to facilitate runninglm_evalspecifically for multimodal models on thechartqadataset.
Using Gemini Code Assist
The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.
Invoking Gemini
You can request assistance from Gemini at any point in your pull request via creating an issue comment (i.e. comment on the pull request page) using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands.
| Feature | Command | Description |
|---|---|---|
| Code Review | /gemini review |
Performs a code review for the current pull request in its current state. |
| Pull Request Summary | /gemini summary |
Provides a summary of the current pull request in its current state. |
| Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in issue comments and review comments. |
| Help | /gemini help |
Displays a list of available commands. |
Customization
To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.
Limitations & Feedback
Gemini Code Assist is currently in preview and may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments to provide feedback.
You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.
Footnotes
-
Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩
|
👋 Hi! Thank you for contributing to the vLLM project. 💬 Join our developer Slack at https://slack.vllm.ai to discuss your PR in #pr-reviews, coordinate on features in #feat- channels, or join special interest groups in #sig- channels. Just a reminder: PRs would not trigger full CI run by default. Instead, it would only run Once the PR is approved and ready to go, your PR reviewer(s) can run CI to test the changes comprehensively before merging. To run CI, PR reviewers can either: Add 🚀 |
There was a problem hiding this comment.
Code Review
This PR introduces a new CI test for multimodal models using lm-eval-harness. The changes include a new test configuration, a script to run the evaluation, and updates to existing configurations and test logic to support different backends. I've suggested adding validation for required arguments in the new shell script and making the Python test logic more robust.
| lm_eval --model vllm-vlm \ | ||
| --model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE" \ | ||
| --tasks chartqa \ | ||
| --batch_size auto \ | ||
| --apply_chat_template \ | ||
| --limit $LIMIT |
There was a problem hiding this comment.
Consider validating that the required arguments (-m, -l, -t) are provided to the script. This will prevent lm_eval from being called with empty variables, which could lead to errors.
| lm_eval --model vllm-vlm \ | |
| --model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE" \ | |
| --tasks chartqa \ | |
| --batch_size auto \ | |
| --apply_chat_template \ | |
| --limit $LIMIT | |
| if [[ -z "$MODEL" || -z "$LIMIT" || -z "$TP_SIZE" ]]; then | |
| echo "Error: -m, -l, and -t are required arguments." | |
| usage | |
| exit 1 | |
| fi | |
| lm_eval --model vllm-vlm \ | |
| --model_args "pretrained=$MODEL,tensor_parallel_size=$TP_SIZE" \ | |
| --tasks chartqa \ | |
| --batch_size auto \ | |
| --apply_chat_template \ | |
| --limit $LIMIT |
| # pip install lm-eval==0.4.9 | ||
|
|
||
| usage() { | ||
| echo`` |
There was a problem hiding this comment.
This line contains a typo: echo`` will print a backtick. Use echoorecho ""` to print an empty line.
| echo`` | |
| echo |
| model=eval_config["backend"], | ||
| model_args=model_args, | ||
| tasks=[task["name"] for task in eval_config["tasks"]], | ||
| num_fewshot=eval_config["num_fewshot"], | ||
| limit=eval_config["limit"], | ||
| batch_size="auto", | ||
| # TODO(yeq): using chat template w/ fewshot_as_multiturn is supposed help | ||
| # text models. however, this is regressing measured strict-match for | ||
| # existing text models in CI, so only apply it for mm. | ||
| apply_chat_template=eval_config["backend"] == "vllm-vlm", |
There was a problem hiding this comment.
To prevent KeyError exceptions, use eval_config.get("backend", "vllm") to provide a default value if the backend key is missing.
results = lm_eval.simple_evaluate(
model=eval_config.get("backend", "vllm"),
model_args=model_args,
tasks=[task["name"] for task in eval_config["tasks"]],
num_fewshot=eval_config["num_fewshot"],
limit=eval_config["limit"],
batch_size="auto",
# TODO(yeq): using chat template w/ fewshot_as_multiturn is supposed help
# text models. however, this is regressing measured strict-match for
# existing text models in CI, so only apply it for mm.
apply_chat_template=eval_config.get("backend") == "vllm-vlm",
)
Essential Elements of an Effective PR Description Checklist
supported_models.mdandexamplesfor a new model.Purpose
None of existing CI tests E2E eval correctness for multimodal models.
vllm/.buildkite/scripts/hardware_ci/run-tpu-v1-test.sh
Lines 136 to 137 in c4cf260
vllm/.buildkite/test-pipeline.yaml
Line 239 in c4cf260
vllm/.buildkite/test-pipeline.yaml
Line 455 in c4cf260
So improve this coverage by running chartqa using "vllm-vlm" backend in lm_eval.
chartqa requires 0.4.9, #19962 to upgrade and see if there's any issues.
For reviewer, most relevant changes are in:
.buildkite/lm-eval-harness/configs/Qwen2.5-VL-7B-Instruct.yaml.buildkite/lm-eval-harness/test_lm_eval_correctness.pyFollow-ups not included in this PR:
Test Plan
(We can wait for ~1 week to add them to CI)
Confirm existing tests still work:
Test Result
New mm test
Right now we run the full test which takes ~3.5 min. 500 examples takes ~90s, but there's a big drop in metrics to ~0.76. This is likely due to the questions are off different complextiy.
Qwen reported 87.3: https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct
I was able to get 86.6 in tp2 but tp1 is around 86. Some potential kernel issues to dig into as a follow-up. But I'll leave them as it is.
*Existing text test
it passes on this PR.
i tried to enable chat template for text models with fewshot_as_multiturn but observed regression in 5 out of 5 test models' strict-match so only apply chat template to mm.