[CI failed]: V1 Test Failed due to "No available memory for the cache blocks" in GitHub Actions

[cynthieye]

Anything you want to discuss about vllm.

I encountered c1 test failure when running CI while initiating the merge request code
code link：#14377
The error is as follows：

[2025-03-10T15:07:20Z] INFO 03-10 08:07:20 [gpu_model_runner.py:1067] Model loading took 0.2389 GB and 0.977005 seconds

  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] EngineCore hit an exception: Traceback (most recent call last):
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 294, in run_engine_core
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] engine_core = EngineCoreProc(*args, **kwargs)
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 249, in init
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] super().init(vllm_config, executor_class, log_stats)
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 59, in init
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] num_gpu_blocks, num_cpu_blocks = self._initialize_kv_caches(
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/engine/core.py", line 102, in _initialize_kv_caches
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] kv_cache_configs = get_kv_cache_configs(vllm_config, kv_cache_specs,
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_utils.py", line 576, in get_kv_cache_configs
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] check_enough_kv_cache_memory(vllm_config, kv_cache_spec,
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] File "/usr/local/lib/python3.12/dist-packages/vllm/v1/core/kv_cache_utils.py", line 468, in check_enough_kv_cache_memory
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] raise ValueError("No available memory for the cache blocks. "
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302] ValueError: No available memory for the cache blocks. Try increasing gpu_memory_utilization when initializing the engine.
  | [2025-03-10T15:07:20Z] ERROR 03-10 08:07:20 [core.py:302]
  | [2025-03-10T15:07:20Z] CRITICAL 03-10 08:07:20 [core_client.py:259] Got fatal signal from worker processes, shutting down. See stack trace above for root cause issue.
  | [2025-03-10T15:07:20Z] bash: line 1: 247 Killed VLLM_USE_V1=1 pytest -v -s v1/engine

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

How should I solve it?

[houseroad]

Seems you don't have enough memory? What model are you trying to load, and what is your GPU memory size?

[github-actions]

This issue has been automatically marked as stale because it has not had any activity within 90 days. It will be automatically closed if no further activity occurs within 30 days. Leave a comment if you feel this issue should remain open. Thank you!