[Bug] Regression: "opea/vllm-gaudi:latest" container in crash loop

[eero-t]

Priority

Undecided

OS type

• OS: Ubuntu 22.04
• Kernel: 5.15.0

Hardware type

• HW: Gaudi2
• driver_ver: 1.16.2-f195ec4

Installation method

• Pull docker images from hub.docker.com

Deploy method

• Helm

Running nodes

Single Node

What's the version?

https://hub.docker.com/layers/opea/vllm-gaudi/latest/images/sha256-d2c0b0aa88cd26ae2084990663d8d789728f658bacacd8a49cc5b81a6a022c58

Description

vllm-gaudi:latest container does not find devices, and is in crash loop.

But if I change latest tag to 1.1, it works fine, i.e. this is regression.

Reproduce steps

Run ChatQnA from GenAIInfra with vLLM:
$ helm install chatqna chatqna/ --skip-tests --values chatqna/gaudi-vllm-values.yaml ...

Raw log

$ kubectl logs chatqna-vllm-75dfb59d66-wp4vs
...
  File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/__init__.py", line 132, in current_device
    init()
  File "/usr/local/lib/python3.10/dist-packages/habana_frameworks/torch/hpu/__init__.py", line 71, in init
    _hpu_C.init()
RuntimeError: synStatus=8 [Device not found] Device acquire failed.

[eero-t]

Only change in OPEA Git since v1.1 is dropping of the eager option, but v1.1 image works fine with that change.
https://github.com/opea-project/GenAIComps/commits/main/comps/llms/text-generation/vllm/langchain/dependency/

However, comparing the latest image layers to ones in earlier v1.1 one:
https://hub.docker.com/layers/opea/vllm-gaudi/1.1/images/sha256-c75d22e05ff23e4c0745e9c0a56ec74763f85c7fecf23b7f62e0da74175ddae7

Shows quite a few differences, also in sizes of the installed layers.

=> I think the problem is in Habana repo side.

Recent Gaudi vLLM dependency changes is one possibility: https://github.com/HabanaAI/vllm-fork/commits/habana_main/requirements-hpu.txt

Maybe new HPU deps do not handle correctly pod's Gaudi plugin device request allowing vLLM (write) access only to one of the 8 devices in the node?

[xiguiw]

Not sure how the docker file are buid.
I find out one build command, but did not find Dockerfile.hpu

Here both v1.1 and latest are built from the same commit id git checkout 3c39626.
because Dockerfile.hpu is missed. And from the behavior, it should be Gaudi vllm service issue. not OPEA level.

@ashahba
Would you let us know the build command for docker image on docker hub?

[eero-t]

Not sure how the docker file are buid. I find out one build command, but did not find Dockerfile.hpu

@xiguiw As you can see from the OPEA script, it git clones Habana repo [1], cds to repo's vllm-fork dir, and builds Dockerfile.hpu from there.

[1] It would be faster to clone specific commit instead of first cloning whole repo and only then checking out that specific commit.

[xiguiw]

Not sure how the docker file are buid. I find out one build command, but did not find Dockerfile.hpu

@xiguiw As you can see from the OPEA script, it git clones Habana repo [1], cds to repo's vllm-fork dir, and builds Dockerfile.hpu from there.

@eero-t
I did not find the exact build command.

I mean if the vllm-gaudi image build from the same commit in v1.1 and latest, it does not make sense that v1.1 works but latest failed.

If v1.1 and latest docker images are built with different commit ID, that possible.
We can try to build vllm-gaudi docker image independently (without OPEA), then to verify if vllm-gaudi service works.

[eero-t]

@eero-t I did not find the exact build command.

Exact build command is this:
https://github.com/opea-project/GenAIComps/blob/main/comps/llms/text-generation/vllm/langchain/dependency/build_docker_vllm.sh#L41

Which uses this exact Dockerfile:
https://github.com/HabanaAI/vllm-fork/blob/3c39626/Dockerfile.hpu

Which uses this exact Python modules requirements file:
https://github.com/HabanaAI/vllm-fork/blob/3c39626/requirements-hpu.txt

I mean if the vllm-gaudi image build from the same commit in v1.1 and latest, it does not make sense that v1.1 works but latest failed. If v1.1 and latest docker images are built with different commit ID, that possible.

There are several components installed during building of those images, which can change although commit IDs for top level code do not change.

Neither Dockerfile.hpu , nor requirements-hpu.txt file used by it specify exact version numbers / SHA values for their dependencies.

And dependencies in requirements-common.txt, that requirements-hpu.txt depends on, pulls in several additional Python modules where exact version numbers are not given either: https://github.com/HabanaAI/vllm-fork/blob/3c39626/requirements-common.txt

=> image build gets latest version of those deps, which can change between builds...

We can try to build vllm-gaudi docker image independently (without OPEA), then to verify if vllm-gaudi service works.

@xiguiw My first guess would be issue to be with drivers in Habalabs "pytorch-installer-2.4.0:latest" image, which is used as base image by Dockerfile.hpu.

According to: https://vault.habana.ai/ui/repos/tree/General/gaudi-docker/1.18.0/ubuntu22.04/habanalabs/pytorch-installer-2.4.0

Its current latest version is from 2024-12-12, whereas OPEA vllm-gaudi:1.1 image was created several weeks earlier, on 2024-11-22: https://hub.docker.com/r/opea/vllm-gaudi/tags

[eero-t]

While opea/llvm-gaudi:1.1 image continues working, the failure reason for opea/llvm-gaudi:11 has changed.

Now these crashes:

NAME                                   READY   STATUS             RESTARTS      AGE
docsum-vllm-56d857cb77-99788           0/1     CrashLoopBackOff   5 (32s ago)   6m31s

Are due to a segfault:

Internal Error: Received signal - Segmentation fault
Task exception was never retrieved
future: <Task finished name='Task-2' coro=<MQLLMEngineClient.run_output_handler_loop() done, defined at /usr/local/lib/python3.10/dist-packages/vllm-0.6.3.dev910+g3c39626f.gaudi000-py3.10.egg/vllm/engine/multiprocessing/client.py:175> exception=ZMQError('Operation not supported')>
...

Which raises the question, why latest OPEA image is not using the latest Gaudi vLLM release, built from vLLM v0.6.4:
https://github.com/HabanaAI/vllm-fork/releases

But uses non-release commit of that repo, using vLLM v0.6.3-dev?

[yinghu5]

@eero-t thank you a lot for discovering. As we run into some issue, so use one dev version , will try the latest version and upgrade it if no CI issue.

[eero-t]

Latest Gaudi driver release is 1.19.1, according to: https://docs.habana.ai/en/latest/Release_Notes/GAUDI_Release_Notes.html

Whereas my test node has 1.16.2, according to output from: head /sys/class/accel/accel0/device/*ver

If newer Gaudi HW drivers are required with latest image builds and v1.2 release, than with OPEA 1.1 release, that would need to be prominently stated in docs.

[ftian1]

But uses non-release commit of that repo, using vLLM v0.6.3-dev?

it's because at that time the vLLM master has bug to use. we have had a PR to update it.

If newer Gaudi HW drivers are required with latest image builds and v1.2 release, than with OPEA 1.1 release, that would need to be prominently stated in docs.

it's Gaudi's limitation which requests HPU driver also be upgraded before using latest Gaudi related docker images.