[Bug] vLLM images missing from Docker Hub

[eero-t]

Priority

Undecided

OS type

Ubuntu

Hardware type

Gaudi2

Installation method

• Pull docker images from hub.docker.com
• Build docker images from source

Deploy method

• Docker compose
• Docker
• Kubernetes
• Helm

Running nodes

Single Node

What's the version?

latest from DockerHub, with Always pull policy.

Description

DocSum and FaqGen images are missing for vLLM: https://hub.docker.com/u/opea?page=1&search=-vllm

Although they are there for TGI: https://hub.docker.com/u/opea?page=1&search=-tgi

And there are vLLM Dockerfiles for them:

• DocSum: https://github.com/opea-project/GenAIComps/tree/main/comps/llms/summarization/vllm/langchain
• FaqGen: https://github.com/opea-project/GenAIComps/tree/main/comps/llms/faq-generation/vllm/langchain

And ChatQnA (text generation) vLLM image is already there: https://hub.docker.com/r/opea/llm-vllm

Because of this, CI fails for: opea-project/GenAIInfra#610

Reproduce steps

docker pull opea/llm-docsum-vllm

Raw log

No response

Attachments

No response

[eero-t]

Note: alternatively llm-vllm image could support also FaqGen & DocSum, but that would require more work than just uploading the built images to DockerHub.

[feng-intel]

”opea/llm-docsum-vllm“ image will be added in version 1.1 and it needs OSPDT now.

[eero-t]

Both FaqGen & DocSum image variants have the same difference to (already existing) text-generation vLLM langchain wrapper image deps:

--- requirements.txt	2024-11-12 14:55:59.952764826 +0200
+++ ../../../summarization/vllm/langchain/requirements.txt	2024-11-14 15:05:44.350115888 +0200
@@ -13,4 +13,3 @@
 shortuuid
 transformers
 uvicorn
-vllm

And similar difference in what Dockerfile installs:

 RUN pip install --no-cache-dir --upgrade pip setuptools && \
-    if [ ${ARCH} = "cpu" ]; then \
-      pip install --no-cache-dir --extra-index-url https://download.pytorch.org/whl/cpu -r /home/user/comps/llms/text-generation/vllm/langchain/requirements.txt; \
-    else \
-      pip install --no-cache-dir -r /home/user/comps/llms/text-generation/vllm/langchain/requirements.txt; \
-    fi
+    if [ ${ARCH} = "cpu" ]; then pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/cpu; fi && \
+    pip install --no-cache-dir -r /home/user/comps/llms/summarization/vllm/langchain/requirements.txt

And all of them install the problematic langserve component at runtime, like the already existing TGI variant images:

$ head */*/*/requirements-runtime.txt
==> faq-generation/tgi/langchain/requirements-runtime.txt <==
langserve

==> faq-generation/vllm/langchain/requirements-runtime.txt <==
langserve

==> summarization/tgi/langchain/requirements-runtime.txt <==
langserve

==> summarization/vllm/langchain/requirements-runtime.txt <==
langserve

==> text-generation/ollama/langchain/requirements-runtime.txt <==
langserve

==> text-generation/vllm/langchain/requirements-runtime.txt <==
langserve

==> text-generation/vllm/llama_index/requirements-runtime.txt <==
langserve

[joshuayao]

@eero-t Could we close this issue if it was fixed?

[feng-intel]

docker pull opea/llm-docsum-vllm is OK now.
Close the issue.

[eero-t]

I can verify image is in DockerHub. Unfortunately GenAIInfra CI still fails to pull it...