Help serving vllm with kuberay

[torsteinelv]

Hello,

have been trying to get this setup working for a long long time and now im about to give up. I have been googling and reading on others issues and deployment, and I see people ask of help on kuberay's github page and the are recommended to ask on vllms page , I hope someone will be able to assist.

Currently I have 3 node 1 gpu in each of them and im able to get the deployment started on 3 nodes but get internal server error when i try to make a request to the openai api. Was hoping someone could help me here :)

current setup:

apiVersion: ray.io/v1
kind: RayService
metadata:
  name: vllm-test
  namespace: ray-operator
spec:
  serveConfigV2: |
    applications:
    - name: llm
      route_prefix: /
      import_path: ray-serve.vllm_engine:build_app
      runtime_env:
        working_dir: "https://github.com/torsteinelv/ray-serve/archive/refs/heads/main.zip"
        pip: ["ray[serve]", "requests"]
      deployments:
      - name: VLLMDeployment
        ray_actor_options:
          num_cpus: 2
      args:
        model: microsoft/Phi-3.5-vision-instruct
        dtype: float16
        pipeline-parallel-size: 3
        accelerator: GPU
        trust_remote_code: True
        enforce-eager: True
        gpu-memory-utilization: 0.8
        max-model-len: 1024
        cpu_offload_gb: 2
  rayClusterConfig:
    headGroupSpec:
      serviceType: LoadBalancer
      rayStartParams:
        dashboard-host: '0.0.0.0'
      template:
        spec:
          volumes:
            - name: model-storage
              persistentVolumeClaim:
                claimName: vllm-test-models
            - name: shm-volume
              emptyDir:
                medium: Memory
          containers:
          - name: ray-head
            image: ghcr.io/sasha0552/vllm:v0.6.6.post1
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
                nvidia.com/gpu: "1"
              requests:
                cpu: "1"
                memory: "1Gi"
                nvidia.com/gpu: "1"
            ports:
            - containerPort: 6379
              name: gcs-server
            - containerPort: 8265
              name: dashboard
            - containerPort: 10001
              name: client
            - containerPort: 8000
              name: serve
            env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: hf_api_token
            volumeMounts:
              - mountPath: /models
                name: model-storage
              - mountPath: /dev/shm
                name: shm-volume
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: ray.io/node-type
                    operator: In
                    values:
                    - worker
                topologyKey: "kubernetes.io/hostname"
    workerGroupSpecs:
    - replicas: 2
      minReplicas: 0
      maxReplicas: 4
      groupName: gpu-group
      rayStartParams: {}
      template:
        spec:
          volumes:
            - name: model-storage
              persistentVolumeClaim:
                claimName: vllm-test-models
            - name: shm-volume
              emptyDir:
                medium: Memory
          containers:
          - name: llm
            image: ghcr.io/sasha0552/vllm:v0.6.6.post1
            env:
            - name: HUGGING_FACE_HUB_TOKEN
              valueFrom:
                secretKeyRef:
                  name: hf-secret
                  key: hf_api_token
            resources:
              limits:
                cpu: "4"
                memory: "16Gi"
                nvidia.com/gpu: "1"
              requests:
                cpu: "1"
                memory: "1Gi"
                nvidia.com/gpu: "1"
            volumeMounts:
              - mountPath: /models
                name: model-storage
              - mountPath: /dev/shm
                name: shm-volume
          tolerations:
            - key: "nvidia.com/gpu"
              operator: "Exists"
              effect: "NoSchedule"
          affinity:
            podAntiAffinity:
              requiredDuringSchedulingIgnoredDuringExecution:
              - labelSelector:
                  matchExpressions:
                  - key: ray.io/node-type
                    operator: In
                    values:
                    - worker
                topologyKey: "kubernetes.io/hostname"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: vllm-test-models
  namespace: ray-operator
spec:
  storageClassName: csi-cephfs-sc
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Gi

code

from typing import Dict, Optional, List
import logging

from fastapi import FastAPI
from starlette.requests import Request
from starlette.responses import StreamingResponse, JSONResponse

from ray import serve

from vllm.engine.arg_utils import AsyncEngineArgs
from vllm.engine.async_llm_engine import AsyncLLMEngine
from vllm.entrypoints.openai.cli_args import make_arg_parser
from vllm.entrypoints.openai.protocol import (
    ChatCompletionRequest,
    ChatCompletionResponse,
    ErrorResponse,
)
from vllm.entrypoints.openai.serving_chat import OpenAIServingChat
from vllm.entrypoints.openai.serving_engine import LoRAModulePath, PromptAdapterPath
from vllm.utils import FlexibleArgumentParser
from vllm.entrypoints.logger import RequestLogger

logger = logging.getLogger("ray.serve")

app = FastAPI()


@serve.deployment(
    autoscaling_config={
        "min_replicas": 1,
        "max_replicas": 10,
        "target_ongoing_requests": 5,
    },
    max_ongoing_requests=10,
)
@serve.ingress(app)
class VLLMDeployment:
    def __init__(
        self,
        engine_args: AsyncEngineArgs,
        response_role: str,
        lora_modules: Optional[List[LoRAModulePath]] = None,
        prompt_adapters: Optional[List[PromptAdapterPath]] = None,
        request_logger: Optional[RequestLogger] = None,
        chat_template: Optional[str] = None,
    ):
        logger.info(f"Starting with engine args: {engine_args}")
        self.openai_serving_chat = None
        self.engine_args = engine_args
        self.response_role = response_role
        self.lora_modules = lora_modules
        self.prompt_adapters = prompt_adapters
        self.request_logger = request_logger
        self.chat_template = chat_template
        self.engine = AsyncLLMEngine.from_engine_args(engine_args)

    @app.post("/v1/chat/completions")
    async def create_chat_completion(
        self, request: ChatCompletionRequest, raw_request: Request
    ):
        """OpenAI-compatible HTTP endpoint.

        API reference:
            - https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html
        """
        if not self.openai_serving_chat:
            model_config = await self.engine.get_model_config()
            # Determine the name of the served model for the OpenAI client.
            if self.engine_args.served_model_name is not None:
                served_model_names = self.engine_args.served_model_name
            else:
                served_model_names = [self.engine_args.model]
            self.openai_serving_chat = OpenAIServingChat(
                self.engine,
                model_config,
                served_model_names,
                self.response_role,
                lora_modules=self.lora_modules,
                prompt_adapters=self.prompt_adapters,
                request_logger=self.request_logger,
                chat_template=self.chat_template,
            )
        logger.info(f"Request: {request}")
        generator = await self.openai_serving_chat.create_chat_completion(
            request, raw_request
        )
        if isinstance(generator, ErrorResponse):
            return JSONResponse(
                content=generator.model_dump(), status_code=generator.code
            )
        if request.stream:
            return StreamingResponse(content=generator, media_type="text/event-stream")
        else:
            assert isinstance(generator, ChatCompletionResponse)
            return JSONResponse(content=generator.model_dump())


def parse_vllm_args(cli_args: Dict[str, str]):
    """Parses vLLM args based on CLI inputs.

    Currently uses argparse because vLLM doesn't expose Python models for all of the
    config options we want to support.
    """
    arg_parser = FlexibleArgumentParser(
        description="vLLM OpenAI-Compatible RESTful API server."
    )

    parser = make_arg_parser(arg_parser)
    arg_strings = []
    for key, value in cli_args.items():
        logger.info(f"Processing argument: --{key} with value: {value}")
        
        if value is True:  # Håndter boolske flagg satt til True
            arg_strings.append(f"--{key}")
        elif value not in (None, "None"):  # Ignorer None eller 'None' som streng
            arg_strings.extend([f"--{key}", str(value)])
        else:
            arg_strings.append(f"--{key}")

    parsed_args = parser.parse_args(args=arg_strings)
    
    return parsed_args


def build_app(cli_args: Dict[str, str]) -> serve.Application:
    """Builds the Serve app based on CLI arguments.

    See https://docs.vllm.ai/en/latest/serving/openai_compatible_server.html#command-line-arguments-for-the-server
    for the complete set of arguments.

    Supported engine arguments: https://docs.vllm.ai/en/latest/models/engine_args.html.
    """  # noqa: E501
    if "accelerator" in cli_args.keys():
        accelerator = cli_args.pop("accelerator")
    else:
        accelerator = "GPU"
    parsed_args = parse_vllm_args(cli_args)
    engine_args = AsyncEngineArgs.from_cli_args(parsed_args)
    engine_args.worker_use_ray = True

    tp = engine_args.tensor_parallel_size
    logger.info(f"Tensor parallelism = {tp}")
    pg_resources = []
    pg_resources.append({"CPU": 1})  # for the deployment replica
    for i in range(tp):
        pg_resources.append({"CPU": 1, accelerator: 1})  # for the vLLM actors

    # We use the "STRICT_PACK" strategy below to ensure all vLLM actors are placed on
    # the same Ray node.
    return VLLMDeployment.bind(
        engine_args,
        parsed_args.response_role,
        parsed_args.lora_modules,
        parsed_args.prompt_adapters,
        cli_args.get("request_logger"),
        parsed_args.chat_template,
    )

error:

INFO 2025-01-11 03:24:22,491 llm_VLLMDeployment 2dqbcyec -- Started initializing replica.
INFO 2025-01-11 03:24:22,492 llm_VLLMDeployment 2dqbcyec -- Starting with engine args: AsyncEngineArgs(model='microsoft/Phi-3.5-vision-instruct', served_model_name=None, tokenizer='microsoft/Phi-3.5-vision-instruct', task='auto', skip_tokenizer_init=False, tokenizer_mode='auto', trust_remote_code=True, allowed_local_media_path=None, download_dir=None, load_format='auto', config_format=<ConfigFormat.AUTO: 'auto'>, dtype='float16', kv_cache_dtype='auto', quantization_param_path=None, seed=0, max_model_len=1024, worker_use_ray=True, distributed_executor_backend=None, pipeline_parallel_size=3, tensor_parallel_size=1, max_parallel_loading_workers=None, block_size=None, enable_prefix_caching=False, disable_sliding_window=False, use_v2_block_manager=True, swap_space=4, cpu_offload_gb=2.0, gpu_memory_utilization=0.8, max_num_batched_tokens=None, max_num_seqs=256, max_logprobs=20, disable_log_stats=False, revision=None, code_revision=None, rope_scaling=None, rope_theta=None, hf_overrides=None, tokenizer_revision=None, quantization=None, enforce_eager=True, max_seq_len_to_capture=8192, disable_custom_all_reduce=False, tokenizer_pool_size=0, tokenizer_pool_type='ray', tokenizer_pool_extra_config=None, limit_mm_per_prompt=None, mm_processor_kwargs=None, disable_mm_preprocessor_cache=False, enable_lora=False, enable_lora_bias=False, max_loras=1, max_lora_rank=16, enable_prompt_adapter=False, max_prompt_adapters=1, max_prompt_adapter_token=0, fully_sharded_loras=False, lora_extra_vocab_size=256, long_lora_scaling_factors=None, lora_dtype='auto', max_cpu_loras=None, device='auto', num_scheduler_steps=1, multi_step_stream_outputs=True, ray_workers_use_nsight=False, num_gpu_blocks_override=None, num_lookahead_slots=0, model_loader_extra_config=None, ignore_patterns=[], preemption_mode=None, scheduler_delay_factor=0.0, enable_chunked_prefill=None, guided_decoding_backend='xgrammar', logits_processor_pattern=None, speculative_model=None, speculative_model_quantization=None, speculative_draft_tensor_parallel_size=None, num_speculative_tokens=None, speculative_disable_mqa_scorer=False, speculative_max_model_len=None, speculative_disable_by_batch_size=None, ngram_prompt_lookup_max=None, ngram_prompt_lookup_min=None, spec_decoding_acceptance_method='rejection_sampler', typical_acceptance_sampler_posterior_threshold=None, typical_acceptance_sampler_posterior_alpha=None, qlora_adapter_name_or_path=None, disable_logprobs_during_spec_decoding=None, otlp_traces_endpoint=None, collect_detailed_traces=None, disable_async_output_proc=False, scheduling_policy='fcfs', override_neuron_config=None, override_pooler_config=None, compilation_config=None, worker_cls='auto', kv_transfer_config=None, generation_config=None, disable_log_requests=False)
INFO 2025-01-11 03:26:05,246 llm_VLLMDeployment 2dqbcyec -- Finished initializing replica.
INFO 2025-01-11 03:26:05,462 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:26:05,469 llm_VLLMDeployment 2dqbcyec 28832e3a-87e5-43c8-a9b4-e7784ed04813 -- GET / 404 15.3ms
INFO 2025-01-11 03:26:30,377 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:30,672 llm_VLLMDeployment 2dqbcyec 6f171a9c-9e57-42c4-8392-ab18991a3262 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:30,867 llm_VLLMDeployment 2dqbcyec 6f171a9c-9e57-42c4-8392-ab18991a3262 -- POST /v1/chat/completions 500 493.2ms
INFO 2025-01-11 03:26:31,273 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:31,289 llm_VLLMDeployment 2dqbcyec 94c1f8f1-00d5-4950-a35c-d856bfe43cef -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:31,294 llm_VLLMDeployment 2dqbcyec 94c1f8f1-00d5-4950-a35c-d856bfe43cef -- POST /v1/chat/completions 500 23.3ms
INFO 2025-01-11 03:26:32,185 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:26:32,203 llm_VLLMDeployment 2dqbcyec f98e1298-7a24-4577-a6bf-85a797ec6263 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:26:32,211 llm_VLLMDeployment 2dqbcyec f98e1298-7a24-4577-a6bf-85a797ec6263 -- POST /v1/chat/completions 500 28.2ms
INFO 2025-01-11 03:27:04,610 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:04,627 llm_VLLMDeployment 2dqbcyec cea64e16-5db2-4670-8833-3eee1e6f21ec -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:04,632 llm_VLLMDeployment 2dqbcyec cea64e16-5db2-4670-8833-3eee1e6f21ec -- POST /v1/chat/completions 500 24.1ms
INFO 2025-01-11 03:27:05,056 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:05,076 llm_VLLMDeployment 2dqbcyec 5675a6b5-f7c6-4c18-b4bc-2d4fd6f5fecd -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:05,081 llm_VLLMDeployment 2dqbcyec 5675a6b5-f7c6-4c18-b4bc-2d4fd6f5fecd -- POST /v1/chat/completions 500 29.8ms
INFO 2025-01-11 03:27:05,882 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 03:27:05,907 llm_VLLMDeployment 2dqbcyec dca2788e-d2c9-4f15-967e-4eec35e3aad7 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 03:27:05,933 llm_VLLMDeployment 2dqbcyec dca2788e-d2c9-4f15-967e-4eec35e3aad7 -- POST /v1/chat/completions 500 53.5ms
INFO 2025-01-11 03:27:35,390 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:27:35,396 llm_VLLMDeployment 2dqbcyec f8e6cc65-fc15-4587-a5fb-f577565c83ab -- GET / 404 8.6ms
INFO 2025-01-11 03:27:46,133 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:27:46,137 llm_VLLMDeployment 2dqbcyec 6d7165c0-0687-45fd-aa22-90f6d189ff44 -- GET / 404 5.0ms
INFO 2025-01-11 03:28:48,411 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 03:28:48,416 llm_VLLMDeployment 2dqbcyec d889f79a-903b-4988-91cb-631c57f07c75 -- GET / 404 9.7ms
INFO 2025-01-11 06:19:08,632 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 06:19:08,640 llm_VLLMDeployment 2dqbcyec 63c3d3f3-069b-4247-a86e-24662f824ad2 -- GET / 404 12.0ms
INFO 2025-01-11 06:19:18,520 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
INFO 2025-01-11 06:19:18,524 llm_VLLMDeployment 2dqbcyec de4da3e5-e057-47c7-a149-2030a0f4c719 -- GET / 404 6.2ms
INFO 2025-01-11 06:19:25,200 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:25,246 llm_VLLMDeployment 2dqbcyec ecf2c282-b83d-4652-a084-657034db315f -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:25,262 llm_VLLMDeployment 2dqbcyec ecf2c282-b83d-4652-a084-657034db315f -- POST /v1/chat/completions 500 63.5ms
INFO 2025-01-11 06:19:25,761 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:25,779 llm_VLLMDeployment 2dqbcyec f1485b1b-f16b-477f-955d-e0e0a805b417 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:25,787 llm_VLLMDeployment 2dqbcyec f1485b1b-f16b-477f-955d-e0e0a805b417 -- POST /v1/chat/completions 500 28.5ms
INFO 2025-01-11 06:19:26,794 llm_VLLMDeployment 2dqbcyec -- Started executing request to method '__call__'.
ERROR 2025-01-11 06:19:26,807 llm_VLLMDeployment 2dqbcyec fde34a5a-a56a-433f-a567-1b3a16c8d903 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 398, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 789, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 573, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 510, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1386, in call_user_method
    await self._call_func_or_gen(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1093, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 497, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 113, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-01-11_02-49-34_762803_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 73, in create_chat_completion
    self.openai_serving_chat = OpenAIServingChat(
                               ^^^^^^^^^^^^^^^^^^
TypeError: OpenAIServingChat.__init__() missing 1 required keyword-only argument: 'chat_template_content_format'
INFO 2025-01-11 06:19:26,814 llm_VLLMDeployment 2dqbcyec fde34a5a-a56a-433f-a567-1b3a16c8d903 -- POST /v1/chat/completions 500 22.2ms

[mdklause]

To fix this TypeError: You can just add chat_template_content_format='auto' in the declaration of OpenAIServingChat

[torsteinelv]

thanks alot this helped me get to the next problem, I cannot get arround this and I see there are other peope having the same problem, it can be a change for a newer version of vllm. Currently Im trying with vllm v0.6.6.post1, I beleave it works if I will go with and older version.

INFO 2025-03-12 22:57:58,688 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- Request: messages=[{'content': '### Task:\nYou are an autocompletion system. Continue the text in `<text>` based on the **completion type** in `<type>` and the given language.  \n\n### **Instructions**:\n1. Analyze `<text>` for context and meaning.  \n2. Use `<type>` to guide your output:  \n   - **General**: Provide a natural, concise continuation.  \n   - **Search Query**: Complete as if generating a realistic search query.  \n3. Start as if you are directly continuing `<text>`. Do **not** repeat, paraphrase, or respond as a model. Simply complete the text.  \n4. Ensure the continuation:\n   - Flows naturally from `<text>`.  \n   - Avoids repetition, overexplaining, or unrelated ideas.  \n5. If unsure, return: `{ "text": "" }`.  \n\n### **Output Rules**:\n- Respond only in JSON format: `{ "text": "<your_completion>" }`.\n\n### **Examples**:\n#### Example 1:  \nInput:  \n<type>General</type>  \n<text>The sun was setting over the horizon, painting the sky</text>  \nOutput:  \n{ "text": "with vibrant shades of orange and pink." }\n\n#### Example 2:  \nInput:  \n<type>Search Query</type>  \n<text>Top-rated restaurants in</text>  \nOutput:  \n{ "text": "New York City for Italian cuisine." }  \n\n---\n### Context:\n<chat_history>\nUSER: test\nASSISTANT: \n</chat_history>\n<type>search query</type>  \n<text>test</text>  \n#### Output:\n', 'role': 'user'}] model='test' frequency_penalty=0.0 logit_bias=None logprobs=False top_logprobs=0 max_tokens=None max_completion_tokens=None n=1 presence_penalty=0.0 response_format=None seed=None stop=[] stream=False stream_options=None temperature=None top_p=None tools=None tool_choice='none' parallel_tool_calls=False user=None best_of=None use_beam_search=False top_k=None min_p=None repetition_penalty=None length_penalty=1.0 stop_token_ids=[] include_stop_str_in_output=False ignore_eos=False min_tokens=0 skip_special_tokens=True spaces_between_special_tokens=True truncate_prompt_tokens=None prompt_logprobs=None echo=False add_generation_prompt=True continue_final_message=False add_special_tokens=False documents=None chat_template=None chat_template_kwargs=None guided_json=None guided_regex=None guided_choice=None guided_grammar=None guided_decoding_backend=None guided_whitespace_pattern=None priority=0 request_id='781b28c4316c405d97bff03d93ed9154' logits_processors=None
ERROR 2025-03-12 22:57:58,690 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 474, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 882, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 648, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 585, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1610, in call_user_method
    result, sync_gen_consumed = await self._call_func_or_gen(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1328, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 502, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-03-12_12-59-05_244804_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 86, in create_chat_completion
    generator = await self.openai_serving_chat.create_chat_completion(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 112, in create_chat_completion
    error_check_ret = await self._check_model(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 206, in _check_model
    if self._is_model_supported(request.model):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in _is_model_supported
    return any(model.name == model_name for model in self.base_model_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in <genexpr>
    return any(model.name == model_name for model in self.base_model_paths)
               ^^^^^^^^^^
AttributeError: 'str' object has no attribute 'name'
INFO 2025-03-12 22:57:58,793 llm_VLLMDeployment 2je6s29h e02a48fd-822a-48aa-a529-b5ba84ea1ff3 -- POST /v1/chat/completions 500 171.7ms
INFO 2025-03-12 22:57:59,067 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- Request: messages=[{'content': 'test', 'role': 'user'}] model='test' frequency_penalty=0.0 logit_bias=None logprobs=False top_logprobs=0 max_tokens=None max_completion_tokens=None n=1 presence_penalty=0.0 response_format=None seed=None stop=[] stream=True stream_options=None temperature=None top_p=None tools=None tool_choice='none' parallel_tool_calls=False user=None best_of=None use_beam_search=False top_k=None min_p=None repetition_penalty=None length_penalty=1.0 stop_token_ids=[] include_stop_str_in_output=False ignore_eos=False min_tokens=0 skip_special_tokens=True spaces_between_special_tokens=True truncate_prompt_tokens=None prompt_logprobs=None echo=False add_generation_prompt=True continue_final_message=False add_special_tokens=False documents=None chat_template=None chat_template_kwargs=None guided_json=None guided_regex=None guided_choice=None guided_grammar=None guided_decoding_backend=None guided_whitespace_pattern=None priority=0 request_id='a5f5d43d9b1a40bc9b35d8209367e187' logits_processors=None
ERROR 2025-03-12 22:57:59,068 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- Request failed.
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 474, in _handle_errors_and_metrics
    yield _status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 882, in _wrap_user_method_call
    yield status_code_callback
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 648, in handle_request_with_rejection
    async for result in self._call_user_generator(
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 585, in _call_user_generator
    raise e from None
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1610, in call_user_method
    result, sync_gen_consumed = await self._call_func_or_gen(
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/replica.py", line 1328, in _call_func_or_gen
    result = await result
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/ray/serve/_private/http_util.py", line 502, in __call__
    await self._asgi_app(
  File "/usr/local/lib/python3.12/dist-packages/fastapi/applications.py", line 1054, in __call__
    await super().__call__(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/applications.py", line 112, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 187, in __call__
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/errors.py", line 165, in __call__
    await self.app(scope, receive, _send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/middleware/exceptions.py", line 62, in __call__
    await wrap_app_handling_exceptions(self.app, conn)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 715, in __call__
    await self.middleware_stack(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 735, in app
    await route.handle(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 288, in handle
    await self.app(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 76, in app
    await wrap_app_handling_exceptions(app, request)(scope, receive, send)
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 53, in wrapped_app
    raise exc
  File "/usr/local/lib/python3.12/dist-packages/starlette/_exception_handler.py", line 42, in wrapped_app
    await app(scope, receive, sender)
  File "/usr/local/lib/python3.12/dist-packages/starlette/routing.py", line 73, in app
    response = await f(request)
               ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 301, in app
    raw_response = await run_endpoint_function(
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/fastapi/routing.py", line 212, in run_endpoint_function
    return await dependant.call(**values)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/tmp/ray/session_2025-03-12_12-59-05_244804_1/runtime_resources/working_dir_files/https_github_com_torsteinelv_ray-serve_archive_refs_heads_main/ray-serve/vllm_engine.py", line 86, in create_chat_completion
    generator = await self.openai_serving_chat.create_chat_completion(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_chat.py", line 112, in create_chat_completion
    error_check_ret = await self._check_model(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 206, in _check_model
    if self._is_model_supported(request.model):
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in _is_model_supported
    return any(model.name == model_name for model in self.base_model_paths)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/dist-packages/vllm/entrypoints/openai/serving_engine.py", line 665, in <genexpr>
    return any(model.name == model_name for model in self.base_model_paths)
               ^^^^^^^^^^
AttributeError: 'str' object has no attribute 'name'
INFO 2025-03-12 22:57:59,073 llm_VLLMDeployment 2je6s29h 0f5253d5-60b0-4838-a944-9f636911d96a -- POST /v1/chat/completions 500 14.5ms

[omrishiv]

@torsteinelv did you ever figure this out? I'm working through a similar issue right now

[torsteinelv]

Yes i did it like this

apiVersion: leaderworkerset.x-k8s.io/v1
kind: LeaderWorkerSet
metadata:
  name: vllm
  namespace: vllm
spec:
  replicas: 0
  leaderWorkerTemplate:
    size: 3
    restartPolicy: RecreateGroupOnPodRestart
    leaderTemplate:
      metadata:
        labels:
          role: leader
      spec:
        nodeSelector:
          kubernetes.io/hostname: k8s-w3
        containers:
          - name: vllm-leader
            image: docker.io/vllm/vllm-openai:v0.8.1
            env:
              - name: VLLM_USE_V1
                value: "0"
              - name: HUGGING_FACE_HUB_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: hf-token-secret
                    key: token
            command:
              - sh
              - -c
              - >
                bash /vllm-workspace/examples/online_serving/multi-node-serving.sh leader --ray_cluster_size=$(LWS_GROUP_SIZE);
                python3 -m vllm.entrypoints.openai.api_server --port 8080 --model unsloth/Phi-3-medium-4k-instruct-bnb-4bit --quantization bitsandbytes --load-format bitsandbytes --max-model-len 1024 --enforce-eager --trust-remote-code --dtype float16 --gpu-memory-utilization 0.8 --tensor-parallel-size 1 --pipeline_parallel_size 3
            resources:
              limits:
                nvidia.com/gpu: "1"
                ephemeral-storage: 10Gi
              requests:
                ephemeral-storage: 10Gi
            ports:
              - containerPort: 8080
            readinessProbe:
              tcpSocket:
                port: 8080
              initialDelaySeconds: 600
              periodSeconds: 10
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 10Gi
    workerTemplate:
      metadata:
        labels:
          role: worker
      spec:
        affinity:
          nodeAffinity:
            preferredDuringSchedulingIgnoredDuringExecution:
              - weight: 1
                preference:
                  matchExpressions:
                    - key: kubernetes.io/hostname
                      operator: In
                      values:
                        - k8s-w1
                        - k8s-w2
        topologySpreadConstraints:
          - maxSkew: 1
            topologyKey: kubernetes.io/hostname
            whenUnsatisfiable: DoNotSchedule
            labelSelector:
              matchLabels:
                role: worker
        containers:
          - name: vllm-worker
            image: ghcr.io/sasha0552/vllm:v0.8.1
            command:
              - sh
              - -c
              - |
                pip install -U ray==2.43.0 ray[default]==2.43.0 && \
                echo "Forced Ray version to 2.43.0" && \
                sleep 15 && \
                bash /vllm-workspace/examples/online_serving/multi-node-serving.sh worker --ray_address=$(LWS_LEADER_ADDRESS) && \
                tail -f /dev/null
            resources:
              limits:
                nvidia.com/gpu: "1"
                ephemeral-storage: 10Gi
              requests:
                ephemeral-storage: 10Gi
            env:
              - name: HUGGING_FACE_HUB_TOKEN
                valueFrom:
                  secretKeyRef:
                    name: hf-token-secret
                    key: token
            volumeMounts:
              - mountPath: /dev/shm
                name: dshm
        volumes:
          - name: dshm
            emptyDir:
              medium: Memory
              sizeLimit: 10Gi
---
apiVersion: v1
kind: Service
metadata:
  name: vllm-leader
  namespace: vllm
spec:
  ports:
    - name: http
      port: 8080
      protocol: TCP
      targetPort: 8080
  selector:
    leaderworkerset.sigs.k8s.io/name: vllm
    role: leader
  type: LoadBalancer

[omrishiv]

Thank you! I actually did manage to sort mine out without LWS. I’ll follow up with my solution

[torsteinelv]

Yea i did also but more easy and i guess better with lws

[omrishiv]

As I know I'll eventually run into this issue again if I don't document it, vLLM ~0.6 version needs
base_model_paths = [BaseModelPath(name=self.engine_args.model, model_path=self.engine_args.model)] for the base_model_paths for OpenAI.