BUG: NCCL error:

[ye7love7]

使用v0.12.0docker镜像部署，启动命令如下：
sudo docker run -d -v /home/tskj/MOD/:/home/MOD/ -e XINFERENCE_HOME=/home/MOD -p 9997:9997 --gpus all xprobe/xinference:v0.12.0 xinference-local -H 0.0.0.0 --log-level debug
我是8卡，选择8后，模型报错：
2024-06-12 05:08:11,767 xinference.core.worker 95 DEBUG Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7f013c4aa700>_uid': 'gpt-3.5-turbo-1-0', 'model_name': 'Qwen1.5-110B-Chat', 'model_size_in_billions': 110, 'model_format': 'pytorch', 'quantization': 'none', 'model_engl_type': 'LLM', 'n_gpu': 8, 'request_limits': None, 'peft_model_config': None, 'gpu_idx': None, 'gpu_memory_utilization': 0.9, 'max_model_len': 32768}
2024-06-12 05:08:11,767 xinference.core.worker 95 DEBUG GPU selected: [0, 1, 2, 3, 4, 5, 6, 7] for model gpt-3.5-turbo-1-0
2024-06-12 05:08:15,436 xinference.model.llm.core 95 DEBUG Launching gpt-3.5-turbo-1-0 with VLLMChatModel
2024-06-12 05:08:15,437 xinference.model.llm.llm_family 95 INFO Caching from URI: /home/MOD/Qwen/Qwen1.5-110B-Chat
2024-06-12 05:08:15,437 xinference.model.llm.llm_family 95 INFO Cache /home/MOD/Qwen/Qwen1.5-110B-Chat exists
2024-06-12 05:08:15,461 xinference.model.llm.vllm.core 210 INFO Loading gpt-3.5-turbo with following model config: {'gpu_memory_utilization': 0.9, 'max 'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 8, 'block_size': 16, 'swap_space': 4, 'max_num_seqs': 256, 'quantization': Nose. Lora count: 0.
2024-06-12 05:08:17,542 WARNING services.py:2009 -- WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 67067904 bytes avharm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by p10.24gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
2024-06-12 05:08:18,666 INFO worker.py:1753 -- Started a local Ray instance.
INFO 06-12 05:08:20 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/home/MOD/Qwen/Qwen1.5-110B-Chat', speculative_config=None, D/Qwen/Qwen1.5-110B-Chat', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=Truat16, max_seq_len=32768, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=False, quantization=None, enforcache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_modeln/Qwen1.5-110B-Chat)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 06-12 05:08:45 utils.py:618] Found nccl from library libnccl.so.2
INFO 06-12 05:08:45 pynccl.py:65] vLLM is using nccl==2.20.5
(RayWorkerWrapper pid=6318) INFO 06-12 05:08:45 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=6318) INFO 06-12 05:08:45 pynccl.py:65] vLLM is using nccl==2.20.5
ERROR 06-12 05:08:46 worker_base.py:148] Error executing method init_device. This might cause deadlock in distributed execution.
ERROR 06-12 05:08:46 worker_base.py:148] Traceback (most recent call last):
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 140, in execute_method
ERROR 06-12 05:08:46 worker_base.py:148] return executor(*args, **kwargs)
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 114, in init_device
ERROR 06-12 05:08:46 worker_base.py:148] init_worker_distributed_environment(self.parallel_config, self.rank,
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 349, in init_worker_distributed_envir
ERROR 06-12 05:08:46 worker_base.py:148] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 239, in ensure_model_par
ERROR 06-12 05:08:46 worker_base.py:148] initialize_model_parallel(tensor_model_parallel_size,
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 191, in initialize_model
ERROR 06-12 05:08:46 worker_base.py:148] _TP_PYNCCL_COMMUNICATOR = PyNcclCommunicator(
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl.py", line 94, in __in
ERROR 06-12 05:08:46 worker_base.py:148] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 244nk
ERROR 06-12 05:08:46 worker_base.py:148] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/pynccl_wrapper.py", line 223
ERROR 06-12 05:08:46 worker_base.py:148] raise RuntimeError(f"NCCL error: {error_str}")
ERROR 06-12 05:08:46 worker_base.py:148] RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details)
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] Error executing method init_device. This might cause deadlock in distributed execution
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] Traceback (most recent call last):
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 140,
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] return executor(*args, **kwargs)
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 114, in i
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] init_worker_distributed_environment(self.parallel_config, self.rank,
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 349, in ited_environment
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] ensure_model_parallel_initialized(parallel_config.tensor_parallel_size,
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", lmodel_parallel_initialized
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] initialize_model_parallel(tensor_model_parallel_size,
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", lize_model_parallel
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] _TP_PYNCCL_COMMUNICATOR = PyNcclCommunicator(
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/, in init
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] self.comm: ncclComm_t = self.nccl.ncclCommInitRank(
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/ line 244, in ncclCommInitRank
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] self.NCCL_CHECK(self._funcs["ncclCommInitRank"](ctypes.byref(comm),
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/device_communicators/ line 223, in NCCL_CHECK
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] raise RuntimeError(f"NCCL error: {error_str}")
(RayWorkerWrapper pid=6318) ERROR 06-12 05:08:46 worker_base.py:148] RuntimeError: NCCL error: unhandled system error (run with NCCL_DEBUG=INFO for details

服务器nccl如下：chatchat) yqga@dnb:~$ dpkg -l|grep nccl

ii libnccl-dev 2.21.5-1+cuda12.5 amd64 NVIDIA Collective Communication Library (NCCL) Development Files
ii libnccl2 2.21.5-1+cuda12.5 amd64 NVIDIA Collective Communication Library (NCCL) Runtime
ii nccl-local-repo-ubuntu2204-2.21.5-cuda12.5 1.0-1 amd64 nccl-local repository configuration files

[ye7love7]

英伟达配置如图

[linqingxu]

+1

[ye7love7]

很急啊，这个产品挺好的，想用，求回复！！！

…

---Original--- From: ***@***.***&gt; Date: Mon, Jun 17, 2024 16:17 PM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [xorbitsai/inference] BUG: NCCL error: (Issue #1622) +1 — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[RandyChen1985]

+1，
以前的版本vllm可以部署，现在都不可以了， 也是报nccl错误。
单卡上用vllm可以，但是选择多卡后就出现nccl错误了。

transform可以正常部署（单卡多卡都可以）。

#环境信息如下

root@41be57132056:/workspace# pip list | grep nccl
nvidia-nccl-cu12 2.20.5
root@41be57132056:/workspace# pip list | grep torch
torch 2.3.0
torchaudio 2.3.0
torchelastic 0.2.2
torchvision 0.18.0
vector-quantize-pytorch 1.14.24
root@41be57132056:/workspace#

(base) root@bm-2203ajt:# dpkg -l|grep nccl
ii libnccl-dev 2.19.3-1+cuda12.3 amd64 NVIDIA Collective Communication Library (NCCL) Development Files
ii libnccl2 2.19.3-1+cuda12.3 amd64 NVIDIA Collective Communication Library (NCCL) Runtime
(base) root@bm-2203ajt:#

#错误如下：

2024-06-19 01:57:01,903 xinference.api.restful_api 1 ERROR [address=0.0.0.0:41855, pid=35109] NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:
Error while creating shared memory segment /dev/shm/nccl-jaXv5Z (size 5767520)
Traceback (most recent call last):
File "/opt/conda/lib/python3.10/site-packages/xinference/api/restful_api.py", line 770, in launch_model
model_uid = await (await self._get_supervisor_ref()).launch_builtin_model(
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
�
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 837, in launch_builtin_model
await _launch_model()
File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 801, in _launch_model
await _launch_one_model(rep_model_uid)
File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 782, in _launch_one_model
await worker_ref.launch_builtin_model(
File "xoscar/core.pyx", line 284, in __pyx_actor_method_wrapper
async with lock:
File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper
result = await result
File "/opt/conda/lib/python3.10/site-packages/xinference/core/utils.py", line 45, in wrapped
ret = await func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 665, in launch_builtin_model
await model_ref.load()
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send
return self._process_result_message(result)
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message
raise message.as_instanceof_cause()
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send
result = await self._run_coro(message.message_id, coro)
File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro
return await coro
File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive
return await super().on_receive(message) # type: ignore
File "xoscar/core.pyx", line 558, in on_receive
�
raise ex
File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive
async with self._lock:
File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive
with debug_async_timeout('actor_lock_timeout',
File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive
result = await result
File "/opt/conda/lib/python3.10/site-packages/xinference/core/model.py", line 277, in load
self._model.load()
File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 230, in load
self._engine = AsyncLLMEngine.from_engine_args(engine_args)
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 386, in from_engine_args
engine = cls(
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 340, in init
self.engine = self._init_engine(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 462, in _init_engine
return engine_class(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 222, in init
self.model_executor = executor_class(
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 317, in init
super().init(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in init
super().init(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in init
self._init_executor()
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 40, in _init_executor
self._init_workers_ray(placement_group)
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 171, in _init_workers_ray
self._run_workers("init_device")
File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 246, in _run_workers
driver_worker_output = self.driver_worker.execute_method(
File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 149, in execute_method
raise e
File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 140, in execute_method
return executor(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 114, in init_device
init_worker_distributed_environment(self.parallel_config, self.rank,
File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 346, in init_worker_distributed_environment
init_distributed_environment(parallel_config.world_size, rank,
File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 122, in init_distributed_environment
torch.distributed.all_reduce(data)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper
return func(*args, **kwargs)
File "/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2219, in all_reduce
work = group.allreduce([tensor], opts)
torch.distributed.DistBackendError: [address=0.0.0.0:41855, pid=35109] NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5
ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error.
Last error:

[ye7love7]

我也是，想换新版本，看它出了fncall

…

---Original--- From: ***@***.***&gt; Date: Wed, Jun 19, 2024 10:00 AM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [xorbitsai/inference] BUG: NCCL error: (Issue #1622) +1， 以前的版本vllm可以部署，现在都不可以了，transform可以部署。 也是报nccl错误。 2024-06-19 01:57:01,903 xinference.api.restful_api 1 ERROR [address=0.0.0.0:41855, pid=35109] NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5 ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. Last error: Error while creating shared memory segment /dev/shm/nccl-jaXv5Z (size 5767520) Traceback (most recent call last): File "/opt/conda/lib/python3.10/site-packages/xinference/api/restful_api.py", line 770, in launch_model model_uid = await (await self._get_supervisor_ref()).launch_builtin_model( File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send result = await self._run_coro(message.message_id, coro) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro return await coro File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive � raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive result = await result File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 837, in launch_builtin_model await _launch_model() File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 801, in _launch_model await _launch_one_model(rep_model_uid) File "/opt/conda/lib/python3.10/site-packages/xinference/core/supervisor.py", line 782, in _launch_one_model await worker_ref.launch_builtin_model( File "xoscar/core.pyx", line 284, in __pyx_actor_method_wrapper async with lock: File "xoscar/core.pyx", line 287, in xoscar.core.__pyx_actor_method_wrapper result = await result File "/opt/conda/lib/python3.10/site-packages/xinference/core/utils.py", line 45, in wrapped ret = await func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/xinference/core/worker.py", line 665, in launch_builtin_model await model_ref.load() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 227, in send return self._process_result_message(result) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/context.py", line 102, in _process_result_message raise message.as_instanceof_cause() File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 659, in send result = await self._run_coro(message.message_id, coro) File "/opt/conda/lib/python3.10/site-packages/xoscar/backends/pool.py", line 370, in _run_coro return await coro File "/opt/conda/lib/python3.10/site-packages/xoscar/api.py", line 384, in on_receive return await super().on_receive(message) # type: ignore File "xoscar/core.pyx", line 558, in on_receive � raise ex File "xoscar/core.pyx", line 520, in xoscar.core._BaseActor.on_receive async with self._lock: File "xoscar/core.pyx", line 521, in xoscar.core._BaseActor.on_receive with debug_async_timeout('actor_lock_timeout', File "xoscar/core.pyx", line 526, in xoscar.core._BaseActor.on_receive result = await result File "/opt/conda/lib/python3.10/site-packages/xinference/core/model.py", line 277, in load self._model.load() File "/opt/conda/lib/python3.10/site-packages/xinference/model/llm/vllm/core.py", line 230, in load self._engine = AsyncLLMEngine.from_engine_args(engine_args) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 386, in from_engine_args engine = cls( File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 340, in init self.engine = self._init_engine(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/async_llm_engine.py", line 462, in _init_engine return engine_class(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/vllm/engine/llm_engine.py", line 222, in init self.model_executor = executor_class( File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 317, in init super().init(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/vllm/executor/distributed_gpu_executor.py", line 25, in init super().init(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/vllm/executor/executor_base.py", line 41, in init self._init_executor() File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 40, in _init_executor self._init_workers_ray(placement_group) File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 171, in _init_workers_ray self._run_workers("init_device") File "/opt/conda/lib/python3.10/site-packages/vllm/executor/ray_gpu_executor.py", line 246, in _run_workers driver_worker_output = self.driver_worker.execute_method( File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 149, in execute_method raise e File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker_base.py", line 140, in execute_method return executor(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 114, in init_device init_worker_distributed_environment(self.parallel_config, self.rank, File "/opt/conda/lib/python3.10/site-packages/vllm/worker/worker.py", line 346, in init_worker_distributed_environment init_distributed_environment(parallel_config.world_size, rank, File "/opt/conda/lib/python3.10/site-packages/vllm/distributed/parallel_state.py", line 122, in init_distributed_environment torch.distributed.all_reduce(data) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/c10d_logger.py", line 75, in wrapper return func(*args, **kwargs) File "/opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py", line 2219, in all_reduce work = group.allreduce([tensor], opts) torch.distributed.DistBackendError: [address=0.0.0.0:41855, pid=35109] NCCL error in: ../torch/csrc/distributed/c10d/ProcessGroupNCCL.cpp:1970, unhandled system error (run with NCCL_DEBUG=INFO for details), NCCL version 2.20.5 ncclSystemError: System call (e.g. socket, malloc) or external library call failed or device error. Last error: — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[RandyChen1985]

我好像找到原因了，docker 启动的时候加个参数 --shm-size 20g， 如下：

docker run -d
-e XINFERENCE_MODEL_SRC=modelscope
-v /data/xinference_llm/.xinference:/root/.xinference
-v /data/xinference_llm/.cache/huggingface:/root/.cache/huggingface
-v /data/xinference_llm/.cache/modelscope:/root/.cache/modelscope
-p 9997:9997
--gpus all
--name xinference
--shm-size 20g \
xprobe/xinference:v0.12.0
xinference-local -H 0.0.0.0 --log-level debug

环境

root@0cf3d74a2d6f:/workspace# pip list | grep nccl
nvidia-nccl-cu12 2.20.5
root@0cf3d74a2d6f:/workspace# pip list | grep torch
torch 2.3.0
torchaudio 2.3.0
torchelastic 0.2.2
torchvision 0.18.0
vector-quantize-pytorch 1.14.24
root@0cf3d74a2d6f:/workspace#

启动日志。。

2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7f0f0d780680>,), kwargs: {'model_uid': 'qwen2-chat-72b-1-0', 'model_name': 'qwen2-instruct', 'model_size_in_billions': 72, 'model_format': 'pytorch', 'quantization': 'none', 'model_engine': 'vLLM', 'model_type': 'LLM', 'n_gpu': 8, 'request_limits': None, 'peft_model_config': None, 'gpu_idx': None}
2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG GPU selected: [0, 1, 2, 3, 4, 5, 6, 7] for model qwen2-chat-72b-1-0
2024-06-19 06:19:07,229 xinference.model.llm.core 95 DEBUG Launching qwen2-chat-72b-1-0 with VLLMChatModel
2024-06-19 06:19:07,229 xinference.model.llm.llm_family 95 INFO Caching from Modelscope: qwen/Qwen2-72B-Instruct
2024-06-19 06:19:07,230 xinference.model.llm.llm_family 95 INFO Cache /root/.xinference/cache/qwen2-instruct-pytorch-72b exists
2024-06-19 06:19:07,251 xinference.model.llm.vllm.core 9209 INFO Loading qwen2-chat-72b with following model config: {'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 8, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': 4096}Enable lora: False. Lora count: 0.
2024-06-19 06:19:10,272 INFO worker.py:1753 -- Started a local Ray instance.
INFO 06-19 06:19:11 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/root/.xinference/cache/qwen2-instruct-pytorch-72b', speculative_config=None, tokenizer='/root/.xinference/cache/qwen2-instruct-pytorch-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/.xinference/cache/qwen2-instruct-pytorch-72b)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5
INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5
WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(RayWorkerWrapper pid=16627) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
INFO 06-19 06:19:34 model_runner.py:146] Loading model weights took 16.9987 GB
(RayWorkerWrapper pid=16627) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 [repeated 6x across cluster]
(RayWorkerWrapper pid=17026) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 6x across cluster]
INFO 06-19 06:19:44 distributed_gpu_executor.py:56] # GPU blocks: 4316, # CPU blocks: 6553
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB [repeated 6x across cluster]
INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
IN

[ye7love7]

是docker的共享内存问题么？我还问了gpt，还以为是nccl宿主机和容器不同导致

…

---Original--- From: ***@***.***&gt; Date: Wed, Jun 19, 2024 14:26 PM To: ***@***.***&gt;; Cc: ***@***.******@***.***&gt;; Subject: Re: [xorbitsai/inference] BUG: NCCL error: (Issue #1622) 我好像找到原因了，docker 启动的时候加个参数 --shm-size 20g， 如下： docker run -d -e XINFERENCE_MODEL_SRC=modelscope -v /data/xinference_llm/.xinference:/root/.xinference -v /data/xinference_llm/.cache/huggingface:/root/.cache/huggingface -v /data/xinference_llm/.cache/modelscope:/root/.cache/modelscope -p 9997:9997 --gpus all --name xinference --shm-size 20g \ xprobe/xinference:v0.12.0 xinference-local -H 0.0.0.0 --log-level debug 环境 ***@***.***:/workspace# pip list | grep nccl nvidia-nccl-cu12 2.20.5 ***@***.***:/workspace# pip list | grep torch torch 2.3.0 torchaudio 2.3.0 torchelastic 0.2.2 torchvision 0.18.0 vector-quantize-pytorch 1.14.24 ***@***.***:/workspace# 启动日志。。 2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7f0f0d780680&gt;,), kwargs: {'model_uid': 'qwen2-chat-72b-1-0', 'model_name': 'qwen2-instruct', 'model_size_in_billions': 72, 'model_format': 'pytorch', 'quantization': 'none', 'model_engine': 'vLLM', 'model_type': 'LLM', 'n_gpu': 8, 'request_limits': None, 'peft_model_config': None, 'gpu_idx': None} 2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG GPU selected: [0, 1, 2, 3, 4, 5, 6, 7] for model qwen2-chat-72b-1-0 2024-06-19 06:19:07,229 xinference.model.llm.core 95 DEBUG Launching qwen2-chat-72b-1-0 with VLLMChatModel 2024-06-19 06:19:07,229 xinference.model.llm.llm_family 95 INFO Caching from Modelscope: qwen/Qwen2-72B-Instruct 2024-06-19 06:19:07,230 xinference.model.llm.llm_family 95 INFO Cache /root/.xinference/cache/qwen2-instruct-pytorch-72b exists 2024-06-19 06:19:07,251 xinference.model.llm.vllm.core 9209 INFO Loading qwen2-chat-72b with following model config: {'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 8, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': 4096}Enable lora: False. Lora count: 0. 2024-06-19 06:19:10,272 INFO worker.py:1753 -- Started a local Ray instance. INFO 06-19 06:19:11 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/root/.xinference/cache/qwen2-instruct-pytorch-72b', speculative_config=None, tokenizer='/root/.xinference/cache/qwen2-instruct-pytorch-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/.xinference/cache/qwen2-instruct-pytorch-72b) Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained. INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 (RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 (RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. (RayWorkerWrapper pid=16627) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. INFO 06-19 06:19:34 model_runner.py:146] Loading model weights took 16.9987 GB (RayWorkerWrapper pid=16627) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB (RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.) (RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 [repeated 6x across cluster] (RayWorkerWrapper pid=17026) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 6x across cluster] INFO 06-19 06:19:44 distributed_gpu_executor.py:56] # GPU blocks: 4316, # CPU blocks: 6553 (RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. (RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage. (RayWorkerWrapper pid=16937) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB [repeated 6x across cluster] INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI. INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage. IN image.png (view on web) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.Message ID: ***@***.***&gt;

[ZimaBlueee]

我好像找到原因了，docker 启动的时候加个参数 --shm-size 20g， 如下：

docker run -d
-e XINFERENCE_MODEL_SRC=modelscope -e XINFERENCE_MODEL_SRC=模型范围
-v /data/xinference_llm/.xinference:/root/.xinference -v /data/xinference_llm/.xinference：/root/.xinference
-v /data/xinference_llm/.cache/huggingface:/root/.cache/huggingface -v /数据/xinference_llm/.cache/huggingface：/root/.cache/huggingface
-v /data/xinference_llm/.cache/modelscope:/root/.cache/modelscope -v /数据/xinference_llm/.cache/modelscope：/root/.cache/modelscope
-p 9997:9997 -第 9997 页：9997
--gpus all --GPU 全部
--name xinference --名称 xinference
--shm-size 20g
xprobe/xinference:v0.12.0 xprobe/xinference：v0.12.0
xinference-local -H 0.0.0.0 --log-level debug

环境

root@0cf3d74a2d6f:/workspace# pip list | grep ncclroot@0cf3d74a2d6f：/workspace# pip 列表 |GREP NCCL公司
nvidia-nccl-cu12 2.20.5 英伟达-NCCL-CU12 2.20.5
root@0cf3d74a2d6f:/workspace# pip list | grep torchroot@0cf3d74a2d6f：/workspace# pip 列表 |GREP 火炬
torch 2.3.0 火炬 2.3.0
torchaudio 2.3.0 火炬音频 2.3.0
torchelastic 0.2.2 火炬弹性 0.2.2
torchvision 0.18.0 火炬视界 0.18.0
vector-quantize-pytorch 1.14.24向量化 pytorch 1.14.24
root@0cf3d74a2d6f:/workspace#root@0cf3d74a2d6f：/工作区#

启动日志。。

2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7f0f0d780680>,), kwargs: {'model_uid': 'qwen2-chat-72b-1-0', 'model_name': 'qwen2-instruct', 'model_size_in_billions': 72, 'model_format': 'pytorch', 'quantization': 'none', 'model_engine': 'vLLM', 'model_type': 'LLM', 'n_gpu': 8, 'request_limits': None, 'peft_model_config': None, 'gpu_idx': None}2024-06-19 06：18：56,050 xinference.core.worker 95 调试 输入launch_builtin_model， args： （<xinference.core.worker.WorkerActor 对象在 0x7f0f0d780680>，）， kwargs： {'model_uid'： 'qwen2-chat-72b-1-0'， 'model_name'： 'qwen2-instruct'， 'model_size_in_billions'： 72， 'model_format'： 'pytorch'， 'quantization'： 'none'， 'model_engine'： 'vLLM'， 'model_type'： 'LLM'， 'n_gpu'： 8， 'request_limits'： 无， 'peft_model_config'： 无， 'gpu_idx'： 无}
2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG GPU selected: [0, 1, 2, 3, 4, 5, 6, 7] for model qwen2-chat-72b-1-02024-06-19 06：18：56,050 xinference.core.worker 95 调试 已选择的 GPU：[0、1、2、3、4、5、6、7] 适用于型号 qwen2-chat-72b-1-0
2024-06-19 06:19:07,229 xinference.model.llm.core 95 DEBUG Launching qwen2-chat-72b-1-0 with VLLMChatModel2024-06-19 06：19：07,229 xinference.model.llm.core 95 DEBUG 使用 VLLMChatModel 启动 qwen2-chat-72b-1-0
2024-06-19 06:19:07,229 xinference.model.llm.llm_family 95 INFO Caching from Modelscope: qwen/Qwen2-72B-Instruct2024-06-19 06：19：07,229 xinference.model.llm.llm_family 95 INFO 从 Modelscope 缓存：qwen/Qwen2-72B-Instruct
2024-06-19 06:19:07,230 xinference.model.llm.llm_family 95 INFO Cache /root/.xinference/cache/qwen2-instruct-pytorch-72b exists2024-06-19 06：19：07,230 xinference.model.llm.llm_family 95 INFO 缓存 /root/.xinference/cache/qwen2-instruct-pytorch-72b 存在
2024-06-19 06:19:07,251 xinference.model.llm.vllm.core 9209 INFO Loading qwen2-chat-72b with following model config: {'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 8, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': 4096}Enable lora: False. Lora count: 0.2024-06-19 06：19：07,251 xinference.model.llm.vllm.core 9209 INFO 加载 qwen2-chat-72b 具有以下模型配置： {'tokenizer_mode'： 'auto'， 'trust_remote_code'： True， 'tensor_parallel_size'： 8， 'block_size'： 16， 'swap_space'： 4， 'gpu_memory_utilization'： 0.9， 'max_num_seqs'： 256， 'quantization'： None， 'max_model_len'： 4096}Enable lora： False.劳拉计数：0。
2024-06-19 06:19:10,272 INFO worker.py:1753 -- Started a local Ray instance.2024-06-19 06：19：10,272 INFO worker.py:1753 -- 启动本地 Ray 实例。
INFO 06-19 06:19:11 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/root/.xinference/cache/qwen2-instruct-pytorch-72b', speculative_config=None, tokenizer='/root/.xinference/cache/qwen2-instruct-pytorch-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/.xinference/cache/qwen2-instruct-pytorch-72b)INFO 06-19 06：19：11 llm_engine.py：161] 初始化LLM引擎 （v0.4.3） config： model='/root/.xinference/cache/qwen2-instruct-pytorch-72b'， speculative_config=None， tokenizer='/root/.xinference/cache/qwen2-instruct-pytorch-72b'， skip_tokenizer_init=False， tokenizer_mode=auto， revision=None， rope_scaling=None， tokenizer_revision=None， trust_remote_code=True， dtype=torch.bfloat16， max_seq_len=4096， download_dir=None， load_format=LoadFormat.AUTO， tensor_parallel_size=8， disable_custom_all_reduce=False， quantization=None， enforce_eager=False， kv_cache_dtype=auto， quantization_param_path=None， device_config=cuda， decoding_config=DecodingConfig（guided_decoding_backend='outlines'）， seed=0， served_model_name=/root/.xinference/cache/qwen2-instruct-pytorch-72b）
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.词汇表中添加了特殊标记，请确保对相关的单词嵌入进行微调或训练。
INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2INFO 06-19 06：19：29 utils.py:618] 从库 libnccl.so.2 找到 nccl
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2（RayWorkerWrapper pid=16720）INFO 06-19 06：19：29 utils.py:618] 从库 libnccl.so.2 找到 nccl
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5（RayWorkerWrapper pid=16720）INFO 06-19 06：19：29 pynccl.py:65] vLLM 正在使用 nccl==2.20.5
INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5INFO 06-19 06：19：29 pynccl.py:65] vLLM 正在使用 nccl==2.20.5
WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.警告 06-19 06：19：29 custom_all_reduce.py：158] 自定义 allreduce 被禁用，因为它在两个以上的仅限 PCIe 的 GPU 上不受支持。若要静音此警告，请显式指定 disable_custom_all_reduce=True。
(RayWorkerWrapper pid=16627) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.（RayWorkerWrapper pid=16627）警告 06-19 06：19：29 custom_all_reduce.py：158] 自定义 allreduce 被禁用，因为它在两个以上的仅限 PCIe 的 GPU 上不受支持。若要静音此警告，请显式指定 disable_custom_all_reduce=True。
INFO 06-19 06:19:34 model_runner.py:146] Loading model weights took 16.9987 GBINFO 06-19 06：19：34 model_runner.py：146] 加载模型权重需要 16.9987 GB
(RayWorkerWrapper pid=16627) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB（RayWorkerWrapper pid=16627）INFO 06-19 06：19：38 model_runner.py：146] 加载模型权重需要 16.9987 GB
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)（RayWorkerWrapper pid=16937）INFO 06-19 06：19：29 utils.py:618] 从库 libnccl.so.2 中找到 nccl [跨集群重复 6 次]（Ray 默认删除重复日志。设置 RAY_DEDUP_LOGS=0 以禁用重复数据删除，或参 https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication 以获取更多选项。
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 [repeated 6x across cluster]（RayWorkerWrapper pid=16937）INFO 06-19 06：19：29 pynccl.py:65] vLLM 正在使用 nccl==2.20.5 [跨集群重复 6 次]
(RayWorkerWrapper pid=17026) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 6x across cluster]（RayWorkerWrapper pid=17026）警告 06-19 06：19：29 custom_all_reduce.py：158] 自定义 allreduce 被禁用，因为它在两个以上的仅限 PCIe 的 GPU 上不受支持。若要静音此警告，请显式指定 disable_custom_all_reduce=True。[跨集群重复 6 次]
INFO 06-19 06:19:44 distributed_gpu_executor.py:56] # GPU blocks: 4316, # CPU blocks: 6553INFO 06-19 06：19：44 distributed_gpu_executor.py：56] # GPU 块数：4316，# CPU 块数：6553
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.（RayWorkerWrapper pid=17026）INFO 06-19 06：19：47 model_runner.py：854] 捕获 CUDA 图形的模型。如果模型不是静态的，这可能会导致意外后果。要在 eager 模式下运行模型，请在 CLI 中设置 'enforce_eager=True' 或使用'--enforce-eager'。
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.（RayWorkerWrapper pid=17026）INFO 06-19 06：19：47 model_runner.py：858] CUDA 图每个 GPU 可以额外占用 13 GiB 内存。如果内存不足，请考虑减少 gpu_memory_utilization 或强制执行急切模式。您还可以根据需要减少内存使用量 max_num_seqs 。
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB [repeated 6x across cluster]（RayWorkerWrapper pid=16937）INFO 06-19 06：19：38 model_runner.py：146] 加载模型权重占用了 16.9987 GB [跨集群重复 6 倍]
INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.INFO 06-19 06：19：47 model_runner.py：854] 捕获 CUDA 图形的模型。如果模型不是静态的，这可能会导致意外后果。要在 eager 模式下运行模型，请在 CLI 中设置 'enforce_eager=True' 或使用'--enforce-eager'。
INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 1INFO 06-19 06：19：47 model_runner.py：858] CUDA 图可以额外 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.每个 GPU 3 GiB 内存。如果内存不足，请考虑减少 gpu_memory_utilization 或强制执行急切模式。您还可以根据需要减少内存使用量 max_num_seqs 。
IN 在

感谢，配置了这个就成功了，看来是docker的锅

[yuanzhiwei]

-d

试过 xinference docker-compose 分布式部署的吗? 怎么修改shm-size

[yuanzhiwei]

我好像找到原因了，docker 启动的时候加个参数 --shm-size 20g， 如下：

docker run -d
-e XINFERENCE_MODEL_SRC=modelscope
-v /data/xinference_llm/.xinference:/root/.xinference
-v /data/xinference_llm/.cache/huggingface:/root/.cache/huggingface
-v /data/xinference_llm/.cache/modelscope:/root/.cache/modelscope
-p 9997:9997
--gpus all
--name xinference
--shm-size 20g
xprobe/xinference:v0.12.0
xinference-local -H 0.0.0.0 --log-level debug

环境

root@0cf3d74a2d6f:/workspace# pip list | grep nccl
nvidia-nccl-cu12 2.20.5
root@0cf3d74a2d6f:/workspace# pip list | grep torch
torch 2.3.0
torchaudio 2.3.0
torchelastic 0.2.2
torchvision 0.18.0
vector-quantize-pytorch 1.14.24
root@0cf3d74a2d6f:/workspace#

启动日志。。

2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG Enter launch_builtin_model, args: (<xinference.core.worker.WorkerActor object at 0x7f0f0d780680>,), kwargs: {'model_uid': 'qwen2-chat-72b-1-0', 'model_name': 'qwen2-instruct', 'model_size_in_billions': 72, 'model_format': 'pytorch', 'quantization': 'none', 'model_engine': 'vLLM', 'model_type': 'LLM', 'n_gpu': 8, 'request_limits': None, 'peft_model_config': None, 'gpu_idx': None}
2024-06-19 06:18:56,050 xinference.core.worker 95 DEBUG GPU selected: [0, 1, 2, 3, 4, 5, 6, 7] for model qwen2-chat-72b-1-0
2024-06-19 06:19:07,229 xinference.model.llm.core 95 DEBUG Launching qwen2-chat-72b-1-0 with VLLMChatModel
2024-06-19 06:19:07,229 xinference.model.llm.llm_family 95 INFO Caching from Modelscope: qwen/Qwen2-72B-Instruct
2024-06-19 06:19:07,230 xinference.model.llm.llm_family 95 INFO Cache /root/.xinference/cache/qwen2-instruct-pytorch-72b exists
2024-06-19 06:19:07,251 xinference.model.llm.vllm.core 9209 INFO Loading qwen2-chat-72b with following model config: {'tokenizer_mode': 'auto', 'trust_remote_code': True, 'tensor_parallel_size': 8, 'block_size': 16, 'swap_space': 4, 'gpu_memory_utilization': 0.9, 'max_num_seqs': 256, 'quantization': None, 'max_model_len': 4096}Enable lora: False. Lora count: 0.
2024-06-19 06:19:10,272 INFO worker.py:1753 -- Started a local Ray instance.
INFO 06-19 06:19:11 llm_engine.py:161] Initializing an LLM engine (v0.4.3) with config: model='/root/.xinference/cache/qwen2-instruct-pytorch-72b', speculative_config=None, tokenizer='/root/.xinference/cache/qwen2-instruct-pytorch-72b', skip_tokenizer_init=False, tokenizer_mode=auto, revision=None, rope_scaling=None, tokenizer_revision=None, trust_remote_code=True, dtype=torch.bfloat16, max_seq_len=4096, download_dir=None, load_format=LoadFormat.AUTO, tensor_parallel_size=8, disable_custom_all_reduce=False, quantization=None, enforce_eager=False, kv_cache_dtype=auto, quantization_param_path=None, device_config=cuda, decoding_config=DecodingConfig(guided_decoding_backend='outlines'), seed=0, served_model_name=/root/.xinference/cache/qwen2-instruct-pytorch-72b)
Special tokens have been added in the vocabulary, make sure the associated word embeddings are fine-tuned or trained.
INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2
(RayWorkerWrapper pid=16720) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5
INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5
WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
(RayWorkerWrapper pid=16627) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly.
INFO 06-19 06:19:34 model_runner.py:146] Loading model weights took 16.9987 GB
(RayWorkerWrapper pid=16627) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 utils.py:618] Found nccl from library libnccl.so.2 [repeated 6x across cluster] (Ray deduplicates logs by default. Set RAY_DEDUP_LOGS=0 to disable log deduplication, or see https://docs.ray.io/en/master/ray-observability/user-guides/configure-logging.html#log-deduplication for more options.)
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:29 pynccl.py:65] vLLM is using nccl==2.20.5 [repeated 6x across cluster]
(RayWorkerWrapper pid=17026) WARNING 06-19 06:19:29 custom_all_reduce.py:158] Custom allreduce is disabled because it's not supported on more than two PCIe-only GPUs. To silence this warning, specify disable_custom_all_reduce=True explicitly. [repeated 6x across cluster]
INFO 06-19 06:19:44 distributed_gpu_executor.py:56] # GPU blocks: 4316, # CPU blocks: 6553
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
(RayWorkerWrapper pid=17026) INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
(RayWorkerWrapper pid=16937) INFO 06-19 06:19:38 model_runner.py:146] Loading model weights took 16.9987 GB [repeated 6x across cluster]
INFO 06-19 06:19:47 model_runner.py:854] Capturing the model for CUDA graphs. This may lead to unexpected consequences if the model is not static. To run the model in eager mode, set 'enforce_eager=True' or use '--enforce-eager' in the CLI.
INFO 06-19 06:19:47 model_runner.py:858] CUDA graphs can take additional 13 GiB memory per GPU. If you are running out of memory, consider decreasing gpu_memory_utilization or enforcing eager mode. You can also reduce the max_num_seqs as needed to decrease memory usage.
IN

请问试过 xinference docker-compose 多worker分布式部署的吗? 怎么修改shm-size