[Bug] IB Error after removing requirement for `CU_DEVICE_ATTRIBUTE_HANDLE_TYPE_FABRIC_SUPPORTED`

[liangyuRain]

Hi, the following code is broken right after a3d8d68

import cupy as cp
from mscclpp import TcpBootstrap, Communicator, Transport
from mscclpp._mscclpp import RawGpuBuffer
bootstrap = TcpBootstrap.create(0, 1)
comm = Communicator(bootstrap)

cp.cuda.Device(0).use()
memory = RawGpuBuffer(1024 * 1024)
data_ptr = memory.data()
comm.register_memory(data_ptr, 1024 * 1024, Transport.IB0)

Traceback (most recent call last):                                                                             
  File "/root/bug.py", line 10, in <module>                                                                    
    comm.register_memory(data_ptr, 1024 * 1024, Transport.IB0)                                                 
mscclpp._mscclpp.IbError: (14, 'ibv_reg_mr failed (errno 14) (Ib failure: Bad address)')    

Environment:

(base) root@xxx:~# nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2024 NVIDIA Corporation
Built on Thu_Mar_28_02:18:24_PDT_2024
Cuda compilation tools, release 12.4, V12.4.131
Build cuda_12.4.r12.4/compiler.34097967_0
(base) root@xxx:~# nvidia-smi
Sun Apr  6 15:37:56 2025       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.216.03             Driver Version: 535.216.03   CUDA Version: 12.4     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA H100 80GB HBM3          On  | 00000000:19:00.0 Off |                    0 |
| N/A   29C    P0              71W / 700W |      0MiB / 81559MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
...8xH100

[Binyang2014]

Thanks for reporting, looks like related to this issue: NVIDIA/gdrcopy#266, will try ibv_reg_dmabuf_mr API

[liangyuRain]

Hi, could you confirm the error is resolved? We still see the same ib error running the script on our hopper machine.

[Binyang2014]

Hi, could you confirm the error is resolved? We still see the same ib error running the script on our hopper machine.

Pls try branch binyli/bug-fix, I think the reason for this is dma buffer is not supported in your platform. You may need to check the kernel version and driver version.

Pls check following things:

1. The NVIDIA driver installed on your machine is the proprietary flavor. You need to “open” flavor to use DMABUF. The CUDA toolkit download page (such as https://developer.nvidia.com/cuda-downloads?target_os=Linux&target_arch=x86_64&Distribution=Ubuntu&target_version=24.04&target_type=deb_network) tells you how to install the open flavor.

2. The nvidia driver is too old. You need 515.43.04+, R535TRD1+ or later.

3. You need bare metel GPU or MIG. vGPU is not supported.

4. You Linux kernel is too old. You need linux kernel 3.x (could not remember the exact minor version) or later.

[liangyuRain]

Unfortunately, we can only use the machine via container environment, and the NVIDIA driver seems to be proprietary version:

root@...:~# cat /proc/driver/nvidia/version
NVRM version: NVIDIA UNIX x86_64 Kernel Module  535.216.03

Is there any workaround? We intend to use both NVLS and IB.

[Binyang2014]

So you cannot access the vm and upgrade the driver to open flavor ? If so, maybe you need to use non-zero copy version: create different GPU buffers for IB and NVLS. And introduce additional copy between these buffers.

[liangyuRain]

I assumed the driver cannot be changed to open flavor inside a container, which is what we can only do. I have tried mscclpp:base-dev-cuda12.8 image and the driver is still in proprietary flavor.

To create separate buffers for IB and NVLS, I see the GpuBuffer always creates cuMemMap allocated memory when NVLS is available. For IB buffer, is the normal cudaMalloc memory OK as in detail::gpuCallocShared?

[Binyang2014]

Yes, the normal cudaMalloc will be OK for IB device