HAMi RoapMap

[archlitchi]

This is the issue for HAMi RoapMap, you can simply reply this PR to submit your ideas

Next Version: v2.7.0
Estimated release: Aug-Sep/2025
Tasks

• Incubating preperations (Incubating Preperations #1057)
• Support kunlunxin sGPU & Scheduling /assign @ouyangluwei163
• Support kunlunxin Lite(scheduling only) /assign @archlitchi
• Support enflame GPU topology /assign @zhaikangqi331
• Support enflame sGPU new plan
• Optimize scheduler events can be access by :31994 @Wangmin362
• Website Optimization /assign @ouyangluwei163 @Nimbus318
• DRA design /assign @Shouren
• HAMi for KAI scheduler
• Support multiple cambricon types (370,590,etc..)
• Optimize project governance on sub-projects(website, volcano-vgpu-device-plugin, etc..) /assign @archlitchi

Bugs:

• An error occurred while create Iluvatar pod #933 @ouyangluwei163
• cuda 13.0 device memory count error #1328 @archlitchi

---

Version v2.6.1/ v2.5.3
Tasks
Fix Bugs:

• [BUG] nvidia-smi shows abnormal GPU memory usage (17TB+) in HAMi containers causing false OOM errors #1181
• libvgpu.so segfault for cuDeviceGet #1055
• seg fault with vllm + PP #1219
• Cannot support vllm= 0.9.0 nvidia-nccl-cu12==2.26.2, tp>1 #1230
• HAMI 2.6.0 can't run vllm 0.9.0-cuda12.8.1 when tp=8 #1191
• Device 0 OOM when multiple call cuMem*Async and cuMemFreeAsync HAMi-core#96
• The app in the container uses more GPU memory than the allocated GPU memory, but the container does not crash #1221

---

Below are the roadmap history
Next Version: v2.6.0
Estimated release: May-June/2025

Tasks:

• Support kunlunxin sGPU /assign @ouyangluwei163
• Support metax sGPU
• Support enflame sGPU
• Optimize scheduler events
• Website Optimization /assign @ouyangluwei163 @Nimbus318
• Add metax to website
• Support driver 570+cuda 12.8 /assign @archlitchi
• DRA design /assign @Shouren
• Support multiple cambricon types (370,590,etc..)

Bugs:

• Scheduled by exclusive GPU, no pod memory usage indicator is included in the 31992 monitoring indicators #931
• An error occurred while create Iluvatar pod #933 @ouyangluwei163
• When using a Cambricon MLU370 with multiple cards, Hami can only schedule them to one physical card. #946 @archlitchi
• Upgrade business pod room after Hami 2.5 #940 @archlitchi

[HJJ256]

Hi @archlitchi

Is it possible to get an intermediate release? I noticed there were a few bug fixes (including #845) which were done in last 2 months since v2.5.0

[ouyangluwei163]

#1023 refactored the filter logs and event messages to enhance their clarity

[MouseSun846]

Can the Volcano scheduler be used to schedule non - GPU virtualization capabilities, such as NPU and DCU? Essentially, we don't want to introduce multiple schedulers into the cluster.

[yangshiqi]

Support enflame sGPU: #1040 by @zhaikangqi331

[AshinWu]

debian11 kernel 5.10 Driver Version 560.35.03 HAMI 2.5.1
curl {GPU 节点 IP}:31992/metrics
缺少Device_memory_desc_of_container、Device_utilization_desc_of_container、vGPU_device_memory_limit_in_bytes、vGPU_device_memory_usage_in_bytes 指标。
有无解决方案？期望能尽快修复，感谢~

[AshinWu]

@archlitchi 你好，目前我们在用Project-HAMi/volcano-vgpu-device-plugin，也同样遇到pod gpu指标获取不到问题。
通过 curl {volcano device plugin pod ip}:9394/metrics发现缺少以下指标：
vGPU_device_memory_limit_in_bytes
vGPU_device_memory_usage_in_bytes

我怀疑也是和HAMI device BUG类似，有无解决方案？期望能尽快修复。这个pod gpu 实时使用量对我们很重要。感谢~

[Wangmin362]

@archlitchi please assign "Optimize scheduler events can be access by :31994" task to me

[jj1kim]

Hi @Shouren !
Do you have any specific plan to implement DRA in HAMi?
My team want to use HAMi with DRA, but it's implementation seems to be your task.
If you are planning to complete that in the near future, we're going to wait until then.
If not, we're going to implement DRA ourselves.
I look forward to your reply! Thanks!