Skip to content

Commit 9a2baf8

Browse files
AscendTransportAscendTransport
committed
Ascend Transport
1 parent a5cf796 commit 9a2baf8

28 files changed

+3499
-0
lines changed

‎doc/en/ascend_transport.md

Lines changed: 141 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,141 @@
1+
# Ascend Transport
2+
3+
The source code path for Ascend Transport is `Mooncake/mooncake-transfer-engine/src/transport/ascend_transport`, which also includes automated build scripts and the README file.
4+
5+
## Overview
6+
7+
Ascend Transport is a high-performance zero-copy NPU data transfer library with one-sided semantics, directly compatible with Mooncake Transfer Engine. To compile and use the Ascend Transport library, please set the `USE_ASCEND` flag to `"ON"` in the `mooncake-common/common.cmake` file.
8+
9+
Ascend Transport supports inter-NPU data transfer using one-sided semantics (currently supports Device to Device; other modes are under development). Users only need to specify the node and memory information at both ends through the Mooncake Transfer Engine interface to achieve high-performance point-to-point data transfer. Ascend Transport abstracts away internal complexities and automatically handles operations such as establishing connections, registering and exchanging memory, and checking transfer status.
10+
11+
### New Dependencies
12+
13+
In addition to the dependencies already required by Mooncake, Ascend Transport needs some HCCL-related dependencies:
14+
15+
**MPI**
16+
```bash
17+
yum install -y mpich mpich-devel
18+
```
19+
20+
**New Version Ascend-cann-toolkit**
21+
22+
```bash
23+
rm -rf /etc/Ascend/ascend_cann_install.info
24+
```
25+
26+
Download `Ascend-cann-toolkit_8.2.RC1.alpha002_linux-x86_64.run` from the Ascend community (choose the `aarch64` version if using ARM architecture), enter the directory containing the file, and execute:
27+
```bash
28+
./Ascend-cann-toolkit_8.2.RC1.alpha002_linux-x86_64.run --install --force
29+
source /usr/local/Ascend/ascend-toolkit/set_env.sh
30+
```
31+
32+
### One-Step Build Script
33+
34+
Ascend Transport provides a one-step build script located at `Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/scripts/build_all_with_dependencies.sh`. Copy this script to the desired installation directory and run it. You can also pass an installation path as an argument; if not provided, it defaults to the current directory:
35+
36+
```bash
37+
./build_all_with_dependencies.sh /path/to/install_directory
38+
```
39+
40+
This script also supports environments where users cannot perform `git clone` directly. Users can place the source code for dependencies and Mooncake in the target directory, and the script will handle the compilation accordingly.
41+
42+
### One-Step Installation Script (Without Building Mooncake)
43+
44+
To avoid potential conflicts when running other processes during Mooncake compilation, Ascend Transport offers a solution that separates the build and runtime environments.
45+
46+
After completing the Mooncake build via build_all_with_dependencies.sh, you can run setup_basic_dependencies.sh to install only the required dependencies. Place the generated Mooncake .whl package and libascend_transport_mem.so into the installation directory.
47+
48+
Copy the script to the installation directory and run:
49+
```bash
50+
./setup_basic_dependencies.sh /path/to/install_directory
51+
```
52+
53+
Before use, ensure that `libascend_transport_mem.so` has been copied to `/usr/local/Ascend/ascend-toolkit/latest/python/site-packages`, then execute:
54+
```bash
55+
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
56+
```
57+
58+
### Build Instructions
59+
60+
Once all dependencies are installed successfully, you can proceed with building Mooncake normally. If errors occur, try setting the following environment variable:
61+
```bash
62+
export CPLUS_INCLUDE_PATH=$(echo $CPLUS_INCLUDE_PATH | tr ':' '\n' | grep -v "/usr/local/Ascend" | paste -sd: -)
63+
```
64+
65+
### Endpoint Management
66+
67+
Each Huawei NPU card has a dedicated parameter-plane NIC and should be managed by a single `TransferEngine` instance responsible for all its data transfers.
68+
69+
### Ranktable Management
70+
Ascend Transport does not rely on global Ranktable information. It only needs to obtain the local Ranktable information of the current NPU card. During the initialization of Ascend Transport, it will automatically parse the /etc/hccn.conf file to acquire this information.
71+
72+
### Initialization
73+
74+
When using Ascend Transport, the `TransferEngine` must still call the `init` function after construction:
75+
76+
```cpp
77+
TransferEngine();
78+
79+
int TransferEngine::init(const std::string &metadata_conn_string,
80+
const std::string &local_server_name,
81+
const std::string &ip_or_host_name,
82+
uint64_t rpc_port)
83+
```
84+
85+
The only difference is that the `local_server_name` parameter must now include the physical NPU card ID. The format changes from `ip:port` to `ip:port:npu_x`, e.g., `"0.0.0.0:12345:npu_2"`.
86+
87+
> **Note**: This extension of the `local_server_name` is used internally by Ascend Transport without modifying Mooncake's external API. The `segment_desc_name` in metadata remains in the original format (`ip:port`). Therefore, each NPU card must use a unique port that is not occupied.
88+
89+
### Metadata Service
90+
91+
Ascend Transport is compatible with all metadata services currently supported by Mooncake, including `etcd`, `redis`, `http`, and `p2phandshake`. Upon initialization, Ascend Transport registers key NPU card information such as `device_id`, `device_ip`, `rank_id`, and `server_ip`.
92+
93+
### Data Transfer
94+
95+
Ascend Transport supports write/read semantics and automatically determines whether cross-HCCS communication is needed, selecting either HCCS or ROCE as the underlying transport protocol. Users can use the standard Mooncake `getTransferStatus` API to monitor the progress of each transfer request.
96+
97+
### Fault Handling
98+
99+
Building upon HCCL’s built-in fault handling mechanisms, Ascend Transport implements comprehensive error recovery strategies across multiple stages, including initialization, connection setup, and data transfer. It incorporates retry logic and returns precise error codes based on HCCL collective communication standards when retries fail. For detailed logs, refer to `/root/Ascend/log/plog`.
100+
101+
### Test Cases
102+
103+
Ascend Transport provides two test files:
104+
- Multi-scenario test: `mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp`
105+
- Performance test: `mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp`
106+
107+
You can configure various scenarios (e.g., 1-to-1, 1-to-2, 2-to-1) and performance tests by passing valid parameters to these programs.
108+
109+
#### Example Commands for Scenario Testing
110+
111+
**Start Initiator Node:**
112+
```bash
113+
./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608
114+
```
115+
116+
**Start Target Node:**
117+
```bash
118+
./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target --block_size=8388608
119+
```
120+
121+
#### Example Commands for Performance Testing
122+
123+
**Start Initiator Node:**
124+
```bash
125+
./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608
126+
```
127+
128+
**Start Target Node:**
129+
```bash
130+
./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target
131+
```
132+
133+
### Print Description
134+
If you need to obtain information about whether each transport request is cross-hccs and its corresponding execution time, you can enable the related logs by setting the environment variable. Use the following command to turn on the logging:
135+
136+
```bash
137+
export ASCEND_TRANSPORT_PRINT=1
138+
```
139+
140+
### Notes
141+
ascned_transport will establish a TCP connection on the host side.This connection uses port (10000 + deviceId). Please avoid using this port for other services to prevent conflicts.

‎doc/zh/ascend_transport.md

Lines changed: 88 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,88 @@
1+
# Ascend Transport
2+
Ascend Transport源代码路径为Mooncake/mooncake-transfer-engine/src/transport/ascend_transport，该路径下还包含自动化编译脚本、README文件。
3+
## 概述
4+
Ascend Transport是一个单边语义的高性能零拷贝NPU数据传输库，直接兼容Mooncake Transfer Engine。要编译使用Ascend Transport库，请在mooncake-common\common.cmake文件中将USE_ASCEND开关置于"ON"。
5+
6+
Ascend Transport支持使用单边语义进行NPU间数据传输（当前版本只支持DEVICE TO DEVICE，其它进行中），用户只需通过Mooncake Transfer Engine接口指定两端传输的节点及内存信息，即可完成点对点高性能传输。Ascend Transport为用户隐去繁琐的内部实现，自动完成一系列如建链、注册和交换内存、检查传输状态等操作。
7+
8+
### 新增依赖
9+
Ascend Transport在Mooncake本身依赖的基础上，新增了一部分HCCL的依赖：
10+
**MPI**
11+
yum install -y mpich mpich-devel
12+
13+
**新版本Ascend-cann-toolkit**
14+
rm -rf /etc/Ascend/ascend_cann_install.info
15+
在昇腾社区下载Ascend-cann-toolkit_8.2.RC1.alpha002_linux-x86_64.run，(arm选择对应的aarch64版本)进入文件目录执行：
16+
./Ascend-cann-toolkit_8.2.RC1.alpha002_linux-x86_64.run --install --force
17+
source /usr/local/Ascend/ascend-toolkit/set_env.sh
18+
19+
### 一键式编译脚本
20+
Ascend Transport提供一键式编译脚本，脚本位置为Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/scripts/build_all_with_dependencies.sh，将脚本复制到想要安装依赖和Mooncake的目录下执行脚本即可，也支持传入参数指定安装路径，不传入时默认为脚本所在目录，命令如下：
21+
./build_all_with_dependencies.sh /path/to/install_directory
22+
一键式编译脚本同样考虑到用户无法直接在环境上git clone的情况，用户可以把给依赖和Mooncake的源码放在安装目录下并指定，脚本会自动编译依赖和Mooncake。
23+
24+
### 一键式安装脚本(不编译Mooncake)
25+
为了避免用户出现在编译Mooncake的环境上，执行其它进程有冲突的可能问题，Ascend Transport给出编译和执行Mooncake分离的一种方案。
26+
在执行build_all_with_dependencies.sh完成Mooncake编译后，用户可执行Mooncake/mooncake-transfer-engine/src/transport/ascend_transport/scripts/setup_basic_dependencies.sh，仅安装依赖。将一键式编译脚本生成的mooncake whl包和libascend_transport_mem.so放在安装目录下
27+
28+
将脚本复制到安装目录下执行脚本即可，命令如下：
29+
./setup_basic_dependencies.sh /path/to/install_directory
30+
31+
在使用前，确保编译生成的libascend_transport_mem.so文件已经复制到/usr/local/Ascend/ascend-toolkit/latest/python/site-packages下，然后执行命令：
32+
export LD_LIBRARY_PATH=/usr/local/Ascend/ascend-toolkit/latest/python/site-packages:$LD_LIBRARY_PATH
33+
34+
### 编译说明
35+
在成功安装所有依赖后，正常编译Mooncake即可，如果报错，可尝试设置环境变量：
36+
export CPLUS_INCLUDE_PATH=$(echo $CPLUS_INCLUDE_PATH | tr ':' '\n' | grep -v "/usr/local/Ascend" | paste -sd: -)
37+
38+
### 端点管理
39+
每张华为NPU卡拥有一张参数面网卡，对应使用一个TransferEngine管理该NPU卡上所有的传输操作。
40+
41+
### Ranktable管理
42+
Ascend Transport不依赖全局的Ranktable信息，只需要获取当前NPU卡的本地Ranktable信息，在Ascend Transport初始化时会解析/etc/hccn.conf文件自动获取。
43+
44+
### 初始化
45+
使用Ascend Transport时，TransferEngine 在完成构造后同样需要调用 `init` 函数进行初始化：
46+
```cpp
47+
TransferEngine();
48+
49+
int TransferEngine::init(const std::string &metadata_conn_string,
50+
const std::string &local_server_name,
51+
const std::string &ip_or_host_name,
52+
uint64_t rpc_port)
53+
```
54+
唯一的区别在于，在TransferEngine init时需要在local_server_name参数中包含TransferEngine所在的NPU物理卡号，local_server_name参数从ip:port改为ip:port:npu_x，例如"0.0.0.0:12345:npu_2"。
55+
56+
```注意```：Ascend Transport只是在不修改Mooncake对外接口的形势下，传入所在的NPU物理卡号，只为Transport内部使用，metadata的segement_desc_name仍然还是原来的形式，即ip:port，因此，每张npu卡的port需要保证互不相同，且端口未被占用。
57+
58+
### metadata服务
59+
Ascend Transport兼容所有Mooncake当前支持的metadata服务，包括 etcd, redis 和 http，以及p2phandshake。Ascend Transport在初始化时会注册当前NPU卡的一系列信息，包括device_id\device_ip\rank_id\server_ip等。
60+
61+
### 数据传输
62+
Ascend Transport支持write/read语义，且会自动判断是否跨HCCS通信，选择HCCS/ROCE的底层通信协议，用户只需采用Mooncake的getTransferStatus接口即可获取每个请求的传输情况。
63+
64+
### 故障处理
65+
Ascend Transport在HCCL本身的故障处理基础上，设计了完善的故障处理机制。针对初始化、建链、数据传输等多个阶段可能出现的故障，新增或沿用了失败重试机制。在重试仍然失败后，沿用了HCCL集合通信相关操作错误码，给出精准的报错信息。为获取更详细的错误信息，也可以查询/root/Ascend/log目录下的plog日志。
66+
67+
### 测试用例
68+
Ascend Transport提供多场景测试mooncake-transfer-engine/example/transfer_engine_ascend_one_sided.cpp和性能测试mooncake-transfer-engine/example/transfer_engine_ascend_perf.cpp两个测试文件，根据测试头部设置的可传入参数传入合法参数，
69+
可以完成一对一、一对二、二对一多种场景和性能测试。
70+
71+
多场景用例执行命令如：
72+
```启动发起节点：```
73+
./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608
74+
```启动目标节点：```
75+
./transfer_engine_ascend_one_sided --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target --block_size=8388608
76+
77+
性能用例执行命令如：
78+
```启动发起节点：```
79+
./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12345 --protocol=hccl --operation=write --segment_id=10.0.0.0:12346 --device_id=0 --mode=initiator --block_size=8388608
80+
```启动目标节点：```
81+
./transfer_engine_ascend_perf --metadata_server=P2PHANDSHAKE --local_server_name=10.0.0.0:12346 --protocol=hccl --operation=write --device_id=1 --mode=target
82+
83+
### 打印说明
84+
如果需要得到每个传输request请求是否跨hccs和耗时情况，可以通过设置环境变量打开相关打印，命令如下：
85+
export ASCEND_TRANSPORT_PRINT=1
86+
87+
### 注意事项
88+
ascned_transport会建立一个host侧的tcp连接，占用端口为10000+deviceId,请注意避开此端口，勿重复占用

‎mooncake-common/common.cmake

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -44,6 +44,7 @@ option(USE_CUDA "option for using gpu direct" OFF)
4444
option(USE_NVMEOF "option for using NVMe over Fabric" OFF)
4545
option(USE_TCP "option for using TCP transport" ON)
4646
option(USE_CXL "option for using cxl protocol" OFF)
47+
option(USE_ASCEND "option for using npu" ON)
4748
option(USE_ETCD "option for enable etcd as metadata server" OFF)
4849
option(USE_ETCD_LEGACY "option for enable etcd based on etcd-cpp-api-v3" OFF)
4950
option(USE_REDIS "option for enable redis as metadata server" OFF)
@@ -83,6 +84,22 @@ if (USE_TCP)
8384
add_compile_definitions(USE_TCP)
8485
endif()
8586

87+
if (USE_ASCEND)
88+
set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DOPEN_BUILD_PROJECT ")
89+
set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -DOPEN_BUILD_PROJECT ")
90+
91+
file(GLOB ASCEND_TOOLKIT_ROOT "/usr/local/Ascend/ascend-toolkit/latest/*-linux")
92+
set(ASCEND_LIB_DIR "${ASCEND_TOOLKIT_ROOT}/lib64")
93+
set(ASCEND_INCLUDE_DIR "${ASCEND_TOOLKIT_ROOT}/include")
94+
95+
message(STATUS "Found lib64 directories: ${ASCEND_LIB_DIR}")
96+
message(STATUS "Found include directories: ${ASCEND_INCLUDE_DIR}")
97+
98+
add_compile_definitions(USE_ASCEND)
99+
include_directories(/usr/local/include /usr/include ${ASCEND_INCLUDE_DIR} /usr/include/jsoncpp)
100+
link_directories(${ASCEND_LIB_DIR})
101+
endif()
102+
86103
if (USE_REDIS)
87104
add_compile_definitions(USE_REDIS)
88105
message(STATUS "Redis as metadata server support is enabled")

‎mooncake-integration/transfer_engine/transfer_engine_py.cpp

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -114,7 +114,9 @@ int TransferEnginePy::initializeExt(const char *local_hostname,
114114
}
115115

116116
free_list_.resize(kSlabSizeKBTabLen);
117+
#ifndef USE_ASCEND
117118
doBuddyAllocate(kMaxClassId);
119+
#endif
118120
return 0;
119121
}
120122

‎mooncake-transfer-engine/CMakeLists.txt

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -8,6 +8,27 @@ if (NOT GLOBAL_CONFIG)
88
include(../mooncake-common/common.cmake)
99
endif() # GLOBAL_CONFIG
1010

11+
if (USE_ASCEND)
12+
file(GLOB ASCEND_TOOLKIT_ROOT "/usr/local/Ascend/ascend-toolkit/latest/*-linux")
13+
set(ASCEND_INCLUDE_DIR "${ASCEND_TOOLKIT_ROOT}/include")
14+
file(GLOB MPI_INCLUDE_DIR "/usr/include/mpich-*/")
15+
file(GLOB MPI_LOCAL_INCLUDE_DIR "/usr/local/mpich-*/include")
16+
include_directories(/usr/local/include
17+
/usr/include
18+
${ASCEND_INCLUDE_DIR}
19+
${ASCEND_INCLUDE_DIR}/experiment
20+
${ASCEND_INCLUDE_DIR}/experiment/hccl
21+
${ASCEND_INCLUDE_DIR}/experiment/slog/toolchain
22+
${ASCEND_INCLUDE_DIR}/experiment/metadef/common/util/error_manager
23+
${ASCEND_INCLUDE_DIR}/experiment/runtime
24+
${ASCEND_INCLUDE_DIR}/experiment/msprof
25+
${MPI_INCLUDE_DIR}
26+
${MPI_LOCAL_INCLUDE_DIR}
27+
/usr/local/Ascend/ascend-toolkit/latest/tools/hccl_test/common/src/
28+
)
29+
endif()
30+
31+
1132
include_directories(include)
1233
add_subdirectory(include)
1334
add_subdirectory(src)

‎mooncake-transfer-engine/example/CMakeLists.txt

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,16 @@
1+
file(GLOB ASCEND_TOOLKIT_ROOT "/usr/local/Ascend/ascend-toolkit/latest/*-linux")
2+
set(ASCEND_LIB_DIR "${ASCEND_TOOLKIT_ROOT}/lib64")
3+
4+
link_directories(${ASCEND_LIB_DIR})
5+
16
add_executable(transfer_engine_bench transfer_engine_bench.cpp)
27
target_link_libraries(transfer_engine_bench PUBLIC transfer_engine)
38

49
add_executable(memory_pool memory_pool.cpp)
510
target_link_libraries(memory_pool PUBLIC transfer_engine)
11+
12+
add_executable(transfer_engine_ascend_one_sided transfer_engine_ascend_one_sided.cpp)
13+
target_link_libraries(transfer_engine_ascend_one_sided PUBLIC transfer_engine)
14+
15+
add_executable(transfer_engine_ascend_perf transfer_engine_ascend_perf.cpp)
16+
target_link_libraries(transfer_engine_ascend_perf PUBLIC transfer_engine)

0 commit comments

Comments
 (0)