[GPUOP] Adding docs for DCM Systemd integration (#851) (#852)

ci-penbot-01 · nikhilsk · sajmera-pensando · commit 319142b8019f · 2025-08-11T17:16:11.000-07:00
(cherry picked from commit 5389d38c6a9efbb7680ac74c28a874aef221728a)

Co-authored-by: nikhilsk &lt;47417007+nikhilsk@users.noreply.github.com&gt;
Signed-off-by: yansun1996 &lt;Yan.Sun3@amd.com&gt;
diff --git a/docs/dcm/applying-partition-profiles.rst b/docs/dcm/applying-partition-profiles.rst
@@ -102,6 +102,9 @@ Below is an example of how to create the `config-manager-config.yaml` file that
                     }
                 ]
             }
+        },
+        "gpuClientSystemdServices": {
+            "names": ["amd-metrics-exporter", "gpuagent"]
         }
         }
     EOF
diff --git a/docs/dcm/device-config-manager-configmap.md b/docs/dcm/device-config-manager-configmap.md
@@ -43,6 +43,9 @@ data:
                   }
               ]
           }
+      },
+      "gpuClientSystemdServices": {
+          "names": ["amd-metrics-exporter", "gpuagent"]
       }
     }
 ```
@@ -57,6 +60,7 @@ Below is an explanation of each field in the ConfigMap:
 | `computePartition`      | Compute partition type                                                                                                                                           |
 | `memoryPartition`       | Memory partition type                                                                                                                                            |
 | `numGPUsAssigned`       | Number of GPUs to be partitioned on the node                                                                                                                     |
+| `gpuClientSystemdServices`       | Defines a list of systemd service unit files to be stopped/restarted on the node                                                                                                                   |
 
 ```{note}
 Users can create a heterogeneous partitioning config profile by specifying more than one `computePartition` scheme in the `profiles` array, however this is not a recommmended or supported configuration by AMD. Note that NPS4 memory partition mode does not work with heterogenous parition schemes and only supports CPX on MI300X systems.
@@ -90,6 +94,9 @@ Users can create a heterogeneous partitioning config profile by specifying more
                 }
             ]
             }
+          },
+          "gpuClientSystemdServices": {
+              "names": ["amd-metrics-exporter", "gpuagent"]
           }
         }
 
diff --git a/docs/dcm/systemd_integration.md b/docs/dcm/systemd_integration.md
@@ -0,0 +1,98 @@
+# DCM Systemd Integration
+
+## Background 
+
+The Device Config Manager (DCM) orchestrates hardware-level tasks such as GPU partitioning. Before initiating partitioning, it gracefully stops specific systemd services defined in a configmap to prevent any processes (gpuagent, etc) from partition interference and ensure consistent device states
+
+## K8S ConfigMap enhancement
+
+The configmap contains a key "gpuClientSystemdServices" which declares the list of services to manage: 
+
+```yaml
+"gpuClientSystemdServices": {
+    "names": ["amd-metrics-exporter", "gpuagent"] 
+}
+```
+- These are the unit names (without the. service suffix) of systemd services related to GPU runtime agents. We add the suffix as a part of the code
+- Users can add/modify services to the above list 
+
+## ConfigMap
+
+```yaml  
+apiVersion: v1
+kind: ConfigMap
+metadata:
+  name: config-manager-config
+  namespace: kube-amd-gpu
+data:
+  config.json: |
+    {
+      "gpu-config-profiles":
+      {
+          "cpx-profile":
+          {
+              "skippedGPUs": {
+                  "ids": []
+              },
+              "profiles": [
+                  {
+                      "computePartition": "CPX",
+                      "memoryPartition": "NPS4",
+                      "numGPUsAssigned": 8
+                  }
+              ]
+          },
+          "spx-profile":
+          {
+              "skippedGPUs": {
+                  "ids": []
+              },
+              "profiles": [
+                  {
+                      "computePartition": "SPX",
+                      "memoryPartition": "NPS1",
+                      "numGPUsAssigned": 8
+                  }
+              ]
+          }
+      },
+      "gpuClientSystemdServices": {
+           "names": ["amd-metrics-exporter", "gpuagent"]
+       }
+    }
+```
+
+## Required Mounts for D-Bus & systemd Integration
+
+| **Mount Name**         | **Mount Path**        | **Purpose**                                                              |
+|------------------------|------------------------|---------------------------------------------------------------------------|
+| `etc-systemd`          | `/etc/systemd`         | Access unit files for service definitions                                |
+| `run-systemd`          | `/run/systemd`         | Enables access to systemd runtime state                                  |
+| `usr-lib-systemd`      | `/usr/lib/systemd`     | Required for systemd libraries and binaries                              |
+| `var-run-dbus`         | `/var/run/dbus`        | Allows DCM to communicate via system D-Bus (`system_bus_socket`)         |
+
+## Workflow
+
+- DCM uses D-Bus APIs to query, stop, and restart systemd services programmatically, ensuring precise service orchestration. 
+
+- Extract Service List: On startup, DCM parses the configmap and retrieves the names array under gpuClientSystemdServices. Each entry is appended with (. service) to form full unit names. 
+
+- Capture Pre-State:
+    - For each service: 
+        - It checks status using D-Bus via `org.freedesktop.systemd1.Manager.GetUnit.` 
+        - Stores current state (e.g. `active`, `inactive`, `not-loaded`) in PreStateDB. 
+        - This DB is used for restoring service state post-partitioning. 
+
+- Stop Services: Services are stopped gracefully using D-Bus APIs. This ensures they release GPU resources and don't disrupt the partitioning operation. We check if the service is present before stopping it using the CheckUnitStatus API. 
+
+- Perform Partitioning: Once services are stopped temporarily, DCM initiates the partitioning logic (using node labels/configmap profiles) and completes the partitioning workflow 
+
+- Restart & Restore State After partitioning: 
+    - DCM checks PreStateDB to determine which services were previously active. 
+    - Only those Services are restarted accordingly using the D-Bus invocation APIs. 
+    - Additionally, PreStateDB is cleared via a CleanupPreState() function to reset the tracker DB for the next run. 
+
+# Conclusion 
+
+- Avoids GPU contention during partitioning (device-busy errors aren’t seen during partition) 
+- Maintains service continuity with minimal downtime 
diff --git a/docs/sphinx/_toc.yml b/docs/sphinx/_toc.yml
@@ -58,6 +58,7 @@ subtrees:
       - file: dcm/device-config-manager
       - file: dcm/device-config-manager-configmap
       - file: dcm/applying-partition-profiles
+      - file: dcm/systemd_integration
   - caption: Specialized Networks
     entries:
       - file: specialized_networks/airgapped-install
diff --git a/docs/sphinx/_toc.yml.in b/docs/sphinx/_toc.yml.in
@@ -58,6 +58,7 @@ subtrees:
       - file: dcm/device-config-manager
       - file: dcm/device-config-manager-configmap
       - file: dcm/applying-partition-profiles
+      - file: dcm/systemd_integration
   - caption: Specialized Networks
     entries:
       - file: specialized_networks/airgapped-install
diff --git a/example/configManager/config.json b/example/configManager/config.json
@@ -27,5 +27,8 @@
                 }          
             ]
         }
+    },
+    "gpuClientSystemdServices": {
+        "names": ["amd-metrics-exporter", "gpuagent"]
     }
   }

Original file line number	Diff line number	Diff line change
@@ -102,6 +102,9 @@ Below is an example of how to create the `config-manager-config.yaml` file that
`102`	`102`	`}`
`103`	`103`	`]`
`104`	`104`	`}`
	`105`	`+ },`
	`106`	`+ "gpuClientSystemdServices": {`
	`107`	`+ "names": ["amd-metrics-exporter", "gpuagent"]`
`105`	`108`	`}`
`106`	`109`	`}`
`107`	`110`	`EOF`
Original file line number	Diff line number	Diff line change
`@@ -43,6 +43,9 @@ data:`
`43`	`43`	`}`
`44`	`44`	`]`
`45`	`45`	`}`
	`46`	`+ },`
	`47`	`+ "gpuClientSystemdServices": {`
	`48`	`+ "names": ["amd-metrics-exporter", "gpuagent"]`
`46`	`49`	`}`
`47`	`50`	`}`
`48`	`51`	```
`@@ -57,6 +60,7 @@ Below is an explanation of each field in the ConfigMap:`
`57`	`60`	\| `computePartition` \| Compute partition type \|
`58`	`61`	\| `memoryPartition` \| Memory partition type \|
`59`	`62`	\| `numGPUsAssigned` \| Number of GPUs to be partitioned on the node \|
	`63`	+\| `gpuClientSystemdServices` \| Defines a list of systemd service unit files to be stopped/restarted on the node \|
`60`	`64`
`61`	`65`	```{note}
`62`	`66`	Users can create a heterogeneous partitioning config profile by specifying more than one `computePartition` scheme in the `profiles` array, however this is not a recommmended or supported configuration by AMD. Note that NPS4 memory partition mode does not work with heterogenous parition schemes and only supports CPX on MI300X systems.
`@@ -90,6 +94,9 @@ Users can create a heterogeneous partitioning config profile by specifying more`
`90`	`94`	`}`
`91`	`95`	`]`
`92`	`96`	`}`
	`97`	`+ },`
	`98`	`+ "gpuClientSystemdServices": {`
	`99`	`+ "names": ["amd-metrics-exporter", "gpuagent"]`
`93`	`100`	`}`
`94`	`101`	`}`
`95`	`102`
Original file line number	Diff line number	Diff line change
`@@ -27,5 +27,8 @@`
`27`	`27`	`}`
`28`	`28`	`]`
`29`	`29`	`}`
	`30`	`+ },`
	`31`	`+ "gpuClientSystemdServices": {`
	`32`	`+ "names": ["amd-metrics-exporter", "gpuagent"]`
`30`	`33`	`}`
`31`	`34`	`}`