Skip to content

Commit d462243

Browse files
mohamedzeidan2021yungwenh-awsmollyheamazonbrianFruitaws-brianxia
authored
Integ test 2 (#326)
* feat: Implement elastic training cli arguments (#273) * feat: Implement elastic training cli arguments * Add elastic training unified config and unit test * Add graceful shutdown and scaling timeout to cli args * Revert "feat: Implement elastic training cli arguments (#273)" This reverts commit 18428ef2b1c0562bf51a9a4b4aa2914eed441259. * Add dev_space_constants.py (#255) Co-authored-by: Brian Xia <[email protected]> * Add dev_space_access_constants.py (#256) Co-authored-by: Brian Xia <[email protected]> * Add space_admin_config_constants.py (#257) Co-authored-by: Brian Xia <[email protected]> * Add template package only (#261) Co-authored-by: Brian Xia <[email protected]> * Add dev_space.py CLI command (#263) * Add dev_space.py CLI command * Add dev space unit tests --------- Co-authored-by: Brian Xia <[email protected]> * Add dev_space_utils.py to work with the dev space template model (#262) * Add dev_space_utils.py * Add unit tests for dev_space_utils --------- Co-authored-by: Brian Xia <[email protected]> * Add dev space CLI (#269) * Rename dev space to space (#272) * Update the Space model and constants per latest operator (#275) * Add space_admin_config.py CLI command (#260) * Add space_admin_config.py CLI command * Update the space admin config to space template --------- Co-authored-by: Brian Xia <[email protected]> * Implement CRUD operations for Space PySDK (#267) * Implement CRUD operations for Space PySDK * Update Space PySDK per new schema * Update Space PySDK per new schema * Implement the pySDK for the Space Template (#282) * Refactor Space CLI using the Space PySDK (#281) * Implement CRUD operations for Space PySDK * Update Space PySDK per new schema * Refactor CLI to use the PySDK * Add dev_space_access.py CLI command (#259) * Add dev_space_access.py CLI command * Add space access creation to pySDK and refactor space access CLI --------- Co-authored-by: Brian Xia <[email protected]> * Listing space will filter out the spaces not created by the current user (#285) * Implement CRUD operations for Space PySDK * Update Space PySDK per new schema * Implement CRUD operations for Space PySDK * Update Space PySDK per new schema * Update Space PySDK per new schema * Implement space list pagination and creator filtering * Refactor space template with PySDK (#286) * Add additional Space parameters for resources including the fractional GPU (#287) * Implement validation for mig profiles for Spaces (#291) * Implement validation for mig profiles when creating/updating spaces * Update Space parameter model * Make Space Template namespaced resource * Parker GA issues (#296) * Update Space Template CLI to be namespaced * Space get-logs default to the workspace container * Remove error handling to bubble up the actual K8s errors * Listing public Spaces * Fix typos, elaborated text, add logic to parse idle-shutdown * Fix the template ref regression (#300) * Update SageMaker Space documentation (#301) * Implement Space integration tests (#298) Inference tests succeeded with parker-cli code - https://quip-amazon.com/fhwhAAMht0Mm/Project-Parker-HyperPod-User-Experience-for-Data-Scientist-persona Parker-cli integ tests pass (shown below) These inference tests failing are known to be flaky- https://w.amazon.com/bin/view/AWS/AmazonAI/Platform/Codex/CodexInfra/Runbooks/HyperPodCLI/TroubleshootInferenceTests#HTroubleshooting ticket has been created to fix these flaky tests - https://t.corp.amazon.com/V1943878058 Parker-cli integ tests passing ============================= test session starts ============================== platform linux -- Python 3.11.14, pytest-8.3.2, pluggy-1.6.0 -- /root/.pyenv/versions/3.11.14/bin/python3.11 cachedir: .pytest_cache rootdir: /codebuild/output/src1458832038/src/github.com/aws/private-sagemaker-hyperpod-cli-staging configfile: setup.cfg plugins: hydra-core-1.3.2, order-1.3.0, dependency-0.6.0, cov-5.0.0 collecting ... collected 39 items test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_create PASSED [ 2%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_table PASSED [ 5%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_json PASSED [ 7%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_yaml PASSED [ 10%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_json PASSED [ 12%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_stop PASSED [ 15%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_start PASSED [ 17%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_update PASSED [ 20%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_get_logs PASSED [ 23%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_delete PASSED [ 25%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_list_empty_namespace PASSED [ 28%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_describe_nonexistent PASSED [ 30%] test/integration_tests/space/cli/test_cli_space.py::TestSpaceCLI::test_space_delete_nonexistent PASSED [ 33%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_create PASSED [ 35%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_table PASSED [ 38%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_json PASSED [ 41%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_yaml PASSED [ 43%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_json PASSED [ 46%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_update PASSED [ 48%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_delete PASSED [ 51%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_list_empty_namespace PASSED [ 53%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_describe_nonexistent PASSED [ 56%] test/integration_tests/space/cli/test_cli_space_template.py::TestSpaceTemplateCLI::test_space_template_delete_nonexistent PASSED [ 58%] test/integration_tests/space/sdk/test_sdk_space.py::test_create_space PASSED [ 61%] test/integration_tests/space/sdk/test_sdk_space.py::test_list_spaces PASSED [ 64%] test/integration_tests/space/sdk/test_sdk_space.py::test_get_space PASSED [ 66%] test/integration_tests/space/sdk/test_sdk_space.py::test_wait_until_running PASSED [ 69%] test/integration_tests/space/sdk/test_sdk_space.py::test_update_space PASSED [ 71%] test/integration_tests/space/sdk/test_sdk_space.py::test_stop_space PASSED [ 74%] test/integration_tests/space/sdk/test_sdk_space.py::test_start_space PASSED [ 76%] test/integration_tests/space/sdk/test_sdk_space.py::test_list_pods PASSED [ 79%] test/integration_tests/space/sdk/test_sdk_space.py::test_get_logs PASSED [ 82%] test/integration_tests/space/sdk/test_sdk_space.py::test_create_space_access SKIPPED [ 84%] test/integration_tests/space/sdk/test_sdk_space.py::test_delete_space PASSED [ 87%] test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_create_template PASSED [ 89%] test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_list_templates PASSED [ 92%] test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_get_template PASSED [ 94%] test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_update_template PASSED [ 97%] test/integration_tests/space/sdk/test_sdk_space_template.py::TestHPSpaceTemplate::test_delete_template PASSED [100%] =============================== warnings summary =============================== * merge conflicts fixed * Update README for fractional gpu support (#294) * Update README for fractional gpu support * update pytorch job example * add example for accelerator partitions * merge conflicts from js template and inference * update changelog * uncommented install req * uncommented * fixed uncomment --------- Co-authored-by: Sophia <[email protected]> Co-authored-by: Molly He <[email protected]> Co-authored-by: Brian Xia <[email protected]> Co-authored-by: Brian Xia <[email protected]> Co-authored-by: Mohamed Zeidan <[email protected]> Co-authored-by: Ophelia Yang <[email protected]>
1 parent 2d6c491 commit d462243

File tree

73 files changed

+9677
-313
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

73 files changed

+9677
-313
lines changed

CHANGELOG.md

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,17 @@
11
# Changelog
22

3+
## v.3.4.0 (2025-11-20)
4+
5+
### Features
6+
7+
* HyperPod Dev Spaces template for data scientists to create, manage, and access interactive ML development environments with configurable resource allocation and namespace isolation
8+
* Support for KVCaching, intelligent routing, tiered storage, MIG
9+
* Support for fractional gpu
10+
* Support KVCache and Intelligent Routing support in template version 1.1
11+
* User can modify jinja template to add parameters supported by CRD through init experience, for further CLI customization
12+
* MIG support for model deployment on SageMaker Hyperpod Inference
13+
14+
315
## v.3.3.1 (2025-10-30)
416

517
### Features

README.md

Lines changed: 296 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -21,10 +21,12 @@ Note: Old `hyperpod`CLI V2 has been moved to `release_v2` branch. Please refer [
2121
- [Inference](#inference)
2222
- [Jumpstart Endpoint](#jumpstart-endpoint-creation)
2323
- [Custom Endpoint](#custom-endpoint-creation)
24+
- [Space](#space)
2425
- [SDK](#sdk)
2526
- [Cluster Management](#cluster-management-sdk)
2627
- [Training](#training-sdk)
2728
- [Inference](#inference-sdk)
29+
- [Space](#space-sdk)
2830
- [Examples](#examples)
2931

3032

@@ -300,6 +302,37 @@ hyp create hyp-pytorch-job \
300302
--volume name=training-output,type=pvc,mount_path=/data2,claim_name=my-pvc,read_only=false
301303
```
302304

305+
**Example with accelerator parititons:**
306+
307+
```bash
308+
hyp create hyp-pytorch-job \
309+
--version 1.1 \
310+
--job-name test-pytorch-job \
311+
--image pytorch/pytorch:latest \
312+
--command '[python, train.py]' \
313+
--args '[--epochs=10, --batch-size=32]' \
314+
--environment '{"PYTORCH_CUDA_ALLOC_CONF": "max_split_size_mb:32"}' \
315+
--pull-policy "IfNotPresent" \
316+
--instance-type ml.p4d.24xlarge \
317+
--tasks-per-node 8 \
318+
--label-selector '{"accelerator": "nvidia", "network": "efa"}' \
319+
--deep-health-check-passed-nodes-only true \
320+
--scheduler-type "kueue" \
321+
--queue-name "training-queue" \
322+
--priority "high" \
323+
--max-retry 3 \
324+
--accelerator-partition-type "mig-1g.5gb" \
325+
--accelerator-partition-count 2 \
326+
--accelerator-partition-limit 4 \
327+
--vcpu 96.0 \
328+
--memory 1152.0 \
329+
--vcpu-limit 96.0 \
330+
--memory-limit 1152.0 \
331+
--preferred-topology "topology.kubernetes.io/zone=us-west-2a" \
332+
--volume name=model-data,type=hostPath,mount_path=/data,path=/data \
333+
--volume name=training-output,type=pvc,mount_path=/data2,claim_name=my-pvc,read_only=false
334+
```
335+
303336
| Parameter | Type | Required | Description |
304337
|-----------|------|----------|-------------|
305338
| `--job-name` | TEXT | Yes | Unique name for the training job (1-63 characters, alphanumeric with hyphens) |
@@ -326,10 +359,21 @@ hyp create hyp-pytorch-job \
326359
| `--accelerators-limit` | INTEGER | No | Limit for the number of accelerators a.k.a GPUs or Trainium Chips |
327360
| `--vcpu-limit` | FLOAT | No | Limit for the number of vCPUs |
328361
| `--memory-limit` | FLOAT | No | Limit for the amount of memory in GiB |
362+
| `--accelerator-partition-type` | TEXT | No | Type of accelerator partition (e.g., mig-1g.5gb, mig-2g.10gb, mig-3g.20gb, mig-4g.20gb, mig-7g.40gb) |
363+
| `--accelerator-partition-count` | INTEGER | No | Number of accelerator partitions to request (minimum: 1) |
364+
| `--accelerator-partition-limit` | INTEGER | No | Limit for the number of accelerator partitions (minimum: 1) |
329365
| `--preferred-topology` | TEXT | No | Preferred topology annotation for scheduling |
330366
| `--required-topology` | TEXT | No | Required topology annotation for scheduling |
331367
| `--debug` | FLAG | No | Enable debug mode (default: false) |
332368

369+
#### List Available Accelerator Partition Types
370+
371+
This command lists the available accelerator partition types on the cluster for a specific instance type.
372+
373+
```bash
374+
hyp list-accelerator-partition-type --instance-type <instance-type>
375+
```
376+
333377
#### List Training Jobs
334378

335379
```bash
@@ -614,6 +658,105 @@ hyp get-operator-logs hyp-custom-endpoint --since-hours 0.5
614658
hyp delete hyp-custom-endpoint --name endpoint-custom
615659
```
616660
661+
### Space
662+
663+
#### Create a Space
664+
665+
```bash
666+
hyp create hyp-space \
667+
--name myspace \
668+
--namespace default \
669+
--display-name "My Space"
670+
```
671+
672+
| Parameter | Type | Required | Description |
673+
|-----------|------|----------|-------------|
674+
| `--name` | TEXT | Yes | Space name |
675+
| `--display-name` | TEXT | Yes | Display Name of the space |
676+
| `--namespace` | TEXT | No | Kubernetes namespace |
677+
| `--image` | TEXT | No | Image specifies the container image to use |
678+
| `--desired-status` | TEXT | No | DesiredStatus specifies the desired operational status |
679+
| `--ownership-type` | TEXT | No | OwnershipType specifies who can modify the space. 'Public' means anyone with RBAC permissions can update/delete the space. 'OwnerOnly' means only the creator can update/delete the space. |
680+
| `--node-selector` | TEXT | No | NodeSelector specifies node selection constraints for the space pod (JSON string) |
681+
| `--affinity` | TEXT | No | Affinity specifies node affinity and anti-affinity rules for the space pod (JSON string) |
682+
| `--tolerations` | TEXT | No | Tolerations specifies tolerations for the space pod to schedule on nodes with matching taints (JSON string) |
683+
| `--lifecycle` | TEXT | No | Lifecycle specifies actions that the management system should take in response to container lifecycle events (JSON string) |
684+
| `--app-type` | TEXT | No | AppType specifies the application type for this workspace |
685+
| `--service-account-name` | TEXT | No | ServiceAccountName specifies the name of the ServiceAccount to use for the workspace pod |
686+
| `--idle-shutdown` | TEXT | No | Idle shutdown configuration. Format: --idle-shutdown enabled=<bool>,idleTimeoutInMinutes=<int>,detection=<JSON string> |
687+
| `--template-ref` | TEXT | No | TemplateRef references a WorkspaceTemplate to use as base configuration. Format: --template-ref name=<name>,namespace=<namespace> |
688+
| `--container-config` | TEXT | No | Container configuration. Format: --container-config command=<cmd>,args=<arg1;arg2> |
689+
| `--storage` | TEXT | No | Storage configuration. Format: --storage storageClassName=<class>,size=<size>,mountPath=<path> |
690+
| `--volume` | TEXT | No | Volume configuration. Format: --volume name=<name>,mountPath=<path>,persistentVolumeClaimName=<pvc_name>. Use multiple --volume flags for multiple volumes. |
691+
| `--accelerator-partition-count` | TEXT | No | Fractional GPU partition count, e.g. '1' |
692+
| `--accelerator-partition-type` | TEXT | No | Fractional GPU partition type, e.g. 'mig-3g.20gb' |
693+
| `--gpu-limit` | TEXT | No | GPU resource limit, e.g. '1' |
694+
| `--gpu` | TEXT | No | GPU resource request, e.g. '1' |
695+
| `--memory-limit` | TEXT | No | Memory resource limit, e.g. '2Gi' |
696+
| `--memory` | TEXT | No | Memory resource request, e.g. '2Gi' |
697+
| `--cpu-limit` | TEXT | No | CPU resource limit, e.g. '500m' |
698+
| `--cpu` | TEXT | No | CPU resource request, e.g. '500m' |
699+
700+
#### List Spaces
701+
702+
```bash
703+
hyp list hyp-space
704+
```
705+
706+
#### Describe a Space
707+
708+
```bash
709+
hyp describe hyp-space --name myspace
710+
```
711+
712+
#### Update a Space
713+
714+
```bash
715+
hyp update hyp-space \
716+
--name myspace \
717+
--display-name "Updated Space Name"
718+
```
719+
720+
#### Start/Stop a Space
721+
722+
```bash
723+
hyp start hyp-space --name myspace
724+
hyp stop hyp-space --name myspace
725+
```
726+
727+
#### Get Logs
728+
729+
```bash
730+
hyp get-logs hyp-space --name myspace
731+
```
732+
733+
#### Delete a Space
734+
735+
```bash
736+
hyp delete hyp-space --name myspace
737+
```
738+
739+
#### Space Template Management
740+
741+
Create reusable space templates:
742+
743+
```bash
744+
hyp create hyp-space-template --file template.yaml
745+
hyp list hyp-space-template
746+
hyp describe hyp-space-template --name <template-name>
747+
hyp update hyp-space-template --name <template-name> --file updated-template.yaml
748+
hyp delete hyp-space-template --name <template-name>
749+
```
750+
751+
#### Space Access
752+
753+
Create remote access to spaces:
754+
755+
```bash
756+
hyp create hyp-space-access --name myspace --connection-type vscode-remote
757+
hyp create hyp-space-access --name myspace --connection-type web-ui
758+
```
759+
617760
## SDK
618761
619762
Along with the CLI, we also have SDKs available that can perform the cluster management, training and inference functionalities that the CLI performs
@@ -993,6 +1136,159 @@ from sagemaker.hyperpod.observability.utils import get_monitoring_config
9931136
monitor_config = get_monitoring_config()
9941137
```
9951138
1139+
### Space SDK
1140+
1141+
#### Creating a Space
1142+
1143+
```python
1144+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1145+
from hyperpod_space_template.v1_0.model import SpaceConfig
1146+
1147+
# Create space configuration
1148+
space_config = SpaceConfig(
1149+
name="myspace",
1150+
namespace="default",
1151+
display_name="My Space",
1152+
)
1153+
1154+
# Create and start the space
1155+
space = HPSpace(config=space_config)
1156+
space.create()
1157+
```
1158+
1159+
#### List Spaces
1160+
1161+
```python
1162+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1163+
1164+
# List all spaces in default namespace
1165+
spaces = HPSpace.list()
1166+
for space in spaces:
1167+
print(f"Space: {space.config.name}, Status: {space.status}")
1168+
1169+
# List spaces in specific namespace
1170+
spaces = HPSpace.list(namespace="your-namespace")
1171+
```
1172+
1173+
#### Get a Space
1174+
1175+
```python
1176+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1177+
1178+
# Get specific space
1179+
space = HPSpace.get(name="myspace", namespace="default")
1180+
print(f"Space name: {space.config.name}")
1181+
print(f"Display name: {space.config.display_name}")
1182+
```
1183+
1184+
#### Update a Space
1185+
1186+
```python
1187+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1188+
1189+
# Get existing space
1190+
space = HPSpace.get(name="myspace")
1191+
1192+
# Update space configuration
1193+
space.update(
1194+
display_name="Updated Space Name",
1195+
)
1196+
```
1197+
1198+
#### Start/Stop a Space
1199+
1200+
```python
1201+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1202+
1203+
# Get existing space
1204+
space = HPSpace.get(name="myspace")
1205+
1206+
# Start the space
1207+
space.start()
1208+
1209+
# Stop the space
1210+
space.stop()
1211+
```
1212+
1213+
#### Get Space Logs
1214+
1215+
```python
1216+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1217+
1218+
# Get space and retrieve logs
1219+
space = HPSpace.get(name="myspace")
1220+
1221+
# Get logs from default pod and container
1222+
logs = space.get_logs()
1223+
print(logs)
1224+
```
1225+
1226+
#### List Space Pods
1227+
1228+
```python
1229+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1230+
1231+
# Get space and list associated pods
1232+
space = HPSpace.get(name="myspace")
1233+
pods = space.list_pods()
1234+
for pod in pods:
1235+
print(f"Pod: {pod}")
1236+
```
1237+
1238+
#### Create Space Access
1239+
1240+
```python
1241+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1242+
1243+
# Get existing space
1244+
space = HPSpace.get(name="myspace")
1245+
1246+
# Create VS Code remote access
1247+
vscode_access = space.create_space_access(connection_type="vscode-remote")
1248+
print(f"VS Code URL: {vscode_access['SpaceConnectionUrl']}")
1249+
1250+
# Create web UI access
1251+
web_access = space.create_space_access(connection_type="web-ui")
1252+
print(f"Web UI URL: {web_access['SpaceConnectionUrl']}")
1253+
```
1254+
1255+
#### Delete a Space
1256+
1257+
```python
1258+
from sagemaker.hyperpod.space.hyperpod_space import HPSpace
1259+
1260+
# Get existing space
1261+
space = HPSpace.get(name="myspace")
1262+
1263+
# Delete the space
1264+
space.delete()
1265+
```
1266+
1267+
#### Space Template Management
1268+
1269+
```python
1270+
from sagemaker.hyperpod.space.hyperpod_space_template import HPSpaceTemplate
1271+
1272+
# Create space template from YAML file
1273+
template = HPSpaceTemplate(file_path="template.yaml")
1274+
template.create()
1275+
1276+
# List all space templates
1277+
templates = HPSpaceTemplate.list()
1278+
for template in templates:
1279+
print(f"Template: {template.name}")
1280+
1281+
# Get specific space template
1282+
template = HPSpaceTemplate.get(name="my-template")
1283+
print(template.to_yaml())
1284+
1285+
# Update space template
1286+
template.update(file_path="updated-template.yaml")
1287+
1288+
# Delete space template
1289+
template.delete()
1290+
```
1291+
9961292
## Examples
9971293
#### Cluster Management Example Notebooks
9981294

doc/cli/cli_index.rst

Lines changed: 10 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,10 +10,11 @@ Complete reference for the SageMaker HyperPod Command Line Interface.
1010
cluster_management/cli_cluster_management
1111
training/cli_training
1212
inference/cli_inference
13+
space/cli_space
1314

1415
.. container::
1516

16-
.. grid:: 1 1 3 3
17+
.. grid:: 1 1 4 4
1718
:gutter: 3
1819

1920
.. grid-item-card:: Cluster Management CLI
@@ -35,4 +36,11 @@ Complete reference for the SageMaker HyperPod Command Line Interface.
3536
:link-type: doc
3637
:class-card: sd-border-secondary
3738

38-
Inference CLI commands, options and parameters.
39+
Inference CLI commands, options and parameters.
40+
41+
.. grid-item-card:: Space CLI
42+
:link: space/cli_space
43+
:link-type: doc
44+
:class-card: sd-border-secondary
45+
46+
Space management commands, options and parameters.

0 commit comments

Comments
 (0)