The script checks for issues that are known to have an impact to the success of a Nexus Dashboard upgrade. Identification and remediation of such issues prior to upgrade can lead to more successful outcomes.
The script can be run from any Linux server with the required dependencies. If ACI APICs are available and have connectivity towards the ND management address, the script can be run from an APIC as these devices already contain all dependencies required to run the script. The script cannot be run from the Nexus Dashboard itself.
- The script can be run against any ND versions from Nexus Dashboard Version 2.1 and later
- If you have feedback or are encountering an issue running the script, please send an email to [email protected]
Note: Some of the checks are version specific and might not apply to the current running version of Nexus Dashboard in your environment. In the event that the check is not relevant, the check will be a "PASS".
| # | Check | Description | Reference |
|---|---|---|---|
| 1 | Techsupport | Verify the script could successfully extract the tech support to perform additional checks | |
| 2 | Version | Verify all cluster nodes are running the same version | |
| 3 | Node status | Verify all cluster nodes are in an Active state | |
| 4 | Ping check | Verify node can ping all Mgmt and Data IPs in the cluster | |
| 5 | Subnet check | Verify Mgmt and Data interfaces are on different subnets | Documentation |
| 6 | Persistent IP check | Verify at lest 5 Persistent IP address are configured in data-external-services | Documentation |
| 7 | Disk space | Verify all directories are under 70% utilization | |
| 8 | Pod status | Verify all Pods and Services are in a healthy state | |
| 9 | System health | Verify all nodes are healthy using 'acs health' command | |
| 10 | NXOS Discovery Service | Verify cisco-ndfc k8 App is not stuck in Disabling | CSCwm97680 |
| 11 | Backup failure check | Verify latest backup is not in Failed or InProgress state | CSCwq57968, CSCwm96512 |
| 12 | Nameserver duplicate check | Verify no duplicate nameservers exist in acs_system_config | |
| 13 | Certificate check | Verify no certificates have non-alphanumeric characters | CSCwm35992 |
| 14 | ISO check | Verify multiple ISOs aren't found in boothook | CSCwn94394 |
| 15 | Lvm Pvs check | Verify no empty ElasticSearch PVs are found | CSCwe91228 |
| 16 | atom0 NVME check | Verify no NVME drive hardware failures are present in Physical node setups | |
| 17 | atom0 vg check | Verify there is more than 50% free space in atom0 virtual group | CSCwr43515 |
The main script requires Python 3.7+ while the worker script is compatible with Python 2.7+ to accommodate older ND node versions.
Most dependencies are part of Python's standard library. For the few external dependencies:
-
Install Python dependencies using pip:
pip install -r requirements.txt
Or for Python 3 specifically:
pip3 install -r requirements.txt
-
Install system dependencies:
- Linux/Unix: Install
sshpassfor SSH authentication:sudo apt-get update sudo apt-get install sshpass
- macOS: Use Homebrew:
brew install hudochenkov/sshpass/sshpass
- Windows: Consider using WSL (Windows Subsystem for Linux) for best compatibility
- Linux/Unix: Install
- ipaddr/ipaddress: For IP address validation and manipulation
ipaddrpackage for Python 2.7 (worker script on older ND nodes)- Built-in
ipaddressmodule for Python 3+ (main script and newer ND nodes)
- sshpass: External system tool for password-based SSH authentication
- All other dependencies are part of Python's standard library
Place the main script ND-Preupgrade-Validation.py and the worker script worker_functions.py in the same directory and execute the main script with the command python ND-Preupgrade-Validation.py or python3 ND-Preupgrade-Validation.py
- The main script is currently only compatible with Python 3.7+ but will be modified in the future.
- The worker script is Python 2.7+ compatible to accommodate ND nodes on versions prior to 3.0.
[root@localhost preupgrade]# python3 ND-Preupgrade-Validation.py
Nexus Dashboard Pre-upgrade Validation Script
Running validation checks on 2025-11-06 13:25:35
Enter Nexus Dashboard IP address: 14.2.29.130
Enter password for rescue-user:
Connecting to Nexus Dashboard at 14.2.29.130...
Successfully connected to Nexus Dashboard at 14.2.29.130
Nexus Dashboard version: 3.2.2f
Discovering Nexus Dashboard nodes...
Found 3 Nexus Dashboard nodes
---------------------------------------
ND1 (14.2.29.130)
ND2 (14.2.29.131)
ND3 (14.2.29.132)
---------------------------------------
Checking disk space on all nodes...
Checking ND1... PASS
Checking ND2... PASS
Checking ND3... PASS
PASS All 3 nodes are active and have healthy disk space (<80% usage).
PASS All nodes are ready for validation.
System resource assessment:
CPU cores: 40
Memory: 62.36 GB
Current load: 0.03
Recommended concurrent nodes: 7
Do you want to generate new tech supports for analysis?
1. Generate new tech supports for analysis
2. Use existing tech supports on all nodes
Enter your choice (1/2): 1
================================================================================
Tech Support Selection/Generation
================================================================================
Generating tech supports for all nodes in parallel.
Starting tech support collection on ND1. This may take several minutes...
Starting tech support collection on ND3. This may take several minutes...
Starting tech support collection on ND2. This may take several minutes...
Tech support collection started successfully on ND3
Tech support collection started successfully on ND1
Tech support collection started successfully on ND2
Found tech support file being generated on ND2, monitoring for completion...
Found tech support file being generated on ND3, monitoring for completion...
Found tech support file being generated on ND1, monitoring for completion...
Tech support still generating on ND2: 1.66 GB (+1179.0 MB)...
Tech support still generating on ND3: 0.85 GB (+270.8 MB)...
Tech support still generating on ND1: 1.47 GB (+1017.5 MB)...
Tech support still generating on ND2: 1.71 GB (+50.1 MB)...
File size stable on ND3, performing verification checks...
File size stable on ND1, performing verification checks...
Tech support generation confirmed complete on ND3
PASS Tech support generated on ND3: /techsupport/2025-11-06T18-52-56Z-system-ts-ND3.tgz
Tech support generation confirmed complete on ND1
PASS Tech support generated on ND1: /techsupport/2025-11-06T18-52-59Z-system-ts-ND1.tgz
Generated tech support on ND3: 2025-11-06T18-52-56Z-system-ts-ND3.tgz
Generated tech support on ND1: 2025-11-06T18-52-59Z-system-ts-ND1.tgz
File size stable on ND2, performing verification checks...
Tech support generation confirmed complete on ND2
PASS Tech support generated on ND2: /techsupport/2025-11-06T18-52-58Z-system-ts-ND2.tgz
Generated tech support on ND2: 2025-11-06T18-52-58Z-system-ts-ND2.tgz
================================================================================
Pre-Flight /tmp Space Validation
================================================================================
Checking /tmp disk space on all nodes before extraction...
This validation ensures sufficient space for tech support extraction (70% threshold).
Checking space on ND1... PASS
Tech support: 1.47 GB | Current: 18.0% | Projected: 45.5%
Checking space on ND2... PASS
Tech support: 1.71 GB | Current: 1.0% | Projected: 33.1%
Checking space on ND3... PASS
Tech support: 0.85 GB | Current: 7.0% | Projected: 23.0%
PASS All nodes passed /tmp space validation!
Proceeding with worker script deployment...
================================================================================
Deploying Worker Scripts and Starting Validation
================================================================================
Deploying worker script to node ND1...
Deploying worker script to node ND2...
Deploying worker script to node ND3...
PASS Worker script deployed to ND1
Starting validation on node ND1...
PASS Worker script deployed to ND2
Starting validation on node ND2...
PASS Worker script deployed to ND3
Starting validation on node ND3...
PASS Validation started on ND3
PASS Validation started on ND1
PASS Validation started on ND2
Monitoring validation progress on 3 nodes in batch 1...
Running validation checks on 3 Nexus Dashboard nodes...
[Node ND3] Validation complete
[Node ND1] Validation complete
[Node ND2] Validation complete
Collecting results and logs from all nodes...
Monitoring completed in 0 minutes 3 seconds
================================================================================
Cleaning Up Temporary Files
================================================================================
Cleaning up temporary files from 3 nodes...
PASS Cleaned up temporary files on ND1
PASS Cleaned up temporary files on ND2
PASS Cleaned up temporary files on ND3
================================================================================
Pre-upgrade Validation Report
================================================================================
Total validation time: 3 min 37 sec
Report generated on: 2025-11-06 13:29:24
[Check 1/17] Techsupport... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS Tech support generated and extracted successfully
ND2 PASS Tech support generated and extracted successfully
ND3 PASS Tech support generated and extracted successfully
[Check 2/17] Version Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS All cluster nodes are on the same version: 3.2.2f
ND2 PASS All cluster nodes are on the same version: 3.2.2f
ND3 PASS All cluster nodes are on the same version: 3.2.2f
[Check 3/17] Node Status... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS All nodes in the cluster are in Active state
ND2 PASS All nodes in the cluster are in Active state
ND3 PASS All nodes in the cluster are in Active state
[Check 4/17] Ping Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS Node can ping all mgmt and data ips in the cluster
ND2 PASS Node can ping all mgmt and data ips in the cluster
ND3 PASS Node can ping all mgmt and data ips in the cluster
[Check 5/17] Subnet Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS Mgmt and Data interfaces are in different subnets
ND2 PASS Mgmt and Data interfaces are in different subnets
ND3 PASS Mgmt and Data interfaces are in different subnets
[Check 6/17] Persistent Ip Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS Found 10 persistent IP addresses (meets minimum requirement of 5)
ND2 PASS Found 10 persistent IP addresses (meets minimum requirement of 5)
ND3 PASS Found 10 persistent IP addresses (meets minimum requirement of 5)
[Check 7/17] Disk Space... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS All directories are under 70% usage
ND2 PASS All directories are under 70% usage
ND3 PASS All directories are under 70% usage
[Check 8/17] Pod Status... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS All Pods and Services are in a healthy state
ND2 PASS All Pods and Services are in a healthy state
ND3 PASS All Pods and Services are in a healthy state
[Check 9/17] System Health... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS acs health indicates Node is healthy
ND2 PASS acs health indicates Node is healthy
ND3 PASS acs health indicates Node is healthy
[Check 10/17] Nxos Discovery Service... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS Deployment Mode is not ndfc-fabric-ndi
ND2 PASS Deployment Mode is not ndfc-fabric-ndi
ND3 PASS Deployment Mode is not ndfc-fabric-ndi
[Check 11/17] Backup Failure Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS All backup jobs completed successfully
ND2 PASS All backup jobs completed successfully
ND3 PASS All backup jobs completed successfully
[Check 12/17] Nameserver Duplicate Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS No duplicate nameservers found
ND2 PASS No duplicate nameservers found
ND3 PASS No duplicate nameservers found
[Check 13/17] Certificate Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS No certificate issues found in SM logs
ND2 PASS No certificate issues found in SM logs
ND3 PASS No certificate issues found in SM logs
[Check 14/17] Iso Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS No multiple ISO issues found
ND2 PASS No multiple ISO issues found
ND3 PASS No multiple ISO issues found
[Check 15/17] Lvm Pvs Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS No empty Elasticsearch PVs found
ND2 PASS No empty Elasticsearch PVs found
ND3 PASS No empty Elasticsearch PVs found
[Check 16/17] Atom0 Nvme Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS NVME drive seen by ND in PVS file
ND2 PASS NVME drive seen by ND in PVS file
ND3 PASS NVME drive seen by ND in PVS file
[Check 17/17] Atom0 Vg Check... PASS
Node Status Details
---------- -------- --------------------------------------------------
ND1 PASS atom0 vg has more than 50G free space
ND2 PASS atom0 vg has more than 50G free space
ND3 PASS atom0 vg has more than 50G free space
Detailed results are available in /root/Downloads/preupgrade/final-results/
Results Bundle: /root/Downloads/preupgrade/nd-preupgrade-validation-results_2025-11-06T13-29-24.tgz
[root@localhost preupgrade]# ls -lh
total 376K
drwxr-xr-x. 2 root root 236 Nov 6 13:29 final-results
-rw-r--r--. 1 root root 178K Nov 6 13:25 ND-Preupgrade-Validation.py
-rw-r--r--. 1 root root 12K Nov 6 13:29 nd-preupgrade-validation-results_2025-11-06T13-29-24.tgz
-rw-r--r--. 1 root root 181K Nov 6 13:25 worker_functions.py
[root@localhost preupgrade]# ls -lh final-results/
total 128K
-rw-r--r--. 1 root root 9.2K Nov 6 13:29 ND1_output.log
-rw-r--r--. 1 root root 2.4K Nov 6 13:29 ND1_results.json
-rw-r--r--. 1 root root 9.2K Nov 6 13:29 ND2_output.log
-rw-r--r--. 1 root root 2.4K Nov 6 13:29 ND2_results.json
-rw-r--r--. 1 root root 9.2K Nov 6 13:29 ND3_output.log
-rw-r--r--. 1 root root 2.4K Nov 6 13:29 ND3_results.json
-rw-r--r--. 1 root root 61K Nov 6 13:29 nd_validation_debug.log
-rw-r--r--. 1 root root 6.2K Nov 6 13:29 validation_details.json
-rw-r--r--. 1 root root 6.6K Nov 6 13:29 validation_summary.txt
- Feedback and questions are welcome.
- If you have feedback or are encountering an issue running the script, please send an email to [email protected]
- If the script is recommending to open a Cisco TAC SR for any specific issue, please open a Cisco TAC SR.
- When opening a TAC SR, upload the
nd-preupgrade-validation-results*.tgzfile which includes all results and diagnostics. - Running this script is not a requirement but it is intended to provide additional reassurance when planning for a Nexus Dashboard upgrade.