Add tool to compare built ipa and master ipa initramfs #1058

peppi-lotta · 2025-07-29T12:06:56Z

This PR introduces a script ipa_debug_tools.sh that provides a set of utilities to assist with debugging and comparing IPA (ironic-python-agent) initramfs builds. The script is particularly useful for identifying differences between locally built IPA images and the official master images.

This tool is meant especially for situations where built IPA size get over a certain limit limit and two file systems need to be compared.

🛠️ What's Included

set_up_debug_dirs
- Sets up a local debug environment under /tmp/debug-initramfs
- Builds the local IPA using jenkins/scripts/dynamic_worker_workflow/build_ipa.sh
- Downloads and extracts the master IPA image from upstream
compare_dir_sizes [path]
- Compares sizes of files/directories between the local build and master IPA
- Provides a colored, sorted table of size differences
- Highlights large discrepancies (10MB+, 1MB+, etc.)
- Lists files exclusive to either initramfs tree

compare_rpm_packages
- Extracts and compares installed RPMs between local and master IPA environments
- Outputs side-by-side view of mismatched or missing packages
- Helps identify unintended package changes

clean_up_debug_dirs
- Deletes all temporary directories and files created during debugging

metal3-io-bot · 2025-07-29T12:07:02Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please assign tuminoid for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Rozzii

In general LGTM,
It is a very useful tool as I have mentioned offline.
I have one improvement proposal but it can be a later addition if you don't feel like implementing it now.

Rozzii · 2025-07-31T06:15:14Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


It would be nice if the upstream IPA URI and tar file name could be customized same as in IPA downloader, you could actually use IPA downloader here instead of your own download logic to keep it simple .

For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

I am fine either way, but the url and the name of the tar file has to be customizable because otherwise only a specific master on a specific base image can be tested. I suggested to use the downloader because that has all the customization options implemented but I am fine with doing it with the one liners too.

Rozzii · 2025-07-31T06:22:40Z

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

metal3-io-bot · 2025-07-31T06:22:45Z

@Rozzii: Overrode contexts on behalf of Rozzii: metal3-ubuntu-e2e-integration-test-main

In response to this:

/override metal3-ubuntu-e2e-integration-test-main
This is a hacktool for manual testing, not used in CI jet.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

tuminoid

This is great addition to debug tools, thanks @peppi-lotta.

Some cleanup to do, some minor syntax things to fix and some suggestions to make it more readable and maintainable.

hack/ipa_debug_tools.sh

tuminoid · 2025-08-05T06:12:15Z

hack/ipa_debug_tools.sh

+    export TEST_IN_CI="false"
+
+    # Build ipa with build_ipa.sh
+    cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" || exit


Suggested change

cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" || exit

cd "${CURRENT_SCRIPT_DIR}/../jenkins/scripts/dynamic_worker_workflow"

|| exit 1 not needed if we're running set -e mentioned earlier, cleans up the code.

tuminoid · 2025-08-05T06:12:40Z

hack/ipa_debug_tools.sh

+    # Unzip ironic-python-agent.initramfs
+    sudo mkdir -p /tmp/debug-initramfs/build-ipa-initramfs
+    sudo cp /tmp/dib/ironic-python-agent.initramfs /tmp/debug-initramfs/
+    cd /tmp/debug-initramfs/build-ipa-initramfs || exit


Suggested change

cd /tmp/debug-initramfs/build-ipa-initramfs || exit

cd /tmp/debug-initramfs/build-ipa-initramfs

tuminoid · 2025-08-05T06:19:00Z

hack/ipa_debug_tools.sh

+    sudo mkdir -p /tmp/debug-initramfs/master-ipa-initramfs
+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz


For debug tool, it might be good enough to be able to supply the URL. get-resource.sh vs one-liner... I'd go for the latter.

tuminoid · 2025-08-05T06:19:09Z

hack/ipa_debug_tools.sh

+    cd /tmp/debug-initramfs || exit
+    sudo wget https://tarballs.opendev.org/openstack/ironic-python-agent/dib/ipa-centos9-master.tar.gz
+    sudo tar -xzf ipa-centos9-master.tar.gz
+    cd master-ipa-initramfs || exit


Suggested change

cd master-ipa-initramfs || exit

cd master-ipa-initramfs

tuminoid · 2025-08-05T06:31:58Z

hack/ipa_debug_tools.sh

+#the built ipa and master.
+# Check if packages are a part of CentOS Stream in https://pkgs.org
+compare_rpm_packages() {
+    basedir1="${1:-/tmp/debug-initramfs/build-ipa-initramfs}" 


Same as above for descriptive dir and files names.

tuminoid · 2025-08-05T06:32:19Z

hack/ipa_debug_tools.sh

+    common_count=$(comm -12 "${compfile1}" "${compfile2}" | wc -l)
+    echo -e "\nNumber of common packages: ${common_count}\n"
+    # Extract base package names and join for fuzzy matching
+    awk -F'-[0-9]' '{print $1}' "${compfile1}" | sort > /tmp/build-ipa-base.txt


Same suggestion for tmpfile names.

tuminoid · 2025-08-05T06:32:49Z

hack/ipa_debug_tools.sh

+        lfull=$(grep -m1 "^${left}-" "${compfile1}" || echo "")
+        rfull=$(grep -m1 "^${right}-" "${compfile2}" || echo "")
+        # Skip printing if both lines are non-empty and exactly the same
+        if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then


Suggested change

if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then

if [[ -n "${lfull}" ]] && [[ "${lfull}" == "${rfull}" ]]; then

tuminoid · 2025-08-05T06:34:02Z

hack/ipa_debug_tools.sh

+function_name="$1"
+shift
+
+# Check if function exists and call it


Why not use standard argument parsing here, much more readable and reliable?

tuminoid · 2025-08-05T06:34:39Z

hack/ipa_debug_tools.sh

+# Usage:
+#   ./ipa_debug_tools.sh clean_up_debug_dirs
+# -----------------------------------------------------------------------------
+clean_up_debug_dirs() {


Cleaning up all the temp files should be done here, and the cleanup should be trapped function so it gets called even if we exit mid run.

Rozzii · 2025-09-29T17:56:01Z

hack/ipa_debug_tools.sh

+# -----------------------------------------------------------------------------
+clean_up_debug_dirs() {
+    # Clean up debug directories
+    sudo rm -rf "${BASE_DEBUG_DIR}" /tmp/dib ipa-file-injector.service 2> /dev/null


I get "${BASE_DEBUG_DIR}" and /tmp/dib but how does ipa-file-injector.service end up here?
I don't see any other reference to that file, and AFAIK that is the systemd unit file of the similarly named DIB module. It could be that I am missing something here but looks like this is some leftover reference maybe.

ipa-file-injector.service appears in the /tmp directory after running the build-ip.sh. That is why I've added that to the clean up as well.

okay, that might be then some sort of residue from the template rendering process. I might remove it later, but then it is fine to remove it here.

tuminoid

IMO colors are fine, but emojis not.

hack/ipa_debug_tools.sh

Signed-off-by: peppi-lotta <[email protected]>

Rozzii

/lgtm

metal3-io-bot requested review from Rozzii and smoshiur1237 July 29, 2025 12:07

metal3-io-bot added the size/L Denotes a PR that changes 100-499 lines, ignoring generated files. label Jul 29, 2025

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 4c2b687 to 9e9d4b4 Compare July 30, 2025 06:15

Rozzii reviewed Jul 31, 2025

View reviewed changes

Rozzii added this to Metal3 - Roadmap Jul 31, 2025

Rozzii moved this to MISC WIP in Metal3 - Roadmap Jul 31, 2025

tuminoid reviewed Aug 5, 2025

View reviewed changes

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch 2 times, most recently from 00054eb to 7a7af56 Compare September 29, 2025 09:51

Rozzii reviewed Sep 29, 2025

View reviewed changes

tuminoid reviewed Sep 30, 2025

View reviewed changes

Add tool to compare built ipa and master ipa initramfs

27494cb

Signed-off-by: peppi-lotta <[email protected]>

peppi-lotta force-pushed the peppi-lotta/add-ipa-debug-tool branch from 7a7af56 to 27494cb Compare October 3, 2025 07:16

Rozzii reviewed Oct 7, 2025

View reviewed changes

metal3-io-bot added the lgtm Indicates that a PR is ready to be merged. label Oct 7, 2025

	cd "$CURRENT_SCRIPT_DIR/../jenkins/scripts/dynamic_worker_workflow" \|\| exit
	cd "${CURRENT_SCRIPT_DIR}/../jenkins/scripts/dynamic_worker_workflow"

	cd /tmp/debug-initramfs/build-ipa-initramfs \|\| exit
	cd /tmp/debug-initramfs/build-ipa-initramfs

	if [[ -n "${lfull}" && "${lfull}" == "${rfull}" ]]; then
	if [[ -n "${lfull}" ]] && [[ "${lfull}" == "${rfull}" ]]; then

Add tool to compare built ipa and master ipa initramfs #1058

Are you sure you want to change the base?

Add tool to compare built ipa and master ipa initramfs #1058

Uh oh!

Conversation

peppi-lotta commented Jul 29, 2025

🛠️ What's Included

Uh oh!

metal3-io-bot commented Jul 29, 2025

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

Rozzii Jul 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rozzii commented Jul 31, 2025

Uh oh!

metal3-io-bot commented Jul 31, 2025

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Rozzii Sep 29, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

tuminoid left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Rozzii left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Rozzii Jul 31, 2025 •

edited

Loading

Rozzii Sep 29, 2025 •

edited

Loading