[build_tools] Add build time analysis tool for CI #2399

WBobby · 2025-12-03T17:13:16Z

Motivation

Add a new tool to analyze Ninja build times and generate HTML reports, helping identify slow-building components in the ROCm build.

Issue Link: #2214

Technical Details

Changes:

Add analyze_build_times.py: parses .ninja_log and generates HTML report
Add report_build_time_template.html: HTML template for the report
Integrate into CI workflow to auto-generate reports after builds

Test Plan

Initiate Continuous Integration (CI) to validate the code.
Execute the test script in the local environment.

Test Result

https://therock-ci-artifacts.s3.amazonaws.com/19685707430-linux/logs/gfx94X-dcgpu/build_time_analysis.html
The script successfully generates the HTML file.

Submission Checklist

Look over the contributing guidelines at https://github.com/ROCm/ROCm/blob/develop/CONTRIBUTING.md#pull-requests.

Add a new tool to analyze Ninja build times and generate HTML reports, helping identify slow-building components in the ROCm build. Changes: - Add analyze_build_times.py: parses .ninja_log and generates HTML report - Add report_build_time_template.html: HTML template for the report - Integrate into CI workflow to auto-generate reports after builds

amd-shiraz · 2025-12-03T17:40:40Z

build_tools/analyze_build_times.py

+# =============================================================================
+
+
+def get_system_info() -> Dict[str, str]:


thanks for adding this to the report.

amd-shiraz · 2025-12-03T17:44:11Z

If possible we also want to add "Resource utilization" per component to the same report i n future. That will help with the work we are doing with @geomin12 on dockers dying due to lack of resources. for ASAN work.

amd-shiraz · 2025-12-04T01:25:11Z

@WBobby i had one more feedback pls if we can also incorporate. can we also add a total build time info at the end for components and dependencies ? thanks!

Calculate wall clock time from ninja log and display it in the Build Information section alongside CPU, memory details. Signed-off-by: Wang, Yanyao <[email protected]>

kiran-thumma · 2025-12-05T22:04:59Z

Link the issue #2399 in PR a mandatory.

.github/workflows/build_portable_linux_artifacts.yml

build_tools/analyze_build_times.py

build_tools/github_actions/post_build_upload.py

build_tools/analyze_build_times.py

kiran-thumma

LGTM. Thank You for adding comments to use the script.

ScottTodd

I'm running ~1 week behind on reviews, sorry. Here's some post-merge feedback.

ScottTodd · 2025-12-08T22:52:49Z

.github/workflows/build_portable_linux_artifacts.yml

+      # Analyze ninja build log to generate per-component timing report
+      - name: Analyze Build Times
+        if: ${{ !cancelled() }}
+        run: |
+          python3 build_tools/analyze_build_times.py --build-dir build


Does this work on Windows too?

Please keep Linux and Windows workflows in sync: https://github.com/ROCm/TheRock/blob/main/.github/workflows/build_windows_artifacts.yml

ScottTodd · 2025-12-08T22:54:06Z

build_tools/github_actions/post_build_upload.py

+    # Build Time Analysis is only generated on Linux
+    if PLATFORM == "linux":
+        analysis_url = f"{bucket_url}/logs/{artifact_group}/build_time_analysis.html"
+        gha_append_step_summary(f"[Build Time Analysis]({analysis_url})")


Why? We should have these for Windows too.

Also note that multiple workflows call this post_build_upload.py script, so this file may not exist. See this for example:

TheRock/.github/workflows/release_portable_linux_packages.yml

Lines 232 to 240 in f9e3277

# TODO: upload logs even if the build fails (separate from artifact upload?)

- name: Post Build Upload

if: ${{ github.repository_owner == 'ROCm' }}

run: |

python3 build_tools/github_actions/post_build_upload.py \

--run-id ${{ github.run_id }} \

--artifact-group "${{ matrix.target_bundle.amdgpu_family }}" \

--build-dir ${{ env.OUTPUT_DIR }}/build \

--upload

This should check if the file was found and uploaded, not if the current platform is Linux

ScottTodd · 2025-12-08T22:55:36Z

build_tools/analyze_build_times.py

+# Build name -> Display name mapping
+NAME_MAPPING = {
+    "clr": "core-hip",
+    "ocl-clr": "core-ocl",
+    "ROCR-Runtime": "core-runtime",
+    "blas": "rocBLAS",
+    "prim": "rocPRIM",
+    "fft": "rocFFT",
+    "rand": "rocRAND",
+    "miopen": "MIOpen",
+    "hipdnn": "hipDNN",
+    "composable-kernel": "composable_kernel",
+    "support": "mxDataGenerator",
+    "host-suite-sparse": "SuiteSparse",
+    "rocwmma": "rocWMMA",
+    "miopen-plugin": "miopen_plugin",
+}
+
+# Top-level directories for ROCm components
+ROCM_COMPONENT_DIRS = {
+    "base",
+    "compiler",
+    "core",
+    "comm-libs",
+    "dctools",
+    "profiler",
+    "ml-libs",
+}


Do we need these mappings in this file? This may be difficult to maintain in the future as projects are moved around (https://github.com/ROCm/TheRock/blob/main/docs/rfcs/RFC0003-Build-Tree-Normalization.md)

ScottTodd · 2025-12-08T22:57:07Z

build_tools/analyze_build_times.py

+    # Get CPU model from /proc/cpuinfo
+    try:
+        with open("/proc/cpuinfo", "r") as f:
+            for line in f:
+                if line.startswith("model name"):
+                    info["cpu_model"] = line.split(":")[1].strip()
+                    break
+    except (FileNotFoundError, IOError):
+        pass


This is definitely Linux-specific.

We have some code for this that is more portable here:

TheRock/build_tools/hack/env_check/device.py

Lines 159 to 199 in f9e3277

def device_cpu_status(self):

"""

**Warning:** This function may broken in Cluster systems.

Return CPU status, include its name, architecture, total cpu count.

-> `(CPU_NAME, CPU_ARCH, CPU_CORES)`

"""

import os, platform, subprocess, re

if self.is_windows:

_cpu_name = get_regedit(

"HKLM",

r"HARDWARE\DESCRIPTION\System\CentralProcessor\0",

"ProcessorNameString",

)

_cpu_arch = platform.machine()

_cpu_core = os.cpu_count()

return (_cpu_name, _cpu_core, _cpu_arch)

elif self.is_linux:

_cpu_name = (

re.search(

r"^\s*Model name:\s*(.+)$",

subprocess.run(

["lscpu"], capture_output=True, text=True, check=True

).stdout,

re.MULTILINE,

)

.group(1)

.strip()

)

_cpu_arch = platform.machine()

_cpu_core = os.cpu_count()

return (_cpu_name, _cpu_core, _cpu_arch)

else:

# <ADD BSD/Intel_MAC ???>

...

pass

Can we just run that env_check and pull that information into these analysis reports instead of duplicating code? Or we can pull those functions into a common module used for both scripts.

ScottTodd · 2025-12-08T23:00:04Z

build_tools/analyze_build_times.py

+def build_table_rows(
+    data: Dict[str, Dict[str, int]], phase_columns: List[str]
+) -> List[tuple]:
+    """Build sorted table rows from project phase data."""
+    rows = []
+    for name, phases in data.items():
+        total = sum(phases.values())
+        cols = [format_duration(phases.get(p, 0)) for p in phase_columns]
+        rows.append((name, cols, format_duration(total), total))
+    rows.sort(key=lambda x: x[3], reverse=True)
+    return [(r[0], r[1], r[2]) for r in rows]


I would like to be able to sort https://therock-ci-artifacts.s3.amazonaws.com/19685707430-linux/logs/gfx94X-dcgpu/build_time_analysis.html based on project name too. Let's add "make the table dynamic" as a feature request for later

ScottTodd · 2025-12-08T23:01:25Z

build_tools/analyze_build_times.py

+def build_table_rows(
+    data: Dict[str, Dict[str, int]], phase_columns: List[str]
+) -> List[tuple]:
+    """Build sorted table rows from project phase data."""
+    rows = []
+    for name, phases in data.items():
+        total = sum(phases.values())
+        cols = [format_duration(phases.get(p, 0)) for p in phase_columns]
+        rows.append((name, cols, format_duration(total), total))
+    rows.sort(key=lambda x: x[3], reverse=True)
+    return [(r[0], r[1], r[2]) for r in rows]


More feature requests:

Link to ninja logs from this report

Link to https://ui.perfetto.dev/ from this report

Embed https://ui.perfetto.dev/ into this report directly

github-project-automation bot added this to TheRock Triage Dec 3, 2025

github-project-automation bot moved this to TODO in TheRock Triage Dec 3, 2025

style: format analyze_build_times.py with black

0812920

WBobby requested review from ScottTodd and amd-shiraz December 3, 2025 17:37

amd-shiraz reviewed Dec 3, 2025

View reviewed changes

amd-shiraz approved these changes Dec 3, 2025

View reviewed changes

WBobby mentioned this pull request Dec 3, 2025

Collect the individual Component Build Times #2214

Open

yanyao-wang and others added 2 commits December 3, 2025 23:52

fix: skip Build Time Analysis link in GHA summary on Windows

8ccd456

Merge branch 'main' into users/yanywang/3-12-build-time-analysis

c1fc904

feat(build_tools): add build duration to HTML report

d86c46c

Calculate wall clock time from ninja log and display it in the Build Information section alongside CPU, memory details. Signed-off-by: Wang, Yanyao <[email protected]>

kiran-thumma reviewed Dec 5, 2025

View reviewed changes

.github/workflows/build_portable_linux_artifacts.yml Show resolved Hide resolved

kiran-thumma reviewed Dec 5, 2025

View reviewed changes

build_tools/analyze_build_times.py Outdated Show resolved Hide resolved

build_tools/github_actions/post_build_upload.py Show resolved Hide resolved

build_tools/analyze_build_times.py Show resolved Hide resolved

build_tools/analyze_build_times.py Show resolved Hide resolved

Add documentation and comments for build time analysis

5a3294c

WBobby requested a review from kiran-thumma December 6, 2025 00:53

kiran-thumma approved these changes Dec 8, 2025

View reviewed changes

WBobby merged commit 2e0faff into main Dec 8, 2025
232 of 241 checks passed

WBobby deleted the users/yanywang/3-12-build-time-analysis branch December 8, 2025 22:21

github-project-automation bot moved this from TODO to Done in TheRock Triage Dec 8, 2025

ScottTodd reviewed Dec 8, 2025

View reviewed changes

		# =============================================================================


		def get_system_info() -> Dict[str, str]:

	# TODO: upload logs even if the build fails (separate from artifact upload?)
	- name: Post Build Upload
	if: ${{ github.repository_owner == 'ROCm' }}
	run: \|
	python3 build_tools/github_actions/post_build_upload.py \
	--run-id ${{ github.run_id }} \
	--artifact-group "${{ matrix.target_bundle.amdgpu_family }}" \
	--build-dir ${{ env.OUTPUT_DIR }}/build \
	--upload

	def device_cpu_status(self):
	"""
	Warning: This function may broken in Cluster systems.
	Return CPU status, include its name, architecture, total cpu count.
	-> `(CPU_NAME, CPU_ARCH, CPU_CORES)`
	"""

	import os, platform, subprocess, re

	if self.is_windows:
	_cpu_name = get_regedit(
	"HKLM",
	r"HARDWARE\DESCRIPTION\System\CentralProcessor\0",
	"ProcessorNameString",
	)
	_cpu_arch = platform.machine()
	_cpu_core = os.cpu_count()

	return (_cpu_name, _cpu_core, _cpu_arch)

	elif self.is_linux:
	_cpu_name = (
	re.search(
	r"^\sModel name:\s(.+)$",
	subprocess.run(
	["lscpu"], capture_output=True, text=True, check=True
	).stdout,
	re.MULTILINE,
	)
	.group(1)
	.strip()
	)
	_cpu_arch = platform.machine()
	_cpu_core = os.cpu_count()

	return (_cpu_name, _cpu_core, _cpu_arch)

	else:
	# <ADD BSD/Intel_MAC ???>
	...
	pass

[build_tools] Add build time analysis tool for CI #2399

[build_tools] Add build time analysis tool for CI #2399

Conversation

WBobby commented Dec 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Motivation

Technical Details

Test Plan

Test Result

Submission Checklist

Uh oh!

amd-shiraz Dec 3, 2025

Choose a reason for hiding this comment

Uh oh!

amd-shiraz commented Dec 3, 2025

Uh oh!

amd-shiraz commented Dec 4, 2025

Uh oh!

kiran-thumma commented Dec 5, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

kiran-thumma left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ScottTodd left a comment

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

ScottTodd Dec 8, 2025

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

WBobby commented Dec 3, 2025 •

edited

Loading

kiran-thumma left a comment •

edited

Loading