Skip to content

(kubernetes.jinja2) Introduce volume for firmware file caching #2884

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
May 29, 2025
Merged

(kubernetes.jinja2) Introduce volume for firmware file caching #2884

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
cdfdb9b
(kubernetes.jinja2) Introduce volume for firmware file caching
nuclearcat May 23, 2025
ff0235c
(kbuild) use cached copy of firmware
nuclearcat May 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions config/runtime/base/kubernetes.jinja2
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,11 @@ spec:
- name: compile-volume
emptyDir: { }

- name: resource-cache
persistentVolumeClaim:
claimName: shared-data # RWX PVC!
readOnly: true

tolerations:
- key: "kubernetes.azure.com/scalesetpriority"
operator: "Equal"
Expand All @@ -40,6 +45,9 @@ spec:
name: compile-volume
- mountPath: "/dev/shm"
name: dev-shm
- mountPath: "/data"
name: resource-cache
readOnly: true

# FIXME: Request safe defaults to not overload node with
# parallel pods
Expand Down
6 changes: 4 additions & 2 deletions kernelci/kbuild.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# SPDX-License-Identifier: LGPL-2.1-or-later

Check warning on line 1 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Too many lines in module (1116/1000)
#
# Copyright (C) 2023 Collabora Limited
# Author: Denys Fedoryshchenko <[email protected]>
Expand Down Expand Up @@ -40,16 +40,16 @@


CIP_CONFIG_URL = \
"https://gitlab.com/cip-project/cip-kernel/cip-kernel-config/-/raw/master/{branch}/{config}" # noqa

Check warning on line 43 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Line too long (104/100)
CROS_CONFIG_URL = \
"https://chromium.googlesource.com/chromiumos/third_party/kernel/+archive/refs/heads/{branch}/chromeos/config.tar.gz" # noqa

Check warning on line 45 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Line too long (129/100)
LEGACY_CONFIG = [
'config/core/build-configs.yaml',
'/etc/kernelci/core/build-configs.yaml',
]
FW_GIT = "https://git.kernel.org/pub/scm/linux/kernel/git/firmware/linux-firmware.git" # noqa

# TODO: find a way to automatically fetch this information

Check warning on line 52 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: find a way to automatically fetch this information
LATEST_LTS_MAJOR = 6
LATEST_LTS_MINOR = 6

Expand Down Expand Up @@ -112,13 +112,13 @@
except requests.exceptions.RequestException as e:
print(f"[_download_file_inner] Error: {e}")
return False
except requests.exceptions.Timeout as e:

Check failure on line 115 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Bad except clauses order (RequestException is an ancestor class of Timeout)
print(f"[_download_file_inner] Timeout: {e}")
return False
except requests.exceptions.ConnectionError as e:

Check failure on line 118 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Bad except clauses order (RequestException is an ancestor class of ConnectionError)
print(f"[_download_file_inner] Connection error: {e}")
return False
except Exception as e:

Check warning on line 121 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Catching too general exception Exception
print(f"[_download_file_inner] Exception: {e}")
return False
if r.status_code == 200:
Expand Down Expand Up @@ -423,8 +423,10 @@
# self.addcmdretry(f"git clone {FW_GIT} --depth 1", 10)
# This file available https://storage.kernelci.org/linux-firmware.tar.gz
# we download it there for better reliability
self.addcmd("wget -c -t 10 --retry-on-host-error " +
"https://storage.kernelci.org/linux-firmware.tar.gz -O linux-firmware.tar.gz")
#self.addcmd("wget -c -t 10 --retry-on-host-error " +
# "https://storage.kernelci.org/linux-firmware.tar.gz -O linux-firmware.tar.gz")
# We should have cached copy of linux-firmware.tar.gz at /data directory
self.addcmd("cp /data/linux-firmware.tar.gz .")
self.addcmd("tar -xzf linux-firmware.tar.gz")
self.addcmd("cd linux-firmware")
self.addcmd("./copy-firmware.sh " + self._firmware_dir)
Expand Down Expand Up @@ -570,7 +572,7 @@
elif fragment.startswith("CONFIG_"):
content = fragment + '\n'
else:
# TODO: implement 'path' option properly

Check warning on line 575 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: implement 'path' option properly
content = self.add_legacy_fragment(fragment)

fragfile = os.path.join(self._fragments_dir, f"{num}.config")
Expand Down Expand Up @@ -620,8 +622,8 @@
for i in range(0, fragnum):
self.addcmd("./scripts/kconfig/merge_config.sh" +
f" -m .config {self._fragments_dir}/{i}.config")
# TODO: olddefconfig should be optional/configurable

Check warning on line 625 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: olddefconfig should be optional/configurable
# TODO: log all warnings/errors of olddefconfig to separate file

Check warning on line 626 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: log all warnings/errors of olddefconfig to separate file
self.addcmd("make olddefconfig")
self.addcmd(f"cp .config {self._af_dir}/")
self.addcmd("cd ..")
Expand All @@ -634,7 +636,7 @@
fragnum = self._parse_fragments(firmware=True)
self._merge_frags(fragnum)
if not self._dtbs_check:
# TODO: verify if CONFIG_EXTRA_FIRMWARE have any files

Check warning on line 639 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: verify if CONFIG_EXTRA_FIRMWARE have any files
# We can check that if fragments have CONFIG_EXTRA_FIRMWARE
self._fetch_firmware()
self._build_kernel()
Expand Down Expand Up @@ -890,7 +892,7 @@
'''
Upload artifacts to storage
'''
# TODO: Upload not using upload_single, but upload as multiple files

Check warning on line 895 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

TODO: Upload not using upload_single, but upload as multiple files
print("[_upload_artifacts] Uploading artifacts to storage")
node_af = {}
storage = self._get_storage()
Expand Down Expand Up @@ -978,7 +980,7 @@
api.node.update(node)
except requests.exceptions.HTTPError as err:
err_msg = json.loads(err.response.content).get("detail", [])
self.log.error(err_msg)

Check failure on line 983 in kernelci/kbuild.py

View workflow job for this annotation

GitHub Actions / Lint 

Instance of 'KBuild' has no 'log' member
return

def submit(self, retcode, dry_run=False):
Expand Down
Loading