Update of most of the first part of the tutorial.

Author: klust

docs/2025-LUST/1_Intro/1_02_Lmod.md

@@ -1,5 +1,7 @@
 # LUMI software stacks (technical)
 
+[easybuild]: https://easybuild.io/
+
 *[[back: The Cray Programming Environment]](1_03_CPE.md)*
 
 ---
@@ -17,11 +19,11 @@ LUMI has different node types providing compute resources:
 -   LUMI has 16 login nodes, though many of those are reserved for special purposes and not
     available to all users. These login nodes have a zen2 CPU. These nodes have a SlingShot 10
     interconnect.
--   There are 1536 regular CPU compute nodes in a partition denoted as LUMI-C. These
+-   There are 2048 regular CPU compute nodes in a partition denoted as LUMI-C. These
     compute nodes have a zen3 CPU and run a reduced version of SUSE Linux optimised
     by Cray to reduce OS jitter. These nodes will in the future be equipped with a 
     SlingShot 11 interconnect card.
--   There are 2560 GPU compute nodes in a partition denoted as LUMI-G. These nodes have
+-   There are 2978 GPU compute nodes in a partition denoted as LUMI-G. These nodes have
     a single zen3-based CPU with optimised I/O die linked to 4 AMD MI250X GPUs. Each node
     has 4 SlingShot 11 interconnect cards, one attached to each GPU.
 -   The interactive data analytics and visualisation partition is really two different partitions 
@@ -32,9 +34,6 @@ LUMI has different node types providing compute resources:
         and are equipped with SlingShot 10 interconnect cards (2 each???)
     -   8 nodes have zen2 CPUs and 8 NVIDIA A40 GPUs each, and have 2 SlingShot 10
         interconnect cards each.
--   The early access platform (EAP) has 14 nodes equipped with a single 64-core
-    zen2 CPU and 4 AMD MI100 GPUS. Each node has a single SlingShot 10 interconnect
-    and also local SSDs.
 
 SlingShot 10 and SlingShot 11 are different software-wise. SlingShot 10 uses a
 Mellanox CX5 NIC that support both OFI and UCX, and hence can also use the
@@ -49,7 +48,7 @@ libfabric and Cray MPICH.
 
 ## CrayEnv and LUMI modules
 
-On LUMI, two types of software stacks are currently offered:
+On LUMI, three types of software stacks are currently offered:
 
   - ``CrayEnv`` (module name) offers the Cray PE and enables one to use
     it completely in the way intended by HPE-Cray. The environment also offers a
@@ -89,14 +88,21 @@ On LUMI, two types of software stacks are currently offered:
     ``LUMI`` module, users can always load a different version of the ``partition``
     module.
 
+  - `spack` offers some setups to install packages with Spack
+
 Note that the ``partition`` modules are only used by the ``LUMI`` module. In the
 ``CrayEnv`` environment, users should overwrite the configuration by loading their
 set of target modules after loading the ``CrayEnv`` module.
 
+There are also local organisations that provide additional pre-installed software,
+often accessible by loading one of the `Local-*` modules. Currently there is software
+offered by CSC with module names that resemble those on their national systems puhti
+and mahti, and some software to access the quantum computers at VTT. Not all of that
+software is available to all LUMI users. There might be license and other restrictions.
 
 ---
 
-## The ``partition`` module
+## The `partition` module
 
 The ``LUMI`` module currently supports five partition modules, but that number may
 be reduced in the future:
@@ -107,22 +113,9 @@ be reduced in the future:
 | ``partition/C``   | ``craype-x86-milan``  | ``craype-accel-host``       |
 | ``partition/G``   | ``craype-x86-trento`` | ``craype-accel-amd-gfx90a`` |
 | ``partition/D``   | ``craype-x86-rome``   | ``craype-accel-nvidia80``   |
-| ``partition/EAP`` | ``craype-x86-rome``   | ``craype-accel-amd-gfx908`` |
 
 All ``partition`` modules also load `craype-network-ofi``.
 
-``pattition/D`` may be dropped in the future as it seems we have no working CUDA setup
-and can only use the GPU nodes in the LUMI-D partition for visualisation and not with CUDA.
-
-Furthermore if it would turn out that there is no advantage in optimizing for Milan
-specifically, or that there are no problems at all in running Milan binaries on Rome 
-generation CPUs, ``partition/L`` and ``partition/C`` might also be united in a single 
-partition.
-
-
-
-
-
 
 
 ---

docs/2025-LUST/1_Intro/1_03_CPE.md

@@ -15,7 +15,8 @@ It is important to be familiar with these terms, so we'll briefly cover them one
 ## Toolchains
 
 A *compiler toolchain* (or just *toolchain* for short) is a **set of [compilers](https://en.wikipedia.org/wiki/Compiler)**,
-which are used to build software from source, together with a set of **additional libraries** that provide further core functionality.
+which are used to build software from source, together with a set of **additional libraries** 
+with pretty standard APIs that provide further core functionality.
 
 We refer to the different parts of a toolchain as **toolchain components**.
 
@@ -52,7 +53,7 @@ The toolchain concept also maps nicely on the Programming Environment concept in
 Programming Environment. Each ``PrgEnv-*`` module in fact provides a full toolchain in a
 typical Cray PE installation, except for the FFTW library. Note that systems could configure 
 the contents of a ``PrgEnv-*`` module differently as the MPI library and scientific library are 
-optional. In the Cray PE, the MPI library is provided by the ``cray-mpic`` module, 
+optional. In the Cray PE, the MPI library is provided by the ``cray-mpich`` module, 
 the BLAS, LAPACK and ScaLAPACK libraries by the ``cray-libsci`` module (for CPU-only nodes)
 and the FFTW library through the ``cray-fftw`` module.
 
@@ -66,11 +67,11 @@ organised in a hierarchy.
 The **`system` toolchain** is a special case which corresponds to using the compilers and libraries
 *provided by the operating system*, rather than using toolchain components that were installed using EasyBuild.
 
-It used sparingly, mostly to install software where no actual compilation is done or
+It is used sparingly, mostly to install software where no actual compilation is done or
 to build a set of toolchain compilers and its dependencies, since the versions of the system compilers
 and libraries are beyond the control of EasyBuild, which could affect the reproducibility of the installation.
 
-On LUMI however it is used a bit more and it takes some of the functions of the ``GCCcore`` toolchain in
+On LUMI however it is used a bit more and it takes some of the functions of the `GCCcore` toolchain in
 other EasyBuild toolchain hierarchies.
 
 
@@ -82,7 +83,7 @@ because they are widely adopted by the EasyBuild community.
 The `foss` toolchain consists of all open source components (hence the name:
 "FOSS" stands for Free & Open Source Software): [GCC](https://gcc.gnu.org/), [Open MPI](https://www.open-mpi.org/), [OpenBLAS](https://www.openblas.net/),
 [ScaLAPACK](https://www.netlib.org/scalapack/) and [FFTW](http://fftw.org/).
-In recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas)
+In more recent versions (since 2021a), [FlexiBLAS](https://www.mpi-magdeburg.mpg.de/projects/flexiblas)
 is used as the BLAS library, with OpenBLAS and LAPACK as the backend.
 
 The `intel` toolchain consists of the 
@@ -102,11 +103,13 @@ A[SYSTEM] --> B[GCCcore];
 B --> C[GCC];
 B --> D[iccifort];
 C --> E[gompi: Adds OpenMPI];
-D --> F[iimpi: Adds Intel MPI];
-D --> G[imkl: Adds Intel MKL];
-E --> H[foss: Adds OpenBLAS, LAPACK, ScaLAPACK, FFTW];
-F --> I[intel];
-G --> I;
+C --> F[gfbf: Adds FlexiBLAS, FFTW];
+D --> G[iimpi: Adds Intel MPI];
+D --> H[imkl: Adds Intel MKL];
+E --> I[foss: Adds ScaLAPACK];
+F --> I;
+G --> J[intel];
+H --> J;
 ```
 
 More information on these toolchains is available [in the EasyBuild documentation](https://docs.easybuild.io/en/latest/Common-toolchains.html).
@@ -116,7 +119,7 @@ More information on these toolchains is available [in the EasyBuild documentatio
 
 ## EasyBuild framework
 
-EasyBuild is written in Python and organised in three layers, represented by the three blocks in the logo:
+EasyBuild is written in Python and organised in three layers, represented by the three lines in the logo:
 the EasyBuild framework, easyblocks and Easyconfig files.
 
 The EasyBuild *framework* consists of a set of Python modules organised in packages (``easybuild.framework``,
@@ -169,20 +172,19 @@ EasyBuild.
 
 Some easyconfig parameters are **mandatory**. The following parameters *must* be defined in *every* easyconfig file:
 
-* ``name`` and ``version``, which specify the name and version of the software to install;
-* ``homepage`` and ``description``, which provide key metadata for the software;
-* ``toolchain``, which specifies the compiler toolchain to use to install the software (see
-  ``toolchains`` tab);
+* `name` and `version`, which specify the name and version of the software to install;
+* `homepage` and `description`, which provide key metadata for the software;
+* `toolchain`, which specifies the compiler toolchain to use to install the software;
 
 Other easyconfig parameters are **optional**: they can be used to provide required information,
 or to control specific aspects of the installation procedure performed by the easyblock.
 
 Some commonly used optional easyconfig parameters include:
 
-* ``easyblock``, which specifies which (generic) easyblock should be used for the installation;
-* ``sources`` and ``source_urls``, which specify the list of source files and where to download them;
-* ``dependencies`` and ``builddependencies``, which specify the list of (build) dependencies;
-* ``configopts``, ``buildopts``, and ``installopts``, which specify options for the configuration/build/install commands, respectively;
+* `easyblock`, which specifies which (generic) easyblock should be used for the installation;
+* `sources` and `source_urls`, which specify the list of source files and where to download them;
+* `dependencies` and `builddependencies`, which specify the list of (build) dependencies;
+* `configopts`, `buildopts`, and `installopts`, which specify options for the configuration/build/install commands, respectively;
 
 If no value is specified for an optional easyconfig parameter, the corresponding default value will be used.
 
@@ -278,6 +280,9 @@ There are three main types of dependencies for computer software:
 
 EasyBuild currently doesn't treat link-time dependencies differently from run-time dependencies
 and hence has no specific easyconfig parameter for them.
+(Even EasyBuild 5 doesn't even though that version defaults to "rpath"-linking so that no
+`LD_LIBRARY_PATH` is needed and hence modules that do not define other variables that are
+needed to run a package don't need to be loaded at runtime.)
 
 ---
 

docs/2025-LUST/1_Intro/1_04_LUMI_software_stack.md

@@ -26,7 +26,7 @@ for the remainder of this tutorial.
 
 !!! summary
 
-    * *Requirements*: Linux, Python 2.7 or 3.5+, environment modules tool (Lmod is recommended).
+    * *Requirements*: Linux, Python 3.6+, environment modules tool (Lmod is recommended).
     * *Installation methods*:
         * <a href=#method-1-using-pip>``pip install easybuild``</a>
         * <a href="#method-2-bootstrapping-easybuild">installing EasyBuild with EasyBuild</a>
@@ -50,15 +50,22 @@ like development or contributing back to the project.
 
 ### Python
 
-EasyBuild is implemented in Python, and is **compatible with both Python 2.7 and Python 3.5+**
-(that is, Python 3.5 or a newer version of Python 3).
+EasyBuild is implemented in Python, and is **requires Python 3.6+**
+(that is, Python 3.6 or a newer version of Python 3), though Python 3.9 or more 
+recent is highly recommended.
 
 To check which Python version you have, use:
 
 ```shell
 python -V
 ```
 
+or, if that command is not available,
+
+```shell
+python3 -V
+```
+
 *No additional Python packages are required by EasyBuild*, the ones that come with the standard
 Python distribution are sufficient. Some additional Python packages *can* be leveraged for specific features.
 More on that later.
@@ -77,7 +84,7 @@ To check if you have a modules tool installed, use:
 module --version
 ```
 
-If this produces output that starts with something like "`Modules based on Lua: Version 8.3.1`" you have Lmod installed,
+If this produces output that starts with something like "`Modules based on Lua: Version 8.7.32`" you have Lmod installed,
 which is the default modules tool used by EasyBuild, and you are all set for installing and using EasyBuild.
 Any sufficiently recent Lmod version (8.x or even 7.x) should be fine. 
 
@@ -99,9 +106,9 @@ we use Lmod as the default module system for users, and EasyBuild is configured
 ## EasyBuild as a Python package
 
 EasyBuild consists of a number of interdependent Python packages,
-and is available via both GitHub at <a href="https://github.com/easybuilders">https://github.com/easybuilders</a>,
+and is available via both GitHub at <a href="github.com/easybuilders">https://github.com/easybuilders</a>,
 as well as via the standard Python Package Index (PyPI) at
-<a href="https://pypi.org/project/easybuild/">https://pypi.org/project/easybuild</a>.
+<a href="pypi.org/project/easybuild/">https://pypi.org/project/easybuild</a>.
 
 As you may be aware the Python packaging ecosystem is bit convoluted,
 which is reflected in the many different ways in which you can install a Python package.
@@ -119,17 +126,20 @@ via the standard *Python Package Index* (PyPI), through one of the standard Pyth
 And since EasyBuild is a software installation tool in its own right, we actually have a couple
 of additional tricks up our sleeve!
 
-### Python 2 or Python 3?
+### Which version of Python?
 
-For EasyBuild it does not matter much whether you install it on top of Python 2 or Python 3. 
-Since version 4.5, Python 3 does offer some optional features (requiring additional packages)
-that are not available with Python 2.
-Since <a href="https://www.python.org/doc/sunset-python-2/">Python 2 is end-of-life</a>,
-we strongly recommend using Python 3 if you have the choice.
+Recent versions of EasyBuild no longer support Python 2.7 and require version 3.6 or
+newer. Some functionality only works on Python 3.9 or more recent.
 
 By default EasyBuild will use the `python` command to run,
 but you <a href="#eb_python">can control this if needed via ``$EB_PYTHON``</a>.
 
+!!! Note "On LUMI"
+    On LUMI, we modify the scripts that run EasyBuild and hard-code the 
+    Python executable and Python path in the script. This avoids accidentally using
+    the wrong Python executable or interference with other Python software that may
+    be loaded.
+
 
 ## Installing EasyBuild
 
@@ -307,14 +317,6 @@ you can consider installing EasyBuild with EasyBuild. This can be done in 3 step
 * Step 2: Using EasyBuild to install EasyBuild as a module
 * Step 3: Loading the EasyBuild module
 
-!!! note
-    A [bootstrap script](https://docs.easybuild.io/en/latest/Installation.html#bootstrapping-easybuild)
-    is available that automates this procedure,
-    but is known to be problematic in some contexts, and is not being actively
-    maintained anymore.
-
-    As a result, we do not recommend using the bootstrap script anymore.
-
 
 #### Step 1: Installing EasyBuild into a temporary location
 
@@ -412,7 +414,7 @@ export EB_PYTHON=python3
 ### Approach on LUMI
 
 *Documentation on the inner workings of the LUMI software stack can be found in the 
-[LUMI-SoftwareStack GitHub, docs subdirectory](https://github.com/Lumi-supercomputer/LUMI-SoftwareStack/tree/main/docs).*
+[LUMI-SoftwareStack texts on the lumi-supercomputer.github.io](https://lumi-supercomputer.github.io/LUMI-SoftwareStack/).*
 
 To keep the different versions of the LUMI software stack as independent from one another as possible,
 EasyBuild is bootstrapped for each software stack by the ``prepare_LUMI_stack.sh`` script. We use the 
@@ -425,7 +427,7 @@ are each untarred and then installed into their location by running
 We do so only for the framework and easyblocks files as the easconfig files are not used to
 install EasyBuild. Instead we create our own EasyConfig file for EasyBuild which contains
 some additional packages that enable extra features in EasyBuild and also provide more information
-to Lmod. Next the configuration module for EasyBuild (see the next section, 
+to Lmod. Next we load the LUMI-specific configuration module for EasyBuild (see the next section, 
 ["Configuring EasyBuild"](../1_07_configuration), for more information) and use the temporary 
 installation of EasyBuild with our own EasyConfig file to do a proper installation of EasyBuild
 with module in the final location.

docs/2025-LUST/1_Intro/1_05_terminology.md

@@ -28,8 +28,10 @@ and to configure EasyBuild according to your preferences and the system on which
 
     There are two more admin-only configuration modules for EasyBuild. The main one is
     ``EasyBuild-production`` which is used for software installations in the central software stack.
-    The other one is ``EasyBuild-infrastructure`` which is used to install infrastructure modules
-    that need to be installed in a pure Lmod hierarchy, e.g., the modules that load the toolchains.
+    The other one is ``EasyBuild-infrastructure`` which can be used to install infrastructure modules
+    that need to be installed in a pure Lmod hierarchy, e.g., the modules that do the EasyBuild
+    configuration (though these are currently not installed through EasyBuild) and the compiler
+    toolchain modules.
 
 
 ---
@@ -42,7 +44,7 @@ in a large amount of available configuration settings.
 
 The full list of configuration settings can be consulted via `eb --help`,
 which shows the corresponding command line option accompanied by a short description.
-At the time of writing, *over 240 different configuration settings* are supported by EasyBuild.
+At the time of writing, *over 270 different configuration settings* are supported by EasyBuild.
 
 For the sake of this tutorial we will focus on a specific subset of configuration settings,
 and cover only the most prominent and important ones.
@@ -105,7 +107,7 @@ and the default `software` and `modules/all` names for the subdirectories are us
     This makes it slightly easier to organise the module tree with user-friendly labeling, but above
     all also makes the synchronisation process of the 4 instances of the software directory more robust
     as it is now easy to synchronise all modules in the last step, which is a much quicker process than
-    syncrhonising the software installations.
+    synchronising the software installations.
 
     We also use short paths for software installations (to avoid overrunning the maximum length of a
     shebang line in scripts) while we use longer, more descriptive names for subdirectories in the 
@@ -139,8 +141,9 @@ or specific mount options like `noexec`).
 
     The configuration modules on LUMI will use a RAM disk for the build path. On the login nodes,
     ``$XDG_RUNTIME_DIR`` is used as that space is automatically cleared when the last session of a user
-    ends. However, on the compute nodes a job- or user-specific subdirectory of ``/dev/shm`` is currently used
-    as ``$XDG_RUNTIME_DIR`` does not exist.
+    ends. However, on the compute nodes a job- or user-specific subdirectory of ``/tmp`` is currently used
+    as ``$XDG_RUNTIME_DIR`` does not exist. The user can also overwrite the build directory by 
+    setting `EBU_WORKDIR` before loading the EasyBuild configuration module (e.g., `EasyBuild-user`).
 
 
 ---
@@ -562,7 +565,7 @@ Leave other configuration settings set to their default value.
 Try running the following command:
 
 ```shell
-eb bzip2-1.0.6.eb
+eb bzip2-1.0.8.eb
 ```
 
 Where do you expect to find the installation?
@@ -584,23 +587,23 @@ Where do you expect to find the installation?
     $ ls $HOME/easybuild/software
     bzip2
     $ ls $HOME/easybuild/software/bzip2
-    1.0.6
-    $ ls $HOME/easybuild/software/bzip2/1.0.6
+    1.0.8
+    $ ls $HOME/easybuild/software/bzip2/1.0.8
     bin  easybuild  include  lib  man
     ```
 
     ```shell
     $ ls $HOME/easybuild/modules/all
     bzip2
     $ ls $HOME/easybuild/modules/all/bzip2
-    1.0.6.lua
+    1.0.8.lua
     ```
 
-    The source file for bzip2 1.0.6 was downloaded to `$HOME/easybuild/sources`:
+    The source file for bzip2 1.0.8 was downloaded to `$HOME/easybuild/sources`:
 
     ```shell
     $ ls $HOME/easybuild/sources/b/bzip2
-    bzip2-1.0.6.tar.gz
+    bzip2-1.0.8.tar.gz
     ```
 
     We will discuss this in more detail in the next part of the tutorial.