Building on Void Linux

[phriv]

I am currently getting a patch for Void Linux up and running and haven't had any issues until I had to compile UCX.

So far I'm getting the following errors:

/media/LI/rocm_sdk_builder/src_projects/ucx/src/tools/perf/perftest.c:222: error: ignoring '#pragma omp barrier' [-Werror=unknown-pragmas] 222 | #pragma omp barrier

/media/LI/rocm_sdk_builder/src_projects/ucx/src/tools/perf/perftest.c:224: error: ignoring '#pragma omp master' [-Werror=unknown-pragmas] 224 | #pragma omp master

/media/LI/rocm_sdk_builder/src_projects/ucx/src/tools/perf/perftest.c:242: error: ignoring '#pragma omp barrier' [-Werror=unknown-pragmas] 242 | #pragma omp barrier

Searching through stackoverflow suggests something about pass '-fopenmp' to gcc but I'll admit I don't know automake/autoconf well enough to figure out which args I need to edit in order to do so.

[lamikr]

Hmm, newer seen that error myself. One thing to try quickly is to add the "-fopenmp" to CFLAGS and LDFLAGS.
File binfo/envsetup.sh has some definitions for those in the end of the file. Could you try to add them there?

Another possibility is that the openmp libraries are not installed in your Linux distro? Do you have for example
ibgomp or ibgomp-dev in the Void Linux?

[jeroen-mostert]

This is an automake usage fail; autoconf detects that OpenMP is not installed but the project fails to do anything with it. Remove your ucx builddir, issue xbps-install libgomp-devel and try again; in my VM this is sufficient. I'm working on getting a complete list of deps for a clean Void install so we can add this to the deps.

Building this revealed another issue that breaks the build on my machine; issued openucx/ucx#10041 to fix that.

[jeroen-mostert]

Well, it's more than just deps, in fact. Void makes the bold choice of symlinking /bin/sh to dash. In theory, this is fine. In practice, there are a lot of nooks and crannies where people assume that bash is everywhere, even in the context of sh. I'll use this comment specifically to track all pull requests I've upstreamed for these fixes (I suggest renaming this one to track overall support for Void instead). Note that some may take quite a long time to get to us as they need to pass through two upstreams (when AMD forks things), so we may need to add local patches.

Of course a simpler alternative is to tell everyone with a distro that doesn't symlink /bin/sh to bash to please do that instead, but I'm guessing that's an even bigger uphill battle. 🙃

• ucx: Fix failing test for MAD that breaks build openucx/ucx#10041
• ucc: Replace bashisms with POSIX syntax openucx/ucc#1006
• openmpi: Use POSIX compatible redirect open-mpi/oac#16
• amd-fftw: Change here-string bashism to POSIX echo amd/amd-fftw#14

[lamikr]

I may have couple of bashism also, is the "declare -A DICTIONARY_PATCHED_PROJECTS" thing from babs.sh working ok with the dash?

[lamikr]

While you are working with the void linux dependencies, here is one problem to think.
I have thinked that the install_deps.sh would only handle the dependencies for the core apps that are build automatically and are listed in the binfo/binfo_list.sh and binfo/core

But as there are now the binfo/extra apps folder for addons, it would be nice to have some dependency handling support also for them. (I consider those apps like a one with little less support and not fully quarantee that they will support on everyblase).

For example binfo/extra/corectrl has quite a many dependencies and so-far I have only but the dependencies I found on mageia and fedora to comments of that binfo file and expect that the user will handle the install of those manually before building the app.

In addition for example on Mageia 9, I needed to backport one app by myself to newer version to be able to build it. Not sure should I print some advice for that or just let the builder to figure it out,

[jeroen-mostert]

I may have couple of bashism also, is the "declare -A DICTIONARY_PATCHED_PROJECTS" thing from babs.sh working ok with the dash?

Your own bashisms are always fine, because... you're using bash. :P If the script uses #!/bin/bash at the top then bash is required. But (AFAIK) automake/autoconf can't be forced to use bash, and even if they could, it would be a bad idea since the whole idea is for them to generate portable files.

Note that bash is installed on Void Linux and is even the default shell, it just chooses not to use it for sh. To, uh, keep people honest, I guess.

[jeroen-mostert]

For the extra deps, there are basically two choices:

• Extend the binfo format to support a dictionary of dependencies for known distros, so they can be automatically installed when building. If the distro is not found/unknown, print a message to that effect to encourage people to figure out the deps and submit a PR. This seems like quite a bit of work, especially in terms of shell scripting, but does give the best experience. We can stuff the dependencies from install-deps.sh in the first core binfo, since it's not worth it to carefully track which project needs which dependency, nor do we want to invoke the package manager every time on every build.
• Don't do any of that, throw up our hands and say "hey, if you want to build this stuff, you need to figure out yourself how to do it, maybe read the comments". Easy, but it also kind of defeats the purpose of including these projects in the repo in the first place if we don't plan on supporting them. The user may as well just clone and build them manually altogether in that case. :P

Note that for rolling distros like Arch and Manjaro, if there is no dependency on ROCm and there is a package for it it's basically not worth building from scratch (corectrl is a good example), since they are realistically close enough to upstream that you don't need to waste time building your own. So rather than figuring out dependencies there, you can just install the distro package and be done with it.

[phriv]

Well, it's more than just deps, in fact. Void makes the bold choice of symlinking /bin/sh to dash. In theory, this is fine. In practice, there are a lot of nooks and crannies where people assume that bash is everywhere, even in the context of sh.

I was about to come back to this thread to mention this. I did follow your advice @jeroen-mostert and libgomp-devel was enough to get ucx to compile without any issues. The following dependency, UCC, will fail during the autogen step. Like you said this comes down to the fact this distro uses dash instead of bash for /bin/sh (Wish they'd put than in the user manual). I'll apply your patch and report back any further issues.

Also yeah its best to rename this thread at this point.

[jeroen-mostert]

For now the build fails on rocBLAS_Tensile/rocBLAS; the latter complains that it can't find Tensile (no module named 'Tensile'), even though the lines just above it say it installed. It seems like it's not building Tensile binaries at all, possibly some platform mismatch or upstream change. Unfortunately most of the rest depends on it (and it's one of the most time-consuming parts of the build).

For now this is the list of dependencies I've recorded (which is not complete, but may allow others to proceed on a build):

xbps-install base-devel cmake babeltrace-devel doxygen elfutils-devel expat-devel gcc-fortran \
    json-c++ gmp-devel libdrm-devel libglvnd-devel libgomp-devel libnuma-devel \
    mpfr-devel msgpack-cxx ncurses-devel openssl-devel protobuf protobuf-devel \
    python3-pybind11 sqlite-devel xxd zlib-devel

Note that although ccache is not formally a dependency of anything, anyone resembling a contributor is encouraged to install it anyway as it really helps speed up those endless rebuilds of llvm, among other things.

[phriv]

I see,

Here's the list of packages I installed. I took arch linux's list (in hindsight, not the best idea) and would replace dependencies that had different/wrong names. Whether or not this is an extensive list of packages remains to be seen.

sudo xbps-install -S libgcc make pkgconf numactl cmake doxygen libelf perl-rename perl-URI perl-File-BaseDir perl-File-Copy-Recursive perl-File-Listing wget gcc gcc-fortran fakeroot libgomp pciutils libdrm vim glew autoconf automake libtool bzip2 xz icu perl libmpack python-pip openssl python3-openssl libffi json-c++ texinfo extra-cmake-modules sqlite git git-lfs valgrind flex byacc gettext ninja texlive-basic ocl-icd protobuf python3-pybind11 libaio gmp mpfr libpng libjpeg-turbo msgpack msgpack-cxx sox ncurses expat babeltrace python3-pip libnuma-devel libgomp-devel

As both Ubuntu and arch call for this dependency in some way, I also added this line for Void (void has no official package).
pip3 install --break-system-packages --user CppHeaderParser

[jeroen-mostert]

There's a lot of fluff in the list; for example all -devel packages for Void also install their base libs as dependencies, so adding those separately is not necessary (by contrast the opposite is true for Arch, all the non dev packages install the dev stuff). Of course there's no real competition going on for getting the smallest dependency lists (unless you're angling for a cloud build, I guess), so whatever works. :P

[jeroen-mostert]

Also the rocBLAS build continues after installing sqlite3 and rebuilding Python (or possibly as a result of things other than sqlite3), showing the importance of having all the deps in place at least before Python gets built, as otherwise it will silently turn off core modules. Conceivably it could probably also be configured to fail the build if certain necessary libs are missing.

Note that some deps we still have left in install-deps, like CppHeaderParser, seem to be leftovers from when we didn't build and use our own Python (which pulls in this package, among others), because aside from pybind11 I haven't found any of them necessary. So far, at least, knock on wood. We shouldn't be installing them if our own packaged Python is doing the work, as it may mask problems where we incorrectly use the system Python. Also, users cannot expect the packages we build to work with their own native Python without sourcing the ROCm environment except by chance, since the versions can mismatch.

There's still something weird going on where rocBLAS builds Tensile (and succeeds) but then hipBLASLt and hipSPARSELt also try to build it, and fail, giving a very strange error involving gcc headers ("non-virtual member function marked 'override' hides virtual member function"). I dread having to do a full rebuild, maybe @phriv can catch up before I have to retry. If not, I'll get there when I get there, it's been a long day's work. :P

[lamikr]

Hipblaslt and hipsparselt use different version of Tensile (something called tensilelite) and these two projects are more for MI-level of cards at the moment.