Cross building from x64 to arm64 fails compiling using clang

[AnInternetTroll]

Describe the Bug

When cross building from x64 to arm64 the process fails as clang "is not able to compile a simple test program."

Steps to Reproduce

In the dotnet/dotnet repo on version v8.0.7 the following command should cause the issue

ROOTFS_DIR=/usr/aarch64-linux-gnu bash ./build.sh --clean-while-building --online --source-repository https://github.com/dotnet/dotnet --source-version 8be139ddde52d33e24c7d82f813248ff9fc54b97 -- -p:TargetArchitecture=arm64 -p:CrossBuild=true

The full void template is available on github (very much WIP)

Other Information

The relevant output log of the build process can be found here
(I have omitted the void-related stuff, such as installing dependencies, as well as the prep.sh output. If it's relevant I can paste it as well)

[ghost]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[ghost]

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

[omajid]

Some relevant bits from the log:

...
        The C compiler

          "/usr/bin/clang-17"

        is not able to compile a simple test program.

        It fails with the following output:
...
          /usr/bin/clang-17   -fstack-clash-protection -D_FORTIFY_SOURCE=2 -O2 -pipe -march=armv8-a  -g -I/usr/aarch64-linux-gnu/usr/include -ffile-prefix-map=/builddir/dotnet-sdk-8.0.7=.  -MD -MT CMakeFiles/cmTC_9081e.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_9081e.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_9081e.dir/testCCompiler.c.o -c /builddir/dotnet-sdk-8.0.7/src/runtime/artifacts/source-build/self/src/artifacts/obj/coreclr/linux.x64.Release/CMakeFiles/CMakeScratch/TryCompile-NWEw92/testCCompiler.c
          error: unknown target CPU 'armv8-a'
          note: valid target CPU values are: nocona, core2, penryn, bonnell, atom, silvermont, slm, goldmont, goldmont-plus, tremont, nehalem, corei7, westmere, sandybridge, corei7-avx, ivybridge, core-avx-i, haswell, core-avx2, broadwell, skylake, skylake-avx512, skx, cascadelake, cooperlake, cannonlake, icelake-client, rocketlake, icelake-server, tigerlake, sapphirerapids, alderlake, raptorlake, meteorlake, sierraforest, grandridge, graniterapids, graniterapids-d, emeraldrapids, knl, knm, k8, athlon64, athlon-fx, opteron, k8-sse3, athlon64-sse3, opteron-sse3, amdfam10, barcelona, btver1, btver2, bdver1, bdver2, bdver3, bdver4, znver1, znver2, znver3, znver4, x86-64, x86-64-v2, x86-64-v3, x86-64-v4

Could it be that the compiler being used can't target arm64 and the compiler that can target arm64 isn't being picked up?

[AnInternetTroll]

Yeah, that seems to be the case. It looks as though void's clang can't cross compile, and they use cross-aarch64-linux-gnu instead as a compiler. If I install clang then dotnet's build script tries to use it and fails. If I don't install it then dotnet's build script will complain it can't find clang. In both cases the $CC variable is set to cross-aarch64-linux-gnu

I don't know how dependent dotnet is on clang, but would it be possible to use something else? Ideally though an env var or flag?

[omajid]

I haven't done any cross-builds for arm64 myself, but when I compare with ppc64le cross builds (which I do build):

I set some env vars:

• CROSSCOMPILE=1
• ROOTFS_DIR=/
• PKG_CONFIG_PATH=/usr/lib/powerpc64le-linux-gnu/pkgconfig

My build command is then literally:

./build.sh --source-only --online --use-mono-runtime /p:TargetArchitecture=ppc64le /p:PortableBuild=true

[AnInternetTroll]

I can't get this to work. This kinda got it closer though, but unknown target CPU 'armv8-a'

Here is a full log with the output of env as well if it helps: dotnet-sdk-aarch64-2.txt

As well as the template used: template

[MichaelSimons]

[Triage] This appears as a discussion with no actionable changes necessary within source-build. Converting to a discussion. Please open follow-up issues if any source changes are needed.

[sec]

Using v10.0.0-preview.5.25277.114 the issue is still present when building from dotnet-dotnet. Trying to make cross build fails with:

    Executing /home/linux/dotnet/src/runtime/src/coreclr/"build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -arm64 -release -ci -keepnativesymbols -os freebsd -hostarch x64 -hostos linux -outputrid freebsd.arm64 -cmakeargs "-DCLR_DOTNET_RID=freebsd-arm64" -subdir x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/home/linux/dotnet/.dotnet/dotnet" -component alljits
    Commencing CoreCLR Repo build
    /home/linux/dotnet/src/runtime/eng/common/native/init-distro-rid.sh: line 40: /bin/freebsd-version: No such file or directory
    __OutputRid: freebsd.arm64
    Setting up directories for build
    Checking prerequisites...
    Commencing build of "  alljits " target in "CoreCLR component" for freebsd.arm64.Release in /home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64
    Invoking "/home/linux/dotnet/src/runtime/eng/native/gen-buildsys.sh" "/home/linux/dotnet/src/runtime/src/coreclr" "/home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64" x64 linux clang Release ""  -DCLR_CMAKE_TARGET_ARCH=arm64 -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="freebsd.arm64" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCLR_DOTNET_HOST_PATH=/home/linux/dotnet/.dotnet/dotnet -DCLR_DOTNET_RID=freebsd-arm64 -DCLR_CROSS_COMPONENTS_BUILD=1  -DCLR_CMAKE_KEEP_NATIVE_SYMBOLS=true
    Not searching for unused variables given on the command line.
    loading initial cache file /home/linux/dotnet/src/runtime/eng/native/tryrun.cmake
    -- The C compiler identification is Clang 18.1.3
    -- The CXX compiler identification is Clang 18.1.3
    -- Detecting C compiler ABI info
    -- Detecting C compiler ABI info - failed
    -- Check for working C compiler: /usr/bin/clang-18
    -- Check for working C compiler: /usr/bin/clang-18 - broken
    CMake Error at /usr/share/cmake-3.28/Modules/CMakeTestCCompiler.cmake:67 (message):
      The C compiler

        "/usr/bin/clang-18"

      is not able to compile a simple test program.

      It fails with the following output:

        Change Dir: '/home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64/CMakeFiles/CMakeScratch/TryCompile-pAaJdp'

        Run Build Command(s): /usr/bin/cmake -E env VERBOSE=1 /usr/bin/gmake -f Makefile cmTC_8faf3/fast
        /usr/bin/gmake  -f CMakeFiles/cmTC_8faf3.dir/build.make CMakeFiles/cmTC_8faf3.dir/build
        gmake[1]: Entering directory '/home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64/CMakeFiles/CMakeScratch/TryCompile-pAaJdp'
        Building C object CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o
        /usr/bin/clang-18 --target=x86_64-unknown-freebsd12 --sysroot=/home/linux/crossfs_arm64_13    -MD -MT CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o -MF CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o.d -o CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o -c /home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64/CMakeFiles/CMakeScratch/TryCompile-pAaJdp/testCCompiler.c
        Linking C executable cmTC_8faf3
        /usr/bin/cmake -E cmake_link_script CMakeFiles/cmTC_8faf3.dir/link.txt --verbose=1
        /usr/bin/clang-18 --target=x86_64-unknown-freebsd12 --sysroot=/home/linux/crossfs_arm64_13  -fuse-ld=lld -fuse-ld=lld  CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o -o cmTC_8faf3
        ld.lld: error: CMakeFiles/cmTC_8faf3.dir/testCCompiler.c.o is incompatible with /home/linux/crossfs_arm64_13/usr/lib/crt1.o
        clang-18: error: linker command failed with exit code 1 (use -v to see invocation)
        gmake[1]: *** [CMakeFiles/cmTC_8faf3.dir/build.make:100: cmTC_8faf3] Error 1
        gmake[1]: Leaving directory '/home/linux/dotnet/src/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64/CMakeFiles/CMakeScratch/TryCompile-pAaJdp'
        gmake: *** [Makefile:127: cmTC_8faf3/fast] Error 2

while doing crossbuild directly from runtime repo is 100% done without errors, the part that fails looks like that on runtime:

  Repo successfully built.^M                                                                                                                                                                                       Product binaries are available at /home/linux/runtime/artifacts/bin/coreclr/freebsd.arm64.Release^M                                                                                                              Executing /home/linux/runtime/src/coreclr/"build-runtime.sh" -cmakeargs "-DCLR_CROSS_COMPONENTS_BUILD=1" -arm64 -release -ci -os freebsd -hostarch x64 -hostos linux -outputrid freebsd-arm64 -cmakeargs "-DCLR_DOTNET_RID=freebsd-arm64" -subdir x64 -cmakeargs "-DCLR_DOTNET_HOST_PATH=/home/linux/runtime/.dotnet/dotnet" -component alljits^M
  Commencing CoreCLR Repo build^M
  /home/linux/runtime/eng/common/native/init-distro-rid.sh: line 40: /bin/freebsd-version: No such file or directory^M
  __OutputRid: freebsd-arm64^M
  Setting up directories for build^M
  Checking prerequisites...^M
  Commencing build of "  alljits " target in "CoreCLR component" for freebsd.arm64.Release in /home/linux/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64^M
  Invoking "/home/linux/runtime/eng/native/gen-buildsys.sh" "/home/linux/runtime/src/coreclr" "/home/linux/runtime/artifacts/obj/coreclr/freebsd.arm64.Release/x64" x64 linux clang Release ""  -DCLR_CMAKE_TARGET_ARCH=arm64 -DCLR_CMAKE_PGO_INSTRUMENT=0 -DCLR_CMAKE_OPTDATA_PATH= -DCLR_CMAKE_PGO_OPTIMIZE=0 -DCLI_CMAKE_FALLBACK_OS="freebsd" -DFEATURE_DISTRO_AGNOSTIC_SSL=1 -DCLR_DOTNET_HOST_PATH=/home/linux/runtime/.dotnet/dotnet -DCLR_DOTNET_RID=freebsd-arm64 -DCLR_CROSS_COMPONENTS_BUILD=1 ^M
  Not searching for unused variables given on the command line.^M
  -- The C compiler identification is Clang 18.1.3^M
  -- The CXX compiler identification is Clang 18.1.3^M
  -- Detecting C compiler ABI info^M
  -- Detecting C compiler ABI info - done^M
  -- Check for working C compiler: /usr/bin/clang-18 - skipped^M
  -- Detecting C compile features^M
  -- Detecting C compile features - done^M
  -- Detecting CXX compiler ABI info^M
  -- Detecting CXX compiler ABI info - done^M
  -- Check for working CXX compiler: /usr/bin/clang++-18 - skipped^M
  -- Detecting CXX compile features^M
  -- Detecting CXX compile features - done^M
  -- The linker identification is Ubuntu LLD 18.1.3 (compatible with GNU linkers)^M
  Detected Linux x86_64^M

Looks like one of the issue comes from https://github.com/dotnet/runtime/blob/e0620bc380fa291867568e653206a9ecb8923d64/eng/native/gen-buildsys.sh#L73 - this line assume that target arch is always host arch - which is not true when doing x64 => arm64 build.