macOS-14-arm64 20250304.1018 much slower than 20250120.774

[odisei369]

Description

XCode tests on macoS-14 image are now taking much more time.

It starts happening after GitHub updates MacOS image version on runners.

The main step of the job:

- name: Run Tests
        run: |
          set -o pipefail && xcodebuild test \
            -project "Project.xcodeproj" \
            -scheme "UnitTests" \
            -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.1' \
            -skipPackagePluginValidation \
            -skipMacroValidation \
            CODE_SIGN_IDENTITY="" \
            CODE_SIGNING_REQUIRED=NO \
            -derivedDataPath .build \
            | xcbeautify --renderer github-actions

Platforms affected

• Azure DevOps
• GitHub Actions - Standard Runners
• GitHub Actions - Larger Runners

Runner images affected

• Ubuntu 20.04
• Ubuntu 22.04
• Ubuntu 24.04
• macOS 13
• macOS 13 Arm64
• macOS 14
• macOS 14 Arm64
• macOS 15
• macOS 15 Arm64
• Windows Server 2019
• Windows Server 2022
• Windows Server 2025

Image version and build link

Image version where builds are fast: 20250120.774
Image version where builds are much slower: 20250304.1018

Is it regression?

Yes, 20250120.774

Expected behavior

The run times for the same job on 20250304.1018 should be close to run time on 20250120.774.

Here is an example of how much time it took on 20250120.774:

Snapshot tests job:

Actual behavior

Here is an example of how much time it took on 20250304.1018 (same code, just reruning CI build):

Snapshot tests job:

You could see that test part takes 2.5 more time comparing to older runner image.

We've set timeout to 40 minutes and you could see that it failed with timeout with some of the jobs.

Repro steps

Because we're using GitHub runners, I could not force runner to use older image version, so I don't see a clear way to reproduce it this way.

Maybe when using self-hosted runner there is a way to reproduce it by:

1. Running some XCode tests on CI with older image.
2. Running some XCode tests on CI with new image.

Maybe I could help by providing some test project with tests so we could run it to do benchmarking and compare how much time it takes for different images.

Let me know how can I help with tackling this issue.

[sureshe456]

Hi @odisei369 We will look in to the issue and keep you posted on any updates. Thanks.

[aartis17]

Hi @odisei369 ,
I cannot reproduce this issue on the old runner images. Therefore, I attempted running the build on the current version of macOS 14 ARM64 (20250317.1117) with IOS 18.2 The build completed successfully within a minute. Below are the relevant snippets for your reference. Could you please try to execute your tests on 20250317.1117 image version?

Also, can you please share the workflow files and some test projects with tests, so that I can try running it on the current version of macOS 14 arm 64.

Workflow:

jobs:
  build:
    strategy:
      fail-fast: false
      matrix:
        os: [ 'macos-14']
    runs-on: ${{ matrix.os }}

    steps:
    - name: Checkout repository
      uses: actions/checkout@v2

    - name: Select Xcode
      run: sudo xcode-select -switch /Applications/Xcode_16.2.app && /usr/bin/xcodebuild -version

    - name: check Xcode Build time
      run: |
        echo "Building project manually.."
        xcodebuild -project sim.xcodeproj -scheme sim -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.2' \
        -skipPackagePluginValidation \
            -skipMacroValidation \
            CODE_SIGN_IDENTITY="" \
            CODE_SIGNING_REQUIRED=NO \
            -derivedDataPath .build \
         build | xcbeautify --renderer github-actions

Output logs:

[odisei369]

Hi @aartis17,

Thanks for checking the issue.

Also, can you please share the workflow files and some test projects with tests so that I can try running them on the current version of macOS 14 ARM 64?

This is a private project, so I cannot share its code.

To provide you with a test project, I would need to create one with enough artificial tests to make the slowdown noticeable. However, I wouldn't be able to run it on 20250120.774 as a reference point, and without a reference, the perceived "slowness" of the workflow would be too subjective.

I ran some experiments with the macOS 15 ARM 64 image and did not experience any slowdowns with this version.

As a result, we have decided to switch from the macOS 14 ARM 64 image to macOS 15 ARM 64.

The macOS 15 ARM 64 20250120.596 image appears to be working fine, so we plan to stick with it.

[aartis17]

Hi @odisei369 ,

That's great to hear that the macOS 15 ARM 64 image is working well for you! In the meantime, I investigated the issue with macOS 14 ARM64, specifically comparing last week's version (20250324.1158) and this week's rollout version (20250331.1204). I ran the same build on both versions and observed that the build completed successfully in 22 minutes with version 20250331.1204. Based on this, it seems likely that this week's rollout should resolve the issue. Thanks.

Please find below snips:

Workflow:

 - name: Run Tests
        run: |
          set -o pipefail && xcodebuild test \
            -project test.xcodeproj \
            -scheme test \
            -destination 'platform=iOS Simulator,name=iPhone 16,OS=18.1' \
            -skipPackagePluginValidation \
            -skipMacroValidation \
            CODE_SIGN_IDENTITY="" \
            CODE_SIGNING_REQUIRED=NO \
            -derivedDataPath .build \
            | xcbeautify --renderer github-actions

[aartis17]

Heads up !
New runners deployed for macOS 14 ARM 64 - 20250331.1204.
@odisei369 Could you please check and do let us know if issue is resolved now?
Thanks.