Azure private container registry config does not work

[mobini-sph]

Bug report

Azure private registry credentials are not used by nextflow running on Azure Batch.

Expected behavior and actual behavior

Pipeline processes should be able to pull the image from the specified registry as explained here.

Steps to reproduce the problem

nextflow.config

process {
  executor = 'azurebatch'
}

azure {
  storage {
    accountName = '<storage account name>'
    accountKey = '<storage key>' 
  }
  batch {
    location = '<location>'
    accountName = '<batch account name>'
    accountKey = '<batch key>'   
    pools {
        auto {
            ...
        }
     }
  }
  registry {
    server = '<registry name>.azurecr.io' 
    userName = '<registry name>'
    password = '<registry key>'
  }
}
...

process

process process_name {

  container '<registry name>.azurecr.io/<image name>:<tag>'

  ...

}

The pipeline is executed from the command line and within Batch Nodes; neither works as expected.

Program output

The Batch Job was created but failed to start with the error below.

Name:
Message 

Value:
Head "https://<registry name>.azurecr.io/v2/<image name>/manifests/<tag>": unauthorized: authentication required, visit https://aka.ms/acr/authorization for more information.

The process reports the error message below.

ERROR ~ Error executing process > 

Caused by:
  One or more container images specified are invalid

Environment

• Nextflow version: [24.10.2]
• Java version: [openjdk 11.0.24 2024-07-16]
• Operating system: [Linux]
• Bash version: 5.0.17(1)-release (x86_64-pc-linux-gnu)

Additional context

Docker login and pull work fine locally and on Batch Node:

docker login <registry name>.azurecr.io -u <registry name> -p <registry key>
docker pull '<registry name>.azurecr.io/<image name>:<tag>'

[zjupNN]

Any news regarding that bug? We face similar problem in our solution.

[bentsherman]

@adamrtalbot have you played with Azure CR at all? Wondering if you have any insights here

[adamrtalbot]

Yes, the authentication is applied to the virtual machines themselves. If the node has attached credentials to access the ACR it should be able to pull images. I've done it before, but I will check for a regression.

[adamrtalbot]

@mobini-sph, regarding this:

Docker login and pull work fine locally and on Batch Node:

docker login <registry name>.azurecr.io -u <registry name> -p <registry key>
docker pull '<registry name>.azurecr.io/<image name>:<tag>'

Are you able to pull the image without running docker login? the node should already be authenticated.

[adamrtalbot]

So I've haven't been able to recreate the problem. Here's my azure.config:

process {
    executor = 'azurebatch'
    queue = 'auto'
    container = 'seqeralabsdev.azurecr.io/wave:sed-4.7--bbe9845d456fc2d3'
}

azure {
    storage {
        accountName   = "$AZURE_STORAGE_ACCOUNT_NAME"
    }
    batch {
        location      = "$AZURE_BATCH_ACCOUNT_REGION"
        accountName   = "$AZURE_BATCH_ACCOUNT_NAME"
        autoPoolMode  = true
        pools {
            auto {
                vmType      = 'Standard_E2d_v4'
                lowPriority = true
                autoScale = true
            }
        }
    }
    registry {
        server = 'seqeralabsdev.azurecr.io'
        userName = "$ACR_USERNAME"
        password = "$ACR_PASSWORD"
    }
}

And you can see the created pool has a container registry attached:

Furthermore, I can see the task is using a container in that private registry.

Could you clarify what you're expecting to see @mobini-sph?

[zjupNN]

@adamrtalbot Maelthe confirmed the setup you've showed works, however with using wave the problem still exists (but I'm not sure if this is an issue at all for us). Thanks!

[adamrtalbot]

Are you using a Seqera Platform workspace with credentials to the private registry added? Wave needs to use Seqera to act as an intermediary to authenticate to the private registry.

Steps:

1. Create an organisation and a workspace
2. Add the Docker registry credentials to the workspace
3. Get the workspace ID (it can be found by looking at the list of workspaces within the organisation)
4. Run Nextflow with the following configuration (in addition to the existing one):

tower {
    enabled = true
    workspaceId = '123456789'
}

wave {
    enabled = true
}

[mobini-sph]

Thanks, Adam. Looks like the issue was with the previously created pool. I didn't find the registry info in the VM config. It worked when I deleted the pool and let nf create a new one. So, whenever we need to enable or change the registry, the pool must be deleted and a new one created.

[adamrtalbot]

Once you create the pool, the container authentication details are fixed and can't edited on Azure, so yep, you need to break the node pool down and recreate it.