Unable to authenticate to Docker hub for pulls when running in Google Cloud Build

[WTPOptAxe]

Summary

Ahead of the dockerhub rate limit enforcement, we're attempting to switch our kaniko builds to use authenticated user for docker hub pulls. We're testing the authentication by pulling a private image. We can pull the image locally using the username and token we're injecting into /kaniko/.docker/config.json

Kaniko fails to pull the private FROM image when building in google cloud build.

I've looked at https://github.com/GoogleContainerTools/kaniko?tab=readme-ov-file#pushing-to-docker-hub as my starting point.

Error

INFO[0000] Resolved base name private/image:20250327 to deps
INFO[0000] Resolved base name private/image:20250327 to runner
INFO[0000] Retrieving image manifest private/image:20250327
INFO[0000] Retrieving image private/image:20250327 from registry index.docker.io
ERRO[0001] Error while retrieving image from cache: private/image:20250327 unable to complete operation after 0 attempts, last error: GET https://index.docker.io/v2/private/image/manifests/20250327: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:private/image Type:repository]]
INFO[0001] Retrieving image manifest private/image:20250327
INFO[0001] Retrieving image private/image:20250327 from registry index.docker.io
error building image: unable to complete operation after 0 attempts, last error: GET https://index.docker.io/v2/private/image/manifests/20250327: UNAUTHORIZED: authentication required; [map[Action:pull Class: Name:private/image Type:repository]]
ERROR
ERROR: build step 0 "gcr.io/kaniko-project/executor:debug" failed: step exited with non-zero status: 1

Auth process and debug

I'm creating a /kaniko/.docker/config.json file from a secret in GCP secret manager. The base64 secret decodes to

{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "user:token_encoded"
    }
  }
}

user:token_encoded is actually the base64 encoded username and token, not clear text as above.
I am able to do echo token | docker login --username username --password-stdin from the cli and then pull images, so I'm confident the token is correct.

Our cloudbuild.yaml is currently

steps:
  - name: "gcr.io/kaniko-project/executor:debug"
    entrypoint: "/busybox/sh"
    args:
      - -c
      - |
        echo "Looking at Kaniko docker config" && \
        if [ -f /kaniko/.docker/config.json ]; then cat /kaniko/.docker/config.json ; else echo "no config"; fi && \
        echo "Adding auth to Kaniko docker config" && \
        echo $$DOCKER_CONFIG | base64 -d > /kaniko/.docker/config.json && \
        echo "Looking at Kaniko docker config after creation" && \
        if [ -f /kaniko/.docker/config.json ]; then cat /kaniko/.docker/config.json ; else echo "no config"; fi && \
        eval /kaniko/executor --dockerfile=path/to/Dockerfile \
        --cleanup  --cache=true --cache-ttl=24h0m0s --reproducible --snapshot-mode=redo \
        --cache-copy-layers=true --compressed-caching=false \
        --destination=europe-west2-docker.pkg.dev/$PROJECT_ID/ourrepo/projectname:$COMMIT_SHA \
        --destination=europe-west2-docker.pkg.dev/$PROJECT_ID/ourrepo/projectname:$_BRANCH_NO_PREFIX-$SHORT_SHA-$(date +%s)
    secretEnv: ["DOCKER_CONFIG"]

availableSecrets:
  secretManager:
    - versionName: projects/gcp_project_id/secrets/secret_name/versions/latest
      env: "DOCKER_CONFIG"

In the output of the cloudbuild job, the echo lines from the cloudbuild.yaml shows that the contents of /kaniko/.docker/config.json are as expected

Looking at Kaniko docker config after creation 
{
  "auths": {
    "https://index.docker.io/v1/": {
      "auth": "user:token_encoded"
    }
  }
}

[mzihlmann]

dont want to spell out the obvious, but your request goes to index.docker.io/v2, your credentials are for index.docker.io/v1

[WTPOptAxe]

dont want to spell out the obvious, but your request goes to index.docker.io/v2, your credentials are for index.docker.io/v1

Is this part of the documentation no longer correct then?

Note: Please use v1 endpoint. See #1209 for more details

{
"auths": {
"https://index.docker.io/v1/": {
"auth": "xxxxxxxxxxxxxxx"
}
}
}