Dependabot is failing to authenticate against a private Hex Repisotiry

[adamtharani]

Is there an existing issue for this?

• I have searched the existing issues

Package ecosystem

Hex

Package manager version

No response

Language version

Elixir

Manifest location and content before the Dependabot update

/directory/mix.exs

dependabot.yml content

version: 2

registries:
  fluxon-hex-repository:
    type: hex-repository
    repo: fluxon
    url: https://repo.fluxonui.com
    auth-key: "${{ secrets.FLUXON_HEX_REPO_AUTH_KEY }}"
    public-key-fingerprint: ${{ secrets.FLUXON_HEX_REPO_PUBLIC_KEY_FINGERPRINT }}
updates:
  - package-ecosystem: mix
    directory: "/"
    registries:
      - fluxon-hex-repository
    insecure-external-code-execution: allow
    schedule:
      interval: weekly
      day: thursday
    groups:
      production-dependencies:
        dependency-type: production
      dev-dependencies:
        dependency-type: development

Updated dependency

No response

What you expected to see, versus what you actually saw

The package manager is failing to update due to private_source_authentication_failure

Handled error whilst updating ash: private_source_authentication_failure {:source=>"fluxon"}

Native package manager behavior

No response

Images of the diff or a link to the PR, issue, or logs

No response

Smallest manifest that reproduces the issue

No response

[frerich]

I'm seeing the same.

Tracing the HTTP calls done to the private repository, I can see that the authorization HTTP header is always empty instead of having whatever value is given to auth-key in the YAML file.

I see there's a unit test for this scenario (https://github.com/dependabot/dependabot-core/blob/main/hex/spec/dependabot/hex/update_checker_spec.rb#L387-L398) and that test seems to do the right thing. Can we debug how the value of auth-key in the YAML file ends up as a Dependabot::Credential key?

[frerich]

@sorentwo can you provide some guidance on how to tackle this? Does dependabot still work with the Oban repo?

[TylerWitt]

Is there a sample repo you could make? If it's just Oban I can try to figure it out.

[TylerWitt]

@frerich Try auth_key instead of the dashed version. I think the documentation is incorrect potentially.

[frerich]

@TylerWitt To reproduce this issue, you can clone https://github.com/frerich/dependabot_testbed -- it's just the result of mix new dependabot_testbed with one dependency added referencing a private acmecorp repo. After cloning it, you should set a HEX_REPO_AUTH_KEY secret in the cloned GitHub repo.

The .github/dependabot.yml file describes how to access that repo: I use a dynamically generated URL (https://webhook.site/d8ea1da7-8afc-4879-9998-65a3d765010f) which lets me inspect all requests sent by Dependabot. This shows how the authorization header is submitted - but it's empty.

Modifying the testspec/dependabot/hex/update_checker_spec.rb:391 however such that it uses the webhook.site URL and then running the test shows that it does submit the auth-key value.

So right now I'm speculating that it's something funny about how the YAML file is turned into a Dependabot::Credential object.

[TylerWitt]

@frerich I can confirm the YAML is being read correctly--but the credentials arg is an empty list when it hits the hex code.

Will keep digging.

[TylerWitt]

It is possible that this issue is happening before dependabot-core. As an example, it seems like dependabot-cli trims certain credential fields and then a proxy service injects them again. The dry-run also does not seem to function well with private repositories (in any language) unless you pass them in with LOCAL_CONFIG_VARIABLES...

[TylerWitt]

When I pass in the contents of my registries from the yaml

LOCAL_CONFIG_VARIABLES="[{\"type\":\"hex_repository\",\"repo\":\"dependabot\",\"url\":\"https://dependabot-private.fly.dev\",\"auth_key\":\"d6fc2b6n6h7katic6vuq6k5e2csahcm4\"}]"

into the dev container, and run the dry run on (my version) of your repository, it works.

So indeed, something about how execution is running for Elixir.

[TylerWitt]

@abdulapopoola Is it possible for me to debug this further? Or is it more likely something internal?

[abdulapopoola]

Thanks @TylerWitt , yes this will require some internal work, I've added this to our tracking board and hope to get back to you soon.

[frerich]

@TylerWitt Much appreciated for tracing this further! I had hoped that this would be something within dependabot-core since then I could hack it into submission. Apparently, that's not the case though.

I wonder how this would ever work for anyone, e.g. users of libraries like Oban Pro which are hosted on a private Hex repository and require an auth key. I suppose this worked at some point after @sorentwo implemented this.

[sorentwo]

@frerich Yes, this certainly worked when I implemented it. The hex repo API hasn't changed since then, so I'm not sure what's changed.

[bgentry]

@sorentwo @abdulapopoola I can confirm that our previously working setup for using Oban Pro's private hex repo w/ Dependabot spontaneously stopped working on its own sometime in the past ~2 months, with no changes to our Dependabot config, secrets, or anything. I even tried using the new Oban Pro repo hoping it would resolve the issues but no luck there.

It seems pretty clear that something regressed in Dependabot on this front. I'm able to manually hit the package endpoints locally with a curl like this:

curl -v https://repo.oban.pro:443/packages/oban_pro -H "Authorization: MY_TOKEN"

But Dependabot is seeing a 401 when trying to fetch it, even though I have my secret set up correctly (and always have):

proxy | 2025/03/04 17:43:23 [086] 401 [https://repo.oban.pro:443/packages/oban_pro](https://repo.oban.pro/packages/oban_pro)

Clearly it's not setting the auth header correctly based on my config, which matches what Oban says to do in its docs.

[probably-not]

@abdulapopoola Is anybody looking at this and is there any timeline of when it will be fixed? This is definitely a regression from the Dependabot side, it's simply not passing the Authentication header at all, I can see in my registry's logs that it's always empty. If this isn't going to be fixed soon - can we at least update the Dependabot docs to say that it's currently broken? The issue has been open and happening for months, the docs should be updated.

This is also not only breaking just private Hex Registries, but because of the failure, I'm seeing that Dependabot is now marking all dependency updates as failed due to authentication failures (including dependencies that are part of the public Hex Registry). This makes Dependabot completely unusable for Elixir projects that use any private packages.

From Dependabot logs:

+---------------------------------------------------------------+
|                 Dependencies failed to update                 |
+-----------------------+---------------------------------------+
| ex_cldr_dates_times   | private_source_authentication_failure |
| earmark_parser        | private_source_authentication_failure |
| dns_cluster           | private_source_authentication_failure |
| hackney               | private_source_authentication_failure |
| tailwind              | private_source_authentication_failure |
| opentelemetry_phoenix | private_source_authentication_failure |
| polymorphic_embed     | private_source_authentication_failure |
| tzdata                | private_source_authentication_failure |
| postgrex              | private_source_authentication_failure |
| ex_cldr_calendars     | private_source_authentication_failure |
| phoenix_live_view     | private_source_authentication_failure |
| req                   | private_source_authentication_failure |
| certifi               | private_source_authentication_failure |
| ecto_sqlite3          | private_source_authentication_failure |
| bandit                | private_source_authentication_failure |
| ex_cldr               | private_source_authentication_failure |
| phoenix_html          | private_source_authentication_failure |
| muontrap              | private_source_authentication_failure |
+-----------------------+---------------------------------------+

None of these are from the private registry that we are using, but they are all failing because of the private_source_authentication_failure.

[ftes]

This is also not only breaking just private Hex Registries, but because of the failure, I'm seeing that Dependabot is now marking all dependency updates as failed due to authentication failures (including dependencies that are part of the public Hex Registry). This makes Dependabot completely unusable for Elixir projects that use any private packages.

I can confirm that this is affecting us in the same way.

[probably-not]

@abdulapopoola Is anybody looking at this? This is a major issue and regression in Dependabot and there's been no answers or activity from maintainers in many months... Can we get any answers on what's going on here?

[abdulapopoola]

Hello everyone,

So sorry about this - we got overwhelmed. The main challenge has been finding someone to investigate this since we are spread thin. Currently exploring some options and should update the thread tomorrow.

[bruce]

👋 Hi everyone; longtime Elixir developer (and hubber) here. This issue was brought to my attention at AlchemyConf today, and @abdulapopoola has been kind enough to accept my offer to help out from our end.

Thanks for all the context above. I may have more questions as I investigate the source of the regression and work on a fix. Please bear with me as I'm traveling back from the conference and need to work with the team to get acquainted with the internals; it's possible they may fix it first, too!

[jakecoffman]

I found an issue in the Dependabot Proxy which was making it inject credentials too conservatively. So I think this PR should have fixed it for Dependabot on Actions: github/dependabot-action#1448

Could someone verify that it's working now?

Thanks to @frerich putting up the open source example, made it easy to track down!

[ftes]

Thanks @jakecoffman
It's working now for us.

[abdulapopoola]

Thanks again @jakecoffman for the fix! Closing this out.

[bgentry]

@abdulapopoola @bruce any chance you could point me at the specific underlying code changes that fixed this issue? I really want to get the GOPROXY issue (#7362) solved and might be able to help with that if I could see how this all works / where a change is needed. Thanks 🙏

[abdulapopoola]

@bgentry , this was an internal tooling fix.