Using GOOGLE_APPLICATION_CREDENTIALS to specify an authorized_user or impersonated_service_account JSON key file giving "missing field `private_key`" error

[liufuyang]

Hi there, thanks for creating this nice package.

Does it support all kinds of different key jsons.

I tried to use the the json generated via
gcloud auth application-default login
and the key is located at /Users/fuyangl/.config/gcloud/application_default_credentials.json

CustomServiceAccountCredentials(Error("missing field `private_key`", line: 11, column: 1))

The key as "client_id", "client_secret", "refresh_token" and "type" as "authorized_user", that is all the fields there.

[hrvolapeter]

Hi @liufuyang did you manage to solve this problem? I'm thinking it could have something to do with newer version of gcloud? I'll investigate

[liufuyang]

I am not quite sure. there is no field of private_key in the json file.

[dacozai]

Hi, @liufuyang did you fix the issue?
I successfully use gcp_auth to fetch the token based on the following env

1. same application_default_credentials as yours
2. Google Cloud SDK 420.0.0
3. gcp_auth = "0.7.5"

Based on my testing, I am able to decrypt based on the token from gcp_auth in the following environments.

1. macOS M1 Ventura 13.2.1 (22D68)
2. debian:bullseye-slim running on GKE

To see if there is any difference from yours, I'd love to check this up. Or, you could provide a repository that reproduces the issue for me. Thanks 🙏🏻

[liufuyang]

Thanks a lot. Very sorry I lost my environment when I tested this and tried again with the setup and it works fine.

But I remember previously I was playing with gcloud auth application-default loginlogin --impersonate-service-account=<Service Account>, perhaps I had a GOOGLE_APPLICATION_CREDENTIALS pointing to another json file when I did the test.

So I tried again to generate a json key with the command above, using --impersonate-service-account to bind an SA.

Then the generated application_default_credentials.json looks like

{
  "delegates": [],
  "service_account_impersonation_url": "https://iamcredentials.googleapis.com/v1/projects/-/serviceAccounts/xxx@xxx.iam.gserviceaccount.com:generateAccessToken",
  "source_credentials": {
    "client_id": "xxxxx.apps.googleusercontent.com",
    "client_secret": "xxxxx",
    "refresh_token": "xxxxx",
    "type": "authorized_user"
  },
  "type": "impersonated_service_account"
}

And when using this impersonated_service_account type key it gives this
Error: CustomServiceAccountCredentials(Error("missing field private_key", line: 11, column: 1))

Perhaps you can take a look in this direction? Maybe it is nice to be able to support impersonated_service_account? Otherwise, it is not anything urgent as it might be a future not many people would use in practice.

[dacozai]

I'm sorry for not getting back to you sooner. I was too busy to check this up for you lately.
After checking, I find this crate should be able to handle your case.
If not, I think it's better to update the crate to the latest version.

Below are the steps on how I tested it.

1. gcloud auth application-default login --impersonate-service-account=<service-account-email> reference
2. check the format cat $HOME/.config/gcloud/application_default_credentials.json which is the same as what you provided with
3. cargo test

gcloud SDK INFO -- Google Cloud SDK 424.0.0

ALL TESTs passed.
Apart from that, I have built a tiny script to confirm the functionality.
Please don't hesitate to check this by yourself.
cargo add reqwest
touch src/main.rs

use reqwest::header::CONTENT_TYPE;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
  let authentication_manager = gcp_auth::AuthenticationManager::new().await?;
  let _token = authentication_manager
    .get_token(&["https://www.googleapis.com/auth/cloud-platform"])
    .await?;

  // Ref: https://cloud.google.com/storage/docs/listing-objects#permissions-rest
  let bucket_name = "REPLACE_THIS_BY_YOURS_BUCKET_NAME".to_string();
  let client = reqwest::Client::new();
  let uri = format!(
    "https://storage.googleapis.com/storage/v1/b/{}/o",
    bucket_name
  );

  let response = client.get(uri)
      .header(CONTENT_TYPE, "application/x-www-form-urlencoded")
      .bearer_auth(&_token.as_str().to_string()).send().await?.text().await?;

  println!("Response: {}", response);

  Ok(())
}

[hrvolapeter]

thanks @dacozai for a suggestion, however I also see value in having direct support for impersonation so I'll leave this open so far

[dacozai]

@liufuyang May I ask which version you use?
gcp_auth: ?
gcloud SDK: ?

thanks 🙏🏻

[liufuyang]

Google Cloud SDK 406.0.0
gcp_auth: 0.8.0

@dacozai Okay, it seems a bug (or just missing a feature) in the code when the GOOGLE_APPLICATION_CREDENTIALS is used for an authorised_user or impersonated_service_account type of JSON key. I did it like this:

1. Create a JSON key. gcloud auth application-default (doesn't matter if impersonation is used or not)
2. Check the key and the type is correct at cat ~/.config/gcloud/application_default_credentials.json, and it shows "type": "authorized_user"
3. Run some sample code (attached below) with cargo run, and it works.
4. But, if run with GOOGLE_APPLICATION_CREDENTIALS=/Users/fuyangl/.config/gcloud/application_default_credentials.json cargo run, basically manually specify the key to use, then it fails.
   Error: CustomServiceAccountCredentials(Error("missing field 'private_key'", line: 6, column: 1))

If an impersonation account JSON is used, then the error shows as a different line 11: Error: CustomServiceAccountCredentials(Error("missing field 'private_key'", line: 11, column: 1))

My testing code is:

use gcp_auth::AuthenticationManager;

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let authentication_manager = gcp_auth::AuthenticationManager::new().await?;
    let token = authentication_manager
        .get_token(&["https://www.googleapis.com/auth/cloud-platform"])
        .await?;

    println!("{}", token.as_str());
    Ok(())
}

[dependencies]
gcp_auth = "0.8.0"
tokio = {version = "1.26.0", features = ["macros", "parking_lot", "rt-multi-thread"]}

So I guess if you just run with cargo run or perhaps cargo test, it doesn't use the /Users/fuyangl/.config/gcloud/application_default_credentials.json file and then uses the gcloud command, as mentioned in the main Readme?

[msdrigg]

This isn't restricted to impersonated service accounts. I am also seeing this with just using the standard application_default_credentials

Version

• gcloud sdk version 441.0.0
• gcp_auth version 0.9.0

Steps to reproduce

• Install google cloud sdk (latest version is 441.0.0, but this was happening at least as early as 438.0.0)
  • brew install --cask google-cloud-sdk
  • gcloud -V "Google Cloud SDK 441.0.0"
• Login
  • gcloud auth application-default login
• Run application
  • GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json cargo run
• See error

0: authorizer error: Application profile provided in `GOOGLE_APPLICATION_CREDENTIALS` was not parsable
1: Application profile provided in `GOOGLE_APPLICATION_CREDENTIALS` was not parsable
2: missing field `private_key` at line 7 column 1', src/main.rs:18:34

• Inspect ~/.config/gcloud/application_default_credentials.json

{
  "client_id": "*******.apps.googleusercontent.com",
  "client_secret": "*******",
  "quota_project_id": "*******",
  "refresh_token": "*******",
  "type": "authorized_user"
}

The reason I need to use GOOGLE_APPLICATION_CREDENTIALS instead of other methods is because I am trying to run my program locally under docker-compose where I won't have gcloud sdk installed.

[msdrigg]

Looking into this further, I found a stack overflow posts from back in 2019 that reference the application_default_credentials matching the format I am seeing, so this isn't an issue with a gcloud sdk update.

The problem is that gcp_auth doesn't support using GOOGLE_APPLICATION_CREDENTIALS to reference application_default_credentials.json. Instead it requires that application_default_credentials.json is found at $HOME/.config/gcloud/application_default_credentials.json.

https://github.com/hrvolapeter/gcp_auth/blob/6e6a44093b0f4db2a1d4d25d32f8a4cb484665cd/src/default_authorized_user.rs#L25-L38

This behavior is contrary to other google tools. For example the cloud_sql_proxy handles GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json just fine.

I believe this issue should be split into 2 issues

• one to support impersonated service accounts
• one to support GOOGLE_APPLICATION_CREDENTIALS=~/.config/gcloud/application_default_credentials.json

[msdrigg]

For others seeing this error due to docker, I can work around this by creating a non-root user with a home dir at /app and then mounting the host's ~/.config/gcloud to the guests /app/.config/gcloud

[djc]

Okay, so it appears there are (at least) two different kinds of JSON being used, one perhaps for users and one for service accounts? The one for service accounts includes a private_key whereas the other one does not.

In order to get this right, it would be good if someone could dig out references to Google documentation (or, to Google source code, for example in their SDKs for other languages) that more clearly shows how these differ and when we're supposed to use one vs the other.

[liufuyang]

@djc Thanks. Perhaps following some idea illustrated here?
https://github.com/golang/oauth2/blob/ac6658e9cb5802cebf9b8fd5f5d58f22bedb527f/google/google.go#L160-L163

Here you also see the private_key is used for Service Account but some clientSecret is used instead of that for User Credential (which typically comes from gcloud auth)
https://github.com/golang/oauth2/blob/ac6658e9cb5802cebf9b8fd5f5d58f22bedb527f/google/google.go#L108-L120