Authentication Issue When Connecting Through Bastion Host and IAP

[apolienko]

Question

Environment

• Google Cloud Platform (GCP)
• Cloud SQL PostgreSQL instance with private IP only
• Bastion host with private IP only
• Identity-Aware Proxy (IAP) for bastion host access
• DBeaver as SQL client

Problem

I'm trying to connect to a PostgreSQL instance in Google Cloud SQL from my local machine. The database has only a private IP address, so I'm using a bastion host to access it.

Bastion has a systemd service that is using

/usr/local/bin/cloud-sql-proxy --auto-iam-authn --private-ip --address 0.0.0.0 --port 3307 [connection-string]

I've set up IAP TCP forwarding to the bastion host with this command:

gcloud compute start-iap-tunnel [bastion-name] 3307 --local-host-port=localhost:5432 --zone=[zone]

When I try to connect through DBeaver using:

• Host: localhost
• Port: 5432
• Database: db_name
• Username: [email protected]
• Password: (empty)

I receive the following error:

FATAL: Cloud SQL IAM user authentication failed for user "[email protected]"

This setup previously worked when I was using cloud-sql-proxy locally. However, now that cloud-sql-proxy is running on the bastion host, the authentication fails.

What I've tried

I've confirmed that my user account has the necessary IAM permissions and has been added to Cloud SQL. The direct connection using cloud-sql-proxy from my local machine worked before, but I need to route through the bastion host now.

What might be causing this authentication failure, and how can I properly configure the connection to maintain IAM authentication while going through the bastion host?

Code

Additional Details

No response

[apolienko]

@hessjcg
Hey, Jonathan!
Could you tell me, please, if this is possible? or what would you recommend to find the best workaround / solution ?
Thank you!

[hessjcg]

Hi @apolienko,

When you enable IAM Auth, the proxy will use the IAM token from the machine where it is running. When you run the proxy locally, it will use your gcloud auth credentials. However, if you run it on the bastion host, it will use the service account from that host.

We recommend running the Auth Proxy locally, however, mainly for security reasons.

-Jonathan

[apolienko]

@hessjcg
Thank you! Is it possible to run cloud-sql-proxy locally using only private ip? (public ip disabled)

[hessjcg]

@apolienko, unfortunately no. Private IP instances may only be accessed from VMs that are attached to the private network. Since you are already using a bastion host, you may need to log in to that host using SSH, authenticate the gcloud CLI, and then run the proxy and database client on the bastion host.

https://cloud.google.com/sql/docs/mysql/connect-to-instance-from-outside-vpc has more details on how to connect to a Private IP instance from outside it's VPC.

[mike-pt]

Basically bumping to the same limitation, I was even thinking of have some wrapper script and somehow pass the auth tho a docker container in the VM... alternatively we can use normal SQL users instead of IAM auth I suppose....

But considering google has IAP for other things I wonder if there are any plans for cloud-sql-proxy to one day support some from of IAP tunnel?

Similar to how for SSH we can do tunnel-trought-IAP, though this might be more like a dream feature than something actually doable :D

[hessjcg]

@mike-pt, we are considering ways to make this easier and an IAP tunnel directly to the DB is one of them.

Until then, you can use IAP to connect to VM and then port forward to the db as described in Connect to an instance from outside vpc I hope that helps.