feat: Support custom BigQuery storage api endpoint

[nj1973]

Description of changes

The PR adds support for a custom googleapis endpoint for the BigQuery Storage API, in addition to the BigQuery and Spanner APIs that we already support.

I have manually tested this in a project with a PSC and firewall rules to block access to standard googleapis.com endpoints and the changes work. I don't think we can realistically add integration tests for this setup though.

Now I have a project I can test in I will be able to test Secret Manager integration and raise an issue for that too. As we have a customer waiting on these changes I didn't plan to test/include that support in this PR.

Issues to be closed

Closes #1369

Checklist

• I have followed the CONTRIBUTING Guide.
• I have commented my code, particularly in hard-to-understand areas
• I have updated any relevant documentation to reflect my changes, if applicable
• I have added unit and/or integration tests relevant to my change as needed
• I have already checked locally that all unit tests and linting are passing (use the tests/local_check.sh script)
• I have manually executed end-to-end testing (E2E) with the affected databases/engines

[nj1973]

/gcbrun

[nj1973]

/gcbrun

[sundar-mudupalli-work]

Neil,

I don't know how this is supposed to work. I am a bit confused.

Sundar Mudupalli

[sundar-mudupalli-work]

Hi,

I don't understand how this is going to work. According to the changes in the documentation, the user passes a string as the endpoint. Here, self.client (a python object) representing the connection to BQ is being assigned a string value. That does not sound right.

If an API endpoint is being used, based on BigQuery client code, you need to pass the client_options kwarg an instance of the ClientOptions class while creating a connection. ClientOptions is defined in API Core Client Options

The latest version of Ibis performs an assignment much like what is being proposed above. However, the documentation, says it is a Client (object) from google.cloud.bigquery package.

Based on the above analysis - one way to accomplish this might be to have DVT environment variables for API endpoints for BQ, GCS, Secret Manager and Spanner. If those are set, then we use those values when connecting to the client API's. I have started a section in the doc on running DVT at on-prem where this explanation may fit. This looks like a more complicated change. I would rather be wrong and we have a simple fix - not sure that is possible here.

I am not aware that we use BigQuery storage API. That is only for Streaming reads.

We might be able to test this by running DVT from a laptop with an IAP tunnel to a PSC endpoint for googleapis.com in pso-kokoro-resources project. It may be too complicated to make the test part of the integration suite.

Thanks.

Sundar Mudupalli