Add forward proto header configuration for cluster monitoring

[raj-manvar]

Description

As documented at https://trino.io/docs/current/security/tls.html#use-a-load-balancer-to-terminate-tls-https, when Trino is behind a loadbalancer or proxy like Trino Gateway, it's common that the TLS is terminated at Trino Gateway.

Correspondingly when Trino Gateway forwards the request to Trino clusters, Gateway adds X-Forwarded-* headers in code here. Relevant documentation is here where users can optionally disable this by setting routing.addXForwardedHeaders to false.

This MR is to add the same Header while making health check calls to get cluster stats like queued queries or running queries. Since it's possible that TLS is terminated at Gateway, a similar header would be required when making the http calls to fetch the cluster stats, for example using the /metrics or /v1/jmx/mbean endpoints

If such a header isn't added, the http call to fetch metrics would fail with an error like:

UnexpectedResponseException{message=Request is missing required keys: 
presto_execution_name_QueryManager_RunningQueries
presto_execution_name_QueryManager_QueuedQueries
trino_metadata_name_DiscoveryNodeManager_ActiveNodeCount
in response: 'Error 403 Forbidden: Authentication over HTTP is not enabled', request=GET http://xxxxxxx/metrics?xxxx

Additional context and related issues

Release notes

( ) This is not user-visible or is docs only, and no release notes are required.
(x) Release notes are required, with the following suggested text:

* Add option to add `X-Forwarded-Proto` when fetching cluster stats.

[Chaho12]

@andythsu Can you check this?

[vishalya]

LGTM

[andythsu]

On a second thought, is this PR necessary? We are currently using UI_API for healthchecks which calls Trino via HTTPS. We had to turn on ssl for this. How is making calls to /metrics different from UI_API?

[raj-manvar]

@andythsu https://trino.io/docs/current/security/tls.html#use-a-load-balancer-to-terminate-tls-https has more details, but when a Trino cluster is behind a LoadBalancer/Gateway it usually just doesn't accept https connections ( and only accepts non-secure http connections) . I think in your case, both Gateway and Trino clusters maybe accepting https connections, but that maynot always be the case.
Copy pasting from the above page.

The load balancer or proxy server accepts TLS connections and forwards them to the Trino coordinator, which typically runs with default HTTP configuration on the default port, 8080.

This is because the Trino clusters don't have a certificate which is globally trusted, only the LoadBalancer/Gateway is mounted with a trusted certificate.
For Trino to accept any non-secure http connection with credentials ( via /metrics endpoints or /v1/query) a X-Forwarded-Proto header is required

[raj-manvar]

We could also consider adding this header by default for all cluster health http calls. For the /v1/statement endpoints this header is added by default here ( unless users specifically set routing.addXForwardedHeaders to false ).
I think having the header maybe harmless incase of secure https connections anyways so we could add it by default

[raj-manvar]

Hello all, is this good to merge now ? Thanks!

[ebyhr]

Could you change to String xForwardedProtoHeader instead?

[raj-manvar]

hi, sorry to clarify, you mean the property would take a string value like http or https ? or the property would take string value of true or false ?

[ebyhr]

The former.

[raj-manvar]

Thanks for the clarification! Could there be any scenario where the value of the header is anything except https ? 🤔 I don't know of any such cases.

[ebyhr]

I'm okay with boolean, but we usually don't use "add" prefix. I think we can simply rename it to "xForwardedProtoHeader"