adapter: wrong resource type set making queries fail

[ocervell]

Hi there,

Trying to configure an HPA based on a managed Prometheus metric like:

"type": "Pods",
"pods": {
	"metric": {
		"name": 'prometheus.googleapis.com|celery_queue_length|gauge',
	},
	"target": {
		"type": "AverageValue",
		"averageValue": int(concurrency / 2),
	}
}

The managed prometheus metric has the resource type "prometheus_target" but the filter builder builds it like:

I0224 22:13:27.933552       1 filter_builder.go:258] Query with filter(s): "metric.labels.pod = one_of(\"celery-worker-c6686897c-flm8h\",\"celery-worker-c6686897c-hlqgv\",\"celery-worker-c6686897c-lzqd2\") AND metric.type = \"prometheus.googleapis.com/celery_queue_length/gauge\" AND resource.labels.cluster = \"secator-cloud-gke\" AND resource.labels.location = \"europe-west1\" AND resource.labels.namespace = \"env-freelabz\" AND resource.labels.project_id = \"<PROJECT_ID>\" AND resource.type = \"prometheus_target\""
I0224 22:13:27.967956       1 filter_builder.go:258] Query with filter(s): "metric.type = \"prometheus.googleapis.com/celery_queue_length/gauge\" AND resource.labels.cluster_name = \"secator-cloud-gke\" AND resource.labels.location = \"europe-west1\" AND resource.labels.namespace_name = \"env-freelabz\" AND resource.labels.pod_name = one_of(\"celery-worker-c6686897c-flm8h\",\"celery-worker-c6686897c-hlqgv\",\"celery-worker-c6686897c-lzqd2\") AND resource.labels.project_id = \"<PROJECT_ID>\" AND resource.type = \"k8s_container\""

Can you please explain why the change of resource.type from "prometheus_target" to "k8s_container" ?

It makes the queries fail afterwards:

E0224 22:13:27.877496 1 status.go:71] apiserver received an error that is not an metav1.Status: &googleapi.Error{Code:400, Message:"The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.", Details:[]interface {}(nil), Body:"{\n "error": {\n "code": 400,\n "message": "The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.",\n "errors": [\n {\n "message": "The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.",\n "domain": "global",\n "reason": "badRequest"\n }\n ],\n "status": "INVALID_ARGUMENT"\n }\n}\n", Header:http.Header{"Content-Type":[]string{"application/json; charset=UTF-8"}, "Date":[]string{"Mon, 24 Feb 2025 22:13:27 GMT"}, "Server":[]string{"ESF"}, "Vary":[]string{"Origin", "X-Origin", "Referer"}, "X-Content-Type-Options":[]string{"nosniff"}, "X-Frame-Options":[]string{"SAMEORIGIN"}, "X-Xss-Protection":[]string{"0"}}, Errors:[]googleapi.ErrorItem{googleapi.ErrorItem{Reason:"badRequest", Message:"The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series."}}, err:(*apierror.APIError)(0xc002516510)}: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badReques

and the HPA message:

  Warning  FailedGetPodsMetric  21s (x373 over 47m)  horizontal-pod-autoscaler  unable to get metric prometheus.googleapis.com|celery_queue_length|gauge: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest

[ocervell]

I got i working by changing the HPA spec to:

"type": "External",
"external": {
	"metric": {
		"name": 'prometheus.googleapis.com|celery_queue_length|gauge',
	},
	"target": {
		"type": "AverageValue",
		"averageValue": int(concurrency / 2),
	}
}

but the documentation here is wrong: https://cloud.google.com/kubernetes-engine/docs/tutorials/autoscaling-metrics#custom-metric_8