Description
Hi there,
Trying to configure an HPA based on a managed Prometheus metric like:
"type": "Pods",
"pods": {
"metric": {
"name": 'prometheus.googleapis.com|celery_queue_length|gauge',
},
"target": {
"type": "AverageValue",
"averageValue": int(concurrency / 2),
}
}
The managed prometheus metric has the resource type "prometheus_target" but the filter builder builds it like:
I0224 22:13:27.933552 1 filter_builder.go:258] Query with filter(s): "metric.labels.pod = one_of(\"celery-worker-c6686897c-flm8h\",\"celery-worker-c6686897c-hlqgv\",\"celery-worker-c6686897c-lzqd2\") AND metric.type = \"prometheus.googleapis.com/celery_queue_length/gauge\" AND resource.labels.cluster = \"secator-cloud-gke\" AND resource.labels.location = \"europe-west1\" AND resource.labels.namespace = \"env-freelabz\" AND resource.labels.project_id = \"<PROJECT_ID>\" AND resource.type = \"prometheus_target\""
I0224 22:13:27.967956 1 filter_builder.go:258] Query with filter(s): "metric.type = \"prometheus.googleapis.com/celery_queue_length/gauge\" AND resource.labels.cluster_name = \"secator-cloud-gke\" AND resource.labels.location = \"europe-west1\" AND resource.labels.namespace_name = \"env-freelabz\" AND resource.labels.pod_name = one_of(\"celery-worker-c6686897c-flm8h\",\"celery-worker-c6686897c-hlqgv\",\"celery-worker-c6686897c-lzqd2\") AND resource.labels.project_id = \"<PROJECT_ID>\" AND resource.type = \"k8s_container\""
Can you please explain why the change of resource.type from "prometheus_target" to "k8s_container" ?
It makes the queries fail afterwards:
E0224 22:13:27.877496 1 status.go:71] apiserver received an error that is not an metav1.Status: &googleapi.Error{Code:400, Message:"The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.", Details:[]interface {}(nil), Body:"{\n "error": {\n "code": 400,\n "message": "The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.",\n "errors": [\n {\n "message": "The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series.",\n "domain": "global",\n "reason": "badRequest"\n }\n ],\n "status": "INVALID_ARGUMENT"\n }\n}\n", Header:http.Header{"Content-Type":[]string{"application/json; charset=UTF-8"}, "Date":[]string{"Mon, 24 Feb 2025 22:13:27 GMT"}, "Server":[]string{"ESF"}, "Vary":[]string{"Origin", "X-Origin", "Referer"}, "X-Content-Type-Options":[]string{"nosniff"}, "X-Frame-Options":[]string{"SAMEORIGIN"}, "X-Xss-Protection":[]string{"0"}}, Errors:[]googleapi.ErrorItem{googleapi.ErrorItem{Reason:"badRequest", Message:"The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series."}}, err:(*apierror.APIError)(0xc002516510)}: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badReques
and the HPA message:
Warning FailedGetPodsMetric 21s (x373 over 47m) horizontal-pod-autoscaler unable to get metric prometheus.googleapis.com|celery_queue_length|gauge: unable to fetch metrics from custom metrics API: googleapi: Error 400: The supplied filter does not specify a valid combination of metric and monitored resource descriptors. The query will not return any time series., badRequest