Malformed PrometheusExporter Metrics #5323
Comments
|
Ok, for reproduction, if I can somehow trigger the behavior, I would bet that it's something around the GraphQL collector here: # lib/graphql/tracing/prometheus_trace/graphql_collector.rb
# Backwards-compat: ([!!] I don't think this is related, old version?)
PrometheusTracing::GraphQLCollector = PrometheusTrace::GraphQLCollector
# lib/graphql/tracing/prometheus_trace.rb
module GraphQL
module Tracing
# ...
PrometheusTrace = MonitorTrace.create_module("prometheus")
module PrometheusTrace
# [!!] Is it possible that this doesn't get hit?
if defined?(PrometheusExporter::Server)
autoload :GraphQLCollector, "graphql/tracing/prometheus_trace/graphql_collector"
end
# ......which I don't entirely get how it gets loaded in considering the collector in # lib/prometheus_exporter/server/collector.rb
module PrometheusExporter::Server
class Collector < CollectorBase
def initialize(json_serializer: nil)
# ...
@collectors = {}
register_collector(WebCollector.new)
register_collector(ProcessCollector.new)
register_collector(SidekiqCollector.new)
register_collector(SidekiqQueueCollector.new)
register_collector(SidekiqProcessCollector.new)
register_collector(SidekiqStatsCollector.new)
register_collector(DelayedJobCollector.new)
register_collector(PumaCollector.new)
register_collector(HutchCollector.new)
register_collector(UnicornCollector.new)
register_collector(ActiveRecordCollector.new)
register_collector(ShoryukenCollector.new)
register_collector(ResqueCollector.new)
register_collector(GoodJobCollector.new)
end
def register_collector(collector)
@collectors[collector.type] = collector
end |
|
Hey, sorry for the trouble! I have never actually used Prometheus so I don't know a lot about how its supposed to work, but I wonder if this isn't part of the puzzle: graphql-ruby/lib/graphql/tracing/prometheus_trace.rb Lines 18 to 28 in ddf2550 That would define the necessary collector, but I'm not sure how its supposed to get passed to The docs for If that helps, then it should definitely be added to the GraphQL-Ruby docs! |
|
Yeah it's a fascinating little issue between the libraries. Once I narrow it down and find a solid way to reproduce it I'll try and get a docs patch in too. They do have an auto-load system over on that side using if custom_type_collectors_filenames.length > 0
custom_type_collectors_filenames.each do |t|
require File.expand_path(t)
end
ObjectSpace.each_object(Class) do |klass|
if klass < PrometheusExporter::Server::TypeCollector
options[:type_collectors] ||= []
options[:type_collectors] << klass
end
end
end...and I believe I saw a similar invocation in our code for runtime. If that's the case it's 100% somewhere on our side as far as the load order issue, and likely something to handle more explicitly in the PrometheusExporter code. I can add a docs patch some time later for it as well, or see if there's a patch we want to put into this code to ensure it's registered at runtime. |
Ah, yes! That would do it. I checked Are you able to |
Describe the bug
PrometheusExporter raises an exception on what appears to be a malformed request from GraphQL Ruby:
...wherein the handler parses it as such, comments prefixed with
[!!]are mine:So the error exists between the two gems somehow. I've filed a PR against PrometheusExporter to try and handle that edge case:
discourse/prometheus_exporter#337
...but wonder if there's something that can be done here as well to mitigate that error.
Versions
graphqlversion: 2.3.2rails(or other framework): 7.1.xother applicable versions (
graphql-batch, etc)Steps to reproduce
Unfortunately this looks to only occur in CodeSpaces, which I do not use, which makes me suspect that this is some form of race condition on loading collectors somehow, but I am not able to validate that at this time.
I'm working on finding another instance of this error that we can delve further into, but most folks now are nuking their environment and rebuilding it to get things going again.
Expected behavior
Sending of a collector metric does not cause an exception.
Actual behavior
Exception is raised on a non-existent metric
Place full backtrace here (if a Ruby exception is involved):
Click to view exception backtrace
Additional context
Add any other context about the problem here.
With these details, we can efficiently hunt down the bug!
The text was updated successfully, but these errors were encountered: