Introduce have_reported_error matcher #2849

skatkov · 2025-06-04T22:53:33Z

rspec-rails is missing support for Rails ErrorReporter, this was introduced to rails in v7 and has been evolving ever since. With my client, we have moved to using ErrorReporter as a unified error reporting interface, so we can easily move from one error tracking software to another with minimal code changes. And we had a need to test this interface with rspec, so we implemented our own matcher to handle this.

I'm suggesting our internal implementation as is. This is probably not suitable as is for this gem, but I'd like to open discussion with this starting point.

Example usage

@example Checking for any error
  expect { Rails.error.report(StandardError.new) }.to have_reported_error

@example Checking for specific error class
  expect { Rails.error.report(MyError.new) }.to have_reported_error(MyError)

@example Checking for specific error instance with message
  expect { Rails.error.report(MyError.new("message")) }.to have_reported_error(MyError.new("message"))

@example Checking error attributes
  expect { Rails.error.report(StandardError.new, context: "test") }.to have_reported_error.with_context(section: "test")

@example Checking error message patterns
  expect { Rails.error.report(StandardError.new("test message")) }.to have_reported_error(/test/)

@example Negation
  expect { "safe code" }.not_to have_reported_error

pirj

Thank you!

spec/rspec/rails/matchers/have_reported_error_spec.rb

pirj · 2025-06-07T21:46:38Z

features/matchers/README.md

+
+```ruby
+# passes if Rails.error.report was called with any error
+expect { Rails.error.report(StandardError.new) }.to have_reported_error


I think this file is supposed to contain just the outline of the matchers defined here (and at a glance it’s rather incomplete). What do you think of keeping just one example of usage of this matcher? The rest will be taken into the same documentation from the feature file.

This is very verbose for us... These lines take up half this file when our primary documentation is the feature file itself...

features/matchers/README.md

lib/rspec/rails/matchers/have_reported_error.rb

spec/rspec/rails/matchers/have_reported_error_spec.rb

features/matchers/README.md

features/matchers/have_reported_error_matcher.feature

JonRowe · 2025-06-25T17:14:51Z

features/matchers/have_reported_error_matcher.feature

+  Scenario: Using in controller specs
+    Given a file named "spec/controllers/users_controller_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe UsersController, type: :controller do
+        describe "POST #create" do
+          it "reports validation errors" do
+            expect {
+              post :create, params: { user: { email: "invalid" } }
+            }.to have_reported_error(ValidationError)
+          end
+        end
+      end
+      """
+    When I run `rspec spec/controllers/users_controller_spec.rb`
+    Then the examples should all pass
+
+  Scenario: Using in request specs
+    Given a file named "spec/requests/users_spec.rb" with:
+      """ruby
+      require "rails_helper"
+
+      RSpec.describe "Users", type: :request do
+        describe "POST /users" do
+          it "reports processing errors" do
+            expect {
+              post "/users", params: { user: { name: "Test" } }
+            }.to have_reported_error.with(context: "user_creation")
+          end
+        end
+      end
+      """
+    When I run `rspec spec/requests/users_spec.rb`
+    Then the examples should all pass
+


This isn't a controller / request specific matcher so these feel out of place, why these, why not every aspect of Rails... I'd just cut them.

spec/rspec/rails/matchers/have_reported_error_spec.rb

JonRowe · 2025-06-25T17:25:27Z

👋 I think this would be a great addition, sorry for the size of the review its been on my todo list for a while, its mostly just grammar / wording tweaks plus a few "fit" things.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

skatkov · 2025-06-27T11:45:47Z

@JonRowe thanks for review.

The one change I do want to see though is dropping instance matching, I don't think it makes sense over just providing class / message / using with.

Which of these you would rather go with?
.have_reported_error(StandardError, /wrong attributes/).with(param1: 'test')

or

.have_reported_error(StandardError).with(message:/wrong attributes/, params1: 'test1')

??

I assume, that first option is prefered to keep similarity with raised_error matcher.
https://rspec.info/features/3-13/rspec-expectations/built-in-matchers/raise-error/

JonRowe · 2025-06-27T11:59:45Z

Given that you already have with, I think:

.have_reported_error(StandardError).with(/wrong attributes/, params1: 'test1')

But if you insisted on message: in with I could be persuaded

pirj · 2025-06-27T14:31:21Z

It makes sense to separate matching of args passed to report and exception’s own args, including the message. Both can receive args, right? It would open a way for ambiguity and confusion.
raise_error accepts have_attributes as an argument matcher. It supports with_message. Wdyt if we reserve with for additional args passed to report, but keep it separate from with_message?

pirj · 2025-08-31T17:42:15Z

lib/rspec/rails/matchers/have_reported_error.rb

+            @expected_error = nil
+            @expected_message = expected_message


If the first argument was not passed at all and defaulted to UNDEFINED, the second, expected_message will also be nil?

Suggested change

@expected_error = nil

@expected_message = expected_message

@expected_error = @expected_message = nil

pirj · 2025-08-31T17:53:08Z

spec/rspec/rails/matchers/have_reported_error_spec.rb

+        Rails.error.report(StandardError.new("test"), context: { user_id: 123, context: "test" })
+      }.to have_reported_error.with_context(user_id: 123, context: "test")


Nested context strikes back.

pirj · 2025-08-31T17:55:14Z

spec/rspec/rails/matchers/have_reported_error_spec.rb

+    it "passes with partial attribute matching" do
+      expect {
+        Rails.error.report(
+          StandardError.new("test"), context: { user_id: 123, context: "test", extra: "data" }


pirj · 2025-08-31T17:58:49Z

spec/rspec/rails/matchers/have_reported_error_spec.rb

+        expect {
+          Rails.error.report(StandardError.new("actual message"))
+        }.to have_reported_error("expected message")
+      }.to fail_with(/Expected error message to be 'expected message', but got: StandardError/)


Is that all the failure message? Does it include the actual exception message? Or is it just skipped for brevity in this expectation?

features/matchers/README.md

pirj · 2025-08-31T18:31:41Z

features/matchers/have_reported_error_matcher.feature

+        end
+
+        def self.process_with_context
+          Rails.error.report(ArgumentError.new("Invalid input"), severity: :error , context: { topic: "user_processing"})


Suggested change

Rails.error.report(ArgumentError.new("Invalid input"), severity: :error , context: { topic: "user_processing"})

Rails.error.report(ArgumentError.new("Invalid input"), severity: :error, context: { topic: "user_processing" })

pirj · 2025-08-31T18:35:55Z

lib/rspec/rails/matchers/have_reported_error.rb

+        private
+
+        def error_matches_expectation?
+          return true if @expected_error.nil? && @expected_message.nil? && @reports.count.positive?


Suggested change

return true if @expected_error.nil? && @expected_message.nil? && @reports.count.positive?

return true if @expected_error.nil? && @expected_message.nil?

pirj · 2025-08-31T18:39:05Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def failure_message_when_negated
+          warn_about_negated_with_qualifiers! if has_qualifiers?


⚠️ This is a bad place to warn.
There might be no failure due to a false positive.
Please move to does_not_match? like here.

pirj · 2025-08-31T18:48:04Z

lib/rspec/rails/matchers/have_reported_error.rb

+          @reports.find do |report|
+            error_class_matches?(report.error) && error_message_matches?(report.error)
+          end


What if there were two errors reported, same class, same message, but the attributes (context) only match on the second one, not the first one that find finds?

There is some repetition between this method and those lines.

pirj · 2025-08-31T18:55:31Z

lib/rspec/rails/matchers/have_reported_error.rb

+          @attributes.all? do |key, value|
+            actual_value = actual[key] || actual[key.to_s] || actual[key.to_sym]
+            if value.respond_to?(:matches?)
+              value.matches?(actual_value)
+            else
+              actual_value == value
+            end
+          end


We have this, should fit nicely here:

Suggested change

@attributes.all? do |key, value|

actual_value = actual[key] || actual[key.to_s] || actual[key.to_sym]

if value.respond_to?(:matches?)

value.matches?(actual_value)

else

actual_value == value

end

end

values_match?(@attributes, actual)

There is a difference, though - we don't coerce keys to strings. I'm more inclined to be strict here and don't coerce unless you have a good argument for being indifferent to keys' type.

pirj · 2025-08-31T18:58:15Z

lib/rspec/rails/matchers/have_reported_error.rb

+        def unmatched_attributes(actual)
+          @attributes.reject do |key, value|
+            actual_value = actual[key] || actual[key.to_s] || actual[key.to_sym]
+            if value.respond_to?(:matches?)
+              value.matches?(actual_value)
+            else
+              actual_value == value
+            end
+          end
+        end


Maybe just report the entire actual context for simplicity, not the diff?

pirj · 2025-08-31T19:01:24Z

I apologise for some code review notes that felt off - I had an outdated commit open in one of the tabs 🙏

pirj · 2025-08-31T19:05:38Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def failure_message
+          if [email protected]? && [email protected]?


Suggested change

if !@reports.empty? && !@attributes.empty?

if @reports.any? && @attributes.any?

pirj · 2025-08-31T19:07:35Z

lib/rspec/rails/matchers/have_reported_error.rb

+
+        def failure_message
+          if [email protected]? && [email protected]?
+            report_context = @reports.last.context


last is incorrect if several errors could have been reported.

If there were no errors reported matching by class/message, even if the expectation has a with_context qualifier, don't even mention context attributes in the failure message.

But if some reports did match by class/message (or ALL if the expectation did not specify class/message/regexp), list those errors with their attributes.

matching_reports (plural) should do better

pirj · 2025-08-31T19:17:00Z

lib/rspec/rails/matchers/have_reported_error.rb

+        end
+
+        def actual_error
+          @actual_error ||= matching_report&.error


Suggested change

@actual_error ||= matching_report&.error

matching_report&.error

Memoization feels redundant

pirj · 2025-08-31T19:26:49Z

lib/rspec/rails/matchers/have_reported_error.rb

+            end
+          else
+            if @expected_error && !actual_error.is_a?(@expected_error)
+              return "Expected error to be an instance of #{@expected_error}, but got #{actual_error.class} with message: '#{actual_error.message}'"


How does this work?
actual_error is matching_report&.error where matching_report is:

@reports.find do |report| error_class_matches?(report.error) && error_message_matches?(report.error)

and error_class_matches? is:

@expected_error.nil? || error.is_a?(@expected_error)

Above, we check that @expected_error is not nil.

How on Earth can actual_error not be of @expected_error class?

Reduced:

actual_error.is_a?(@expected_error) && !actual_error.is_a?(@expected_error)

How does that spec pass?

Co-authored-by: Phil Pirozhkov <[email protected]>

skatkov force-pushed the have-reported-error branch from 8b837a8 to d2d0e51 Compare June 7, 2025 20:44

pirj reviewed Jun 7, 2025

View reviewed changes

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Show resolved Hide resolved

pirj reviewed Jun 21, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

skatkov force-pushed the have-reported-error branch 2 times, most recently from b38ff1c to 87ebd2a Compare June 22, 2025 20:50

skatkov added 12 commits June 22, 2025 22:59

first pass at have_reported_error matcher

5e6efcc

Adding a feature

5dd2c5c

simplify matcher/readmer.md

ce1a162

Remove mocks out of tests

550c25e

test failure messages better

e074991

rewrite specs

1fe7e91

should require a block matcher

6b41d2b

Create ErrorEvent struct to store errors

3859b04

simplify matcher

8352668

disable matcher chaining

030c27a

improved argument error message in case block is not passed to matcher

2b9bd02

further clean-up

6135d5d

skatkov force-pushed the have-reported-error branch from 71685ad to 6135d5d Compare June 22, 2025 20:59

pirj reviewed Jun 23, 2025

View reviewed changes

lib/rspec/rails/matchers/have_reported_error.rb Outdated Show resolved Hide resolved

JonRowe requested changes Jun 25, 2025

View reviewed changes

skatkov and others added 2 commits June 25, 2025 23:21

Apply suggestions from code review

8335664

Commit grammar improvements Co-authored-by: Jon Rowe <[email protected]>

clean-up features

a14f5c7

pirj and others added 5 commits August 8, 2025 19:43

Update features/matchers/README.md

8b89342

Define supports_value_expectations? to false

3f54ca2

remove log/development.log

9eba59b

Don't use default values in HaveReportedError initializer

139c055

Don't allow to redefine attributes with chaining in with_context method

6a14230

skatkov force-pushed the have-reported-error branch 3 times, most recently from 605ab75 to bd4e048 Compare August 31, 2025 11:37

pirj reviewed Aug 31, 2025

View reviewed changes

features/matchers/README.md Outdated Show resolved Hide resolved

pirj reviewed Aug 31, 2025

View reviewed changes

Stanislav (Stas) Katkov and others added 5 commits September 1, 2025 15:39

adjust .feature file to properly illustrate report usage

f01a731

handle hegated matchers safely, not allowing any attributes

d298e7f

Remove from features/matchers/README.md

5d9f34b

Co-authored-by: Phil Pirozhkov <[email protected]>

cleaning up a class

90ff6af

remove context attribute in specs

b5568a0

skatkov force-pushed the have-reported-error branch from ee09016 to b5568a0 Compare September 1, 2025 13:39

	@expected_error = nil
	@expected_message = expected_message
	@expected_error = @expected_message = nil

		Rails.error.report(StandardError.new("test"), context: { user_id: 123, context: "test" })
		}.to have_reported_error.with_context(user_id: 123, context: "test")

	Rails.error.report(ArgumentError.new("Invalid input"), severity: :error , context: { topic: "user_processing"})
	Rails.error.report(ArgumentError.new("Invalid input"), severity: :error, context: { topic: "user_processing" })

	return true if @expected_error.nil? && @expected_message.nil? && @reports.count.positive?
	return true if @expected_error.nil? && @expected_message.nil?

	if !@reports.empty? && !@attributes.empty?
	if @reports.any? && @attributes.any?

	@actual_error \|\|= matching_report&.error
	matching_report&.error

Uh oh!

Introduce have_reported_error matcher #2849

Are you sure you want to change the base?

Introduce have_reported_error matcher #2849

Uh oh!

Conversation

skatkov commented Jun 4, 2025 • edited by pirj Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Example usage

Uh oh!

pirj left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

JonRowe commented Jun 25, 2025

Uh oh!

skatkov commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JonRowe commented Jun 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

pirj commented Jun 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pirj Aug 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

pirj commented Aug 31, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

skatkov commented Jun 4, 2025 •

edited by pirj

Loading

skatkov commented Jun 27, 2025 •

edited

Loading

JonRowe commented Jun 27, 2025 •

edited

Loading

pirj Aug 31, 2025 •

edited

Loading