Skip to content

Add model conditions to equality test #1001

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: main
Choose a base branch
from

Conversation

james-johnston-thumbtack

resolves #704

Problem

In #704, it's expressed that it would be useful to limit the rows compared in the equality test. The issue was closed due to inactivity, but the discussion seems to indicate it was a reasonable request, and the only reason the issue was closed was because nobody got around to writing a PR:

I guess we could also add a compare_model_condition parameter to this test, to accompany the where param. I'd be happy to review a PR in that vein if you wanted to open one.

Although we are closing this issue as stale, it's not gone forever. Issues can be reopened if there is renewed community interest. Just add a comment to notify the maintainers.

Solution

This adds the following parameters to generic_tests/equality.sql:

  • model_condition
  • compare_model_condition

It allows one to filter out rows from either model before comparing the remaining rows in the models as usual.

The rationale and usage is very similar to the relationships_where test. For example, you can exclude recently-modified rows which might not be fully updated in the other table due to e.g. ELT limitations, etc.

Note that model_condition is introduced, because the normal 'where' clause can't be used because it means the given "model" is no longer a relation, and so the dbt_utils._is_relation check will fail. This was discussed at #704 (comment)

Also note that by using two parameters, the user has full control over the filtering, and can choose different filters for each table.

Checklist

  • This code is associated with an issue which has been triaged and accepted for development.
  • I have read the contributing guide and understand what's expected of me
  • I have run this code in development and it appears to resolve the stated issue
  • This PR includes tests, or tests are not required/relevant for this PR
  • I have updated the README.md (if applicable)

NOTE: i tested by writing integration tests, and testing using the postgres target.

This adds the following parameters to generic_tests/equality.sql:

- model_condition
- compare_model_condition

It allows one to filter out rows from either model before comparing the
remaining rows in the models as usual.

The rationale and usage is very similar to the relationships_where test.

Note that model_condition is introduced, because the normal 'where'
clause can't be used because it means the given "model" is no longer a
relation, and so the dbt_utils._is_relation check will fail.
@joellabes
Copy link
Contributor

@james-johnston-thumbtack a quick message to say I see all the stuff you've done here! We will check this out but probably a couple of weeks away sorry 😬

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

equality test with where config
2 participants