Skip to content

fix: Issue 1497 generate partitions custom query #1500

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

sundar-mudupalli-work
Copy link
Collaborator

@sundar-mudupalli-work sundar-mudupalli-work commented Apr 11, 2025

Description of changes

These changes fix a number of issues, standardize the generate partitions tests across the different engines and have some minor cleanup.

Generate Partition tests
In order to not break prior builds, a new table pso_data_validator.test_generate_partitions_v2 was created on all engines. In nearly every engine a composite key consisting of string, integer, timestamp, date and boolean value is used to generate partitions. There are issues with engines including Impala and MS SQL Server, MySQL, Oracle, and Teradata

Cleanup

  • moved the function to generate SQL from Ibis tables into util.py
  • Replaced usage of ibis TableExpr type with Table type - they are synonymous and the latter is documented.

Issues to be closed

1497 Generate Partitions With (complex) custom query
1183 generate table partitions replaces whitespace in primary keys with a single space.
923 generate-table-partitions generates invalid SQL in filter clause when a timestamp field is part of the primary-key

Checklist

  • I have followed the CONTRIBUTING Guide.
  • I have commented my code, particularly in hard-to-understand areas
  • I have updated any relevant documentation to reflect my changes, if applicable
  • I have added unit and/or integration tests relevant to my change as needed
  • I have already checked locally that all unit tests and linting are passing (use the tests/local_check.sh script)
  • I have manually executed end-to-end testing (E2E) with the affected databases/engines

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

Copy link
Contributor

@nj1973 nj1973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly nit pick comments.

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

@sundar-mudupalli-work
Copy link
Collaborator Author

/gcbrun

Copy link
Contributor

@nj1973 nj1973 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM.

@sundar-mudupalli-work sundar-mudupalli-work merged commit 4e0e890 into develop Apr 16, 2025
5 checks passed
@sundar-mudupalli-work sundar-mudupalli-work deleted the issue-1497-generate-partitions-custom-query branch April 16, 2025 13:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants