Feature: migrate to uv #442

marcellodesales · 2025-10-09T20:53:52Z

Note

This is just in case there are plans to use uv, but I've been playing with nodestream locally with uv

🎉 New Feature

Using uv to improve test performance

🔊 Migration Logs

Moved from poetry to uv
Updating Makefile commands
- TODO: propose dockerized builds to isolate support

uvx migrate-to-uv --package-manager poetry
Locking dependencies with "uv lock"...
Using CPython 3.10.16
Resolved 84 packages in 1.96s
Locking dependencies with "uv lock" again to remove constraints...
Using CPython 3.10.16
Resolved 84 packages in 54ms
Successfully migrated project from Poetry to uv!

✅ Test cases

uv run pytest -m "not e2e"
========================================================================================= test session starts ==========================================================================================
platform darwin -- Python 3.10.16, pytest-8.4.0, pluggy-1.6.0 -- /Users/mdesales/dev/github.com/marcellodesales/nodestream/.venv/bin/python3
cachedir: .pytest_cache
rootdir: /Users/mdesales/dev/github.com/marcellodesales/nodestream
configfile: pytest.ini
testpaths: tests
plugins: snapshot-0.9.0, anyio-4.9.0, httpx-0.30.0, mock-3.14.1, cov-6.1.1, asyncio-1.0.0
asyncio: mode=strict, asyncio_default_fixture_loop_scope=None, asyncio_default_test_loop_scope=function
collected 2435 items                                                                                                                                                                                   

tests/data/test_snapshot_handling_during_errors.py::test_snapshot_handling_during_errors PASSED                                                                                                  [  0%]
tests/integration/test_interpreter_schema_generation.py::test_schema_introspection PASSED                                                                                                        [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_interpretation_snapshot[fifa_2021_player_data.yaml] PASSED                                                             [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_interpretation_snapshot[airports.yaml] PASSED                                                                          [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_interpretation_snapshot[source_match_only.yaml] PASSED                                                                 [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_interpretation_snapshot[source_eager.yaml] PASSED                                                                      [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[fifa_2021_player_data.yaml-plain] PASSED                                                              [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[fifa_2021_player_data.yaml-graphql] PASSED                                                            [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[airports.yaml-plain] PASSED                                                                           [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[airports.yaml-graphql] PASSED                                                                         [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[people.yaml-plain] PASSED                                                                             [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[people.yaml-graphql] PASSED                                                                           [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[dns.yaml-plain] PASSED                                                                                [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[dns.yaml-graphql] PASSED                                                                              [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[multiple_passes.yaml-plain] PASSED                                                                    [  0%]
tests/integration/test_pipeline_and_data_interpretation.py::test_pipeline_schema_inference[multiple_passes.yaml-graphql] PASSED                                                                  [  0%]
tests/integration/test_pipeline_flush_handling.py::test_flush_handling PASSED                                                                                                                    [  0%]
tests/unit/cli/commands/test_copy.py::test_get_target_from_user_from_option PASSED                                                                                                               [  0%]
tests/unit/cli/commands/test_copy.py::test_get_target_from_user_from_prompt PASSED                                                                                                               [  0%]
tests/unit/cli/commands/test_copy.py::test_get_target_from_user_from_option_unknown_target PASSED                                                                                                [  0%]
tests/unit/cli/commands/test_copy.py::test_get_type_selection_from_user_from_option PASSED                                                                                                       [  0%]
tests/unit/cli/commands/test_copy.py::test_get_type_selection_from_user_from_prompt PASSED                                                                                                       [  0%]
tests/unit/cli/commands/test_copy.py::test_get_type_selection_from_user_from_all_flag PASSED                                                                                                     [  0%]
tests/unit/cli/commands/test_copy.py::test_get_type_selection_from_user_from_option_unknown_type PASSED                                                                                          [  0%]
tests/unit/cli/commands/test_copy.py::test_handle_async_unknown_target_error PASSED                                                                                                              [  1%]
tests/unit/cli/commands/test_copy.py::test_handle_async PASSED                                                                                                                                   [  1%]
tests/unit/cli/commands/test_make_migrations.py::test_handle_a
...


tests/unit/schema/migrations/test_operations.py::test_create_node_type_as_node_type PASSED                                                                                                       [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_relationship_type_as_relationship_type PASSED                                                                                       [ 94%]
tests/unit/schema/migrations/test_operations.py::test_node_key_part_renamed_proposed_index_name PASSED                                                                                           [ 94%]
tests/unit/schema/migrations/test_operations.py::test_node_key_extended_proposed_index_name PASSED                                                                                               [ 94%]
tests/unit/schema/migrations/test_operations.py::test_drop_rel_index_proposed_index_name PASSED                                                                                                  [ 94%]
tests/unit/schema/migrations/test_operations.py::test_drop_additional_node_property_index_name PASSED                                                                                            [ 94%]
tests/unit/schema/migrations/test_operations.py::test_rename_node_type_old_name PASSED                                                                                                           [ 94%]
tests/unit/schema/migrations/test_operations.py::test_rename_node_type_new_name PASSED                                                                                                           [ 94%]
tests/unit/schema/migrations/test_operations.py::test_drop_node_type_index_name PASSED                                                                                                           [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_node_type_reduce PASSED                                                                                                             [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_node_type_reduce_no_match PASSED                                                                                                    [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_node_type_reduce_irrelevant_type PASSED                                                                                             [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_relationship_type_reduce PASSED                                                                                                     [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_relationship_type_reduce_no_match PASSED                                                                                            [ 94%]
tests/unit/schema/migrations/test_operations.py::test_create_relationship_type_reduce_irrelevant_type PASSED                                                                                     [ 94%]
tests/unit/schema/migrations/test_operations.py::test_drop_node_type_reduce PASSED                                                                                                               [ 95%]
tests/unit/schema/migrations/test_operations.py::test_drop_node,,,

,,,



    @pytest.mark.asyncio
    async def test_warn_policy(mocker):
        context = mocker.Mock()
        context.object_store.get_pickled.return_value = None
>       subject = SchemaEnforcer(
            inference_sample_size=0, enforcement_policy="warn"
        )

tests/unit/pipeline/test_filters.py:213: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <nodestream.pipeline.filters.SchemaEnforcer object at 0x1284d0a60>, kwargs = {'enforcement_policy': 'warn', 'inference_sample_size': 0}

    def __init__(self, **kwargs):
        super().__init__(**kwargs)
>       raise ImportError(
            "SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra."
        )
E       ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.

nodestream/pipeline/filters.py:350: ImportError
------------------------------------------------ generated xml file: /Users/mdesales/dev/github.com/marcellodesales/nodestream/build/reports/junit.xml -------------------------------------------------
============================================================================================ tests coverage ============================================================================================
__________________________________________________________________________ coverage: platform darwin, python 3.10.16-final-0 ___________________________________________________________________________

Name                                                                          Stmts   Miss  Cover   Missing
-----------------------------------------------------------------------------------------------------------
nodestream/__init__.py                                                            0      0   100%
nodestream/cli/__init__.py                                                        3      0   100%
nodestream/cli/application.py                                                    25     14    44%   15, 19-25, 29-38
nodestream/cli/commands/__init__.py                                              14      0   100%
nodestream/cli/commands/audit_command.py                                         18      7    61%   14-15, 19-24
nodestream/cli/commands/copy.py                                                  56      0   100%
nodestream/cli/commands/make_migrations.py                                       14      0   100%
nodestream/cli/commands/new.py                                                   17      0   100%
nodestream/cli/commands/nodestream_command.py                                    40      5    88%   13, 22, 41, 56, 60
nodestream/cli/commands/print_schema.py                                          13      2    85%   36-37
nodestream/cli/commands/remove.py                                                14      0   100%
nodestream/cli/commands/run.py                                                   25      0   100%
nodestream/cli/commands/run_migrations.py                                        20      0   100%
nodestream/cli/commands/scaffold.py                                              18      2    89%   35-36
nodestream/cli/commands/shared_options.py                                        14      0   100%
nodestream/cli/commands/show.py                                                  11      0   100%
nodestream/cli/commands/show_migrations.py                                       36      0   100%
nodestream/cli/commands/squash_migrations.py                                     14      0   100%
nodestream/cli/operations/__init__.py                                            18      0   100%
nodestream/cli/operations/add_pipeline_to_project.py                             20      0   100%
nodestream/cli/operations/commit_project_to_disk.py                              10      0   100%
nodestream/cli/operations/execute_migration.py                                   12      0   100%
nodestream/cli/operations/generate_migration.py                                  36      0   100%
nodestream/cli/operations/generate_pipeline_scaffold.py                          26      0   100%
nodestream/cli/operations/generate_squashed_migration.py                         17      0   100%
nodestream/cli/operations/initialize_logger.py                                   26      0   100%
nodestream/cli/operations/initialize_project.py                                   6      0   100%
nodestream/cli/operations/intitialize_metrics.py                                 17      0   100%
nodestream/cli/operations/operation.py                                            9      0   100%
nodestream/cli/operations/print_project_schema.py                                23      0   100%
nodestream/cli/operations/remove_pipeline_from_project.py                        10      0   100%
nodestream/cli/operations/run_audit.py                                           25      0   100%
nodestream/cli/operations/run_copy.py                                            27      0   100%
nodestream/cli/operations/run_pipeline.py                                       116      6    95%   59-61, 92-94
nodestream/cli/operations/run_project_cookiecutter.py                            14      0   100%
nodestream/cli/operations/show_pipelines.py                                      39      0   100%
nodestream/compat.py                                                             15      0   100%
nodestream/databases/__init__.py                                                  5      0   100%
nodestream/databases/copy.py                                                     37      0   100%
nodestream/databases/database_connector.py                                       30      5    83%   43-46, 49
nodestream/databases/debounced_ingest_strategy.py                                41      0   100%
nodestream/databases/ingest_strategy.py                                          19      1    95%   7
nodestream/databases/null.py                                                     30      0   100%
nodestream/databases/operation_debouncer.py                                      44      0   100%
nodestream/databases/query_executor.py                                           24      0   100%
nodestream/databases/query_executor_with_statistics.py                           51      0   100%
nodestream/databases/writer.py                                                   30      6    80%   19-22, 39-43
nodestream/file_io.py                                                            72      0   100%
nodestream/interpreting/__init__.py                                               3      0   100%
nodestream/interpreting/interpretation_passes.py                                 81      0   100%
nodestream/interpreting/interpretations/__init__.py                               7      0   100%
nodestream/interpreting/interpretations/conditions.py                            81      0   100%
nodestream/interpreting/interpretations/extract_variables_interpretation.py      11      0   100%
nodestream/interpreting/interpretations/interpretation.py                        30      0   100%
nodestream/interpreting/interpretations/properties_interpretation.py             18      0   100%
nodestream/interpreting/interpretations/property_mapping.py                      41      0   100%
nodestream/interpreting/interpretations/relationship_interpretation.py          107      0   100%
nodestream/interpreting/interpretations/source_node_interpretation.py            45      0   100%
nodestream/interpreting/interpretations/switch_interpretation.py                 36      0   100%
nodestream/interpreting/interpreter.py                                           34      0   100%
nodestream/interpreting/record_decomposers.py                                    23      0   100%
nodestream/metrics.py                                                           190      0   100%
nodestream/model/__init__.py                                                      8      0   100%
nodestream/model/creation_rules.py                                               12      0   100%
nodestream/model/desired_ingestion.py                                            64      5    92%   11, 29, 32, 50, 134
nodestream/model/graph_objects.py                                               115      6    95%   12, 77, 136, 175, 178, 200
nodestream/model/ingestion_hooks.py                                              10      0   100%
nodestream/model/ttl.py                                                          18      3    83%   7, 20-21
nodestream/pipeline/__init__.py                                                  12      0   100%
nodestream/pipeline/argument_resolvers/__init__.py                                5      0   100%
nodestream/pipeline/argument_resolvers/argument_resolver.py                      23      1    96%   20
nodestream/pipeline/argument_resolvers/configuration_argument_resolver.py        23      0   100%
nodestream/pipeline/argument_resolvers/environment_variable_resolver.py           6      0   100%
nodestream/pipeline/argument_resolvers/include_file_resolver.py                  10      0   100%
nodestream/pipeline/channel.py                                                   51      0   100%
nodestream/pipeline/class_loader.py                                              45      0   100%
nodestream/pipeline/extractors/__init__.py                                        6      0   100%
nodestream/pipeline/extractors/apis.py                                           28      0   100%
nodestream/pipeline/extractors/credential_utils.py                               41      1    98%   40
nodestream/pipeline/extractors/extractor.py                                      31      0   100%
nodestream/pipeline/extractors/files.py                                         280      1    99%   140
nodestream/pipeline/extractors/iterable.py                                       23      0   100%
nodestream/pipeline/extractors/queues/__init__.py                                 1      0   100%
nodestream/pipeline/extractors/queues/extractor.py                               45      0   100%
nodestream/pipeline/extractors/queues/sqs.py                                     49      3    94%   61-63
nodestream/pipeline/extractors/stores/__init__.py                                 0      0   100%
nodestream/pipeline/extractors/stores/aws/__init__.py                             3      0   100%
nodestream/pipeline/extractors/stores/aws/athena_extractor.py                    89      3    97%   49, 66-67
nodestream/pipeline/extractors/stores/aws/dynamodb_extractor.py                  42      0   100%
nodestream/pipeline/extractors/streams/__init__.py                                3      0   100%
nodestream/pipeline/extractors/streams/extractor.py                              61      5    92%   57-62
nodestream/pipeline/extractors/streams/kafka.py                                  57      0   100%
nodestream/pipeline/extractors/ttls.py                                           11      0   100%
nodestream/pipeline/filters.py                                                  171     97    43%   155-343
nodestream/pipeline/flush.py                                                      1      0   100%
nodestream/pipeline/normalizers/__init__.py                                       6      0   100%
nodestream/pipeline/normalizers/lowercase_strings.py                              5      0   100%
nodestream/pipeline/normalizers/normalizer.py                                    43      0   100%
nodestream/pipeline/normalizers/remove_trailing_dots.py                           5      0   100%
nodestream/pipeline/normalizers/trim_whitespace.py                                5      0   100%
nodestream/pipeline/normalizers/uppercase_strings.py                              5      0   100%
nodestream/pipeline/object_storage.py                                           168      0   100%
nodestream/pipeline/pipeline.py                                                 104      0   100%
nodestream/pipeline/pipeline_file_loader.py                                     100      0   100%
nodestream/pipeline/progress_reporter.py                                         41      3    93%   18, 27-28
nodestream/pipeline/scope_config.py                                              17      0   100%
nodestream/pipeline/step.py                                                      37      0   100%
nodestream/pipeline/transformers/__init__.py                                      4      0   100%
nodestream/pipeline/transformers/expand_json_field.py                            31      0   100%
nodestream/pipeline/transformers/transformer.py                                  84      1    99%   88
nodestream/pipeline/transformers/value_projection.py                             13      0   100%
nodestream/pipeline/value_providers/__init__.py                                  12      0   100%
nodestream/pipeline/value_providers/cast_value_provider.py                       28      2    93%   36-37
nodestream/pipeline/value_providers/context.py                                   12      0   100%
nodestream/pipeline/value_providers/jmespath_value_provider.py                   60      0   100%
nodestream/pipeline/value_providers/mapping_value_provider.py                    30      1    97%   37
nodestream/pipeline/value_providers/normalizer_value_provider.py                 22      0   100%
nodestream/pipeline/value_providers/regex_value_provider.py                      32      0   100%
nodestream/pipeline/value_providers/split_value_provider.py                      23      0   100%
nodestream/pipeline/value_providers/static_value_provider.py                     18      0   100%
nodestream/pipeline/value_providers/string_format_value_provider.py              28      1    96%   35
nodestream/pipeline/value_providers/value_provider.py                            44      0   100%
nodestream/pipeline/value_providers/variable_value_provider.py                   21      1    95%   34
nodestream/pipeline/writers.py                                                   23      1    96%   32
nodestream/pluggable.py                                                          19      0   100%
nodestream/project/__init__.py                                                    6      0   100%
nodestream/project/audits/__init__.py                                             4      0   100%
nodestream/project/audits/audit.py                                               22      2    91%   50-51
nodestream/project/audits/audit_printer.py                                        7      1    86%   11
nodestream/project/audits/audit_ttls.py                                          27      3    89%   41-43
nodestream/project/pipeline_definition.py                                        93      0   100%
nodestream/project/pipeline_scope.py                                             69      2    97%   161-166
nodestream/project/plugin.py                                                     51      0   100%
nodestream/project/project.py                                                   157      7    96%   355, 370-377
nodestream/project/run_request.py                                                18      0   100%
nodestream/project/storage.py                                                    46      0   100%
nodestream/project/target.py                                                     30      0   100%
nodestream/schema/__init__.py                                                     2      0   100%
nodestream/schema/migrations/__init__.py                                          6      0   100%
nodestream/schema/migrations/auto_change_detector.py                            238      2    99%   450, 460
nodestream/schema/migrations/auto_migration_maker.py                             22      0   100%
nodestream/schema/migrations/migrations.py                                       93      0   100%
nodestream/schema/migrations/migrator.py                                         47      0   100%
nodestream/schema/migrations/operations.py                                      295      0   100%
nodestream/schema/migrations/project_migrations.py                               41      0   100%
nodestream/schema/migrations/state_providers.py                                  87      0   100%
nodestream/schema/printers/__init__.py                                            5      0   100%
nodestream/schema/printers/cypheresque.py                                        22      0   100%
nodestream/schema/printers/graphql_schema_printer.py                             45      2    96%   74-75
nodestream/schema/printers/plain_text_schema_printer.py                           5      0   100%
nodestream/schema/printers/schema_printer.py                                     18      1    94%   22
nodestream/schema/state.py                                                      309      0   100%
nodestream/subclass_registry.py                                                  43      3    93%   40, 70, 81
nodestream/utils.py                                                              74      0   100%
-----------------------------------------------------------------------------------------------------------
TOTAL                                                                          6174    206    97%
Coverage HTML written to dir build/reports/html
Coverage XML written to file build/reports/coverage.xml
======================================================================================= short test summary info ========================================================================================
FAILED tests/unit/pipeline/test_filters.py::test_schema_enforcer_with_fetch_schema - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_schema_enforcer_with_infer_schema - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_schema_enforcement_modes - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_infer_mode_schema_already_exists - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_enforce_mode_schema_not_present - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_invalid_enforcement_policy - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
FAILED tests/unit/pipeline/test_filters.py::test_warn_policy - ImportError: SchemaEnforcer requires genson and jsonschema to be installed. Install the `validation` extra.
============================================================================= 7 failed, 2428 passed, 3 warnings in 22.09s ==============================================================================
g

Issuing the regular command to migrate uvx migrate-to-uv --package-manager poetry Locking dependencies with "uv lock"... Using CPython 3.10.16 Resolved 84 packages in 1.96s Locking dependencies with "uv lock" again to remove constraints... Using CPython 3.10.16 Resolved 84 packages in 54ms Successfully migrated project from Poetry to uv!

uv add prometheus-client It was failing because of the missing dependency

Update all commands from Makefile to use uv

Add pytest.ini to set build directories for tests and coverage report so that it's easier for CICD integration.

Adjusting to make sure tests run

zprobst · 2025-10-16T20:08:01Z

@marcellodesales looks like CI failed here due to caching on poetry.lock. Do you mind changing the ci rules to use uv.lock?

marcellodesales added 5 commits October 9, 2025 14:27

🐍 📦 add prometheus-client needed

5b56894

uv add prometheus-client It was failing because of the missing dependency

🐚 ♻️ migrate poetry to uv in Makefile

1465606

Update all commands from Makefile to use uv

🐍 ✅ add pytest.ini

a4f06c1

Add pytest.ini to set build directories for tests and coverage report so that it's easier for CICD integration.

🐍 ✅ adjust test_filters.py uv

7f7885e

Adjusting to make sure tests run

marcellodesales force-pushed the feature/migrate-to-uv branch from fa0b5d9 to 7f7885e Compare October 9, 2025 21:29

zprobst marked this pull request as ready for review October 15, 2025 11:19

zprobst requested review from ccloes and zprobst as code owners October 15, 2025 11:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Feature: migrate to uv #442

Feature: migrate to uv #442

Uh oh!

marcellodesales commented Oct 9, 2025 •

edited

Loading

Uh oh!

zprobst commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Feature: migrate to uv #442

Are you sure you want to change the base?

Feature: migrate to uv #442

Uh oh!

Conversation

marcellodesales commented Oct 9, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎉 New Feature

🔊 Migration Logs

✅ Test cases

Uh oh!

zprobst commented Oct 16, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

marcellodesales commented Oct 9, 2025 •

edited

Loading