Skip to content

Conversation

timsaucer
Copy link
Contributor

@timsaucer timsaucer commented Sep 10, 2025

Which issue does this PR close?

Closes #15342

Rationale for this change

Reduces the duplicate dependencies.

We currently depend on bzip2 in two different ways. In attempting to reduce this, I needed to update async-compression which caused two different libraries to link to the system lzma library. This is not allowed in rust.

This PR updates avro-rs, but we cannot merge this PR until that crate merges apache/avro-rs#284 and we remove the crates.io patch this PR contains.

What changes are included in this PR?

Update avro-rs and switch from the unmaintained xz2 crate to liblzma.

Are these changes tested?

Unit tests.

Are there any user-facing changes?

None. This is simply a dependency update to a more recent crate.

@github-actions github-actions bot added core Core DataFusion crate datasource Changes to the datasource crate labels Sep 10, 2025
@alamb
Copy link
Contributor

alamb commented Sep 10, 2025

btw there is a bunch of work upstream to make an arrow-avro crate which would remove the dependency on avro

@findepi
Copy link
Member

findepi commented Sep 11, 2025

is this "fixes #15342" ?

@timsaucer
Copy link
Contributor Author

Thank you @findepi . I should have searched the issues to find it reported.

The upstream PR is merged, so once it's next release comes out I can update this PR and mark it ready.

@timsaucer
Copy link
Contributor Author

It looks like upstream is on a 2 month release cadence, so we'll see if that lands first or the arrow-avro

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
core Core DataFusion crate datasource Changes to the datasource crate
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Dependency conflict with rquest due to async-compression and xz2 linking to lzma
3 participants