lddtree: support alternative LD_LIBRARY_PATH called AUDITWHEEL_LD_LIBRARY_PATH #597
Conversation
Be explicit in the test-case matrix when to set env variable. This enables to check if environment variable is actually correctly reset. This commit passes unit tests.
Add a failing test case confiring that environment is not being correctly reset.
xnox
changed the title
It seems that in addition to reloading lddtree, reload of auditwheel.wheel_abi is needed to full flush and reset analyze_wheel_abi() lru_caches.
…RARY_PATH Sometimes wheels are different only just by shared libraries that are vendored in. Sometimes the wheel being repaired and the libraries that need to be vendored in are accessible on the host that was not used for building the wheel. In such cases, it could be the case that shared libraries used by python that is executing the auditwheel script, are incompatible (too old or too new) than those that need to be vendored. In such cases, LD_LIBRARY_PATH is not convenient to use, as the python interpreter for the auditwheel script is crashing. Add an AUDITWHEEL_LD_LIBRARY_PATH whichis used by lddtree, but otherwise does not affect the python interpreter executing auditwheel script. This helps vendoring in older libraries from an alternative path, into a wheel built against an older python, whilst otherwise running auditwheel repair on a modern system with a much newer python. This is particularly useful when stripping a wheel of vendored libraries, and repair it again to update the shared library dependencies coming from an unpacked chroot. Separately it also helps creating alternative wheels with shared libraries rebuilt with higher march settings. Given that python upstream doesn't support loading optimised libraries as has been implemented and supported in Intel ClearLinux Python which allows one to have alternative march setting shared library deps.
Codecov ReportAll modified and coverable lines are covered by tests ✅
Additional details and impacted files@@ Coverage Diff @@
## main #597 +/- ##
==========================================
- Coverage 92.82% 92.77% -0.06%
==========================================
Files 21 21
Lines 1771 1771
Branches 333 333
==========================================
- Hits 1644 1643 -1
Misses 77 77
- Partials 50 51 +1 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
Cherrypick: - pypa/auditwheel#597 Which adds support for alternative LD_LIBRARY_PATH such that one can link wheels against 2.28 abi, whilst the system otherwise uses higher abi libraries. A pseudo sysroot.
|
I wonder (out loud and will try soon myself if I remember), if #598 would be solved by this PR.. |
it would help indeed. Because /usr/bin/python is linked against libexpat, and if you want a different mangled libexpat in your wheel, indeed currently there is no way to achieve repair with auditwheel, which this PR would enable to specify alternative library path for repair, that doesn't affect the libraries used to execute python that is running auditwheel. |
|
Ah that's great information @xnox, thanks! Then I'll have a closer look at your PR and report back if I made it work with my packaging problems. |
my usecase is different however; i have multiple sets of shared libraries compiled; and i need to create different wheels for each set of libraries. As this way i can create multiple wheels, quickly, without rebuilding them. But packing in updated libraries, and those built with otherwise different properties / platforms. |
|
Sorry, don't want to spam your PR comments, but one thing seems odd: the verbose |
it is correct that 2_5 image is missing libexpat; as that is not allowed dependency (not in the whitelist) and likely those older systems do not have libexpat or build python without libexpat. Thus if a wheel wants to use libexpat; for higher manylinux platforms it can assume for it to be there, but on older ones it has to ensure it installs it somehow (asks the wheel builder to install headers and runtime libraries using their system package manager, or somehow fetch and self-build a copy of expat). 2_28 is different, if the whitelisted libexpat is good enough one can use that. If a forked or higher version is needed, one has to compile it or install pre-built one, change its soname, and revendor it. And when doing so, care has to be taken to keep /usr/bin/python executing with libexpat it needs (nothing or LD_LIBRARY_PATH), but the higher/incompatible/forked libexpat is vendored into the wheel from the appropriate location such as AUDITWHEEL_LD_LIBRARY_PATH introduced here. Note I am not well versed in such ancient systems as manylinux 2_5 which is no longer supported. Only 2_17 and up, it might be worth removing such policy from auditwheel now. |
|
OK I see, thanks!
The thing is that in our case we don't have multiple Though I'd strongly assume that whoever builds other e.g. I'm not yet sure this PR will really fix my specific packaging problem. I mostly read some of the PR description when asking, hadn't looked yet at the code. But you definitely gave a lot of good hints how things work under the hood! I'll just go and try a few things:) |
test_analyze_wheel_abi: refactor test with explicit env
Be explicit in the test-case matrix when to set env variable. This
enables to check if environment variable is actually correctly reset.
This commit passes unit tests.
test_analyze_wheel_abi: add failing test case
Add a failing test case confiring that environment is not being
correctly reset.
test_analyze_wheel_abi: reload and reimport wheel_abi
It seems that in addition to reloading lddtree, reload of
auditwheel.wheel_abi is needed to full flush and reset
analyze_wheel_abi() lru_caches.
lddtree: support alternative LD_LIBRARY_PATH called AUDITWHEEL_LD_LIBRARY_PATH
Sometimes wheels are different only just by shared libraries that are
vendored in. Sometimes the wheel being repaired and the libraries that
need to be vendored in are accessible on the host that was not used
for building the wheel. In such cases, it could be the case that
shared libraries used by python that is executing the auditwheel
script, are incompatible (too old or too new) than those that need to
be vendored.
In such cases, LD_LIBRARY_PATH is not convenient to use, as the python
interpreter for the auditwheel script is crashing.
Add an AUDITWHEEL_LD_LIBRARY_PATH whichis used by lddtree, but
otherwise does not affect the python interpreter executing auditwheel
script.
This helps vendoring in older libraries from an alternative path, into
a wheel built against an older python, whilst otherwise running
auditwheel repair on a modern system with a much newer python. This is
particularly useful when stripping a wheel of vendored libraries, and
repair it again to update the shared library dependencies coming from
an unpacked chroot.
Separately it also helps creating alternative wheels with shared
libraries rebuilt with higher march settings. Given that python
upstream doesn't support loading optimised libraries as has been
implemented and supported in Intel ClearLinux Python which allows one
to have alternative march setting shared library deps.
This fixes bug in test fixtures, all commits pass pre-commit, and all self-service workflows pass in my own repository xnox#3