Skip to content

Added docs for new compression plugin and created a plugin subfolder #4107

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 4 commits into
base: develop2
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions conf.py
Original file line number Diff line number Diff line change
Expand Up @@ -436,6 +436,8 @@
def copy_legacy_redirects(app, docname): # Sphinx expects two arguments
# FILL in this dicts the necessary redirects
redirect_files = {
"reference/extensions/profile_plugin.html" : "plugins/profile_plugin.html",
"reference/extensions/authorization_plugins.html" : "plugins/authorization_plugins.html",
}

redirect_template = """<!DOCTYPE html>
Expand Down
3 changes: 1 addition & 2 deletions reference/extensions.rst
Original file line number Diff line number Diff line change
Expand Up @@ -46,9 +46,8 @@ Contents:
extensions/python_api
extensions/deployers
extensions/hooks
extensions/plugins
extensions/binary_compatibility
extensions/profile_plugin
extensions/authorization_plugins
extensions/command_wrapper
extensions/package_signing

32 changes: 32 additions & 0 deletions reference/extensions/plugins.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,32 @@
.. _reference_plugins:

Plugins
==========

Conan plugins are a powerful mechanism to customize and extend Conan's built-in
behavior. They are Python scripts loaded dynamically from the user's local
Conan cache, allowing users to intercept and augment specific parts of the
Conan workflow without modifying the Conan source code.

Plugins are ideal for advanced scenarios where teams or organizations need to:

- Enforce custom authorization or access control rules.
- Dynamically modify profiles or settings based on complex logic.
- Customize compression or packaging formats to optimize bandwidth or storage.

Plugins offer a clean, maintainable, and shareable way to implement advanced
behavior, particularly in CI/CD pipelines or large-scale deployments.

They can be distributed and shared using ``conan config install``, making it easy to apply
consistent behavior across all users in a team.

The following types of plugins are currently supported:

.. toctree::
:maxdepth: 2

plugins/authorization_plugins
plugins/profile_plugin
plugins/compression_plugin

Each plugin type has a specific purpose and interface, which is documented in its corresponding section.
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,7 @@
Authorization plugins
=====================

.. include:: ../../common/experimental_warning.inc
.. include:: ../../../common/experimental_warning.inc

Regarding authorization, we have two plugins: one focused on remote :ref:`Conan servers <setting_up_conan_remotes>`
authorization, ``auth_remote.py``, and another focused on authorization for source file servers, ``auth_source.py``.
Expand Down
98 changes: 98 additions & 0 deletions reference/extensions/plugins/compression_plugin.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,98 @@
.. _reference_extensions_compression_plugin:

Compression plugin
------------------

.. include:: ../../../common/experimental_warning.inc

The ``compression.py`` plugin is a Conan extension that allows users to
customize the compression and extraction processes for all files managed by Conan.

To activate it, place the plugin at: ``<conan_home>/extensions/plugins/compression.py``.

This plugin provides flexibility in how Conan packages are compressed and
extracted, making it especially useful in scenarios such as:

- Replacing the default ``gzip`` compression algorithm with a more efficient one like ``zstd`` or ``xz``.
- Embedding custom metadata into the compressed archive.
- Modifying the internal structure of the package content.
- Modifying the file or directory permissions inside the compressed archive.
- Manage symbolic links inside archives.

These capabilities can help organizations reduce bandwidth and storage usage,
or enforce specific packaging policies.

.. important::

Once this plugin is enabled, all operations involving ``conan upload`` and
``conan download`` will use the user-defined compression and extraction functions.
This implies that **all users** within the same organization **must install** the
plugin to avoid incompatibilities during extraction.

You can distribute and synchronize your configuration by packaging it and installing it via ``conan config install``.

Plugin Interface
++++++++++++++++

To implement a custom compression plugin, define the following two functions in ``compression.py``:

.. code-block:: python

def tar_extract(archive_path, dest_dir, conf=None, ref=None, *args, **kwargs) -> None:
...

def tar_compress(archive_path, files, recursive=False, conf=None, ref=None, *args, **kwargs) -> None:
...


- ``archive_path``: Path to the final archive file. This value is immutable.

.. important::

Even if you use a different compression algorithm, the output file must retain
the ``.tgz`` extension. This is required for Conan to correctly handle archives.
Changing the extension will **break** the workflow.

- ``files``: Dictionary of files to be compressed in the form ``{name_in_tar: absolute_path}``.
- ``recursive``: Whether to include subdirectories when adding files.
- ``conf``: Conan configuration object with user-defined options. It can be used to retrieve custom settings, such as compression level in the following way:

.. code-block:: python

compresslevel = conf.get("core.gzip:compresslevel", check_type=int) if conf else None


Also, the ``conf`` object can be used to retrieve other custom configuration options that might be relevant for the compression process.

- ``ref``: Optional Conan reference (e.g., package or recipe reference) useful for logging.



Example: Compression Plugin Using xz
++++++++++++++++++++++++++++++++++++

This example shows how to implement a plugin using the ``xz`` compression format:

.. code-block:: python

import os
import tarfile
from conan.api.output import ConanOutput

def tar_compress(archive_path, files, recursive, conf=None, *args, **kwargs):
name = os.path.basename(archive_path)
ConanOutput().info(f"Compressing {name} using compression plugin (xz)")
compresslevel = conf.get("core.gzip:compresslevel", check_type=int) if conf else None
tar_kwargs = {"preset": compresslevel} if compresslevel else {}
with tarfile.open(archive_path, f"w:xz", **tar_kwargs) as txz:
for filename, abs_path in sorted(files.items()):
txz.add(abs_path, filename, recursive=True)

def tar_extract(archive_path, dest_dir, conf=None, *args, **kwargs):
ConanOutput().info(f"Decompressing {os.path.basename(archive_path)} using compression plugin (xz)")
with open(archive_path, mode='rb') as file_handler:
txz = tarfile.open(fileobj=file_handler)
txz.extraction_filter = (lambda member, path: member)
txz.extractall(path=dest_dir)
txz.close()

Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
.. _reference_extensions_profile_plugin:

Profile plugin
---------------
===============

The ``profile.py`` extension plugin is a Python script that receives one profile and allow
checking and modifying it.
Expand Down