feat: add pyproject.toml support to generate-recipe command #1849

millsks · 2025-08-25T04:36:04Z

🎯 Summary

This PR adds native support for generating conda recipes from pyproject.toml files to the rattler-build generate-recipe command. This integrates the core functionality from pyrattler-recipe-autogen directly into rattler-build, eliminating the need for a separate tool.

🚀 Features Added

New Command

rattler-build generate-recipe pyproject <pyproject.toml> [--output <recipe.yaml>]

Core Functionality

✅ Full pyproject.toml parsing - Extract project metadata, dependencies, and build system requirements
✅ Dependency conversion - Convert Python package dependencies to conda format with proper version constraints
✅ Schema version support - Configurable schema version with YAML language server headers
✅ Conda overrides - Support for tool.conda.recipe.* configuration sections
✅ Entry points handling - Convert project.scripts to conda recipe entry points
✅ Build system integration - Add build system requirements to host dependencies

📝 Example Usage

Input `pyproject.toml`:

[project]
name = "my-package"
version = "1.0.0"
description = "A sample Python package"
dependencies = [
    "requests>=2.25.0",
    "click>=8.0.0"
]

[project.scripts]
my-tool = "my_package.cli:main"

[build-system]
requires = ["setuptools", "wheel"]
build-backend = "setuptools.build_meta"

[tool.conda.recipe]
schema_version = 1

[tool.conda.recipe.about]
license = "MIT"
homepage = "https://github.com/example/my-package"

Generated `recipe.yaml`:

# yaml-language-server: $schema=https://raw.githubusercontent.com/prefix-dev/recipe-format/main/schema.json
schema_version: 1

context:
  name: my-package
  version: "1.0.0"

package:
  name: ${{ name }}
  version: ${{ version }}

source:
  path: .

build:
  script: python -m pip install . -vv --no-deps --no-build-isolation
  entry_points:
    - my-tool = my_package.cli:main

requirements:
  host:
    - python
    - pip
    - setuptools
    - wheel
  run:
    - python
    - requests >=2.25.0
    - click >=8.0.0

about:
  description: A sample Python package
  license: MIT
  homepage: https://github.com/example/my-package

tests:
  - python:
      imports:
        - my_package
      commands:
        - my-tool --help

🔧 Implementation Details

Files Added/Modified

src/recipe_generator/pyproject.rs - Complete pyproject.toml parsing and recipe generation logic
src/recipe_generator/serialize.rs - Added schema_version field support to Recipe struct
src/recipe_generator/luarocks/mod.rs - Integrated pyproject command into CLI
rust-tests/src/lib.rs - Added comprehensive integration tests

Key Components

Dependency Conversion: Converts Python package names and version constraints to conda format
Schema Support: Configurable schema version with proper YAML headers for VS Code integration
Conda Overrides: Full support for tool.conda.recipe.* sections to customize any part of the recipe
Build System Integration: Automatically adds build system requirements to host dependencies

🧪 Testing

Unit Tests (11 tests - all passing)

test_convert_python_to_conda_dependency - Package name conversion logic
test_format_python_constraint - Version constraint formatting
test_build_package_section - Package metadata generation
test_build_requirements_section - Dependencies and build requirements
test_build_about_section - About section with conda overrides
test_build_context_section - Context variables handling
test_build_context_section_dynamic_version - Dynamic version resolution
test_build_schema_version - Schema version configuration
test_resolve_dynamic_version - Dynamic version extraction from files
test_apply_package_name_mapping - Python to conda package mapping
test_format_yaml_with_schema - YAML output with schema headers

Integration Tests (2 tests - all passing)

test_generate_recipe_pyproject_basic - End-to-end basic functionality test
test_generate_recipe_pyproject_with_conda_overrides - Advanced conda overrides test

🎁 Benefits

Unified tooling - No need for separate pyrattler-recipe-autogen tool
Better developer experience - Single command for all recipe generation needs
Consistency - Unified behavior and output format across all recipe types
Performance - Native Rust implementation for better performance
Maintainability - Single codebase to maintain and update

🔗 Related Issues

Closes #1848 - Generate a recipe.yaml from pyproject.toml using generate-recipe subcommand

📋 Checklist

Implementation follows existing code patterns and conventions
Comprehensive unit test coverage (11 tests)
Integration test coverage (2 tests)
All tests passing
Documentation through examples and test cases
Follows Rust best practices and error handling
Schema version support for forward compatibility
Conda override support for advanced use cases

🚦 Testing Instructions

Create a test pyproject.toml file:

cat > test_pyproject.toml << 'EOF'
[project]
name = "test-package"
version = "1.0.0"
dependencies = ["requests>=2.25.0"]

[build-system]
requires = ["setuptools", "wheel"]
EOF

Generate a recipe:

rattler-build generate-recipe pyproject test_pyproject.toml

Verify the output contains proper conda dependencies and schema headers

📚 Additional Notes

The implementation is designed to be extensible for future enhancements
Full backward compatibility with existing recipe generation functionality
Follows the same patterns as the existing luarocks recipe generator
Ready for production use with comprehensive error handling

🚀 Quick Demo

Want to test this feature quickly? Here's a complete example:

# Create a sample pyproject.toml
cat > sample_pyproject.toml << 'EOF'
[project]
name = "demo-package"
version = "0.1.0"
description = "A demo package for testing pyproject recipe generation"
dependencies = [
    "click>=8.0.0",
    "requests>=2.25.0"
]

[project.scripts]
demo-tool = "demo_package.main:cli"

[build-system]
requires = ["setuptools>=45", "wheel"]
build-backend = "setuptools.build_meta"

[tool.conda.recipe.about]
license = "MIT"
license_file = "LICENSE"
EOF

# Generate the conda recipe
rattler-build generate-recipe pyproject --input sample_pyproject.toml --output demo_recipe.yaml

# View the generated recipe
cat demo_recipe.yaml

This will generate a complete conda recipe with:

Proper dependency conversion (click>=8.0.0 → click >=8.0.0)
Build system requirements in host dependencies
Entry points configuration
Schema version and VS Code language server support
Conda-specific metadata from tool.conda.recipe.* sections

- Add new 'pyproject' subcommand to generate-recipe CLI - Integrate core functionality from pyrattler-recipe-autogen - Support dependency extraction from project.dependencies with version constraints - Add tool.conda.recipe.* configuration overrides for all recipe sections - Include schema version support with default schema header - Set default output to recipe/recipe.yaml directory structure - Convert Python version operators to conda format - Support entry points conversion from project.scripts This enables generating conda recipes directly from pyproject.toml files following the same patterns as pyrattler-recipe-autogen.

Add patterns to ignore test files generated during pyproject recipe development to prevent accidental commits of temporary test artifacts.

- Add 11 unit tests covering all core functionality - Test dependency conversion with version constraints and environment markers - Test schema version handling (default and custom) - Test YAML formatting with schema header - Test context, package, requirements, and about section generation - Test dynamic version resolution for different build backends - Test Python constraint formatting and package name mapping All tests pass and provide good coverage of the pyproject module functionality.

- test_generate_recipe_pyproject_basic: Tests basic pyproject.toml parsing and recipe generation - test_generate_recipe_pyproject_with_conda_overrides: Tests conda-specific overrides - Both tests verify schema header generation, dependency conversion, and build system integration - Fixed directory creation issue in test framework by explicitly creating temp directories

- Replace disallowed std::fs methods with fs_err - Fix wildcard pattern in match expression - Initialize structs directly instead of field reassignment - Remove unnecessary let binding before return - Move functions before test module to fix items-after-test-module - Change PathBuf parameter to Path reference - Remove duplicate if branches in JSON serialization All clippy warnings have been resolved and the code now passes lint checks.

… options

…ated logic for stdout support

…f not specified

millsks · 2025-08-27T05:39:44Z

@wolfv would you or anyone else on the team mind reviewing and providing feedback? I believe this feature may be useful for those behind corporate networks that need to generate a recipe for their python project that may not be published anywhere. This option will allow them to create the recipe with their local pyproject.toml.

Even if it doesn't get merged in it was a good learning experience.

Thanks!

wolfv · 2025-08-27T12:36:35Z

I like the idea, will have to check out the code :)

wolfv · 2025-09-01T10:44:10Z

src/recipe_generator/pyproject.rs

+
+/// Convert Python dependency format to conda dependency format
+/// Following the same pattern as pyrattler-recipe-autogen
+fn convert_python_to_conda_dependency(dep: &str) -> String {


Would it be possible to re-use / share more code with the pypi implementation?

wolfv · 2025-09-01T10:44:30Z

src/recipe_generator/pyproject.rs

+}
+
+/// Apply common Python package name to conda package name mappings
+fn apply_package_name_mapping(dep: &str) -> String {


Same here - we would need the same code in pypi, right?

wolfv · 2025-09-01T10:45:58Z

Hey @millsks this is definitely functionality that we are interested in!

I am wondering - currently it looks like not a lot of code is shared with the PyPI generator. Do you think you could change that? Ideally we'd only have one version of these functions / mappings.

@zelosleone maybe you can also take a look?

zelosleone · 2025-09-01T11:02:15Z

src/recipe_generator/pyproject.rs

+}
+
+/// Generate a recipe from a pyproject.toml file
+pub async fn generate_pyproject_recipe(opts: &PyprojectOpts) -> miette::Result<()> {


does this need to be pub? it doesnt really being used anywhere besides the internal module via mod.rs anyways

zelosleone · 2025-09-01T11:16:22Z

src/recipe_generator/pyproject.rs

+fn assemble_recipe(
+    toml_data: HashMap<String, Value>,


I don't think we need a function like this like wolf suggested you can use pypi code here, since you are already mapping the toml to json before, it shouldn't be a problem to actually use the pypi code to generate the recipe with correct datas. And the custom logic here with 400 lines could be like simplified to maybe just one call.

zelosleone · 2025-09-01T11:21:13Z

src/recipe_generator/pyproject.rs

+fn build_context_section(
+    project: &serde_json::Map<String, Value>,


This also suffers the same fate as assemble_recipe as pypi code should already handle the normalization with correct mapping

zelosleone · 2025-09-01T11:23:40Z

src/recipe_generator/pyproject.rs

+fn build_package_section(
+    _project: &serde_json::Map<String, Value>,


Same as functions before this, please use pypi code for these after the correct mapping via toml to json

zelosleone · 2025-09-01T11:23:55Z

src/recipe_generator/pyproject.rs

+fn build_source_section(
+    project: &serde_json::Map<String, Value>,


Same as functions before this, please use pypi code for these after the correct mapping via toml to json

zelosleone · 2025-09-01T11:24:42Z

src/recipe_generator/pyproject.rs

+    // Default to PyPI source
+    let package_name = name.to_lowercase().replace("-", "_");
+    let pypi_url = format!(
+        "https://pypi.org/packages/source/{}/{}/{}-${{{{ version }}}}.tar.gz",
+        &package_name[..1],
+        package_name,
+        package_name
+    );


this defaulting could be also removed if we can just share code with pypi module.

zelosleone · 2025-09-01T11:34:35Z

src/recipe_generator/pyproject.rs

+}
+
+/// Build the build section
+fn build_build_section(toml_data: &HashMap<String, Value>) -> miette::Result<serialize::Build> {


this is also redudant and pypi code for this is better that you can use

to be more specific pypi uses the correct extraction methods with additional wheel analysis, you can get it from there

zelosleone · 2025-09-01T11:37:36Z

src/recipe_generator/pyproject.rs

+fn build_requirements_section(
+    project: &serde_json::Map<String, Value>,
+    toml_data: &HashMap<String, Value>,


i dont get why we are always extracting full toml data for specific sections of the build. you might want to map them correctly to pypi code via toml to json and just use the specific ones. but yeah this is also redudant and can be replaced with correct code sharing/calls to pypi module

zelosleone · 2025-09-01T11:47:21Z

Some problems that needs fixing:

We don't need to load the full toml data after you map them to json. You can get the specific parts instead for building requirements, platform etc.
I pointed some functions that can be replaced with pypi module we already have and this should be easy to do since you are with the first function in the module mapping the entire toml to json correctly (assuming)
We don't need to hardcode noarch either, we can just parse and map it to json correctly and let it just be handled by the code.

millsks · 2025-09-02T17:46:10Z

Hi @wolfv and @zelosleone. Thanks for the feedback and suggestions. I'll start working on it this week.

millsks added 4 commits August 24, 2025 23:03

chore: Update .gitignore to exclude pyproject test files

bb2941c

Add patterns to ignore test files generated during pyproject recipe development to prevent accidental commits of temporary test artifacts.

millsks marked this pull request as draft August 25, 2025 04:36

millsks added 4 commits August 26, 2025 22:26

chore: Remove trailing newline in pixi.toml

4d97f5a

fix: Update boto3 and botocore versions in pixi.lock

45d09d4

feat: Add support for generating recipes from pyproject.toml with new…

a5e3f2a

… options

millsks changed the title ~~Add pyproject.toml support to generate-recipe command~~ feat: add pyproject.toml support to generate-recipe command Aug 27, 2025

millsks added 5 commits August 26, 2025 23:39

test: Add end-to-end tests for pyproject.toml recipe generation

7329484

refactor: Update PyprojectOpts to make output optional and adjust rel…

6a54224

…ated logic for stdout support

test: Enhance stdout output test for pyproject.toml recipe generation

7d87d02

docs: Update output option for recipe generation to print to stdout i…

fe21c15

…f not specified

refactor: Remove redundant '--write' argument from command args in tests

e0ddb6c

millsks marked this pull request as ready for review August 27, 2025 05:33

wolfv requested changes Sep 1, 2025

View reviewed changes

zelosleone reviewed Sep 1, 2025

View reviewed changes

		fn build_context_section(
		project: &serde_json::Map<String, Value>,

		fn build_package_section(
		_project: &serde_json::Map<String, Value>,

		fn build_source_section(
		project: &serde_json::Map<String, Value>,

feat: add pyproject.toml support to generate-recipe command #1849

Are you sure you want to change the base?

feat: add pyproject.toml support to generate-recipe command #1849

Uh oh!

Conversation

millsks commented Aug 25, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

🎯 Summary

🚀 Features Added

New Command

Core Functionality

📝 Example Usage

Input pyproject.toml:

Generated recipe.yaml:

🔧 Implementation Details

Files Added/Modified

Key Components

🧪 Testing

Unit Tests (11 tests - all passing)

Integration Tests (2 tests - all passing)

🎁 Benefits

🔗 Related Issues

📋 Checklist

🚦 Testing Instructions

📚 Additional Notes

🚀 Quick Demo

Uh oh!

millsks commented Aug 27, 2025

Uh oh!

wolfv commented Aug 27, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

wolfv commented Sep 1, 2025

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

zelosleone commented Sep 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

millsks commented Sep 2, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

millsks commented Aug 25, 2025 •

edited

Loading

Input `pyproject.toml`:

Generated `recipe.yaml`:

zelosleone commented Sep 1, 2025 •

edited

Loading