Skip to content

feat: add ResourceOptimization workspace with conditional compute environment deployment #185

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 5 commits into
base: main
Choose a base branch
from
Open

feat: add ResourceOptimization workspace with conditional compute environment deployment #185

wants to merge 5 commits into from

Conversation

@edmundmiller
Copy link
Contributor

@edmundmiller edmundmiller commented Oct 22, 2025 •
edited
Loading

Summary

Adds support for the ResourceOptimization Seqera Platform workspace with conditional compute environment deployment capabilities for the multi-workspace architecture.

Changes

Core Infrastructure Improvements

  • Conditional Compute Environment Deployment: Workspaces can now selectively enable/disable CPU, GPU, and ARM compute environments via workspace_config.py
  • Workspace-Specific IAM Resources: IAM users and policies now include workspace name suffix to prevent conflicts between workspaces
  • Flexible S3 Infrastructure: Support both importing existing buckets and creating new ones

ResourceOptimization Workspace

Successfully deployed new workspace for Florian's resource optimization testing:

  • Workspace ID: 80750985193419
  • Compute Environments: CPU only (GPU and ARM disabled)
  • S3 Bucket: nf-core-resource-optimization
  • IAM Resources: TowerForge-AWSMegatests-ResourceOptimization (user + 3 policies)
  • Deployment Time: 1m2s (13 resources created)

Technical Details

  1. Updated compute_environments.py:

    • Added workspace_config parameter to deploy_seqera_environments_terraform()
    • Conditional deployment based on workspace_config["compute_environments"][env_type]["enabled"]
    • Updated get_compute_environment_ids_terraform() to handle missing environments gracefully

  2. Updated credentials.py:

    • Added workspace_name parameter to create_towerforge_credentials()
    • Workspace suffix applied to IAM user and policy names
    • Prevents conflicts when multiple workspaces exist

  3. Updated s3.py:

    • Added bucket_name and import_existing parameters
    • Supports creating new buckets for new workspaces
    • Maintains backward compatibility for existing AWSMegatests bucket import

  4. Updated constants.py:

    • Changed config file paths to absolute paths
    • Uses Path(__file__).parent.parent for shared directory reference

  5. Added Seqera SDK:

    • Copied pulumi_seqera v0.25.2 to both workspaces' sdks/ directories
    • Removed from pyproject.toml dependencies (not on PyPI)
    • Loaded via sys.path manipulation in __main__.py

  6. Documentation:

    • Added manual setup commands to pulumi_state/README.md
    • Documented 1Password credential retrieval
    • Included environment variable requirements and import commands

Deployment Output

compute_env_ids:
  cpu: teiXlOfccCoaPubY9sp6a

s3_bucket:
  arn: arn:aws:s3:::nf-core-resource-optimization
  name: nf-core-resource-optimization
  region: eu-west-1

workspace:
  name: ResourceOptimization
  organization: nf-core
  workspace_id: "80750985193419"

Testing

  • ✅ Pulumi preview successful (17 resources planned)
  • ✅ Deployment successful (13 resources created in 1m2s)
  • ✅ CPU compute environment created and visible in Seqera Platform
  • ✅ GPU and ARM environments correctly skipped
  • ✅ IAM resources created with workspace-specific names (no conflicts)
  • ✅ S3 bucket created successfully
  • ✅ Credentials uploaded to Seqera Platform

Related

edmundmiller and others added 5 commits October 21, 2025 16:22
@edmundmiller @claude
feat: add multi-workspace Seqera Platform structure
1f25214 
Create new pulumi/seqera_platform directory with support for multiple
independent Seqera Platform workspaces. This enables managing separate
workspaces with different configurations while sharing common code.

Structure:
- shared/: Reusable modules for providers, infrastructure, and integrations
- awsmegatests/: Full workspace with CPU, GPU, and ARM compute environments
- resource_optimization/: Focused workspace with CPU-only for resource testing

Each workspace is an independent Pulumi project with workspace-specific
configuration defined in workspace_config.py. The shared modules are
imported via sys.path manipulation to avoid code duplication.

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@edmundmiller @claude
feat: add Pulumi state storage bootstrap project
69e4191 
Create S3-backed state storage for all Pulumi projects. This bootstrap
project creates and manages the nf-core-pulumi-state bucket in eu-north-1.

Infrastructure:
- S3 bucket with versioning enabled (nf-core-pulumi-state)
- Server-side encryption (AES-256)
- Public access blocked
- Lifecycle policy (90-day version retention)
- Tags for resource tracking

Backend usage:
  pulumi login 's3://nf-core-pulumi-state?region=eu-north-1&awssdk=v2'

This project uses local backend to avoid circular dependency, while
all other projects use the S3 backend.

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@edmundmiller @claude
feat: add Pulumi state storage bootstrap project
3388276 
Create S3-backed state storage for all Pulumi projects. This bootstrap
project creates and manages the nf-core-pulumi-state bucket in eu-north-1.

Infrastructure:
- S3 bucket with versioning enabled
- Server-side encryption (AES-256)
- Public access blocked
- Lifecycle policy (90-day version retention)
- Tags for resource tracking

Backend usage:
  pulumi login 's3://nf-core-pulumi-state?region=eu-north-1&awssdk=v2'

🤖 Generated with Claude Code (https://claude.com/claude-code)

Co-Authored-By: Claude <[email protected]>
@edmundmiller
docs: add manual setup documentation for Pulumi state storage
abc64d0 
- Document AWS CLI commands for manual S3 bucket creation
- Add 1Password credential retrieval instructions
- Document environment variable requirements
- Add Pulumi import commands for existing bucket resources
- Provide troubleshooting guidance for bucket setup
@edmundmiller
feat: add ResourceOptimization workspace with conditional compute env…
e2d7ab9 
…ironment deployment

- Add support for workspace-specific configurations in multi-workspace setup
- Implement conditional compute environment deployment (CPU/GPU/ARM can be toggled per workspace)
- Add workspace-specific IAM resource naming to avoid conflicts between workspaces
- Add pulumi-seqera SDK (v0.25.2) to both awsmegatests and resource_optimization workspaces
- Update S3 infrastructure to support both import (existing buckets) and create (new buckets)
- Add absolute paths for seqerakit config files in constants
- Copy seqerakit configs to shared module for workspace reuse
- Update credentials.py to accept workspace_name parameter for unique IAM resources
- Remove pulumi-seqera from pyproject.toml dependencies (loaded via sdks/ directory)
- Document manual setup commands and 1Password credential retrieval in pulumi_state/README.md

Deployed ResourceOptimization workspace:
- Workspace ID: 80750985193419
- CPU compute environment only (GPU and ARM disabled)
- S3 bucket: nf-core-resource-optimization
- IAM resources: TowerForge-AWSMegatests-ResourceOptimization (user and policies)
- Successfully deployed 13 resources in 1m2s
@edmundmiller edmundmiller requested review from a team and maxulysse as code owners October 22, 2025 07:39
@edmundmiller edmundmiller self-assigned this Oct 22, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@maxulysse maxulysse Awaiting requested review from maxulysse maxulysse is a code owner

@FloWuenne FloWuenne Awaiting requested review from FloWuenne

At least 1 approving review is required to merge this pull request.

Assignees

@edmundmiller edmundmiller

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@edmundmiller