Skip to content

Conversation

@HereThereBeDragons
Copy link
Contributor

@HereThereBeDragons HereThereBeDragons commented Dec 4, 2025

Script to upload releases (or any kind of .whls and .tar.gz) to some S3 bucket.
Expects directory structure to be the same as created by the download_prerelease_packages.py

Has 2 safety features:

  • dry-run
  • and upload first to some testing bucket. need a second flag or manual setting from the bucket to use the release bucket.

Part of Issue #1891:
This is the third step of package promotion from release candidate (e.g.
7.10.0rc2) to release (7.10.0).

  1. Download prerelease packages (Script to download prereleases by version #2382)
  2. Promote packages to release (Python script to promote a "rc" version packages to final release #1973)
  3. Upload release packages to S3 release bucket (this one here)
  4. Update index in S3 release bucket (use existing script manage.py)

@HereThereBeDragons
Copy link
Contributor Author

example output upload from tarballs
================================================================================
Upload Release Packages
================================================================================
Input directory: /home/lpromber/TheRockVanilla/test_promotion/download
Python bucket: therock-testing-bucket
Python bucket prefix: release-upload-testing/
Tarball bucket: therock-testing-bucket
Tarball bucket prefix: release-upload-testing/tarball/
Execution mode: EXECUTE (will upload)
================================================================================

Found 2 architecture(s):
  - gfx1151
  - gfx950-dcgpu

Uploading tarballs to S3...
================================================================================
  Found 3 tarball file(s) to upload:
    Uploading therock-dist-linux-gfx1151-7.10.0.tar.gz ... done
    Uploading therock-dist-linux-gfx950-dcgpu-7.10.0.tar.gz ... done
    Uploading therock-dist-windows-gfx1151-7.10.0.tar.gz ... done

================================================================================
SUMMARY
================================================================================
Execution mode: EXECUTED
Tarballs uploaded: 3 (location: s3://therock-testing-bucket/release-upload-testing/tarball/)

Next steps:
  - Verify uploads in S3 bucket
  - Use manage.py to generate and upload index files
================================================================================

@HereThereBeDragons
Copy link
Contributor Author

HereThereBeDragons commented Dec 4, 2025

example output for dryrun (ignore the different versions. i was running the promotion script in between)
================================================================================
Upload Release Packages
================================================================================
Input directory: /home/lpromber/TheRockVanilla/test_promotion/download
Python bucket: therock-testing-bucket
Python bucket prefix: release-upload-testing/
Tarball bucket: therock-testing-bucket
Tarball bucket prefix: release-upload-testing/tarball/
Execution mode: DRY-RUN (no uploads)
================================================================================

⚠️  SAFETY: Dry-run mode active - no actual uploads will be performed
⚠️          Use --execute flag to perform actual uploads
⚠️          Additionally use --use-release-buckets flag for production buckets (therock-release-python and therock-release-tarball)


Found 2 architecture(s):
  - gfx1151
  - gfx950-dcgpu

Uploading Python packages to S3...
================================================================================
DRY-RUN MODE: No actual uploads will be performed
================================================================================

Architecture: gfx1151
  Found 21 file(s) to upload:
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp311-cp311-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp311-cp311-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp313-cp313-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp313-cp313-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp311-cp311-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp311-cp311-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp313-cp313-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp313-cp313-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.5.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/pytorch_triton_rocm-3.5.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm-7.10.0rc2.tar.gz -> s3://therock-testing-bucket/release-upload-testing/gfx1151/rocm-7.10.0rc2.tar.gz
    [DRY-RUN] Would upload: rocm_sdk_core-7.10.0-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/rocm_sdk_core-7.10.0-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm_sdk_devel-7.10.0rc2-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/rocm_sdk_devel-7.10.0rc2-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm_sdk_libraries_gfx1151-7.10.0-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/rocm_sdk_libraries_gfx1151-7.10.0-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.7.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torch-2.7.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.8.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torch-2.8.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.8.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torch-2.8.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.9.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torch-2.9.1+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.7.1a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchaudio-2.7.1a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.8.0a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchaudio-2.8.0a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.9.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchaudio-2.9.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.22.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchvision-0.22.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.23.0a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchvision-0.23.0a0+rocm7.10.0-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.24.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx1151/torchvision-0.24.0+rocm7.10.0-cp312-cp312-linux_x86_64.whl

Architecture: gfx950-dcgpu
  Found 16 file(s) to upload:
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/pytorch_triton_rocm-3.3.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/pytorch_triton_rocm-3.4.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: pytorch_triton_rocm-3.5.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/pytorch_triton_rocm-3.5.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm-7.10.0rc2.tar.gz -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/rocm-7.10.0rc2.tar.gz
    [DRY-RUN] Would upload: rocm_sdk_core-7.10.0rc2-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/rocm_sdk_core-7.10.0rc2-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm_sdk_devel-7.10.0rc2-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/rocm_sdk_devel-7.10.0rc2-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: rocm_sdk_libraries_gfx950_dcgpu-7.10.0rc2-py3-none-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/rocm_sdk_libraries_gfx950_dcgpu-7.10.0rc2-py3-none-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.7.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torch-2.7.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.8.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torch-2.8.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torch-2.9.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torch-2.9.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.7.1a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchaudio-2.7.1a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.8.0a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchaudio-2.8.0a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchaudio-2.9.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchaudio-2.9.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.22.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchvision-0.22.1+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.23.0a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchvision-0.23.0a0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl
    [DRY-RUN] Would upload: torchvision-0.24.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl -> s3://therock-testing-bucket/release-upload-testing/gfx950-dcgpu/torchvision-0.24.0+rocm7.10.0rc2-cp312-cp312-linux_x86_64.whl

Uploading tarballs to S3...
================================================================================
DRY-RUN MODE: No actual uploads will be performed
================================================================================
  Found 3 tarball file(s) to upload:
    [DRY-RUN] Would upload: therock-dist-linux-gfx1151-7.10.0.tar.gz -> s3://therock-testing-bucket/release-upload-testing/tarball/therock-dist-linux-gfx1151-7.10.0.tar.gz
    [DRY-RUN] Would upload: therock-dist-linux-gfx950-dcgpu-7.10.0.tar.gz -> s3://therock-testing-bucket/release-upload-testing/tarball/therock-dist-linux-gfx950-dcgpu-7.10.0.tar.gz
    [DRY-RUN] Would upload: therock-dist-windows-gfx1151-7.10.0.tar.gz -> s3://therock-testing-bucket/release-upload-testing/tarball/therock-dist-windows-gfx1151-7.10.0.tar.gz

================================================================================
SUMMARY
================================================================================
Execution mode: DRY-RUN
  Python packages that would be uploaded: 37
  Tarballs that would be uploaded: 3

Next steps:
  - Review the dry-run output above
  - Run again with --execute to perform actual uploads to testing bucket
  - Add --use-release-buckets flag when ready for production upload
================================================================================

python build_tools/packaging/upload_release_packages.py --input-dir ./promotion/download/ --no-upload-python --upload-tarballs --execute --use-release-buckets
```

## 4. Update index files for the release bucket
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

TODO: step 5 sanity check the uploaded files?

Comment on lines +1 to +3
# How to make a release available

The steps for the release are:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We may want a warning here saying these instructions are only for release team members 🤔
(permissions should be set up that way, but just to deter anyone casually browsing)

Copy link
Member

@ScottTodd ScottTodd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fine to merge without addressing comments

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HereThereBeDragons HereThereBeDragons merged commit 2d21446 into main Dec 5, 2025
11 checks passed
@HereThereBeDragons HereThereBeDragons deleted the users/lpromber/release_upload_script_and_doc branch December 5, 2025 19:02
@github-project-automation github-project-automation bot moved this from TODO to Done in TheRock Triage Dec 5, 2025
@HereThereBeDragons
Copy link
Contributor Author

comments can be addressed in a future pr

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

3 participants