Skip to content

This package provides a unified entry point and organized task modules for managing N5/Zarr dataset conversions and downsampling. It centralizes your workflow into a single pipeline to reduce manual work and errors.

License

Notifications You must be signed in to change notification settings

JaneliaSciComp/tensorswitch

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

15 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

TensorSwitch

This package provides a unified entry point and organized task modules for managing N5/Zarr dataset conversions and downsampling. It centralizes your workflow into a single pipeline to reduce manual work and errors.

Folder Structure

tensorswitch/
├── pixi.lock
├── pyproject.toml
├── README.md
├── src
│   └── tensorswitch
│       ├── __init__.py
│       ├── __main__.py # Main dispatcher script
│       ├── tasks
│       │   ├── __init__.py
│       │   ├── downsample_shard_zarr3.py # Downsample using shards
│       │   ├── n5_to_n5.py               # N5 to N5 conversion logic
│       │   ├── n5_to_zarr2.py            # N5 to Zarr V2 conversion logic
│       │   ├── tiff_to_zarr3_s0.py       # TIFF to Zarr V3 level s0 conversion logic  
│       ├── utils.py                      # Common utilities (chunk domain calculation)
│       ├── z_to_chunk_index.py           # Print chunk index ranges for resubmit failed or left over jobs
├── re_submit_jobs.ipynb                  # Jupyter notebook to re-submit failed chunk jobs
└── tests
    ├── test_n5_to_n5.py
    ├── test_n5_to_zarr2.py
    ├── test_n5_to_zarr3_downsample_shard.py
    ├── test_zarr3_to_downsample_noshard_zarr3.py
    └── test_zarr3_to_downsample_shard_zarr3.py

Installation

Pip

pip install git+https://github.com/JaneliaSciComp/tensorswitch

Pixi

pixi add python
pixi add --pypi "tensorswitch @ git+https://github.com/JaneliaSciComp/tensorswitch"

How to Use

1. Run the Main Pipeline

Use the python -m tensorswitch as the entry point. Example:

python -m tensorswitch --task n5_to_zarr2     --base_path /path/to/n5     --output_path /path/to/zarr2     --level 0

2. Submit Cluster Jobs (Optional)

You can also use the --submit flag to dispatch jobs to an LSF cluster:

python -m tensorswitch --task downsample_shard_zarr3 \
  --base_path /path/to/zarr/s0 \
  --output_path /path/to/zarr \
  --level 1 \
  --downsample 1 \
  --use_shard 1 \
  --submit \
  --project your_project_name

3. Supported Tasks

Task Name Description
n5_to_zarr2 Convert N5 to Zarr V2
n5_to_n5 Convert N5 to N5 (new chunking)
downsample_zarr Downsample existing Zarr dataset
submit_downsample Submit downsampling jobs to the cluster
submit_n5_zarr Submit N5 to Zarr jobs to the cluster
submit_n5_n5 Submit N5 to N5 jobs to the cluster
tiff_to_zarr3_s0 Convert TIFF stack to Zarr V3 (s0)

4. Example Commands

Convert N5 to Zarr V2 locally

python -m tensorswitch --task n5_to_zarr2     --base_path /path/to/n5     --output_path /path/to/zarr2     --level 0

Downsample Zarr locally

python -m tensorswitch  --task downsample_shard_zarr3     --base_path /path/to/zarr/s0     --output_path /path/to/zarr     --level 1     --use_shard 1

Submit N5 to Zarr jobs to cluster

python -m tensorswitch --task n5_to_zarr2 --base_path /path/to/n5 --output_path /path/to/zarr2 --submit --project your_project_name

Convert TIFF to Zarr v3 s0

python -m tensorswitch --task tiff_to_zarr3_s0 --base_path /path/to/tiff_folder --output_path /path/to/zarr3 --use_shard 0

Resubmit Failed Chunks

Use re_submit_jobs.ipynb to debug chunk failures and resubmit specific chunk ranges using z_to_chunk_index.py.


Requirements

  • Python 3.10+
  • TensorStore
  • NumPy
  • psutil
  • requests (for N5 over HTTP)
  • Dask + tifffile (for TIFF conversion)

Support

If you need more enhancements (like adding logging or progress tracking), feel free to extend the tasks modules.

For resubmissions, consider using the interactive notebook re_submit_jobs.ipynb and CLI helpers in z_to_chunk_index.py.


About

This package provides a unified entry point and organized task modules for managing N5/Zarr dataset conversions and downsampling. It centralizes your workflow into a single pipeline to reduce manual work and errors.

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published