RHOShift - OpenShift Operator Installation Toolkit

A comprehensive, enterprise-grade toolkit for managing OpenShift operators with enhanced stability features, automatic dependency resolution, and Red Hat OpenShift AI (RHOAI) integration.

📋 Table of Contents

Features
Enhanced Stability Features
Supported Operators
Installation
Usage
Advanced Usage
Dependency Management
RHOAI Integration
Configuration
Troubleshooting
Contributing

✨ Features

🚀 Core Functionality

7 Enterprise Operators: Complete operator stack for modern OpenShift deployments
Enhanced Stability System: 3-tier stability levels with comprehensive error handling
Automatic Dependency Resolution: Smart installation order with dependency detection
Pre-flight Validation: Cluster readiness and permission verification
Health Monitoring: Real-time operator status tracking and reporting
Auto-recovery: Intelligent error classification and automatic retry logic

🛡️ Enterprise-Grade Reliability

Comprehensive Error Handling: 59+ exception handlers throughout codebase
Webhook Certificate Resilience: Automatic timing issue resolution for RHOAI
Resource Conflict Detection: Prevention of operator namespace conflicts
Smart Retry Logic: Exponential backoff with contextual error recovery
Parallel Installation: Optimized performance for multiple operators

🔧 Advanced Integration

RHOAI DSC/DSCI Management: Complete DataScienceCluster lifecycle control
Kueue Management States: Dynamic DSC integration with Managed/Unmanaged modes
KedaController Automation: Automatic KEDA controller creation and validation
Configurable Timeouts: Flexible timing control for enterprise environments

🛡️ Enhanced Stability Features

RHOShift includes a comprehensive stability system designed for enterprise deployments:

Stability Levels

🟢 Enhanced (Default): Pre-flight checks + health monitoring + auto-recovery
🔵 Comprehensive: Maximum resilience with advanced error classification
⚪ Basic: Standard installation with basic error handling

Pre-flight Validation

✅ Cluster connectivity and authentication
✅ Required permissions verification
✅ Resource quota validation
✅ Operator catalog accessibility
✅ Namespace conflict detection
✅ DSCI compatibility validation for RHOAI installations

Health Monitoring

📊 Real-time operator status tracking
🔍 Multi-resource health validation
📈 Installation progress reporting
⚡ Performance metrics and timing

Auto-recovery Features

🔄 Intelligent retry mechanisms
🧠 Error classification (transient vs. permanent)
⏰ Exponential backoff strategies
🛠️ Automatic resource cleanup and recreation

📦 Supported Operators

Operator	Package	Namespace	Channel	Dependencies
OpenShift Serverless	`serverless-operator`	`openshift-serverless`	`stable`	None
Service Mesh	`servicemeshoperator`	`openshift-operators`	`stable`	None
Authorino	`authorino-operator`	`openshift-operators`	`stable`	None
cert-manager	`openshift-cert-manager-operator`	`cert-manager-operator`	`stable-v1`	None
Kueue	`kueue-operator`	`openshift-kueue-operator`	`stable-v1.0`	cert-manager
KEDA	`openshift-custom-metrics-autoscaler-operator`	`openshift-keda`	`stable`	None
RHOAI/ODH	`opendatahub-operator`	`openshift-operators`	`stable`	None

🚀 Installation

Quick Install

git clone https://github.com/mwaykole/O.git
cd O
pip install -e .

Verify Installation

rhoshift --help
rhoshift --summary

💻 Usage

Basic Commands

# Install single operator with enhanced stability
rhoshift --serverless

# Install multiple operators with batch optimization
rhoshift --serverless --servicemesh --authorino

# Install with dependency resolution (Kueue + cert-manager)
rhoshift --kueue

# Install all operators (includes DSCI validation for RHOAI)
rhoshift --all

# Install all with RHOAI channel preference
rhoshift --all --rhoai-channel=odh-nightlies

# Show detailed operator summary
rhoshift --summary

# Clean up all operators
rhoshift --cleanup

RHOAI with DSC/DSCI

# Install RHOAI with complete setup
rhoshift --rhoai \
  --rhoai-channel=odh-nightlies \
  --rhoai-image=brew.registry.redhat.io/rh-osbs/iib:1049242 \
  --deploy-rhoai-resources

# Install RHOAI with Kueue integration
rhoshift --rhoai --kueue Managed \
  --rhoai-channel=stable \
  --rhoai-image=quay.io/rhoai/rhoai-fbc-fragment:rhoai-2.25-nightly \
  --deploy-rhoai-resources

Kueue Management States

# Install Kueue as Managed (RHOAI controls it)
rhoshift --kueue Managed

# Install Kueue as Unmanaged (independent) - Default
rhoshift --kueue Unmanaged
rhoshift --kueue  # Same as above

# Switch management states (updates existing DSC)
rhoshift --kueue Managed    # Switch to Managed
rhoshift --kueue Unmanaged  # Switch to Unmanaged

🔧 Advanced Usage

Enterprise Deployment

# Complete ML/AI stack with queue management
rhoshift --all --kueue Managed \
  --rhoai-channel=stable \
  --rhoai-image=brew.registry.redhat.io/rh-osbs/iib:1049242 \
  --deploy-rhoai-resources \
  --timeout=900

# High-availability setup with service mesh
rhoshift --serverless --servicemesh --keda --authorino

# Development environment setup
rhoshift --cert-manager --kueue Unmanaged --keda

Custom Configuration

# Custom timeouts and retries for enterprise clusters
rhoshift --all \
  --timeout=1200 \
  --retries=5 \
  --retry-delay=15

# Custom oc binary path
rhoshift --serverless --oc-binary=/usr/local/bin/oc

# Verbose output for debugging
rhoshift --kueue Managed --verbose

🔗 Dependency Management

RHOShift automatically handles operator dependencies:

Automatic Resolution

Kueue → cert-manager: Installing Kueue automatically includes cert-manager
Installation Order: Dependencies installed first, primary operators second
Conflict Detection: Prevents namespace and resource conflicts

Smart Validation

# This command installs BOTH cert-manager AND Kueue in correct order:
rhoshift --kueue
# Output:
# 🔍 Pre-flight checks passed. Cluster is ready for installation.
# ⚠️  Missing dependency: kueue-operator requires openshift-cert-manager-operator
# 🚀 Installing 2 operators with enhanced stability...
# ✅ cert-manager installed successfully
# ✅ kueue installed successfully

🤖 RHOAI Integration

DataScienceCluster Management

RHOShift provides complete DSC/DSCI lifecycle management:

# Create RHOAI with DSC/DSCI
rhoshift --rhoai --deploy-rhoai-resources

# RHOAI with Kueue integration
rhoshift --rhoai --kueue Managed --deploy-rhoai-resources

DSC Behavior

Existing DSC: Automatically updates Kueue managementState
No DSC: State applied when DSC is created via --deploy-rhoai-resources
Webhook Resilience: Automatic handling of certificate timing issues

Output Examples

# When DSC exists and gets updated:
🔄 Updating DSC with Kueue managementState: Managed
✅ Successfully updated DSC with Kueue managementState: Managed

# When no DSC exists:
ℹ️  No existing DSC found. Kueue managementState will be applied when DSC is created.

⚙️ Configuration

CLI Options

Operator Selection:
  --serverless          Install OpenShift Serverless Operator
  --servicemesh         Install Service Mesh Operator
  --authorino           Install Authorino Operator
  --cert-manager        Install cert-manager Operator
  --rhoai               Install RHOAI Operator
  --kueue [{Managed,Unmanaged}]  Install Kueue with DSC integration
  --keda                Install KEDA (Custom Metrics Autoscaler)
  --all                 Install all operators
  --cleanup             Clean up all operators
  --summary             Show operator summary

Configuration:
  --oc-binary OC_BINARY     Path to oc CLI (default: oc)
  --retries RETRIES         Max retry attempts (default: 3)
  --retry-delay RETRY_DELAY Delay between retries (default: 10s)
  --timeout TIMEOUT         Command timeout (default: 300s)

RHOAI Options:
  --rhoai-channel CHANNEL   RHOAI channel (stable/odh-nightlies)
  --rhoai-image IMAGE       RHOAI container image
  --raw RAW                 Enable raw serving (True/False)
  --deploy-rhoai-resources  Create DSC and DSCI

Environment Variables

export LOG_FILE_LEVEL=DEBUG      # File logging level
export LOG_CONSOLE_LEVEL=INFO    # Console logging level

Logging

Location: /tmp/rhoshift.log
Rotation: 10MB max size, 5 backup files
Levels: DEBUG (file) / INFO (console)
Colors: Supported in compatible terminals

🔍 Troubleshooting

Common Issues

Permission Errors

# Verify cluster access
oc whoami
oc auth can-i create subscriptions -n openshift-operators

Installation Failures

# Check logs
tail -f /tmp/rhoshift.log

# Verify operator catalogs
oc get catalogsource -n openshift-marketplace

# Check with enhanced timeouts
rhoshift --kueue --timeout=900 --retries=5

Dependency Issues

# Verify dependencies are resolved
rhoshift --summary

# Manual dependency installation
rhoshift --cert-manager
rhoshift --kueue

RHOAI/DSC Issues

# Check DSC status
oc get dsc,dsci -A

# Verify webhook certificates
oc get pods -n opendatahub-operators

# Manual DSC creation
rhoshift --rhoai --deploy-rhoai-resources --timeout=900

DSCI Immutable Field Conflicts

# Error: MonitoringNamespace is immutable
# This happens when existing DSCI has different monitoring namespace

# Check existing DSCI configuration
oc get dsci default-dsci -o yaml

# Solution 1: Force recreate DSCI (recommended)
rhoshift --rhoai --deploy-rhoai-resources

# Solution 2: Use existing DSCI configuration
# RHOShift will automatically detect and adapt to existing DSCI

Debug Mode

# Enable verbose output
rhoshift --all --verbose

# Check stability report
rhoshift --summary

🛠️ Development

Prerequisites

Python 3.8+
OpenShift CLI (oc)
OpenShift cluster access
cluster-admin privileges

Project Structure

rhoshift/
├── rhoshift/
│   ├── cli/              # Command-line interface
│   ├── logger/           # Logging system
│   ├── utils/
│   │   ├── operator/     # Operator management
│   │   ├── resilience.py # Error handling & recovery
│   │   ├── health_monitor.py # Health monitoring
│   │   ├── stability_coordinator.py # Stability management
│   │   └── constants.py  # Operator configurations
│   └── main.py          # Entry point
├── scripts/
│   ├── cleanup/         # Cleanup utilities
│   └── run_upgrade_matrix.sh # Upgrade testing
└── tests/               # Test suite

Running Tests

pytest tests/

🤝 Contributing

Fork the repository
Create a feature branch: git checkout -b feature-name
Commit changes: git commit -am 'Add feature'
Push to branch: git push origin feature-name
Create Pull Request

Development Guidelines

Follow Python PEP 8 standards
Add tests for new features
Update documentation
Ensure backward compatibility

📄 License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.

🆘 Support

Issues: GitHub Issues
Documentation: This README and --help output
Logs: /tmp/rhoshift.log for detailed debugging

RHOShift - Enterprise-grade OpenShift operator management with enhanced stability and reliability features.

Name		Name	Last commit message	Last commit date
Latest commit History 71 Commits
rhoshift		rhoshift
scripts		scripts
tests		tests
.gitignore		.gitignore
.pre-commit-config.yaml		.pre-commit-config.yaml
LICENSE		LICENSE
MANIFEST.in		MANIFEST.in
README.md		README.md
__init__.py		__init__.py
pyproject.toml		pyproject.toml
requirements.txt		requirements.txt
setup.py		setup.py
upgrade_matrix_usage.md		upgrade_matrix_usage.md

License

mwaykole/O

Folders and files

Latest commit

History

Repository files navigation