Automatic agent redeployment when TLS secret changes

[lets-call-n-walk]

Problem

Currently, agents do not automatically redeploy when the Kubernetes Secrets referenced in their ModelConfig's TLS configuration change. This means that if a TLS certificate is rotated or updated in a Secret, agents will continue using the old certificate until they are manually redeployed.

This creates operational overhead and potential security issues:

• Certificate rotations require manual intervention
• Agents may fail to connect if old certificates expire
• No automatic propagation of updated CA certificates or client certificates

Solution

Implement automatic agent redeployment when TLS secrets referenced in ModelConfig resources are updated.

Technical Approach

1. Extend Secret Watcher in Controller

The controller already watches Secrets for API key changes. Extend this functionality to also monitor TLS-related secrets:

• Watch for changes to Secrets referenced in ModelConfig.spec.tls.caCertSecretRef
• Trigger reconciliation when these secrets are updated

2. Extend findModelsUsingSecret Function

Update the existing findModelsUsingSecret function to check TLS secret fields in addition to API key secrets:

// Check both API key secrets and TLS secrets
if model.Spec.APIKeySecretRef == secretName || 
   (model.Spec.TLS != nil && model.Spec.TLS.CACertSecretRef == secretName) {
    // Add to list of models using this secret
}

3. Trigger Agent Redeployment

When a TLS secret changes:

1. Option A - Pod Template Hash Update: Update the agent deployment's pod template annotations with a hash of the secret content, forcing Kubernetes to roll out new pods
2. Option B - ConfigMap/Secret Version Tracking: Track secret versions in agent deployment annotations and update when secrets change
3. Option C - Direct Pod Restart: Delete existing agent pods to force recreation with updated secret mounts

Recommended: Option A (hash-based annotation) as it provides declarative state and proper rollout behavior.

4. Ensure Cascading Updates

Make sure the redeployment propagates correctly:

• ModelConfig changes should trigger updates to all dependent Agents
• Secret changes should trigger ModelConfig reconciliation
• ModelConfig reconciliation should trigger Agent reconciliation
• Agent reconciliation should update deployment with new hash/version

Implementation Notes

• Follow the existing pattern used for API key secret watching
• Ensure reconciliation is idempotent
• Add unit tests for secret change detection
• Add E2E tests for automatic redeployment on secret update
• Consider rate limiting to avoid excessive redeployments
• Document the automatic redeployment behavior for users

Reference

• PR feat: Add SSL/TLS Configuration Support to ModelConfig CRD #1059: feat: Add SSL/TLS Configuration Support to ModelConfig CRD
• Review Comment add the "generate_resource" for the istio agent as tool #11 from @inFocus7
• Related to examples/modelconfig-with-tls.yaml line 384

Scope Note

This enhancement is out of scope for PR #1059. This issue is created for future implementation and tracking purposes.

[lets-call-n-walk]

Note, waiting on completion of this PR: #1059