Graceful Deployment Handling for Edge Workers

[jumski]

The Problem

When deploying a new Edge Function version on Supabase:

• New worker instances are created but old ones aren't terminated
• Old workers continue running for up to 150s (free tier) or 400s (paid tier)
• Both versions run simultaneously, processing messages from the same queues
• Old workers are removed from HTTP routing but their polling loops continue until resource limits hit
• This forces backward AND forward compatibility requirements
• Impacts MTTR and development iteration speed

Why This Matters

This deployment overlap creates several challenges:

1. Schema migrations: If you change message format or database schema, both versions need to handle both formats
2. Business logic changes: Bug fixes or logic changes take minutes to fully roll out
3. Resource waste: Old workers consume resources while processing with outdated code
4. Debugging complexity: Logs show both versions processing, making issues harder to trace

Proposed Solution: Deployment-Aware Workers

In short: Workers check if a newer deployment exists and stop polling for new work, allowing immediate cutover to new code.

Workers can detect when a newer deployment exists and gracefully stop polling while completing current work.

Implementation Approach

1. Deployment ID tracking:

   • Use DENO_DEPLOYMENT_ID environment variable (available in production)
   • Format: <identifier>_<deployment_number> (e.g., abc123xyz_1, def456uvw_2)
   • Deployment numbers are consecutive, enabling simple comparison
   • Workers store deployment_id (or just deployment_number) in pgflow.workers table when registering

2. Deployment check during heartbeat:

   • Extract deployment number from ID (split on _, take last part)
   • Simple comparison: if any worker has deployment_number > mine, stop polling
   • No need for timestamp comparison - deployment numbers tell the story
   • Complete current batch then idle until termination

3. Benefits:

   • Immediate cutover to new code for message processing
   • No forced kill needed - workers drain naturally
   • Maintains graceful shutdown for in-flight work

Example Implementation

// Extract deployment number from DENO_DEPLOYMENT_ID
const deploymentId = Deno.env.get("DENO_DEPLOYMENT_ID"); // e.g., "abc123_42"
const myDeploymentNum = deploymentId ? parseInt(deploymentId.split("_").pop()!) : null;

// During heartbeat, check for newer deployments
if (myDeploymentNum !== null) {
  const hasNewer = await sql`
    SELECT 1 FROM pgflow.workers
    WHERE deployment_number > ${myDeploymentNum}
      AND last_heartbeat_at > NOW() - INTERVAL '30s'
    LIMIT 1
  `;
  
  if (hasNewer) {
    console.log(`🛑 Stopping polling - newer deployment detected`);
    stopPolling();
  }
}

Integration with pg_cron Approach

The pg_cron-based polling approach (discussion #170) makes deployment detection even simpler: the HTTP request body can include the current deployment ID, allowing workers to compare against their own ID without any SQL queries.

Implementation Details to Discuss

1. What to store in workers table: Should we store:

   • Full deployment ID (e.g., abc123xyz_1) - preserves all information
   • Just deployment number (e.g., 1) - simpler comparisons, less storage
   • Both in separate columns - most flexible but redundant

2. Environment detection: Need to reliably distinguish local dev from production:

   • DENO_DEPLOYMENT_ID is missing locally, but is this enough to detect local env?
   • Should we check multiple env vars (e.g., SUPABASE_URL pattern, IS_LOCAL, etc.)?
   • Current thinking: Disable deployment checking when DENO_DEPLOYMENT_ID is missing AND other local indicators present
   • Need confirmation: What env vars reliably indicate local vs production?

3. Check frequency: When should workers check for new deployments?

   • Every heartbeat (current worker implementation)
   • Before each polling iteration (upcoming cron worker implementation)

Questions

1. Are there edge cases where immediate polling cessation could cause issues?

Other ideas

1. Maybe workers can use supabase realtime presence to notify each other about new deployments immediately when starting?

Related Discussions

• Worker Spawning Control: pg_cron-Based Alternative (#170) - The pg_cron approach provides natural deployment checkpoints

[nel]

Great stuff !!!!

Biggest problem to this approach, I did not see previously.

I just noticed an issue I did not see, we won't store new DEPLOYMENT_ID until code is deployed AND new WORKER is started. The restart is mandatory to store new DEPLOYMENT_ID in the DB. So unless you deploy AND curl or wait for next cron, this is not as good as new DEPLOY = stop polling 😭. Still a big improvement to current situation but not total elimination of forward compatibility requirement unless you add manually in your deploy the curl to reboot a process.

Are there edge cases where immediate polling cessation could cause issues?

If you have multiple workers and new deployment is detected then polling stop, no new tasks are being processed. The question is when new Worker are getting started, there will probably be a delay between deployment detected (via curl e.g.) and new process being started.

Unless similar to current implementation of imminent death detection, worker process start a new one.

I am not sure what is happening if it is cron pinging for new process, I guess it depends on impementation (current or new proposed one or no cron + safety cf https://github.com/orgs/pgflow-dev/discussions/170. Not trivial.

KEY or DEPLOYMENT_ID

For portability reason, just using the string DEPLOYMENT_ID and comparing it to current process DEPLOYMENT_ID seems sufficient. My 2 cents.