bug/blocker: apisix-Ingress-controller: Upstream/Route state not refreshed after pod restart → 502s (adc v0.21.1, AIC 2.0.0-rc4) - helm-chart 2.11.6

[sopacifics]

Hi team, can anyone help with this bug? It’s blocking the upgrade of APISIX Helm chart from 2.10.1 to 2.11.6, as well as the application upgrade from 3.11 (no longer supported) to 3.13

Current Behavior

Current Behavior

Summary
After a workload pod restarts and receives a new IP, APISIX continues routing to the old endpoint. In addition, route-level custom plugins (auth/credentials) are not refreshed. This leads to 502 Bad Gateway responses until a manual resync.

Also, when a new route is created via CDR, it does not appear in the admin route list (returns 404), even though the route creation itself succeeds. Deleting and recreating the route does not resolve the issue.

Environment

• helm-chart: 2.11.6
• adc: 0.21.1 (custom image for linux/arm64)
• apisix-ingress-controller: 2.0.0-rc4
• apisix: 3.13
• Traffic config: APISIX CRDs (ApisixRoute, etc.)

2025-09-29T10:17:21Z  Service restarted
New pod IP:           10.13.195.107

APISIX admin view (after restart) still shows old node:
"upstream": {
  "nodes":[{"host":"10.13.197.151","port":3000,"weight":100,"priority":0}]
}

APISIX then attempts to route to a non-existent pod → 502 Bad Gateway.

Expected

• New routes created via CRDs should appear in admin and work as expected.
• Upstream nodes should update promptly to reflect the new pod IP(s).
• Route custom plugins (auth/credentials) should reconcile with the current CRD state.

Actual

• New routes are missing in admin (https://127.0.0.1:9180/apisix/admin/routes) → 404 Not Found.
• Upstream continues pointing to the previous pod IP.
• Route custom plugins are not refreshed.
• Requests return 502 until a manual resync.

Notes

• We build our own adc:0.21.1 image for arm64 (binary pulled from the linked release asset).

Versions

• APISIX: 3.13
• Helm-chart: 2.11.6
• Ingress-controller: 2.0.0-rc4 (arm64)
• ADC: 0.21.1 (arm64)
• OS: (output of uname -a)

Related Links

• PR: fix(apisix-standalone): conflict between upstream nodes and service discovery api7/adc#344
• Release: https://github.com/api7/adc/releases/tag/v0.21.1

Expected Behavior

No response

Error Logs

No response

Steps to Reproduce

Steps to Reproduce

502 Bad Gateway.

• Deploy a Service/Deployment behind an ApisixRoute.
• Confirm upstream node(s) in APISIX match the running pod IP(s).
• Restart the pod (e.g., kubectl rollout restart deploy/).
• Observe new pod IP via kubectl get pod -o wide.
• Query APISIX admin or /status to inspect upstream nodes.
• Send traffic to the route.

404 Not Found

• Create a new route
• Confirm that the route is created successfully
  `status:
  conditions:
  • message: Sync Successfully
    observedGeneration: 1
    reason: ResourcesSynced
    status: "True"
    type: ResourcesAvailable`
• Observe route via kubectl get apisixroute -A
• Route is not appearing in APISIX admin route
• 404 when try to reach the new route.

Environment

• APISIX Ingress controller version (run apisix-ingress-controller version --long)
• Kubernetes cluster version (run kubectl version)
• OS version if running APISIX Ingress controller in a bare-metal environment (run uname -a)

[dosubot[bot]]

This is a known issue in the APISIX ecosystem: after a pod restart, upstream nodes and routes sometimes aren't refreshed, causing 502 errors and missing routes in the admin API. Multiple recent threads confirm that APISIX may continue routing to old pod IPs until a manual resync or controller restart, and new routes may not appear immediately in the admin API (#12294, #12283, #12436).

Restarting the apisix-ingress-controller pod usually forces a resync and resolves the stale IP issue temporarily. The root cause is a lag or bug in the reconciliation logic between Kubernetes CRDs, ADC, and APISIX, especially after dynamic changes like pod restarts. The maintainers recommend using APISIX Ingress Controller v2.0.0 and standalone mode to avoid some of these synchronization issues (#12283).

There is no documentation or merged PR confirming that the fix from adc#344 is included in adc v0.21.1, and no upgrade guides for Helm chart 2.11.6 affecting route/upstream reconciliation were found. The ARM64 ADC build should behave like the official release if built from the provided assets.

For now, the most reliable workaround is to manually restart the ingress-controller pod after pod restarts or route changes to force a resync. If you see "Key not found" in the admin API, ensure etcd keys are initialized (run apisix init_etcd if needed). If possible, monitor for updates to the ingress-controller and ADC that address this bug directly.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[sopacifics]

dosubot[bot]

Is there an ETA for a stable ingress-controller 2.0.0 release with these sync bugs fixed? Unfortunately, restarting the apisix-ingress-controller doesn't resolve the issue in my case.
We can keep using Helm chart 2.10.1 (APISIX 3.11) with ingress-controller 1.8.4, but my concern is that APISIX 3.11 is no longer actively maintained.
Alternative question: Are there known compatibility issues that prevent using APISIX 3.13 with ingress-controller 1.8.4, or would this combination just have missing features? If it's only missing features (not critical bugs), we might prefer running the stable controller with the maintained APISIX version.

[dosubot[bot]]

There’s no official ETA for a stable ingress-controller 2.0.0 release yet, but it’s in active development and maintainers expect sync bugs to be resolved in the stable version. The 1.x series (including 1.8.4) will be discontinued soon, and users are encouraged to test the 2.0.0 release candidates in their environments to help surface remaining issues source.

As for APISIX 3.13 with ingress-controller 1.8.4: there’s no published compatibility matrix, but multiple threads confirm that the 1.x controller series has sync bugs and missing features with newer APISIX releases, especially around upstream IP refresh and route reconciliation after pod restarts source source. If you use this combination, you may encounter stale routing, missing routes, and plugin refresh issues—these are not just missing features but can be critical bugs in production scenarios.

If stability is your priority, running the maintained APISIX version with the stable controller is reasonable, but be aware that sync issues may persist until the new ingress-controller is released. Standalone mode (without etcd) is recommended for production to avoid some of these problems source.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}    

[sopacifics]

@dosu Thanks for the clarification. Just to confirm my understanding about standalone mode:

The new APISIX Ingress Controller 2.0.0+ supports API-driven Standalone Mode (https://docs.api7.ai/apisix/reference/api-standalone-usage), which:

• ✅ Works with CRDs via the ingress controller
• ❌ Is currently experimental and not recommended for production

The traditional File-driven Standalone Mode:

• ✅ Stable and production-ready
• ❌ Does NOT support CRDs (requires manual apisix.yaml configuration)

My concern: If the 1.x ingress-controller series (including 1.8.4) will be discontinued soon (#12506), but the 2.0.0 API-driven standalone mode is still experimental, what's the migration path for production users who rely on CRDs?

Key question: Will the 1.x controller continue to be supported until API-driven standalone mode reaches production stability? We depend on the Admin API since our ingress controller reads CRDs and updates APISIX accordingly.

Should we:

• Continue with 1.x and accept the sync bugs until 2.0.0 stable is released?
• Test 2.0.0-RC in production despite the experimental warning?
• Use traditional etcd-backed mode instead of standalone for now?

[dosubot[bot]]

The ingress-controller 1.x series (including 1.8.4) is being phased out and will not receive ongoing support until API-driven standalone mode is fully production-ready—maintainers recommend migrating to the 2.0.0 series, which is actively developed and aims to resolve sync bugs and CRD issues source source.

For production users who rely on CRDs and the Admin API, the safest migration path is to use traditional etcd-backed mode for now. This mode is stable, supports CRDs, and avoids the risks of experimental features. The API-driven standalone mode works with CRDs but is not recommended for production yet source.

Testing ingress-controller 2.0.0-RC in a staging or production-like environment is encouraged to help surface issues, but be aware that fixes are still being merged and stability is not guaranteed. Continuing with 1.x means accepting known sync bugs and reduced support. If you need reliability and CRD support, stick with etcd-backed mode until the new controller and API-driven standalone mode are proven stable.

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}