feat(kubernetes): support HA gateway rebalancing by TaylorMutch · Pull Request #1868 · NVIDIA/OpenShell

TaylorMutch · 2026-06-11T04:47:21Z

Summary

Adds HA gateway rebalancing support for Kubernetes deployments so client and supervisor traffic can survive gateway replica scale-up, scale-down, and pod rotation.

This PR targets main directly. The reconciler lease work from #1577 has already landed, so this PR now focuses on peer authentication/routing, supervisor relay handoff, Kubernetes ownership behavior, Helm/Skaffold HA wiring, CLI retry hardening, and HA validation.

How it works

The Gateway now exposes a peer Service to let gateway replicas discover and call each other to reach supervisors. When a client request lands on a replica that does not currently own the target sandbox supervisor session, that gateway resolves the owning replica and relays the supervisor traffic to the peer instead of failing the request. Kubernetes lease/reconciler ownership keeps sandbox supervision coordinated as gateway pods scale, roll, or disappear, while the CLI retries transient sync probes during those handoffs.

Related Issue

Closes #1021

Related: #1012, #1429, #1577, #1731, #1488

Changes

Adds gateway peer authentication and peer routing for HA supervisor relay handoff.
Adds Kubernetes compute lease/reconciler ownership behavior for multi-replica gateways.
Adds Helm peer Service/RBAC rendering and Skaffold HA/Envoy dev profile support.
Adds Kubernetes HA rebalancing e2e coverage and removes the noisy readyz e2e smoke.
Retries CLI sandbox file sync and transient sync probe failures after gateway rollouts.
Fixes z3 cross-builds by using Zig archive tools for generated gateway artifacts.
Updates architecture and local cluster/debug skills for HA gateway development.

Testing

cargo fmt --all -- --check
mise run helm:lint
cargo check -p openshell-server --features test-support
mise run pre-commit
Local Kubernetes HA validation with Envoy Gateway, external PostgreSQL, and gateway scale/rotation
GitHub Branch Checks passed
GitHub Helm Lint passed
GitHub Kubernetes HA E2E smoke passed with test:e2e-kubernetes

Checklist

Follows Conventional Commits
Commits are signed off (DCO)

github-actions · 2026-06-11T04:47:33Z

Label test:e2e-kubernetes applied for ad9f04d. Open the existing run and click Re-run all jobs to execute with the label set. The run will execute Kubernetes HA E2E after building the required gateway and supervisor images once. This is an optional proof-of-life suite; failures are visible in the workflow run but do not publish a required CI gate status.

elezar · 2026-06-29T13:02:18Z

+```bash
+kubectl -n openshell get svc openshell-peer -o wide
+kubectl -n openshell get endpoints openshell-peer
+kubectl -n openshell get pod -l app.kubernetes.io/instance=openshell \


Question: Does naming the envvar OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_FILE instead of OPENSHELL_PEER_SERVICE_ACCOUNT_TOKEN_PATH have precedence?

elezar · 2026-06-29T13:07:57Z

+`kubernetes_ha_rebalancing`. That coverage validates sandbox create/watch and
+exec through the Envoy proxy while gateway replicas scale up, scale down, and
+rotate.


Does it also run a longer running sandbox and then perform the operations? Are those relevant / in scope?

elezar · 2026-07-10T14:37:36Z


        match mode {
-            AuthMode::Unauthenticated | AuthMode::Sandbox => {
+            AuthMode::Unauthenticated | AuthMode::Sandbox | AuthMode::Peer => {


Is this an exhaustive list? Does matching on _ (after the supported AuthModes) make sense?

elezar · 2026-07-10T14:38:27Z

            value.span(),
            format!(
-                "invalid auth mode `{other}`; expected one of `unauthenticated`, `sandbox`, `bearer`, `dual`"
+                "invalid auth mode `{other}`; expected one of `unauthenticated`, `sandbox`, `bearer`, `dual`, `peer`"


Nit: Is expanding this list every time we add a new auth mode really what we want to do?

elezar · 2026-07-10T14:40:19Z

            AuthMode::Bearer => quote! { crate::auth::method_authz::AuthMode::Bearer },
            AuthMode::Dual => quote! { crate::auth::method_authz::AuthMode::Dual },
+            AuthMode::Peer => quote! { crate::auth::method_authz::AuthMode::Peer },
        };


Nit: Is this juse the reverse of parse_auth_mode? Should it be a function? (Also what happens when auth.mode is not one of the match arms?

Signed-off-by: Taylor Mutch <taylormutch@gmail.com>

TaylorMutch requested review from a team, derekwaynecarr and mrunalp as code owners June 11, 2026 04:47

TaylorMutch added the test:e2e-kubernetes Requires Kubernetes end-to-end coverage label Jun 11, 2026

TaylorMutch force-pushed the 1021-ha-gateway-rebalancing/tm branch from 24c1003 to 3e590e6 Compare June 11, 2026 17:35

elezar reviewed Jun 12, 2026

View reviewed changes

Comment thread deploy/helm/openshell/skaffold.yaml Outdated

TaylorMutch force-pushed the 1021-ha-gateway-rebalancing/tm branch 3 times, most recently from fb46193 to a60c79c Compare June 16, 2026 20:59

TaylorMutch requested a review from maxamillion as a code owner June 16, 2026 20:59

TaylorMutch force-pushed the 1021-ha-gateway-rebalancing/tm branch 2 times, most recently from e93a30d to 493f3da Compare June 23, 2026 19:23

TaylorMutch requested a review from elezar June 23, 2026 19:30

TaylorMutch force-pushed the 1021-ha-gateway-rebalancing/tm branch from 493f3da to a831deb Compare June 23, 2026 20:39

This was referenced Jun 25, 2026

feat: gateway inter-pod session forwarding for HA multi-replica deployments #1994

Closed

feat: add first-class sandbox-to-sandbox network policy primitive #1993

Closed

TaylorMutch force-pushed the 1021-ha-gateway-rebalancing/tm branch from a831deb to 236c8ab Compare June 26, 2026 16:55