Disclaimer: No artificial intelligence tools or automated content generation systems were used in the creation of this content.
To reduce infrastructure costs, our production environment ran on a single-zone Kubernetes cluster.
At the time, that was a reasonable trade-off: traffic and operational scope were modest, and budget constraints made a lean footprint attractive.
The decision was defensible on paper. In practice, it concentrated every dependency (compute, networking, and control-plane operations) into one failure domain.
A full outage was not a question of if, but when.
How did it happen?
A single-zone cluster means all nodes, workloads, and the cloud resources they depend on live in one availability zone.
There is no geographic fallback within the cluster itself. When that zone cannot satisfy a scheduling or provisioning request, the blast radius is the entire production stack.
The incident unfolded in the middle of the day during a routine deployment.
ArgoCD attempted to sync a new manifest that required the creation of a GCP resource.
The underlying zone had exhausted available capacity, so the resource could not be provisioned.
ArgoCD offered no clear signal that the failure was capacity-related. The application remained in a perpetual Progressing state (the sync appeared to be underway, but it never completed).
We tried to recover by recreating the related gateway manifest. That operation failed as well.
With the zone still unable to allocate resources, the cluster entered a degraded state: delete operations hung, kubectl and other control interactions became unreliable, and orphaned services began to accumulate behind the scenes.
Because the gateway had already been removed and could not be recreated, external traffic had no path into the environment.
Production was unreachable.
How a multi-zonal cluster saved the day
With the existing cluster effectively unusable, recovery required a clean path forward rather than incremental repair inside an exhausted zone.
We stood up a separate multi-zonal cluster, bootstrapped the platform components, and synced the ArgoCD applications against the new environment.
Once workloads were healthy, we updated DNS so the domain pointed to the new load balancer IP address.
Service was restored.
The fix worked because capacity and scheduling were no longer bound to a single zone.
The same manifests that could not converge in the old cluster could be applied across zones in the new one, and the entry point (the gateway and its load balancer) could be recreated where resources were actually available.
Multi-zonal is a must for production
For production workloads, a multi-zonal cluster should be treated as a baseline requirement, not an optional upgrade.
Spreading nodes across zones does not eliminate operational risk, but it removes the single-zone dependency that turned a routine sync into a total outage.
Cost pressure is real, and not every team can run large node pools in every zone.
Even on a tight budget, distributing a minimal footprint (such as one node per zone) buys meaningful resilience compared with concentrating everything in a single zone and waiting for capacity exhaustion or zone-level failure to take production offline.