Kubernetes Probes: Why Healthy Pods Can Still Cause Outages
A retrospective of Kubernetes health checks, traffic readiness, startup behavior, and how poorly tuned probes can amplify production incidents.
A running process is not a working service
A container running at the operating-system level does not guarantee that a visitor can complete a request. An application may be deadlocked, its connection pool exhausted, or critical initialization still underway. Kubernetes health probes aim to make those states observable. However, confusing the questions that probes answer can turn a helpful recovery mechanism into an outage amplifier. When reviewing the liveness and readiness approach associated with the 2017 Kubernetes era, distinguish historical capabilities from features that matured in later releases. A successful HTTP health response does not prove that every external service or user journey is functioning. A probe should verify a small, explicitly defined property rather than pretending to measure the entire system.
When should liveness restart a container?
Liveness checks determine when an application should be restarted because it can no longer make progress. A worker trapped in an internal deadlock may retain its process ID while never completing another job; a restart could recover that condition. A brief database outage is different. Restarting every service instance during a transient dependency failure often makes matters worse: each new instance creates connections, warms caches, and competes for resources just when the dependency is struggling. A liveness endpoint should be lightweight, describe the progress of its own process, and avoid treating every temporary failure in a remote system as an unrecoverable container defect. The cost of a false restart can exceed the original incident.
When should readiness remove traffic?
Readiness answers a different question: should this instance receive user traffic right now? A service that is loading configuration, validating schema compatibility, or warming a critical cache may be running but not yet ready. When readiness fails, Kubernetes can remove the Pod from the Service's available endpoints, allowing other healthy instances to handle requests. That is less disruptive than immediately killing the process. Readiness checks can themselves become a bottleneck if they are expensive or excessively frequent. Include dependencies only when their state genuinely determines the ability to serve requests. Measure rapid changes between ready and unready during overload; unstable signaling can create oscillating traffic patterns across the remaining instances.
Modern startup probes and initialization
A slow-starting application can be restarted repeatedly if liveness begins checking too early. In current Kubernetes releases, a startup probe can explicitly guard the initialization phase; liveness and readiness probes wait until startup succeeds. It would be historically incorrect to assume that every Kubernetes cluster in 2017 already supported this modern mechanism. Older environments must use the startup delays and thresholds available in their exact version. For a modern design, measure real initialization time distributions, account for high-percentile delays, and avoid masking a broken startup path with excessively generous timeouts. Initial readiness and a service that becomes unhealthy after hours of work are separate operational questions and deserve separate signals.
Example: one failing instance in a three-Pod service
Imagine a web API running with three Pods. One Pod intermittently loses its database connection and begins failing user requests. If readiness detects that state and removes only the affected instance from traffic, the other two may continue serving within their capacity limits. If liveness instead causes all three instances to restart around the same time, the entire service can become unavailable. Probe settings must therefore be considered together with rolling-update strategy, resource limits, disruption budgets, and capacity planning. In a test environment, briefly interrupt the database and record restart counts, endpoint membership, p95 latency, and failed requests. Recovery should be evaluated by its cost to actual users, not merely by whether a container restarted successfully.
A practical production checklist
Start by writing one sentence for each probe: can the process make progress, can this instance safely accept traffic, and has initialization completed? Choose periodSeconds, timeoutSeconds, failureThreshold, and initialDelaySeconds from measurements rather than arbitrary defaults. Verify that probe endpoints are reachable under your network and authorization policies without exposing sensitive configuration. Record restart counts, endpoint changes, response latency, and error rates before and after deployment. Rehearse conditions that resemble production incidents instead of testing only the happy path. Most importantly, retain a rollback plan and prove that the health-check configuration does not amplify an otherwise recoverable dependency failure into a wider outage.
Comments
No comments yet. Be the first to contribute.
Reading “Kubernetes sağlıklı görünüyor ama hizmet neden kesiliyor? Probe tasarımı” raised this question for me: compatibility and debugging impact deserve review.
Editor note: Product claims and measured outcomes should be assessed separately. A small pilot with defined metrics is a useful first step.
We should also discuss maintenance cost. An easy rollout does not guarantee trouble-free upgrades.
I would start with a pilot and measurable success criteria rather than assuming a launch means proven value.
✎ Write a comment