Definition
A probe is a periodic check whose result Kubernetes uses to make a specific lifecycle decision: allow startup to continue, mark a workload ready for traffic, or restart a container.
01Read the report
A restart count is a symptom. Find the decision behind it.
The report is illustrative: requests fail while the database is unavailable, and restart counts rise. These observations fit several explanations. A dependency-sensitive liveness check might be ordering restarts. Memory exhaustion might be killing the containers. The application might be exiting on its own. Changing probes before distinguishing those explanations can leave the fault in place.
Inspect the affected container’s last termination reason and exit status, its previous logs, and Pod events in the same time window. Match the Pod identity and container: a new Pod is not another restart of the old container. A “Killing … failed liveness probe” event supports a different action from an OOMKilled termination.
Choose the next observationWhat would weaken the probe hypothesis?
If the termination record says OOMKilled and the memory limit was reached, increasing the probe threshold does not address that evidence. If probe failures appear without a matching restart, check thresholds and timing before claiming they killed the process. If the configured endpoint responds locally but the kubelet times out, investigate reachability, binding, port and load. A YAML file tells you the intended policy; the active Pod and events tell you which policy ran.
02Separate the decisions
Wait, withdraw traffic, and restart solve different problems.
The ticket API needs the database to serve every ordinary request. It reconnects when the database returns. Initialization loads a local index; a particular local progress failure is repaired by a new process. These are the workload’s assumptions. A service that can serve cached responses during the outage would need a different readiness contract.
| Probe | Question | Action after its failure threshold |
|---|---|---|
| Startup | Has this process finished initialization? | Restart under this Deployment’s Always restart policy. Until startup succeeds, regular readiness and liveness checks wait. |
| Readiness | Can this replica serve its promised requests now? | Mark it not ready for ordinary Service traffic; keep the process running. |
| Liveness | Has this process reached a failure that restarting can repair? | Restart the container under the Deployment’s restart policy. |
Readiness does not finish work already accepted. Endpoint updates and traffic routing take time, and direct Pod access or existing connections can follow another path. Keep shutdown and connection draining as a separate lifecycle contract.
03Predict the consequence
What happens when a database check can order a restart?
Experiment
One outage, two liveness policies
Both model runs start ready. They observe every five seconds; two failed readiness observations withdraw traffic and three failed liveness observations request a restart. Startup after a restart gates the other checks. Compare the same database outage under two policies.
04Configure the contract
Give each endpoint one question to answer.
The HTTP server delegates its decisions to the same TypeScript handler you can step through
in the lab. The fixture exposes three endpoints. /startupz returns success
after local initialization. /readyz also requires a working dependency and
local progress. /livez checks local progress without consulting the database. An HTTP listener accepting
a connection does not prove that these contracts hold.
export type HealthState = { initialized: boolean; stuck: boolean; databaseDown: boolean };
/** The HTTP server and recorded calls use this same handler. */
export function statusFor(path: string | undefined, state: HealthState) {
if (path === '/startupz') {
return state.initialized ? 200 : 503;
}
if (path === '/livez') {
return state.stuck ? 503 : 200;
}
if (path === '/readyz' || path === '/tickets') {
const canServe = state.initialized && !state.stuck && !state.databaseDown;
return canServe ? 200 : 503;
}
return 404;
}
The server samples initialization, local progress and dependency state before calling this function. It converts the returned status into an HTTP response. The handler decides what to report; the Kubernetes configuration decides what authority that report has.
startupProbe:
httpGet:
path: /startupz
port: http
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 24
readinessProbe:
httpGet:
path: /readyz
port: http
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 2
successThreshold: 1
livenessProbe:
httpGet:
path: /livez
port: http
periodSeconds: 5
timeoutSeconds: 1
failureThreshold: 3 Initialization normally takes 50 seconds in this fixture. The startup probe allows 24 failures at a five-second period: a nominal 120-second allowance, leaving room for slower starts. It remains bounded if initialization never finishes. This product is a sizing aid, not an exact restart timestamp; scheduling, probe execution and termination affect what you observe.
The readiness threshold tolerates one failed observation before traffic eligibility changes; a success restores it. Liveness waits for three failures. Choose periods, timeouts and thresholds against the workload’s tolerable failure exposure and measured behavior. Raising every threshold trades slower failure detection for fewer reactions; it does not fix a check asking the wrong question.
Inspect the complete HTTP fixtureNode 22 · simulated dependency and progress faults
The files named /tmp/database-down and /tmp/stuck simulate endpoint
observations inside disposable containers. There is no real database or deadlock. An independent
progress check is useful here because the HTTP handler still responds during the simulated stall.
A real blocked event loop may instead fail by timeout.
// Disposable drill fixture. Marker files simulate observations, not a real DB or deadlock.
import { createServer } from 'node:http';
import { existsSync } from 'node:fs';
import { pathToFileURL } from 'node:url';
import { statusFor } from './health.ts';
export { statusFor } from './health.ts';
/** @param {() => import('./health.ts').HealthState} readState */
export function createFixtureServer(readState) {
return createServer((request, response) => {
const status = statusFor(request.url, readState());
response.writeHead(status, {
'Content-Type': 'text/plain',
'Cache-Control': 'no-store',
'X-Replica': process.env.HOSTNAME ?? 'local'
});
response.end(status === 200 ? 'ok\n' : 'unavailable\n');
});
}
if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
const startedAt = Date.now();
createFixtureServer(() => ({
initialized: Date.now() - startedAt >= Number(process.env.STARTUP_MS ?? 50000),
stuck: existsSync('/tmp/stuck'),
databaseDown: existsSync('/tmp/database-down')
})).listen(Number(process.env.PORT ?? 8080), '0.0.0.0');
}
05Verify failure and recovery
Follow the request path as well as the process.
| Fault | Expected evidence | Recovery check |
|---|---|---|
| Dependency unavailable for one replica | Ready becomes false; ordinary EndpointSlice ready condition becomes false; no new restart. | Clear the fault on that exact replica. Ready returns; requests through the Service succeed and can reach it again. |
| Local progress failure | Liveness fails; matching event orders termination; the same Pod’s container restart count rises. | New container initializes, then becomes ready; Service requests succeed. The container-local fault marker is gone. |
| Initialization exceeds its allowance | Startup failures trigger repeated restarts before initialization completes. | Use a justified bounded allowance; observe startup succeed and regular probes begin, then test requests. |
Run the disposable Kubernetes drillDocker, kind and kubectl · complete recipe
Download Dockerfile, app.mjs, deployment.yaml and drill.sh and health.ts into one fresh directory. Run the commands from that directory. The recipe below targets
only kind-probes-lesson and namespace probes-lesson. Run one group at a time, saving Pod identity,
events, restart counts and request results. It creates a named local cluster; use a fresh
one for this exercise.
A separate client Pod sends requests through the Service. A port-forward to a Pod cannot establish Service routing behavior. Inject into one replica, identify it, clear its marker and verify recovery before injecting the next fault. Delete only the named lesson cluster after collecting your evidence.
# Run from this lesson's examples directory. Requires Docker, kind, and kubectl.
# All cluster commands explicitly target a disposable local cluster.
kind create cluster --name probes-lesson
docker build -t probes-lesson:v1 .
kind load docker-image probes-lesson:v1 --name probes-lesson
kubectl --context kind-probes-lesson create namespace probes-lesson
kubectl --context kind-probes-lesson apply -f deployment.yaml
kubectl --context kind-probes-lesson -n probes-lesson rollout status deployment/tickets --timeout=180s
# Observe before injecting a failure. Record restart counts for each Pod.
kubectl --context kind-probes-lesson -n probes-lesson get pods -l app=tickets
kubectl --context kind-probes-lesson -n probes-lesson get endpointslices -l kubernetes.io/service-name=tickets -o yaml
# Start an in-cluster client. Its image is already loaded; this adds no new image.
kubectl --context kind-probes-lesson -n probes-lesson run request-check --image=probes-lesson:v1 --image-pull-policy=Never --restart=Never --command -- node -e 'setInterval(() => {}, 1000)'
kubectl --context kind-probes-lesson -n probes-lesson wait --for=condition=Ready pod/request-check --timeout=60s
# Repeat this request before, during, and after the drill. Record status and replica.
kubectl --context kind-probes-lesson -n probes-lesson exec request-check -- node -e "fetch('http://tickets/tickets').then(async r => console.log(r.status, r.headers.get('x-replica'), await r.text()))"
# A marker simulates an unavailable dependency for ONE replica.
kubectl --context kind-probes-lesson -n probes-lesson exec deployment/tickets -- node -e "require('node:fs').writeFileSync('/tmp/database-down', '')"
# Wait at least 15 seconds, then inspect readiness, restarts, and endpoint conditions.
# No restart is expected. Save the affected Pod name from this output.
kubectl --context kind-probes-lesson -n probes-lesson get pods -l app=tickets
kubectl --context kind-probes-lesson -n probes-lesson get endpointslices -l kubernetes.io/service-name=tickets -o yaml
# Replace AFFECTED_POD with that name; do not let exec choose a different replica.
kubectl --context kind-probes-lesson -n probes-lesson exec AFFECTED_POD -- node -e "require('node:fs').unlinkSync('/tmp/database-down')"
# After recovery, simulate a local progress failure in that same replica.
kubectl --context kind-probes-lesson -n probes-lesson exec AFFECTED_POD -- node -e "require('node:fs').writeFileSync('/tmp/stuck', '')"
kubectl --context kind-probes-lesson -n probes-lesson describe pod AFFECTED_POD
# Expect a liveness failure, one container restart, startup gating, then Ready.
# Marker files live in the container writable layer and disappear on restart.
# Cleanup only this disposable cluster when observations are complete.
kind delete cluster --name probes-lesson
The browser model checks policy consequences under declared assumptions. HTTP fixture tests check endpoint responses. A cluster drill checks controller and traffic behavior. A passing result at one layer does not establish the others.
06Make the next call
Change the action only when the evidence warrants it.
Keep the operating rule: ask what this observation warrants, what the action can repair, and which evidence would show recovery. Revisit the rule when startup time, dependency behavior or the serving contract changes.
Practice with an audio-guide catalogue
Carry one configuration through a dependency outage, a larger startup index, and a restart investigation. Each level starts from the preceding reference configuration.
Kubernetes referencesProbe behavior checked 17 October 2026
See Liveness, Readiness, and Startup Probes for the decisions attached to each check, and Configure probes for HTTP behavior, fields and failure considerations. Use the documentation for your cluster version when adapting the drill.