← DevOps
Technique Kubernetes · workload lifecycle

Startup, readiness, and liveness probes

Give each observation the right authority.

Three ticket API replicas share a database. During a database outage, all three begin restarting. The alert says “unhealthy”, but a new process cannot bring the database back. Before changing a threshold, find out what ordered the restart and what that action could repair.

TypeScript TypeScript handler · Kubernetes configuration · offline replay

Definition

A probe is a periodic check whose result Kubernetes uses to make a specific lifecycle decision: allow startup to continue, mark a workload ready for traffic, or restart a container.

01Read the report

A restart count is a symptom. Find the decision behind it.

The report is illustrative: requests fail while the database is unavailable, and restart counts rise. These observations fit several explanations. A dependency-sensitive liveness check might be ordering restarts. Memory exhaustion might be killing the containers. The application might be exiting on its own. Changing probes before distinguishing those explanations can leave the fault in place.

Inspect the affected container’s last termination reason and exit status, its previous logs, and Pod events in the same time window. Match the Pod identity and container: a new Pod is not another restart of the old container. A “Killing … failed liveness probe” event supports a different action from an OOMKilled termination.

Choose the next observationWhat would weaken the probe hypothesis?

If the termination record says OOMKilled and the memory limit was reached, increasing the probe threshold does not address that evidence. If probe failures appear without a matching restart, check thresholds and timing before claiming they killed the process. If the configured endpoint responds locally but the kubelet times out, investigate reachability, binding, port and load. A YAML file tells you the intended policy; the active Pod and events tell you which policy ran.

02Separate the decisions

Wait, withdraw traffic, and restart solve different problems.

The ticket API needs the database to serve every ordinary request. It reconnects when the database returns. Initialization loads a local index; a particular local progress failure is repaired by a new process. These are the workload’s assumptions. A service that can serve cached responses during the outage would need a different readiness contract.

Observation → authority → consequence
ProbeQuestionAction after its failure threshold
StartupHas this process finished initialization?Restart under this Deployment’s Always restart policy. Until startup succeeds, regular readiness and liveness checks wait.
ReadinessCan this replica serve its promised requests now?Mark it not ready for ordinary Service traffic; keep the process running.
LivenessHas this process reached a failure that restarting can repair?Restart the container under the Deployment’s restart policy.

Readiness does not finish work already accepted. Endpoint updates and traffic routing take time, and direct Pod access or existing connections can follow another path. Keep shutdown and connection draining as a separate lifecycle contract.

03Predict the consequence

What happens when a database check can order a restart?

Experiment

One outage, two liveness policies

Both model runs start ready. They observe every five seconds; two failed readiness observations withdraw traffic and three failed liveness observations request a restart. Startup after a restart gates the other checks. Compare the same database outage under two policies.

Both policies use database availability for readiness. Policy A also uses it for liveness; policy B checks only local progress for liveness. The database is unavailable from model second 20 through 95 and recovers at 100. Which policy requests more restarts?

04Configure the contract

Give each endpoint one question to answer.

The HTTP server delegates its decisions to the same TypeScript handler you can step through in the lab. The fixture exposes three endpoints. /startupz returns success after local initialization. /readyz also requires a working dependency and local progress. /livez checks local progress without consulting the database. An HTTP listener accepting a connection does not prove that these contracts hold.

health.ts · the same handler the lab steps
export type HealthState = { initialized: boolean; stuck: boolean; databaseDown: boolean };

/** The HTTP server and recorded calls use this same handler. */
export function statusFor(path: string | undefined, state: HealthState) {
	if (path === '/startupz') {
		return state.initialized ? 200 : 503;
	}
	if (path === '/livez') {
		return state.stuck ? 503 : 200;
	}
	if (path === '/readyz' || path === '/tickets') {
		const canServe = state.initialized && !state.stuck && !state.databaseDown;
		return canServe ? 200 : 503;
	}
	return 404;
}

The server samples initialization, local progress and dependency state before calling this function. It converts the returned status into an HTTP response. The handler decides what to report; the Kubernetes configuration decides what authority that report has.

deployment.yaml · container probe excerpt
startupProbe:
  httpGet:
    path: /startupz
    port: http
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 24
readinessProbe:
  httpGet:
    path: /readyz
    port: http
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2
  successThreshold: 1
livenessProbe:
  httpGet:
    path: /livez
    port: http
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 3

Initialization normally takes 50 seconds in this fixture. The startup probe allows 24 failures at a five-second period: a nominal 120-second allowance, leaving room for slower starts. It remains bounded if initialization never finishes. This product is a sizing aid, not an exact restart timestamp; scheduling, probe execution and termination affect what you observe.

The readiness threshold tolerates one failed observation before traffic eligibility changes; a success restores it. Liveness waits for three failures. Choose periods, timeouts and thresholds against the workload’s tolerable failure exposure and measured behavior. Raising every threshold trades slower failure detection for fewer reactions; it does not fix a check asking the wrong question.

Inspect the complete HTTP fixtureNode 22 · simulated dependency and progress faults

The files named /tmp/database-down and /tmp/stuck simulate endpoint observations inside disposable containers. There is no real database or deadlock. An independent progress check is useful here because the HTTP handler still responds during the simulated stall. A real blocked event loop may instead fail by timeout.

app.mjs · endpoint fixture
// Disposable drill fixture. Marker files simulate observations, not a real DB or deadlock.
import { createServer } from 'node:http';
import { existsSync } from 'node:fs';
import { pathToFileURL } from 'node:url';

import { statusFor } from './health.ts';
export { statusFor } from './health.ts';

/** @param {() => import('./health.ts').HealthState} readState */
export function createFixtureServer(readState) {
	return createServer((request, response) => {
		const status = statusFor(request.url, readState());
		response.writeHead(status, {
			'Content-Type': 'text/plain',
			'Cache-Control': 'no-store',
			'X-Replica': process.env.HOSTNAME ?? 'local'
		});
		response.end(status === 200 ? 'ok\n' : 'unavailable\n');
	});
}

if (process.argv[1] && import.meta.url === pathToFileURL(process.argv[1]).href) {
	const startedAt = Date.now();
	createFixtureServer(() => ({
		initialized: Date.now() - startedAt >= Number(process.env.STARTUP_MS ?? 50000),
		stuck: existsSync('/tmp/stuck'),
		databaseDown: existsSync('/tmp/database-down')
	})).listen(Number(process.env.PORT ?? 8080), '0.0.0.0');
}

05Verify failure and recovery

Follow the request path as well as the process.

Collect a baseline, inject one fault, then observe recovery
FaultExpected evidenceRecovery check
Dependency unavailable for one replicaReady becomes false; ordinary EndpointSlice ready condition becomes false; no new restart.Clear the fault on that exact replica. Ready returns; requests through the Service succeed and can reach it again.
Local progress failureLiveness fails; matching event orders termination; the same Pod’s container restart count rises.New container initializes, then becomes ready; Service requests succeed. The container-local fault marker is gone.
Initialization exceeds its allowanceStartup failures trigger repeated restarts before initialization completes.Use a justified bounded allowance; observe startup succeed and regular probes begin, then test requests.
Run the disposable Kubernetes drillDocker, kind and kubectl · complete recipe

Download Dockerfile, app.mjs, deployment.yaml and drill.sh and health.ts into one fresh directory. Run the commands from that directory. The recipe below targets only kind-probes-lesson and namespace probes-lesson. Run one group at a time, saving Pod identity, events, restart counts and request results. It creates a named local cluster; use a fresh one for this exercise.

A separate client Pod sends requests through the Service. A port-forward to a Pod cannot establish Service routing behavior. Inject into one replica, identify it, clear its marker and verify recovery before injecting the next fault. Delete only the named lesson cluster after collecting your evidence.

drill.sh · run one group at a time
# Run from this lesson's examples directory. Requires Docker, kind, and kubectl.
# All cluster commands explicitly target a disposable local cluster.
kind create cluster --name probes-lesson
docker build -t probes-lesson:v1 .
kind load docker-image probes-lesson:v1 --name probes-lesson
kubectl --context kind-probes-lesson create namespace probes-lesson
kubectl --context kind-probes-lesson apply -f deployment.yaml
kubectl --context kind-probes-lesson -n probes-lesson rollout status deployment/tickets --timeout=180s

# Observe before injecting a failure. Record restart counts for each Pod.
kubectl --context kind-probes-lesson -n probes-lesson get pods -l app=tickets
kubectl --context kind-probes-lesson -n probes-lesson get endpointslices -l kubernetes.io/service-name=tickets -o yaml

# Start an in-cluster client. Its image is already loaded; this adds no new image.
kubectl --context kind-probes-lesson -n probes-lesson run request-check --image=probes-lesson:v1 --image-pull-policy=Never --restart=Never --command -- node -e 'setInterval(() => {}, 1000)'
kubectl --context kind-probes-lesson -n probes-lesson wait --for=condition=Ready pod/request-check --timeout=60s
# Repeat this request before, during, and after the drill. Record status and replica.
kubectl --context kind-probes-lesson -n probes-lesson exec request-check -- node -e "fetch('http://tickets/tickets').then(async r => console.log(r.status, r.headers.get('x-replica'), await r.text()))"

# A marker simulates an unavailable dependency for ONE replica.
kubectl --context kind-probes-lesson -n probes-lesson exec deployment/tickets -- node -e "require('node:fs').writeFileSync('/tmp/database-down', '')"
# Wait at least 15 seconds, then inspect readiness, restarts, and endpoint conditions.
# No restart is expected. Save the affected Pod name from this output.
kubectl --context kind-probes-lesson -n probes-lesson get pods -l app=tickets
kubectl --context kind-probes-lesson -n probes-lesson get endpointslices -l kubernetes.io/service-name=tickets -o yaml

# Replace AFFECTED_POD with that name; do not let exec choose a different replica.
kubectl --context kind-probes-lesson -n probes-lesson exec AFFECTED_POD -- node -e "require('node:fs').unlinkSync('/tmp/database-down')"
# After recovery, simulate a local progress failure in that same replica.
kubectl --context kind-probes-lesson -n probes-lesson exec AFFECTED_POD -- node -e "require('node:fs').writeFileSync('/tmp/stuck', '')"
kubectl --context kind-probes-lesson -n probes-lesson describe pod AFFECTED_POD
# Expect a liveness failure, one container restart, startup gating, then Ready.
# Marker files live in the container writable layer and disappear on restart.

# Cleanup only this disposable cluster when observations are complete.
kind delete cluster --name probes-lesson

The browser model checks policy consequences under declared assumptions. HTTP fixture tests check endpoint responses. A cluster drill checks controller and traffic behavior. A passing result at one layer does not establish the others.

06Make the next call

Change the action only when the evidence warrants it.

The API can now serve its promised catalogue from a local snapshot during a database outage. Should readiness still fail solely because the database is unreachable?

Keep the operating rule: ask what this observation warrants, what the action can repair, and which evidence would show recovery. Revisit the rule when startup time, dependency behavior or the serving contract changes.

Practice with an audio-guide catalogue

Carry one configuration through a dependency outage, a larger startup index, and a restart investigation. Each level starts from the preceding reference configuration.

Kubernetes referencesProbe behavior checked 17 October 2026

See Liveness, Readiness, and Startup Probes for the decisions attached to each check, and Configure probes for HTTP behavior, fields and failure considerations. Use the documentation for your cluster version when adapting the drill.