This is article 5 of 10 in Beyond Kubernetes Defaults. Read article 4: Kubernetes Secrets Are Distribution, Not Custody first, or start with article 1: When Kubernetes Primitives Aren’t Enough. Later articles cover deduplication, desired-state authority and more.

Kubernetes gives us several useful states for a workload: scheduled, Running, Ready, unhealthy, terminating.

Those states are operationally important, and they are precisely defined. They are also narrower than many application state machines.

The misleading green Pod

It is a common pattern — illustrative, not a report about any specific system — to meet a workload that is Running at the Kubernetes level while still unable to participate safely in its application.

The process exists. The container started. A health endpoint answers.

But useful participation can still depend on facts Kubernetes has not been told about:

enrollment into the application having completed;
workload identity being valid and current;
tenant and placement binding being correct;
current credentials being available;
a remote admission decision having succeeded;
desired state having arrived and been applied;
session or heartbeat state being current;
recovery state being available if the process restarts.

The process was healthy. The workload was not yet admitted.

Health, readiness, admission, authority and convergence are different

A more precise model separates five questions:

Liveness
  Is the process functioning well enough that a restart is not warranted?

Kubernetes readiness
  Should this Pod be eligible for Service traffic?

Application admission
  Has this workload joined the system it belongs to, correctly?

Application authority
  May it perform this class of operation now?

Convergence
  Has it applied the state and generation it is expected to have?

Those answers can legitimately differ.

A follower replica in a replicated controller can be healthy and authenticated but not authoritative. An agent that lost its management connection can be alive but unable to receive work. A runtime whose credential expired can be reachable but should fail closed rather than serve.

What the probes actually measure

Kubernetes probes are well-defined, and it is worth being precise about them. The probe documentation describes three kinds:

A startup probe gates the others: liveness and readiness checks do not run until it first succeeds, which protects slow-starting containers from being killed or marked unready during initialization.
A failing liveness probe causes the container to be restarted (subject to the pod’s restart policy).
A failing readiness probe does not restart anything. It makes the Pod ineligible for a normal Service: the endpoints controller removes it from the matching Service’s endpoint list until readiness returns. (Exact routing behavior depends on how traffic reaches the Pod — headless Services, direct Pod access and some ingress setups do not consult readiness the same way.)

These are process and traffic answers. None of them, by themselves, tells you whether the workload has been enrolled, whether its credential is current, or whether it has applied the expected generation.

That is not a platform limitation. Kubernetes can express more: the API supports custom Pod conditions and readiness gates, so extra admission signals can participate in the Ready computation, and Status conditions can carry structured state. API-side admission — admission controllers, Pod Security Admission — validates objects at creation time and answers a different question again: whether this Pod spec is allowed to run at all, not whether the running application has joined its system.

The observation is about design, not platform capability:

Define application readiness first. Then decide how probes, conditions and status should reflect it.

Do not start with a probe endpoint and let its boolean become the architecture.

One boolean often loses the reason

When several conditions collapse into Ready=true|false, operators lose the cause.

We prefer bounded, semantic reason codes alongside the boolean, such as:

disconnected from the management plane;
heartbeat stale;
credential expired or revoked;
desired generation ahead;
recovery state unavailable;
retiring;
authorization no longer valid.

Kubernetes conditions and readiness gates give you a place to put some of this; metrics and status APIs can carry the rest. The point is not the mechanism — it is refusing to let a generic “not ready” be the only observable state.

Do not import the wrong health into the wrong layer

The opposite failure mode is making one component’s readiness depend on everything downstream.

A lifecycle or control component whose job is to stay up and manage state should not necessarily become not-ready because a customer delivery target is returning HTTP 500. That is application-layer state, not lifecycle state — and restarting the management process would not heal the target. Conversely, a component whose responsibility is delivering to that target may be exactly the component that should report itself not ready until the target recovers.

There is no universal rule. There is an ownership rule:

Each layer derives readiness from the facts it owns.

This keeps one red dependency from turning the whole system red while hiding which authority actually failed.

Startup ordering is not semantic readiness

Another trap: assuming that because init containers completed and dependencies are reachable, the workload is ready for its application role.

Admission into a distributed system can require remote facts that arrive after process startup:

identity issued
-> trust verified
-> binding validated
-> desired state delivered
-> generation applied and admitted
-> workload ready for its application role

That sequence can legitimately take longer than container initialization. Encoding it explicitly avoids the race where the Service starts sending work as soon as the process opens a port.

Readiness should survive restarts predictably

A semantic readiness model also makes restart behavior testable.

An ordinary restart should preserve the workload’s existing identity through valid, durable recovery: the checkpoint is intact, the credential is still current, and the workload resumes its role without re-applying for membership. Re-enrollment is a different, explicit lifecycle action — an authorized transition with its own audit trail — not a repair path to fall back on when recovery fails. If the checkpoint is corrupt, the workload should fail closed and say so rather than silently present itself as a fresh install; if the credential was revoked while the Pod was down, process health must not override the revocation.

This turns restart tests from “the Pod returned to Running” into real assertions:

identity is continuous across the restart, with re-enrollment only as a distinct authorized action — never as a silent fallback, and never via bootstrap replay or exported credentials;
no generation rollback;
no resurrection of disabled or retired state;
no leader-only authority exercised while the replica is a follower;
a bounded reason code is exposed while convergence is incomplete.

A short illustrative scenario

A queue worker behind a Service: image pulled, init containers done, /healthz returning 200, readiness probe green. The Service begins sending it events. The worker accepts one, then discovers its session credential has expired while it was starting — a fact none of its probes measured.

With a single boolean, the incident looks like “Pod is Ready but errors under load”, and the instinct is to restart it.

With layered state, the worker reports credential expired, exits its traffic-eligible state through a readiness gate or its own status, and the diagnosis is visible before anyone touches the Pod: restart will not help; re-issuance will.

Separate observation from eligibility

Sometimes you want a degraded component to remain observable even though it should not receive traffic.

Kubernetes gives you the pieces: readiness for Service eligibility, separate endpoints for observation, metrics, Status conditions, and distinct Services with different selectors. A common split:

/healthz  -> process can be observed
/readyz   -> eligible for this Service's traffic
status    -> richer admission/convergence state

The exact endpoint names matter less than refusing to make one boolean carry every operational meaning.

The diagnostic payoff

“Pod not Ready” is a symptom.

“Credential revoked,” “heartbeat stale,” “desired generation ahead,” or “recovery state unavailable” is a diagnosis.

A practical checklist for your own workloads:

Does the liveness endpoint answer only process questions?
Does readiness derive from facts this component owns — not from the health of everything downstream?
Are admission and convergence states observable with bounded reason codes (conditions, metrics, status), instead of collapsed into one bit?
Is recovery from restart distinguishable from re-enrollment, in state and in audit?
Does a downstream failure change only the layers that own it?
Can an operator tell “process down” apart from “admitted but not authoritative” from a dashboard?

Running and Ready are not meaningless — they are precise about a scope you should understand. The mistake is letting their scope quietly become the whole model. Define what admission and authority mean for your application, expose them as first-class state, and let Kubernetes probes reflect that model instead of replacing it.