← All posts

When Kubernetes Primitives Aren't Enough: Production Guarantees We Had to Build Above K8s

Kubernetes gives us excellent primitives for scheduling, leases, secrets, health checks, networking and reconciliation. Building a distributed delivery system taught us where those primitives stop—and where application-level guarantees have to begin.

When Kubernetes Primitives Aren’t Enough

Kubernetes is one of the best pieces of infrastructure engineering we use.

It schedules workloads, reconciles desired state, exposes health probes, provides a Lease API for leader election, stores configuration and Secrets, applies network policy, and gives controllers a powerful declarative model.

But while building Zen Mesh, we kept encountering the same category of mistake:

A Kubernetes primitive was being treated as if it guaranteed more than it actually did.

That is not a Kubernetes defect. In most cases, Kubernetes is deliberately providing a lower-level mechanism and leaving application semantics to the application.

The difference becomes important when a system has to answer questions such as:

  • Which replica is authorized to act, not merely elected?
  • Does returning HTTP 200 mean an event can survive a process crash?
  • Is a running Pod actually admitted into the application trust model?
  • Does a Secret being mounted mean the workload is the correct identity?
  • Can replay protection survive a request landing on another replica?
  • Who owns a field when desired, observed and applied state disagree?
  • Can we prove that the container running is the exact container we qualified?

These are the guarantees we had to make explicit.

This article is the first in a series about those boundaries: what Kubernetes gives us, where the guarantee stops, and what we built above it.

1. Garbage collection: deletion is easy; policy is harder

Kubernetes already has several good cleanup mechanisms.

Owner references let child resources disappear with their owner. Jobs can use ttlSecondsAfterFinished. CronJobs have history limits. Finalizers let controllers delay deletion until cleanup is complete.

For many systems, those primitives are exactly enough.

Our problem appeared when cleanup crossed resource classes and safety boundaries.

A qualification Job may be disposable after a short TTL. A generated ConfigMap may have a different lifecycle. An abandoned temporary environment may be best handled by destroying the whole environment. Evidence, a dead-letter record, or customer data should never enter a generic cleanup policy merely because it is old.

That changes the question from:

“How do I delete Kubernetes objects?”

into:

“Which classes may be collected, under what policy, with which protections, and who is allowed to decide?”

That was the motivation for Zen GC: not replacing Kubernetes garbage collection, but adding a policy layer where native lifecycle mechanisms are too narrow.

The design rule is intentionally conservative: use native Kubernetes cleanup when it is sufficient. Use a higher-level collector only when the lifecycle rule is genuinely cross-resource or policy-driven.

2. Leader election: holding the Lease is not the entire authority model

Kubernetes Lease objects and controller-runtime leader election are useful and well understood.

They answer an important question:

Which participant currently holds leadership?

Our harder question was:

How does every authoritative operation prove that an old leader is no longer allowed to act?

Imagine two replicas, A and B.

A is the leader at generation 12. A loses leadership. B becomes leader at generation 13. A is still alive long enough to execute one delayed reconciliation.

If application code only asks, “Was A once a valid authenticated replica?”, the answer is yes.

That is not sufficient.

The application needs a fencing rule such as:

A / leadership generation 12 -> DENY
B / leadership generation 13 -> ACCEPT

If A later becomes leader again, that should happen under a newer authority generation—not by reviving stale authority.

This led us to separate several concepts that are easy to accidentally collapse:

  • workload authentication;
  • leadership selection;
  • leadership generation;
  • session generation;
  • credential generation;
  • application authority.

A follower can have a perfectly valid identity and still be forbidden from performing leader-only operations.

Kubernetes solves leader selection. The application still has to solve stale-authority rejection where that matters.

3. Secrets: storage and distribution are not custody

Kubernetes Secrets are useful. We use them where they are appropriate.

The dangerous leap is treating “the Secret exists and is mounted” as a complete security architecture.

A Secret answers questions about storing and distributing sensitive bytes. It does not, by itself, answer:

  • Should this private key ever leave a custody boundary?
  • Is possession of the key equivalent to workload identity?
  • Who is authorized to use the key?
  • What happens during rotation?
  • What happens after local state loss?
  • Can a stale workload continue using old authority?
  • Can we prove which identity performed the operation?

Earlier versions of our architecture relied more heavily on possession of shared HMAC material and Kubernetes Secret distribution. That model can be secure when keys are random, scoped, rotated and protected, but it also makes secret possession carry a lot of semantic weight.

Our newer design separates those responsibilities.

Zen Lock is a custody boundary. Workload identity is a separate concern. Authorization is a separate concern. Enrollment and recovery are separate protocols.

The important lesson is not “Kubernetes Secrets are insecure.” It is:

Secret distribution, key custody, workload identity and authorization are different security problems.

Collapsing them makes rotation and recovery much harder to reason about.

4. Pod readiness: a process can be healthy and the workload can still be unusable

Kubernetes probes are process- and service-health primitives.

They are not automatically application admission proofs.

We saw this clearly in our own runtime history. An ingester or egress process could be up, its container could be running, and its HTTP health endpoint could respond—while the workload was still not functionally ready to participate.

Application readiness depended on facts such as:

  • identity being valid;
  • enrollment/admission having completed;
  • tenant and Plane binding being correct;
  • credentials being current;
  • heartbeat state being fresh;
  • desired state being received;
  • observed and applied generations being compatible;
  • required persistence being available.

Those states deserve their own semantics.

A useful mental model is:

process alive
    != service reachable
    != workload authenticated
    != workload admitted
    != workload authoritative
    != workload ready for customer traffic

Kubernetes probes remain important. We simply stopped asking them to represent every higher-level state.

5. Replicas: high availability can expose correctness bugs

Adding a second replica often improves availability while exposing assumptions that were invisible with one process.

Replay protection is a good example.

Suppose an ingress request contains a valid signature and nonce. Replica A accepts it and stores that nonce in an in-memory or local database cache.

If the same signed request is replayed to replica B, what prevents B from accepting it?

Nothing—unless replay state is shared, replicated, partitioned with a safe ownership model, or the routing guarantee itself is strong enough to make that impossible.

We found this class of defect during pre-production qualification of our Traffic runtime.

The fix was not “use more Kubernetes.” The missing guarantee was application-level:

Once a nonce has been successfully consumed, no live replica may accept it again inside its replay window.

The same principle applies to idempotency, retries and durable acknowledgements.

A Deployment with multiple replicas gives us multiple processes. It does not automatically give us one correctness authority.

6. HTTP 200: acknowledgement is a contract, not a status code

Another useful example is durable acceptance.

A system can easily do this:

receive event
-> put event in memory
-> return HTTP 200
-> process dies
-> event disappears

Kubernetes may restart the process exactly as designed. The event is still gone.

The infrastructure behaved correctly. The application contract did not.

We therefore had to define what successful acknowledgement actually means.

For a delivery system, a useful invariant is:

If we return success, the event has crossed the declared durable acceptance boundary.

The exact durability class can vary by topology. A single-node edge installation may have a narrower failure model than a replicated runtime. But the response must not imply durability the architecture does not provide.

This is a recurring theme in distributed systems: infrastructure recovery and application correctness are related, but they are not interchangeable.

7. Desired state: reconciliation needs explicit ownership

Kubernetes made declarative desired state mainstream, and that is one of its biggest contributions.

But complex systems often have more than one meaningful state:

  • desired state: what the authority wants;
  • observed state: what the remote/runtime reports;
  • last-applied state: what an authorized controller knows it successfully applied.

Without explicit ownership, those fields can blur together.

Who owns generation 8? Can the runtime advance it? Can an old controller write generation 7 after reconnect? What does “drift” mean if desired, observed and applied state are all stored in different places?

Our newer state model makes the separation explicit and uses generations, digests, connection state and bounded reason codes to classify convergence.

The important rule is simple:

Every authoritative field should have one owner.

Kubernetes reconciliation is the mechanism. Application state authority is the contract layered on top.

8. NetworkPolicy: reachability is not identity

NetworkPolicy is extremely useful for reducing who can talk to whom.

But packet reachability does not tell an application who the caller is.

A strong internal boundary may need both:

NetworkPolicy:
    is this network path allowed?
Workload identity / mTLS:
    which cryptographic principal is calling?
Authorization:
    is this principal allowed to perform this operation now?

We use those as complementary controls rather than substitutes.

That distinction matters during failover and rotation. A network path can remain valid while a credential becomes stale. A workload can possess a valid identity while being a follower. A component can be reachable but not authorized for a tenant or operation.

Again, Kubernetes is doing its job. The application has to supply the semantic layer.

9. Container tags: deployability is not provenance

Kubernetes will happily deploy an image tag.

That is convenient for development.

It is weaker as qualification evidence.

If we test service:candidate, the tag can later point at different bytes. The label is the same while the artifact changed.

For release qualification, we instead care about a chain like:

source revision
-> built image
-> registry digest
-> deployed image reference by digest
-> runtime image ID

The runtime should be able to prove it is executing the artifact that was actually qualified.

This is also why shortcuts such as preloading an image directly into cluster nodes can be dangerous in a release test: the system may appear healthy while bypassing the same registry/provenance path production will use.

Tags are excellent names. Digests are better evidence.

10. The most important boundary: some problems are not Kubernetes problems

There is a trap at the other extreme: once you start finding places where infrastructure primitives are narrower than application guarantees, it becomes tempting to call every application problem a Kubernetes gap.

That would be wrong.

Kubernetes should not be expected to understand:

  • our tenant authorization model;
  • what a Flow revision means;
  • whether a customer event is durably accepted;
  • our provider-specific authentication contract;
  • our sovereignty policy;
  • what evidence a customer needs to verify a delivery.

Those belong in the product architecture.

The goal is not to replace Kubernetes.

The goal is to know precisely where its responsibility ends.

A practical rule we now use

When we encounter an infrastructure primitive, we ask four questions:

  1. What does this primitive actually guarantee?
  2. What stronger guarantee does our product need?
  3. Which component owns that stronger guarantee?
  4. How do we test the failure case, not just the happy path?

That discipline has influenced how we build garbage collection, leadership fencing, key custody, workload admission, replay protection, durable event acceptance, state reconciliation and artifact qualification.

Kubernetes remains the substrate underneath all of it.

The lesson is not that Kubernetes falls short.

The lesson is that a production system becomes easier to trust when it stops asking infrastructure primitives to carry application semantics they were never designed to own.

This is the first article in the Beyond Kubernetes Defaults series. The next article examines a deceptively simple question: if Kubernetes already has garbage collection, why would we build another garbage collector?