Kubernetes Secrets Are Distribution, Not Custody

This is article 4 of 10 in Beyond Kubernetes Defaults. Read article 3: A Kubernetes Lease Is Not a Fencing Token first, or start with article 1: When Kubernetes Primitives Aren’t Enough. Later articles cover workload admission and more.

Kubernetes Secrets are not the villain of this article.

They are a useful API for delivering sensitive configuration to workloads, and they can be protected with encryption at rest, RBAC, admission controls and careful operational practice.

The mistake is asking a Secret to answer security questions that are outside its job.

The overloaded-secret problem

A system often starts with a simple pattern:

create credential
-> store credential in Secret
-> mount Secret into Pod
-> possession means caller is trusted

That can work.

Over time, however, “the Secret” can begin representing several different concepts simultaneously:

• identity;
• authorization;
• enrollment state;
• key custody;
• tenant context;
• rotation generation;
• recovery state.

When those concepts collapse into possession of one blob, failures become difficult to classify.

Does a lost Secret mean identity loss? Does rotation require re-enrollment? Can a copied Secret impersonate another replica? Is the key allowed to leave the component? Which authority can issue a replacement?

Distribution is not custody

Custody is a stronger statement than storage.

A custody boundary can be designed so that a private key is never exported to the caller at all. The caller requests an operation, proves its own identity and authorization, and receives a result—not the underlying private key.

That changes the compromise surface.

Instead of every workload filesystem or mounted Secret becoming a copy of the durable identity key, the component can hold only the material it needs to authenticate to the custody service.

The exact design depends on the threat model, but the conceptual separation is valuable:

Secret distribution
    != private-key custody
    != workload identity
    != authorization

What our own enrollment contract says

To make this concrete: our enrollment security contract for production workloads is a candidate document, explicitly marked as not yet qualified. It states the custody law we are building toward, and it is deliberately honest about its own status—we are not claiming that every live workload already conforms to it today.

What the contract requires for enrolled production workloads is unambiguous, though:

• credentials for enrolled workloads are held in authorized, encrypted, CRD-backed custody, not Kubernetes Secret backends;
• enrollment signing uses a non-exporting custody interface—the signing key is an architectural requirement, not an exportable file with a policy suggestion attached;
• the forbidden list is explicit: Secret volumes, secretKeyRef/envFrom injection for enrolled workload credentials, legacy Secret sync, and local-key fallback are all out, with no selectable weaker adapter;
• private key material must not be serialized into enrollment transport or left in durable plaintext copies, and must stay out of logs, receipts, environment variables and error strings.

The non-exportable part is worth stressing as a design requirement: it is a capability the custody architecture is required to provide, not a property we claim every current deployment already has. Some of the migration work is still open, and the candidate status exists precisely so that gap is stated rather than papered over.

Identity should survive operational changes cleanly

Separation also buys clean recovery semantics.

A Pod restart, Pod replacement, local state loss and full re-enrollment are different events.

If the architecture treats them all as “recreate the Secret,” it becomes easy to accidentally turn routine recovery into identity replacement—or to restore stale authority without noticing.

We prefer explicit recovery states and generation checks so a corrupted checkpoint is not silently treated as a fresh install.

That is especially important in replicated systems where each replica should hold an independently issued key rather than copying another replica’s private material.

Authorization is separate again

Even a perfectly protected key does not answer:

Is this workload allowed to perform this operation now?

A valid workload identity can be a follower rather than a leader. It can belong to the correct service but wrong tenant scope. It can be valid cryptographically but stale for the current generation.

This is why our architecture keeps custody, workload identity and authorization as distinct boundaries.

Kubernetes Secrets still have a role

Kubernetes Secrets remain a reasonable choice for plenty of platform and application configuration where distribution is the actual job.

But they are not the governed credential path for enrolled production workloads. In our architecture, that role belongs to the custody boundary described above: credentials are issued into encrypted CRD-backed custody, consumed through authorized projection, and never exported to the workload as a copyable Secret.

The rule is not “never use Secrets.”

The rule is:

Do not let the convenience of Secret distribution decide the rest of your security model by accident.

If a private key must never leave custody, design a custody boundary. If a workload needs a cryptographic identity, give it an identity protocol. If an action needs authorization, verify authorization explicitly.

Kubernetes Secrets can participate in that design without being forced to become all of it.

A useful threat-model question: what can the compromised workload extract?

Suppose an attacker compromises the application container.

If the durable identity private key is mounted into that container, the attacker may be able to copy it and continue using it elsewhere until revocation or expiration.

If the workload instead talks to a narrow custody interface, the attacker may still misuse the workload while the compromise is active, but extracting the long-lived key is designed to be substantially harder—or impossible, because the key never exists where the attacker is standing.

That does not eliminate the need for authorization, rotation or incident response. It changes the blast radius of credential theft.

Rotation becomes clearer when custody is separate

A rotation protocol can then answer explicit questions:

Which identity is rotating?
Which generation is current?
What proves possession/authority?
What state is persisted for restart?
Can an old generation still sign?
What happens if the process crashes halfway through activation?

Those are application/security protocol questions. A Secret update alone cannot define all of them.

We found it useful to model activation as a crash-consistent transition rather than “write new files and hope startup sorts it out.” A restart should either recover one coherent generation or fail closed.

Channel credentials and identity keys can have different treatment

Not every secret deserves the same custody mechanism.

A short-lived channel credential used to authenticate to a local security service may reasonably be distributed differently from the durable private key representing workload identity.

This is another reason not to make “Secret or no Secret” the architecture question.

Classify the secret by purpose, lifetime and consequence of extraction.

When Kubernetes Secrets are enough

For many applications, Kubernetes Secrets plus good cluster security are entirely appropriate.

A dedicated custody service becomes more compelling when:

• private keys should never be exportable to application workloads;
• several components need controlled cryptographic operations;
• rotation/recovery needs strong generation semantics;
• audit/evidence of key usage matters;
• the blast radius of copied key material is unacceptable.

Additional machinery should follow the threat model, not fashion.