This is article 6 of 10 in Beyond Kubernetes Defaults. Read article 5: A Running Pod Is Not an Admitted Workload first, or start with article 1: When Kubernetes Primitives Aren’t Enough. Later articles cover desired-state authority and more.

Single-process correctness can hide distributed-system bugs.

Replay protection is a particularly clean example.

The one-replica version

A signed request arrives with a signature, a timestamp and a nonce. The application verifies the signature, checks that the timestamp falls inside an acceptable window, and records the nonce so the same signed message cannot be accepted twice.

With one process — and all nonce state in that process’s memory or its local database — this can look correct.

It is correct for exactly one topology: one process that never loses state and never gains a sibling.

Add replica B

Now Kubernetes runs two healthy replicas behind one Service.

The first request lands on replica A. A records nonce N locally.

The replay lands on replica B.

B has never seen N. If replay authority is process-local, B accepts the exact same signed request.

Nothing about a Deployment or a Service turns local security state into shared state. Scaling is not state synchronization.

Sticky routing is not a security boundary

A tempting mitigation is consistent hashing or sticky sessions so the same nonce normally reaches the same Pod. Kubernetes Services can express some of this — for example sessionAffinity: ClientIP — and it can be operationally useful.

It becomes unsafe when a security guarantee silently depends on membership never changing during the replay window.

Pods restart. Pools scale up and down. Nodes disappear. Endpoints change. If membership movement can make a consumed nonce fresh again, sticky routing is only reducing replay probability, not preventing replay. A guarantee that breaks on the first reschedule is not a guarantee; it is a hope with a config key.

Define the invariant before choosing the mechanism

The useful statement is precise:

After the first successful use of a nonce, every later use inside the replay window must be denied, regardless of which live replica receives it.

Several mechanisms can satisfy that invariant, with different trade-offs:

a shared replay store every replica consults;
replicated nonce state with safe transfer on membership change;
stable partition ownership, where each key has exactly one current owner;
a cryptographic protocol that removes the need for shared replay state.

Whichever you choose, three properties need explicit decisions:

Scope. A nonce is not globally unique by accident of formatting. Uniqueness is defined per scope — typically per tenant or per source channel — and the stored key must include that scope, or one tenant’s traffic can collide with, or clear, another’s. Tenant scoping is part of the key, not a filter applied afterward.

Atomicity. A shared store is not automatically sufficient. Check-then-record done as two steps races: two replicas can both look, both see nothing, both accept. The test-and-record must be one atomic operation — a compare-and-set or a uniqueness constraint the store itself enforces.

Lifetime. The replay window is bounded by the timestamp tolerance, and the nonce state should have a matching TTL: entries older than the window can be pruned. But expiry rules must survive membership changes the same way acceptance does — a pruned-on-replica-A entry must not resurrect as fresh on replica B, and the window must be sized so clock skew between replicas cannot legally replay inside it.

Durable acknowledgement has the same shape

A related single-process habit breaks the same way: returning success before durable acceptance.

receive
-> enqueue in process memory
-> HTTP 200
-> process dies

Kubernetes restarts the Pod. The accepted event is gone. The sender believed it was delivered.

For a reliable delivery system, a successful acknowledgement should mean the event crossed the declared durability boundary — persistence that at least one surviving component can recover — before the 200 leaves the building. In an HA topology that boundary is usually outside the single process. If the durable authority is briefly unavailable, failing closed with a retryable response preserves a stronger property than acknowledging an event that may later vanish: no sender is ever told “done” for something that is not durable.

HA qualification means hostile cross-replica tests

A second replica changes the correctness model for more than replay protection:

replay protection and nonce state;
idempotency keys;
leader-only operations;
local queues and in-memory buffers;
SQLite or other node-local state;
rate limits;
caches;
retry ownership.

“Kill a Pod and see another Pod stay up” tests availability. It says nothing about whether guarantees remain true once state is distributed across processes. That requires hostile cross-replica tests — traffic aimed at the seams deliberately.

Membership change belongs in those tests. Even a shared-pool design can look correct if tests never resize the pool:

1. send signed request with nonce N -> ACCEPT
2. replay N to another replica      -> DENY
3. remove the replica that recorded N
4. replay N                         -> DENY
5. add a replacement replica
6. replay N                         -> DENY
7. scale the pool up and down
8. replay N                         -> DENY

If a new replica starts with an empty replay view, every membership change has reopened the window. That is what state transfer or shared authority is for.

Fail-closed beats silently-open during transitions

Suppose replay authority is briefly unavailable mid-transition. Two behaviors are possible:

A. accept without the replay check
B. refuse/retry until replay authority is available

For a security-sensitive signed ingress path, B is usually the safer default: a short retryable availability window is recoverable, a silently accepted replay is not. This is an explicit product trade-off — latency and availability against enforcement completeness — and it should be a measured decision, not whatever the error path happens to do.

Duplicate authority appears in more places than replicas

A delivery topology can accidentally hold several: the ingestion side retries, the delivery side retries, and two dead-letter queues record the same logical failure. Each component behaves reasonably on its own, while the end-to-end result is duplicated attempts and ambiguous terminal state.

We prefer one logical failure owner per topology: a single component is authoritative for “this acceptance failed terminally,” and retries hang off that decision rather than sprouting independently at each layer.

Arm idempotency at the right boundary

One more ordering trap: marking an event idempotently complete before the downstream effect has actually succeeded. If the process records “seen” and then the target call fails, the retry that would have delivered the event is suppressed — a permanent miss disguised as deduplication.

The contract has to define the order — durable acceptance, attempt, downstream result, terminal state — such that a crash at any point cannot convert an incomplete delivery into a permanent duplicate suppression.

What this buys — and what it does not

None of this adds up to exactly-once delivery. At-least-once transport plus idempotent, correctly-ordered consumption is the honest combination: deduplication shrinks duplicate effects at the boundaries you control, and the remaining duplicates are handled by consumers rather than wished away. Claims beyond that need evidence, not adjectives.

A practical review checklist for your own HA topology:

Is the dedup key scoped (tenant/source) and is that scope part of the stored key?
Is check-and-record atomic under concurrent access from two replicas?
Does the nonce/replay state outlive membership changes, with a TTL matched to the signature window?
Is any security property silently relying on sticky routing?
Does an acknowledgement imply durable acceptance, and is the durability boundary outside the single process?
Is exactly one component authoritative for terminal failure and retry ownership?
Do your tests include replay across replica replacement and pool resize — not just pod-kill?

A second replica is an availability feature and a correctness question at the same time. The question is not whether the service stays reachable. It is whether its guarantees stay true once its state lives in more than one place.