Skip to main content

Key Rotation

How to rotate the age master key with zero pod downtime, using zen-lock's dual-key grace window.

The Dual-Key Model​

Rotation can't be instantaneous: ZenLock resources in Git and in the cluster are encrypted with the old public key, and they must all be re-encrypted with the new one before the old key can be destroyed. zen-lock handles this with a two-key decrypt state machine:

  • Active key — the new key; all new encryption uses it
  • Previous key — the old key, kept only for decrypting not-yet-re-encrypted ZenLocks

While both are configured, the webhook tries the active key first and falls back to the previous one (a "grace hit"). A leader-elected controller re-encrypts every ZenLock with the active key in the background. When no grace hits remain, the previous key is safe to remove.

Rotation phases: idle → rotating → idle (with a deprecated marker when the previous key is removed).

Step by Step​

1. Generate the new keypair​

zen-lock rotate --previous-key /path/to/current.key --output /tmp/rotation/
# writes:
#   /tmp/rotation/age-identity   (new active private key)
#   /tmp/rotation/age-previous   (old private key)
#   /tmp/rotation/rotation.yaml  (timestamp + both public keys — commit this)

2. Load both keys into the cluster​

kubectl create secret generic zen-lock-master-key \
  --namespace zen-lock-system \
  --from-file=key.txt=/tmp/rotation/age-identity \
  --from-file=age-previous=/tmp/rotation/age-previous \
  --dry-run=client -o yaml | kubectl apply -f -

kubectl rollout restart deployment/zen-lock-webhook -n zen-lock-system

Both webhook and controller now run in dual-key mode (ROTATION: rotating in kubectl get zenlock -o wide).

3. Wait for re-encryption​

The rotation controller (leader replica only) re-encrypts every ZenLock with the active key:

kubectl get zenlocks -o wide
# NAME             PHASE   ROTATION   GRACEHITS   ...
# db-credentials   Ready   rotating   2

Monitor progress with metrics (see Operations) — the signal to watch is:

zenlock_rotation_safe_to_deprecate == 1

meaning zero grace hits across the fleet and re-encryption complete.

Update Git too

Re-encryption changes ciphertext in the cluster's ZenLocks. If you manage ZenLocks in Git, re-encrypt and commit the updated manifests during this window (encrypt with the new public key), so your repo doesn't resurrect old-key ciphertext on the next sync.

4. Deprecate the previous key​

zen-lock deprecate --namespace zen-lock-system

The command verifies the active key is the one you expect and prints the exact kubectl commands to drop age-previous from the Secret. After removing it and restarting the webhook, ROTATION returns to idle.

5. Rollback (if something goes wrong mid-rotation)​

If re-encryption hasn't completed and you need the old key as primary again:

  1. Restore the old key as key.txt and the new key as age-previous (swap active/previous) — this keeps dual-key decrypt working in both directions
  2. If rotation had completed, simply restore the old key alone as key.txt; every ZenLock was already re-encrypted with the new key though, so prefer completing forward over rolling back

What Rotates and What Doesn't​

Rotated by this procedureNot rotated
The age master key (and thus all ZenLock ciphertext)Individual secret values — rotating a credential means encrypting a new value into the ZenLock
CSI provider key (same Secret)The custody key used by the retrieval server — separate key, rotated independently

Suggested Alerts​

  • ZenLockRotationStalled — in rotating phase but no re-encryptions succeeding
  • ZenLockRotationTooLong — more than 72 hours in rotating

Both rules ship in deploy/prometheus/prometheus-rules.yaml in the zen-lock repository.