Kubernetes Has Garbage Collection. Why Did We Build Another One?

This is article 2 of 10 in Beyond Kubernetes Defaults. Read article 1: When Kubernetes Primitives Aren’t Enough first. Later articles cover lease fencing, custody, admission and more.

Kubernetes already garbage-collects resources.

That makes “we built a Kubernetes garbage collector” sound suspiciously like reinventing a solved problem.

In many cases, it would be.

Kubernetes gives us several excellent lifecycle mechanisms: owner references, cascading deletion, Job TTLs, CronJob history limits and finalizers. If a resource has a natural owner or a single lifecycle rule, those mechanisms should usually be the first choice.

Our need appeared when cleanup stopped being a property of one resource and became policy across resource classes.

Native cleanup is good at structural lifecycle

Owner references are excellent when the relationship is structural: delete the parent and its children should disappear.

ttlSecondsAfterFinished is excellent when every completed Job can follow the same lifecycle.

CronJob history limits are excellent when the controller already owns the history set.

Finalizers are excellent when deletion must wait for an explicit cleanup action.

The key question is not “can Kubernetes delete this?” It obviously can.

The key question is “who owns the decision that this object is now disposable?”

Policy changes the problem

Consider a qualification environment that creates several classes of temporary resources:

• test Jobs;
• generated ConfigMaps;
• short-lived credentials or references;
• transient namespaces;
• evidence artifacts;
• dead-letter records;
• customer-owned resources.

They may all be old. They should not all be collected by the same rule.

A Job can be disposable after one hour while its evidence must remain. A generated ConfigMap can be safely deleted after a run, while a ConfigMap carrying customer configuration may be owned by another controller entirely. A dead-letter entry is old data, but age alone does not make it garbage.

This is the distinction that led to Zen Cleaner.

The useful abstraction was not “delete old Kubernetes things.” It was:

Evaluate an explicit cleanup policy across allowed resource classes, while preserving protection and ownership boundaries.

The safety property matters more than the deletion feature

A garbage collector can cause more damage than the resource leak it fixes.

The first design question should therefore be the negative one:

What must this collector never delete?

In our classification, evidence, DLQ/customer data and component-owned durable state do not become generic-GC targets merely because they are old. Their owning component remains responsible for retention and deletion semantics.

That turns protection into a first-class policy rather than an operator convention.

Why not encode everything in finalizers?

Finalizers are powerful, but they solve a different problem. They let an owning controller block deletion until it has completed a required action.

They do not naturally express a global policy such as:

completed qualification Jobs: collect after 2h
transient generated ConfigMaps: collect after 6h
resources with protection label: never collect
resources owned by customer-data controllers: out of scope

You can build a controller that interprets those policies using Kubernetes APIs — and that is effectively what the higher-level GC controller becomes.

The point is not that Kubernetes lacks extensibility. The point is that the policy is application/operator logic implemented on top of Kubernetes primitives.

Why not a cronjob with kubectl?

A shell script is tempting:

list old resources
-> grep timestamps
-> kubectl delete

It is also difficult to make safe over time.

The moment the policy requires selectors, protected classes, per-kind TTLs, bounded retries, reason codes, rate limiting or dry-run evidence, the script is becoming an undocumented controller.

At that point a controller with an explicit policy API is easier to test and reason about.

The rule we use

Use Kubernetes-native lifecycle whenever it is sufficient.

Add a policy controller only when at least one of these becomes true:

• lifecycle spans unrelated resource kinds;
• different classes require different TTLs;
• some resources require explicit protection;
• ownership must be checked before deletion;
• deletion decisions need durable/auditable reason codes;
• cleanup must be rate-limited or retried predictably.

If none of those are true, a custom GC controller is probably unnecessary.

Protection semantics are the real feature

A generic collector needs a way to distinguish “eligible for cleanup” from “safe to delete.” Those are not equivalent.

For example, age can be one input without being the authority. A resource can be older than its nominal TTL and still be protected because it represents evidence, an active investigation, a recovery checkpoint, or state owned by another controller.

A robust collector therefore needs an explicit protection model. Depending on the environment, that can be labels, annotations, policy objects, owner classifications or another bounded mechanism. The important property is that protection is inspectable and testable rather than tribal knowledge.

The negative cases are as important as the positive ones. The policy model is designed so that:

eligible + unprotected -> collect
eligible + protected   -> preserve
unknown class          -> ignore (fail closed)
foreign owner          -> preserve
malformed policy       -> do not broaden deletion

A collector must also behave well as an operational API: bounded concurrency, retry with backoff, rate limiting, idempotent handling of already-deleted objects, metrics for eligible/protected/deleted/failed resources, and reason codes explaining why an object was or was not collected.

These are not exotic features. They are the normal cost of turning a cleanup idea into a safe controller.

A useful ownership test

Before adding a resource class to generic GC, ask:

1. Who created it?
2. Who understands its semantic lifecycle?
3. Can generic age/policy safely decide it is disposable?
4. Does deletion destroy evidence, recovery state or customer data?
5. Is there already a better native Kubernetes lifecycle mechanism?

If the owning component understands the lifecycle better than a generic collector, the generic collector should stay out.

That restraint is one of the most important properties of a GC system.

Why this matters beyond Zen

The same pattern shows up in CI clusters, preview environments, ephemeral test infrastructure and internal platforms. Teams often begin with ad hoc cleanup scripts and eventually discover that the difficult part is not deletion — it is policy ownership.

Kubernetes provides the deletion machinery. A dedicated controller is justified only when the organization needs a reusable, explicit answer to “what is garbage?”

Later in this series we’ll look at desired-state authority and at the discipline of deciding what a controller should never manage — two boundaries that shaped this design.