The Principle of Reconciliation

Reconciling the world not as it is, but as it should be.

TL;DR

At the heart of resilient systems lies a deceptively simple feedback loop: observe reality, compare it against an ideal, and act to align the two. This principle, reconciliation, is what makes systems self-healing, adaptable, and robust in the face of change and failure.

Desired vs. Actual: A Systemic Tension

Every resilient system embodies a tension: the difference between the desired state and the actual state. This delta isn’t just a bug, it’s the whole point. Reconciliation exists because divergence is inevitable. But more importantly, because it’s recoverable.

We often talk about reconciliation in infrastructure terms, but it's more fundamental than that. It’s a control loop. It's Kubernetes. It's your thermostat. It's the immune system.

From Roots in Kubernetes

Reconciliation has deep roots in the design of Kubernetes, which introduced a powerful pattern: controllers continuously observe actual system state and work to make it match a declarative desired state. This is not just configuration management, it’s resilience as architecture.

The genius of the Kubernetes controller pattern lies in its simplicity:

1. Watch: Observe relevant state.

2. Compare: Compute the delta between desired and actual.

3. Act: Make changes to move actual toward desired.

And then repeat. Forever.

Workqueues: The Backbone of Reconciliation

Central to this architecture is the workqueue. It decouples events from processing, ensuring that:

• Work is retried until it succeeds.

• Transient errors don’t result in lost updates.

• Concurrency is bounded and controlled.

The workqueue doesn’t hold events, it holds keys. This subtle difference enables level-based reconciliation. Rather than reacting to events (edge-triggered), we simply treat the events as just another nudge to check that the entire state of the world referenced by our key is in its desired state (not just whatever change prompted the event). This shift in how we think about processing data is what makes the system idempotent and resilient. The same input key can be processed repeatedly with no side effects when no change is needed.

Idempotency and No-Ops as a Feature

A well-architected reconciler leans hard on idempotency: doing the same thing twice has the same effect as doing it once. This makes reconciliation loops safe to retry and compose, but it also makes the efficiency of “no-op” reconciliations paramount.

Segmenting Desired State: Layers of Abstraction

In Kubernetes, we often model complex systems as a stack of controllers, each owning a level of abstraction. One controller reconciles a Custom Resource Definition (CRD), another reconciles lower-level Deployment or StatefulSet objects, and yet another manages Pods.

This layering of desired state enables:

• Separation of concerns.

• Reuse of common abstractions.

• Modularity and composability.

Each layer has its own desired and actual state, forming a nested chain of reconciliations. Together, they converge toward an eventual system-wide desired state.

This technique of segmentation can be used to help make complex reconciliation loops simpler and partition “expensive” aspects from other facets that are less expensive or “free.”

Why This Matters

Reconciliation is the foundation of systems that survive change. Systems that react, not just to inputs, but to reality. It’s what allows infrastructure to adapt, self-heal, and maintain integrity without manual intervention.

Like immutability, ephemerality, and minimalism, reconciliation is a principle that scales. It’s the scaffolding on which we build resilience.

This principle is central to how we build and distribute container images at Chainguard. From making sure that we have built and published images with all of the latest updates, to making sure our customers have received all of the images to which they are entitled.

Closing Thought

If we accept that systems will drift, then reconciliation is not optional, it’s essential. And the best reconciliations don’t just react to change; they are resilient in the face of change.