Reconciling Everything

Queues. The backbone of distributed systems, our old friends that we can rely on, and the cause of a lot of grief and on-call worries as they inevitably back up, overflow, replay, or duplicate items.

There is a different (and sometimes better) way to build distributed systems, though - the reconciliation loop, a system where stateless programs talks to a central datastore and try to progress the state in small, incremental actions.

We'll take a look at what reconciliation loops are, exactly, how they compare to both queues and other distributed system messaging options, and then dive into their active use as part of the Takahē ActivityPub/Fediverse server - and see the good, the bad, and the strange behaviours that can result.