Skip to content
go-sequentCQRS + Event Sourcing for Go financial systems

Operations

PostgreSQL failover runbook

Keep a single writer for the go-sequent event store through a PostgreSQL failover: routing rules, a six-step checklist, application guidance and what the library does not do.

PostgreSQL failover runbook

Goals

  • A single-writer primary for the event store, with no silent split-brain.
  • RPO close to zero when you use synchronous commit or a quorum. That is an operations choice.
  • Projection reads use replicas only where asynchronous lag is acceptable.

Where each path should connect

Path Recommendation
Command plus SyncInTxn projectors Always the primary. Balances need read-your-writes.
Async outbox consumers The primary, or a lag-tolerant replica for non-ledger views.
Replay and rebuild workers The primary, or a dedicated rebuild role with a consistent snapshot.

Use Patroni, Cloud SQL HA, RDS Multi-AZ or an equivalent. Put PgBouncer or your cloud pooler in front to absorb connection storms.

Failover checklist

  1. Detect primary loss through Patroni, cloud health checks or missing heartbeats.
  2. Promote a synchronous standby. Confirm pg_is_in_recovery() returns false on the new primary.
  3. Repoint the pooler or DSN, or update service discovery. Do a rolling restart of application pods if the DSN is sticky.
  4. Verify writes. Run a go-sequent append and one SyncInTxn projector update.
  5. Verify there are no dual writers. The old primary must not accept writes. Use fencing or STONITH.
  6. Rebuild standbys from the new primary and watch replication lag.

Application guidance

  • Keep database transactions short. One CommandService.Execute batch is one transaction.
  • On ErrOptimisticConcurrency, retry with backoff. These errors are common during catch-up storms after a failover.
  • Do not share a UnitOfWork across goroutines.
  • Health checks should degrade to read-only only if the SyncInTxn path is not required for that probe.
for attempt := 0; attempt < 5; attempt++ {
    err := app.Service.Execute(ctx, cmd)
    if !errors.Is(err, eventstore.ErrOptimisticConcurrency) {
        return err
    }
    time.Sleep(backoff(attempt))
}

Retrying is safe when each command carries an idempotency key. See Idempotency.

What go-sequent does not do

  • There is no multi-master event store and no CRDT merge.
  • There is no automatic DSN failover inside the library. Rely on your pooler or platform.

Docs track the go-sequent library. When APIs drift, the source wins.