Barriers

A Grackle barrier is a named N-party rendezvous point. A fixed number of peers agree to synchronize at it: each one arrives, then waits, and they are all released together once the last one has arrived. It is the distributed equivalent of a pthread_barrier or Java’s CyclicBarrier — useful when several workers need to finish a phase before any of them moves on to the next.

A barrier lives inside a namespace and is referenced by name. It must be created explicitly with CreateBarrier before processes can arrive or wait on it.

Core concepts

Expected and arrived processes

A barrier has two numbers:

  • expected_processes — how many distinct peers must arrive before the barrier releases. Set at creation and changeable via UpdateBarrier; it must stay greater than 0, and lowering it to the current arrived_processes trips the barrier immediately (handy when a peer is decommissioned and will never arrive).
  • arrived_processes — how many distinct peers have arrived so far in the current generation.

The barrier releases when arrived_processes == expected_processes. Unlike a wait group there is no producer/observer split — every participant both contributes to and waits on the same barrier.

Process IDs

Each participant identifies itself with a process_id — a free-form string the caller chooses. Grackle stores one participant record per (generation, process_id), which makes ArriveAtBarrier idempotent: a worker that retries its arrival call is counted only once per generation. process_id values are scoped to a single barrier.

This process_id identifies a peer participating in the rendezvous; it is unrelated to the process_id used by lock and semaphore leases.

You can enumerate who has arrived in a given generation with ListBarrierParticipants.

Generations

A barrier is reusable. Each completed rendezvous bumps the barrier’s generation counter, and the next cycle starts fresh with arrived_processes = 0. Every ArriveAtBarrier and WaitAtBarrier call carries an expected_generation field so a slow client can’t accidentally contribute to a later cycle, and so a fast client can wait specifically for the cycle it cares about.

Generations advance by exactly one per trip, so peers compute the next round themselves: after a call that waited at expected_generation: N releases, the next round is expected_generation: N+1. The barrier’s actual current generation is always available on barrier.generation.

Arrive vs. Wait

There are two ways to interact with a barrier:

  • ArriveAtBarrier — records the caller’s arrival and returns immediately. The response carries the current barrier state and all_arrived so a caller that does not need to block can fire-and-go.
  • WaitAtBarrier — blocks until the barrier releases for the requested generation or timeout_seconds elapses. The response carries an outcome enum: BARRIER_WAIT_OUTCOME_TRIPPED when all expected processes arrived and the barrier advanced a generation, or BARRIER_WAIT_OUTCOME_TIMED_OUT when timeout_seconds elapsed before the barrier tripped. The wait call does not register an arrival — peers usually call ArriveAtBarrier first, then WaitAtBarrier.

Metadata

A barrier carries an optional metadata map (string → string) set on CreateBarrier and replaced by UpdateBarrier. Each arriving process can also attach its own metadata on ArriveAtBarrier, which is returned with the participant by ListBarrierParticipants. Metadata is opaque to Grackle — see Metadata for the shared semantics and limits.

Lifecycle

Each barrier carries a last_activity_at timestamp — the time of the most recent activity on it, namely creation or a process arriving via ArriveAtBarrier. It is set at creation and is not changed by UpdateBarrier, WaitAtBarrier, or other reads. It is the base from which the auto-deletion time (last_activity_at + delete_inactive_after_seconds) is computed.

Barriers do not expire. Instead they are auto-deleted after a period of inactivity: delete_inactive_after_seconds, and garbage collection removes the barrier (and its participant records) once last_activity_at + delete_inactive_after_seconds has passed. A barrier that is actively used therefore lives indefinitely, while a forgotten one is reclaimed delete_inactive_after_seconds after its last use. This is the backstop for crashed peers — a half-arrived barrier cannot linger forever.

delete_inactive_after_seconds is the barrier’s own inactivity window; it is not tied to any lease. Barriers do not use leases.

If needed, any barrier can be removed manually with DeleteBarrier.

Example workflow

The coordinator creates a barrier for 4 worker shards that need to meet at end-of-phase, and asks Grackle to auto-delete it after an hour (3600 seconds) of inactivity. (Assuming that a namespace pipelines already exists.)

CreateBarrierRequest:

{
  "namespace_name": "pipelines",
  "barrier_name": "phase_1_complete",
  "description": "End of map phase",
  "expected_processes": 4,
  "delete_inactive_after_seconds": 3600
}

CreateBarrierResponse:

{
  "barrier": {
    "name": "phase_1_complete",
    "description": "End of map phase",
    "expected_processes": 4,
    "arrived_processes": 0,
    "generation": 1,
    "version": 1,
    "created_at": 1718150400000000000,
    "updated_at": 1718150400000000000,
    "last_activity_at": 1718150400000000000,
    "delete_inactive_after_seconds": 3600
  }
}

Each worker reports arrival with its own process_id and the current generation. The call is non-blocking and idempotent — a retry with the same (process_id, expected_generation) is a no-op.

ArriveAtBarrierRequest:

{
  "namespace_name": "pipelines",
  "barrier_name": "phase_1_complete",
  "process_id": "shard-0",
  "expected_generation": 1
}

ArriveAtBarrierResponse (not yet released):

{
  "barrier": {
    "name": "phase_1_complete",
    "expected_processes": 4,
    "arrived_processes": 1,
    "generation": 1
  },
  "all_arrived": false
}

A worker that needs to block until the whole group has caught up calls WaitAtBarrier. Many peers can wait on the same generation at once; they are all released together.

WaitAtBarrierRequest:

{
  "namespace_name": "pipelines",
  "barrier_name": "phase_1_complete",
  "expected_generation": 1,
  "timeout_seconds": 300
}

WaitAtBarrierResponse (all peers arrived before timeout — the barrier has advanced to generation 2 and arrived_processes reset to 0):

{
  "barrier": {
    "name": "phase_1_complete",
    "expected_processes": 4,
    "arrived_processes": 0,
    "generation": 2
  },
  "outcome": "BARRIER_WAIT_OUTCOME_TRIPPED"
}

WaitAtBarrierResponse (timeout fired first — no trip, so the barrier is unchanged):

{
  "barrier": {
    "name": "phase_1_complete",
    "expected_processes": 4,
    "arrived_processes": 3,
    "generation": 1
  },
  "outcome": "BARRIER_WAIT_OUTCOME_TIMED_OUT"
}

A timed-out caller can simply call WaitAtBarrier again — late arrivals continue to accumulate in the background. Once the barrier trips, the next phase uses expected_generation: 2 (one more than the round that just completed) and the cycle repeats.

Inspect the barrier at any time with GetBarrier, or enumerate who has arrived in a given generation with ListBarrierParticipants. DeleteBarrier removes the barrier and its participant records; one that sits idle is deleted automatically once delete_inactive_after_seconds elapse since its last activity.

When to use what

  • Barrier — N peers all need to meet at a synchronization point and continue together. Symmetric: every participant both contributes and waits. Reusable across phases via generations.
  • Wait group — one or more producers fan out N jobs, one or more observers want to know when the fan-in is done. Asymmetric: producers grow the counter, workers report completions, observers block.

Reach for a barrier when peers are equals and need to march in lockstep through phases; reach for a wait group when one party is fanning out work and another party (or parties) need to know when the fan-in is done.

API reference