Skip to content

Pub/Sub Delivery and Scaling

How a published event reaches a worker, where consumer groups exist, and the three shapes for running copies of a worker: broadcast, partitioning and sharding.

The delivery tree

flowchart TB
    S[("Redis stream<br/>one per topic")]
    S --> A1["api pod 1<br/>consumer group = pod name"]
    S --> A2["api pod 2<br/>consumer group = pod name"]
    S --> A3["api pod 3<br/>consumer group = pod name"]
    A1 -->|in-process fan-out| R1["Runtime pod A<br/>one Subscribe stream per topic"]
    A2 -->|in-process fan-out| R2["Runtime pod B"]
    A1 -->|in-process fan-out| R3["Runtime pod C"]
    R1 --> I1["worker 1<br/>inbox per topic"]
    R1 --> I2["worker 2<br/>inbox per topic"]
    R3 --> I3["worker 3<br/>inbox per topic"]
    I1 --> O1["worker 1 outbox"]
    I2 --> O2["worker 2 outbox"]
    I3 --> O3["worker 3 outbox"]
    O1 --> P["PublishEvents"]
    O2 --> P
    O3 --> P

Four hops, each with its own unit of consumption:

Hop Unit Delivery semantics
Redis → daprd one consumer group per API pod per topic (consumerID: {podName}) Competing inside the group, broadcast across groups: every API pod receives every message
API process → gRPC subscribers one Dapr subscription per topic per API process, shared by every Pubsub.Subscribe caller In-process fan-out: every subscriber receives every message
Runtime pod → workers one gRPC subscription per topic per Runtime pod Every local worker registered for the topic receives every message
Worker one Dapr actor, one inbox per topic One actor turn at a time; order preserved within a topic

The WorkManager and its Runtime do not have consumer groups of their own. Only the API pods hold Dapr/Redis consumer groups; everything below them is fan-out.

One worker: per-topic inboxes, one turn at a time

A worker never reads the stream itself. Each of its topics enqueues into its own bounded inbox, and a single processing loop drains them through the actor:

flowchart LR
    TA["topic a stream"] --> IA["inbox: topic a<br/>4096 messages"]
    TB["topic b stream"] --> IB["inbox: topic b<br/>4096 messages"]
    IA -->|enqueue a signal| SIG["signals<br/>at most one per topic"]
    IB -->|enqueue a signal| SIG
    SIG --> LOOP["processing loop"]
    LOOP -->|one message| ACT["actor turn<br/>(serialized per worker)"]
    ACT --> OB["outbox<br/>ordered response batches"]
  • The topic's reader only enqueues into the inbox of (worker, topic) and raises a signal; it never waits for the actor.
  • The loop picks topics in arrival order with at most one pending signal per topic, so a flooding or slow topic gets one turn in the rotation, not one turn per queued message. A backlog on topic a cannot hold topic b behind it.
  • Actor turns stay serialized per worker (Dapr's contract); each topic's events stay in order.
  • The inbox is bounded at 4096 messages per topic. When it is full that topic's reader waits rather than drops, which back-pressures the API's subscription for that topic alone; a topic over 80% full logs "falling behind".
  • Responses go to the worker's ordered outbox, published in the background as ordered PublishEvents batches.

Consumer groups

  • Groups are per API pod, not per Runtime pod or per worker. The count is roughly API replicas × topics with at least one subscriber.
  • A group is created lazily with the first subscriber for a topic and is not deleted when the subscription closes. Dapr's Redis component never removes it, so every rollout leaves groups named after pods that no longer exist; the deployment role removes them (consumer_groups.yml in virtufin/k3s-common).
  • Because every API pod has its own group, the group's "competing consumer" behavior stays inside one pod. Across pods the effect is broadcast — which is what makes every Runtime pod see everything.

Acknowledgment: the broker is not a retry mechanism

TopicDaprSubscriptionRegistry.Deliver writes the event into every subscriber's channel and returns success. Dapr acknowledges the group entry right after that, before any worker processes it. Consequences:

  • a live pending entry exists for microseconds;
  • a worker failure is not redelivered by the broker; the Runtime publishes it once to dlq.<topic> (see Worker Lifecycle);
  • a handler-level fault returns Retry: the entry stays pending, and the sidecar redelivers it once it has been pending processingTimeout (60 s), checking every redeliverInterval (15 s; helm pubsub.*). Dapr's Redis pub/sub has no retry limit and no dead-letter topic, so that repeats until the entry is acknowledged; the gateway's sidecar sweep acknowledges it within seconds when a subscription on the pod closes (common#28);
  • a subscriber whose channel is full logs Failed to enqueue and is reaped by the subscription health sweep (one minute); its Runtime reconnects.

No replay from the group: subscriptions start at the tail

Dapr would resume an existing group at its last-delivered-id, and create a missing group at the stream's start ("0"). Either way a new subscription would be fed old entries. Before opening a subscription, the gateway therefore

  1. creates its own group at $ (or moves it there with XGROUP SETID), and
  2. acknowledges every entry still pending for that group,

and refuses to subscribe if that fails rather than replaying silently. This is also why a worker that subscribes late does not receive history: catch-up is explicit, through the API's replay_since (bounded to 10 000 entries / 64 MB by default), which reads the stream without moving any consumer group. The Runtime uses it for per-worker cursors (Worker Lifecycle).

Operational consequences, all handled by the deployment:

  • Orphan groups after a rollout are removed by virtufin_backend's consumer-group cleanup.
  • Parked readers: a closed subscription's XREADGROUP ... BLOCK 0 reader can take one message into the pending list and never acknowledge it. The gateway wakes parked sidecar readers (CLIENT UNBLOCK) when subscriptions close and acknowledges entries pending longer than five seconds, matching the no-catch-up policy.
  • Memory: streamTTL: 1m trims only when something publishes; a CronJob trims idle streams above a Redis memory threshold, never past what an active group still needs.

Scaling

Add What it adds What it costs
API replica fan-out headroom and another broker consumer group every message is delivered once more, per API pod
Runtime pod (one per node in the reference deployment) capacity to host workers, engines, inboxes no new consumer groups; one more gRPC subscriber per topic it hosts
WorkManager pod catalog/API throughput nothing on the pub/sub path

What does not scale: a single worker. One worker is one actor, one turn at a time — copying it does not share its load unless the copies are put in a shard or on separate topics. And ordering is guaranteed only within a topic; two topics interleave freely.

Three shapes for copies of a worker

flowchart TD
    Q1{"Must every copy see<br/>every event?"}
    Q1 -->|yes| B["broadcast<br/>one topic, copies without a shard"]
    Q1 -->|no| Q2{"Can the publisher<br/>route by key?"}
    Q2 -->|yes| P["partition<br/>one topic per key"]
    Q2 -->|no| S["shard<br/>one topic + shard/shard_size/shard_index"]
Shape Delivery Ordering Failure
Broadcast every copy gets every event per copy, per topic only the failed copy misses
Partition one copy per topic per topic that topic waits
Shard one member per event per entity (subject key) that slot waits; no stealing

Fan-out: copies on one topic

Three copies subscribe to one topic, without a shard. One event is published; each copy processes it and publishes its own output. This is broadcast, by design: two deployments of the same package duplicate the work and its outputs.

Exactly one: one topic per copy

Three copies, each subscribed to its own topic, and one event published per topic. Each event is processed by exactly one worker. This is partitioning, and the upstream publisher (or a router worker) has to decide the topic.

flowchart LR
    E1["event A"] --> T1["x.part.0"] --> W1["worker 0"]
    E2["event B"] --> T2["x.part.1"] --> W2["worker 1"]
    E3["event C"] --> T3["x.part.2"] --> W3["worker 2"]

Sharded groups

A worker can join a shard: several copies of it then split a topic instead of every copy processing every event. The default (no shard) is unchanged: a dedicated worker receives every event.

A shard is not a broker consumer group. The Redis/Dapr consumer groups live one per API pod (above), which is why every worker still receives every event. A shard is a deterministic dispatch rule inside the Runtime; it does not use Redis groups, and it does not steal work.

Configuration

CreateWorkerRequest field Meaning
shard Shard name. Empty means dedicated (broadcast). Letters, digits, -, _, .; up to 64 characters.
shard_size Number of members, 1..1024. Every live member must use the same value.
shard_index This member's slot, 0 <= shard_index < shard_size. Must be unique among live members that share a topic.

All three are immutable after creation and are returned by ListWorkers.

The slot rule

An event belongs to slot hash(routing key) % shard_size, and only the member whose shard_index equals that slot processes it. The routing key is the CloudEvent subject when present, else its id:

  • with a subject (the platform's per-entity key), all events of one entity land on the same member, so per-entity order is preserved;
  • without one, the event id is used: each event is still handled exactly once, but one entity's events may be spread over different members.

For shard_size = 3, for example:

Subject Slot
BTCUSD 0
ETHUSD 1
SOLUSD 2
order/1001 2
order/1002 1

The hash (FNV-1a over UTF-8 plus the Murmur3 avalanche step) is pinned by tests; changing it would remap every event.

sequenceDiagram
    participant P as Publisher
    participant S as Topic stream
    participant M0 as member 0
    participant M1 as member 1
    participant M2 as member 2
    P->>S: ETHUSD event
    S-->>M0: fan-out (every member)
    S-->>M1: fan-out
    S-->>M2: fan-out
    M0->>M0: slot 1 ≠ 0 → drop
    M1->>M1: slot 1 = 1 → enqueue, actor turn
    M2->>M2: slot 1 ≠ 2 → drop

Two consequences follow from "exactly one by construction":

  • there is no coordination: no claim, no lock, no per-message state write;
  • every member still receives every event (delivery stays a broadcast) and simply drops what it does not own before enqueueing it. The fan-out cost is unchanged; the actor work is divided.

Command topics (wrk.<type>.command.<id>) bypass the split: a command addresses one worker, so the slot filter never applies to it.

Choosing a shard index

shard_index is just the member's position in the shard — assign each copy a distinct value from 0 to shard_size - 1 once, when you create it:

for (var i = 0u; i < 3; i++)
{
    await client.DeployAsync(new WorkerSpec(Code, WorkerMimeTypes.CSharpSource, ["orders"])
    {
        Shard = "orders",
        ShardSize = 3,
        ShardIndex = i,
    });
}
for index in range(3):
    await wm.CreateWorker({
        "code_source": code,
        "mime_type": "text/x-python",
        "topics": ["orders"],
        "shard": "orders",
        "shard_size": 3,
        "shard_index": index,
    })
for (let index = 0; index < 3; index++) {
  await client.createWorker(
    "text/x-python", ["orders"], undefined, code,
    undefined, undefined, undefined,
    "orders", 3, index,
  );
}

Practical rules:

  • Size: shard_size is the number of copies you will run. Pick ceil(peak rate ÷ one member's rate) plus headroom; too many members adds fan-out cost and makes a missing slot hurt more.
  • Uniqueness: a slot (shard, topic, shard_index) has one holder. StartWorker claims it atomically, on any replica, and refuses a duplicate with FAILED_PRECONDITION, also while the worker is still LOADING, so the error comes from StartWorker itself, not from a later lifecycle event. Two concurrent starts of one slot leave one running and one refused.
  • Same size: members must agree on shard_size; a mismatched start is refused.
  • Resizing: the fields are immutable. Stop the shard and recreate it with the new size (a short gap), or run the new generation under a new name (e.g. orders-v2) and switch over.
  • Missing member: events for an index with no live member are dropped by the other members — the same tail policy as any worker that is not subscribed.
  • Keeping a slot: stop, delete, a failed start and a failed load give the slot back. A library reload or an activation move keeps it: the worker stays Running. A claim left by a crash is recovered by the next start of that slot once its holder is gone, or is neither Running nor loading with a start requested, and 5 minutes have passed since it claimed. Until then a start is refused with already claimed by worker <id>.
  • Different sizes: two members with different shard_size on different slots are told apart by the running members, so two such starts at the same moment can both pass. Create a shard's members from one definition.

What you get per event

Setup Workers that process one event Outputs published for it
Dedicated worker(s), no shard every copy subscribed to the topic one set per copy
Shard exactly one member one set, from that member
Partition (one topic per copy) the copy subscribed to that topic one set
Shard with a missing member none none (dropped)

A worker may still return several events per input; all of them are published by the owner. A failed input is dead-lettered once (dlq.<topic>), by the owner. Members are assumed to be identical code; the WorkManager validates size/index, not code equality.

Lifecycle and observability

  • Incomplete coverage logs a warning: shard X on [topics] has N of M members running.
  • ListWorkers returns shard, shard_size and shard_index; lifecycle events do not carry them yet (tracked in issue #106).
  • Sharded workers do not use input cursors: a skipped event would pin a cursor, and a replay would re-enter the slot race. On a move they start at the tail, exactly like replay=off. Commands keep their own cursor.

Work stealing

Sharding is static: a slot belongs to one member, and a slow or dead member's share waits. There is no work stealing in shard mode.

Two different things are sometimes called stealing:

  • Admission balancing — every member races for each event, and whoever is free takes it. A slow member simply takes fewer.
  • Failover stealing — a dead member's queued work is picked up by a survivor.

Admission balancing can be built on top of the platform today: every member already receives every event, so a worker can claim it in the shared state store before processing (create-only with a TTL; the Python client exposes both, C#/TypeScript currently need a Get + etag compare-and-swap). That costs one state round trip per event, loses per-entity order, and only gives failover if the losers hold the event and retry after the claim expires — at-least-once.

A first-class claim mode (shard_mode: slot | claim), and the queue-based alternative (stream → per-shard queue → competing consumers), are tracked in issue #107. Until then, prefer finer partitioning or a smaller shard_size before adding claims.

Runnable example

examples/scaling_workers.py deploys all three shapes against a live API and counts the markers each worker publishes:

pip install virtufin-api
VIRTUFIN_API_HOST=api.virtufin.com VIRTUFIN_API_PORT=443 VIRTUFIN_API_TLS=1 \
VIRTUFIN_API_KEY=... python3 examples/scaling_workers.py

Real output from the demo deployment:

fan-out: 1 event -> 3 marker(s) from 3 worker(s) (fanout-0, fanout-1, fanout-2)
exactly-one: 3 events -> 3 marker(s); per-event counts [1, 1, 1]
  part-0 handled 7febd601, part-1 handled b1f1353e, part-2 handled 7d70cd8e
shard: 5 events -> 5 marker(s); per-event counts [1, 1, 1] for the id-routed ones
  scaling-demo/0 handled 41065da2, scaling-demo/0 handled 408ab2dd, scaling-demo/2 handled fdf9933a, scaling-demo/2 handled 89329789, scaling-demo/2 handled 4c91a13b
shard: subject 'entity/BTCUSD' -> 1 member (scaling-demo/0)

fan-out: PASS
exactly-one: PASS
shard: PASS

The script prints PASS/FAIL from the marker counts, so it doubles as a regression check for the delivery model. The last phase also checks the routing key: two events carrying the same subject are handled by the same member.

A single worker still sees each message once

Within one worker, the actor instance is unique and the inbox is FIFO, so a message is processed once. The only repeats are deliberate: input or command cursors are batched (every few hundred milliseconds), so an event finished in the last batch before a crash is replayed to the next activation. Workers are expected to tolerate a repeated input.

Inspecting it on a cluster

Against the Redis primary (namespace redis in the reference deployment):

# one group per API pod per topic, with its last-delivered id and pending count
redis-cli XINFO GROUPS <topic>

# anything a closed subscription's parked reader left behind
redis-cli XPENDING <topic> <api-pod-name>

The worker lifecycle event stream (sys.workmanager.worker.lifecycle) carries worker_id and topic, which is the quickest way to see which workers a topic reaches. Shard membership is in ListWorkers (shard, shard_size, shard_index).

History: worker groups

Before 0.3.0 a worker could carry a group, and members of a group coordinated through a distributed Redis lock (GroupLockOptions, AcquireGroupLockAsync, the VIRTUFIN_WORKER_GROUP environment variable) so that exactly one member processed each message. That lock — and its failure modes — was removed in 0.3.0, and the old group proto field was removed (its number was reused by config).

Competing consumers came back in 0.39.0 as deterministic slotting, named group/group_size/group_index. 0.41.0 renamed them to shard/shard_size/shard_index: same mechanism, but the name no longer suggests a broker consumer group (which supports work stealing) or the old locked workgroups. The slot hash is unchanged across the rename.