Worker Lifecycle
A worker is created through the WorkManager's gRPC API and runs in its WorkManager.Runtime actor. This page follows a worker from creation to deletion; the RPCs are in the API reference, and how messages reach the worker in Pub/Sub Delivery and Scaling.
Worker States
stateDiagram-v2
[*] --> Loading : CreateWorker
Loading --> Stopped : code loaded
Loading --> Running : code loaded, StartWorker was requested
Loading --> LoadFailed : load failed
Stopped --> Running : StartWorker
Running --> Stopped : StopWorker
Stopped --> [*] : DeleteWorker
Running --> [*] : DeleteWorker
LoadFailed --> [*] : DeleteWorker
The worker contract
Every worker implements IWorker (Virtufin.Worker.Host), and every host
(the Runtime, the devkit's LocalWorkManager, BehaviorTestHost,
WorkerObservables.Run) drives it the same way:
public interface IWorker
{
Task StartAsync(IWorkerContext ctx, CancellationToken stop); // returns when running; long work is the worker's own task
Task<IReadOnlyList<CloudEvent>> ProcessAsync(CloudEvent input, CancellationToken ct); // ct: this call's timeout
Task StopAsync(CancellationToken deadline); // release everything; the host waits this long
}
- Start before the first Process, Stop after the last. The context is
handed over once, at
StartAsync, and is the activation's:WorkerId,Api,Logger, theConfigandSecretssnapshots (the values the host stamps on inputs; an event's own attributes still win over config),GetSecretsAsync()(the secrets resolved now, for a credential used long after Start such as a reconnect's login; see Worker Secrets),State(the worker's own keys,sys.workmanager.worker.<id>.state.<key>through the gateway's State service, listed in….statekeys, refused when a value contains a secret value, deleted with the worker),PublishAsync(CloudEvent…)(through the Runtime's outbox: the output rules of a returned output, in order with the responses, gated by the lease) andReportStatusAsync(state, detail)(astatuslifecycle event and the Runtime record, never the catalog). The context'sApiroutesPublishEvent(s)into the same gate; publishing past the host is not supported. stopis cancelled the moment the Runtime loses ownership: the lease is lost, the worker is stopped, reloaded, deleted, its actor deactivated, the pod drains. From then on nothing of the worker's is published (aPublishAsyncthrows),ProcessAsyncis never called again, andStopAsync(deadline)follows with a 5 s budget (RuntimeShutdown.WorkerStopTimeout).ProcessAsyncon a worker that is not started is refused (FAILED_PRECONDITIONthroughInvokeWorkerCommand).- A
requires=<type>/<name>config attribute makes a devkit worker attach to a resource worker at start (attach/detachcommands on the resource's command topic; see the worker-devkit docs).
The lifecycle is pinned by the conformance vectors
(conformance/worker-event-rules.v1.json, lifecycle, since version 2): Start
before Process, host publishes routed like outputs, lease loss stops the worker
and publishes nothing, the stop token fires before StopAsync, no Process
after Stop. The Runtime, the Worker.Host reference host and the local
WorkManager run them.
Every engine gives its worker the lifecycle (workmanager#103 phases 2 and 5):
the DotNetDll engine in process and through the CoreCLR bridge with a host
callback vtable; the C# source and Python engines over the subprocess
protocol (start/stop frames, and the context's publish, status, state and
fresh secrets as host calls; a Python module's optional Start(ctx)/Stop());
the native engine through ABI v3 (Start/Stop exports, host publish
and report_status; a v2 library gets no lifecycle calls). The lifecycle
vectors run through the Runtime logic on each of these engines
(EngineLifecycleConformanceTests). See engines.
Creation
CreateWorker returns before the code is loaded:
- The WorkManager validates the request: a supported mime type, a non-empty code source, an HTTP(S) URL that passes the SSRF guard, valid secret references.
- It calls
CreateAsyncon the worker's Runtime actor, which records the worker asLoading, and persists the catalog record asLOADING. - It returns the new worker id.
- In the background (
RuntimeWorkerLoadCoordinator), the actor'sLoadAsyncfetches the code (a URL source is fetched by the Runtime, pod to registry), resolves its dependencies and loads the engine. The WorkManager commits the outcome:STOPPED, orRUNNINGif aStartWorkerarrived meanwhile, orLOAD_FAILED, and publishesloadedorload-failed(thenstartingandonline, orfailed, for a deferred start). A deferred start that fails leaves the workerSTOPPED.
A WorkManager replica that dies mid-load leaves the worker LOADING; every
replica re-drives such loads at startup (LoadAsync is idempotent and only one
commit wins).
Lifecycle events
Lifecycle events go to sys.workmanager.worker.lifecycle (ce-type
com.virtufin.workmanager.worker.lifecycle, subject worker/<id>, source
urn:com.virtufin:workmanager); the payload carries state, worker_id and
topic (the worker's topics, comma-separated). Each is published once, when
the transition happens, and only then:
state |
When | Published by | Extra payload |
|---|---|---|---|
registered |
CreateWorker accepted and persisted the worker |
WorkManager | |
loaded |
the background load succeeded | WorkManager | libraries: resolved Virtufin.* versions |
load-failed |
the background load failed | WorkManager | error_type, error_message |
starting |
a start begins: StartWorker on a stopped worker, or a deferred start when the load finishes |
WorkManager | |
online |
the start succeeded (the worker is subscribed), or a faulted worker self-healed | WorkManager, Runtime | |
failed |
a start failed (the worker stays STOPPED), or a running worker's self-heal failed |
WorkManager, Runtime | error_type, error_message |
stopped |
StopWorker stopped a running worker, or DeleteWorker deleted one |
WorkManager | |
deleted |
DeleteWorker removed the worker (after its stopped, when it ran) |
WorkManager | |
updated |
a listed worker changed without a state transition: its tags (SetTag, SetTags) or its code (LoadCode) |
WorkManager | changed (["tags"] or ["code"]); tags (the whole tag map) or codeurl (null for inline code) |
status |
the worker reported its own health (ReportStatusAsync), or the Runtime lost the worker's lease to another activation (status = lease-lost) |
Runtime | status, detail |
gap |
input published on one of the worker's topics between from and to may not have reached it: a catch-up after a move was refused or failed, or a command topic's replay was refused (Inputs across a move) |
Runtime | topic (that one topic), from, to (ISO 8601), reason |
A typical worker publishes registered, loaded, starting, online, then
stopped, and deleted when it is removed. A call that changes nothing
publishes nothing: StartWorker on a running worker, StopWorker on a stopped,
loading or LOAD_FAILED one, a StartWorker that is only recorded while the
worker loads (its events come with the load), and an actor reactivation that
self-heals a healthy worker. A call that fails publishes nothing either.
Every change of what ListWorkers returns has an event (registered, deleted,
updated and the state transitions; workmanager#111), so a client can follow the
workers instead of polling: one ListWorkers when its subscription to the
lifecycle topic is live, then the events (virtufin-tui does this). Delivery is
best effort: a publish that fails is logged, not retried, so read
ListWorkers again after a resubscription.
Self-heal failure. When a Runtime actor reactivates a running worker (pod
restart, idle deactivation) and cannot reload its code or resubscribe its topics,
the worker's Runtime record becomes faulted and the Runtime publishes failed
once; the activation still fails, so every later activation (an inbound message,
the WorkManager's poke) retries it, without publishing again. The activation
that succeeds publishes online. The catalog keeps the worker RUNNING
meanwhile.
degraded, stopping and unknown are reserved: nothing publishes them. See
the
worker-management spec.
Dead letters
A message a worker fails on, because it throws or a native worker returns an
error status, is not retried. The Runtime publishes it once to the dead-letter
topic of the topic it arrived on, dlq.<topic> (the naming library's
Names.DeadLetter), and the worker goes on with its next message. Several
workers on one topic share its dead-letter topic.
The dead letter is the original CloudEvent, unchanged (id, type, source, subject, data and its own extensions), plus:
| Extension | Value |
|---|---|
dlqworkerid |
the id of the worker that failed |
dlqactortype |
the Runtime actor type hosting it, i.e. its engine (DotnetDllWorker, CSharpSourceWorker, PythonWorker, NativeDllWorker) |
dlqerrortype |
the exception type the Runtime saw |
dlqerrormessage |
its message, cut at 1024 characters; never a stack trace |
dlqfailedtime |
when it failed (timestamp) |
The names are WorkerAttributes.DeadLetter* in Virtufin.Worker.Host, and
WorkerEventRules.DeadLetter builds the dead letter (see
one event). A
worker's failure crosses the Dapr actor call, so dlqerrortype is usually the
transport's exception and dlqerrormessage carries the worker's own message
inside it. The input event is dead-lettered as it arrived, before the Runtime
adds the worker's config and secrets, so no secret value reaches it; a worker
that puts a secret into its exception message would leak it, as it would into
the logs.
To read dead letters, subscribe to dlq.<topic> for the worker's input topic
(its topics are in ListWorkers), for example through the API's
Pubsub.Subscribe, and filter on dlqworkerid; a new subscription sees only
dead letters published after it opened. If publishing a dead letter fails, the
Runtime logs it at error level and moves on; nothing is retried. The broker
itself dead-letters nothing: Dapr's Redis pub/sub has no dead-letter topic, so
dlq.pubsub (a reserved name) receives no messages.
Start and stop
StartWorker makes the actor claim the worker's lease, start the worker with
its context, subscribe its topics and persist RUNNING; on a LOADING worker
it records the request instead, and the load commit starts it. StartWorker
on a LOAD_FAILED worker is FAILED_PRECONDITION. StopWorker cancels the
worker's stop token, waits for its StopAsync (5 s), revokes the lease,
unsubscribes and persists STOPPED; on a LOADING worker it withdraws a
pending start.
Ownership: the lease
Dapr's placement keeps one activation per worker id; the lease is the
correctness guard around it. On every start (and every self-heal) the Runtime
claims the worker's lease through the API's publication fence
(Pubsub.ChangePublicationFence, keyed by the worker id; a fresh token
superseding the one persisted in the worker's Runtime record, lease 10 s),
persists the token, starts the worker, and renews the lease every second
(WORKER_LEASE_RENEW_SECONDS, WORKER_LEASE_SECONDS). The lease is checked
locally before every outbox batch and host publish, measured from before the
last successful claim or renewal: no round trip on the publish path.
- Taken over (a renewal answers
ABORTED: another activation claimed the worker, a rebalance or a replaced pod): this activation closes its gate, cancels the stop token, callsStopAsync, leaves its topics, publishesstatus = lease-lost; the record is the successor's. What was queued is dropped; the zombie window is one renewal interval plus one in-flight batch. It starts the worker again only once nobody holds the lease: a poke or call that reaches it (at most once per renewal interval) claims under its own superseded token, which the API refuses while the successor's lease is live. Without that, an actor moved back here, or a successor deactivated while calls still reach this instance, stays listedRUNNINGand answersnot started on this Runtimefor good (workmanager#114). - Lapsed (renewals failed for a whole lease, the API unreachable): the
worker is stopped the same way, the record becomes faulted (
failed), and the next poke or call claims and starts it again (online). - Revoked on stop, delete, deactivation, reload and the shutdown drain, so a successor claims at once instead of waiting for the lease to expire.
Commands
InvokeWorkerCommand(id, command, payload, content_type, attributes, timeout_ms)
hands a running worker one command synchronously: a CloudEvent of type
com.virtufin.workmanager.worker.command (source urn:com.virtufin:workmanager,
subject worker/<id>, the command in the command extension, the attributes
as string extensions, the payload as data, application/json by default) goes
through the actor's ProcessAsync like any input (its config and secrets
stamped, serialized with the inbox turns by the actor's single turn), and the
outputs come back in the response, flattened (WorkerEvent: id, type, source,
subject, content type, data, extensions as strings), not published. A
worker that wants both publishes through its context. A worker that is not
running answers FAILED_PRECONDITION; a worker that overruns timeout_ms
(default 30 s) DEADLINE_EXCEEDED. The asynchronous path, a command event on
wrk.<type>.command.<id>, is unchanged.
Code updates
LoadCode validates the new source like CreateWorker, makes the actor
reload its engine with it, and appends it to the worker's history. A started
worker is never reloaded in place: it is stopped (stop token, StopAsync,
lease revoked), its code reloaded, and it is started again with a fresh
context and lease; its in-memory state starts fresh. The same happens when a
Virtufin.* library (a TypeScript worker: an @virtufin/* npm package; a
wheel worker: a virtufin-* Python package) it uses gets a newer release
(LibraryUpdateTracker),
unless the worker's config says autoreload=false: a worker that owns a
resource (a connection) sets it, and is reloaded only by LoadCode or a
Runtime release. The tracker is per pod and follows the worker's activation:
a deactivation (the worker moved, idled out, or its pod stops) and a delete
untrack it there, so a pod never reloads a worker that now runs elsewhere or
no longer exists (before 0.36.1 such a pod retried a reload of a deleted worker
every five minutes, failing Worker <id> was not found.).
Deletion
DeleteWorker stops the worker (stop token, StopAsync, lease revoked),
deletes its own state (sys.workmanager.worker.<id>.state.*), deletes it from
its actor and from the catalog, and deletes the triggers the worker owns. A running worker publishes
stopped, then every deleted worker publishes deleted (before 0.41.7 there was
no deleted event).
Recovery
There is no startup recovery pass:
- A Runtime actor self-heals on activation: it reloads its code from its record
and, if the record says running, claims the lease (with the persisted
token), starts the worker and resubscribes its topics. If that fails, the
worker is faulted (
failed, once) until an activation succeeds (online). - An actor activates only when invoked, so the WorkManager's
RuntimeWorkerPokeServicetouches every running worker's actor once at startup and everyRUNTIME_WORKER_POKE_INTERVAL_SECONDS(default 60 s). A touch also makes sure the worker is started (a lapsed lease is claimed again). After a Runtime pod restart (or the Runtime's 24 h actor idle timeout) a quiet worker is running again within that interval; a worker with traffic on its next invocation. RecoverWorkersruns the same poke on demand.
Deactivation and shutdown
Dapr deactivates a worker's actor on a Runtime pod when placement moves the
worker to another pod (a rebalance after a pod joined or left), when the actor
idled for the Runtime's 24 h ActorIdleTimeout, or when the pod stops. The
deactivation stops the worker here (stop token, StopAsync, lease revoked so
the successor claims at once) and ends this pod's delivery for it: it
unsubscribes the worker's topics here (closing a topic's API stream when no
other worker on the pod uses it) and releases the engine. The worker's Runtime record keeps its
status: the worker is still running wherever its next activation lands, and
that activation reloads the code and resubscribes the topics, on the new pod
or on the same one. Before 0.34.6 the old pod kept its subscriptions and kept
calling the actor, which placement routed to the new pod, so a rebalanced
worker saw every message twice.
The idle timeout never reaches a running worker: the WorkManager pokes every running worker's actor every 60 s (see Recovery), which resets the idle clock. A stopped worker's actor deactivates after a day and reloads on its next call.
Rebalance drain is explicit (DrainRebalancedActors): an actor placement
moves away finishes the call it is in (up to DrainOngoingCallTimeout, 30 s,
the engines' message timeout) before it is deactivated here.
Shutdown (SIGTERM). The Runtime drains before it exits
(RuntimeDrainService, the first hosted service the host stops):
- stop every started worker (stop token,
StopAsync, lease revoked so a successor claims at once), all at once, bounded by 5 s; their last publishes still reach their outboxes; - close every topic stream, so nothing new enters a worker's topic inboxes;
- let each worker finish the messages it already holds (in order within each topic, across topics in a fair rotation, so nothing queued behind a backlog is skipped) and publish everything in its outbox (responses, dead letters), bounded by 10 s in total; what is still queued at the deadline is dropped and logged with a count per worker;
- release every engine (unload contexts, end subprocesses), bounded by 5 s.
The host's stop budget is 25 s (HostOptions.ShutdownTimeout). The helm chart
gives the pod 45 s (terminationGracePeriodSeconds) and keeps the sidecar up
for the drain (dapr.io/block-shutdown-duration: 25s, dapr.io/graceful-shutdown-seconds: 10),
since the drain's actor calls go through it. A message the API delivers after
the streams closed goes to the worker's next activation on another pod only
if it was published after that pod subscribed: the API starts every
subscription at the stream's tail. The worker's own command topic is the
exception (next section).
Commands across a move
A worker moves on a placement rebalance, when its pod is replaced or killed,
or when a topic stream is reopened (the API pods replaced). The old pod has
unsubscribed by then, and the new activation subscribes only when the
WorkManager's poke reaches it (up to a minute). A command published on
wrk.<type>.command.<id> in that gap is still in the Redis stream, but before
0.36.1 nobody received it: the new subscription started at the tail.
The Runtime keeps a command cursor per worker
(sys.workmanager-runtime.worker.<id>.commands): since when, and the ids of the
commands handled since (processed or dead-lettered). It is saved after each
handled command, before the next one is taken. A worker that continues (an
activation's self-heal, a repeated start, a reloaded worker, a reopened stream)
opens its command topic with the API's replay_since from that time less a
5 s clock margin, and skips what the cursor lists, so each command published
during the move reaches the worker once, in order, before the live ones. The
cursor's start moves up to a minute behind the clock whenever no command is
queued, and the ids older than that are forgotten.
- A worker that is stopped and started again, or newly started, starts at
the tail:
StopWorkerandDeleteWorkerdelete its cursor (commands sent to a stopped worker are not its business). - Every other input topic catches up through its own cursor (next section),
unless the worker's config says
replay=off. - A replay the API refuses (a backend without stream history, or more than 1000 commands retained since the cursor) is logged as an error naming the worker and the time, and the topic opens at the tail.
- A command that reaches a worker the host already stopped on that pod (the drain stops workers before the inboxes run out) is refused and not counted as handled: the next activation replays it. On a stop and start in place (a reload) such a command is not replayed; it is logged as a warning.
- A command a pod handled but had not recorded when it died is delivered again: exactly once on a move or drain, at least once on a crash.
Inputs across a move
Since 0.38.0 a worker's normal input topics survive the same moves.
The Runtime keeps input cursors per worker
(sys.workmanager-runtime.worker.<id>.inputs, naming entity
runtime_worker_inputs): per topic, the CloudEvent id of the last event the
worker finished (processed or dead-lettered) and a time at or before that
event's stream entry (when the topic's reader was last caught up with the API
before it received the event).
- Persistence is batched: a finished event only updates the cursor in memory; one State write per worker carries all its topics, at most every 250 ms (the first change after a quiet spell at once, so a low-rate topic is saved right after each event). The worker's stop settles it while the lease is still held, so a move, a drain or a stop loses nothing. A pod that no longer owns the worker (lease taken over or lapsed) writes nothing.
- Catch-up: when a worker continues (the same cases as commands: a move, a
self-heal, a reload, a repeated start, a reopened stream), the topic's
shared stream (one per topic and pod, opened at the tail as before) keeps
this worker's events aside, and a catch-up asks the API for what the topic's
Redis stream retained since the cursor's time less 5 s (
replay_since+replay_only, api 0.15.4), skips through the cursor's event id, hands the rest to the worker, then the events kept aside less those the replay already delivered (by CloudEvent id), and goes live. Other workers on the topic are not affected. A reopened stream catches every worker on it up from the last event it received. When the cursor's event is not in that window, the catch-up looks back once more, 2 min before the cursor's time (Redis's clock further behind the pod's than the margin); found there, it continues exactly once from it. - Retention bounds the catch-up: the API's Redis streams evict entries
older than the chart's
pubsub.streamTTL(1 min on medusa) whenever a topic is published to. A worker that is down longer than that (on 2026-11-01 a pod kill during a Runtime rollout kept one 70 s) finds its last event evicted: the catch-up delivers every entry the stream still retains (all newer than that event, in order, once) and reports a gap from the cursor's time to the catch-up for what was evicted in between. - Gaps are visible: a catch-up the API refuses (more than
PubsubReplayMaxEntries, default 10000, orPubsubReplayMaxBytes, default 64 MiB, retained; no stream history; an API withoutreplay_only), one that fails three times or takes over 30 s, and a stream that no longer retains the cursor's event (the reason names it and the oldest retained one), log a warning and publish a lifecycle event with stategap(topic,from,to,reason); the worker continues live. - A refused actor call is not a worker failure: when the actor call that
hands an input to the worker fails with
Unavailable(the pod's Dapr sidecar is gone or restarting, as while a pod is deleted), the Runtime retries it with backoff while it owns the worker and never dead-letters or finishes that input; if it loses the worker first, the next activation catches the input up. - At least once on a crash: events a pod finished in the last 250 ms before
it died (or whose save was in flight) are delivered again to the next
activation. Workers must tolerate a repeated event on their inputs. Since
worker-devkit 0.41.0 every
BehaviorWorkerBaseworker drops an input it already took (samesourceandid) and derives its output ids from its input, so the events a crashed worker republishes keep their ids (Inputs can arrive twice); what each worker does: market data BinanceTrades, AlpacaTrades, BinanceCandles, BinanceOrderBook, AlpacaOrderBook; indicators (bridge, e.g. SimpleMovingAverage); strategies (bridge, workers); execution (bridge, SimulatedExecutor, ShadowExecutor, LiveExecutor); a worker of your own: the author guide. replay=off(worker config, read by the Runtime, not the worker; alsofalse,0,no): no input cursor, every input topic starts at the tail after a move, as before 0.38.0. For workers that resynchronise themselves: the order-book workers (BinanceOrderBook,AlpacaOrderBook) detect a gap by sequence number and rebuild from a snapshot, and replaying thousands of stale deltas would only delay that. Default: on. Commands always replay.StopWorkerandDeleteWorkerdelete the input cursors: a stopped worker starts at the tail again.
Sharded workers do not use input cursors. When a worker is in a shard
(shard/shard_size/shard_index, see
Delivery and Scaling), its slot decides
which of a topic's events it handles, so a per-worker cursor would be pinned by
the events it skips and a replay would re-enter the slot race. A grouped worker
therefore starts every input topic at the tail after a move, exactly like
replay=off; commands keep their own cursor.
Replicas
Any WorkManager replica serves any RPC for any worker: every write to a worker's record is an etag compare-and-swap with retry. Each Runtime pod holds one subscription per topic for the workers placed on it; Dapr's actor placement keeps each worker on exactly one Runtime pod.
History
Each worker records its code sources: the one it was created with and one
entry per LoadCode. GetWorkerHistory returns them, oldest first. Execution
errors go to the logs and the dead letters, not the history.