Skip to content

Worker Lifecycle

A worker is created through the WorkManager's gRPC API and runs in its WorkManager.Runtime actor. This page follows a worker from creation to deletion; the RPCs are in the API reference, and how messages reach the worker in Pub/Sub Delivery and Scaling.

Worker States

stateDiagram-v2
    [*] --> Loading : CreateWorker
    Loading --> Stopped : code loaded
    Loading --> Running : code loaded, StartWorker was requested
    Loading --> LoadFailed : load failed
    Stopped --> Running : StartWorker
    Running --> Stopped : StopWorker
    Stopped --> [*] : DeleteWorker
    Running --> [*] : DeleteWorker
    LoadFailed --> [*] : DeleteWorker

The worker contract

Every worker implements IWorker (Virtufin.Worker.Host), and every host (the Runtime, the devkit's LocalWorkManager, BehaviorTestHost, WorkerObservables.Run) drives it the same way:

public interface IWorker
{
    Task StartAsync(IWorkerContext ctx, CancellationToken stop);                            // returns when running; long work is the worker's own task
    Task<IReadOnlyList<CloudEvent>> ProcessAsync(CloudEvent input, CancellationToken ct);  // ct: this call's timeout
    Task StopAsync(CancellationToken deadline);                                             // release everything; the host waits this long
}
  • Start before the first Process, Stop after the last. The context is handed over once, at StartAsync, and is the activation's: WorkerId, Api, Logger, the Config and Secrets snapshots (the values the host stamps on inputs; an event's own attributes still win over config), GetSecretsAsync() (the secrets resolved now, for a credential used long after Start such as a reconnect's login; see Worker Secrets), State (the worker's own keys, sys.workmanager.worker.<id>.state.<key> through the gateway's State service, listed in ….statekeys, refused when a value contains a secret value, deleted with the worker), PublishAsync(CloudEvent…) (through the Runtime's outbox: the output rules of a returned output, in order with the responses, gated by the lease) and ReportStatusAsync(state, detail) (a status lifecycle event and the Runtime record, never the catalog). The context's Api routes PublishEvent(s) into the same gate; publishing past the host is not supported.
  • stop is cancelled the moment the Runtime loses ownership: the lease is lost, the worker is stopped, reloaded, deleted, its actor deactivated, the pod drains. From then on nothing of the worker's is published (a PublishAsync throws), ProcessAsync is never called again, and StopAsync(deadline) follows with a 5 s budget (RuntimeShutdown.WorkerStopTimeout).
  • ProcessAsync on a worker that is not started is refused (FAILED_PRECONDITION through InvokeWorkerCommand).
  • A requires=<type>/<name> config attribute makes a devkit worker attach to a resource worker at start (attach/detach commands on the resource's command topic; see the worker-devkit docs).

The lifecycle is pinned by the conformance vectors (conformance/worker-event-rules.v1.json, lifecycle, since version 2): Start before Process, host publishes routed like outputs, lease loss stops the worker and publishes nothing, the stop token fires before StopAsync, no Process after Stop. The Runtime, the Worker.Host reference host and the local WorkManager run them.

Every engine gives its worker the lifecycle (workmanager#103 phases 2 and 5): the DotNetDll engine in process and through the CoreCLR bridge with a host callback vtable; the C# source and Python engines over the subprocess protocol (start/stop frames, and the context's publish, status, state and fresh secrets as host calls; a Python module's optional Start(ctx)/Stop()); the native engine through ABI v3 (Start/Stop exports, host publish and report_status; a v2 library gets no lifecycle calls). The lifecycle vectors run through the Runtime logic on each of these engines (EngineLifecycleConformanceTests). See engines.

Creation

CreateWorker returns before the code is loaded:

  1. The WorkManager validates the request: a supported mime type, a non-empty code source, an HTTP(S) URL that passes the SSRF guard, valid secret references.
  2. It calls CreateAsync on the worker's Runtime actor, which records the worker as Loading, and persists the catalog record as LOADING.
  3. It returns the new worker id.
  4. In the background (RuntimeWorkerLoadCoordinator), the actor's LoadAsync fetches the code (a URL source is fetched by the Runtime, pod to registry), resolves its dependencies and loads the engine. The WorkManager commits the outcome: STOPPED, or RUNNING if a StartWorker arrived meanwhile, or LOAD_FAILED, and publishes loaded or load-failed (then starting and online, or failed, for a deferred start). A deferred start that fails leaves the worker STOPPED.

A WorkManager replica that dies mid-load leaves the worker LOADING; every replica re-drives such loads at startup (LoadAsync is idempotent and only one commit wins).

Lifecycle events

Lifecycle events go to sys.workmanager.worker.lifecycle (ce-type com.virtufin.workmanager.worker.lifecycle, subject worker/<id>, source urn:com.virtufin:workmanager); the payload carries state, worker_id and topic (the worker's topics, comma-separated). Each is published once, when the transition happens, and only then:

state When Published by Extra payload
registered CreateWorker accepted and persisted the worker WorkManager
loaded the background load succeeded WorkManager libraries: resolved Virtufin.* versions
load-failed the background load failed WorkManager error_type, error_message
starting a start begins: StartWorker on a stopped worker, or a deferred start when the load finishes WorkManager
online the start succeeded (the worker is subscribed), or a faulted worker self-healed WorkManager, Runtime
failed a start failed (the worker stays STOPPED), or a running worker's self-heal failed WorkManager, Runtime error_type, error_message
stopped StopWorker stopped a running worker, or DeleteWorker deleted one WorkManager
deleted DeleteWorker removed the worker (after its stopped, when it ran) WorkManager
updated a listed worker changed without a state transition: its tags (SetTag, SetTags) or its code (LoadCode) WorkManager changed (["tags"] or ["code"]); tags (the whole tag map) or codeurl (null for inline code)
status the worker reported its own health (ReportStatusAsync), or the Runtime lost the worker's lease to another activation (status = lease-lost) Runtime status, detail
gap input published on one of the worker's topics between from and to may not have reached it: a catch-up after a move was refused or failed, or a command topic's replay was refused (Inputs across a move) Runtime topic (that one topic), from, to (ISO 8601), reason

A typical worker publishes registered, loaded, starting, online, then stopped, and deleted when it is removed. A call that changes nothing publishes nothing: StartWorker on a running worker, StopWorker on a stopped, loading or LOAD_FAILED one, a StartWorker that is only recorded while the worker loads (its events come with the load), and an actor reactivation that self-heals a healthy worker. A call that fails publishes nothing either.

Every change of what ListWorkers returns has an event (registered, deleted, updated and the state transitions; workmanager#111), so a client can follow the workers instead of polling: one ListWorkers when its subscription to the lifecycle topic is live, then the events (virtufin-tui does this). Delivery is best effort: a publish that fails is logged, not retried, so read ListWorkers again after a resubscription.

Self-heal failure. When a Runtime actor reactivates a running worker (pod restart, idle deactivation) and cannot reload its code or resubscribe its topics, the worker's Runtime record becomes faulted and the Runtime publishes failed once; the activation still fails, so every later activation (an inbound message, the WorkManager's poke) retries it, without publishing again. The activation that succeeds publishes online. The catalog keeps the worker RUNNING meanwhile.

degraded, stopping and unknown are reserved: nothing publishes them. See the worker-management spec.

Dead letters

A message a worker fails on, because it throws or a native worker returns an error status, is not retried. The Runtime publishes it once to the dead-letter topic of the topic it arrived on, dlq.<topic> (the naming library's Names.DeadLetter), and the worker goes on with its next message. Several workers on one topic share its dead-letter topic.

The dead letter is the original CloudEvent, unchanged (id, type, source, subject, data and its own extensions), plus:

Extension Value
dlqworkerid the id of the worker that failed
dlqactortype the Runtime actor type hosting it, i.e. its engine (DotnetDllWorker, CSharpSourceWorker, PythonWorker, NativeDllWorker)
dlqerrortype the exception type the Runtime saw
dlqerrormessage its message, cut at 1024 characters; never a stack trace
dlqfailedtime when it failed (timestamp)

The names are WorkerAttributes.DeadLetter* in Virtufin.Worker.Host, and WorkerEventRules.DeadLetter builds the dead letter (see one event). A worker's failure crosses the Dapr actor call, so dlqerrortype is usually the transport's exception and dlqerrormessage carries the worker's own message inside it. The input event is dead-lettered as it arrived, before the Runtime adds the worker's config and secrets, so no secret value reaches it; a worker that puts a secret into its exception message would leak it, as it would into the logs.

To read dead letters, subscribe to dlq.<topic> for the worker's input topic (its topics are in ListWorkers), for example through the API's Pubsub.Subscribe, and filter on dlqworkerid; a new subscription sees only dead letters published after it opened. If publishing a dead letter fails, the Runtime logs it at error level and moves on; nothing is retried. The broker itself dead-letters nothing: Dapr's Redis pub/sub has no dead-letter topic, so dlq.pubsub (a reserved name) receives no messages.

Start and stop

StartWorker makes the actor claim the worker's lease, start the worker with its context, subscribe its topics and persist RUNNING; on a LOADING worker it records the request instead, and the load commit starts it. StartWorker on a LOAD_FAILED worker is FAILED_PRECONDITION. StopWorker cancels the worker's stop token, waits for its StopAsync (5 s), revokes the lease, unsubscribes and persists STOPPED; on a LOADING worker it withdraws a pending start.

Ownership: the lease

Dapr's placement keeps one activation per worker id; the lease is the correctness guard around it. On every start (and every self-heal) the Runtime claims the worker's lease through the API's publication fence (Pubsub.ChangePublicationFence, keyed by the worker id; a fresh token superseding the one persisted in the worker's Runtime record, lease 10 s), persists the token, starts the worker, and renews the lease every second (WORKER_LEASE_RENEW_SECONDS, WORKER_LEASE_SECONDS). The lease is checked locally before every outbox batch and host publish, measured from before the last successful claim or renewal: no round trip on the publish path.

  • Taken over (a renewal answers ABORTED: another activation claimed the worker, a rebalance or a replaced pod): this activation closes its gate, cancels the stop token, calls StopAsync, leaves its topics, publishes status = lease-lost; the record is the successor's. What was queued is dropped; the zombie window is one renewal interval plus one in-flight batch. It starts the worker again only once nobody holds the lease: a poke or call that reaches it (at most once per renewal interval) claims under its own superseded token, which the API refuses while the successor's lease is live. Without that, an actor moved back here, or a successor deactivated while calls still reach this instance, stays listed RUNNING and answers not started on this Runtime for good (workmanager#114).
  • Lapsed (renewals failed for a whole lease, the API unreachable): the worker is stopped the same way, the record becomes faulted (failed), and the next poke or call claims and starts it again (online).
  • Revoked on stop, delete, deactivation, reload and the shutdown drain, so a successor claims at once instead of waiting for the lease to expire.

Commands

InvokeWorkerCommand(id, command, payload, content_type, attributes, timeout_ms) hands a running worker one command synchronously: a CloudEvent of type com.virtufin.workmanager.worker.command (source urn:com.virtufin:workmanager, subject worker/<id>, the command in the command extension, the attributes as string extensions, the payload as data, application/json by default) goes through the actor's ProcessAsync like any input (its config and secrets stamped, serialized with the inbox turns by the actor's single turn), and the outputs come back in the response, flattened (WorkerEvent: id, type, source, subject, content type, data, extensions as strings), not published. A worker that wants both publishes through its context. A worker that is not running answers FAILED_PRECONDITION; a worker that overruns timeout_ms (default 30 s) DEADLINE_EXCEEDED. The asynchronous path, a command event on wrk.<type>.command.<id>, is unchanged.

Code updates

LoadCode validates the new source like CreateWorker, makes the actor reload its engine with it, and appends it to the worker's history. A started worker is never reloaded in place: it is stopped (stop token, StopAsync, lease revoked), its code reloaded, and it is started again with a fresh context and lease; its in-memory state starts fresh. The same happens when a Virtufin.* library (a TypeScript worker: an @virtufin/* npm package; a wheel worker: a virtufin-* Python package) it uses gets a newer release (LibraryUpdateTracker), unless the worker's config says autoreload=false: a worker that owns a resource (a connection) sets it, and is reloaded only by LoadCode or a Runtime release. The tracker is per pod and follows the worker's activation: a deactivation (the worker moved, idled out, or its pod stops) and a delete untrack it there, so a pod never reloads a worker that now runs elsewhere or no longer exists (before 0.36.1 such a pod retried a reload of a deleted worker every five minutes, failing Worker <id> was not found.).

Deletion

DeleteWorker stops the worker (stop token, StopAsync, lease revoked), deletes its own state (sys.workmanager.worker.<id>.state.*), deletes it from its actor and from the catalog, and deletes the triggers the worker owns. A running worker publishes stopped, then every deleted worker publishes deleted (before 0.41.7 there was no deleted event).

Recovery

There is no startup recovery pass:

  • A Runtime actor self-heals on activation: it reloads its code from its record and, if the record says running, claims the lease (with the persisted token), starts the worker and resubscribes its topics. If that fails, the worker is faulted (failed, once) until an activation succeeds (online).
  • An actor activates only when invoked, so the WorkManager's RuntimeWorkerPokeService touches every running worker's actor once at startup and every RUNTIME_WORKER_POKE_INTERVAL_SECONDS (default 60 s). A touch also makes sure the worker is started (a lapsed lease is claimed again). After a Runtime pod restart (or the Runtime's 24 h actor idle timeout) a quiet worker is running again within that interval; a worker with traffic on its next invocation.
  • RecoverWorkers runs the same poke on demand.

Deactivation and shutdown

Dapr deactivates a worker's actor on a Runtime pod when placement moves the worker to another pod (a rebalance after a pod joined or left), when the actor idled for the Runtime's 24 h ActorIdleTimeout, or when the pod stops. The deactivation stops the worker here (stop token, StopAsync, lease revoked so the successor claims at once) and ends this pod's delivery for it: it unsubscribes the worker's topics here (closing a topic's API stream when no other worker on the pod uses it) and releases the engine. The worker's Runtime record keeps its status: the worker is still running wherever its next activation lands, and that activation reloads the code and resubscribes the topics, on the new pod or on the same one. Before 0.34.6 the old pod kept its subscriptions and kept calling the actor, which placement routed to the new pod, so a rebalanced worker saw every message twice.

The idle timeout never reaches a running worker: the WorkManager pokes every running worker's actor every 60 s (see Recovery), which resets the idle clock. A stopped worker's actor deactivates after a day and reloads on its next call.

Rebalance drain is explicit (DrainRebalancedActors): an actor placement moves away finishes the call it is in (up to DrainOngoingCallTimeout, 30 s, the engines' message timeout) before it is deactivated here.

Shutdown (SIGTERM). The Runtime drains before it exits (RuntimeDrainService, the first hosted service the host stops):

  1. stop every started worker (stop token, StopAsync, lease revoked so a successor claims at once), all at once, bounded by 5 s; their last publishes still reach their outboxes;
  2. close every topic stream, so nothing new enters a worker's topic inboxes;
  3. let each worker finish the messages it already holds (in order within each topic, across topics in a fair rotation, so nothing queued behind a backlog is skipped) and publish everything in its outbox (responses, dead letters), bounded by 10 s in total; what is still queued at the deadline is dropped and logged with a count per worker;
  4. release every engine (unload contexts, end subprocesses), bounded by 5 s.

The host's stop budget is 25 s (HostOptions.ShutdownTimeout). The helm chart gives the pod 45 s (terminationGracePeriodSeconds) and keeps the sidecar up for the drain (dapr.io/block-shutdown-duration: 25s, dapr.io/graceful-shutdown-seconds: 10), since the drain's actor calls go through it. A message the API delivers after the streams closed goes to the worker's next activation on another pod only if it was published after that pod subscribed: the API starts every subscription at the stream's tail. The worker's own command topic is the exception (next section).

Commands across a move

A worker moves on a placement rebalance, when its pod is replaced or killed, or when a topic stream is reopened (the API pods replaced). The old pod has unsubscribed by then, and the new activation subscribes only when the WorkManager's poke reaches it (up to a minute). A command published on wrk.<type>.command.<id> in that gap is still in the Redis stream, but before 0.36.1 nobody received it: the new subscription started at the tail.

The Runtime keeps a command cursor per worker (sys.workmanager-runtime.worker.<id>.commands): since when, and the ids of the commands handled since (processed or dead-lettered). It is saved after each handled command, before the next one is taken. A worker that continues (an activation's self-heal, a repeated start, a reloaded worker, a reopened stream) opens its command topic with the API's replay_since from that time less a 5 s clock margin, and skips what the cursor lists, so each command published during the move reaches the worker once, in order, before the live ones. The cursor's start moves up to a minute behind the clock whenever no command is queued, and the ids older than that are forgotten.

  • A worker that is stopped and started again, or newly started, starts at the tail: StopWorker and DeleteWorker delete its cursor (commands sent to a stopped worker are not its business).
  • Every other input topic catches up through its own cursor (next section), unless the worker's config says replay=off.
  • A replay the API refuses (a backend without stream history, or more than 1000 commands retained since the cursor) is logged as an error naming the worker and the time, and the topic opens at the tail.
  • A command that reaches a worker the host already stopped on that pod (the drain stops workers before the inboxes run out) is refused and not counted as handled: the next activation replays it. On a stop and start in place (a reload) such a command is not replayed; it is logged as a warning.
  • A command a pod handled but had not recorded when it died is delivered again: exactly once on a move or drain, at least once on a crash.

Inputs across a move

Since 0.38.0 a worker's normal input topics survive the same moves.

The Runtime keeps input cursors per worker (sys.workmanager-runtime.worker.<id>.inputs, naming entity runtime_worker_inputs): per topic, the CloudEvent id of the last event the worker finished (processed or dead-lettered) and a time at or before that event's stream entry (when the topic's reader was last caught up with the API before it received the event).

  • Persistence is batched: a finished event only updates the cursor in memory; one State write per worker carries all its topics, at most every 250 ms (the first change after a quiet spell at once, so a low-rate topic is saved right after each event). The worker's stop settles it while the lease is still held, so a move, a drain or a stop loses nothing. A pod that no longer owns the worker (lease taken over or lapsed) writes nothing.
  • Catch-up: when a worker continues (the same cases as commands: a move, a self-heal, a reload, a repeated start, a reopened stream), the topic's shared stream (one per topic and pod, opened at the tail as before) keeps this worker's events aside, and a catch-up asks the API for what the topic's Redis stream retained since the cursor's time less 5 s (replay_since + replay_only, api 0.15.4), skips through the cursor's event id, hands the rest to the worker, then the events kept aside less those the replay already delivered (by CloudEvent id), and goes live. Other workers on the topic are not affected. A reopened stream catches every worker on it up from the last event it received. When the cursor's event is not in that window, the catch-up looks back once more, 2 min before the cursor's time (Redis's clock further behind the pod's than the margin); found there, it continues exactly once from it.
  • Retention bounds the catch-up: the API's Redis streams evict entries older than the chart's pubsub.streamTTL (1 min on medusa) whenever a topic is published to. A worker that is down longer than that (on 2026-11-01 a pod kill during a Runtime rollout kept one 70 s) finds its last event evicted: the catch-up delivers every entry the stream still retains (all newer than that event, in order, once) and reports a gap from the cursor's time to the catch-up for what was evicted in between.
  • Gaps are visible: a catch-up the API refuses (more than PubsubReplayMaxEntries, default 10000, or PubsubReplayMaxBytes, default 64 MiB, retained; no stream history; an API without replay_only), one that fails three times or takes over 30 s, and a stream that no longer retains the cursor's event (the reason names it and the oldest retained one), log a warning and publish a lifecycle event with state gap (topic, from, to, reason); the worker continues live.
  • A refused actor call is not a worker failure: when the actor call that hands an input to the worker fails with Unavailable (the pod's Dapr sidecar is gone or restarting, as while a pod is deleted), the Runtime retries it with backoff while it owns the worker and never dead-letters or finishes that input; if it loses the worker first, the next activation catches the input up.
  • At least once on a crash: events a pod finished in the last 250 ms before it died (or whose save was in flight) are delivered again to the next activation. Workers must tolerate a repeated event on their inputs. Since worker-devkit 0.41.0 every BehaviorWorkerBase worker drops an input it already took (same source and id) and derives its output ids from its input, so the events a crashed worker republishes keep their ids (Inputs can arrive twice); what each worker does: market data BinanceTrades, AlpacaTrades, BinanceCandles, BinanceOrderBook, AlpacaOrderBook; indicators (bridge, e.g. SimpleMovingAverage); strategies (bridge, workers); execution (bridge, SimulatedExecutor, ShadowExecutor, LiveExecutor); a worker of your own: the author guide.
  • replay=off (worker config, read by the Runtime, not the worker; also false, 0, no): no input cursor, every input topic starts at the tail after a move, as before 0.38.0. For workers that resynchronise themselves: the order-book workers (BinanceOrderBook, AlpacaOrderBook) detect a gap by sequence number and rebuild from a snapshot, and replaying thousands of stale deltas would only delay that. Default: on. Commands always replay.
  • StopWorker and DeleteWorker delete the input cursors: a stopped worker starts at the tail again.

Sharded workers do not use input cursors. When a worker is in a shard (shard/shard_size/shard_index, see Delivery and Scaling), its slot decides which of a topic's events it handles, so a per-worker cursor would be pinned by the events it skips and a replay would re-enter the slot race. A grouped worker therefore starts every input topic at the tail after a move, exactly like replay=off; commands keep their own cursor.

Replicas

Any WorkManager replica serves any RPC for any worker: every write to a worker's record is an etag compare-and-swap with retry. Each Runtime pod holds one subscription per topic for the workers placed on it; Dapr's actor placement keeps each worker on exactly one Runtime pod.

History

Each worker records its code sources: the one it was created with and one entry per LoadCode. GetWorkerHistory returns them, oldest first. Execution errors go to the logs and the dead letters, not the history.