Architecture
The WorkManager runs workers: units of user code in several languages that subscribe to pub/sub topics, process incoming CloudEvents and publish responses. It is two services:
- WorkManager (
Virtufin.WorkManager): the gRPC API and the worker catalog. It validates requests, persists each worker's record, publishes lifecycle events and routes every worker to its Runtime actor. It runs no worker code. - WorkManager.Runtime (
Virtufin.WorkManager.Runtime): one Dapr actor per worker, each owning one engine instance, plus the topic subscriptions that feed them.
State and pub/sub go through the virtufin API (State, Pubsub); the Dapr
sidecars carry the actor calls, service-invocation mTLS, tracing and metrics.
System Overview
flowchart TB
C[Client] -->|gRPC| WM
subgraph WM["WorkManager (replicas)"]
G[WorkManagerGrpcService] --> CAT[Worker catalog<br/>WorkerConfig records]
G --> LC[RuntimeWorkerLoadCoordinator<br/>async load + commit]
P[RuntimeWorkerPokeService]
end
G -->|Dapr actor calls| RT
LC -->|LoadAsync| RT
P -->|TouchAsync| RT
subgraph RT["WorkManager.Runtime (replicas)"]
A[One actor per worker<br/>CSharpSource / DotnetDll / NativeDll / Python] --> E[Engine]
POOL[RuntimeTopicSubscriptionPool] --> A
end
CAT --> API[(virtufin API<br/>State / Pubsub)]
POOL <--> API
A --> API
Engines and actors
All engines implement IEngine:
public interface IEngine
{
Task LoadCodeAsync(byte[] code, string workerId, CancellationToken cancellationToken = default);
Task<IReadOnlyList<CloudEvent>> ProcessAsync(CloudEvent input, CancellationToken cancellationToken = default);
}
An engine only loads code and processes one CloudEvent at a time, handing the
worker its IWorkerContext (worker id, the Runtime's api client, logger,
cancellation) on each call; it knows nothing about topics or persistence. Everything else (identity,
persisted record, topic subscriptions, status, surviving pod restarts) is the
job of the worker's actor. Each execution mechanism has its own actor type:
| MIME Type | Engine | Runtime Actor Type |
|---|---|---|
text/x-python, application/x-python |
PythonEngine (source or wheel, auto-detected) |
PythonWorker |
text/x-typescript, application/x-npm |
TypeScriptEngine (module or npm package, auto-detected) |
TypeScriptWorker |
text/x-csharp |
CSharpSourceEngine |
CSharpSourceWorker |
application/x-dotnet-dll |
DotNetDllEngine |
DotnetDllWorker |
application/x-native-dll |
NativeDllEngine |
NativeDllWorker |
The mapping is RuntimeActorTypeNames in Virtufin.WorkManager.Shared, which
also holds the actor contract (IWorkerRuntimeActor, its payloads and
serializer) both services compile against. An unsupported mime type is
INVALID_ARGUMENT.
Each actor instance (keyed by worker id) owns exactly one engine instance for its lifetime, and delegates to it:
flowchart LR
subgraph Actor["One actor instance, e.g. PythonWorkerActor (per worker id)"]
Eng["PythonEngine<br/>(owned 1:1, does the actual work)"]
Store["RuntimeWorkerStore<br/>(persisted RuntimeWorkerRecord)"]
Pool["RuntimeTopicSubscriptionPool<br/>(topic subscribe/unsubscribe)"]
end
Actor -->|"LoadCodeAsync / ProcessAsync"| Eng
Actor -->|"Save/Load on Create/Start/Stop/Delete"| Store
Actor -->|"Subscribe/Unsubscribe on Start/Stop"| Pool
The WorkManager hosts no actors; it calls them, resolving the actor type
from the worker's mime type for CreateWorker, StartWorker, StopWorker,
DeleteWorker and LoadCode. Tags, history and ListWorkers are catalog
reads and writes and never reach the Runtime.
Engine Process Model
Engines fall into two groups by how isolated a worker's code is from the host process:
flowchart TB
subgraph Host["WorkManager.Runtime process (Native AOT)"]
DDE[DotNetDllEngine]
NDE[NativeDllEngine]
SEB["SubprocessEngineBase<br/>(shared lifecycle + Unix-socket IPC)"]
PE["PythonEngine<br/>(source or wheel, auto-detected)"]
CSE[CSharpSourceEngine]
TSE["TypeScriptEngine<br/>(module or npm package)"]
SEB --- PE
SEB --- CSE
SEB --- TSE
end
DDE -->|hostfxr: embedded CoreCLR,<br/>same OS process| BR[DotNetDll.Bridge]
PE -->|spawns| PY1[python3 subprocess]
CSE -->|spawns| CSB[CSharpSource.Bridge subprocess<br/>Roslyn compile + AssemblyLoadContext]
PY1 <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| PE
CSB <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| CSE
TSE -->|spawns| NB[node subprocess<br/>worker-bridge.mts, type stripping]
NB <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| TSE
DotNetDllEngine/NativeDllEngine run worker code inside the Runtime process — a crash there
takes that Runtime pod down, with every worker it hosts (the WorkManager and the other Runtime
pods keep running, and the workers are re-created elsewhere). PythonEngine/CSharpSourceEngine both run
worker code in a genuinely separate child OS process instead, via the shared
SubprocessEngineBase (process lifecycle, health-pinging, auto-restart-on-crash, and the wire
protocol are all implemented once, in the base class): binary frames over a Unix domain socket (a
4-byte big-endian length, a 1-byte frame type, the payload), with
ready/process/result/error/ping/pong/shutdown frame types and CloudEvents
protobuf payloads. CSharpSourceEngine's own child process (CSharpSourceEngine.Bridge) is a real, separately
published, JIT-capable (non-AOT) executable — required because compiling C# source with Roslyn and
then dynamically loading/executing the result needs a JIT, which a Native-AOT-published process
does not have.
Because the Runtime is a separate service, a crash inside a worker's engine can take down only a Runtime pod, never the WorkManager's API.
Delivery. Each Runtime pod holds one API subscription per topic and fans every message out to the workers it hosts on that topic. The topic's reader only enqueues: every worker keeps an ordered inbox per topic it subscribes to, and one processing loop drains those inboxes through its actor, one message at a time. Actor turns therefore stay serialized per worker (Dapr's contract) while each topic's events stay in order. The loop picks across topics in arrival order with at most one pending signal per topic, so a flooding or slow topic gets one turn in the rotation, not one turn per queued message; its backlog cannot hold another topic's messages behind it. Each worker also has its own ordered outbox of responses, published in the background: a slow worker, or a slow publish, holds up only itself.
The inbox is bounded at 4096 messages per topic; when it is full that topic's read loop waits rather than drops, which back-pressures the API's subscription for that topic alone. A topic over 80% full logs a warning ("falling behind"). A publish the API reports as failed is logged. How events reach this point — consumer groups, fan-out and scaling — is in Pub/Sub Delivery and Scaling. Several copies of a worker can form a shard there, so each event is handled by one member instead of all of them.
A worker's delivery (its topic inboxes and its outbox) lives on the pod that hosts its actor. It
exists while the worker has a topic there, or while the host pinned it (EnsureQueueAsync, for a
worker that publishes without input topics), and its outbox takes several writers in one order:
the worker's responses and dead letters, and the host's own publishes (PublishAsync).
Deactivating the actor (placement moved it, idle timeout, pod shutdown) ends the pod's delivery
for the worker: its topics are unsubscribed there and the engine released, the record untouched,
so the next activation resubscribes wherever it lands and no pod keeps feeding an actor it no
longer hosts (docs/v1/worker-lifecycle.md#deactivation-and-shutdown).
Because a Dapr actor activates only when something invokes it, a Runtime worker whose topics go
quiet has nothing to reactivate it (no subscription -> no messages -> no invocations): its
subscriptions die with its pod restart or with the Runtime's 24h ActorIdleTimeout.
The WorkManager closes that gap with RuntimeWorkerPokeService: it sweeps the persisted catalog
(default every 60 s, RUNTIME_WORKER_POKE_INTERVAL_SECONDS to override, plus once on
startup) and calls TouchAsync on each running worker's actor -- a live actor
gets a no-op (and its idle clock restarts, so a running worker never idles out), a deactivated
one reactivates and its self-heal re-subscribes its topics. The re-subscription gap after a
Runtime pod restart is thereby bounded by the poke interval instead of being unbounded.
On SIGTERM the Runtime drains before it exits: streams closed first, then every worker finishes
what it holds and flushes its outbox (10 s), then the engines are released (5 s); the chart's
grace periods cover that budget (worker-lifecycle.md#deactivation-and-shutdown).
Data Flow
Worker creation
sequenceDiagram
participant C as Client
participant WM as WorkManager
participant RA as Runtime actor
participant Eng as Engine
C->>WM: CreateWorker(mime_type, code_source, topics)
WM->>WM: validate (mime type, SSRF guard, secrets)
WM->>RA: CreateAsync(payload)
RA->>RA: persist RuntimeWorkerRecord (Loading)
WM->>WM: persist WorkerConfig (Loading)
WM-->>C: worker id
WM->>RA: LoadAsync() (background)
RA->>Eng: LoadCodeAsync(code)
RA-->>WM: RuntimeLoadResult
WM->>WM: commit Stopped / LoadFailed (or start a deferred StartWorker)
WM->>WM: publish loaded / load-failed
StartWorker makes the actor subscribe the worker's topics; from then on the
Runtime's pool delivers each message to the actor's ProcessAsync and
publishes the responses.
A market-data frame, end to end
A typical strategy call: a market-data frame arrives, the WebSocketFeed worker that owns the connection publishes it through the gateway, the Runtime hands it to the worker subscribed to it, and the worker's output is published back through the gateway.
sequenceDiagram
participant EX as Exchange
participant F as WebSocketFeed worker
participant API as API Gateway
participant DAPR as Dapr PubSub
participant RT as WorkManager Runtime
participant W as Worker
EX->>F: WS frame (order-book update)
F->>API: PublishEvents via the Runtime outbox (act.exchange.binance.orderbook.update.BTC_USDT)
API->>DAPR: publish
Note over RT,API: the Runtime subscribes via Pubsub.Subscribe<br/>while it hosts a worker for the topic
DAPR->>API: deliver
API->>RT: server-streaming gRPC
RT->>W: Process(CloudEvent)
W-->>RT: output events
RT->>API: PublishEvents(sc.LIVE.trading.order.submitted)
API->>DAPR: publish
One event
Every message goes through the same per-event rules, whatever the engine. They
live in Virtufin.Worker.Host as WorkerEventRules, so the Runtime and a local
host (see Running workers locally) share one copy.
The actor applies PrepareInput and FinishOutputs around the engine call; the
subscription pool applies Route to each output and DeadLetter when the call
fails.
flowchart LR
IN["Input topic<br/>(API Pubsub.Subscribe)"] --> PREP
subgraph Rules["WorkerEventRules (Virtufin.Worker.Host)"]
PREP["PrepareInput<br/>datacontenttype default,<br/>config fills gaps, secrets win"]
FIN["FinishOutputs<br/>id + source defaults,<br/>secrets stripped"]
ROUTE["Route<br/>publishtopic, else ce-type"]
DLQ["DeadLetter<br/>original event + dlq* attributes"]
end
PREP --> ENG["Engine and worker<br/>(C# source, DLL, native, Python)"]
ENG -->|outputs, in order| FIN --> ROUTE
ROUTE -->|topic| OUT["Worker outbox<br/>(API Pubsub.PublishEvents)"]
ROUTE -->|no topic| SKIP["Logged, not published"]
ENG -.->|exception| DLQ -->|"dlq.<topic>"| OUT
| Rule | What it does |
|---|---|
PrepareInput |
An input without datacontenttype gets application/json. The worker's config then fills the attributes the event leaves out (the event's own attributes win; an entry the CloudEvents SDK rejects is logged and dropped). The worker's secrets are set last, over everything. |
ValidateConfig |
Applied once, when the worker is created (CreateWorker answers INVALID_ARGUMENT, naming the key): a config entry that could never reach the worker is refused. Config travels as CloudEvent attributes, so a key must be lower-case letters and digits (strategyid, not strategy_id; no dots, dashes or capitals), must not be data, time or dataschema (no string value), and must not be an attribute every input already carries (id, source, specversion, type, datacontenttype). Before 0.38.3 such an entry was created and then dropped on every event with only a log warning. |
FinishOutputs |
An output without an id gets a fresh one, without a source the input's. Every secret attribute is removed. The ce-type has no default: falling back to the input's type would republish the output to the topic it came from. |
Route |
An output goes to its publishtopic (a non-empty string; removed before publishing), else to its ce-type. An output with neither is logged and skipped; the worker's other outputs still go out. |
DeadLetter |
A failed message is published once to dlq.<topic>: the event as it arrived (before config and secrets), with the dlq* attributes and the message cut at 1024 characters. It is never retried. See dead letters. |
The dead-letter topic name comes from the naming library (WorkManagerNames.DeadLetter), which
Virtufin.Worker.Host does not depend on, so the caller supplies it.
Conformance vectors
conformance/worker-event-rules.v1.json (repo root) pins these rules for every host and
engine. It ships in the Virtufin.Worker.Host package under conformance/, so other hosts
can test against the same file. It has these parts:
- Codec cases: an event in dict form (the shape a Python worker sees) and its CloudEvents protobuf bytes. Decoding must give the event, and encoding must give it back. They cover text and binary data, booleans, integers, timestamps and bytes.
- Config validation (version 3): a worker's config and whether a host accepts it at
creation (
WorkerEventRules.ValidateConfig); a refused case names the key. - Rule vectors: an input event, the worker's config and secrets, and what the worker
returns or raises. Each vector expects what the worker saw, what was published (topic and
event, in order), which outputs were not routed, and the dead letters. Expected events
match as subsets:
"*"means present,nullmeans absent, and timestamps compare as instants.
Where the Python engine differs, a vector has expect_python. A Python worker returns a dict,
which the codec decodes with defaults: a missing type becomes unknown, so the output is
published to the topic unknown instead of being skipped, and a missing source becomes
urn:unknown instead of the input's. The Python engine reports every worker exception as
System.InvalidOperationException, so the vectors' scripted failures use that type in .NET too.
The vectors run in four test suites:
| Test | What it runs |
|---|---|
Worker.Host.Tests WorkerEventRulesConformanceTests |
the rules alone, as an in-process host applies them |
Worker.Host.Tests CloudEventProtoCodecConformanceTests |
the codec cases against CloudEventProtoCodec |
Runtime.Tests ConformanceTests |
the real actor logic and subscription pool, with a scripted engine, publishes captured on the wire |
Runtime.Tests PythonEngineConformanceTests |
the same with the real PythonEngine (bootstrap, codec, frames) and a scripted Python worker; also the codec cases. Skipped when python3 is missing. |
Runtime.Tests TypeScriptEngineConformanceTests |
the same with the real TypeScriptEngine (Node bridge, codec, frames) and a scripted TypeScript worker; expect_python applies (the dict-form wire path both subprocess engines share). Skipped without node 22.18+. |
Runtime.Tests EngineLifecycleConformanceTests |
the lifecycle vectors on every real engine (C# source, Python, TypeScript, native ABI v3) |
Running workers locally
A LocalWorkManager in Virtufin.Behavior.DevKit (repo worker-devkit, from release
0.34.0) runs workers in-process: they subscribe to topics on a local bus, and their
outputs route as they would on pubsub. It applies the same WorkerEventRules and is
checked against the same conformance vectors, so a local run is a rehearsal of the
cluster. See Running workers locally
in the worker-devkit docs, and the worker workflow
for develop → test → local run → deploy.
Recovery
There is no startup recovery pass. A Runtime actor self-heals on activation:
it reloads its code from its record and resubscribes its topics if the record
says running. The WorkManager's RuntimeWorkerPokeService (and the
RecoverWorkers RPC) touch every running worker's actor so a deactivated one
reactivates; the load coordinator re-drives loads a WorkManager replica died
in the middle of.
Persistence
sys.workmanager.worker.index: the list of worker idssys.workmanager.worker.<worker_id>: the catalog record (WorkerConfig: code source, topics, config, tags, history, status, secret references)sys.workmanager.shard-slot.<slot>: who holds a shard slot (ShardSlotClaim;<slot>hashes shard, topic and index), written by compare-and-swapsys.workmanager-runtime.worker.<worker_id>: the Runtime's record (RuntimeWorkerRecord: what execution needs)
Every read-modify-write of a catalog record is an etag compare-and-swap with
retry, so any replica may write any record. All names come from
naming/virtufin.service.workmanager.yaml (WorkManagerNames); see
Naming.
Code Source Types
Workers support two code source types:
CodeSourceUrl: Code fetched from an HTTP(S) URL at worker creation time
{ "url": "https://example.com/worker.py" }
CodeSourceContent: Base64-encoded inline code
{ "content": "aW1wb3J0IGpzb24..." }
Adding an engine
See Engines: Adding an Engine.
Custom Worker Logic
Workers implement IWorker.ProcessAsync(CloudEvent input, IWorkerContext context) (C#, usually
through Worker.DevKit's bases; the context carries the host's API, logger, worker id and
cancellation) or a module-level Process(event) (Python). A C# source text may instead
declare the legacy static form shown below:
Python:
def Process(event):
return {
"type": "com.example.output",
"source": "my-worker",
"data": { "result": event["data"] }
}
C#:
public static CloudEvent ProcessAsync(CloudEvent input)
{
return new CloudEvent
{
Type = "com.example.output",
Source = new Uri("urn:my-worker"),
Data = input.Data
};
}