Skip to content

Architecture

The WorkManager runs workers: units of user code in several languages that subscribe to pub/sub topics, process incoming CloudEvents and publish responses. It is two services:

  • WorkManager (Virtufin.WorkManager): the gRPC API and the worker catalog. It validates requests, persists each worker's record, publishes lifecycle events and routes every worker to its Runtime actor. It runs no worker code.
  • WorkManager.Runtime (Virtufin.WorkManager.Runtime): one Dapr actor per worker, each owning one engine instance, plus the topic subscriptions that feed them.

State and pub/sub go through the virtufin API (State, Pubsub); the Dapr sidecars carry the actor calls, service-invocation mTLS, tracing and metrics.

System Overview

flowchart TB
    C[Client] -->|gRPC| WM
    subgraph WM["WorkManager (replicas)"]
        G[WorkManagerGrpcService] --> CAT[Worker catalog<br/>WorkerConfig records]
        G --> LC[RuntimeWorkerLoadCoordinator<br/>async load + commit]
        P[RuntimeWorkerPokeService]
    end
    G -->|Dapr actor calls| RT
    LC -->|LoadAsync| RT
    P -->|TouchAsync| RT
    subgraph RT["WorkManager.Runtime (replicas)"]
        A[One actor per worker<br/>CSharpSource / DotnetDll / NativeDll / Python] --> E[Engine]
        POOL[RuntimeTopicSubscriptionPool] --> A
    end
    CAT --> API[(virtufin API<br/>State / Pubsub)]
    POOL <--> API
    A --> API

Engines and actors

All engines implement IEngine:

public interface IEngine
{
    Task LoadCodeAsync(byte[] code, string workerId, CancellationToken cancellationToken = default);
    Task<IReadOnlyList<CloudEvent>> ProcessAsync(CloudEvent input, CancellationToken cancellationToken = default);
}

An engine only loads code and processes one CloudEvent at a time, handing the worker its IWorkerContext (worker id, the Runtime's api client, logger, cancellation) on each call; it knows nothing about topics or persistence. Everything else (identity, persisted record, topic subscriptions, status, surviving pod restarts) is the job of the worker's actor. Each execution mechanism has its own actor type:

MIME Type Engine Runtime Actor Type
text/x-python, application/x-python PythonEngine (source or wheel, auto-detected) PythonWorker
text/x-typescript, application/x-npm TypeScriptEngine (module or npm package, auto-detected) TypeScriptWorker
text/x-csharp CSharpSourceEngine CSharpSourceWorker
application/x-dotnet-dll DotNetDllEngine DotnetDllWorker
application/x-native-dll NativeDllEngine NativeDllWorker

The mapping is RuntimeActorTypeNames in Virtufin.WorkManager.Shared, which also holds the actor contract (IWorkerRuntimeActor, its payloads and serializer) both services compile against. An unsupported mime type is INVALID_ARGUMENT.

Each actor instance (keyed by worker id) owns exactly one engine instance for its lifetime, and delegates to it:

flowchart LR
    subgraph Actor["One actor instance, e.g. PythonWorkerActor (per worker id)"]
        Eng["PythonEngine<br/>(owned 1:1, does the actual work)"]
        Store["RuntimeWorkerStore<br/>(persisted RuntimeWorkerRecord)"]
        Pool["RuntimeTopicSubscriptionPool<br/>(topic subscribe/unsubscribe)"]
    end
    Actor -->|"LoadCodeAsync / ProcessAsync"| Eng
    Actor -->|"Save/Load on Create/Start/Stop/Delete"| Store
    Actor -->|"Subscribe/Unsubscribe on Start/Stop"| Pool

The WorkManager hosts no actors; it calls them, resolving the actor type from the worker's mime type for CreateWorker, StartWorker, StopWorker, DeleteWorker and LoadCode. Tags, history and ListWorkers are catalog reads and writes and never reach the Runtime.

Engine Process Model

Engines fall into two groups by how isolated a worker's code is from the host process:

flowchart TB
    subgraph Host["WorkManager.Runtime process (Native AOT)"]
        DDE[DotNetDllEngine]
        NDE[NativeDllEngine]
        SEB["SubprocessEngineBase&lt;br/&gt;(shared lifecycle + Unix-socket IPC)"]
        PE["PythonEngine&lt;br/&gt;(source or wheel, auto-detected)"]
        CSE[CSharpSourceEngine]
        TSE["TypeScriptEngine&lt;br/&gt;(module or npm package)"]
        SEB --- PE
        SEB --- CSE
        SEB --- TSE
    end
    DDE -->|hostfxr: embedded CoreCLR,<br/>same OS process| BR[DotNetDll.Bridge]
    PE -->|spawns| PY1[python3 subprocess]
    CSE -->|spawns| CSB[CSharpSource.Bridge subprocess<br/>Roslyn compile + AssemblyLoadContext]
    PY1 <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| PE
    CSB <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| CSE
    TSE -->|spawns| NB[node subprocess<br/>worker-bridge.mts, type stripping]
    NB <-->|Unix socket:<br/>binary frames, CloudEvents protobuf| TSE

DotNetDllEngine/NativeDllEngine run worker code inside the Runtime process — a crash there takes that Runtime pod down, with every worker it hosts (the WorkManager and the other Runtime pods keep running, and the workers are re-created elsewhere). PythonEngine/CSharpSourceEngine both run worker code in a genuinely separate child OS process instead, via the shared SubprocessEngineBase (process lifecycle, health-pinging, auto-restart-on-crash, and the wire protocol are all implemented once, in the base class): binary frames over a Unix domain socket (a 4-byte big-endian length, a 1-byte frame type, the payload), with ready/process/result/error/ping/pong/shutdown frame types and CloudEvents protobuf payloads. CSharpSourceEngine's own child process (CSharpSourceEngine.Bridge) is a real, separately published, JIT-capable (non-AOT) executable — required because compiling C# source with Roslyn and then dynamically loading/executing the result needs a JIT, which a Native-AOT-published process does not have.

Because the Runtime is a separate service, a crash inside a worker's engine can take down only a Runtime pod, never the WorkManager's API.

Delivery. Each Runtime pod holds one API subscription per topic and fans every message out to the workers it hosts on that topic. The topic's reader only enqueues: every worker keeps an ordered inbox per topic it subscribes to, and one processing loop drains those inboxes through its actor, one message at a time. Actor turns therefore stay serialized per worker (Dapr's contract) while each topic's events stay in order. The loop picks across topics in arrival order with at most one pending signal per topic, so a flooding or slow topic gets one turn in the rotation, not one turn per queued message; its backlog cannot hold another topic's messages behind it. Each worker also has its own ordered outbox of responses, published in the background: a slow worker, or a slow publish, holds up only itself.

The inbox is bounded at 4096 messages per topic; when it is full that topic's read loop waits rather than drops, which back-pressures the API's subscription for that topic alone. A topic over 80% full logs a warning ("falling behind"). A publish the API reports as failed is logged. How events reach this point — consumer groups, fan-out and scaling — is in Pub/Sub Delivery and Scaling. Several copies of a worker can form a shard there, so each event is handled by one member instead of all of them.

A worker's delivery (its topic inboxes and its outbox) lives on the pod that hosts its actor. It exists while the worker has a topic there, or while the host pinned it (EnsureQueueAsync, for a worker that publishes without input topics), and its outbox takes several writers in one order: the worker's responses and dead letters, and the host's own publishes (PublishAsync). Deactivating the actor (placement moved it, idle timeout, pod shutdown) ends the pod's delivery for the worker: its topics are unsubscribed there and the engine released, the record untouched, so the next activation resubscribes wherever it lands and no pod keeps feeding an actor it no longer hosts (docs/v1/worker-lifecycle.md#deactivation-and-shutdown).

Because a Dapr actor activates only when something invokes it, a Runtime worker whose topics go quiet has nothing to reactivate it (no subscription -> no messages -> no invocations): its subscriptions die with its pod restart or with the Runtime's 24h ActorIdleTimeout. The WorkManager closes that gap with RuntimeWorkerPokeService: it sweeps the persisted catalog (default every 60 s, RUNTIME_WORKER_POKE_INTERVAL_SECONDS to override, plus once on startup) and calls TouchAsync on each running worker's actor -- a live actor gets a no-op (and its idle clock restarts, so a running worker never idles out), a deactivated one reactivates and its self-heal re-subscribes its topics. The re-subscription gap after a Runtime pod restart is thereby bounded by the poke interval instead of being unbounded.

On SIGTERM the Runtime drains before it exits: streams closed first, then every worker finishes what it holds and flushes its outbox (10 s), then the engines are released (5 s); the chart's grace periods cover that budget (worker-lifecycle.md#deactivation-and-shutdown).

Data Flow

Worker creation

sequenceDiagram
    participant C as Client
    participant WM as WorkManager
    participant RA as Runtime actor
    participant Eng as Engine

    C->>WM: CreateWorker(mime_type, code_source, topics)
    WM->>WM: validate (mime type, SSRF guard, secrets)
    WM->>RA: CreateAsync(payload)
    RA->>RA: persist RuntimeWorkerRecord (Loading)
    WM->>WM: persist WorkerConfig (Loading)
    WM-->>C: worker id
    WM->>RA: LoadAsync() (background)
    RA->>Eng: LoadCodeAsync(code)
    RA-->>WM: RuntimeLoadResult
    WM->>WM: commit Stopped / LoadFailed (or start a deferred StartWorker)
    WM->>WM: publish loaded / load-failed

StartWorker makes the actor subscribe the worker's topics; from then on the Runtime's pool delivers each message to the actor's ProcessAsync and publishes the responses.

A market-data frame, end to end

A typical strategy call: a market-data frame arrives, the WebSocketFeed worker that owns the connection publishes it through the gateway, the Runtime hands it to the worker subscribed to it, and the worker's output is published back through the gateway.

sequenceDiagram
    participant EX as Exchange
    participant F as WebSocketFeed worker
    participant API as API Gateway
    participant DAPR as Dapr PubSub
    participant RT as WorkManager Runtime
    participant W as Worker

    EX->>F: WS frame (order-book update)
    F->>API: PublishEvents via the Runtime outbox (act.exchange.binance.orderbook.update.BTC_USDT)
    API->>DAPR: publish
    Note over RT,API: the Runtime subscribes via Pubsub.Subscribe<br/>while it hosts a worker for the topic
    DAPR->>API: deliver
    API->>RT: server-streaming gRPC
    RT->>W: Process(CloudEvent)
    W-->>RT: output events
    RT->>API: PublishEvents(sc.LIVE.trading.order.submitted)
    API->>DAPR: publish

One event

Every message goes through the same per-event rules, whatever the engine. They live in Virtufin.Worker.Host as WorkerEventRules, so the Runtime and a local host (see Running workers locally) share one copy. The actor applies PrepareInput and FinishOutputs around the engine call; the subscription pool applies Route to each output and DeadLetter when the call fails.

flowchart LR
    IN["Input topic<br/>(API Pubsub.Subscribe)"] --> PREP
    subgraph Rules["WorkerEventRules (Virtufin.Worker.Host)"]
        PREP["PrepareInput<br/>datacontenttype default,<br/>config fills gaps, secrets win"]
        FIN["FinishOutputs<br/>id + source defaults,<br/>secrets stripped"]
        ROUTE["Route<br/>publishtopic, else ce-type"]
        DLQ["DeadLetter<br/>original event + dlq* attributes"]
    end
    PREP --> ENG["Engine and worker<br/>(C# source, DLL, native, Python)"]
    ENG -->|outputs, in order| FIN --> ROUTE
    ROUTE -->|topic| OUT["Worker outbox<br/>(API Pubsub.PublishEvents)"]
    ROUTE -->|no topic| SKIP["Logged, not published"]
    ENG -.->|exception| DLQ -->|"dlq.&lt;topic&gt;"| OUT
Rule What it does
PrepareInput An input without datacontenttype gets application/json. The worker's config then fills the attributes the event leaves out (the event's own attributes win; an entry the CloudEvents SDK rejects is logged and dropped). The worker's secrets are set last, over everything.
ValidateConfig Applied once, when the worker is created (CreateWorker answers INVALID_ARGUMENT, naming the key): a config entry that could never reach the worker is refused. Config travels as CloudEvent attributes, so a key must be lower-case letters and digits (strategyid, not strategy_id; no dots, dashes or capitals), must not be data, time or dataschema (no string value), and must not be an attribute every input already carries (id, source, specversion, type, datacontenttype). Before 0.38.3 such an entry was created and then dropped on every event with only a log warning.
FinishOutputs An output without an id gets a fresh one, without a source the input's. Every secret attribute is removed. The ce-type has no default: falling back to the input's type would republish the output to the topic it came from.
Route An output goes to its publishtopic (a non-empty string; removed before publishing), else to its ce-type. An output with neither is logged and skipped; the worker's other outputs still go out.
DeadLetter A failed message is published once to dlq.<topic>: the event as it arrived (before config and secrets), with the dlq* attributes and the message cut at 1024 characters. It is never retried. See dead letters.

The dead-letter topic name comes from the naming library (WorkManagerNames.DeadLetter), which Virtufin.Worker.Host does not depend on, so the caller supplies it.

Conformance vectors

conformance/worker-event-rules.v1.json (repo root) pins these rules for every host and engine. It ships in the Virtufin.Worker.Host package under conformance/, so other hosts can test against the same file. It has these parts:

  • Codec cases: an event in dict form (the shape a Python worker sees) and its CloudEvents protobuf bytes. Decoding must give the event, and encoding must give it back. They cover text and binary data, booleans, integers, timestamps and bytes.
  • Config validation (version 3): a worker's config and whether a host accepts it at creation (WorkerEventRules.ValidateConfig); a refused case names the key.
  • Rule vectors: an input event, the worker's config and secrets, and what the worker returns or raises. Each vector expects what the worker saw, what was published (topic and event, in order), which outputs were not routed, and the dead letters. Expected events match as subsets: "*" means present, null means absent, and timestamps compare as instants.

Where the Python engine differs, a vector has expect_python. A Python worker returns a dict, which the codec decodes with defaults: a missing type becomes unknown, so the output is published to the topic unknown instead of being skipped, and a missing source becomes urn:unknown instead of the input's. The Python engine reports every worker exception as System.InvalidOperationException, so the vectors' scripted failures use that type in .NET too.

The vectors run in four test suites:

Test What it runs
Worker.Host.Tests WorkerEventRulesConformanceTests the rules alone, as an in-process host applies them
Worker.Host.Tests CloudEventProtoCodecConformanceTests the codec cases against CloudEventProtoCodec
Runtime.Tests ConformanceTests the real actor logic and subscription pool, with a scripted engine, publishes captured on the wire
Runtime.Tests PythonEngineConformanceTests the same with the real PythonEngine (bootstrap, codec, frames) and a scripted Python worker; also the codec cases. Skipped when python3 is missing.
Runtime.Tests TypeScriptEngineConformanceTests the same with the real TypeScriptEngine (Node bridge, codec, frames) and a scripted TypeScript worker; expect_python applies (the dict-form wire path both subprocess engines share). Skipped without node 22.18+.
Runtime.Tests EngineLifecycleConformanceTests the lifecycle vectors on every real engine (C# source, Python, TypeScript, native ABI v3)

Running workers locally

A LocalWorkManager in Virtufin.Behavior.DevKit (repo worker-devkit, from release 0.34.0) runs workers in-process: they subscribe to topics on a local bus, and their outputs route as they would on pubsub. It applies the same WorkerEventRules and is checked against the same conformance vectors, so a local run is a rehearsal of the cluster. See Running workers locally in the worker-devkit docs, and the worker workflow for develop → test → local run → deploy.

Recovery

There is no startup recovery pass. A Runtime actor self-heals on activation: it reloads its code from its record and resubscribes its topics if the record says running. The WorkManager's RuntimeWorkerPokeService (and the RecoverWorkers RPC) touch every running worker's actor so a deactivated one reactivates; the load coordinator re-drives loads a WorkManager replica died in the middle of.

Persistence

  • sys.workmanager.worker.index: the list of worker ids
  • sys.workmanager.worker.<worker_id>: the catalog record (WorkerConfig: code source, topics, config, tags, history, status, secret references)
  • sys.workmanager.shard-slot.<slot>: who holds a shard slot (ShardSlotClaim; <slot> hashes shard, topic and index), written by compare-and-swap
  • sys.workmanager-runtime.worker.<worker_id>: the Runtime's record (RuntimeWorkerRecord: what execution needs)

Every read-modify-write of a catalog record is an etag compare-and-swap with retry, so any replica may write any record. All names come from naming/virtufin.service.workmanager.yaml (WorkManagerNames); see Naming.

Code Source Types

Workers support two code source types:

CodeSourceUrl: Code fetched from an HTTP(S) URL at worker creation time

{ "url": "https://example.com/worker.py" }

CodeSourceContent: Base64-encoded inline code

{ "content": "aW1wb3J0IGpzb24..." }

Adding an engine

See Engines: Adding an Engine.

Custom Worker Logic

Workers implement IWorker.ProcessAsync(CloudEvent input, IWorkerContext context) (C#, usually through Worker.DevKit's bases; the context carries the host's API, logger, worker id and cancellation) or a module-level Process(event) (Python). A C# source text may instead declare the legacy static form shown below:

Python:

def Process(event):
    return {
        "type": "com.example.output",
        "source": "my-worker",
        "data": { "result": event["data"] }
    }

C#:

public static CloudEvent ProcessAsync(CloudEvent input)
{
    return new CloudEvent
    {
        Type = "com.example.output",
        Source = new Uri("urn:my-worker"),
        Data = input.Data
    };
}