Skip to content

Engine System

The WorkManager uses a pluggable engine architecture to support multiple programming languages. Each engine handles code execution for a specific MIME type.

IEngine Interface

All engines implement the IEngine interface:

public interface IEngine
{
    Task LoadCodeAsync(byte[] code, string workerId, CancellationToken cancellationToken = default);
    Task<IReadOnlyList<CloudEvent>> ProcessAsync(CloudEvent input, CancellationToken cancellationToken = default);
    WorkerPackageDependencies? ReadPackageDependencies(byte[] code) => null;
}
Method Description
LoadCodeAsync Loads the worker code into the engine. Called once at worker creation or when code is updated; workerId is what the worker sees as IWorkerContext.WorkerId.
ProcessAsync Processes an incoming CloudEvent into zero or more response CloudEvents, published in order; an empty list publishes nothing.
ReadPackageDependencies The Virtufin.* packages the code declares, for library auto-reload; null when the format has none.
NpmDependencies After a load: the @virtufin/* npm packages the worker asked for and the versions npm installed, for library auto-reload; null for engines without npm and workers without such a package.
PyPiDependencies After a load: the virtufin-* Python packages a wheel's Requires-Dist declares and every virtufin-* version uv installed, for library auto-reload; null for source workers, engines without pip, and wheels without a virtufin-* requirement.

Start and stop

IEngine.StartAsync(IWorkerContext, CancellationToken stop) and StopAsync(CancellationToken deadline) carry the worker lifecycle (docs/v1/worker-lifecycle.md#the-worker-contract) into the engine; both default to no-ops. DotNetDllEngine implements them: in process it calls the worker's StartAsync/StopAsync directly; through the CoreCLR bridge it sends the context as a WorkerStartPayload (config, secrets, the state keys) plus a host callback vtable (DotNetHostVtable, ABI 2: publish, status, secrets, free_buffer) through which the bridge-side context's publishes, status reports and GetSecretsAsync calls come back to the host's context (secrets returns the freshly resolved map as JSON in a host-allocated buffer the bridge releases with free_buffer; on failure the buffer holds the error, which names a reference at most), and SignalStop/StopWorker entry points for the stop token and StopAsync. Each StartWorker creates a fresh stop token (a stopped worker started again never sees the earlier stop), and the engine forwards the worker's publishes from the StartWorker call on (a worker may publish inside its own StartAsync) until StopAsync begins; the Runtime's gate decides ownership.

The subprocess engines (CSharpSourceEngine, PythonEngine, TypeScriptEngine) send the context as a start frame (EngineStartPayload JSON: worker id, config, secrets) and wait for started under the engine's startup timeout (never the 30 s message timeout: a start may open a connection and log in). One reader task demultiplexes the socket: the worker's host calls (publish, status, state, secrets frames, each with a call id and answered by reply) are carried out against the host's context while the engine's own requests (start, process, ping, stop) stay one at a time, so a worker may publish from its own loop, inside its start or while a process is in flight. stop carries a budget; the subprocess cancels the worker's stop token first, then runs its stop for at most the budget and answers stopped. The host's stop token closes the engine's side of the publish gate at once. A subprocess that dies or hangs is replaced, and a started worker is started again in the new one with the same context. A host publish without a route (no publishtopic, no type) is dropped, as in process.

The native engine calls a v3 library's Start and Stop exports, and the host vtable's publish and report_status reach the host's context (native ABI v3); a v2 library gets no lifecycle calls.

The worker context

Every engine hands the worker an IWorkerContext on each call (IWorker.ProcessAsync(input, context)): the worker id, the Runtime's api client (WorkerContexts.EnvironmentApi, from VIRTUFIN_API_HOST/_GRPC_PORT/ _KEY/_TLS), a logger (the Runtime's, the bridge's stderr, or the native log callback) and the call's cancellation token. Nothing is ambient: a worker that uses state, pubsub or triggers reads context.Api, which is why the same worker also runs under a LocalWorkManager with an in-memory api (Worker.Host).

Architecture

The WorkManager maps a worker's mime type to a WorkManager.Runtime actor type (RuntimeActorTypeNames); each actor type owns one engine class, and each worker's actor owns one engine instance for its activation.

flowchart TD
    subgraph WM["WorkManager (catalog, routing)"]
        C["text/x-csharp"] --> A1[CSharpSourceWorker]
        D["application/x-dotnet-dll"] --> A2[DotnetDllWorker]
        N["application/x-native-dll"] --> A3[NativeDllWorker]
        P["text/x-python, application/x-python"] --> A4[PythonWorker]
        T["text/x-typescript, application/x-npm"] --> A5[TypeScriptWorker]
    end
    A1 --> E1[CSharpSourceEngine]
    A2 --> E2[DotNetDllEngine]
    A3 --> E3[NativeDllEngine]
    A4 --> E4[PythonEngine]
    A5 --> E5[TypeScriptEngine]

Engine Resolution

When a worker is created:

  1. The WorkManager picks the Runtime actor type for the mime type (an unsupported one is INVALID_ARGUMENT) and calls CreateAsync on the worker's actor
  2. In the background, LoadAsync makes the actor construct its engine and call LoadCodeAsync() with the fetched code
  3. On StartWorker the actor subscribes the worker's topics; every message reaches the engine through the actor's ProcessAsync

Each worker has its own engine instance; no state is shared between workers.

Built-in Engines

PythonEngine

MIME Types: text/x-python (raw source) and application/x-python (a wheel) -- one engine handles both, since they're the same execution mechanism differing only in how the entry file gets prepared. LoadCodeAsync detects which format it was given by sniffing the code bytes for a ZIP local-file-header signature (every non-empty wheel has one; Python source never does), rather than needing the mime type threaded through. Formerly two engine classes (PythonEngine/PythonWheelEngine) and two Runtime actor types; merged since neither was a genuinely distinct execution mechanism.

Executes Python code in a persistent sandboxed python3 subprocess. For raw source, code is written to a .py temp file at startup; for a wheel, see "Wheel format" below. The subprocess is started with a Unix domain socket (python3 --socket <path> --code <path>). The subprocess stays alive for the worker's lifetime, communicating over the Unix domain socket (not stdin/stdout) in the worker wire protocol's binary frames (see Python Engine Internals).

Data flow:

sequenceDiagram
    autonumber
    participant Sub as Pub/Sub
    participant WM as WorkManager<br/>(PythonEngine)
    participant Py as python3 subprocess
    participant Code as Worker's Process()

    Sub->>WM: input CloudEvent
    WM->>WM: write worker code to .py temp file
    WM->>Py: start python3 X.py --socket S
    Py->>WM: ready frame (over the Unix domain socket)
    Note over WM,Py: engine waits for ready. timeout triggers relaunch.
    WM->>Py: process frame (protobuf CloudEvent)
    Py->>Code: Process(ce_dict)
    Code-->>Py: result dict, list of dicts, or None
    Py->>WM: result frame (protobuf CloudEventBatch)
    WM->>WM: decode batch → publish each, in order<br/>(empty → no reply)
    WM->>Sub: reply CloudEvent(s)
    Note over WM,Py: 10s health-check ping (empty heartbeat).<br/>Unresponsive subprocess is auto-restarted.

Code Requirements:

def Process(event):
    # event is a dict with CloudEvent fields
    # Return a dict representing the output CloudEvent, a list of them
    # (published in order), or None
    return {
        "type": "com.example.output",
        "source": "my-worker",
        "data": { "result": "processed" }
    }

Lifecycle (optional): a module-level Start(ctx) runs once per activation before the first event, Stop() after the last.

import threading

_ctx = None

def _loop():
    while not _ctx.stopping.is_set():          # set the moment the host stops or loses the worker
        _ctx.publish({"type": "com.example.tick", "source": "urn:ticker", "data": "{}"})
        _ctx.stopping.wait(1.0)

def Start(ctx):
    global _ctx
    _ctx = ctx                                  # worker_id, config, secrets, state, ...
    ctx.report_status("connected", ctx.config.get("venue"))
    threading.Thread(target=_loop, daemon=True).start()

def Stop():
    pass                                        # release what Start acquired; ctx.publish now raises HostRefused

ctx has worker_id, config, secrets (the snapshot at start), stopping (a threading.Event), publish(event_or_list) (routed like a returned output, through the host's outbox and ownership gate), report_status(state, detail=None), state.get(key) → (value, etag) or None, state.save(key, value, etag=None), state.delete(key), state.keys(), and get_secrets() (resolved now). A refused call raises HostRefused (a RuntimeError), a stale etag StateConflict, a value that contains a secret ValueError; both exception classes are globals of the worker module. The module runs in its own namespace, never the bootstrap's.

CloudEvent Mapping:

The bootstrap decodes the protobuf CloudEvent into a dict shaped like the CloudEvents JSON format:

{
    "id": "event-123",
    "type": "com.example.input",
    "source": "urn:source",
    "specversion": "1.0",
    "datacontenttype": "application/json",
    "time": "2026-10-24T12:00:00.123Z",
    "sequence": 7,
    "data": "{\"key\": \"value\"}"
}

  • data is the event's text data as a str (JSON is not parsed; the Python devkit's CommandWorker does that); binary data arrives base64-encoded in data_base64 instead.
  • Every other attribute is a top-level key, keeping its type: integers as int, booleans as bool, time as an RFC 3339 string.
  • A returned dict maps back the same way: a non-string data is JSON-encoded, data_base64 becomes binary data, int/bool values become typed attributes, and a missing id or source gets a default.

Error Handling: - An exception in Process() is sent back as an error frame; the engine throws it as the event's error and the subprocess stays alive - stderr output is captured as warning-level log entries (not treated as errors); stdout is discarded - Execution timeout is 30 seconds by default - The subprocess is health-checked every 10 seconds; if unresponsive, it is automatically restarted

Wheel format (application/x-python): the code file is a wheel (a .whl, which is just a zip with a .dist-info/ folder) instead of a bare .py file, allowing third-party pip dependencies (raw source only supports stdlib-only single-file scripts). The published wheel itself is tiny — just worker.py plus a Requires-Dist dependency manifest, no vendored third-party code. LoadCodeAsync resolves and installs those dependencies at worker-load time via uv pip install --target <dir>, which installs both the wheel's own worker.py and its resolved dependencies flat into the same directory — exactly the layout the bootstrap script expects on sys.path (worker.py sitting alongside its now-installed dependencies). Otherwise the subprocess lifecycle, Unix-socket frame protocol, and health-check/auto-restart behavior are identical to the raw-source format.

Layout (fixed naming convention for the entry point; everything else is whatever the wheel's own build step put there):

worker-1.0.0-py3-none-any.whl
├── worker.py            # entry point; module-level def Process(cloud_event): ...
└── <name>-<version>.dist-info/
    └── METADATA         # Requires-Dist: <dep> lines declare third-party dependencies

Code Requirements: same Process(event) contract as PythonEngine (see above) -- by the time worker.py is exec'd, uv pip install --target has already resolved and installed its declared dependencies into the same flat directory, already on sys.path, so imports of those dependencies resolve normally.

No import-allowlist equivalent for wheels. The raw-source path's ValidatePythonImports/blocked-modules check assumes decodable UTF-8 source text; it doesn't apply to zip bytes, and isn't applied to a wheel's worker.py. Dependencies are expected to be vetted via their Requires-Dist declaration, not restricted at runtime -- a deliberate asymmetry kept as-is when the two formats were merged into one engine, not something the merge changed.

Dependencies are installed at worker-load time, not build time. This keeps the published wheel itself KB-scale (Gitea's PyPI package registry silently fails to index uploads above ~8-10MB, well under what a vendored grpcio/protobuf wheel would need) and means the cluster does need outbound access to a package index at load time. Two indexes are consulted: - Public/default index — PYTHON_DEFAULT_INDEX_URL env var (default https://pypi.org/simple/), configurable, used for everything except virtufin-* packages. - Private virtufin index — fixed at Gitea's PyPI feed, authenticated via VIRTUFIN_PYPI_USERNAME/VIRTUFIN_PYPI_API_KEY env vars (Basic auth, embedded in the index URL), used for virtufin-* packages (e.g. virtufin-workmanager).

Installs run with uv's DEFAULT first-index index strategy, with the private index in the --extra-index-url slot: uv always ranks extra indexes ABOVE the default index and stops at the first index containing a package, so virtufin-* names always resolve from the private index and can never be shadowed by a public package of the same name (dependency-confusion protection). A version the private index lacks fails the install closed instead of falling back to the public index. (The earlier --index-strategy unsafe-best-match flag disabled exactly that protection — under it, a public package shadowing a virtufin-* name with a higher version would win; removed. Both indexes are NOT equally trusted: the default index is public pypi.) A fixed UV_CACHE_DIR (/tmp/worker-sockets/uv-cache) means repeated loads of the same dependency versions within a pod's lifetime hit cache instead of re-downloading. Install timeout: TimeoutConstants.PythonDependencyInstallTimeoutSeconds (60s). See general-workers' Hello/GetTime workers and their @common/scripts/{build_python_wheel,publish_worker_package}.py build/publish pipeline for the reference implementation.

Library updates. A wheel worker is reloaded when a virtufin-* package it runs on is released, direct or transitive. At load the engine records the wheel's own virtufin-* requirements and what uv pip install installed (the install directory's *.dist-info, the worker's own distribution excluded). Every LIBRARY_UPDATE_POLL_SECONDS the Runtime looks up each installed virtufin-* package on the private index's PEP 503 simple pages (uncached, so a release minutes old counts), picking the newest final version that satisfies the wheel's PEP 440 specifier, if it declares one; a result other than what uv installed reloads the worker (stop, a fresh uv pip install, start), unless its config says autoreload=false. The index has no dependency metadata, so only uv sees a cap set by another package: a reload that doesn't install the version it was for isn't repeated for that version, and the next release is a new attempt. A capped package therefore costs one reload per release, never a loop. Third-party packages are never followed.

TypeScriptEngine

MIME Types: text/x-typescript (one ES module) and application/x-npm (an npm pack tarball). One engine and one Runtime actor type (TypeScriptWorker) handle both; LoadCodeAsync tells them apart by the gzip header, as the Python engine tells a wheel from source.

Execution: one persistent node subprocess per worker running the engine's embedded bridge (worker-bridge.mts), over the same Unix-socket frame protocol as the Python and C# source engines (start/started, process/result, stop/stopped, host calls publish/status/state/secrets answered by reply, one demultiplexing reader on each side). Node runs TypeScript with its built-in type stripping: no compiler, no build step, about 25 ms once per load. So a worker uses erasable syntax only (tsc --erasableSyntaxOnly): no enum, namespace or constructor parameter properties. Node's --experimental-transform-types, which would allow them, was removed in Node 26, so the engine does not use it. The Runtime image ships Node 24 LTS (the node:<version>-bookworm-slim binary and npm, about 140 MB unpacked, 45 MB compressed, as layers of their own that stay the same until the Node version changes).

Code format. A worker module exports any of

export async function Start(ctx: WorkerContext): Promise<void> { ... }   // once per activation, before the first event
export async function Process(event: WorkerEvent): Promise<Outputs> { ... }
export async function Stop(): Promise<void> { ... }                      // ctx.stopping is already aborted

or default-exports a worker object (or a class, constructed once) with start(ctx), process(event) and stop(), which is how the @virtufin/worker-devkit bases (WorkerBase, CommandWorker) are deployed (export default new MyWorker()). Every entry point may be async. event is the dict form every subprocess worker sees (the conformance vectors' convention): id, source, specversion, type and every other attribute as a top-level key, text data in data, binary data base64 in data_base64, timestamps as RFC 3339 text. Process returns an event, a list (published in order) or nothing; a non-string data is JSON-encoded, a Uint8Array is binary. ctx is the context of every engine: workerId, config, secrets, stopping (an AbortSignal), publish(events), reportStatus(state, detail), state.get/save/delete/keys and getSecrets(), all returning promises; a refused call rejects with an error named HostRefused (StateConflict for a stale etag). Worker logs (console.*) go to stderr, which the Runtime logs.

Dependencies install with npm at load, into the worker's own directory:

  • text/x-typescript: the module's bare import specifiers are its dependencies (import { WorkerBase } from "@virtufin/worker-devkit" installs @virtufin/worker-devkit), each at its latest version. Node built-ins and relative imports are not packages. To pin a third-party version, deploy an npm package instead.
  • application/x-npm: the tarball's package/package.json main is the worker module (.js, .mjs, .ts or .mts; TypeScript in the package itself is stripped like a module, TypeScript inside node_modules is not, so dependencies ship JavaScript). Its dependencies are installed, every @virtufin/* one at its latest version whatever range it names (workers always run the newest Virtufin libraries); devDependencies and scripts are dropped.

@virtufin/* resolve from the private Gitea npm registry with the Runtime's VIRTUFIN_REGISTRY_TOKEN (handed to npm in its environment and referenced as ${VIRTUFIN_NPM_TOKEN} from a temporary npmrc, never written to disk), every other package from NPM_DEFAULT_REGISTRY (default https://registry.npmjs.org/). npm's scope mapping is absolute, so a public package can never shadow a @virtufin/* name. npm runs with --ignore-scripts --omit=dev: packages that build native addons at install are not supported. The Runtime's registry credentials are removed from the worker subprocess's environment.

Library updates. Like NuGet workers, a TypeScript worker is reloaded when a @virtufin/* package it uses is released. At load the engine records the worker's own @virtufin/* dependencies and every @virtufin/* package npm installed (node_modules, nested copies included). Every LIBRARY_UPDATE_POLL_SECONDS the Runtime re-resolves that closure from the registry's packuments (uncached, with the same token): a range picks like npm, the latest tag when it satisfies the range, else the highest satisfying version. A result other than what npm installed reloads the worker (stop, a fresh npm install, start), unless its config says autoreload=false. Only @virtufin/* packages are followed: a third-party release never reloads a worker.

Lifecycle, ownership, redelivery are the Runtime's, as for every engine; the TypeScript devkit's WorkerBase derives output ids and drops repeated inputs like the C# and Python bases.

CSharpSourceEngine

MIME Type: text/x-csharp

Deploys a worker from C# source text, no nupkg needed: a notebook can send the code it just wrote. The Runtime starts one CSharpSource.Bridge child process per worker (a self-contained JIT process; the Runtime itself is NativeAOT and cannot load compiled code). The bridge resolves the source's NuGet packages, compiles it with Roslyn and runs it.

Example (the greeter worker; the same file is the test fixture tests/Virtufin.WorkManager.Tests/Fixtures/greeter.worker.cs):

#r "nuget: Virtufin.Worker.DevKit"

using System.Text.Json.Nodes;
using System.Threading.Tasks;
using CloudNative.CloudEvents;
using Virtufin.Worker.DevKit;
using Virtufin.Worker.Host;

// Answers {"name": "Ada"} with {"greeting": "Hello, Ada!"}
// (ce-type com.virtufin.greeter.response, topic wrk.greeter.response).
public sealed class Greeter() : WorkerBase<JsonNode, JsonObject>("greeter", new StrictJsonCodec())
{
    protected override Task<JsonObject?> ProcessAsync(CloudEvent input, JsonNode payload, IWorkerContext context) =>
        Task.FromResult<JsonObject?>(new JsonObject
        {
            ["greeting"] = $"Hello, {payload["name"]?.GetValue<string>() ?? "world"}!",
        });
}

Sent {"name": "Ada"}, it publishes {"greeting": "Hello, Ada!"} as com.virtufin.greeter.response on wrk.greeter.response.

Data flow:

sequenceDiagram
    autonumber
    participant Sub as Pub/Sub
    participant RT as Runtime<br/>(CSharpSourceEngine)
    participant Br as CSharpSource.Bridge<br/>(child process)
    participant Feed as NuGet feeds
    participant W as Compiled worker

    Note over RT,W: === LoadCodeAsync (once per worker) ===
    RT->>Br: start, code file path
    Br->>Feed: resolve #r "nuget: …" packages (+ transitive)
    Feed-->>Br: package DLLs
    Br->>Br: Roslyn compile (BCL + host-shared + package DLLs)
    Br->>W: load into a WorkerLoadContext, find the IWorker
    Br-->>RT: ready (or error: the compiler/resolver message)

    Note over Sub,W: === Per message ===
    Sub->>RT: input CloudEvent
    RT->>Br: process frame (protobuf CloudEvent)
    Br->>W: ProcessAsync
    W-->>Br: outputs
    Br-->>RT: result frame (protobuf CloudEventBatch)

Package directives. Lines at the top of the source, before any code (blank and // lines may sit between them):

Directive Meaning
#r "nuget: Virtufin.Worker.DevKit" the latest stable release
#r "nuget: Some.Package, 1.2.3" a NuGet version range; a bare version is a minimum
Virtufin.Strategy.DevKit / Some.Package 1.2.3 the same, in Verso's syntax (* = latest)

They are resolved by the DLL engine's installer (NuGetDependencyInstaller), transitively, from the same two feeds with the same credentials: Virtufin.* from the org feed at the newest release in the range, everything else from DOTNET_DEFAULT_NUGET_SOURCE_URL (default nuget.org) at the lowest version in the range (latest when there is no version). Only NuGet references are supported; #r "some.dll" fails the load. The directive lines are blanked before compiling, so compiler line numbers match your source.

References. The worker compiles against the .NET 10 reference assemblies, CloudNative.CloudEvents, Virtufin.Worker.Host, and every resolved package DLL. There are no implicit usings; write them out. Nullable annotations are enabled.

Host-shared assemblies. A package assembly the bridge ships itself (Virtufin.Worker.Host, CloudNative.CloudEvents, Virtufin.Api.Client, Google.Protobuf, Virtufin.Naming, …) is compiled against and loaded from the bridge's copy, not the package's, so the worker's CloudEvent and IWorker are the host's types. This is the same WorkerLoadContext rule DLL workers follow. Everything else (Worker.DevKit, Core, the domain DevKit packages) loads from the feed.

Entry point. Exactly one non-abstract class implementing IWorker (directly or through a devkit base such as WorkerBase<,>, BehaviorWorkerBase or StrategyWorkerBase), with a public parameterless constructor. It runs as a DLL worker does: every output is published in order. Two such classes fail the load; make helpers abstract. Without one, the original form still works: a static or instance method CloudEvent ProcessAsync(CloudEvent input) (single output, null for none).

Library updates. A source worker with package directives is tracked like a DLL worker (LIBRARY_UPDATE_POLL_SECONDS): when a newer Virtufin.* release is in range, the Runtime reloads it, which resolves the packages and compiles the source again.

Per-event rules. Around every call the Runtime applies the shared rules in Virtufin.Worker.Host (WorkerEventRules), the same for every engine:

flowchart LR
    IN[Input event] --> PREP["PrepareInput<br/>content type, config, secrets"]
    PREP --> W["Your IWorker<br/>(in the bridge)"]
    W -->|outputs| FIN["FinishOutputs<br/>id + source, secrets stripped"]
    FIN --> ROUTE["Route<br/>publishtopic, else ce-type"]
    ROUTE --> OUT[Pubsub]
    W -.->|exception| DLQ["DeadLetter<br/>dlq.&lt;topic&gt;"]

What each rule does, and the conformance vectors that pin them (conformance/worker-event-rules.v1.json, also in the Virtufin.Worker.Host package), are in Architecture: one event.

Running workers locally. Before deploying, the same worker can run in-process: the LocalWorkManager in Virtufin.Behavior.DevKit (worker-devkit 0.34.0) subscribes workers to topics on a local bus and applies the same WorkerEventRules. See Running workers locally in the worker-devkit docs.

Errors.

  • A malformed directive, an unresolvable package, a compile error (worker.cs(line,col): error CS…) or a missing/ambiguous entry point fails the load (LOAD_FAILED, with the message).
  • An exception the worker throws fails that message only; the next one is processed.
  • The load may take up to 90 s cold (60 s NuGet install + 30 s); later loads hit the packages cache in /tmp/worker-sockets/nuget-cache.

Limits. One source file. Package DLLs only (no native assets, no analyzers or source generators). The worker shares the bridge's version of every host-shared assembly listed above.

NativeDllEngine

MIME Type: application/x-native-dll

Loads per-architecture native shared libraries (.so / .dylib / .dll) via NativeLibrary.Load inside the AOT-compiled WorkManager.Runtime process that hosts the worker. Exchanges CloudEvents with the worker's Process export in the worker wire protocol (C ABI v2: a protobuf CloudEvent in, a protobuf CloudEventBatch out) and supports host callbacks (log, gateway_call, free_response, and under ABI v3 publish and report_status) on the VirtufinHost struct. A library declaring ABI v3 also exports Start and Stop, the worker lifecycle (native ABI v3).

Data flow:

sequenceDiagram
    autonumber
    participant Sub as Pub/Sub
    participant WM as WorkManager<br/>(NativeDllEngine,<br/>AOT)
    participant Lib as Native worker<br/>(lib*.so)
    participant Base as "AotNative<W>"
    participant Ctx as WorkerContext

    Note over WM,Lib: === LoadCodeAsync (once per worker) ===
    WM->>WM: extract zip to temp dir
    WM->>WM: find runtimes/<rid>/native/lib*.so
    WM->>Lib: NativeLibrary.Load(path)
    WM->>Lib: GetExport("Process") → fn-ptr
    WM->>Lib: GetExport("FreeResult") → fn-ptr
    WM->>WM: build VirtufinHost vtable<br/>(log, gateway_call, free_response)
    Note over WM,Lib: --- one-time, <10ms startup ---

    Note over Sub,Lib: === Per-message ProcessAsync ===
    Sub->>WM: input CloudEvent
    WM->>WM: CloudEventProtoCodec.Encode(ce) → inBytes
    WM->>WM: inBuf = Marshal.AllocHGlobal(inBytes)
    WM->>Lib: Process(host, inBuf, inLen, &outBuf, &outLen)
    Lib->>Base: ProcessStatic<T>(host, inBuf, inLen, ...)
    Base->>Base: CloudEventProtoCodec.Decode(inBuf) → ce
    Base->>Base: _instances.GetOrAdd(host) → worker
    Base->>Ctx: worker.ProcessAsync(ce, context) (Api = EnvironmentApi, Logger = native log callback)
    Ctx-->>Base: output CloudEvents (zero or more)
    Base->>Base: CloudEventProtoCodec.EncodeBatch(outputs) → outBytes
    Base->>WM: status 0, outBuf = AllocHGlobal(outBytes), outLen = outBytes.Length
    WM->>WM: Marshal.Copy(outBuf, span)
    WM->>Lib: FreeResult(outBuf)
    WM->>WM: CloudEventProtoCodec.DecodeBatch(span)
    WM->>Sub: publish each output, in order

Isolation caveat: the engine runs inside the Runtime process; an unrecoverable native error terminates that Runtime pod and every worker it hosts (the WorkManager itself is unaffected). See NativeDllEngine Workers for the full author guide, C ABI, and defensive-coding requirements.

Performance: low-latency (~50-200 µs/message) without a subprocess boundary. Suitable for trusted, pre-compiled native code.

Code requirements: - Worker is a per-architecture shared library at runtimes/<rid>/native/lib<library>.so (or .dylib / .dll), packaged in a standard NuGet .nupkg. - The worker's .nuspec declares virtufin* extension elements (virtufinAbiVersion, virtufinEntryPoint, virtufinFreeResult, virtufinLibrary) inside <metadata>. - Two exported C functions: Process(const VirtufinHost*, const uint8_t* in, int32_t in_len, uint8_t** out, int32_t* out_len) and FreeResult(uint8_t*). - See NativeDllEngine Workers for a full example, the vendored virtufin_worker_api_c.h and the header-only C codec virtufin_worker_api_pb.h.

Error handling: - ABI mismatch (NotSupportedException). - Missing / malformed .nuspec, invalid library basename, missing RID entry (InvalidWorkerNuspecException / UnsupportedArchitectureException). - Process returns 1 (VIRTUFIN_STATUS_WORKER_ERROR) → engine throws InvalidOperationException with the worker's UTF-8 message; the event fails like any worker error. - Process returns 2 or any other status, or a malformed batch → engine throws InvalidWorkerResponseException.

DotNetDllEngine (inside the Runtime process, default)

MIME Type: application/x-dotnet-dll

Default engine as of LIBRARY_VERSION 0.0.59. The engine is AOT-compiled into the WorkManager.Runtime binary and embeds the .NET runtime (CoreCLR) on first use via libhostfxr. Worker DLLs are loaded into a per-worker, collectible AssemblyLoadContext. Per-call latency: sub-microsecond (direct in-process method dispatch, no socket, no JSON).

See In-Process DotNet DLL Workers for the full architecture, setup contract, and trade-offs.

Data flow:

sequenceDiagram
    autonumber
    participant Sub as Pub/Sub
    participant WM as WorkManager<br/>(DotNetDllEngine,<br/>AOT + CoreCLR)
    participant Ctx as AssemblyLoadContext
    participant W as IWorker impl

    Note over WM: === First LoadCodeAsync (one-time) ===
    WM->>WM: HostFxrBootstrap<br/>ensure CoreCLR loaded
    Note over WM,Ctx: === Per-worker LoadCodeAsync ===
    WM->>WM: WorkerNupkgLoader.Extract
    WM->>WM: NuGetDependencyInstaller<br/>resolve+download nuspec deps
    WM->>Ctx: new AssemblyLoadContext
    WM->>Ctx: LoadFromStream(worker.dll)
    WM->>Ctx: scan for IWorker impl
    WM->>W: Activator.CreateInstance

    Note over Sub,W: === Per-message ProcessAsync ===
    Sub->>WM: input CloudEvent
    WM->>W: worker.ProcessAsync(ce, context)<br/>(direct call, <1 µs)
    W-->>WM: output CloudEvents (0..n)
    WM->>Sub: reply CloudEvent

Code Requirements: Same as before — workers implement Virtufin.Worker.Host.IWorker (usually via Worker.DevKit's WorkerBase). The worker DLL is published as a NuGet .nupkg with a lib/<tfm>/<id>.dll entry and a <virtufinLibrary> extension in the nuspec. See DotNet DLL Workers for the full author guide.

Dependencies are resolved at worker-load time, not vendored into the nupkg. The published nupkg's lib/<tfm>/ contains only the worker's own DLL; third-party dependencies are declared in the nuspec's <dependencies> element (standard NuGet convention — populated from the worker's PackageReferences at build time) and resolved+downloaded by NuGetDependencyInstaller (Virtufin.WorkManager.Engine.DotNetDll.Bridge) the first time the worker loads, using NuGet.Protocol/NuGet.Resolver driven in-process (no dotnet SDK is available in the runtime image, unlike a build machine, so this can't shell out to dotnet restore). This is the .NET analog of PythonEngine's wheel-format load-time uv pip install — see that engine's section above for the motivating size-limit issue this design avoids. Two sources are consulted: a configurable public source (DOTNET_DEFAULT_NUGET_SOURCE_URL, default nuget.org) for everything, and the existing authenticated private virtufin Gitea feed (reusing VIRTUFIN_REGISTRY_USER/VIRTUFIN_REGISTRY_TOKEN, the same credentials already used to fetch the nupkg itself) for Virtufin.* packages. Resolved packages are cached under a fixed /tmp/worker-sockets/nuget-cache so repeated loads of the same versions within a pod's lifetime don't re-download. Vendored sibling DLLs (old-style nupkgs published before this change) still take precedence on a filename conflict, so already-published nupkgs continue to load unchanged.

Discovery: The engine scans the loaded assembly for the first non-abstract, non-interface type implementing IWorker. If no implementation is found, an InvalidOperationException is thrown.

Virtufin.Worker.Host provides the IWorker contract, the worker context (IWorkerContext), the AOT-native bridge (AotNative<W>) and the CloudEventProtoCodec (the worker wire protocol); see Worker.Host. Workers are authored with Worker.DevKit.

The same loading pipeline runs in-process for notebooks and tests: Virtufin.WorkManager.Engines.Local provides DotNetDllWorkerLoader (nupkg + nuspec dependencies + WorkerLoadContext) and CSharpSourceWorkerLoader (the bridge's WorkerCompiler, no child process) behind the client's IWorkerLoader.

Adding an Engine

  1. Implement IEngine (in Virtufin.WorkManager.Engines) in a new Virtufin.WorkManager.Engine.* project.
  2. Add a WorkerRuntimeActorBase<TEngine> subclass with its own [DaprActor] type name in Virtufin.WorkManager.Runtime/Actors.
  3. Add the mime type to WorkerMimeTypes (Virtufin.WorkManager.Client) and map it to that actor type in RuntimeActorTypeNames (Virtufin.WorkManager.Shared).
  4. Optionally, an IWorkerLoader in Virtufin.WorkManager.Engines.Local so the format also runs under a LocalWorkManager.

Engine Lifecycle

Engines are created fresh for each worker's actor activation. This means:

  • Engines should be stateless (or handle their own state initialization)
  • Engine instances are not reused across workers
  • When an actor reactivates (idle timeout, Runtime restart), a new engine is created and the code loaded again

Thread Safety

Engine instances are specific to a single worker and accessed sequentially (one message at a time). However:

  • PythonEngine and CSharpSourceEngine stop their subprocess on dispose
  • DotNetDllEngine unloads the worker's load context on dispose
  • NativeDllEngine implements IAsyncDisposable; dispose frees the loaded library and releases the host struct

Performance Characteristics

The figures below are estimates. Measured numbers per engine (engine overhead, pipeline latency and saturation), from the nightly benchmark runs, are at benchmarks.docs.virtufin.com.

Engine Startup Per-Message Throughput Recommended For
CSharpSourceEngine ~2s (Roslyn compilation) + package resolution (cold: network-bound) < 1ms High High-throughput topics, complex logic
DotNetDllEngine < 10ms < 1ms Highest Pre-compiled .NET assemblies
PythonEngine (raw source) ~500ms (subprocess start) 1-5ms Medium Prototyping, scripting, stdlib-only single-file Python
PythonEngine (wheel) ~500ms + uv pip install (cold cache: network-bound; warm cache: fast) 1-5ms Medium Python workers needing third-party pip dependencies
TypeScriptEngine (module) ~0.5 s (node start + type stripping; measured 464 ms, first load) + npm install when it imports packages (one @virtufin/* import measured 2.0 s cold, 0.75 s with a warm npm cache) ~50 µs (measured p50 46 µs, p99 134 µs; Python 39/135 µs on the same machine) Medium TypeScript/JavaScript workers, npm ecosystem
NativeDllEngine < 10ms (per-RID resolve + dlopen) ~50-200 µs High Pre-compiled native libraries (C/C++/Rust/Zig) where latency matters and the worker is trusted

Python Engine Internals

The Python engine uses a persistent subprocess that communicates via a Unix domain socket:

  • The subprocess is started once at LoadCode and stays alive for the worker's lifetime
  • Code is written to temp files under /tmp/worker-sockets/code/<id>/: code.py (the worker source) and code.bootstrap.py (a thin bootstrap that exec()s the code file and runs the socket frame loop)
  • A Unix domain socket at /tmp/worker-sockets/<id>.sock is bound by the C# side; the Python subprocess connects to it as a client (--socket)
  • Engine and subprocess exchange binary frames (SubprocessFrames): a 4-byte big-endian length (of type byte + payload), a 1-byte frame type, then the payload
  • Frame types: ready (1, subprocess → engine after init), process (2, payload: a protobuf CloudEvent), result (3, payload: a protobuf CloudEventBatch, empty = no output), error (4, payload: a UTF-8 message), ping/pong (5/6, heartbeat), shutdown (7, graceful stop), start/started (8/9, payload: EngineStartPayload JSON), stop/stopped (10/11, payload: the budget in ms, 4 bytes big-endian), and the worker's host calls publish (12, a CloudEventBatch), status (13), state (14), secrets (15), each prefixed by a 4-byte call id and answered by reply (16: the call id, a status byte ok/refused/invalid/conflict/failed, the result)
  • A reader thread demultiplexes the socket (replies to the waiting call, ping answered at once, requests queued) and the main thread handles start/process/stop one at a time
  • The subprocess sends ready before entering the processing loop; the engine blocks until it receives it (or an error frame on init failure)
  • The CloudEvents protobuf codec is pure Python inside the bootstrap, so workers stay stdlib-only
  • A health check pings the subprocess every 10 seconds with an empty ping; if the subprocess fails to respond, it is automatically restarted on the next ProcessAsync call
  • Stderr is captured as a Warning-level log channel, not treated as an error
  • Stdout is discarded — all protocol communication goes over the Unix domain socket

Recommendations

  • Use CSharpSourceEngine for high-throughput or latency-sensitive topics
  • Use PythonEngine for rapid prototyping, scripting, or when Python-specific packages are needed
  • Use TypeScriptEngine for TypeScript/JavaScript workers and the npm ecosystem; per-message cost is the Python engine's
  • The first Roslyn compilation in CSharpSourceEngine incurs a ~2s startup cost; subsequent messages are sub-millisecond
  • Python worker subprocess overhead is ~500ms at startup and 1-5ms per message