Skip to content

Development Guide

Guide for building, testing, and running the Virtufin WorkManager locally.

Table of Contents


Prerequisites

Required Software

Software Version Purpose
.NET SDK 10.0+ Runtime and build
Dapr CLI 1.18+ Local development sidecar
Docker Latest Redis/Valkey for state + pub/sub
Python 3.9+ Python engine workers (subprocess)
buf 1.40+ Protobuf code generation (optional)

Install .NET SDK

# macOS
brew install dotnet

# Linux (Ubuntu)
wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt update
sudo apt install dotnet-sdk-10.0

# Windows
winget install Microsoft.DotNet.SDK.10

Install Dapr CLI

# macOS
brew install dapr/tap/dapr

# Linux
curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | bash

No dapr init is needed: deploy-local's scripts/deploy.sh installs the pinned Dapr runtime into its own .local/dapr (see below).

Project Structure

src/
├── Virtufin.WorkManager/                  # WorkManager: gRPC API, worker catalog, routing
├── Virtufin.WorkManager.Runtime/          # Runtime: one actor per worker, hosts the engines
├── Virtufin.WorkManager.Shared/           # Actor contract + host setup shared by both
├── Virtufin.WorkManager.Protos/           # Proto + generated stubs (packed inside Virtufin.WorkManager.Client)
├── Virtufin.WorkManager.Client/           # IWorkManagerClient, WorkerSpec, IWorkerLoader, GatewayWorkManagerClient
├── Virtufin.WorkManager.Engines.Local/    # in-process nupkg / C# source loaders for LocalWorkManager
├── Virtufin.WorkManager.Engines/          # IEngine, subprocess frames, nupkg layout, VirtufinRegistry
├── Virtufin.WorkManager.Engine.Python/    # Python engine (subprocess)
├── Virtufin.WorkManager.Engine.CSharpSource/  # C# source engine (Roslyn)
├── Virtufin.WorkManager.Engine.DotNetDll/     # .NET DLL engine (hostfxr + ALC)

tests/
├── Virtufin.WorkManager.Tests/            # Service-level tests
├── Virtufin.WorkManager.Runtime.Tests/    # Runtime tests
├── Virtufin.WorkManager.Client.Tests/     # Client library tests
├── python/                                # Python client tests + DevKit tests
└── typescript/                            # TypeScript client tests

Building

# Build the entire solution
dotnet build Virtufin.WorkManager.slnx

# Build a specific project
dotnet build src/Virtufin.WorkManager/Virtufin.WorkManager.csproj

The build output is placed in src/*/bin/Debug/net10.0/.

Running Locally

Run the backend with virtufin/deploy-local, cloned as a sibling of workmanager/ (with api and helm). It builds this checkout from source and starts the API, the WorkManager and the Runtime natively with dapr run, configured from the helm chart. WebSocket connections are WebSocketFeed workers on the Runtime, not a service of their own:

cd ../deploy-local
./scripts/deploy.sh                                  # Redis, placement, scheduler and all services
./scripts/undeploy.sh --service workmanager && ./scripts/deploy.sh --service workmanager   # pick up a change
./scripts/undeploy.sh                                # stop everything

deploy.sh runs scripts/render.sh, which renders the chart into the Dapr components (components/), the gateway's service registry and each service's env and sidecar flags; edit the chart or values-local.yaml, not the generated files. Logs go to .local/logs/<name>.log.

Service gRPC HTTP health
WorkManager (workmanager) :25002 (with reflection) :25001 (/health, /healthz, /ready)
Runtime (workmanager-runtime) :35002 :35001

The WorkManager reaches the Runtime's actors through Dapr.

3. Verify

# Health check
curl http://localhost:25001/health

# gRPC reflection service list
grpcurl -plaintext localhost:25002 list

# Interactive explorer (common's launcher)
bash ../common/scripts/run_grpcui_local.sh localhost:25002

Testing

Run All Tests

dotnet test Virtufin.WorkManager.slnx

Run Tests with Coverage

dotnet test Virtufin.WorkManager.slnx \
  --collect:"XPlat Code Coverage" \
  --results-directory ./TestResults

Run Specific Tests

# By class name
dotnet test --filter "FullyQualifiedName~PythonEngineTests"

# By trait/category
dotnet test --filter "Category=Integration"

Test Projects

Test Project Description
Virtufin.WorkManager.Tests Service-level tests (engines, runtime, recovery)
Virtufin.WorkManager.Client.Tests Client library tests (hand-written + generated)
tests/python/ Python client + DevKit tests
tests/typescript/ TypeScript client tests

Writing Tests

  • Place unit tests next to the class they test in *.Tests/
  • Engine tests use mock I/O — they do not spawn real Python subprocesses
  • Runtime tests use WebApplicationFactory<Program> for in-process integration testing

Debugging

Visual Studio Code

.vscode/launch.json includes a debug profile that runs the WorkManager under Dapr:

# In VS Code
code .
# Press F5 → "Launch WorkManager (Dapr)" profile

JetBrains Rider

Rider auto-detects the launchSettings profile. Right-click Virtufin.WorkManager → "Run/Debug" → choose the "Dapr" launch profile.

Common Debug Scenarios

  • Python worker not starting: set a breakpoint in PythonEngine.StartProcessAsync and verify the AcceptTcpClientAsync call completes
  • A worker doesn't come back after a Runtime restart: check the Runtime's activation logs for the worker and the state store contents (docker exec -it virtufin-local-redis redis-cli --scan --pattern '*sys.workmanager*')
  • gRPC serialization errors: enable EnableDetailedErrors in Program.cs (gated on IsDevelopment())

Generating Protobuf Files

Proto files live in src/Virtufin.WorkManager.Protos/proto/. To regenerate stubs after editing a .proto file:

cd src/Virtufin.WorkManager.Protos/proto
buf generate

This regenerates:

  • C# stubs → src/Virtufin.WorkManager/Generated/
  • Python stubs → src/python/virtufin/workmanager/workmanager_pb2*.py
  • TypeScript stubs → src/typescript/src/generated/workmanager_pb.js

Generated files are excluded from git.

Regenerate Client Tests

After regenerating stubs, also regenerate the cross-language client tests:

# The repos are NOT a monorepo, so the script is fetched
# directly from the `common` Gitea repo at runtime.
python3 <(curl -L https://git.haenerconsulting.com/virtufin/common/raw/branch/master/scripts/generate-client-tests.py) \
  src/Virtufin.WorkManager.Protos/proto/workmanager.proto \
  --prefix M

Troubleshooting

"Python worker process exited with code N"

Check the Python engine logs for the StartProcessAsync call. Common causes:

  • Python 3 not on PATH — the engine calls EnsurePythonAvailableAsync with a 10s timeout; verify with python3 --version
  • Sandbox module blocked — the worker's import statement tries to load a blocked module (see PYTHON_BLOCKED_MODULES config)
  • Stdin closed — the worker process may have crashed; check stderr capture in the logs

"Dapr sidecar not responding"

Check the sidecar and placement logs in deploy-local's .local/logs/, and that the virtufin-local-redis container runs (docker ps). ./scripts/deploy.sh starts whatever is missing (Redis, placement, scheduler, services) and leaves the rest alone.

"Worker not found" after restart

The WorkManager reads workers from the catalog in the state store; it keeps none in memory. If a worker is missing:

  1. Verify the state store contains the worker record (sys.workmanager.worker.<id>)
  2. Check the Runtime's logs for the worker's load (LoadAsync) errors

Port conflicts

If 5001/5002 are in use, override via env vars:

HttpPort=5101 GrpcPort=5102 dapr run --app-port 5102 -- dotnet run

Or command-line switches:

dotnet run -- --http-port 6001 --grpc-port 6102

Memory leaks (long-running workers)

Each worker holds an engine instance with a Python subprocess. If the worker count grows without bound, check:

  • The reclaim hosted service is running and reclaiming dead workers
  • The state store TTL on worker records is set (recommended: 24h)
  • Your application code is calling DeleteWorker when workers are no longer needed