Development Guide
Guide for building, testing, and running the Virtufin WorkManager locally.
Table of Contents
- Prerequisites
- Project Structure
- Building
- Running Locally
- Testing
- Debugging
- Generating Protobuf Files
- Troubleshooting
Prerequisites
Required Software
| Software | Version | Purpose |
|---|---|---|
| .NET SDK | 10.0+ | Runtime and build |
| Dapr CLI | 1.18+ | Local development sidecar |
| Docker | Latest | Redis/Valkey for state + pub/sub |
| Python | 3.9+ | Python engine workers (subprocess) |
| buf | 1.40+ | Protobuf code generation (optional) |
Install .NET SDK
# macOS
brew install dotnet
# Linux (Ubuntu)
wget https://packages.microsoft.com/config/ubuntu/22.04/packages-microsoft-prod.deb
sudo dpkg -i packages-microsoft-prod.deb
sudo apt update
sudo apt install dotnet-sdk-10.0
# Windows
winget install Microsoft.DotNet.SDK.10
Install Dapr CLI
# macOS
brew install dapr/tap/dapr
# Linux
curl -fsSL https://raw.githubusercontent.com/dapr/cli/master/install/install.sh | bash
No dapr init is needed: deploy-local's scripts/deploy.sh installs the pinned
Dapr runtime into its own .local/dapr (see below).
Project Structure
src/
├── Virtufin.WorkManager/ # WorkManager: gRPC API, worker catalog, routing
├── Virtufin.WorkManager.Runtime/ # Runtime: one actor per worker, hosts the engines
├── Virtufin.WorkManager.Shared/ # Actor contract + host setup shared by both
├── Virtufin.WorkManager.Protos/ # Proto + generated stubs (packed inside Virtufin.WorkManager.Client)
├── Virtufin.WorkManager.Client/ # IWorkManagerClient, WorkerSpec, IWorkerLoader, GatewayWorkManagerClient
├── Virtufin.WorkManager.Engines.Local/ # in-process nupkg / C# source loaders for LocalWorkManager
├── Virtufin.WorkManager.Engines/ # IEngine, subprocess frames, nupkg layout, VirtufinRegistry
├── Virtufin.WorkManager.Engine.Python/ # Python engine (subprocess)
├── Virtufin.WorkManager.Engine.CSharpSource/ # C# source engine (Roslyn)
├── Virtufin.WorkManager.Engine.DotNetDll/ # .NET DLL engine (hostfxr + ALC)
tests/
├── Virtufin.WorkManager.Tests/ # Service-level tests
├── Virtufin.WorkManager.Runtime.Tests/ # Runtime tests
├── Virtufin.WorkManager.Client.Tests/ # Client library tests
├── python/ # Python client tests + DevKit tests
└── typescript/ # TypeScript client tests
Building
# Build the entire solution
dotnet build Virtufin.WorkManager.slnx
# Build a specific project
dotnet build src/Virtufin.WorkManager/Virtufin.WorkManager.csproj
The build output is placed in src/*/bin/Debug/net10.0/.
Running Locally
Run the backend with virtufin/deploy-local,
cloned as a sibling of workmanager/ (with api and helm). It builds this checkout
from source and starts the API, the WorkManager and the Runtime natively with
dapr run, configured from the helm chart. WebSocket connections are
WebSocketFeed workers on the
Runtime, not a service of their own:
cd ../deploy-local
./scripts/deploy.sh # Redis, placement, scheduler and all services
./scripts/undeploy.sh --service workmanager && ./scripts/deploy.sh --service workmanager # pick up a change
./scripts/undeploy.sh # stop everything
deploy.sh runs scripts/render.sh, which renders the chart into the Dapr components
(components/), the gateway's service registry and each service's env and sidecar
flags; edit the chart or values-local.yaml, not the generated files. Logs go to
.local/logs/<name>.log.
| Service | gRPC | HTTP health |
|---|---|---|
WorkManager (workmanager) |
:25002 (with reflection) |
:25001 (/health, /healthz, /ready) |
Runtime (workmanager-runtime) |
:35002 |
:35001 |
The WorkManager reaches the Runtime's actors through Dapr.
3. Verify
# Health check
curl http://localhost:25001/health
# gRPC reflection service list
grpcurl -plaintext localhost:25002 list
# Interactive explorer (common's launcher)
bash ../common/scripts/run_grpcui_local.sh localhost:25002
Testing
Run All Tests
dotnet test Virtufin.WorkManager.slnx
Run Tests with Coverage
dotnet test Virtufin.WorkManager.slnx \
--collect:"XPlat Code Coverage" \
--results-directory ./TestResults
Run Specific Tests
# By class name
dotnet test --filter "FullyQualifiedName~PythonEngineTests"
# By trait/category
dotnet test --filter "Category=Integration"
Test Projects
| Test Project | Description |
|---|---|
Virtufin.WorkManager.Tests |
Service-level tests (engines, runtime, recovery) |
Virtufin.WorkManager.Client.Tests |
Client library tests (hand-written + generated) |
tests/python/ |
Python client + DevKit tests |
tests/typescript/ |
TypeScript client tests |
Writing Tests
- Place unit tests next to the class they test in
*.Tests/ - Engine tests use mock I/O — they do not spawn real Python subprocesses
- Runtime tests use
WebApplicationFactory<Program>for in-process integration testing
Debugging
Visual Studio Code
.vscode/launch.json includes a debug profile that runs the WorkManager under Dapr:
# In VS Code
code .
# Press F5 → "Launch WorkManager (Dapr)" profile
JetBrains Rider
Rider auto-detects the launchSettings profile. Right-click Virtufin.WorkManager → "Run/Debug" → choose the "Dapr" launch profile.
Common Debug Scenarios
- Python worker not starting: set a breakpoint in
PythonEngine.StartProcessAsyncand verify theAcceptTcpClientAsynccall completes - A worker doesn't come back after a Runtime restart: check the Runtime's activation logs for the worker and the state store contents (
docker exec -it virtufin-local-redis redis-cli --scan --pattern '*sys.workmanager*') - gRPC serialization errors: enable
EnableDetailedErrorsinProgram.cs(gated onIsDevelopment())
Generating Protobuf Files
Proto files live in src/Virtufin.WorkManager.Protos/proto/. To regenerate stubs after editing a .proto file:
cd src/Virtufin.WorkManager.Protos/proto
buf generate
This regenerates:
- C# stubs →
src/Virtufin.WorkManager/Generated/ - Python stubs →
src/python/virtufin/workmanager/workmanager_pb2*.py - TypeScript stubs →
src/typescript/src/generated/workmanager_pb.js
Generated files are excluded from git.
Regenerate Client Tests
After regenerating stubs, also regenerate the cross-language client tests:
# The repos are NOT a monorepo, so the script is fetched
# directly from the `common` Gitea repo at runtime.
python3 <(curl -L https://git.haenerconsulting.com/virtufin/common/raw/branch/master/scripts/generate-client-tests.py) \
src/Virtufin.WorkManager.Protos/proto/workmanager.proto \
--prefix M
Troubleshooting
"Python worker process exited with code N"
Check the Python engine logs for the StartProcessAsync call. Common causes:
- Python 3 not on PATH — the engine calls
EnsurePythonAvailableAsyncwith a 10s timeout; verify withpython3 --version - Sandbox module blocked — the worker's
importstatement tries to load a blocked module (seePYTHON_BLOCKED_MODULESconfig) - Stdin closed — the worker process may have crashed; check stderr capture in the logs
"Dapr sidecar not responding"
Check the sidecar and placement logs in deploy-local's .local/logs/, and that the
virtufin-local-redis container runs (docker ps). ./scripts/deploy.sh starts
whatever is missing (Redis, placement, scheduler, services) and leaves the rest alone.
"Worker not found" after restart
The WorkManager reads workers from the catalog in the state store; it keeps none in memory. If a worker is missing:
- Verify the state store contains the worker record (
sys.workmanager.worker.<id>) - Check the Runtime's logs for the worker's load (
LoadAsync) errors
Port conflicts
If 5001/5002 are in use, override via env vars:
HttpPort=5101 GrpcPort=5102 dapr run --app-port 5102 -- dotnet run
Or command-line switches:
dotnet run -- --http-port 6001 --grpc-port 6102
Memory leaks (long-running workers)
Each worker holds an engine instance with a Python subprocess. If the worker count grows without bound, check:
- The reclaim hosted service is running and reclaiming dead workers
- The state store TTL on worker records is set (recommended: 24h)
- Your application code is calling
DeleteWorkerwhen workers are no longer needed