Skip to content

Troubleshooting

Common failure modes and recovery procedures for the Virtufin WorkManager.

See also: Development guide → Troubleshooting for service-startup, port-conflict, Python-engine, and protoc-generation issues. Deployment guide → Troubleshooting for Kubernetes pod, image-pull, and worker-recovery issues.

Worker lifecycle failures

Worker stuck in LOADING, or LOAD_FAILED (never running)

Symptom: ListWorkers shows a worker as LOADING long after CreateWorker, or LOAD_FAILED; a StartWorker sent meanwhile never took effect.

Likely causes: - The worker code is unreachable (URL fetch failed, host not allow-listed) - A dependency could not be resolved, or a secret reference is missing - The C# engine failed to compile the source (Roslyn error) - The .NET DLL or native engine failed to load the package - The Python engine subprocess crashed during startup

Diagnose: the load-failed lifecycle event on sys.workmanager.worker.lifecycle carries error_type and error_message (see Worker Lifecycle). The engines run in the WorkManager.Runtime, so their errors are in its logs, not the WorkManager's:

# Load, compile and subprocess errors (all engines)
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "<worker id>\|failed to load"

# For Python workers, the subprocess stderr
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "python\|subprocess"

# For C# workers, the Roslyn compilation errors
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "roslyn\|error CS"

A worker left LOADING by a WorkManager replica that died mid-load is re-driven when a WorkManager replica starts.

"Worker not found" after restart

Symptom: After a WorkManager restart, workers that were Running are no longer found.

Likely cause: "not found" means the catalog has no record for the id (sys.workmanager.worker.<id>); the WorkManager keeps no worker in memory. A worker that exists but stopped receiving after a Runtime restart is not yet reactivated: its actor resubscribes on its next invocation, or within RUNTIME_WORKER_POKE_INTERVAL_SECONDS (default 60 s) when the WorkManager pokes it.

Diagnose:

# Check the Runtime's activation (self-heal) logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "<worker id>"

A running worker whose self-heal fails publishes a failed lifecycle event (once) and retries on every later activation; see Inspecting worker records for the records themselves.

Fix: Call RecoverWorkers to poke every running worker now (see "Force recovery re-run" below).

Sharded groups

FAILED_PRECONDITION: shard 'x' slot N is already running

Likely cause: two live members of the same shard share a shard_index and at least one topic. StartWorker refuses the second one.

Diagnose: ListWorkers returns shard, shard_size and shard_index.

# via the SDK or the gateway: list workers and group them by shard

Fix: delete or re-index the copy that should not run. Indices must be unique per (shard, topic); two shards with disjoint topics may reuse a number.

FAILED_PRECONDITION: shard 'x' slot N is already claimed by worker <id>

Likely cause: that worker is starting or loading with a start requested, so the slot is taken although it isn't Running yet.

Fix: as above. If the holder really is stopped (a WorkManager died before it released the slot), its claim expires 5 minutes after it was taken; a start after that takes the slot over. A claim whose worker no longer exists is taken over at once.

FAILED_PRECONDITION: shard 'x' already runs with shard_size M

Likely cause: members of one shard disagree about shard_size. Two sizes compute different slots for the same event, so it could be processed twice or not at all.

Fix: recreate the member with the size the rest of the shard uses. The fields are immutable; resizing means recreating the whole shard (or running a new generation name and switching over).

shard X on [...] has N of M members running

Symptom: the warning appears at StartWorker, and some events on the topic are never processed (no output, no dead letter).

Likely cause: the shard is only partly deployed, or a member is stopped. For every missing shard_index, the other members receive those events and drop them — by design, there is no failover.

Fix: start the missing members. Check ListWorkers for the shard's indices.

A sharded worker did not catch up after a move

Not a failure: sharded workers ignore input cursors (a skipped event would pin a cursor), so on a move they start every input topic at the tail, exactly like replay=off. Commands keep their cursor.

A sharded worker processed the same event twice

Likely cause: two members ran with the same index for a time — usually two starts racing, or a rolling deployment where an old pod and a new one both subscribed. The duplicate check is advisory against concurrent starts.

Fix: start a shard's members sequentially (or from one client) and stop the old generation before starting the new one under the same name.

Python engine failures

"Python worker process exited with code N"

Common causes:

Exit code Meaning Fix
127 Python not found The WorkManager.Runtime image ships python3; check a custom image
1 Worker code raised an unhandled exception Check the worker code for runtime errors
-1 Killed by SIGTERM The engine called StopProcessAsync (expected during shutdown)
137 Killed by OOM Increase memory limits or reduce worker memory footprint

Diagnose:

# Check the Python engine logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "python\|sandbox"

Sandbox module import errors

Symptom: Worker logs show ImportError: <module> is not allowed.

Likely cause: The worker code imports a module the Python engine blocks (subprocess, os, shutil, socket, requests, urllib, http.client and the mail/FTP/telnet clients; the list is built in).

Fix: Either remove the import from the worker code, or add the module to PYTHON_ALLOWED_PACKAGES (note: this only allows pip packages, not stdlib modules).

Code fetched from URL fails

Symptom: LoadCode returns success but the worker fails to start.

Likely cause: The fetched content doesn't match the expected MIME type (loaded but unparseable), or the SHA-256 hash doesn't match (if ContentSha256 is set).

Diagnose:

# The Runtime fetches URL sources itself
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "fetch\|url\|sha256"

# Verify the host is in the WorkManager's allow-list
kubectl get deploy -n virtufin workmanager --context medusa -o yaml | grep -A1 ALLOWED_CODE_SOURCE_HOSTS

C# / .NET engine failures

C# source compilation errors

Symptom: Worker is created but the C# engine returns a compilation error when loading the source.

Likely cause: The C# source has syntax errors (the message carries worker.cs(line,col): error CS…), references types not available to it (add a #r "nuget: <Id>" directive), or a #r package that cannot be resolved ("Could not resolve the worker's NuGet packages"). See engines.md.

Diagnose:

# Check the Roslyn compilation error in the logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "error CS\|roslyn"

Fix: Fix the worker source code and call LoadCode with the updated bytes.

.NET DLL load errors

Symptom: LoadCode returns success but the worker fails with InvalidOperationException: Assembly load failed.

Likely cause: A dependency DLL in the ZIP is missing, has a version conflict with the WorkManager's assemblies, or is for the wrong platform (any CPU vs x64 vs ARM).

Diagnose:

# Check the assembly load logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "assembly\|dependency\|resolving"

State store failures

Worker state not persisting across restarts

Symptom: Workers created before a restart are not recovered.

Likely cause: The state store (Redis/Valkey) behind the API was unavailable or lost its data.

Diagnose: check that the records exist; see Inspecting worker records.

Memory and resource issues

Memory leak in long-running workers

Symptom: A WorkManager.Runtime pod's memory usage grows over time without bound.

Likely cause: Each worker holds an engine instance with a Python subprocess. If workers are not being deleted when no longer needed, the subprocess count grows.

Diagnose:

# Count Python subprocesses
ps aux | grep -c "python3.*worker"

# Compare with the worker count
grpcurl -plaintext localhost:5002 virtufin.WorkManager/ListWorkers | grep -c '"id"'

Fix:

  1. Ensure your application calls DeleteWorker when workers are no longer needed
  2. Increase the WorkManager.Runtime's memory limits if legitimate load

Recovery procedures

Inspecting worker records

The records live in the Redis behind the API's Dapr state store (redis-primary.redis.svc.cluster.local), under the API's app id as the key prefix: the catalog at api||sys.workmanager.worker.<id> (and the list at api||sys.workmanager.worker.index), the Runtime's own at api||sys.workmanager-runtime.worker.<id>. Each is a hash whose data field holds the JSON. Read them, never write or delete them by hand: the WorkManager's etag compare-and-swap depends on them.

# In a redis-cli session on the Redis primary (namespace redis), read-only:
SCAN 0 MATCH 'api||sys.workmanager.worker.*' COUNT 1000
HGET 'api||sys.workmanager.worker.<id>' data

Restart a worker

Stop and start it (StopWorker, then StartWorker); the Runtime keeps its code loaded. To reload its code, LoadCode it with the same source. To restart the WorkManager itself:

kubectl rollout restart deployment/workmanager -n virtufin --context medusa
kubectl rollout status deployment/workmanager -n virtufin --context medusa

Force recovery re-run

# Poke every running worker's Runtime actor (a deactivated one reactivates,
# reloads its code and resubscribes)
grpcurl -plaintext localhost:5002 virtufin.WorkManager/RecoverWorkers

Reporting issues

If none of the above resolves your issue, gather the following and file a Gitea issue:

  • WorkManager version (LIBRARY_VERSION from the build info endpoint)
  • Worker ID(s) affected
  • Engine type (Python / C# source / .NET DLL)
  • Worker code (or a minimal reproduction)
  • Full error message and gRPC status code
  • Relevant log lines from the WorkManager and the WorkManager.Runtime