Troubleshooting
Common failure modes and recovery procedures for the Virtufin WorkManager.
See also: Development guide → Troubleshooting for service-startup, port-conflict, Python-engine, and protoc-generation issues. Deployment guide → Troubleshooting for Kubernetes pod, image-pull, and worker-recovery issues.
Worker lifecycle failures
Worker stuck in LOADING, or LOAD_FAILED (never running)
Symptom: ListWorkers shows a worker as LOADING long after
CreateWorker, or LOAD_FAILED; a StartWorker sent meanwhile never took
effect.
Likely causes: - The worker code is unreachable (URL fetch failed, host not allow-listed) - A dependency could not be resolved, or a secret reference is missing - The C# engine failed to compile the source (Roslyn error) - The .NET DLL or native engine failed to load the package - The Python engine subprocess crashed during startup
Diagnose: the load-failed lifecycle event on
sys.workmanager.worker.lifecycle carries error_type and error_message
(see Worker Lifecycle). The engines run
in the WorkManager.Runtime, so their errors are in its logs, not the
WorkManager's:
# Load, compile and subprocess errors (all engines)
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "<worker id>\|failed to load"
# For Python workers, the subprocess stderr
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "python\|subprocess"
# For C# workers, the Roslyn compilation errors
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "roslyn\|error CS"
A worker left LOADING by a WorkManager replica that died mid-load is
re-driven when a WorkManager replica starts.
"Worker not found" after restart
Symptom: After a WorkManager restart, workers that were Running are no longer found.
Likely cause: "not found" means the catalog has no record for the id
(sys.workmanager.worker.<id>); the WorkManager keeps no worker in memory. A
worker that exists but stopped receiving after a Runtime restart is not yet
reactivated: its actor resubscribes on its next invocation, or within
RUNTIME_WORKER_POKE_INTERVAL_SECONDS (default 60 s) when the WorkManager
pokes it.
Diagnose:
# Check the Runtime's activation (self-heal) logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "<worker id>"
A running worker whose self-heal fails publishes a failed lifecycle event
(once) and retries on every later activation; see
Inspecting worker records for the records
themselves.
Fix: Call RecoverWorkers to poke every running worker now (see
"Force recovery re-run" below).
Sharded groups
FAILED_PRECONDITION: shard 'x' slot N is already running
Likely cause: two live members of the same shard share a shard_index and at
least one topic. StartWorker refuses the second one.
Diagnose: ListWorkers returns shard, shard_size and shard_index.
# via the SDK or the gateway: list workers and group them by shard
Fix: delete or re-index the copy that should not run. Indices must be unique
per (shard, topic); two shards with disjoint topics may reuse a number.
FAILED_PRECONDITION: shard 'x' slot N is already claimed by worker <id>
Likely cause: that worker is starting or loading with a start requested, so the
slot is taken although it isn't Running yet.
Fix: as above. If the holder really is stopped (a WorkManager died before it released the slot), its claim expires 5 minutes after it was taken; a start after that takes the slot over. A claim whose worker no longer exists is taken over at once.
FAILED_PRECONDITION: shard 'x' already runs with shard_size M
Likely cause: members of one shard disagree about shard_size. Two sizes
compute different slots for the same event, so it could be processed twice or
not at all.
Fix: recreate the member with the size the rest of the shard uses. The fields are immutable; resizing means recreating the whole shard (or running a new generation name and switching over).
shard X on [...] has N of M members running
Symptom: the warning appears at StartWorker, and some events on the topic
are never processed (no output, no dead letter).
Likely cause: the shard is only partly deployed, or a member is stopped. For
every missing shard_index, the other members receive those events and drop
them — by design, there is no failover.
Fix: start the missing members. Check ListWorkers for the shard's indices.
A sharded worker did not catch up after a move
Not a failure: sharded workers ignore input cursors (a skipped event would pin a
cursor), so on a move they start every input topic at the tail, exactly like
replay=off. Commands keep their cursor.
A sharded worker processed the same event twice
Likely cause: two members ran with the same index for a time — usually two starts racing, or a rolling deployment where an old pod and a new one both subscribed. The duplicate check is advisory against concurrent starts.
Fix: start a shard's members sequentially (or from one client) and stop the old generation before starting the new one under the same name.
Python engine failures
"Python worker process exited with code N"
Common causes:
| Exit code | Meaning | Fix |
|---|---|---|
| 127 | Python not found | The WorkManager.Runtime image ships python3; check a custom image |
| 1 | Worker code raised an unhandled exception | Check the worker code for runtime errors |
| -1 | Killed by SIGTERM | The engine called StopProcessAsync (expected during shutdown) |
| 137 | Killed by OOM | Increase memory limits or reduce worker memory footprint |
Diagnose:
# Check the Python engine logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "python\|sandbox"
Sandbox module import errors
Symptom: Worker logs show ImportError: <module> is not allowed.
Likely cause: The worker code imports a module the Python engine blocks
(subprocess, os, shutil, socket, requests, urllib, http.client
and the mail/FTP/telnet clients; the list is built in).
Fix: Either remove the import from the worker code, or add the module
to PYTHON_ALLOWED_PACKAGES (note: this only allows pip packages, not stdlib
modules).
Code fetched from URL fails
Symptom: LoadCode returns success but the worker fails to start.
Likely cause: The fetched content doesn't match the expected MIME type
(loaded but unparseable), or the SHA-256 hash doesn't match (if
ContentSha256 is set).
Diagnose:
# The Runtime fetches URL sources itself
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "fetch\|url\|sha256"
# Verify the host is in the WorkManager's allow-list
kubectl get deploy -n virtufin workmanager --context medusa -o yaml | grep -A1 ALLOWED_CODE_SOURCE_HOSTS
C# / .NET engine failures
C# source compilation errors
Symptom: Worker is created but the C# engine returns a compilation error when loading the source.
Likely cause: The C# source has syntax errors (the message carries
worker.cs(line,col): error CS…), references types not available to it (add a
#r "nuget: <Id>" directive), or a #r package that cannot be resolved ("Could not
resolve the worker's NuGet packages"). See
engines.md.
Diagnose:
# Check the Roslyn compilation error in the logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "error CS\|roslyn"
Fix: Fix the worker source code and call LoadCode with the
updated bytes.
.NET DLL load errors
Symptom: LoadCode returns success but the worker fails with
InvalidOperationException: Assembly load failed.
Likely cause: A dependency DLL in the ZIP is missing, has a version conflict with the WorkManager's assemblies, or is for the wrong platform (any CPU vs x64 vs ARM).
Diagnose:
# Check the assembly load logs
kubectl logs -n virtufin -l app=workmanager-runtime --context medusa | grep -i "assembly\|dependency\|resolving"
State store failures
Worker state not persisting across restarts
Symptom: Workers created before a restart are not recovered.
Likely cause: The state store (Redis/Valkey) behind the API was unavailable or lost its data.
Diagnose: check that the records exist; see Inspecting worker records.
Memory and resource issues
Memory leak in long-running workers
Symptom: A WorkManager.Runtime pod's memory usage grows over time without bound.
Likely cause: Each worker holds an engine instance with a Python subprocess. If workers are not being deleted when no longer needed, the subprocess count grows.
Diagnose:
# Count Python subprocesses
ps aux | grep -c "python3.*worker"
# Compare with the worker count
grpcurl -plaintext localhost:5002 virtufin.WorkManager/ListWorkers | grep -c '"id"'
Fix:
- Ensure your application calls
DeleteWorkerwhen workers are no longer needed - Increase the WorkManager.Runtime's memory limits if legitimate load
Recovery procedures
Inspecting worker records
The records live in the Redis behind the API's Dapr state store
(redis-primary.redis.svc.cluster.local), under the API's app id as the key
prefix: the catalog at api||sys.workmanager.worker.<id> (and the list at
api||sys.workmanager.worker.index), the Runtime's own at
api||sys.workmanager-runtime.worker.<id>. Each is a hash whose data field
holds the JSON. Read them, never write or delete them by hand: the
WorkManager's etag compare-and-swap depends on them.
# In a redis-cli session on the Redis primary (namespace redis), read-only:
SCAN 0 MATCH 'api||sys.workmanager.worker.*' COUNT 1000
HGET 'api||sys.workmanager.worker.<id>' data
Restart a worker
Stop and start it (StopWorker, then StartWorker); the Runtime keeps its
code loaded. To reload its code, LoadCode it with the same source. To
restart the WorkManager itself:
kubectl rollout restart deployment/workmanager -n virtufin --context medusa
kubectl rollout status deployment/workmanager -n virtufin --context medusa
Force recovery re-run
# Poke every running worker's Runtime actor (a deactivated one reactivates,
# reloads its code and resubscribes)
grpcurl -plaintext localhost:5002 virtufin.WorkManager/RecoverWorkers
Reporting issues
If none of the above resolves your issue, gather the following and file a Gitea issue:
- WorkManager version (
LIBRARY_VERSIONfrom the build info endpoint) - Worker ID(s) affected
- Engine type (Python / C# source / .NET DLL)
- Worker code (or a minimal reproduction)
- Full error message and gRPC status code
- Relevant log lines from the WorkManager and the WorkManager.Runtime