Troubleshooting
Choose among six symptoms, inspect evidence and outcomes, and expand full commands or advanced tools as needed.
Record time, impact, and recent changes, then choose a symptom.
Readiness fails
The process responds, but /readyz returns 503.
Connection or reconnect failures
Check protocol, advertised address, network, and gateway.
Send failures or backlog
Compare errors, latency, queues, and node hot spots.
Unhealthy cluster state
Inspect nodes, Controller, Slots, and task evidence.
Disk or data problems
Record disk and data paths before offline read-only checks.
Backup or upgrade failed
Preserve tasks and archives; follow stop conditions.
Pause changes when state is unclear
Pause releases and topology changes if node, Controller, Slot, Channel, or task state is missing, stale, or conflicting. Preserve evidence; avoid bulk restarts, leader movement, and data deletion.
The first ten minutes
Complete collection order and read-only commands
- Pause other changes: pause releases, scaling, restore, and concurrent cluster operations. Record every active task.
- Record the impact: note the start time and affected users, Channels, nodes, and entry points (TCP, WebSocket, or HTTP).
- Check liveness and readiness: save the status code and full response from
/healthzand/readyz; the/readyzfailure reason matters most. - Open Manager: inspect nodes, Controller, Slots, Channels, and tasks. Mark anything unhealthy or no longer updating.
- Align the time range: compare metrics, error logs, and recent release or configuration changes from the same period.
curl -sS -i http://127.0.0.1:5001/healthz
curl -sS -i http://127.0.0.1:5001/readyz
wkcli top --server http://127.0.0.1:5001 --once --jsonReplace the address with the real API address. 0.0.0.0 is a listen setting, not a server address that clients can use.
For metric names, copyable PromQL, and result interpretation, see Health & Monitoring. Check collection health before comparing data from the same node and time interval.
Continue from the symptom
/healthz succeeds but /readyz returns 503
Save the full /readyz response and reason. Inspect maintenance, Controller, Slots, and active tasks in Manager, then compare other nodes.
Check: Identify the specific readiness failure. Do not restore traffic before the restore, scaling, or upgrade stop conditions are cleared.
Readiness: complete check list
- Save the complete
/readyzresponse andreason. - In Manager, check maintenance state, Controller, Slots, and active restore or control tasks.
- Compare time, version, configuration, and readiness with other nodes.
Do not return traffic to this node or bypass safety checks in restore, scaling, or upgrade procedures.
Clients cannot connect or keep reconnecting
Confirm the client protocol and advertised address. Check DNS, load balancer, TLS, firewall, and gateway logs over the same interval.
Check: Narrow the failure to an entry, node, or connection step and retain evidence. Do not disconnect or restart in bulk without evidence.
Connections: complete check list
- Identify whether the user connects through TCP, WebSocket, or HTTP. Do not use the Manager address as a client address.
- Check the advertised address, DNS, load balancer, TLS, and firewall.
- Compare connection counts, connection errors, file descriptors, memory, and network by node. Read gateway logs for the same user and time window.
Do not disconnect sessions in bulk or restart every node without evidence.
Messages fail, slow down, or back up
Inspect send errors and high-percentile latency, then compare node queues. Use an exact Channel lookup when its ID is known.
Check: Identify the node or Channel hot spot. Verify commit, peer receipt, and history recovery separately; averages cannot replace high percentiles.
Messaging: complete check list
- Check send error rate and high-percentile latency, then see whether Channel, delivery, or push queues keep growing.
- Compare nodes to find a single-machine or large-group hot spot.
- When you know the Channel ID, use an exact lookup instead of listing every Channel.
- Use pprof for a short, bounded window only after metrics and logs point to CPU, memory, or goroutines.
See Message Flow for the full path. With large groups and high message rates, averages can look normal while maximums and high percentiles are unhealthy.
Controller, Slot, or node state is unhealthy
Save state timestamps, control revision, blocked_reasons, tasks, and Slot evidence. Pause changes for missing, stale, or conflicting state.
Check: A diagnostic recommendation is not authorization to write. Node removal still requires safe_to_remove=true.
Cluster state: read-only diagnostic commands
wkcli node ls --context production
wkcli node diagnose 4 --context production --jsonSave state timestamps, control revision, blocked_reasons, tasks, and Slot evidence from the output. A diagnostic recommendation is not permission to perform a write. Node removal must still wait for safe_to_remove=true. Follow Scaling for the procedure.
Disk is low or data looks inconsistent
Record free disk, IO, errors, node ID, and data paths. Do not delete data files or unknown logs to make space.
Check: wkcli db reads only stopped nodes, snapshots, or copied stores. An import must write to an explicitly offline target; it does not replace backup and restore.
Disk and data: offline inspection boundaries
First record free disk, IO latency, errors, node ID, data paths, and configuration. Do not make space by deleting data files or unknown logs.
wkcli db can inspect only a stopped node, filesystem snapshot, or copied data directory. Start with read-only query, diff, or export. import writes data, must target an explicitly offline store, and does not replace Manager backup and restore.
Backup, restore, or upgrade failed
Preserve Manager tasks and audits, archive inventory, physical Slot verification, and version digests. Follow the relevant task stop conditions.
Check: A live /healthz during restore does not permit traffic. Mixed-version operation requires explicit release-note support.
Maintenance failures: archives and stop conditions
Save Manager tasks and audits, archive inventory, verification for all 256 physical Slots, and program versions and digests. Do not return traffic because /healthz succeeds during restore, and do not assume arbitrary versions may remain mixed after a failed upgrade. Return to the stop conditions in Backup & Restore or Upgrade & Migration.
Use tools in this order
Start with /readyz, Manager, and metrics and logs over the same interval. Narrow to one node or an exact Slot, Channel, or task. Time-box and target pprof, then disable Debug; use wkcli bench only in an isolated test cluster.
Complete tool order and usage boundaries
- Start with
/readyzand Manager; they are the lowest-cost checks. - Read Prometheus metrics and error logs from the same time range.
- Use Top or
wkcli topfor one node's current resources and short history. - Use Diagnostics only after the problem is narrowed to an exact node, Slot, Channel, or task.
- Time-box and target pprof, then disable Debug access when finished.
wkcli benchcreates real traffic and data. Reproduce only in an isolated test cluster; never run an unbounded load against production.
Stop increasing diagnostic cost when an earlier step already explains the problem.
What to send when asking for help
Provide time, impact, versions and redacted configuration, full readiness responses, and matching metrics with a short log excerpt. Remove passwords, full tokens, and message bodies before sharing.
Help request: complete evidence list
- Incident start time, impact, and recent changes.
- Node inventory, versions, cluster identity, and relevant redacted configuration.
- Complete
/healthzand/readyzresponses and Manager state. - Metric screenshots or exports and a small relevant log excerpt from the same time range.
- Relevant node, Slot, Channel, task, or trace IDs.
- Checks already performed, their results, and high-risk actions deliberately not performed.
Do not send passwords, complete tokens, user message bodies, or a full unreviewed configuration file.
Troubleshooting is complete only after the product works again, queues are controlled, nodes pass /readyz, monitoring has recovered, and temporary Debug access, sampling, and test traffic have been removed.
Completion criteria: The product works, queues are controlled, nodes pass /readyz, monitoring has recovered, and temporary Debug access, sampling, and test traffic are removed.