Troubleshooting

Troubleshooting does not mean restarting as soon as an error appears. First answer three questions: when did it start, who is affected, and which layer became unhealthy first? Begin with read-only checks and change the system only after the evidence is clear.

The first ten minutes

1. Pause other changes: pause releases, scaling, restore, and concurrent cluster operations. Record every active task.
2. Record the impact: note the start time and affected users, Channels, nodes, and entry points (TCP, WebSocket, or HTTP).
3. Check liveness and readiness: save the status code and full response from /healthz and /readyz; the /readyz failure reason matters most.
4. Open Manager: inspect nodes, Controller, Slots, Channels, and tasks. Mark anything unhealthy or no longer updating.
5. Align the time range: compare metrics, error logs, and recent release or configuration changes from the same period.

curl -sS -i http://127.0.0.1:5001/healthz
curl -sS -i http://127.0.0.1:5001/readyz
wkcli top --server http://127.0.0.1:5001 --once --json

Replace the address with the real API address. 0.0.0.0 is a listen setting, not a server address that clients can use.

For metric names, copyable PromQL, and result interpretation, see Health & Monitoring. Check collection health before comparing data from the same node and time interval.

Continue from the symptom

/healthz succeeds but /readyz returns 503

1. Save the complete /readyz response and reason.
2. In Manager, check maintenance state, Controller, Slots, and active restore or control tasks.
3. Compare time, version, configuration, and readiness with other nodes.

Do not return traffic to this node or bypass safety checks in restore, scaling, or upgrade procedures.

Clients cannot connect or keep reconnecting

1. Identify whether the user connects through TCP, WebSocket, or HTTP. Do not use the Manager address as a client address.
2. Check the advertised address, DNS, load balancer, TLS, and firewall.
3. Compare connection counts, connection errors, file descriptors, memory, and network by node. Read gateway logs for the same user and time window.

Do not disconnect sessions in bulk or restart every node without evidence.

Messages fail, slow down, or back up

1. Check send error rate and high-percentile latency, then see whether Channel, delivery, or push queues keep growing.
2. Compare nodes to find a single-machine or large-group hot spot.
3. When you know the Channel ID, use an exact lookup instead of listing every Channel.
4. Use pprof for a short, bounded window only after metrics and logs point to CPU, memory, or goroutines.

See Message Flow for the full path. With large groups and high message rates, averages can look normal while maximums and high percentiles are unhealthy.

Controller, Slot, or node state is unhealthy

wkcli node ls --context production
wkcli node diagnose 4 --context production --json

Save state timestamps, control revision, blocked_reasons, tasks, and Slot evidence from the output. A diagnostic recommendation is not permission to perform a write. Node removal must still wait for safe_to_remove=true. Follow Scaling for the procedure.

Disk is low or data looks inconsistent

First record free disk, IO latency, errors, node ID, data paths, and configuration. Do not make space by deleting data files or unknown logs.

wkcli db can inspect only a stopped node, filesystem snapshot, or copied data directory. Start with read-only query, diff, or export. import writes data, must target an explicitly offline store, and does not replace Manager backup and restore.

Backup, restore, or upgrade failed

Save Manager tasks and audits, archive inventory, verification for all 256 physical Slots, and program versions and digests. Do not return traffic because /healthz succeeds during restore, and do not assume arbitrary versions may remain mixed after a failed upgrade. Return to the stop conditions in Backup & Restore or Upgrade & Migration.

Use tools in this order

1. Start with /readyz and Manager; they are the lowest-cost checks.
2. Read Prometheus metrics and error logs from the same time range.
3. Use Top or wkcli top for one node's current resources and short history.
4. Use Diagnostics only after the problem is narrowed to an exact node, Slot, Channel, or task.
5. Time-box and target pprof, then disable Debug access when finished.
6. wkcli bench creates real traffic and data. Reproduce only in an isolated test cluster; never run an unbounded load against production.

Stop increasing diagnostic cost when an earlier step already explains the problem.

What to send when asking for help

• Incident start time, impact, and recent changes.
• Node inventory, versions, cluster identity, and relevant redacted configuration.
• Complete /healthz and /readyz responses and Manager state.
• Metric screenshots or exports and a small relevant log excerpt from the same time range.
• Relevant node, Slot, Channel, task, or trace IDs.
• Checks already performed, their results, and high-risk actions deliberately not performed.

Do not send passwords, complete tokens, user message bodies, or a full unreviewed configuration file.

Troubleshooting is complete only after the product works again, queues are controlled, nodes pass /readyz, monitoring has recovered, and temporary Debug access, sampling, and test traffic have been removed.