Health & Monitoring

When checking WuKongIM, start with /readyz and then inspect monitoring. Do not only check whether the process is running.

Run these three commands first

curl -sS -i http://127.0.0.1:5001/healthz
curl -sS -i http://127.0.0.1:5001/readyz
curl -sS http://127.0.0.1:5001/metrics

Replace 127.0.0.1:5001 with the node's real API address. The third endpoint may be unavailable when Prometheus metrics are disabled.

Remember one rule: a successful /healthz only means that the process is alive; only a successful /readyz means that the node can receive product traffic. A failed /readyz returns HTTP 503 and a reason. Save the complete response.

The minimum monitoring set

Start with these four groups. You do not need a complicated dashboard on day one.

Manager Realtime and Top are useful for a quick view of one node. Prometheus shows trends over time. They complement one another, but neither replaces /readyz.

Connect an existing Prometheus

Download prometheus.yml and alerts.yml into one directory. Replace the three private node endpoints, keep one target for a single-node cluster, and set a unique cluster label. Enable node metrics through Observability.

On the Prometheus host, validate the files before loading them through your existing service management process:

promtool check config prometheus.yml
promtool check rules alerts.yml

Confirm every target is UP in Prometheus Targets. For an existing configuration, merge only the relevant scrape_configs and rule_files; do not overwrite other jobs or scrape the same node twice. Configure notification receivers through your existing Prometheus/Alertmanager setup.

Four queries to start with

These queries use the sample's job="wukongim" and scrape-added cluster label; node_id comes from each node. Replace the selector if your job name differs. Check up{job="wukongim"} first: 0 means collection failed, while 1 only means a successful scrape, not successful /readyz. An empty result can mean an unconfigured target, disabled metrics, or an event that has not occurred; do not automatically interpret it as a healthy zero.

Connections: compare nodes

sum by (cluster, node_id) (wukongim_gateway_connections_active{job="wukongim"})

The unit is connections, not distinct users. A skewed node suggests checking routing and connection distribution. For a sudden drop, compare gateway errors and client reconnects over the same interval.

Send failures: inspect reasons

sum by (cluster, node_id, reason) (increase(wukongim_gateway_sendacks_total{job="wukongim",reason!="success"}[5m]))

Shows unsuccessful SENDACKs emitted in the last 5 minutes; it does not count all client timeouts. For auth_fail, check UID, device category, and token. For other reasons, consult Reason Code and the relevant node logs. Metric labels use names; protocol results use numeric codes.

Slow appends: inspect per-node P99

histogram_quantile(0.99,
  sum by (cluster, node_id, le) (
    rate(wukongim_message_append_duration_seconds_bucket{job="wukongim"}[5m])
  )
)

Values are seconds for server-side message append latency, not end-to-end receive latency. If P99 rises, compare disk IO, replication latency, and traffic on that node. Sparse traffic does not provide enough samples for a capacity conclusion.

Delivery backlog: inspect queued batches

max by (cluster, node_id) (wukongim_delivery_recipient_worker_queue_depth{job="wukongim"})

The unit is queued delivery batches, not unread messages or undelivered users. For sustained growth, check delivery workers, downstream nodes, and connection writes before changing capacity.

The downloadable rules cover scrape failures, unsuccessful SENDACKs, and sustained delivery backlogs. The 2m, 10m, and 80% thresholds are starter examples to tune against your baseline; they do not cover all readiness, storage, or cluster failures.

Create these alerts first

• /readyz keeps returning 503 or repeatedly changes state.
• Send errors or latency keep rising.
• A queue keeps growing, or rejection, drops, or delivery failures appear.
• Disk space is low, or memory, file descriptors, or goroutines keep growing.
• Controller, Slot leader, replica, or ISR state is unhealthy.
• Monitoring collection stops, so cluster state becomes unknown.

Do not copy alert numbers from another environment. Observe the normal range for this version under a real product peak, then set thresholds. Large groups, high message rates, and many connections can create one-node or one-channel hot spots, so inspect maximums and high-percentile latency as well as averages.

What to do after an alert

1. Record the start time, impact, and complete /readyz response.
2. Align metrics, Manager state, and error logs from the same time range.
3. If they do not explain the problem, follow Troubleshooting and add Top, diagnostics, or pprof one step at a time.

See Logs & Observability for monitoring and logging settings. Never let an alert automatically move leaders, delete nodes, or perform another topology change.