Production Checklist

This checklist is a production admission gate, not a collection of suggestions. Pause traffic cutover whenever a critical item has no owner, evidence, or rollback path.

This page is for the first launch of a new environment. For a version upgrade, first determine whether old and new versions may run together, then use Upgrade & Migration to choose a rolling update, maintenance window, or separate migration.

Artifact and configuration

• Every node is pinned to the same commit SHA and artifact digest.
• The build toolchain, dependency acquisition, and artifact retention are reproducible.
• Every node uses explicit -config, and the configuration version is in the release record.
• The current binary loads wukongim.toml with no unknown fields.
• WK_* overrides are audited; list values replace the complete list as JSON.
• Configuration, logs, and diagnostics do not expose complete secrets.

Topology and storage

• Every deployment is treated as a cluster; node IDs, data directories, and transport addresses are unique.
• hash_slot_count = 256, and replica counts fit the node count and failure domains.
• Node inventory, cluster ID, and replica policy match on every node.
• Node transport is mutually reachable but closed to unrelated networks.
• Every node has an independent persistent disk with capacity, IO latency, inode, and disk-full alerts.
• Recreating a container or process does not remove data, logs, or required state.

Network and security

• Advertised client TCP/WS/WSS addresses are reachable from target networks.
• TLS terminates and validates correctly at the load balancer, proxy, or service mesh.
• Product HTTP APIs are on a trusted network or behind an external authentication proxy.
• Manager authentication is enabled and all fixed users, passwords, and JWT secrets are replaced.
• Join tokens and other capability secrets are distributed by a secret system and can rotate.
• Manager, metrics, debug, benchmark, diagnostics, and transport each follow least exposure.
• Benchmark and debug APIs are disabled by default; temporary diagnostics have approval, expiry, and audit.

Readiness, observability, and alerts

• Load balancers use /readyz, never /healthz, for traffic admission.
• A /readyz 503 removes the node and triggers an alert.
• Process, CPU, memory, disk, FD, goroutine, and network signals are visible per node.
• /metrics is collected, with alerts for critical Gateway, Controller, Slot, Channel, storage, and delivery signals.
• Logs are centralized, identify the node, have bounded retention, and omit sensitive payloads.
• Alerts have an on-call owner, escalation path, and corresponding runbook.

Data protection and recovery

• Recovery point objective (RPO) and recovery time objective (RTO) are defined.
• Backups cover the correct nodes, data scope, and cluster generation rather than an arbitrary directory.
• Recovery has been exercised in isolation and verified with message reads and new writes.
• Restore maintenance cannot admit business traffic through /readyz.
• Current configuration, artifact digest, node inventory, and secret-recovery procedure are retained.

Use Backup & Restore to verify the current plan revision and complete archive. A backup without a recovery drill is not complete evidence.

Capacity and failure

• Capacity tests approximate real online users, message rates, channel counts, and large-group sizes.
• Results record version, configuration, hardware, workload, latency, error rate, and resource headroom.
• Queues, concurrency, memory, FDs, disk, and network have explicit headroom and backpressure expectations.
• Pre-production exercises node exit, restart, disk pressure, and downstream webhook failure.
• A single peak-QPS result is not treated as sustainable production capacity.

First rollout and rollback

1. Freeze the artifact, configuration, and change window.
2. Complete readiness, end-to-end messaging, and failure checks in pre-production.
3. Start nodes according to the approved first-launch plan; wait for /readyz before proceeding.
4. Send a small traffic percentage and observe errors, latency, queues, and storage.
5. Before expanding traffic, verify product sends, client send/receive, reconnect sync, and webhook boundaries.
6. On anomalies, stop progression, remove new nodes from traffic, and execute the rehearsed rollback.

After all items pass, retain the checklist, logs, metric snapshots, artifact digest, and approvals as release evidence.