Scaling

Follow node states to add or remove a node; expand full configuration, commands, and migration actions.

This example starts with nodes 1–3, adds node 4 at 10.0.0.14, and then removes it. Open Manager → Cluster → Nodes.

Add node 4

Join, activate, migrate, then admit traffic.

Remove node 4

Drain and migrate before safe removal.

Requires cluster.node:w. The cluster keeps 256 physical hash slots when nodes change.

Scale-out: add node 4

1. Configure and start the new node

Match cluster.id, replica counts, and join_token to the existing cluster. /healthz checks process liveness; it does not admit traffic.

Full configuration and startup commands

Copy an existing node's configuration and keep unrelated settings unchanged. Update [node], remove nodes from [cluster], and use:

[node]
id = 4
data_dir = "/var/lib/wukongim"

[cluster]
id = "prod-im-a"
listen_addr = "0.0.0.0:7000"
advertise_addr = "10.0.0.14:7000"
seeds = ["10.0.0.11:7000"]
join_token = "the-same-secret-as-existing-nodes"
hash_slot_count = 256
slot_replica_n = 3
channel_replica_n = 3

Match the existing cluster's cluster.id, replica counts, and join_token. Start the process and check liveness:

sudo systemctl enable --now wukongim
curl --fail http://10.0.0.14:5001/healthz

2. Activate it in Manager

Wait for joining, select Activate node, then confirm active.

Complete activation actions

Wait for node 4 to show joining.
Open node 4 and select Activate node.
Wait for status active.

3. Move data in

Set Max slot moves to 1. Plan and start onboarding; advance again only after active tasks reach 0, until no new task is created.

Complete onboarding actions

Set Max slot moves to 1.
Select Plan slot onboarding and confirm that node 4 is the target.
Select Start onboarding. When active tasks reach 0, select Advance onboarding.
Repeat until no new task is created.

4. Verify

Check /readyz on all four nodes and test message send and receive. Only then add 10.0.0.14:5100 and 10.0.0.14:5200 to the load balancer.

Readiness commands and traffic admission

curl --fail http://10.0.0.11:5001/readyz
curl --fail http://10.0.0.12:5001/readyz
curl --fail http://10.0.0.13:5001/readyz
curl --fail http://10.0.0.14:5001/readyz

After all four requests pass and messages work, add 10.0.0.14:5100 and 10.0.0.14:5200 to the load balancer.

Scale-in: remove node 4

Do not stop node 4 first

Remove and stop it only after Manager shows Safe to remove: Yes (safe_to_remove=true). Keep the node running when blocked.

Phase	Required result
Remove traffic → Mark leaving → Enable drain mode	Other nodes carry the load; active, closing, and pending connections are all 0
Plan → Advance scale-in → Refresh	Slot, Channel, and active task counts are 0; Safe to remove: Yes

Set Scale-in max slot moves to 1; advance again after active tasks reach 0. Remove the node only after the gate passes, then stop its service.

Done when: Remaining nodes pass /readyz and messaging works. Retain node 4's data until the product is stable, then follow your retention policy.

Complete scale-in actions and final stop command

Do not stop node 4 first

Remove and stop the server only after Manager shows Safe to remove: Yes.

Remove node 4 from the load balancer and confirm that the other nodes can carry the traffic.
Open node 4, select Mark leaving, and wait for status leaving.
Select Enable drain mode. Wait for active, closing, and pending connection counts to reach 0.
Set Scale-in max slot moves to 1, then select Plan scale-in → Advance scale-in. Advance again after active tasks reach 0.
Select Refresh scale-in status. When Slot, Channel, and active task counts are 0 and Safe to remove: Yes appears, select Remove node.
Finally, run on node 4:

sudo systemctl disable --now wukongim

Check /readyz on the remaining nodes and test messages again. Keep node 4's data directory until the product is stable, then handle it under your data-retention process.

If an action is disabled or Manager shows a blocking reason, stop and keep the node running, then follow Troubleshooting. This procedure requires cluster.node:w.

Stop if blocked: Disabled actions or blocking reasons mean keep the node running and preserve evidence. Continue through Troubleshooting.