CKS: scale-down with separate etcd revokes wrong SSH rules and fails (4.22.1.1)

[gmautner]

problem

On CloudStack 4.22.1.1, a fresh CKS cluster with three separate etcd nodes can be created and scaled from one to two workers successfully, but scaling back to one worker fails with API error 530 while updating SSH network rules.

The backend has already removed the worker when it fails. It revokes the first etcd SSH firewall rule (50000), leaves the control/worker firewall range (2222–2224) active, and then conflicts when creating the replacement range (2222–2223). It also removes the control/worker SSH forwards and the first etcd SSH forward. The cluster becomes Alert with a stale worker count.

This controlled reproduction uses three sequential native API mutations on one fresh cluster, with Kubernetes 1.33.13 throughout. Creation and scale-up were allowed to finish and pass node/pod/API readiness checks before proceeding. There was no Kubernetes upgrade, concurrent lifecycle operation, UI involvement, autoscaling, manual firewall customization, or per-node offering override.

versions

• CloudStack management server and KVM agent: 4.22.1.1. Installed package verification reports only local configuration differences; backend binaries are unchanged.
• Advanced zone, isolated network, CloudStack virtual router, shared NFS primary storage.
• SystemVM Template (KVM), Debian 12 guest OS; uniform offering: 2 vCPU, 4 GiB RAM, fixed 80 GiB root disk for all node types.
• Kubernetes 1.33.13, Calico 3.31.5. The ISO is custom-built with official component releases, includes the native etcd 3.5.24 binaries needed for separate-etcd bootstrap, and includes Headlamp and an upgrade-readiness hook. No upgrade was performed in this reproduction.
• cloud.kubernetes.etcd.node.start.port=50000 (default).
• One control node, three separate etcd VMs, initially one worker. Normal tenant API credentials.

The steps to reproduce the bug

The following are the complete application parameters for the three mutations, submitted via HTTP POST to /client/api. Standard response=json and API-key/signature authentication are omitted here. UUIDs identify this lab's zone, registered 1.33.13 ISO, offering and cluster; substitute equivalent IDs in another environment. No existing network was supplied: CloudStack created the isolated network and its rules automatically.

{
  "command": "createKubernetesCluster",
  "name": "cks-etcd-minimal-20260912",
  "zoneid": "5ebb765f-3f58-4574-acde-4df589dd478e",
  "kubernetesversionid": "759912e1-1fd1-48c9-9d1c-da3de44834cc",
  "serviceofferingid": "db6694af-7532-414d-b5f2-0300d95656f4",
  "noderootdisksize": 80,
  "clustertype": "CloudManaged",
  "hypervisor": "kvm",
  "controlnodes": 1,
  "etcdnodes": 3,
  "size": 1
}

{
  "command": "scaleKubernetesCluster",
  "id": "3b74274b-6554-40c2-b2cf-4329d4308544",
  "size": 2
}

{
  "command": "scaleKubernetesCluster",
  "id": "3b74274b-6554-40c2-b2cf-4329d4308544",
  "size": 1
}

Wait for each async job before proceeding:

Operation	Job ID	Result / completion (UTC−03)
Create, size=1	821341c9-4fdd-45a5-ab0c-7838790880fb	jobstatus=1, 2026-09-12 00:27:59; health passed
Scale, size=2	927ddb07-b482-4b16-a568-926249c79bec	jobstatus=1, 00:29:52; health passed
Scale, size=1	b860f7d3-cd52-4396-b9ce-b3ee03eb4d5d	jobstatus=2, error 530, 00:30:08

queryAsyncJobResult(jobid=...) was polled every five seconds. Read-only listKubernetesClusters, getKubernetesClusterConfig, listPublicIpAddresses, listFirewallRules and listPortForwardingRules calls captured state between steps; kubectl checked nodes, pods and /readyz. No fourth cluster mutation was submitted after the failure. The cluster remains available for investigation.

Expected result

Scale-down succeeds, the cluster reports one worker and Running, and SSH forwarding for the remaining control/worker and all three etcd VMs is preserved.

Actual results and relevant logs

The complete nested error states that 2222–2223 conflicts with the still-active 2222–2224 firewall rule. These are verbatim relevant management-server log lines from the failed job (unrelated lines omitted):

2026-09-12 00:29:57,354 INFO  [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) Removing vm VM instance {"id":519,"instanceName":"i-39-519-VM","state":"Running","type":"User","uuid":"939ddd32-c36f-461b-8771-c3a55d51647f"} from cluster KubernetesCluster {"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"}
2026-09-12 00:30:07,904 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The SSH firewall rule null with the id 1600 was revoked
2026-09-12 00:30:07,905 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1601] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1607] was mark as revoked.
2026-09-12 00:30:07,907 DEBUG [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) The Port forwarding rule [null] with the id [1608] was mark as revoked.
2026-09-12 00:30:08,761 DEBUG [c.c.n.f.FirewallManagerImpl] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f, ctx-0f06110e]) (logid:b860f7d3) Rules ([FirewallRule {"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"}] and [FirewallRule {"id":1610,"networkId":281,"purpose":"Firewall","state":"Staged","uuid":"d2e0ba0c-931d-45a8-a093-770dc73e636e"}]) have conflicting port ranges.
2026-09-12 00:30:08,768 ERROR [c.c.k.c.a.KubernetesClusterScaleWorker] (API-Job-Executor-95:[ctx-4e85bf73, job-11296, ctx-bd0fd81f]) (logid:b860f7d3) Scaling failed for Kubernetes cluster KubernetesCluster {"id":79,"name":"cks-etcd-minimal-20260912","uuid":"3b74274b-6554-40c2-b2cf-4329d4308544"}, unable to update network rules com.cloud.exception.ManagementServerException: Failed to provision firewall rules for SSH access for the Kubernetes cluster : cks-etcd-minimal-20260912
Caused by: com.cloud.exception.NetworkRuleConflictException: The range specified, 2222-2223, conflicts with rule FirewallRule {"id":1606,"networkId":281,"purpose":"Firewall","state":"Active","uuid":"9439cbc3-faeb-4b6a-b7a6-66fdd167343b"} which has 2222-2224
	at com.cloud.network.firewall.FirewallManagerImpl.detectRulesConflict(FirewallManagerImpl.java:486)
	at com.cloud.network.firewall.FirewallManagerImpl.lambda$createFirewallRule$0(FirewallManagerImpl.java:265)
	at com.cloud.utils.db.Transaction.execute(Transaction.java:38)
	at com.cloud.network.firewall.FirewallManagerImpl.createFirewallRule(FirewallManagerImpl.java:255)

Rule identities reconstructed from the rule-application commands and API snapshots:

Rule	Purpose / ports	Observed change
1595	Initial control/worker firewall, 2222–2223	Correctly revoked during scale-up
1600	First etcd firewall, 50000	Incorrectly revoked during scale-down
1602 / 1604	Other etcd firewalls, 50001 / 50002	Remain active
1606	Replacement control/worker firewall, 2222–2224	Created during scale-up; incorrectly left active during scale-down
1601	First etcd port forward, 50000→22	Removed during scale-down
1607 / 1608	Control / retained-worker port forwards, 2222→22 / 2223→22	Removed during scale-down
1610	Attempted replacement firewall, 2222–2223	Rejected due to overlap with 1606

After failure, listPortForwardingRules returns only 50001→22 and 50002→22. listFirewallRules returns 2222–2224, 6443, 50001 and 50002. CloudStack reports Alert and size=2, while only one worker VM remains. The five remaining VMs are Running. kubectl shows the control and one worker Ready on v1.33.13, and /readyz still returns ok.

What to do about it?

In 4.22.1.1 removeSshFirewallRule, the selector accepts either a firewall starting at 2222 or any corresponding port forward whose destination starts at 22, then breaks after the first match:

if (Objects.equals(firewallRule.getSourcePortStart(), CLUSTER_NODES_DEFAULT_START_SSH_PORT)
    || (Objects.nonNull(pfRule) && pfRule.getDestinationPortStart() == DEFAULT_SSH_PORT)) {
    rule = firewallRule;
    firewallService.revokeIngressFwRule(firewallRule.getId(), true);
    break;
}

The first etcd SSH rule also satisfies that condition. The preceding scale-up recreated the control/worker rule after the etcd rules; the next scale selected etcd rule 1600 first.

scaleKubernetesClusterIsolatedNetworkRules then uses the selected rule's end port as the upper bound for port-forward cleanup, yielding 2222–50000. This explains the removal of both regular-node SSH forwards and the first etcd forward. Recreating 2222–2223 then conflicts with the untouched general rule.

Please distinguish control/worker SSH rules from separate-etcd rules during selection and bound the forwarding cleanup accordingly. A regression test covering fresh three-etcd creation → worker scale 1→2→1, including rule ordering after recreation, should catch this. The partial VM removal and stale cluster count on a network-rule failure also merit recovery handling.

[boring-cyborg]

Thanks for opening your first issue here! Be sure to follow the issue template!