Skip to content

SKE cluster deletion hangs for up to 90 minutes when the cluster left a load balancer behind #1646

Description

@FabianHardt

Description

Deleting an SKE cluster that ran a Service of type LoadBalancer leaves the
cluster in STATE_DELETING for a long time — in our case over an hour, and the
provider would have waited up to its 90 minute timeout.

The cause is the load balancer the cloud controller created inside the
cluster. It lives in the project, but it belongs to no Terraform state:
terraform destroy removes the cluster, nobody removes the load balancer, and
the shoot waits for it.

Nothing in the API says so. STATE_DELETING is the only signal, and the
provider's eventual error is Cluster deletion waiting: ..., which does not
name a cause.

Steps to reproduce

resource "stackit_ske_cluster" "example" {
  project_id         = var.project_id
  name               = "example"
  kubernetes_version_min = "1.35"

  node_pools = [{
    name         = "pool-01"
    machine_type = "c1a.2d"
    minimum       = 1
    maximum       = 2
  }]
}
  1. terraform apply
  2. In the cluster, install anything that creates a Service of type
    LoadBalancer — an ingress controller, for example. A load balancer appears
    under stackit load-balancer list -p <project>.
  3. terraform destroy

Actual behavior

terraform destroy triggers DeleteCluster and then waits. The cluster stays
in STATE_DELETING; stackit load-balancer list still shows the load balancer
in STATUS_READY. Nothing indicates that one is waiting for the other. After
90 minutes the provider gives up with

Error deleting cluster: Cluster deletion waiting: ...

Expected behavior

Either of these would resolve it:

  1. The shoot cleans up the load balancers it caused when it is deleted, the way
    it created them.
  2. Or the cluster status names what the deletion is waiting for, so that both
    a human and the provider can report something better than "still deleting".

The second one alone would already help a lot — the state is currently
indistinguishable from "slow".

Environment

  • OS: macOS 26.6 (arm64)
  • Terraform version: v1.15.8
  • Version of the STACKIT Terraform provider: v0.106.0

Additional information

I do not think the provider should delete the load balancers itself: it never
created them, and picking which ones belong to a given cluster would mean
deleting resources it does not manage. This is why I am filing it as a service
behaviour issue rather than a provider bug.

The known workaround, and what we do in our offboarding, is to delete all
LoadBalancer services in the cluster first, wait until
stackit load-balancer list is empty for the project, remove leftovers via the
CLI, and only then run terraform destroy.

It is the same class of problem as AWS EKS with ELBs, so the workaround will be
familiar to many — but it costs an hour of confusion the first time, because
nothing points at it. I am happy to open a separate PR adding a note to the
stackit_ske_cluster documentation and a clearer timeout message, if that is
useful while the underlying behaviour is looked at.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions