RDS MySQL instance intermittently stuck in error state on create (container readiness race)

[Ranotronarison]

RDS MySQL instance intermittently stuck in error state on create (container readiness race)

Issue Description

Creating an aws_db_instance with engine = "mysql" via Terraform intermittently fails: the underlying LocalStack-managed MySQL sibling container is started successfully and becomes healthy, but LocalStack's own readiness probe gives up too early and marks the DB instance error before the container is actually accepting connections. The instance never recovers from this state afterwards (DescribeDBInstances keeps reporting error indefinitely), even though the MySQL container itself is confirmed running and reachable minutes later. All other resources in the same terraform apply (VPC, subnets, ALB, ACM, Route53, S3) create successfully every time — only the RDS MySQL engine is affected.

Environment Information

• LocalStack image: localstack/localstack:2026.08.1 (Pro, via Docker Compose)
• lstk CLI: 0.23.0
• Terraform: v1.16.1, hashicorp/aws provider 5.100.0
• Docker: 25.0.3
• Host: WSL2, Linux 6.6.87.2-microsoft-standard-WSL2 (x86_64)
• docker-compose.yml: PERSISTENCE=1, volume-mounted /var/run/docker.sock (sibling-container mode)
• Failing resource config (aws_db_instance): engine = "mysql", engine_version unpinned tag resolving to mysql:8.4, db_subnet_group_name pointing at 2 private subnets, manage_master_user_password = true, skip_final_snapshot = true

Reproduction Steps

1. docker compose up -d (LocalStack starts healthy)
2. lstk terraform init -backend-config=backend.hcl
3. lstk terraform apply tfplan — plan includes module.rds[0].aws_db_instance.this (MySQL engine) alongside VPC/ALB/ACM/Route53/S3 resources
4. ✅ All non-RDS resources create successfully, several within the same run (VPC ~11s, ALB ~60s, ACM/Route53 ~15s)
5. ❌ module.rds[0].aws_db_instance.this polls "Still creating..." for 100-110s, then fails:

   Error: waiting for RDS DB Instance (terraform-...) create: unexpected state 'error', wanted target 'available, storage-optimization'. last error: %!s(<nil>)
   

6. lstk aws rds describe-db-instances confirms DBInstanceStatus: error, and stays error on repeated checks minutes later — no self-recovery.
7. docker ps shows the sibling MySQL container (ls-mysql-<hash>, image mysql:8.4) in state Up and has been running fine the whole time, both on this attempt and a prior one on the same day with a different generated instance identifier — i.e. this has reproduced twice in the same session with two different instance IDs.

Troubleshooting Attempts

• Verified mysql:8.4 was already pulled/cached locally (5+ weeks old) before this run, ruling out a slow image-pull as the cause.
• Confirmed via docker ps that the sibling container the failed instance was backed by is still Up and running well past the point LocalStack reported error — the container itself never crashed or exited.
• Retried the whole apply a second time (fresh random_string suffix, brand-new aws_db_instance/new sibling container) — reproduced with a different, more specific error the second time (see logs below): mysqladmin connect refused on localhost:3306.
• Confirmed this is not a Terraform config or backend/provider issue: terraform.tf, backend.hcl, and the provider lock file are correctly resolved, and every non-RDS resource in the same plan creates without issue.
• Confirmed the DB instance, once flagged error, does not transition on its own even after the container is demonstrably ready — there appears to be no retry/re-check after the initial readiness window is exhausted.

Root Cause Analysis

LocalStack's MySQL RDS engine emulation launches a sibling Docker container (per the RDS docs: "A MySQL server will be launched in a new Docker container upon requesting the MySQL engine") and then polls it with mysqladmin ping/exec calls to decide when the instance becomes available. This appears to be a fixed-budget polling loop (roughly ~100-110s observed across two runs) rather than one tied to actual mysqld readiness. When the container takes longer than that budget to finish MySQL's first-run initialization (data directory setup + entrypoint script) and start accepting TCP connections on 3306, the probe's mysqladmin/exec call fails once, and LocalStack immediately marks the instance error and never retries — even though the container goes on to become fully healthy seconds/minutes later. Because DescribeDBInstances.DBInstanceStatus isn't an attribute Terraform's aws_db_instance resource tracks for drift, a plain re-apply won't detect or fix the stuck instance either; it has to be deleted and recreated out of band.

The two failures observed differ slightly in their surfaced error, which points at two edges of the same race rather than two separate bugs:

• 1st run: ContainerStateError: Container not yet started. — the readiness check ran before the container object itself was even in a "started" state.
• 2nd run: mysqladmin: connect to server at 'localhost' failed ... Can't connect to MySQL server on 'localhost:3306' (111) — the container was started, but mysqld wasn't accepting connections yet when the last poll attempt ran.

Detailed Logs

Terraform apply output (2nd reproduction):

module.rds[0].aws_db_instance.this: Creating...
module.rds[0].aws_vpc_security_group_ingress_rule.mysql_cidr["10.0.0.0/16"]: Creating...
module.rds[0].aws_vpc_security_group_ingress_rule.mysql_cidr["10.0.0.0/16"]: Creation complete after 0s [id=sgr-ba065390d13c65572]
...
module.rds[0].aws_db_instance.this: Still creating... [00m10s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [00m20s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [00m30s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [00m40s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [00m50s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m00s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m10s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m20s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m30s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m40s elapsed]
module.rds[0].aws_db_instance.this: Still creating... [01m50s elapsed]
╷
│ Error: waiting for RDS DB Instance (terraform-20260907122355832600000004) create: unexpected state 'error', wanted target 'available, storage-optimization'. last error: %!s(<nil>)
│
│   with module.rds[0].aws_db_instance.this,
│   on ../../modules/rds/main.tf line 1, in resource "aws_db_instance" "this":
│    1: resource "aws_db_instance" "this" {
│
╵

LocalStack container logs, 1st reproduction:

2026-09-07T10:33:57.336  INFO --- [et.reactor-5] localstack.request.aws     : AWS rds.CreateDBSubnetGroup => 200
2026-09-07T10:33:57.339  INFO --- [et.reactor-4] localstack.request.aws     : AWS rds.DescribeDBSubnetGroups => 200
2026-09-07T10:33:57.343  INFO --- [et.reactor-2] localstack.request.aws     : AWS rds.ListTagsForResource => 200
2026-09-07T10:33:57.568  INFO --- [-functhread5] l.p.c.s.rds.engine_mysql   : Starting MySQL server on port 4510 - database "devops_practice_db", user "db_user"
2026-09-07T10:33:57.619  INFO --- [-functhread5] l.u.container_networking   : Determined main container network: devops-practice_default
2026-09-07T10:34:07.572  INFO --- [et.reactor-0] localstack.request.aws     : AWS rds.CreateDBInstance => 200
2026-09-07T10:34:07.959  INFO --- [read-4 (run)]  l.p.c.persistence.manager  : Saving state for rds took 234 ms
2026-09-07T10:35:07.579  INFO --- [et.reactor-1] localstack.request.aws     : AWS rds.DescribeDBInstances => 200
2026-09-07T10:35:17.583  INFO --- [et.reactor-5] localstack.request.aws     : AWS rds.DescribeDBInstances => 200
2026-09-07T10:35:27.590  INFO --- [et.reactor-5] localstack.request.aws     : AWS rds.DescribeDBInstances => 200
2026-09-07T10:35:32.635  WARN --- [-functhread5] l.p.c.s.rds.db_utils       : Unable to startup DB instance: Container not yet started. Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 64, in A
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 83, in start_db_instance
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/pro/core/services/rds/engine_mysql.py.enc", line 82, in V
    def V():A='mysqladmin ping --user root -p$MYSQL_ROOT_PASSWORD --protocol tcp';A=[S,'-c',A];assert E.container.exec(command=A)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/utils/container/container.py", line 22, in B
    if not self.is_started():raise ContainerStateError('Container not yet started.')
localstack.utils.container.container.ContainerStateError: Container not yet started.
2026-09-07T10:35:37.595  INFO --- [et.reactor-5] localstack.request.aws     : AWS rds.DescribeDBInstances => 200

LocalStack container logs, 2nd reproduction (~2h later, same day, different instance id):

2026-09-07T12:25:37.600  WARN --- [-functhread8] l.p.c.s.rds.db_utils       : Unable to startup DB instance: ('Exec command returned with exit code 1', None, b"mysqladmin: [Warning] Using a password on the command line interface can be insecure.\n\x07mysqladmin: connect to server at 'localhost' failed\nerror: 'Can't connect to MySQL server on 'localhost:3306' (111)'\nCheck that mysqld is running on localhost and that the port is 3306.\nYou can check this by doing 'telnet localhost 3306'\n") Traceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/pro/core/services/rds/db_utils.py.enc", line 64, in A

docker ps — both sibling containers running fine after LocalStack had already given up on them:

ls-mysql-84acfc36   mysql:8.4   Up About an hour   2026-09-07 13:36:03 +0300 EAT
ls-mysql-674e65b6   mysql:8.4   Up 10 minutes      2026-09-07 15:23:56 +0300 EAT

lstk aws rds describe-db-instances — status never self-heals:

+-----------+----------------------------------------+-------+----------+
| endpoint  |                  id                    | port  | status   |
+-----------+----------------------------------------+-------+----------+
|  localhost|  terraform-20260907103357353900000004  |  4510 |  error   |
+-----------+----------------------------------------+-------+----------+