Skip to content

fix(chart): drain s3proxy pods on scale-in instead of killing in-flight uploads#136

Merged
ServerSideHannes merged 1 commit into
mainfrom
graceful-drain-on-scale-in
Jul 20, 2026
Merged

fix(chart): drain s3proxy pods on scale-in instead of killing in-flight uploads#136
ServerSideHannes merged 1 commit into
mainfrom
graceful-drain-on-scale-in

Conversation

@ServerSideHannes

Copy link
Copy Markdown
Owner

Problem

The KEDA connection-demand trigger (argocd-system #807, live since 2026-07-19 ~14:30 UTC) fixed scale-in decisions — the fleet no longer reads idle while busy. But the 2026-07-19 backup run still died at 23:49 UTC with InvalidPart, because scale-in execution is still a hard kill:

  • Top-of-hour cron bursts (clickhouse/postgres backups) spike sessions to ~97 → KEDA correctly scales to 30.
  • Burst ends → HPA scales 30→6 within ~15 min while Scylla still holds 20–40 long-lived upload sessions.
  • Victims get SIGTERM + 30s grace, no preStop. A large UploadPart/UploadPartCopy takes longer than 30s → killed mid-request → orphaned multipart state → InvalidPart on CompleteMultipartUpload.

Fix

Let a terminating pod finish what it's serving:

  • preStop sleep 10 — pod keeps accepting while it drops out of the headless-service DNS, so HAProxy (server-template + resolvers) stops routing new connections to it before uvicorn stops listening. Existing connections are untouched by HAProxy when a resolved server goes away.
  • terminationGracePeriodSeconds: 600 (was 30, now a values knob) — uvicorn on SIGTERM stops accepting and waits for in-flight requests, then exits immediately. The grace period only needs to cover the longest single request; 600s bounds a slow multi-minute part copy. Pods exit as soon as they're idle — the ceiling costs nothing in the common case.

Chart-only change, no image change needed.

@ServerSideHannes
ServerSideHannes merged commit cd91e22 into main Jul 20, 2026
11 checks passed
@ServerSideHannes
ServerSideHannes deleted the graceful-drain-on-scale-in branch July 20, 2026 06:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant