feat: Orchestrator - Terminate PENDING-wedged executions with a deadline watchdog

[morgan-wowk]

What this changes

Teaches the orchestrator to give up on a task that can never start. Today a pod stuck in Pending (e.g. a gcsfuse mount wedge: MountVolume.SetUp failed ... code = Unauthenticated → CreateContainerConfigError) is polled forever with no deadline and no terminal state — one such run sat stuck for 6 days. This adds a pending deadline that terminates the pod and fails the execution with the real kubelet reason.

Tracking: Shopify/oasis-backend#413

Before / after

Before

• Pod wedged in Pending → orchestrator logs remains in PENDING state and returns, every poll, indefinitely.
• The execution never reaches a terminal state; started_at stays None; downstream tasks wait forever.
• No signal to the user that anything is wrong — the run simply hangs.

After (once a deployment sets max_pending_duration)

• Past the deadline, the orchestrator terminates the pod and marks the execution SYSTEM_ERROR, skipping downstream — the run fails fast instead of hanging.
• The error message carries the actual cause, e.g.:

  Task pending 0:27:13, never started (deadline 0:20:00).
  CreateContainerConfigError: MountVolume.SetUp failed for volume "gcsfuse-prd-oasis-tmp":
    code = Unauthenticated desc = failed to prepare storage service
  

The changes

PENDING-deadline watchdog (orchestrator_sql.py)

• New optional orchestrator param max_pending_duration: timedelta | None. Defaults to None (disabled) — this PR is a no-op in production until oasis-backend passes a value (separate submodule bump). Merging it carries no behavior change or risk.
• New pure helper _pending_deadline_exceeded(created_at, now, max_pending_duration): returns False when disabled or when created_at is unknown, so legacy rows are never force-failed. Boundary is strict (>), so it fires only past the deadline.
• In internal_process_one_running_execution, before the PENDING == PENDING early return: when the deadline is exceeded, upload logs, call terminate(), and raise OrchestratorError(<message>).
• Reuses existing teardown. The raise is caught by the existing outer handler, which already sets SYSTEM_ERROR, records the orchestration error on each node, and calls _mark_all_downstream_executions_as_skipped. No new state-machine code, no new failure path to maintain.
• created_at is the correct anchor because started_at is None for a never-started container.

Real kubelet reason in the error message (kubernetes_launchers.py)

• New pending_diagnostics property on LaunchedKubernetesContainer returns the main container's waiting reason + message, read from the already-cached _debug_pod — no extra Kubernetes API calls. This is the signal that carries the wedge cause (CreateContainerConfigError + the mount failure).
• Fixes the root gap: the existing launcher_error_message reads only the terminated state, which never exists on a boot wedge, so the reason was being dropped.
• No interface change. The property lives only on the K8s launcher; the orchestrator reads it via getattr(..., "pending_diagnostics", None), so the LaunchedContainer interface and its other implementations are untouched. When unavailable, the message falls back to the generic timeout line.

Threshold guidance

max_pending_duration must exceed real Kueue admission / scheduling latency or it will false-positive on healthy queued work. A value around 20–30 min is a safe starting point. It is configurable precisely so it can be tuned in oasis-backend without a code change.

Tests

tests/test_pending_deadline_watchdog.py:

• _pending_deadline_exceeded: disabled (None), unknown created_at, under threshold, past threshold, strict boundary.
• pending_diagnostics: gcsfuse mount-wedge message surfaced; no main-container status → None.

All green; black clean.

Out of scope (follow-ups)

• oasis-backend wiring to set max_pending_duration and actually enable the watchdog (submodule bump).
• Stop cache-reuse of stale PENDING nodes, so a resubmit doesn't re-link to the dead node.

[morgan-wowk]

Needs review. Must be careful with interface changes.

[morgan-wowk]

Reverted — no interface change. pending_diagnostics now lives only on LaunchedKubernetesContainer, and the orchestrator reads it via getattr(..., "pending_diagnostics", None), so the base LaunchedContainer interface and its other implementations are untouched.

[morgan-wowk]

This is a lot of lines diff. Consider how much value / impact is really being achieved here and whether there's a much shorter and simpler solution.

[morgan-wowk]

Trimmed from ~64 lines to 8. It now returns just the main container's waiting reason + message, which is the single signal that carries the actual wedge (CreateContainerConfigError + the MountVolume.SetUp failed ... Unauthenticated text from the incident). Dropped the init-container and pod-condition handling — the watchdog still fires and terminates in those cases; the message just falls back to the generic timeout line.