Skip to content

fix(producer): type video extraction failures#2776

Open
jrusso1020 wants to merge 1 commit into
fix/atomic-video-download-retryfrom
fix/video-extraction-failure-retry
Open

fix(producer): type video extraction failures#2776
jrusso1020 wants to merge 1 commit into
fix/atomic-video-download-retryfrom
fix/video-extraction-failure-retry

Conversation

@jrusso1020

Copy link
Copy Markdown
Collaborator

Summary

  • classify per-source video download/probe/decode/extraction failures with a bounded taxonomy and safe producer-facing summaries
  • add candidate-only, at-most-one transient retry with cleanup and retry telemetry
  • preserve default engine/producer behavior when the policy is off
  • carry allowlisted extraction error codes through blocking JSON and SSE responses

Stack

Depends on #2774 for atomic remote downloads and its single owned download retry. This PR is intentionally based on fix/atomic-video-download-retry; rebase/change the base to main after #2774 merges.

Default compatibility

HF_VIDEO_EXTRACTION_FAILURE_MODE defaults to off and forces maxTransientRetries=0.

With the feature off:

  • metadata probe failures keep the legacy Promise rejection
  • grouped extraction keeps the existing grouped-to-direct fallback
  • no new producer failure gate is enforced
  • render-plan schema, Plan v1 artifacts, chunk routing, and distributed execution are unchanged

Typed metadata aggregation is explicit and enabled only by the candidate enforce lane.

Retry ownership

  • remote downloads: exactly one retry owned by fix(engine): make remote video downloads atomic #2774
  • metadata/FFmpeg extraction: at most one retry only when HF_VIDEO_EXTRACTION_MAX_RETRIES=1
  • invalid, missing, rejected, out-of-range, zero-output, cancellation, and unknown/internal failures do not retry
  • non-finite or invalid runtime retry budgets fail closed to zero
  • the superset optimization is never retried; on failure it preserves direct-member fallback, and only the individual ranges can use the bounded retry
  • retry counters increment when a retry is scheduled, including exhausted retries

The internal sidecar and Experiment Framework must treat both exhausted stage codes as workflow-terminal after the producer-local budget. Candidate enforcement must not be enabled until those companion mappings are deployed, or Temporal can multiply producer attempts.

Failure contract

  • VIDEO_SOURCE_UNRENDERABLE: at least one deterministic/unknown source failure
  • VIDEO_EXTRACTION_FAILED: all source failures are transient but the producer-local budget is exhausted

Only the allowlisted code and kind/count summaries cross JSON/SSE. Raw diagnostics remain engine-local because they may contain signed URLs or local paths.

Rollout

  1. merge and deploy with stable/candidate both off
  2. candidate observe, retries 0
  3. candidate observe, retries 1
  4. deploy internal + EF terminal transport mappings
  5. candidate enforce, retries 1
  6. keep stable off until success delta, retry counts, extraction latency, CPU/disk, and queue backlog are acceptable

Validation

  • engine focused suites: 105 passed
  • producer focused suites: 15 passed
  • full engine suite: 1,176 passed, 3 skipped
  • full producer unit lane: 32 Vitest files / 393 tests plus all classified Bun unit tests
  • engine and producer typechecks passed
  • oxlint, oxfmt, Fallow, tracked-artifact, and commit hooks passed
  • independent review: approved for merge default-off; candidate enforcement held on companion transport rollout

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant