Skip to content

Stop the stale-job reaper from revoking jobs that are created but not yet started #1355

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mihow merged 2 commits into from
Jun 26, 2026
Merged

Stop the stale-job reaper from revoking jobs that are created but not yet started #1355

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
df5af6f
fix(jobs): stop the reaper from revoking created-but-unstarted jobs
mihow Jun 26, 2026
a8ac1a9
docs(jobs): tighten JobState comments after dropping CREATED from run…
mihow Jun 26, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
9 changes: 6 additions & 3 deletions ami/jobs/models.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -79,7 +79,9 @@ class JobState(str, OrderedEnum):

@classmethod
def running_states(cls):
return [cls.CREATED, cls.PENDING, cls.STARTED, cls.RETRY, cls.CANCELING, cls.UNKNOWN]
# Dispatched states the reapers/reconcilers select from. CREATED is not
# here: a not-yet-enqueued job has no Celery task to reap. See #1354.
return [cls.PENDING, cls.STARTED, cls.RETRY, cls.CANCELING, cls.UNKNOWN]

@classmethod
def final_states(cls):
Expand All @@ -96,8 +98,9 @@ def active_states(cls):

@classmethod
def finalizable_states(cls):
# running_states() minus CANCELING (don't resurrect a cancel in progress)
# and UNKNOWN (never served; shouldn't auto-finalize). See #1337.
# States a job may auto-finalize from. Excludes CANCELING (don't resurrect
# a cancel in progress) and UNKNOWN (never served; shouldn't auto-finalize).
# See #1337.
return [cls.CREATED, cls.PENDING, cls.STARTED, cls.RETRY]


Expand Down
18 changes: 18 additions & 0 deletions ami/jobs/tests/test_update_stale_jobs.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -121,3 +121,21 @@ def test_skips_recent_and_final_state_jobs(self, mock_cleanup):

self.assertEqual(results, [])
mock_cleanup.assert_not_called()

@patch("ami.jobs.tasks.cleanup_async_job_if_needed")
def test_skips_created_but_unstarted_jobs(self, mock_cleanup):
"""A job created but never enqueued is left alone regardless of age.

Jobs can be pre-configured and started later, so a CREATED job has no
Celery task and no async resources to reconcile. The staleness cutoff
(measured against ``updated_at``) does not apply to it — it waits in
CREATED until a user starts it. See #1354.
"""
job = self._create_job(status=JobState.CREATED, minutes_ago=300)

results = check_stale_jobs()

self.assertEqual(results, [])
job.refresh_from_db()
self.assertEqual(job.status, JobState.CREATED.value)
mock_cleanup.assert_not_called()
Loading