fix: cap uvicorn graceful shutdown so in-app restart cannot wedge#1772
Open
King0James0 wants to merge 1 commit into
Open
fix: cap uvicorn graceful shutdown so in-app restart cannot wedge#1772King0James0 wants to merge 1 commit into
King0James0 wants to merge 1 commit into
Conversation
The restart flow (/api/restart -> process.reload) signals uvicorn to shut down gracefully and then calls sys.exit(0). The SystemExit is raised inside the ASGI request and swallowed by uvicorn's BaseException handler in run_asgi, so the process does not exit that way. Meanwhile uvicorn's graceful shutdown has already closed the listeners and waits for in-flight requests with timeout_graceful_shutdown=None, i.e. forever. Any long-running request at that moment (an agent message task, a slow download) wedges the process permanently: the UI port is dead, run_ui never exits, and the supervisor never respawns it until it is killed manually. Cap the graceful shutdown at 10 seconds (env-overridable via A0_SHUTDOWN_TIMEOUT_SECONDS). uvicorn then cancels lingering tasks, serve() returns, run_ui exits normally, and the standard respawn path restarts the server. Repro on the stock v2.4 image: hold a slow request open (e.g. rate-limited download of a large static file), POST /api/restart -> the old process stays alive indefinitely with the port closed. With this change the same scenario recovers in seconds.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
fix: cap uvicorn graceful shutdown so in-app restart cannot wedge
Problem
Clicking Restart (or anything that calls
process.reload(), including self-update) can permanently wedge the server whenever a long-running request is in flight — for example an agent message task that runs for minutes, or a slow file download. The symptom: the web UI port goes dead, the container stays "Up", the oldrun_ui.pyprocess stays alive indefinitely, and the supervisor never respawns it. The only recovery is a manualkill -9of the run_ui PID inside the container.Mechanism (verified on the current
mainand the v2.4 image, uvicorn 0.49)/api/restart→process.reload()setsserver.should_exit = True(graceful shutdown) and then callssys.exit(0).SystemExitis raised inside the ASGI request, and uvicorn'srun_asgicatchesBaseException("Exception in ASGI application", the client sees a 500) — so the process does not exit viasys.exit. This has always been the case; the process is expected to exit through uvicorn's shutdown instead._wait_tasks_to_completewithtimeout=self.config.timeout_graceful_shutdown— which is never set, i.e.None= wait forever.Short requests drain fine, and an idle websocket also drains (uvicorn closes it during shutdown). The killer is any long-running request task: with the synchronous message API a single in-flight agent task holds shutdown for its entire duration — effectively forever from the user's point of view.
Reproduction (stock image, no plugins)
Result on stock: the old run_ui PID stays alive with the port closed — observed 150+ seconds with no recovery (unbounded). With this patch, same scenario: the old PID exits at the 10s cap and the UI is answering again in ~25 seconds total.
Fix
Pass
timeout_graceful_shutdownto theuvicorn.Config(default 10s, overridable viaA0_SHUTDOWN_TIMEOUT_SECONDS, following the existingA0_STARTUP_*env pattern). uvicorn then cancels lingering tasks after the cap,serve()returns,run_uiexits normally, and the standard supervisor respawn restarts the server.One line of behavior, no API change. Operators who want a longer drain window can raise the env var.
Notes
sys.exit(0)inprocess.reload()is left as-is — it is harmless, and with the shutdown cap the process exits through the normal path. Happy to remove it or add a comment if preferred.CancelledErrorcan still delay exit past the cap; the cap bounds the common case and restores the restart button. A hard-exit backstop was considered but left out to keep this minimal — can add if maintainers want it.