Skip to content

fix: cap uvicorn graceful shutdown so in-app restart cannot wedge#1772

Open
King0James0 wants to merge 1 commit into
agent0ai:mainfrom
King0James0:fix/restart-graceful-shutdown-timeout
Open

fix: cap uvicorn graceful shutdown so in-app restart cannot wedge#1772
King0James0 wants to merge 1 commit into
agent0ai:mainfrom
King0James0:fix/restart-graceful-shutdown-timeout

Conversation

@King0James0

Copy link
Copy Markdown

fix: cap uvicorn graceful shutdown so in-app restart cannot wedge

Problem

Clicking Restart (or anything that calls process.reload(), including self-update) can permanently wedge the server whenever a long-running request is in flight — for example an agent message task that runs for minutes, or a slow file download. The symptom: the web UI port goes dead, the container stays "Up", the old run_ui.py process stays alive indefinitely, and the supervisor never respawns it. The only recovery is a manual kill -9 of the run_ui PID inside the container.

Mechanism (verified on the current main and the v2.4 image, uvicorn 0.49)

  1. /api/restartprocess.reload() sets server.should_exit = True (graceful shutdown) and then calls sys.exit(0).
  2. The SystemExit is raised inside the ASGI request, and uvicorn's run_asgi catches BaseException ("Exception in ASGI application", the client sees a 500) — so the process does not exit via sys.exit. This has always been the case; the process is expected to exit through uvicorn's shutdown instead.
  3. uvicorn's graceful shutdown closes the listeners (port dead) and then waits for in-flight requests in _wait_tasks_to_complete with timeout=self.config.timeout_graceful_shutdown — which is never set, i.e. None = wait forever.

Short requests drain fine, and an idle websocket also drains (uvicorn closes it during shutdown). The killer is any long-running request task: with the synchronous message API a single in-flight agent task holds shutdown for its entire duration — effectively forever from the user's point of view.

Reproduction (stock image, no plugins)

# 1. run a stock container
docker run -d --name a0test -p 127.0.0.1:50083:80 -v a0test-vol:/a0/usr agent0ai/agent-zero:v2.4

# 2. create a large static file and hold a slow download open (stand-in for a long agent task)
docker exec a0test dd if=/dev/zero of=/a0/webui/big.bin bs=1M count=64
curl -s -o /dev/null --limit-rate 1k http://127.0.0.1:50083/big.bin &

# 3. restart via the API (same as the UI button)
#    (fetch /api/csrf_token with an Origin header, then POST /api/restart)

Result on stock: the old run_ui PID stays alive with the port closed — observed 150+ seconds with no recovery (unbounded). With this patch, same scenario: the old PID exits at the 10s cap and the UI is answering again in ~25 seconds total.

Fix

Pass timeout_graceful_shutdown to the uvicorn.Config (default 10s, overridable via A0_SHUTDOWN_TIMEOUT_SECONDS, following the existing A0_STARTUP_* env pattern). uvicorn then cancels lingering tasks after the cap, serve() returns, run_ui exits normally, and the standard supervisor respawn restarts the server.

One line of behavior, no API change. Operators who want a longer drain window can raise the env var.

Notes

  • The swallowed sys.exit(0) in process.reload() is left as-is — it is harmless, and with the shutdown cap the process exits through the normal path. Happy to remove it or add a comment if preferred.
  • On heavily loaded instances a cancelled task that swallows CancelledError can still delay exit past the cap; the cap bounds the common case and restores the restart button. A hard-exit backstop was considered but left out to keep this minimal — can add if maintainers want it.

The restart flow (/api/restart -> process.reload) signals uvicorn to shut
down gracefully and then calls sys.exit(0). The SystemExit is raised inside
the ASGI request and swallowed by uvicorn's BaseException handler in
run_asgi, so the process does not exit that way. Meanwhile uvicorn's
graceful shutdown has already closed the listeners and waits for in-flight
requests with timeout_graceful_shutdown=None, i.e. forever. Any long-running
request at that moment (an agent message task, a slow download) wedges the
process permanently: the UI port is dead, run_ui never exits, and the
supervisor never respawns it until it is killed manually.

Cap the graceful shutdown at 10 seconds (env-overridable via
A0_SHUTDOWN_TIMEOUT_SECONDS). uvicorn then cancels lingering tasks, serve()
returns, run_ui exits normally, and the standard respawn path restarts the
server.

Repro on the stock v2.4 image: hold a slow request open (e.g. rate-limited
download of a large static file), POST /api/restart -> the old process stays
alive indefinitely with the port closed. With this change the same scenario
recovers in seconds.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant