Add opt-in crawl_timeout and shield page cleanup from cancellation

[SohamKukreti]

Fixes #2205

A page whose JS thread goes busy after navigation hung arun() forever. page_timeout only covers goto; page.evaluate and page.content() have no timeout and are not covered by set_default_timeout, so every later step waited indefinitely and the page was never released. A task cancel landing inside the finally cleanup also skipped page.close(), leaking the page.

This PR:

• Adds CrawlerRunConfig.crawl_timeout (ms, default None = no limit). It wraps the whole page visit, from navigation to final HTML including js_code and hooks, in asyncio.wait_for. On expiry the page is force-closed (a session is dropped, with a warning) and the crawl fails with Crawl exceeded crawl_timeout of N ms.
• Runs the finally cleanup as a shielded task and re-awaits it on cancel, so page.close() always completes before the cancel propagates. The two console cleanup calls are bounded to 5 s so a hung page cannot block the cancel.
• Docker: base_config sets crawl_timeout to 180 s. Untrusted requests are capped at 60 s, same as the other timeouts.

Known gaps, by design:

• The Docker default reaches only the /crawl batch path; the other endpoints build CrawlerRunConfig() directly (follow-up).
• Docker hooks that take more than 180 s now fail where they previously ran under the 300 s server deadline.

List of files changed and why

• crawl4ai/async_configs.py - new crawl_timeout parameter (docstring, constructor, to_dict), added to the untrusted allowlist and the timeout cap.
• crawl4ai/async_crawler_strategy.py - _crawl_web wraps the new _crawl_page in asyncio.wait_for when crawl_timeout is set; finally cleanup moved into a shielded _cleanup() task with bounded console calls; new _close_unresponsive_page to force-close the page or drop the session on timeout.
• deploy/docker/config.yml - crawl_timeout: 180000 under crawler.base_config.
• docs/md_v2/api/parameters.md - parameter table row.
• docs/md_v2/core/browser-crawler-config.md - field list entry.
• docs/md_v2/core/page-interaction.md - "Timing Control" entry.
• tests/test_crawl_timeout.py - new tests (see below).

How Has This Been Tested?

tests/test_crawl_timeout.py (7 tests, all pass) uses a local HTTP server that serves a "trap" page whose JS thread goes busy after load:

• trap page fails within crawl_timeout with the expected error, page closed, context refcount 0
• trap page with remove_overlay_elements=True does not hang
• trap page in a session drops the session and the same session_id works again
• crawl_timeout=None keeps the old behaviour (crawl still running after 8 s), cancel leaves refcount 0
• keep-last-page rule for headless/managed browsers unchanged
• cancel landing inside the cleanup still closes the page (refcount 0)
• an evaluate inside cleanup on a hung page does not block the timeout, refcount 0

Regression: tests/browser (excluding the docker dir) and the config tests give an identical pass/fail list on this branch and on clean develop; the failures there are pre-existing (missing asyncio markers, import errors).

Live crawls (headless, Python 3.13): 8 real sites with and without crawl_timeout=30000 succeed with the same HTML; js_code, wait_for, screenshot, console and network capture, scan_full_page, overlay removal and delay_before_return_html all work under the timer; trap page fails in 5.5 s and the next crawl works; session trap drops the session and the session id is reusable; arun_many with a trap mixed in finishes in about the timeout with only the trap failing; 5 mid-crawl cancels leave refcount 0. No orphan Chrome processes after the runs.

Also verified: headed managed Chrome survives a forced close of its last tab (a blank tab is opened first), and dump/load/clone/from_kwargs preserve crawl_timeout.

Checklist:

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• I have made corresponding changes to the documentation
• I have added/updated unit tests that prove my fix is effective or that my feature works
• New and existing unit tests pass locally with my changes

[ntohidi]

Ran it locally. New tests pass, config + unit suites pass. The shielded finally looks right, and the blank-tab trick in _close_unresponsive_page is a good catch.

Two things though.

The session path still hangs. _crawl_web does await page.evaluate("window.stop()") before the wait_for, so if a session page went busy after its last crawl, the next arun() on that session waits forever. Same bug as #2205.

I hit it with a page that spins up a while(true) 4s after load: first crawl fine, second one never returns (I gave it 25s, crawl_timeout was 5s). Bounding that evaluate to 2s makes it fail in 7.5s with the right error. The existing except Exception: pass already swallows the TimeoutError, so it's a one-liner. Would add the case to test_crawl_timeout.py.

Second, the Docker default only lands on /crawl. /md, /screenshot, /pdf, /execute_js and the two spots in api.py (269, 393) build CrawlerRunConfig() themselves and get nothing. That's the path that pins a renderer, so I'd rather not leave it to a follow-up.

Minor stuff:

• The RuntimeError goes through the proxy/retry loop in async_webcrawler.py:527, so with retries or a proxy list you pay the timeout once per attempt. Fine at the Docker defaults, but worth a line in the docs.
• _cleanup() bounds the console calls but not release_page_with_context() / page.close(). Cancel now waits on those, so a wedged close makes arun() uncancellable where before it returned right away.
• create_task inside wait_for is redundant.

One thing I checked because it worried me: an untrusted crawl_timeout: null gets capped at 60s rather than passing through as "no limit". Good.

[SohamKukreti]

@ntohidi thanks for the review. Pushed a7913ca on top with the fixes.

Session path hang. Reproduced with a page that goes busy 4 s after load: first crawl fine, second arun() on the same session hung on the window.stop() evaluate. It is now asyncio.wait_for(page.evaluate("window.stop()"), 2); the existing except Exception: pass swallows the timeout. Added test_session_page_hung_between_crawls_fails_within_crawl_timeout for the case (second crawl fails in ~7.5 s with the crawl_timeout error and the session is dropped).

Docker default on every endpoint. Added apply_base_config(cfg, config, keys=None) in api.py (the inline loop from handle_crawl_request, moved into a function) and call it at every place that builds its own CrawlerRunConfig: /md, /llm (both the QA route and /llm/job), /html, /screenshot, /pdf, /execute_js, and /crawl/stream. /crawl batch and stream apply the full base_config as before; the specialized endpoints apply only crawl_timeout (keys=("crawl_timeout",)) so their behaviour does not change beyond the timeout. Same fill-only-if-unset rule, so a value the request sends always wins.

Retries. Added a sentence to parameters.md and page-interaction.md: the timeout applies per attempt when max_retries or a proxy list is set.

Minor points.

• create_task inside wait_for: not redundant in practice. Playwright cancels its pending protocol call only when the task that made it ends as cancelled; with a bare coroutine, wait_for converts the cancel to TimeoutError, the pending call is left dangling and Python logs "Future exception was never retrieved" on every timeout. Wrapping in a task makes the inner task end cancelled and the noise goes away. Happy to add a short comment if you want it visible.
• page.close() / release_page_with_context() unbounded in cleanup: agreed on the reasoning. Left as is since the close is handled by the browser process and fails fast on a dead connection.

Verification.

• tests/test_crawl_timeout.py: 8 pass. tests/browser and the config suites give an identical pass/fail list to clean develop.
• Library live runs: 8 real sites with and without crawl_timeout, js_code/wait_for/screenshot/capture/scroll/overlay, trap pages, sessions (including hung-between-crawls), arun_many with a trap, 5 mid-crawl cancels; refcount 0 and no orphan pages after each.
• Docker image built from this branch, config.yml mounted with crawl_timeout: 8000, trap page served from the host: every endpoint above fails at ~8.6 s with Crawl exceeded crawl_timeout of 8000 ms (for /html, /screenshot, /pdf, /execute_js the message is in the server log since those return a generic 500); a request that sends crawl_timeout: 3000 or 15000 keeps its value; per-URL crawler_configs keep theirs; 999999 and null are capped to 60000 by the untrusted clamp; real sites on /md, /crawl, /screenshot, /execute_js all fine; Chrome process count in the container did not grow across repeated trap runs.