fix: bound sitemap decompression size and nesting depth to prevent DoS

[vdusek]

Problem

The sitemap parsing pipeline processes content from arbitrary crawled hosts, which are untrusted. Two vectors let a malicious host exhaust crawler memory or trap it in unbounded work:

• Gzip decompression bomb. parse_sitemap decompressed gzipped sitemaps with no output cap, so a ~50 KB payload could inflate to gigabytes.
• Unbounded nested-sitemap recursion. parse_sitemap defaulted max_depth to float('inf') with only exact-URL dedup, so a chain of unique nested-sitemap URLs recursed forever.

Changes

• Gzip decompression bomb. Decompression and parsing now run in bounded pieces capped at MAX_SITEMAP_SIZE (50 MB, the sitemap-protocol limit). The read loop also stops once the gzip stream ends (otherwise trailing bytes accumulate unbounded in the decompressor's unused_data) and once too many raw bytes are read (guarding a stream that produces little or no output). On a repro, peak memory dropped from ~680 MB to ~15 MB.
• Unbounded nested-sitemap recursion. The default is now DEFAULT_MAX_DEPTH = 10. SitemapRequestLoader — the main consumer — drives nesting itself (it passes max_depth=0), so it now tracks per-sitemap depth in its persisted state and stops following chains past the same limit.

Out of scope

A third listed vector, unbounded robots.txt reads, is intentionally not addressed here. Capping it requires switching RobotsTxtFile.load from send_request to streaming, which regresses CurlImpersonateHttpClient (a multi-second stall in the crawl pipeline that fails test_respect_robots_txt_with_problematic_links[curl]). Since send_request reads the whole body eagerly and there is no size-limited read primitive in the HTTP client layer, there is no safe non-streaming cap. It is the lowest-severity vector (no compression amplification — the attacker must transmit the full bytes), so it is left as a follow-up: add a size-limited read to the HttpClient streaming layer, then cap robots.txt there.

Tests

Adds regression tests for gzip-bomb truncation, stop-after-gzip-member, parse_sitemap depth, and loader nested-chain depth.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 93.38%. Comparing base (7780e78) to head (7edced5).
⚠️ Report is 2 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #2012      +/-   ##
==========================================
+ Coverage   93.35%   93.38%   +0.03%     
==========================================
  Files         179      179              
  Lines       12482    12516      +34     
==========================================
+ Hits        11652    11688      +36     
+ Misses        830      828       -2     

Flag	Coverage Δ
unit	93.38% <100.00%> (+0.03%)	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.