Crawler-invented malformed URLs become immortal Targets: no validation at discovery + 403 exempt from suppression

[harper-joseph]

Summary

Crawlers with broken link extractors invent malformed URLs from our rendered HTML. Discovery creates a
Target for each with no URL validation, the origin's WAF rejects them with 403, and 401/403 is
deliberately exempt from suppression — so they can never be retired. They accumulate permanently and
retry forever.

Measured on a production deployment over 24h: 1,264 distinct URLs, producing ~2,950 renders/day
that can never succeed. All of them: sitemapUrl null (never submitted), strikes at max, and no
PrerenderedPage for any device cacheKey — i.e. not one has ever rendered successfully.

Why it happens

Representative captured URLs (product ids redacted):

/product/prd-XXXXXXX/target=
/product/prd-XXXXXXX/url(%22https:/fonts.googleapis.com/css2
/product/prd-XXXXXXX/....jsp'%20defer='defer
/product/prd-XXXXXXX/name@example-vendor.com
/product/prd-XXXXXXX                      (bare id, slug truncated)

These are artefacts of third-party crawlers mis-parsing our own output:

• url(%22https:/fonts… is a CSS url("https://fonts…") declaration resolved as a relative page
  URL — note https:// collapsed to https:/ by the crawler's normalizer. A single sampled snapshot
  contained 200 url( occurrences.
• target= is the target attribute concatenated onto the href.
• ' defer='defer is script attributes concatenated.

Verified NOT our bug: the same sampled snapshot had 162 hrefs, 0 containing suspicious tokens
and 0 unquoted attributes. Our emitted HTML is well-formed; the mis-parsing is crawler-side.

Worth noting that prerendering enlarges the attack surface: a rendered page is ~2.5x the size of the
origin SSR document with substantially more markup and CSS, so bad parsers have more to mangle than
they would against the origin.

Then the chain closes:

1. crawler requests the malformed URL
2. cache miss → handlePageScheduling creates a Target — no URL validation
3. render → the CDN/WAF answers 403 (these read as injection payloads: quotes, url(, =
   fragments), not 404
4. RenderQueue classifies 401/403 as auth-shaped and deliberately does not suppress — correct in
   general, because an auth failure is usually a broken renderer credential and striking would
   mass-delete healthy targets during an outage
5. failureRetry retries each ~2x/day, forever

Each decision is individually right. Together they make an immortal class of targets.

Impact

Compute cost is trivial — ~0.2% of render throughput. The real damage is diagnostic:

RenderQueue logs, per occurrence:

Prerender got 403 for <cacheKey> — auth-shaped, NOT suppressing. If these are widespread, check the renderer's origin-bypass credential and the CDN/origin access rules.

That is exactly the alarm you need for a genuine bypass-token break or a CDN rule change — and it
currently fires ~2,950 times/day as steady-state noise, so a real incident has to be spotted against
that floor. This matters more as crawl volume grows: the junk population scales with crawl traffic.

Proposed fix (two parts, complementary)

1. Validate URLs at discovery

Reject non-canonical shapes in handlePageScheduling before creating a Target. This prevents the
row, the schedule entries, and every downstream render — the cheapest possible point to stop it.

Prefer a positive rule over a blocklist: for a known route, require the path to match that route's
canonical shape (e.g. product slugs restricted to a safe charset). A blocklist of url(, quotes,
whitespace, =-in-segment etc. would work today but invites an endless game of catch-up.

Should be config-driven and default-on, with a metric/counter for rejected discoveries so the rule's
selectivity is observable (and a too-strict rule is visible rather than silent).

2. Let 403 suppress under a never-successful predicate

Keep the mass-outage protection, but narrow it. Suppress on 401/403 only when all hold:

• sitemapUrl is null (never submitted), and
• strikes >= maxStrikes, and
• no PrerenderedPage exists for any device cacheKey (never rendered successfully in its lifetime)

A target meeting all three has never produced a page and was never submitted, so retiring it cannot
lose anything — and any submitted URL keeps the current unconditional protection, which is the case the
exemption was written for. Deletion is self-healing regardless: a later request proxies to origin and
rediscovers the URL.

Part 1 stops new arrivals; part 2 retires what slips past (and anything already accumulated).

Interim action taken

1,106 of these targets were deleted via Target.delete() (which also drops the RenderSchedule rows
and PrerenderedPage rows — a raw table delete would orphan the schedule row, and claim builds jobs
from the schedule alone without checking the target still exists). Every deletion was gated on the
three predicates above; 0 were in a sitemap, 0 had a cached page, 0 errors.

That is a cleanup, not a fix — without part 1 they simply come back on the next crawl.