`lookup()` returns providers nested *inside* the matching entry, not just those containing the query

[chrisdromey]

Summary

The two-pass merge in build_data_structures() files every nested CIDR/domain under its
outermost container's key. A lookup for any target inside that container then returns the
nested provider too. The blast radius is the whole container, not the nested subtree.

This looks like the over-correction of a known problem rather than an unnoticed bug — the
comment at src/lib.rs:306-311 shows the second pass was added deliberately, to stop ACL mode
destroying the child node when a broader entry is inserted. That fix works, but it widens the
key further than the comment's reasoning implies: the example reasons about
blob.core.windows.net, while the key actually used is windows.net.

Reproduction

Three synthetic providers — BigCloud owns 10.0.0.0/8 and bigcloud.example, TinyCdn owns
only 10.1.2.3/32 and node.bigcloud.example, Elsewhere owns 192.0.2.0/24:

10.9.9.9              -> ['BigCloud', 'TinyCdn']     expected ['BigCloud']
10.1.2.4              -> ['BigCloud', 'TinyCdn']     expected ['BigCloud']
www.bigcloud.example  -> ['BigCloud', 'TinyCdn']     expected ['BigCloud']
192.0.2.5             -> ['Elsewhere']               correct

Deterministic across runs (30/30 identical), despite HashMap iteration order being randomised
per process — the double pass converges the tree to its maximal entries either way.

Against the live signature file

cloud_providers_v3.json from stable, md5 bf1cf1c043047e909a0a26d1090dc2be. "Contained by"
is computed from that same file with Python's ipaddress — suffix match for domains,
containment for cidrs, which is the semantic radixtarget's own matcher implements
(src/ip.rs:44-93, src/dns.rs:59-76):

query	lookup()	contained by, same file
18.195.165.195	Amazon, Cloudfront, Quiccloud	Amazon
52.67.31.86	Amazon, Cloudfront, Gocache	Amazon
108.132.147.156	Amazon	Amazon (correct)
jlrweb...nlb.elb.eu-west-1.amazonaws.com	Amazon, GitHub, HPE, Microsoft	Amazon
foo.s3.amazonaws.com	Amazon, GitHub, HPE, Microsoft	Amazon
asdf.windows.net	GitHub, Microsoft, Microsoft365	Microsoft

The clearest cases are unrelated third parties:

• QUIC.cloud (LiteSpeed CDN) is returned for all of 18.128.0.0/9 — 8.4M addresses — because
  of one entry, 18.192.146.200/32.
• GoCache (Brazilian CDN) for all of 52.64.0.0/12, from 52.67.255.165/32.
• HPE for every *.amazonaws.com hostname, from hpefonts.s3.amazonaws.com.

Note the domain key is amazonaws.com, not s3.amazonaws.com — Amazon's only entry in that
subtree is the bare domain — so ELB, RDS, Lambda and SQS hostnames are affected too, not just S3.

Mechanism

RadixTarget::new(&[], ScopeMode::Acl) at src/lib.rs:303. In ACL mode, radixtarget
deliberately deduplicates:

• insert(child) when a parent covers it -> returns None, stores nothing (ip.rs:23, dns.rs:35)
• insert(parent) when children exist -> node.clear() wipes them (ip.rs:39, dns.rs:51)
• get(child) -> returns the ancestor's key

build_data_structures() then files the provider under whatever get() returned
(src/lib.rs:323-324, 342-343), so the nested provider lands on the container's key. lookup()
(src/lib.rs:456-458) resolves the target to that same key and returns the merged list. Because
ProvidersMap is keyed on the merged key, the association between a provider and its own
sub-entry is gone by the time anything could filter on it.

This is correct behaviour for an ACL — it is the wrong structure for provider attribution.

Scale

From the live file, replicating the ACL convergence: 292 merged IPv4 keys return more than one
provider, covering ~130.2M addresses. Affected root domains include amazonaws.com,
windows.net, azure.com, github.com, akamaized.net, outlook.com, office.com.

Test that encodes the current behaviour

test_lookup_windows_blob_domain (src/lib.rs:509-527) asserts asdf.blob.core.windows.net
returns GitHub. GitHub lists only specific hosts — copilotprodattachments.blob.core.windows.net,
productionresultssa0-19.blob.core.windows.net — not the blob.core.windows.net suffix. So the
asserted host is not one GitHub claims. Worth revisiting alongside a fix.

Downstream effect in bbot

bbot 3.0.2 pins cloudcheck>=11.1.0,<12 and passes provider["name"] and provider["tags"]
straight to event.add_tag() (bbot/modules/internal/cloudcheck.py:144-163), with no
containment re-check. Tags then propagate to descendant events via the inheritance path.

QUIC.cloud, GoCache and CloudFront all carry tags: ["cdn"]. In scans that enable portfilter
(the nuclei/* and web/lightfuzz-* presets do; it is not on by default), a cdn tag drops
every open port outside 80/443 — bbot/modules/portfilter.py:34-42. So one unrelated /32 inside
18.128.0.0/9 can suppress non-web ports across that whole range. That is a false negative in
attack-surface discovery, which is why we chased it.

Suggested direction

Return providers whose own entry contains the query — ancestors only, never siblings or
descendants. That likely means not keying attribution on an ACL-deduplicated tree: either a
non-ACL tree that keeps every node and walks ancestors collecting matches, or the pre-11
arrangement of one tree per provider.

Notes

• build_data_structures is byte-identical in 11.0.0 and 11.1.0; both are affected.
• No commit after v11.1.0 touches src/ — all are daily signature updates. No branch carries a fix.
• Pre-Rust 7.x used one radix tree per provider (cloudcheck/providers/base.py:66,
  providers/__init__.py:86-93), so it could not merge across providers. This is a regression
  introduced by the Rust rewrite, not by 11.x specifically — cloudcheck-v8 already had the
  single-pass form of the same filing pattern.

---

This was pretty much all Claude's work - blame him if its wrong... I'll take the credit if its useful!