feat: build-generate AI retrieval layer (knowledge graph + index)

[eugenia-scandit]

Add a Docusaurus postBuild plugin that turns the rendered docs into an AI-consumable layer, generated at build time:

/assets/knowledge-retrieval-index.json (fast lookup)
/assets/knowledge-graph.jsonld (concept graph)

Parses the final rendered HTML (so imported partials/MDX are captured in full), splits each current-version page into ~1400-char knowledge modules, and derives per-chunk metadata: summary (from the frontmatter description), rule-based intents/audiences, framework, product, and the real URL. The graph mines real edges — BelongsToProduct, CitesApi, SeeAlso, and per-product AvailableOn / NotAvailableOn. Current docs only (frozen versions, the external API reference, and *.html redirect stubs excluded). Extraction failures are non-fatal so they never block a deploy.

[lucatorella]

🤖 Two build/deploy-integrity findings inline.

[lucatorella]

🤖 cheerio is imported directly but is not a declared dependency — it resolves today only by npm hoisting, and losing it breaks the whole site build, not just this plugin.

On this branch package.json has no cheerio in dependencies or devDependencies (0 matches), and neither lockfile is touched by the PR. In package-lock.json, node_modules/cheerio (1.0.0-rc.12) has exactly one requirer:

required by: ['node_modules/@docusaurus/plugin-content-blog']

So the import works only because a transitive dep of preset-classic happens to hoist cheerio to the root. @docusaurus/core and @docusaurus/preset-classic are pinned as ^3.4.0, so any Docusaurus 3.x bump that stops pulling cheerio into plugin-content-blog — or a npm dedupe, a lockfile regeneration, or the yarn.lock path with a stricter resolver — removes it.

The consequence is worse than a broken artifact. This is a top-level import in a module that docusaurus.config.ts imports, so resolution failure happens at config load, before any plugin lifecycle runs. The try/catch at the bottom of postBuild — which the PR description relies on for "extraction failures are non-fatal so they never block a deploy" — cannot catch it. The failure mode is npm ci && npm run build dying on every branch, not a skipped artifact.

Adding cheerio to dependencies (and letting the lockfile update) fixes it.

[lucatorella]

🤖 Still open at d948727b. package.json on the head has no cheerio in either dependency block (the only change is the new docs:retrieval-evals script), so the import continues to resolve purely through hoisting from @docusaurus/plugin-content-blog.

Worth noting the new Docs Retrieval Evals job raises the stakes slightly: it runs yarn build on every PR touching docs/** or the extractor, so once cheerio stops being hoisted, that job fails alongside Build Docusaurus Site rather than the failure being confined to this plugin.

[lucatorella]

🤖 Nothing asserts that anything was actually extracted, so this can publish empty artifacts over good ones with a fully green build and no signal.

index is written unconditionally. Combined with the catch-all at the end of postBuild:

} catch (err) {
  console.warn(`[knowledge-extractor] skipped (non-fatal): ${(err as Error)?.message || err}`);
}

there is no build outcome that distinguishes "extracted 9000 modules" from "extracted 0". Two concrete paths get there:

1. Selector drift. Every page is admitted via $("article .markdown") / $(".theme-doc-markdown") / $("article"), and a non-match is continue (if (!root.length) continue;). A Docusaurus or theme upgrade that renames those classes makes every page fall through, index becomes [], and the build succeeds — deploying an empty index and a graph with no nodes.
2. Mid-write failure. The two writeFileSync calls are sequential and share one catch. A throw between them ships an index with no matching graph, again with a passing build.

Because these are build-time generated and served from outDir, a bad run doesn't leave the previous good copy in place — consumers get empty JSON or a 404, and the only trace is a console.warn in build logs nobody greps.

The rest of this config is deliberately fail-loud about build integrity (onBrokenLinks: "throw", onBrokenAnchors: "throw", onBrokenMarkdownLinks: "throw"), so a floor check would match that convention: throw (or at minimum refuse to overwrite) when pagesProcessed or index.length is 0, or below some fraction of the pages walked. "Non-fatal" is a reasonable goal for one bad page; it shouldn't extend to "extracted nothing at all".

[lucatorella]

🤖 Partially mitigated at d948727b, but not closed. The new eval script does hard-fail on an absent or empty index:

if (!fs.existsSync(INDEX)) { console.error(`retrieval-evals: index not found at ${INDEX} …`); process.exit(1); }
if (!Array.isArray(index) || !index.length) { console.error("retrieval-evals: empty or invalid index."); process.exit(1); }

so a build that silently extracts nothing now gets caught — on PRs. Two gaps remain:

1. docs-retrieval-evals.yml triggers only on pull_request with a paths filter (docs/**, the extractor, the eval scripts, its own workflow). There is no push trigger, so nothing checks the index on main or on the branch that actually deploys. A build that starts emitting an empty index for a reason unrelated to those paths — a Docusaurus upgrade renaming article .markdown, say — ships with every job green.
2. The plugin itself still writes whatever it has and swallows its own errors, so the artifact pair can still go out mismatched (index written, graph not) without the eval noticing: it only reads the index.

A floor check inside the plugin still seems worth it, since it's the only place that knows how many pages were walked versus how many produced modules.

[lucatorella]

🤖 Re-reviewed 65676e09..d948727b. One finding on the new metric; notes on the two earlier threads.

[lucatorella]

🤖 recall@k is algebraically identical to precision@k for every query in the gold set, so --min-recall adds a blocking threshold that provides no protection beyond the one already there.

const precision = relCount / Math.max(hits.length, 1);
const denom = Math.min(totalRelevant(g.expect), K) || 1;
const recall = relCount / denom;

hits comes from search(index, g.query, K), so hits.length === K for any index with at least K modules. And denom === K whenever totalRelevant >= K. Both denominators are then K, both numerators are relCount, and the two metrics are the same number — query by query, not just on average.

totalRelevant >= K holds for all 20 gold entries. Counting pages under each expect path-class in docs/ (and every page yields at least one module):

/sparkscan/                     34      /id-capture/               44
/matrixscan-count/              34      /id-bolt/                  13
/matrixscan-find/               34      /express/                  18
/label-capture/                 43      barcode-symbologies        24
/barcode-capture/get-started    11      /agent-skills               9

The smallest is 9, against K = 3. The || 1 guard and the min(total, K) cap only bite when a path-class has fewer than 3 relevant modules in the whole corpus, which no current entry does — so the cap never engages and the commit's stated "capped recall" degenerates to precision.

Two consequences worth heading off:

• The report and the console will print precision@3 and recall@3 as two equal numbers that move in lockstep. That reads as two independent signals corroborating each other, when it's one signal printed twice.
• MIN_RECALL = 0.6 is really a precision floor. Precision currently has no threshold, so if a precision floor is what's wanted, that's a reasonable thing to add — but it should be named that, and MIN_SUCCESS/MIN_MRR shouldn't be joined by a third gate that can only fail when precision already did.

A recall metric that actually measures recall needs an uncapped denominator (relCount / totalRelevant, reported at a k large enough to be meaningful), or a gold set that names specific pages rather than path-classes so totalRelevant is small and the cap has something to cap.

[raffaelefarinaro]

🤖 This review was AI-assisted. I verified each blocking finding by hand against the code and a local build before posting.

Starting with what works, because it is genuinely careful. The exclusion logic is correct, and I checked it against the built output rather than reading it: zero URLs from the frozen versions, the external data-capture-sdk reference, or any of the 235 *.html redirect stubs make it into the index. Build cost is not a regression, Build Documentation on this branch ran 3m29s to 4m08s against 3m36s to 4m00s on recent main. And the eval gates sit far above chance, since a random retriever over this corpus would score success@3 = 0.158 against your floor of 0.8, with 0.900 actual.

Your three open threads are already fixed, so close them

You have probably been looking at three red threads for six days. They are stale, not outstanding:

• cheerio and gray-matter are both declared in dependencies as of 1748eaf, with matching entries in both lockfiles.
• The zero-modules floor check at index.ts:697 and the .tmp plus rename writes at index.ts:712 landed in the same commit, which is exactly what the thread asked for.
• Recall was removed and replaced with an honestly named MIN_PRECISION in 972e6cb, plus a comment explaining the algebra.

Luca's re-confirm was against d948727b, which is six commits behind the current head 2a6edd5. I checked the recall claim against the built corpus rather than taking it on trust, and he was right: every one of the 20 gold entries resolves to between 31 and 1075 relevant modules against K = 3, so capped recall did reduce to precision. The fix is the right one.

Rebase before responding to anything else here

The last four commits, which add the entire frontmatter-ingestion feature, have zero CI runs. Here is why. main deleted package-lock.json in f23a28f (#416), and this branch still edits it, so git merge-tree reports a modify/delete conflict on package-lock.json plus a content conflict in package.json. GitHub cannot build the refs/pull/418/merge ref for a conflicted PR, so no pull_request run gets created. Runs exist and are green for the first five commits and then stop dead on 2026-08-05 at 12:53Z. gh pr checks 418 reports "no checks" because it queries head 2a6edd5 specifically.

A rebase onto main plus deleting package-lock.json fixes the conflict, restores CI, and picks up docs-gate.yml, which does not exist on this branch at all. Do that first, because CI will probably surface more than any human reader can, and it would be wasteful to work through review comments against a head that has never been built.

Provenance of the numbers

The corpus statistics below (4523 records, 89% at the 400-character cap, 219 Titanium modules, 26.9 MB, the random-retriever baseline) come from my own local build on Node 26 with npm ci, not from CI on Node 18 with yarn. My counts came out at 4523 modules against CI's 4526 at 972e6cb, a 0.07% difference I attribute to the four intervening commits. Treat the figures as indicative of magnitude, not exact.

What is in the inline comments

Five blocking: the package-lock.json conflict, the Titanium trees, the chunk truncation, the contradictory core availability edges, and the division of labour with docusaurus-plugin-llms. Three smaller ones: a substring match in pickIntents, MIN_AUTO_SUCCESS defaulting to zero, and last_verified being the build timestamp.

Two of these are product decisions more than defects, so I have put them as questions about intent rather than verdicts. I also could not see who consumes these artifacts, so my point about truncation argues from the plugin's own stated purpose in its header comment, not from a requirement I can verify. If a consumer exists that only needs pointers plus a snippet, that finding weakens a lot and I would rather hear that from you than assume it.

I have left a longer writeup with the non-blocking observations out of this review to keep it readable. Happy to walk through it whenever suits.

[raffaelefarinaro]

Blocking: this file is why the PR cannot merge and why CI has gone quiet.

main deleted package-lock.json in f23a28f (#416). This branch still modifies it, which produces a modify/delete conflict, and a conflicted PR gets no pull_request workflow runs at all because GitHub cannot build the merge ref.

The repo is yarn-only: every workflow runs yarn install --frozen-lockfile. Please delete package-lock.json in the rebase rather than updating it. Keeping both lockfiles guarantees drift, and I hit that directly while reviewing: running npm ci on this branch rewrote yarn.lock with 251 insertions and 130 deletions of unrelated Algolia version churn.

The yarn.lock side of your diff is correct and minimal, so nothing to do there beyond stripping the two unrelated registry-URL flips on fsevents and search-insights that came along for the ride.

[raffaelefarinaro]

Blocking, as a question rather than a defect: what is the division of labour between this and docusaurus-plugin-llms?

Seven lines above this, the same config registers docusaurus-plugin-llms, which already emits an AI-consumable layer from the same docs on the same build: llms.txt at 0.07 MB and llms-full.txt at 2.03 MB in my build. That plugin's setup here is deliberately curated, with shared-partial dedup, non-Web-SDK root trimming, and measured before/after numbers in the header comment.

This plugin adds a second layer with different curation rules and 26.9 MB of output. I am not assuming one should go, but two parallel AI exports with no stated boundary is the thing most likely to rot, and the next person to touch either one will not know which is authoritative. Could you add a line to the PR description saying what each is for, and whether the intent is for one to eventually replace the other?

[raffaelefarinaro]

Blocking: this exclusion set does not honour a curation decision made elsewhere in the same config.

docusaurus.config.ts:49 declares llmsIgnoredSdkTrees = ["docs/sdks/titanium/**"] with the comment "Entire platform omitted from llms export (deprecated / not needed for assistant context)". build/llms.txt has zero /titanium/ references. This index has 219 modules under /sdks/titanium/.

Nothing secret leaks, so this is not a security problem, but the repo has already decided which trees it wants an assistant reading, and this artifact overrides that decision silently and publishes the result to the public site.

Was that deliberate? If the new layer should have wider coverage than the llms export, that is a defensible answer and worth writing down. If not, reusing llmsIgnoredSdkTrees and the shared-partial dedup list would keep the two layers consistent for free.

[raffaelefarinaro]

Blocking: the index does not ship the content it chunks.

CHUNK_TARGET_CHARS is 1400 at line 30, but these two lines truncate to 400 and 300. In my build, 4031 of 4523 records (89%) sit exactly at the 400-character cap and 4475 (99%) sit exactly at the 300-character cap. So roughly 70% of every chunk gets parsed, chunked, held in memory, and then discarded, and each published record is a fragment cut mid-sentence.

The header comment describes this as the artifact "an assistant / in-docs search consume". Against that goal, a consumer gets a snippet and still has to fetch the URL, which llms-full.txt already covers.

I want to be careful here, because I could not see who actually consumes these files. If the design intent is a pointer index where a 400-character preview is enough, this is fine as built and I would just note it in the header. If the intent is self-contained retrieval modules, then either raise the caps or drop the chunking to page level, because the 1400-character chunking currently buys nothing that survives into the output. Which is it?

[raffaelefarinaro]

Blocking: this "core" fallback produces five self-contradicting availability edges.

core is a bucket for framework-level pages, not a product, so it collects both real pages (/sdks/linux/add-sdk/) and "not available" stubs (/sdks/linux/ai-powered-barcode-scanning/). Because availability is aggregated per product across modules at lines 560 to 580, the graph then emits both edge types for the same pair.

Confirmed in the built graph: urn:product:core carries both AvailableOn and NotAvailableOn for linux, net-android, net-ios, titanium and web. That is 5 of the 29 NotAvailableOn edges contradicting themselves, and these are the edges the PR description calls out as the ones that "directly answer what's available where".

Simplest fix is to skip availability mining when product === "core". Alternatively let NotAvailableOn win when both are present, though excluding the pseudo-product seems more honest.

Worth noting the stub detection itself is sound, which is why I looked here rather than there: all 70 files matching is not available (on|for) the are 298 to 394 byte stubs, and the six files with other "not available" phrasings are legitimate prose that correctly does not match.

[raffaelefarinaro]

text.includes(" auth") fires on any word starting with "auth", not just authentication. Every one of the 31 modules tagged with the secure intent in my build matched on authorized, authenticity or authority.

A word-boundary regex fixes it, something like /\bauth(entication|orization|n)?\b/. Small blast radius, easy fix.

[raffaelefarinaro]

updatedAt is new Date().toISOString() from the top of postBuild, so last_verified is the build timestamp. Every module in every deploy claims it was verified today. All 4523 records in my build carry 2026-08-11.

A consumer using this for freshness gets noise shaped like signal, which is worse than an absent field. Either drop it, or wire it to the source file's git mtime, which you already have the path for via readFrontMatter.

Same applies to updated_at in the metadata block just below.

[raffaelefarinaro]

MIN_AUTO_SUCCESS defaults to 0, and the gate at line 161 is page_success_at_k < MIN_AUTO_SUCCESS, so --auto can never fail. Nothing in the workflow passes --auto today, so this is latent rather than live, but if the intent is a corpus-wide floor it needs a real default.

Two related notes while you are in this file. --auto is O(pages x modules), roughly 2.6 million score computations at the current corpus size, so it is worth timing before wiring it into CI. And the header comment at line 5 points at docs/assets/knowledge-retrieval-index.json, while the actual default at line 32 is build/assets/....