fix(search): restore the API reference to search and stop docs-version renames from breaking it - #428
Open
eugenia-scandit wants to merge 4 commits into
Open
fix(search): restore the API reference to search and stop docs-version renames from breaking it#428eugenia-scandit wants to merge 4 commits into
eugenia-scandit wants to merge 4 commits into
Conversation
Releasing 8.5.3 renamed the docs version from `current` to `8.5.3`. Docusaurus builds `docusaurus_tag` from the version NAME, so every page at the site root flipped from `docs-default-current` to `docs-default-8.5.3` without a single URL changing. Two things broke, both silently: 1. The generated API reference under /data-capture-sdk/ is crawled into the same index but is not a Docusaurus version - it is tagged `docs-default-current`. While lastVersion was "current" it happened to share the guides' tag and rode along for free. After the rename it no longer matched the contextual filter, and ~3,200 pages left every search result. A representative query returned 86 hits where the index held 106 reachable ones. 2. buildVersionTagByMajor let `current` win its major unconditionally. With 8.6.0-beta as current and 8.5.3 as lastVersion, typing "v8" routed readers at the unreleased beta's tag - which still returned results, because that tag held the API reference, so nothing looked wrong. Neither failed loudly. Search just got worse, and the monitoring could only report "too few records". Changes: - docusaurus.config.ts is now the single source of truth for versions and every tag derived from them. DOCS_LAST_VERSION is declared once beside docsVersions instead of being restated in the preset, docVersionTag is the only place a `docs-default-*` string is built, and ALWAYS_ON_SEARCH_TAGS names the indexed content Docusaurus cannot tag for us. - buildVersionTagByMajor now follows lastVersion and skips unreleased versions, so a typed major routes where the reader actually is. - SearchBar widens the contextual OR group with the always-on tags, and the typed-version rewrite no longer clobbers them. - A build plugin writes build/search-tags.json, and scripts/verify-search-tags.cjs (yarn verify:search-tags, wired into CI) checks it three ways: the routed tag for the served major must be the served tag, the build must actually emit every tag the config names, and no tag holding real content may sit outside what search can reach. Crawl lag warns; config drift fails. The gate reproduces both bugs on the pre-fix config and passes on this one. It also surfaces two orphan cohorts left by earlier renames - docs-default-8.0.0 (10 pages) and docs-default-6.28.1 (18) - which want purging from the index. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
My previous commit restored the API reference to search by OR-ing its tag in on every page - including the frozen 7.6.14 and 6.28.11 trees, where it had never appeared before. Measured on a 7.6.14 page, one query went from 53 hits to 73, and all 20 extras were current-SDK API pages that do not apply to a reader on that version. Before the 8.5.3 rename this scoping happened by accident: the API reference shared the served version's tag, so it appeared there and nowhere else. This makes it explicit - alwaysOnScopeTags names the versions the extra tags apply to (the served version and the in-development one), and the widget injects them only when the page's own tag is in that set. Also starts the crawler migration. The API reference should not be tagged with a version name at all: `docs-default-current` is the 8.6.0 beta's own tag, which is exactly why a release could rename the API reference out of search. ALWAYS_ON_SEARCH_TAGS now lists `api-reference` alongside it, so retagging the Algolia crawler's /data-capture-sdk/** action can land before or after this commit without breaking search either way. The gate reports an empty always-on tag as a pending migration, and only fails if no always-on tag carries the content. Adds scripts/test-search-facets.cjs (yarn test:search-facets, wired into CI). It reads the two functions out of the shipped module rather than copying them, and pins both failure modes: too narrow and content vanishes from search, too wide and legacy readers get results that do not apply. Neither throws in production, so only an assertion catches them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
I had this wrong twice, and the correction matters: the API reference IS versioned. It is published per major.minor line at /<major.minor>/data-capture-sdk/<framework>/ - /6.28/, /7.6/, /8.4/, /8.5/, /8.6/ all return 200. My earlier probes used /6.28.11/, which 404s, and I concluded from that there was only one API reference. There is one per line, and each docs version has a matching one. That makes the previous design wrong in both directions: OR-ing one API reference into every version polluted legacy search with current-SDK pages, and scoping it to the served version alone hid the 6.28 and 7.6 API references from the readers they belong to. So the API reference is now mapped per version rather than bolted on. buildApiReferenceTags derives, from docsVersions alone, the tag each docs version's API reference carries: docs-default-6.28.11 -> api-reference-6.28, docs-default-7.6.14 -> api-reference-7.6, docs-default-8.5.3 -> api-reference-8.5, docs-default-current -> api-reference-8.6. A reader on 6.28.11 gets the 6.28 API reference and never the 8.x one, and typing "v7" moves the guides and the API reference together. Nothing here needs maintaining per release. The repo reads the line out of docsVersions; the crawler is expected to read it out of the URL (/8.5/data-capture-sdk/... -> api-reference-8.5). Neither side hard-codes a version, which is the whole point: tagging the API reference with a docs version NAME is what broke search when 8.5.3 renamed it. Also fixes a regression I introduced. Hoisting DOCS_LAST_VERSION left scripts/update-version.py reading the `lastVersion:` line, whose last quoted string is now "current" - so the next release would have aborted with "Already in production state". All four read/write sites now target the constant, which is the value both the docs plugin and the tag derivation actually use. Tests rewritten around the version mapping: a legacy version gets ITS API reference and not the current one, an unknown version adds nothing rather than guessing, and a typed version moves both together. 7 passing. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…links
Correcting my own previous commit. It mapped the served version to
`api-reference-8.5`, on the assumption that every docs version has a
matching /<major.minor>/data-capture-sdk/ tree the crawler can reach. It
does not, and the assumption would have dropped the current API reference
out of search a second time.
What the site really does, verified against production:
- the sitemap contains ZERO /data-capture-sdk/ URLs, so the crawler can
only find that tree by following links
- the 7.6.14 guides link to /7.6/data-capture-sdk/...
- the guides served at the root link to the UNVERSIONED
/data-capture-sdk/... - nothing anywhere links to /8.5/
So /8.5/data-capture-sdk/ exists but is undiscoverable, and pointing the
served version at it would have pointed at an empty tag.
The mapping now mirrors the site's own linking instead of assuming a
symmetry that isn't there: the version served at the root and the
in-development one take the unversioned tree (`api-reference-latest`),
every frozen version takes its own line (`api-reference-7.6`,
`api-reference-6.28`). That is what the crawler will actually produce from
the URL, and it stays automatic - this file reads the version out of
docsVersions, the crawler out of the URL, neither hard-codes one.
Records are only added by this change, never removed: the unversioned tree
keeps its URLs and just changes tag. maxLostRecordsPercentage does not need
raising.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What was broken
Releasing 8.5.3 renamed the docs version, so
docusaurus_tagon every page atthe site root changed from
docs-default-currenttodocs-default-8.5.3—while not a single URL changed.
Two consequences, both silent:
it carried the tag
docs-default-current. While that tag matched the guides'tag, it was found alongside them for free. After the rename ~3,200 pages left
every result. A query with 106 matching pages in the index returned 86.
buildVersionTagByMajorletcurrentwin its major unconditionally. With8.6.0-beta as
currentand 8.5.3 aslastVersion, typing "v8" sent readersat the unreleased beta's tag — and it still returned results, so nothing
looked wrong.
The monitoring could only report "too few records".
What this changes
docusaurus.config.tsis the only place versions are declared and tags arebuilt.
DOCS_LAST_VERSIONis declared once besidedocsVersions(it used tobe restated in the preset), and
docVersionTagis the only constructor ofdocs-default-*strings.buildVersionTagByMajorfollowslastVersionand skips unreleased versions.version gets the API tree it belongs with —
7.6.14 → api-reference-7.6,6.28.11 → api-reference-6.28, and the version served at the root → theunversioned tree. A reader on 6.28.11 no longer gets 8.x API pages.
DOCS_LAST_VERSIONleft
scripts/update-version.pyreading thelastVersion:line, so the nextrelease would have aborted with "Already in production state". Verified by
running a real release end to end.
Why the mapping mirrors the site rather than assuming symmetry
The sitemap contains zero
/data-capture-sdk/URLs, so the crawler can onlyreach that tree by following links. The 7.6.14 guides link to
/7.6/data-capture-sdk/…, but the guides served at the root link to theunversioned
/data-capture-sdk/…— nothing anywhere links to/8.5/.Pointing the served version at
api-reference-8.5would have pointed at a treethe crawler never reaches, dropping the current API reference out of search a
second time.
Preventing a repeat
yarn verify:search-tags(in CI) compares the build against the live index:the tag a typed major routes to must be the tag the site serves; the build must
emit every tag the config names; and content under a tag search cannot reach is
an error. Crawler lag warns instead of failing.
yarn test:search-facets(in CI) — 7 assertions on what the widget actuallysends to Algolia, including that a legacy version gets its API reference and
not the current one.
Both reproduce both bugs against the pre-fix config.