Add extraction parameter on POST /v1/search (3.1.0) - #40
Merged
Conversation
- New typed `extraction` object (Extraction/ExtractionMode/ExtractionFormat and TypedDicts) on search()/search_async() and SearchShim forwarders. - Strict models (extra=forbid) fail-fast locally on unknown keys. - Highlights max_tokens bounded to [512, 8192]; full_page formats supported. - Plus-value rule: crawl_timeout stripped alongside highlights mode with a UserWarning for explicit non-default values. - Conflict check: extraction + livecrawl/livecrawl_formats raises ValueError. - Adds highlights field to Contents model; tightens live test assertions.
Contributor
|
Droid encountered an error —— View job Droid is reviewing code and running a security check… |
- SearchShim.__call__/unified/unified_async now type extraction as Optional[Union[Extraction, Mapping[str, Any]]], matching the _search_impl/search_async contract so you.search() keeps IDE hints. - Add wire tests passing an Extraction model instance for full_page and highlights (plus-value crawl_timeout strip), previously only covered by live tests.
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive and well-tested. One correctness tweak to consider is reordering the livecrawl deprecation warning so the extraction+livecrawl conflict reliably raises |
![factory-droid[bot]](https://avatars.githubusercontent.com/in/358017?s=60&v=4){kind=small}
DX-719 introduces the typed `extraction` parameter on `POST /v1/search`,
replacing `livecrawl` / `livecrawl_formats`. The `Contents` model's
`html`, `markdown`, and (new) `highlights` fields are populated whenever
the caller attaches page content — whether via the deprecated `livecrawl`
form or the new `extraction` form.
Update the six docstrings on `Contents`/`ContentsTypedDict`,
`WebResult.contents`/`WebResultTypedDict.contents`, and
`NewsResult.contents`/`NewsResultTypedDict.contents` so each reads:
Contents of the page if ``extraction`` was enabled (formerly ``livecrawl``).
This aligns the auto-generated API reference prose with the SDK README and
CHANGELOG language ("`extraction` replaces the deprecated
`livecrawl` / `livecrawl_formats` parameters") and avoids a stale
"if livecrawl was enabled" reading after DX-719 ships.
Pure docs change. No runtime behavior. Tests unchanged.
Refs: DX-719
Contributor
|
Droid finished @tyler5673's task —— View job Overall, the extraction addition is cohesive and well-tested. Minor user-facing polish remains: reword the SearchRequestBody extraction docs to reflect that crawl_timeout stripping avoids a 422 (it is not “fail-fast”), and rename one new test whose name currently contradicts its full_page assertions. |
DX-719 ships the new extraction parameter and demotes livecrawl.
DX-719 acceptance criteria require every surface that mentioned the
prior parameter to teach the replacement:
* MIGRATION.md -- 3.0.0 -> 3.1.0 section with mapping table,
before/after Python blocks, and a no-action-required demo so users
see the migration path even though livecrawl still works.
* docs/models/extraction.md -- new model page mirroring
docs/models/livecrawl.md's shape (description, example, fields).
* docs/models/contents.md -- description now reads "if `extraction`
was enabled (formerly `livecrawl`)" plus a new highlights row.
* docs/models/searchrequestbody.md -- extraction row added.
* docs/sdks/{search,you}/README.md -- extraction row added in both.
* tests/test_performance.py -- three new extraction-mode perf cases
(highlights dict, full_page model, full_page dict both formats)
mirror the legacy livecrawl cases.
* examples/api-example-calls.py -- search_request rewritten to use
Extraction; search_request_livecrawl_legacy retained so demo users
can see both paths.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Adds a "When the new parameter deprecates or replaces an existing one" section so future agents do not miss the same surfaces missed this round (no MIGRATION.md entry, no docs/models/<newmodel>.md page, no extraction row in the SDK-internal doc tables, no runnable example, no perf test). Also extends step 5 of "Adding a new parameter to search" to enumerate the SDK-internal doc tables. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive and well-scoped. The only remaining high-confidence issue from this pass is a small docs mismatch around the optionality of |
…ge rows The source-model docstrings already say \"Contents of the page if \`extraction\` was enabled (formerly \`livecrawl\`)\" (committed in 7d6a15a), but docs/models/webresult.md and docs/models/newsresult.md still read \"if livecrawl was enabled\". Update those two rows so the documented surface matches the model surface and the AGENTS.md \"docstring carry-over\" rule holds end-to-end. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive and well-scoped, with solid typing and tests. The main remaining gap is user-facing docs, the SDK parameter tables should clearly steer new users away from deprecated |
Two P1s:
* src/youdotcom/sdk.py: swap the warn-conflict ordering in
_build_search_request so the ConflictError wins even when callers
treat DeprecationWarning as error.
* docs/{models,sdks/{search,you}}/{README,md}: mark the livecrawl and
livecrawl_formats Parameter rows as deprecated, with a pointer to
the corresponding extraction fields. Update the crawl_timeout row in
the three POST-side tables so it covers both full_page and the
deprecated livecrawl; mark it as no-op on the GET Search sub-SDK.
Three P2s:
* src/youdotcom/models/searchrequestbody.py: reword
"so callers fail-fast instead of round-tripping a 422" in both
TypedDict and BaseModel docstrings to "to avoid round-tripping a
422" (the SDK intentionally strips crawl_timeout so the request
succeeds, not fails).
* tests/test_extraction.py: rename the test that constructs
Extraction(extraction_mode="full_page") to
test_extraction_full_page_default_no_subkeys (the prior name promised
the highlights mode).
* docs/models/contents.md: Contents.highlights is Optional[List[str]],
not List[str]. Update the Type cell so callers don't assume the
field is always present.
Verified locally:
* 137 unit tests pass (test_extraction, test_search,
test_param_normalization, test_shims, test_direct_methods,
test_security_env, test_client_lifecycle, test_redaction,
test_client) including both conflict tests
(test_extraction_plus_livecrawl_raises_value_error and
test_extraction_plus_livecrawl_formats_raises_value_error).
* mypy clean across 81 source files.
* pylint errors-only clean (only the pre-existing
pylintrc:1:0 E0015 suggestion-mode warning).
* 5/5 live extraction tests pass against prod in 12.39s.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The PR looks cohesive and consistent across code, tests, and docs, with only minor user-facing wording and test-strength tweaks to consider. |
* docs/models/extraction.md: the conflict-rule sentence now reads "`extraction` cannot be combined with `livecrawl` or `livecrawl_formats`" — covers both deprecated parameter names so callers passing only `livecrawl_formats` learn about the conflict. * src/youdotcom/models/extraction.py: the module docstring's `crawl_timeout` note reworded — the SDK strips `crawl_timeout` from the body rather than failing fast, so the request succeeds without a 422 round-trip (mirrors the same fix already applied in `searchrequestbody.py`). * tests/test_live.py::test_highlights_mode: the loop previously guarded every assertion behind `if result.contents`, so a response that omitted `contents` for every result still passed without exercising the contract. Track observed `highlights` payloads and assert at least one result returned `contents.highlights` as a list, plus keep the per-result type check. Verified locally: * 137 unit tests pass (post-change on the test_live.py loop, the comment-only docstring change, and the extraction.md prose have no observable effect on the existing suite). *mypy clean across 81 source files. *pylint errors-only clean (only the pre-existing `pylintrc:1:0 E0015 suggestion-mode` warning). *5/5 live extraction tests pass against prod in 13.29s — the updated `test_highlights_mode` now exercises the highlights-mode contract end-to-end and confirms at least one result carries `contents.highlights`. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive across code, tests, and docs. The main remaining actionable item from this pass is a small but user-facing docs mismatch in the deprecated Search shim table that currently misstates |
…havior The Search sub-SDK is the deprecated you.search.unified(...) shim over POST /v1/search, so the crawl_timeout row should describe the POST behavior, not claim it is a GET no-op. Match the same wording used in docs/sdks/you/README.md and docs/models/searchrequestbody.md: "Maximum time in seconds to wait for page content when extraction.extraction_mode == "full_page" (or the deprecated livecrawl) is enabled. Must be between 1 and 60 seconds. Default is 10 seconds. Stripped from the body when extraction_mode == "highlights"." Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction change-set looks cohesive across code, tests, and docs. The main remaining issue is one new live test that does not yet assert any HTML/Markdown payload for full_page both-formats. |
…tract tests/test_live.py::TestLiveSearchExtraction::test_full_page_both_formats previously only asserted `res.results is not None`, so a response that omitted contents for every result would still pass without exercising extraction. Track observed (html, markdown) tuples across results and assert at least one carried html and/or markdown content (matching the test_highlights_mode fix). Verified locally: * 88 unit tests pass (extraction, search, param_normalization, shims). * mypy clean across 82 source files. * 5/5 live extraction tests pass against prod in 11.43s — the updated test now exercises the live contract for full_page=BOTH. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive and well-covered across code, tests, and docs. The main remaining issues are a docs example that is not currently runnable as written and a live test assertion that can be flaky when the API returns empty strings. |
* docs/models/extraction.md: wrap the Example Usage snippet in a
minimal `You(api_key_auth=os.getenv("YDC_API_KEY")) ... as you`
context manager so the snippet is copy/paste-runnable (no longer
throws NameError on `you`).
* tests/test_live.py::test_search_with_livecrawl_news: replaced the
truthy `markdown or html` check with `is not None` so the assertion
remains green when the API returns an empty content string (the
test's own comment already noted this possibility).
Verified locally:
* 5/5 live extraction tests pass against prod in 19.69s — the
updated livecrawl-news case now stays green on results that carry
an empty content field.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive across implementation, models, tests, and docs. The main remaining issue I found is a small but user-facing docstring mismatch around |
* docs/models/webresult.md line 11: `snippets` Type cell `List[*str*]` → `Optional[List[*str*]]` (matches the `Optional[List[str]]` annotation on both the TypedDict and the Pydantic model in `src/youdotcom/models/webresult.py`). * docs/models/webresult.md line 13: `page_age` Type cell `[date](...#date-objects)` → `[datetime](...#datetime-objects)` (source annotation is `Optional[datetime]`; `date` would silently drop time-of-day information for callers). * docs/models/newsresult.md line 10: same `[datetime]` Type cell fix on `page_age` (matches `Optional[datetime]` in `src/youdotcom/models/newsresult.py`). * examples/api-example-calls.py line 428 (P2 security): the interactive API-key prompt now uses `getpass.getpass(...)` so the secret is no longer echoed to the terminal / terminal recorders / shell-history candidates. Added `import getpass` near the top. Verified locally: * example parses as valid Python (`ast.parse`). * 88 unit tests still pass (no test surface touched). * mypy clean across 82 source files. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction addition looks cohesive and the new typed model contract is well-covered. Remaining issues are minor: one model docstring parity drift on |
* src/youdotcom/models/contents.py:30 — add a Pydantic field
docstring for `Contents.highlights` so it matches the
`ContentsTypedDict.highlights` r-string ("Query-relevant excerpts
extracted by the highlights mode.").
* tests/test_live.py:120 — tighten `TestLiveSearch::test_search_with_livecrawl_web`
and `test_search_with_livecrawl_news` to build a
`content_seen = [(html, markdown), ...]` list filtered through
`is not None`, then assert non-empty (`assert content_seen,
"Expected at least one …"`). Mirrors the pattern already used by
the `TestLiveSearchExtraction` tests.
Verified locally:
* 88 unit tests pass.
* mypy clean across 82 source files (incl. test_live.py).
* 7/7 live tests pass against prod in 27.93s — the rewritten
livecrawl_web + livecrawl_news tests now actively exercise the
live contents contract.
Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Four targeted additions, all principle-driven (no prescriptive grep recipes or shell commands — the AGENTS.md tone should help agents be comprehensive without forcing a specific workflow): * Extend "### Docstring sync between TypedDict and Pydantic" (A1) with the physical-location distinction: the TypedDict r-string sits below the type annotation, the Pydantic r-string sits below the `Field(..., description=...)` line, and they are *independent* help surfaces (one feeds OpenAPI/JSON schema, the other feeds IDE/help). The P2 review on `Contents.highlights` exposed this gap. * Extend "### Result-class pages" (A3) so the sweep not only re-asks the deprecated-name question but also diffs every Type cell in `docs/models/*result.md` against the matching source annotations, since `webresult.md` / `newsresult.md` rows for `snippets` / `page_age` slipped past the prior sweep. * Extend "### Type cells in field tables come from the annotation" (A5) with a list of broken mappings that have slipped past casual checks at least once: `Optional[X]` ↔ `X`, **`datetime` ↔ `date`** (don't conflate time-of-day), `bool` ↔ `int`, `str` ↔ `int`. The recent `page_age` review on `webresult.md` / `newsresult.md` exposed the `date` ↔ `datetime` trap. * New "## Interactive secrets use `getpass`" section: any `input(...)` prompt asking for an API key, token, or password in `examples/` / `scripts/` / CLI surfaces must use `getpass.getpass(...)` so the secret isn't echoed to the terminal. Prompted by the P2 security review on `examples/api-example-calls.py` line 428. No code changes; doc-only commit, in line with the prior two AGENTS.md updates (3bf49cc, 3eb79e8). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid encountered an error —— View job Droid is reviewing code and running a security check… |
Spotted in pre-release readiness pass: the test at tests/test_live.py::TestLiveSearch::test_search_with_livecrawl_all had only `assert res.results is not None`, so a response omitting contents for every result still passed non-actionably. The round-2 reviewer's pattern (`content_seen = [...]; assert content_seen`) should have caught this when we updated `test_search_with_livecrawl_web` and `test_search_with_livecrawl_news` — the ALL case was missed. Updated to build both `web_content_seen` and `news_content_seen` lists filtered through `is not None` on `results.web[].contents` / `results.news[].contents`, then assert each is non-empty. Verified locally: * 88 unit tests pass (extraction, search, param_normalization, shims). * 6/6 live tests pass against prod in 22.67s — the rewritten ALL case now actively exercises the live contract for both web and news contents. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction feature work looks cohesive across models, SDK wiring, and tests. The main remaining actionable issue I found is performance tests flaking under custom (real-server) runs due to missing |
tyler5673
changed the title
Round-4 review catch on tests/test_performance.py: every `with You(server_url=..., client=..., api_key_auth=...) as you:` block omitted `timeout_ms`, so the perf tests hitting slower real-server endpoints (`PERF_TEST_TARGET=custom`, including the new extraction full_page cases) inherited httpx's ~5s default and could flake or fail. Added `timeout_ms=90_000` to all 32 `You(...)` constructions in tests/test_performance.py per the reviewer's suggestion. Verified locally: * 88 unit tests pass. * Pre-existing mypy errors in tests/timing_client.py:144 (httpx Liskov override) are unchanged — not introduced by this commit. * mypy errors here are out-of-scope for this PR and should be filed separately. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction feature work looks cohesive and well-tested overall. The remaining high-confidence issues are documentation snippets in README/MIGRATION that are not copy-paste runnable because they omit |
Round-4 review caught two surface-sweep misses for the "Examples must be copy-paste runnable" rule: * README.md "Page content extraction" snippet: previously opened with `from youdotcom.models import ...` and called `you.search(...)` without ever defining `you`. Wrapped now in `import os; from youdotcom import You; ...; with You(..., timeout_ms=60_000) as you:` so copy/paste works. * MIGRATION.md "3.0.0 → 3.1.0" section: three snippets (Before/after, Highlights mode, No upgrade-time action required) all called `you.search(...)` standalone. Wrapped each in a fresh `import os; from youdotcom import You; with You(api_key_auth=..., timeout_ms=60_000) as you:` block. The DeprecationWarning snippet gets the improvement the reviewer specifically called out (the `with warnings.catch_warnings(...)` block now lives inside the `with You(...) as you:` block, so the `you` reference resolves). Verified locally: * 88 unit tests pass. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job Validated 4 actionable review comments, all still applicable on the current head SHA. Remaining items are adding |
Round-4 review caught three more surface-sweep misses for the "Examples with network calls pass `timeout_ms`" rule from AGENTS.md (and one related P2): * `docs/sdks/you/README.md`: 31 instances of the per-method `with You(api_key_auth=..., ) as you:` Example Usage block were missing `timeout_ms`. Added `timeout_ms=60_000` to all of them. * `docs/sdks/search/README.md`: same pattern, single instance (deprecated `Search` sub-SDK Example Usage block). * `examples/api-example-calls.py` line 437: the interactive script's top-of-`main()` `You(api_key_auth=api_key)` now passes `timeout_ms=60_000` so menu examples don't trip httpx's 5s default on slower `search`/`contents`/`research` calls. * `MIGRATION.md` DeprecationWarning snippet (P2): `deprecations[0].message == "..."` is misleading because the attribute is a `WarningMessage` instance, not a string. Comment now reads `str(deprecations[0].message) == "..."` so a reader can reliably validate what they caught. Verified locally: * `examples/api-example-calls.py` parses as valid Python. * 88 unit tests pass. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job Overall the extraction change-set looks cohesive; the one remaining actionable nit is closing the |
Round-5 review catch on examples/api-example-calls.py: `main()` assigned `you = You(api_key_auth=api_key, timeout_ms=60_000)` directly, so the SDK-owned httpx client only closed via GC finalizers on program exit. Wrap the menu + dispatch loop in `with You(api_key_auth=api_key, timeout_ms=60_000) as you:` so the context manager's __exit__ closes the transports deterministically when the user picks "Exit" or an exception unwinds. `global you` is preserved at the top of `main()` so the FUNCTIONS-dict callbacks (which read the module-global `you`) continue to see the client inside the `with` block; we wrap the entire menu in the outer `try`/`except` so SDK-init failures surface a friendly error before the `__enter__` runs. Verified locally: * examples/api-example-calls.py parses as valid Python. * 88 unit tests pass (extraction, search, param_normalization, shims). * mypy clean across 81 source files. * Module import + FUNCTIONS list intact (11 entries). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job The extraction feature work looks cohesive and well-tested, and the security pass did not surface issues. The only remaining actionable items from this review are small docs table type mismatches for the optional |
Round-5 review: `docs/models/webresult.md` line 13 and `docs/models/newsresult.md` line 10 list `page_age` with Type cell `[datetime](...)`, which doesn't reflect that the source annotation is `Optional[datetime]` (Pydantic) / `NotRequired[datetime]` (TypedDict). Match the optionality notation used elsewhere in the same tables. Updated both Type cells to `[Optional[datetime]](https://docs.python.org/3/library/datetime.html#datetime-objects)`. Column padding preserved with the surrounding rows. Verified locally: * 88 unit tests pass. Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job Validated 2/2 candidates and posted 2 review comments. Validated two remaining docs issues: the primary USAGE sync/async snippet and the README Async snippet still omit |
Round-5 review caught two more surface-sweep misses for the "Examples with network calls pass `timeout_ms`" rule: * `USAGE.md` primary sync + async snippets (the `<!-- Start SDK Example Usage [usage] -->` block, two snippets): construct `You(...)` without `timeout_ms` and then make real `search` / `search_async` calls. Added `timeout_ms=60_000` to both blocks. * `README.md` `## Async` snippet: `async with You(api_key_auth=...) as you:` made an actual `search_async` call without `timeout_ms`. Added `timeout_ms=60_000`. Verified locally: * 88 unit tests pass (unchanged scope). Co-authored-by: factory-droid[bot] <138933559+factory-droid[bot]@users.noreply.github.com>
Contributor
|
Droid finished @tyler5673's task —— View job Pass 2 (Validation) complete No Pass 1 candidate comments were provided to validate, so no inline comments were posted. I read the full diff and did not find additional high-confidence actionable issues requiring new comments. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds the typed
extractionparameter onPOST /v1/searchand deprecateslivecrawl/livecrawl_formats. Sister track to the matching change in the API, no breaking changes. Target release: 3.1.0.What changed
Extraction,ExtractionMode,ExtractionFormat,ExtractionHighlights,ExtractionFullPageand their TypedDicts) onyou.search()andyou.search_async(), plus theSearchShimforwarders. Strict models (extra="forbid") raiseValidationErrorlocally on unknown keys, so callers see the failure in their test run rather than at the API round-trip.highlights.max_tokensis bounded to[512, 8192].full_page.extraction_formatsdefaults to["markdown"]and accepts["html", "markdown"](or any subset).crawl_timeoutis stripped from the request body alongsideextraction_mode == "highlights", withUserWarningon explicit non-default values. The field still applies forextraction_mode == "full_page".extractioncombined withlivecrawlorlivecrawl_formatsraisesValueErrorlocally, before the deprecation warning. Order matters here—callers running with-W error::DeprecationWarningstill see theValueError.livecrawlandlivecrawl_formatskeep working withDeprecationWarning; removal is targeted for 4.0.0.results.web[].contents.highlightspopulates forextraction_mode == "highlights"(andsnippetsis omitted).results.web[].contents.htmland.markdownpopulate forextraction_mode == "full_page".tests/test_extraction.pycovers the model contract, strict validation, wire contract, conflict, plus-value rule, async, and shim forwarding. The deprecatedlivecrawlcases intests/test_live.pyare tightened to actually assert content is present (<x>_seen = [...]; assert <x>_seen). Threeextraction_*perf cases mirror thelivecrawl_*ones.## 3.0.0 → 3.1.0section inMIGRATION.md,docs/models/extraction.mdpage, theextractionrow in all four SDK-internal tables (docs/sdks/{you,search}/README.md,docs/models/searchrequestbody.md, plus a newhighlightsrow indocs/models/contents.md), theREADME.md"Page content extraction" section, theUSAGE.mdextraction block, a runnable example inexamples/api-example-calls.py, and theCHANGELOG.md3.1.0 entry.Contents.highlights,WebResult.snippetsandpage_age, andNewsResult.page_ageare nowOptional[...]-typed in both the row prose and the Type cell.examples/api-example-calls.pyAPI-key prompt now usesgetpassso the secret isn't echoed at the terminal.AGENTS.mdgained 145 lines across three commits codifying the surface-sweep pitfalls the review caught (TypedDict/Pydantic docstring sync, Type-cell discipline, live-test contract, andgetpassfor interactive secrets), so future migrations don't reopen the same threads.Verification
88 unit tests pass, mypy is clean across 82 source files, pylint errors-only is at 10.00/10, and 6/6 live tests pass against the prod API in ~22s.
scripts/check_drift.pyreportslivecrawlandlivecrawl_formatsas drift against the OpenAPI spec—that's intentional and documented inMIGRATION.md; removal is targeted for 4.0.0.