Skip to content

feat(web-search): add Tavily provider - #237

Open
savvadesogle wants to merge 1 commit into
AtomicBot-ai:mainfrom
savvadesogle:feat/web-search-tavily
Open

feat(web-search): add Tavily provider#237
savvadesogle wants to merge 1 commit into
AtomicBot-ai:mainfrom
savvadesogle:feat/web-search-tavily

Conversation

@savvadesogle

Copy link
Copy Markdown

Summary

Adds Tavily as a first-class os.web.search provider alongside the existing duckduckgo, searxng, exa and brave providers.

Motivation

os.web.search was unreliable on proxied / cloud-egress setups:

  1. Exa without an API key. When EXA_API_KEY is unset, Exa falls back to its keyless MCP tier (https://mcp.exa.ai/mcp), which returns 502 / 403 / 429 under load. The project already tracks this degradation mode in Keyless Exa search returns HTTP 429 continuously and degrades to a weaker provider silently #179 (a GAIA campaign logged 1341 Exa returned HTTP 429 failures).
  2. DuckDuckGo anti-bot wall. The keyless fallback (html.duckduckgo.com) intermittently serves an anti-bot challenge (HTTP 202 + challenge-form / anomaly-modal). This happens consistently from datacenter / VPN / proxy egress IPs, so the fallback is not a reliable backstop in proxied environments (verified 5/5 attempts, both direct and through a proxy).
  3. Tavily is a stable, developer-friendly search API that requires an API key but behaves predictably and returns structured results[].

Changes

File Change
src/tools/os/web-search/providers/tavily-provider.ts New. createTavilyProvider() POSTs to https://api.tavily.com/search with { api_key, query, max_results }, throws on HTTP >= 400, and maps the results[] payload (title, url, content, publishedDate) to WebSearchResult[]. parseTavilyJson is exported for tests.
src/tools/os/web-search/web-search-provider.ts Add "tavily" to the WebSearchProviderName union.
src/tools/os/web-search/providers/provider-registry.ts Register createTavilyProvider under the "tavily" case in resolveProviderByName.
src/tools/os/web-search/providers/search-orchestrator.ts isProviderUsable("tavily") requires TAVILY_API_KEY to be present in the environment (mirrors the brave gate), so a keyless Tavily is skipped rather than failing every search.
src/config/config-schema.ts Add web.search.tavily.apiKeyEnv to WebSearchConfig, its default "TAVILY_API_KEY" in USER_CONFIG_DEFAULTS, parsing of the provider name union, and the webSearchTavily extraction in parseUserConfigFile.
src/tools/os/web-search/tool/warn-missing-search-key.ts Treat Tavily as a keyed provider (add to KEYED_PROVIDERS and resolve its apiKeyEnv), so a missing key produces the same one-line startup warning as Exa/Brave.

Usage

// <stateDir>/config.json
"web": {
  "search": {
    "provider": "tavily",
    "fallback": []
  }
}
# <stateDir>/.env
TAVILY_API_KEY=tvly-...

Verification

  • npm run lint — clean.
  • npm test — full suite: 20 failed / 5939 passed. This is identical to the pre-existing baseline on main (all 20 failures are environment-specific Windows tests: clear-stale-chrome-locks, fs-approval-scope, open-terminal-window, send-message-concurrency, logo-art.generated, bootstrap, stream-cli-completion, skill-script-runner, skill-tools). No new failures are introduced.
  • Manual: os.web.search returns Tavily results through a proxy.

Notes

  • Tavily requests go through searchHttp, which shells out to curl. curl reads HTTP_PROXY / HTTPS_PROXY / NO_PROXY from the environment itself, so Tavily works behind a proxy without any extra wiring.
  • Follows the exact shape of the existing brave / searxng providers, so the orchestrator, cache, retry policy, and error surface behave the same.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant