Skip to content

fix(fetch_fred): warm S&P 500 covariates and skip delisted gold series#167

Merged
ethancjackson merged 1 commit into
mainfrom
fix-fred-script
Jul 7, 2026
Merged

fix(fetch_fred): warm S&P 500 covariates and skip delisted gold series#167
ethancjackson merged 1 commit into
mainfrom
fix-fred-script

Conversation

@ethancjackson

Copy link
Copy Markdown
Collaborator

Summary

Unified fix for FRED-related items. fetch_fred.py only warmed the food-price series, so the S&P 500 implementation's FRED covariates were never cached — even though fetch_sp500_market.py explicitly defers to it for them.

  • Warm both experiments' covariates. Import FRED_PREFETCH_REGISTRY from sp500_forecasting.data (whose own comment already declared it was "for scripts/fetch_fred.py registration" — it was just never consumed) and fetch the union of the food-price and S&P 500 series. FRED ids are unique across the two sets, so no de-duplication needed.
  • Per-series frequency. Catalogue entries now carry a frequency field, so daily S&P 500 series (DGS10, DGS2, DFF, DCOILWTICO, DTWEXBGS) register as D instead of the old hardcoded MS. The S&P 500 covariate builders handle alignment themselves; this script only warms the raw parquet cache.
  • Delisted gold series. Both London gold fixing series (GOLDAMGBD228NLBM, GOLDPMGBD228NLBM) are permanently discontinued by FRED (HTTP 400 "series does not exist") with no equivalent daily USD replacement. Marked them known-unavailable via KNOWN_UNAVAILABLE_FRED_IDS so a clean run reports 13 registered (0 failed, 2 skipped) instead of two spurious [failed] lines. Documented the discontinuation in sp500's registry where the ids live. No modeling behavior change — gold stays in the covariate list and still degrades to absent at runtime via the existing first-available fallback (strict_covariates=False).
  • Regenerated the concierge-context mirrors for the two changed source files.

Verification

  • uv run python scripts/fetch_fred.pyRegistered 13 series (0 failed, 2 skipped).
  • ruff check passes on both source files.
  • pytest implementations/tests/sp500_forecasting/test_data.py aieng-forecasting/tests/aieng/forecasting/data/test_fred_adapter.py → 7 passed.

Out of scope (deliberately)

  • Regenerating the concierge context surfaced ~25 unrelated stale mirrors + a volatile catalog-summary.yaml from earlier merged PRs. That mirror set isn't regenerated per-PR, so this PR keeps only the two mirrors it actually changed — a full refresh belongs in a separate maintenance PR.
  • No daily FRED gold replacement exists (GVZCLS is volatility, not price; IR14270/ID7108 are monthly indices that don't fit the daily log-return builder). A real substitute (e.g. GLD via yfinance) would be a modeling change, not a quick win.

🤖 Generated with Claude Code

fetch_fred.py only cached the food-price FRED series, so the S&P 500
implementation's FRED covariates were never warmed even though
fetch_sp500_market.py defers to it for them. Import sp500's
FRED_PREFETCH_REGISTRY (which already documented itself as the source for
this script) and fetch the union of both experiments' series, carrying a
per-series frequency so daily series register as D rather than a hardcoded
MS.

Both London gold fixing series (GOLDAMGBD228NLBM/GOLDPMGBD228NLBM) are
permanently discontinued by FRED with no daily USD replacement. Mark them
known-unavailable so a clean run reports 0 failures, and document the
discontinuation in sp500's registry where the ids live (gold already
degrades to absent at runtime via the first-available fallback).

Regenerate the concierge context mirrors for the two changed files.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@ethancjackson ethancjackson merged commit 086a71e into main Jul 7, 2026
2 checks passed
@ethancjackson ethancjackson deleted the fix-fred-script branch July 7, 2026 18:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant