fix(fetch_fred): warm S&P 500 covariates and skip delisted gold series#167
Merged
Conversation
fetch_fred.py only cached the food-price FRED series, so the S&P 500 implementation's FRED covariates were never warmed even though fetch_sp500_market.py defers to it for them. Import sp500's FRED_PREFETCH_REGISTRY (which already documented itself as the source for this script) and fetch the union of both experiments' series, carrying a per-series frequency so daily series register as D rather than a hardcoded MS. Both London gold fixing series (GOLDAMGBD228NLBM/GOLDPMGBD228NLBM) are permanently discontinued by FRED with no daily USD replacement. Mark them known-unavailable so a clean run reports 0 failures, and document the discontinuation in sp500's registry where the ids live (gold already degrades to absent at runtime via the first-available fallback). Regenerate the concierge context mirrors for the two changed files. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Unified fix for FRED-related items.
fetch_fred.pyonly warmed the food-price series, so the S&P 500 implementation's FRED covariates were never cached — even thoughfetch_sp500_market.pyexplicitly defers to it for them.FRED_PREFETCH_REGISTRYfromsp500_forecasting.data(whose own comment already declared it was "forscripts/fetch_fred.pyregistration" — it was just never consumed) and fetch the union of the food-price and S&P 500 series. FRED ids are unique across the two sets, so no de-duplication needed.frequencyfield, so daily S&P 500 series (DGS10,DGS2,DFF,DCOILWTICO,DTWEXBGS) register asDinstead of the old hardcodedMS. The S&P 500 covariate builders handle alignment themselves; this script only warms the raw parquet cache.GOLDAMGBD228NLBM,GOLDPMGBD228NLBM) are permanently discontinued by FRED (HTTP 400 "series does not exist") with no equivalent daily USD replacement. Marked them known-unavailable viaKNOWN_UNAVAILABLE_FRED_IDSso a clean run reports13 registered (0 failed, 2 skipped)instead of two spurious[failed]lines. Documented the discontinuation in sp500's registry where the ids live. No modeling behavior change — gold stays in the covariate list and still degrades to absent at runtime via the existing first-available fallback (strict_covariates=False).Verification
uv run python scripts/fetch_fred.py→Registered 13 series (0 failed, 2 skipped).ruff checkpasses on both source files.pytest implementations/tests/sp500_forecasting/test_data.py aieng-forecasting/tests/aieng/forecasting/data/test_fred_adapter.py→ 7 passed.Out of scope (deliberately)
catalog-summary.yamlfrom earlier merged PRs. That mirror set isn't regenerated per-PR, so this PR keeps only the two mirrors it actually changed — a full refresh belongs in a separate maintenance PR.GVZCLSis volatility, not price;IR14270/ID7108are monthly indices that don't fit the daily log-return builder). A real substitute (e.g. GLD via yfinance) would be a modeling change, not a quick win.🤖 Generated with Claude Code