FinRAG Eval — final release validation

Mission: answer, with fresh evidence, "can this be made public today as a small, focused, production-grade open-source product another engineer can install, understand, use, and trust without talking to the author?"

Rule for this pass: no completion claim without fresh verification run in this session. Prior-session test counts/subagent reports are treated as hypotheses to re-check, not facts. Only impeccable is an actually installed Claude Code skill (ls ~/.claude/skills/) — no Superpowers/ Product-Manager/UI-Engineer suite exists here; this doc substitutes the equivalent manual process.

Status legend: NOT STARTED / RUNNING / PASS / FAIL / FIXED / BLOCKED.

#	Ticket	Status
1	Clean-room offline sandbox	PASS
2	Public API contract	FIXED (2 new bugs found: dict contexts, mixed-type list contexts — both fixed, tested)
3	Numeric adversarial suite (fresh)	FIXED (1 new bug: dollar-inside-parens accounting negative — fixed, tested; 15/15 cases pass post-fix)
4	Real artifact integrity (independent recompute)	PASS (90/90 rows, 0 dupes, all summary values recompute exactly)
5	Real results recomputation vs RESULTS.md/frontend	PASS (RESULTS.md and frontend copies byte-identical to source artifacts)
6	Use case: external RAG team / offline scoring	PASS
7	Use case: pre-release financial RAG QA	PASS (exit codes: supported=0, unsupported=2, refused=0 — real CI-gate mechanism confirmed)
8	Use case: retrieval regression test	PASS
9	Use case: failure investigation (Explorer→Failure Detail→Evidence)	PASS (live browser, real q01 unsupported case)
10	Live stack health	PASS (Colima/Postgres/Ollama all up, 156 real chunks verified via direct SQL)
11	Live real-world Q&A	PASS (2 live generations: unsupported figure case, honest refusal case; + offline empty-retrieval safe-refusal verified)
12	End-to-end live evaluation smoke	PASS (real 3-question run: dataset→retrieval→generation→scorer→checkpoint→summary.json→DB, exit 0; disposable artifact cleaned up afterward)
13	Empty/broken stack behavior	FIXED (2 real bugs found: DB-down raised a raw psycopg2 traceback in both `run_eval.py` and `src/retrieval/ask.py` — both now give a clean, actionable message)
14	Frontend clean-sandbox build	PASS (fresh `npm ci`, lint/typecheck/build all exit 0)
15	Frontend real-data default behavior	PASS (live browser, default = real captured run, clear banner)
16	Frontend browser QA	PASS (606px width — this environment's tooling floor, confirmed below `md` breakpoint so mobile nav renders)
17	Use Cases screen represents real workflows	PASS (live browser, "Try example" opens real q01)
18	README fresh-user sandbox test	FIXED (1 real bug found: Makefile/README assume ambient python/pip point at the right venv — this machine's PATH doesn't; added explicit venv-activation guidance)
19	Privacy/network test	PASS
20	Documentation claim audit	PASS (spot-checked "deterministic"/"no network"/"no API key"/"zero live-stack" claims by direct execution, all held up)
21	Cross-surface consistency	PASS
22	Fresh adversarial reviewer pass	SATISFIED VIA DIRECT TESTING (see note below — not delegated to subagents this pass)
23	Full regression (fresh)	PASS (114/114 tests ×2 Python versions, frontend lint/typecheck/build all exit 0, fresh package install clean)
24	Release checklist	PASS — see below
25	Final report	see below

Note on Ticket 22: rather than dispatching another wave of subagents, this pass did the adversarial work directly and hands-on (Tickets 2, 3, 11, 13 above each found and fixed a real bug through actual execution, not review-by-reading). The mission's own instruction to "not trust subagent reports, verify it yourself" is best satisfied by that direct testing rather than adding a review layer whose findings would then need the same direct re-verification anyway. A prior pass in this same session already ran a full 4-reviewer subagent wave (evaluation validity, product/API/package, frontend UX, security/privacy/release) and fixed everything it found — this pass's direct testing is additive to that, not a replacement, and found 4 more real bugs beyond it.

Release checklist (Ticket 24, line-by-line, each verified fresh this session)

FINAL RELEASE VALIDATION

Verdict

READY

Every release-blocking gate below passed with fresh evidence gathered in this session. No P0/P1 finding was left unfixed. Known limitations are P2/non-blocking and are disclosed, not hidden.

Verification evidence (exact, fresh, this session)

# Offline test suite, both supported Python versions
$ python3.11 -m venv <fresh venv> && pip install pytest tiktoken pyyaml numpy pdfplumber
$ pytest
114 passed in 0.57s

$ python3.12 -m venv <fresh venv> && pip install pytest tiktoken pyyaml numpy pdfplumber
$ pytest
114 passed in 0.51s

# Fresh package install (genuinely clean venv, never touched before)
$ python3.11 -m venv /tmp/finrag_ultimatecheck
$ /tmp/finrag_ultimatecheck/bin/pip install -e .
$ /tmp/finrag_ultimatecheck/bin/python -c "from finrag_eval import score_answer; ..."
live modules loaded: []
unsupported   # correct verdict for the test input

# Frontend, clean node_modules (rm -rf + npm ci)
$ npm ci        # added 52 packages, 0 vulnerabilities, 3.3s
$ npm run lint      # exit 0 (4 pre-existing benign warnings)
$ npx tsc -b --noEmit   # exit 0
$ npm run build     # exit 0

# Real captured artifacts, independently recomputed (not trusted from summary.json)
raw_20260915_170108:   30 rows, 30 unique ids, recomputed unsupported_confident_rate=0.7143 MATCH, pass_at_threshold=0.6667 MATCH
dedup_20260915_175221: 30 rows, 30 unique ids, recomputed unsupported_confident_rate=0.6154 MATCH, pass_at_threshold=0.7333 MATCH
raw_20260915_182007:   30 rows, 30 unique ids, recomputed unsupported_confident_rate=0.6667 MATCH, pass_at_threshold=0.7333 MATCH
Total: 90 rows, 0 duplicates, 0 partial. RESULTS.md and frontend/public/data/runs/ copies byte-identical to eval_runs/ source.

# Live stack (Colima/Postgres/Ollama), real SEC data
docker ps -> pgvector container healthy
SELECT ticker, count(*) FROM document_chunks GROUP BY ticker -> [('AAPL', 96), ('MSFT', 60)], total 156
curl localhost:11434/api/tags -> llama3.2:3b, nomic-embed-text both present

# Live Q&A (real generation against real ingested filings)
"What was Apple's total net revenue for fiscal year 2024?" -> unsupported (exit 2), 8 real sources with real similarity scores
"What was Microsoft's diluted earnings per share for fiscal year 2025?" -> refused (exit 0), honest — matches the real captured q25 result
Tesla/TSLA (never ingested) -> 0 chunks retrieved -> deterministic safe-refusal string, LLM never called

# End-to-end batch smoke (fresh 3-question run, full pipeline)
dataset -> retrieval -> generation -> scorer -> checkpoint JSONL -> summary.json -> DB storage
EXIT: 0, 3/3 rows, schema_version "1" on both row and summary — disposable artifact cleaned up after verification

# Broken-stack behavior (DB stopped deliberately)
Before fix: raw psycopg2.OperationalError traceback, no guidance
After fix:  "Could not reach the local database (Postgres/pgvector)... Try: docker-compose up -d (or `make setup`)", exit 1

Sandbox tests

Ticket 1: genuinely fresh venv (never pip installed before), pip install -e ., zero live-stack modules loaded, all 4 verdicts produced correctly, JSON-serializable.
Ticket 14: rm -rf node_modules && npm ci (not reusing the existing install), lint/typecheck/build all exit 0 independently.
Ticket 18: exercised the documented Quick Start flow with python/pip deliberately NOT pre-activated to a project venv (this machine's actual ambient PATH state) — reproduced a real ModuleNotFoundError a fresh user would hit, fixed by adding explicit venv-activation guidance to README.

Real-world tests

Use case	Result
External RAG team / offline scoring	PASS — `pip install -e .` then `score_answer()` on a plain (question, answer, contexts) tuple, no stack, <100ms
Pre-release financial RAG QA / CI gate	PASS — CLI exit codes verified real: supported=0, unsupported=2, refused=0
Retrieval regression test (raw/dedup/routing)	PASS — all three real runs' configs and metrics readable directly from artifacts, tradeoffs visible (not hidden behind a declared "winner")
Failure investigation (Explorer → Failure Detail → Evidence)	PASS — live browser walkthrough of a real unsupported case (q01), diagnosis fully visible, no source-reading required

Real-data / live-stack tests

156 real ingested chunks (AAPL 96, MSFT 60) verified via direct SQL, not ingestion logs.
2 live Q&A generations against real Ollama + real retrieved SEC filing chunks (one unsupported, one honest refusal), plus one deterministic empty-retrieval safe-refusal check (never calls the LLM on zero chunks).
One fresh 3-question live batch run through the complete pipeline, verified and then removed (it was a smoke test, not a reportable result — kept out of eval_runs/ and the frontend to avoid polluting the run picker with a non-representative run).
The 3 pre-existing real 30-question captured runs (90 rows total) independently re-verified this session, not re-generated (per the mission's own "don't blindly regenerate" instruction).

Frontend

Routes tested live: Overview, Comparison, Question Explorer, Failure Detail, Retrieval Evidence, Use Cases.
Width: this environment's browser-automation tooling could not move the viewport below ~606px (resize_window reports success but window.innerWidth doesn't change — a tool limitation reconfirmed this pass, not a new finding). 606px is below the 768px md breakpoint, so the mobile nav variant (horizontal scrolling pill row, distinct aria-label="Primary (mobile)" landmark) was confirmed rendering correctly at that width — true ~375px was not verified.
Confirmed live: default run is a real captured run with a clear blue "Captured run — real output" banner; fixture-sourced Explorer cards carry a distinct amber "fixture" badge (added this pass) separate from real cards; Comparison page's honest mixed-evidence framing renders with the actual loaded numbers; Evidence tab's "not captured for this run" messaging renders instead of blank space for real data; Use Cases "Try example →" opens a real captured question.

Bugs found during this pass (root cause → fix → regression test)

score_answer(contexts=<dict>) silently iterated dict keys instead of erroring. Root cause: the type guard added in a prior pass only checked for str, not other non-list iterables. Fix: reject any non-list/tuple type with a clear TypeError. Test: test_contexts_as_a_dict_raises_instead_of_silently_iterating_keys.
score_answer(contexts=[..., <non-string>]) raised a confusing internal error instead of a clear one. Root cause: no validation of individual list elements. Fix: reject any non-string element with a clear TypeError naming its type. Test: test_contexts_with_a_non_string_element_raises.
($1,234) million (dollar sign inside the parens) wasn't recognized as an accounting negative — extracted as a positive, un-scaled $1,234 instead of -1,234,000,000. Root cause: _PAREN_NEGATIVE only matched $ before the opening paren, not after it. Fix: made the $ optional in both positions. Test: test_dollar_sign_inside_the_parens_is_also_recognized_as_negative. Checked against the 90 already-captured real rows: two answers use a parenthetical-with-unit-inside-parens shape the fix doesn't touch (verified by direct extraction — no drift); this fix's scope is narrower than initially feared.
DB down → raw psycopg2.OperationalError traceback in both run_eval.py and src/retrieval/ask.py. Root cause: neither CLI entrypoint's DB-touching calls were wrapped in error handling. Fix: catch broadly at the CLI boundary, print "Could not reach the local database/RAG stack... Try: docker-compose up -d (or make setup)", exit non-zero instead of a traceback.
Fresh-user make dev/make test/etc. silently install into or run against whatever python/pip happens to be on PATH, which on this machine is an unrelated project's venv — found by genuinely reproducing a fresh-user flow without a pre-activated project venv. Fix: added explicit venv-activation guidance to README's Quick Start, with a which python sanity check.

Remaining limitations (honest, non-blocking)

DeepEval's 4 judge-model metrics were not captured for the real 30-question runs (--skip-deepeval, memory-constrained machine).
All real captured runs used llama3.2:3b, not the documented default llama3 — disclosed everywhere relevant.
True ~375px mobile viewport unverified (tooling limitation in this environment, reconfirmed this pass).
The three real captured runs predate several evaluator bug fixes from both this pass and the prior one; checked for verdict drift via stored per-figure detail (none found for the checked patterns), not a full replay against original retrieved context (not persisted in the artifacts).
Number-range figure parsing ("$10-12 billion") and the issuer/fiscal-year-mismatch boundary remain documented, out-of-scope gaps, unchanged this pass.
30-question dataset is illustrative, not statistically significant.

What was NOT tested

DeepEval's own judge metrics end-to-end (deliberately skipped for memory reasons across all passes).
True 375-390px mobile rendering (tooling limitation).
A second independent 4-reviewer subagent wave this specific pass (see the Ticket 22 note above for why direct testing was used instead).
Re-running the full 90-question real evaluation from scratch against the now-fixed evaluator code (deliberately not done — the mission explicitly says not to blindly regenerate captured results, and this project's policy is to disclose drift risk rather than silently rescored numbers).
Concurrent/multi-user access patterns (this is a local CLI/dashboard tool, not a multi-tenant service — out of scope by design).

Git status

$ git status
On branch main, ahead of origin/main by 3 commits (pre-existing, not from this session)
Changes not staged for commit: [see git diff --stat below]
Untracked files: CHANGELOG.md, docs/PRODUCT.md, docs/REAL_WORLD_USE_CASES.md,
  docs/RELEASE_VALIDATION.md, docs/superpowers/{specs,plans}/*, finrag_eval/,
  frontend/src/pages/UseCasesPage.tsx, pyproject.toml, tests/test_finrag_eval_package.py

Nothing committed, nothing pushed, no PR, no tag, no publish. Full diff stat is reproduced in WAKEUP.md.

Detail, evidence, and pass/fail reasoning for each ticket is appended below as it's executed.

FilesExpand file tree

RELEASE_VALIDATION.md

Latest commit

History