Skip to content

PRODUCT.md

Latest commit

 

History

History
175 lines (140 loc) · 7.48 KB

PRODUCT.md

File metadata and controls

175 lines (140 loc) · 7.48 KB
Raw

FinRAG Eval — Product

What is FinRAG Eval?

A focused evaluation toolkit for one specific, dangerous RAG failure mode: a financial RAG system confidently asserting a dollar or percent figure that the evidence it actually retrieved does not support. It answers one question, deterministically and offline:

Did the model assert a financial number that the retrieved evidence does not support?

It ships as three layers that work independently:

  1. An importable Python evaluator (finrag_eval / src/eval/) — no Docker, Ollama, database, or LLM required. This is the product's core.
  2. A reference RAG pipeline (src/ingestion/, src/retrieval/) over real SEC EDGAR filings, used to exercise and validate the evaluator against real financial documents, and to produce the CLI workflows (make ask, make eval) documented in docs/USE_CASES.md.
  3. A local dashboard (frontend/) for inspecting evaluation runs — reads the JSON the evaluator produces, never re-implements scoring logic itself.

Who uses it?

See docs/REAL_WORLD_USE_CASES.md for the full write-up. In short:

  • A financial RAG engineer doing pre-release QA on their own system's answers.
  • A team changing chunking/retrieval config who wants to know whether a change that improves generic retrieval metrics quietly makes numeric hallucination worse.
  • An engineer debugging one specific bad answer, trying to tell whether the failure came from retrieval, table extraction, unit normalization, or generation.
  • A RAG-eval framework maintainer or researcher interested in the overlapping-chunk precision/recall fix this project contributed upstream to DeepEval (PR #2743).

What exact problem does it solve?

Standard RAG eval metrics (contextual precision/recall, answer relevancy) don't distinguish "the model refused" from "the model gave a precise, confident, wrong number." In financial documents specifically, a wrong number is far more dangerous than an honest refusal — and metrics built for general RAG don't catch it, because a confidently-wrong figure can still score well on relevance and fluency. FinRAG Eval separates two questions that are easy to accidentally conflate (and were, in fact, conflated in an earlier version of this project's own code — see PLAN.md/RESULTS.md for that history):

  • Context grounding — is the asserted figure actually present in the retrieved evidence?
  • Ground-truth correctness — does it match the expected answer?

A wrong number that happens to match a gold answer must never read as "grounded" just because it's correct; a correct number that the system never actually retrieved must never read as "grounded" just because it's right. Both are checked independently.

Product interfaces

Python API

from finrag_eval import score_answer

result = score_answer(
    question="What was Apple's R&D spend?",
    answer="Apple's R&D expense was $14.2 billion in fiscal 2024.",
    contexts=["Research and development expense was approximately $31.4 billion."],
)
result["verdict"]  # "unsupported"

No Docker, Ollama, or network. Optional ground_truth and document_type arguments — see README.md's "Python API" section.

contexts must be a list[str] — one string per retrieved chunk, not one big string. Passing a bare string raises a clear TypeError rather than silently corrupting the context (found by adversarial review: a bare string used to get joined character-by-character downstream, which could flip the verdict with no error at all). result["metrics"] always has numeric_grounding; it additionally gains section_aware_precision/ section_aware_recall only when both ground_truth and contexts are non-empty — the key set is additive, never a fixed shape, so don't assume every call returns the same set of metric keys.

CLI

python -m src.eval.score_answer --question "..." --answer "..." --context "..." [--ground-truth "..."] [--document-type ...]

Prints stable JSON. Exit code 2 on unsupported, 0 otherwise — scriptable as a CI gate (echo $?).

Batch evaluation

make eval

Runs the full dataset against the live RAG pipeline, writes eval_runs/<run_id>.jsonl (one row per question) and eval_runs/<run_id>.summary.json (aggregate rates, including the headline unsupported_confident_rate) — see docs/USE_CASES.md.

Dashboard

make ui

Reads eval_runs/ artifacts (or bundled fixture data when none exist yet) — see frontend/README.md.

What it does NOT try to be

  • Not a general-purpose RAG evaluation framework — it does one thing (numeric grounding for financial figures) well, and leaves contextual precision/recall/faithfulness to DeepEval, which it uses alongside its own metrics rather than replacing.
  • Not an LLM judge — score_answer() is deterministic regex/unit- normalization heuristics, documented as such (see src/eval/hallucination.py's module docstring), not a model call. This is why it needs no API key and produces identical output on repeat runs.
  • Not a production RAG pipeline to adopt wholesale — src/ingestion/ and src/retrieval/ exist to exercise and validate the evaluator against real filings, not as a system meant to be deployed as-is. A team with its own RAG pipeline should only need the Python API.
  • Not a hosted service — everything runs locally; there is no server component, telemetry, or account system.

How another team integrates it

The smallest integration is a function call: pipe your own system's (question, answer, retrieved_contexts) through score_answer() and act on result["verdict"]. Nothing about the RAG pipeline, database, or dashboard in this repo is required for that. See pyproject.toml for the importable package; requirements.txt's live-stack dependencies (ollama, psycopg2-binary, supabase, deepeval) are not needed for this integration path.

What is proven today

  • Context-grounding vs. ground-truth-correctness independence: enforced by code structure and covered by regression tests (see tests/test_hallucination.py, tests/test_score_answer.py).
  • Unit/scale normalization ($31.4B == $31.4 billion == 31,400 million), negative-sign and parenthetical-accounting-negative handling, bare-calendar-year filtering, refusal/no-figure classification — all unit-tested, offline, deterministic (see RESULTS.md's "Verified now" table for the exact test names).
  • The reference RAG pipeline genuinely runs end-to-end against real SEC EDGAR filings on local infrastructure (Postgres/pgvector via Docker, Ollama for embeddings and generation) — see RESULTS.md for whether a captured live run exists as of the current state of this repo.
  • The DeepEval upstream contribution (issue #2594, merged fix PR #2743) is real — independently verified against the live DeepEval GitHub repo during a prior review pass (see PLAN.md).

What is not yet proven

  • Statistical significance: the evaluation dataset is 30 questions across two issuers — illustrative of the failure mode, not a large-scale benchmark.
  • Broad filing-format coverage: HTML section-splitting is regex-based (Item N. markers) and has only been exercised against Apple and Microsoft 10-Ks; non-standard filing formatting could break it.
  • Scale: this has not been run against a large question set, multiple issuers beyond AAPL/MSFT, or under production load — see RESULTS.md's Known Gaps for the current, honest state.