feat: frontmatter validation, per-model reporting, new verify checks, and auto-repair - #5
Open
MatMatt wants to merge 42 commits into
Open
feat: frontmatter validation, per-model reporting, new verify checks, and auto-repair#5MatMatt wants to merge 42 commits into
MatMatt wants to merge 42 commits into
Conversation
…nnet path - call_openrouter streams output across calls when truncated, trimming seams to block boundaries (never mid-table/fence) with dedup; PDF cached across calls - text_coverage: covered/reworded/missing classifier + letter-digit split so restructured sentences aren't false gaps - detect: skip malformed bboxes instead of crashing - gemini-2.5-flash is the only convert model; drop legacy claude max-token caps
phase 2.5b insertion cost was discarded and hard-coded to 0; thread real usage cost up through run_phase25 into phase_cost['rescue'] and cost_usd.
Reconcile the two rescue lines: keep develop's output-continuation, figure rescue + cost, reworded-aware text coverage, sonnet removal; take main's postfix repair pass (--postfix/--improve), 5 new verify checks, single-quote YAML frontmatter, and template-merge post-processing. Conflicts: - app.py: drop page-splitting (_split_convert) — continuation supersedes it; keep both figure-rescue (2.5) and postfix (4.5); take main's batch try/finally - pass2.py/resolve.py: keep develop's --template qmd gating, feed the real path to main's _apply_template_frontmatter - text_coverage.py: strip postfix section (main) then letter-digit split (develop)
Feed each continuation a bounded output tail + resume anchor instead of the whole output-so-far, so per-call input stays flat and page count is no longer capped by the context window (~500 -> ~1200 page ceiling, lower cost). Anchor = deterministic last-heading/caption (regex-extracted, no model trust) + a best-effort <!-- pdf2md:cursor next/page --> marker the model emits, which is parsed for the forward hint and stripped so it never lands in the document. Raise _MAX_CONTINUATIONS 8->40 (each call is now cheap; the real wall is the PDF fitting context).
Live probe over 6 docs / 24 truncated chunks: the cursor fired 0 times — truncation always cut it off before the end-marker, so it never served as a forward anchor. The deterministic tail + last-heading anchor gave clean seams (no dup, coherent heading order) on its own. Remove the directive, parser, and threading; less code, cleaner conversion prompt, no measured loss.
Warn upfront when a doc's estimated text footprint (~text_chars/4 tokens) approaches the model's context window — the binding wall for tail-only continuation. Uses estimate_file's existing signals (no API call). Warn only, no split fallback; verify's text_coverage still catches any truncation.
Tail alone lost progress awareness: the 131-page doc stopped at p106/131 and reported complete. Feed each continuation the heading outline + page count + a no-early-stop instruction. Retest: p131/131.
text_coverage now reports strict (in-place) and effective (counting the recovery appendix); convert_one records before/after and writes postfix_report.md. Makes the real impact of post-conversion fixes visible instead of a flat number.
The converter splits/merges tables, so scoring each source table against a single best-matching .qmd table counted present content as missing. Match against a bounded union of .qmd tables instead; report content and single-table aligned. On the 131-page manual: 90.9% -> 97.5%, thin-table findings 7 -> 1, warn -> ok.
Table rows re-segmented into cells failed the prose local-window check and read as missing, though table_coverage already scores them. Skip table regions when scoring prose. On the 131-page manual: 96.5% -> 97.8%, missing 88 -> 43.
Recover prose per page (not a bullet digest) and put it back in flow instead of an end-of-document appendix. Positions are resolved against the original text and applied back-to-front so inserts can't anchor onto each other; pages with no trustworthy anchor are appended rather than guessed. 131-page manual: text 97.8% -> 98.6%, no duplication introduced.
Convert the gap page together with its neighbours in one call, then anchor on the model's own rendering of the neighbour: the .qmd is model-rendered, so raw PDF text rarely matches it. Anchors must be unique in the .qmd — headings recur in the table of contents and chrome on every page, which pinned inserts to the top. 131-page manual: text 97.8% -> 99.2%, 4/5 pages placed in the right section.
Bare numbers and units recur across tables, so a table that never converted could collect them from unrelated tables and score as covered. Weight tokens by inverse document frequency, with absent tokens weighted highest. 131-page manual: an inflated 97.5% becomes an honest 96.4% and surfaces 2 genuinely thin tables.
Re-emit source tables whose data never reached the .qmd, straight from find_tables cells — exact values, no LLM. Detection uses each table's distinctive values, since shared boilerplate is supplied by sibling tables and hid 13 pages of missing tables behind an 85% token-bag score. Degenerate and partially-present tables are skipped. 131-page manual: table coverage 96.4% -> 98.9%.
The exact grid from find_tables preserves values but loses structure: filler columns, split cells, no caption. Re-convert the table region with the LLM for structure, then verify every extracted cell value survives; drop the rendering and keep the plain grid if any value is lost. 131-page manual: table 98.9% -> 99.3% (0 thin), text 99.2% -> 99.4% as captions return. 15 of 16 used the rendering, 1 hit the fallback.
The PDF renders bullets as a literal 'o' and carries its own section numbering; the converter emits markdown bullets and lets Quarto number headings. Comparing those as words reported identical content as missing. Strip both before scoring. 131-page manual: text 99.4% -> 99.8%, missing 11 -> 3.
A lone page-number block is kept out of chrome_keys because its digits differ per page, but it was then counted as BODY — pushing the no-cross guard below the footer so the whole bottom band was discarded. Footers were therefore neither stripped before conversion nor excluded from verification, and find_tables reported one 'table' per page footer. Treat page-number blocks as neutral, and skip chrome regions when extracting source tables. hrsi-snow-atbd: 43 source tables -> 14, table coverage 87.8% -> 97.1%; the other five documents unchanged.
A PDF keeps the href in a link annotation, not the page text, so the converter never sees it — across 6 documents not one URL reached the .qmd that was not already visible as text. Re-attach them deterministically: inline where the anchor text occurs exactly once, otherwise in a Source links list so no citation is lost. Four of six documents go to 100%; the worst goes 41/97 -> 94/97.
- Report: consolidate before/after into one verify_report.md (drop postfix_report.md); header shows conversion+repair cost split - Figures: adopt model-referenced images detection missed; unresolved tokens render as markers, not broken <img> tags - Footnotes: prompt [^n] syntax; ignore unit/exponent superscripts - Headings: restore dropped source-outline headings (deterministic postfix) - Math: tighten equation detection (drop error-bar/threshold/set-notation false positives) - Text coverage: join wrapped PDF lines before matching; skip TOC leaders - Detect: abort loudly when all pages fail; keep detections.json for reruns - Missing-text repair: per-page LLM recovery, no page cap, chrome stripped
Repair now anchors recovered content at its true location instead of appending to the end, and declines rather than guessing when there is no unique anchor. - Missing text: anchor each gap on the surviving sentence before it, fill a placeholder marker; a heavily collapsed region (e.g. a references section) recovers as one region gap; grow the anchor until unique - Tables: recover in place at the caption anchor, and run after text recovery so a table in a collapsed region can anchor on the prose just restored - Anchor context and the coverage checks use the chrome-stripped working.pdf the converter actually saw, not the raw source - Estimate recalibrated to real gemini-flash costs; self-calibration reads result.json so it survives cleanup - API key checked up front with an actionable message instead of a mid-run 401 - Verify report shows one terse problem line per issue, with the cost split
- Merge postfix + review into single iterative repair loop - --postfix N (default 3 iterations), --no-postfix to disable - --review flag removed (vision repair built into loop) - 1:1 conversion by default, --strip-chrome for optional chrome removal - repair_report.md with per-iteration summary table - 439 tests passing
Append 3 new pages (sections 10-13) to pdf2md_edgecases.pdf covering 6 previously untested scenarios: - Clickable hyperlink annotations (PDF link annotations) - \DATA / <sub> / <sup> / <br> raw patterns (_sanitise_for_typst) - 130171 display math + ambiguous notation (math false-positive guards) - Running header variants per page (chrome.py detection) - Page-N-of-M footer pattern (lint.py chrome-leak guard) - Nested image paths + doubled URL schemes (lint.py) Regeneration: python3 src/pdf2md/tests/gen_edgecase_supplement.py
Remove 1990-2018_PUM_v1_short.pdf (real CLMS PUM snippet) and port the 2 pagefilter regression tests to pdf2md_edgecases.pdf. The synthetic edgecases PDF covers both scenarios: - recall: pages 9 (aerial photo raster) and 11 (vector figure) detected - skip rate: 7 text-only pages correctly skipped
- --main-llm <slug> replaces --model (--model kept as deprecated alias) - --figure-llm <slug> for Phase 1 figure detection - pdf2md --setup now fetches model catalogue from OpenRouter API and presents a numbered pick list (multimodal first, sorted by price) for both conversion LLM and figure-detection LLM - Config saved to ~/.pdf2md/config.json under 'model' and 'figure_llm'
# Conflicts: # src/pdf2md/app_cli.py # src/pdf2md/postfix.py
…lint gate
Root cause of Quarto/Typst emitting #box(image(""))
instead of #box(image("img.png")): the .qmd's markdown image reference
was itself nested inside another image wrapper ()), a
known artifact from phase2/rescue/tablefix. Quarto's typst writer does
not detect this hazard — it just stringifies the whole nested markdown
as the literal image path.
lint.py already had _unwrap_nested_images() to fix exactly this, but it
only ran in Phase 4.6 (the *final* emit-time gate), which executes AFTER
render (Phase 3), verify (Phase 4), and postfix (Phase 4.5). So the
render always saw the un-sanitized .qmd.
Fix: run the same deterministic sanitize pass right before render
(Phase 2.9), in addition to keeping the final Phase 4.6 gate (postfix
can still reintroduce the hazard after render, so both passes are
needed).
Reproduced by isolating the pattern with:
quarto render test.qmd --to typst --metadata-file src/pdf2md/render_assets/_typst.yml -M keep-typ:true
Confirmed against /tmp/eea-out real-run output, which had this exact
artifact for 2 figures (img-1174619926e23b52307165b105d52675.png,
img-ed92374ebbd81963432e40b81edd8e25.png) that had been nested-image
wrapped upstream.
… and auto-repair - Add YAML validity check to normalize_frontmatter (invalid YAML blocks are discarded and replaced with a proper skeleton) - Add title: '' and subtitle: '' fallbacks for PR-gate compliance - Fix date fallback to always emit today's date as last resort - Add per-phase model reporting (cover/detect/convert/repair) to verify report - Rename --cover-model to --cover-llm for consistency with --main-llm/--figure-llm - Add 3 new verify checks: yaml_boundary (frontmatter leaks), duplication (near-duplicate paragraphs), artifacts (Office ref errors) - Add artifact auto-repair to the deterministic postfix pass - Fix duplication check to exclude tbl-caption boilerplate (false positives)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Improvements to pdf2md's frontmatter handling, verify checks, and model reporting.
Frontmatter fixes
---but fills with body content) are now detected and replaced with a proper skeletontitle: ''/subtitle: ''when the converter supplies neitherModel reporting
--cover-model→--cover-llmfor consistency with--main-llmand--figure-llmNew verify checks
Auto-repair