Skip to content
34 changes: 26 additions & 8 deletions docs/agent.md
Original file line number Diff line number Diff line change
Expand Up @@ -298,24 +298,42 @@ wrong-calls ↓) and its **knee** (best quality per unit cost).
### Real-run prompt optimization (`--optimize`)

GEPA prompt optimization is a first-class CLI path. `tablassert agent --optimize` (`-o`) runs
`dspy.GEPA` with a real reflection LM (built from the same `--model-id`/`--api-base`/`--api-key`
config) and **persists the optimized instructions** instead of running the supervisor:
`dspy.GEPA` and **persists the optimized instructions** instead of running the supervisor.

Following GEPA best practice, the optimizer splits the models: a **strong reflection LM** (`--model-id`)
proposes the few instruction edits, and an optional **fast task LM** (`--task-model`) runs the many
candidate program evaluations. Pointing `--task-model` at a cheap model (e.g. a flash model) keeps the
run fast while the strong model does the thinking; without `--task-model` the reflection LM is used for
both. `--gepa-threads` parallelizes GEPA's candidate **LM forward passes** only — the coverage-scoring
builds stay serialized on the process-wide `_GEPA_BUILD_LOCK` (`agent.py`, since `os.chdir` is
process-global), so a higher thread count does not speed up the expensive build/coverage step.

```bash
# optimize the agent prompt over a dataset of examples, writing the result to a file
tablassert agent PMC11708054 --fullmap ./fullmap --optimize \
--dataset examples/gepa-dataset.yaml --instructions-out .tablassert/agent/optimized_instructions.yaml
--dataset examples/gepa-dataset.yaml --task-model qwen-flash \
--max-metric-calls 30 --gepa-threads 4 \
--instructions-out .tablassert/agent/optimized_instructions.yaml

# later, run the supervisor with the optimized prompt
tablassert agent PMC11708054 --fullmap ./fullmap \
--instructions-file .tablassert/agent/optimized_instructions.yaml
```

`--dataset` is a YAML/JSON list of `{table_summary, coverage_feedback}` examples; `--max-metric-calls`
bounds the GEPA metric budget. `save_optimized_instructions` / `load_optimized_instructions` persist and
reload the prompt (a `{instructions, descriptions}` mapping). Without `--instructions-file` the built-in
`INSTRUCTIONS` prompt is used. (A real optimization run needs a live model; the offline suite exercises
this path via an injectable `gepa_cls` stub.)
`--dataset` is a YAML/JSON list of examples. Each example carries `table_summary` and
`coverage_feedback` (the program inputs); it MAY also carry:

- `fullmap` — a fullmap path. When present, the GEPA metric scores each proposed config with **real
fullmap coverage** (via a `build_and_audit` head-sample), so GEPA optimizes the genuine objective
rather than a validity-only proxy.
- `workdir` — the directory a proposed config's relative `source.local` resolves against (LLMs mimic the
exemplar's `./downloads/...` paths), so coverage is measured on the actual table.
- `head` — default `true`: score a fast 5-row preview; set `false` for full-fidelity coverage builds.

`--max-metric-calls` bounds the GEPA metric budget. `save_optimized_instructions` /
`load_optimized_instructions` persist and reload the prompt (a `{instructions, descriptions}` mapping).
Without `--instructions-file` the built-in `INSTRUCTIONS` prompt is used. (A real optimization run needs a
live model; the offline suite exercises this path via an injectable `gepa_cls` stub.)

### Golden fixture

Expand Down
4 changes: 2 additions & 2 deletions docs/api/qc.md
Original file line number Diff line number Diff line change
Expand Up @@ -115,15 +115,15 @@ return similarity >= 0.5

### BioBERT Model

**Model:** `pritamdeka/BioBERT-mnli-snli-scitail-mednli-stsb`
**Model:** `pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb`

**Backend:** [sentence-transformers](https://www.sbert.net/) (PyTorch). Embeddings are compared with scikit-learn's `cosine_similarity`.

**Lazy-loaded** on the first `fullmap_audit()` call that reaches the embedding stage via `get_biobert()`, then cached globally for the lifetime of the process.

### Model Caching

`get_biobert()` loads the model from the local cache when present; otherwise it downloads `pritamdeka/BioBERT-mnli-snli-scitail-mednli-stsb` and saves it for future runs.
`get_biobert()` loads the model from the local cache when present; otherwise it downloads `pritamdeka/BioBERT-mnli-snli-scinli-scitail-mednli-stsb` and saves it for future runs.

**Cache location:** `.tablassert/biobert/` on disk (`qc.MODEL`); the loaded model object is also cached in memory for the lifetime of the process.

Expand Down
4 changes: 3 additions & 1 deletion docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,9 @@ PMC ids are passed positionally (also accepted as `--pmc-ids`). This page lists
| `--instructions-file` | Path | No | `None` | Load GEPA-optimized instructions from a prior `--optimize` run |
| `--instructions-out` | Path | No | `None` | Where `--optimize` writes optimized instructions (default `<state-dir>/optimized_instructions.yaml`) |
| `--max-metric-calls` | int | No | `8` | GEPA metric-call budget for `--optimize` |
| `--dataset` | Path | No | `None` | YAML/JSON list of `{table_summary, coverage_feedback}` examples for `--optimize` |
| `--dataset` | Path | No | `None` | YAML/JSON list of `{table_summary, coverage_feedback}` examples for `--optimize` (an example may also carry `fullmap`, `workdir`, and `head` to score each proposed config with real coverage) |
| `--task-model` | str | No | `None` | Fast model id for GEPA's many program evaluations (cheap task LM + strong reflection LM); `--model-id` is the reflection LM. Defaults to the reflection LM |
| `--gepa-threads` | int | No | `None` | Thread count for GEPA's evaluation pool (`--optimize`) — parallelizes candidate LM forward passes only; coverage-scoring builds stay serialized on `_GEPA_BUILD_LOCK` |

```bash
tablassert agent PMC11708054 --fullmap ./fullmap
Expand Down
Loading
Loading