beam: scale the async-operation timeout to the corpus instead of a fixed 300s - #38
Open
nicoloboschi wants to merge 3 commits into
Open
beam: scale the async-operation timeout to the corpus instead of a fixed 300s#38nicoloboschi wants to merge 3 commits into
nicoloboschi wants to merge 3 commits into
Conversation
Ingest cost on a large split is dominated by fact extraction, and extraction cost is `corpus_chars / retain_chunk_size` LLM calls. BEAM-10M is 10 conversations totalling ~468M characters, so at the server-side default chunk size of 3000 it is **~156,000 extraction calls for a single run** — which is what makes that split impractical to ingest rather than merely slow. The provider had no way to influence either. `_bank_kwargs` sent only `enable_observations` and, for BEAM, a retain mission, so every bank ran the server default (`concise`, i.e. full LLM extraction) at the default chunk size. Two env vars now pass through to `create_bank`, which has accepted both all along: - `AMB_HINDSIGHT_EXTRACTION_MODE=chunks` skips the LLM entirely and stores each chunk as its own unit. That makes a run of this size tractable, but it is **not the same measurement**: `_BEAM_RETAIN_MISSION` is an extraction prompt, so chunks mode ignores it and stores raw text with no fact extraction and no entities. A chunks-mode score is not comparable to an extracted one. - `AMB_HINDSIGHT_CHUNK_SIZE` trades the same axis more gently — doubling it roughly halves the call count while keeping extraction, at coarser granularity. Both are UNSET by default, so every existing result stays exactly the run it was. This is deliberately a new capability rather than a change of default: which mode BEAM should be scored under is a benchmark-semantics decision, and this only makes the choice expressible. `scripts/test_bank_kwargs.py` checks the default sends neither control and keeps the BEAM mission, that each control arrives when set (with the chunk size coerced to int), and that non-BEAM datasets get the controls without the BEAM mission.
`load_documents` caps documents at `_MAX_DOC_CHARS = 100_000` and splits sessions
to stay under it. BEAM-10M never reached that code. It nests two levels deeper than
the other splits:
chat[i]["plan-N"][batch]["turns"] -> list of turn GROUPS, each a list of turn dicts
Nothing there is a `list` at the top level, so `sessions` came out empty and the
conversation fell through to the "unusual structure" branch, which emits it whole:
10m: 10 documents, median 47,280,119 chars <- 470x the cap
100k: 170 documents, median 92,351 chars
That is 470x the limit this same function enforces on every other split, for exactly
the reason the limit was introduced.
The consequence was not a slow ingest but an impossible one. Retain cost is per-CALL,
not per-byte -- measured against a live API, one item takes 1.29s and fifty take 1.39s
-- and a backend that serializes retains per document gets no parallelism at all when
a 10-conversation split is only 10 documents. Ingest ran at ~1,300 chars/s, ~100 hours
for the split, so the harness hit its 300s-per-operation timeout, gave up, and scored
a corpus that was 0.27% loaded. That is what the published 10m result
(`ingested_docs: 1`, `accuracy: 0.0`) is.
`_sessions_from_plans` flattens each batch's turn groups into one session, so the
existing chunk loop applies unchanged:
10m: 10 -> 5,265 documents, median 97,409 chars
100k: 170 documents (unchanged)
1m: 1,830 documents (unchanged)
Only the split that was taking the fallback moves; every other split loads byte-identically,
so existing results stay the runs they were. Total content is preserved to within 10,510
chars of 468,288,866 (0.002%), the difference being per-session formatting separators.
This is necessary but not sufficient: at ~1,300 chars/s a 5-document batch is still
~375s, over the 300s `_await_operation` timeout, so the ingest rate has to come down
too before a 10m score means anything.
Claude-Session: https://claude.ai/code/session_01CJvQzszqZck74S5oC8dEAZ
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
…xed 300s Reverts the client-side document chunking from the previous commit. Splitting the conversation in the harness changes the document shape the benchmark measures, and it was aimed at the wrong problem: nothing actually stops a 47MB document. Measured in-cluster against a chunks-mode bank with consolidation off: async submit 100,000 chars 0.1s async submit 1,000,000 chars 0.3s async submit 5,000,000 chars 1.0s async submit 20,000,000 chars 3.9s The submit returns an operation id immediately and scales fine -- a 47MB document submits in about 9s. The work behind it then runs at ~31,000 chars/s, so a BEAM-10M conversation (median 47,280,119 chars) needs ~25 MINUTES to complete, and the whole 468,288,866-char split about 4.2 hours. `_await_operation` waited a fixed 300s. So it gave up five minutes in, logged a warning, and let the harness query a corpus that was still loading. That is what the published 10m result is: `ingested_docs: 1`, `accuracy: 0.0` -- not a failed run, a run that scored an empty bank. Two changes: - The timeout defaults to 7200s and is overridable with `AMB_OPERATION_TIMEOUT_S`, which covers the largest single document in any current split with room to spare. - Abandoning an operation is now an ERROR that says the score is invalid, not a warning followed by "continuing anyway". A run that silently scores a partially loaded corpus is worse than one that fails, because the number looks real. Claude-Session: https://claude.ai/code/session_01CJvQzszqZck74S5oC8dEAZ
nicoloboschi
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
load_documentscaps documents at_MAX_DOC_CHARS = 100_000and splits sessions to stay under it. BEAM-10M never reached that code.It nests two levels deeper than the other splits:
Nothing there is a
listat the top level, sosessionscame out empty and the conversation fell through to the "unusual structure" branch, which emits it whole:That is 470x the limit this same function enforces on every other split, for exactly the reason the limit exists.
Why it matters
The consequence was not a slow ingest but an impossible one. Retain cost is per-call, not per-byte — measured against a live API:
50x the content for +8% time. And a backend that serializes retains per document gets no parallelism when a 10-conversation split is only 10 documents. Ingest ran at ~1,300 chars/s — ~100 hours for the split — so the harness hit its 300 s
_await_operationtimeout, gave up, and scored a corpus that was 0.27% loaded (1,048 of ~390,000 chunks on unit 1, units 2-10 empty). That is what the published 10m result (ingested_docs: 1,accuracy: 0.0) is measuring.The change
_sessions_from_plansflattens each batch's turn groups into one session, so the existing chunk loop applies unchanged:Only the split that was taking the fallback moves — every other split loads byte-identically, so existing results stay the runs they were. Content is preserved to within 10,510 chars of 468,288,866 (0.002%), the difference being per-session formatting separators.
Necessary but not sufficient
At ~1,300 chars/s a 5-document batch is still ~375 s, over the 300 s
_await_operationtimeout, so a 10m run would still abandon operations. The ingest rate has to come down before a 10m score means anything — I have not fixed that here.https://claude.ai/code/session_01CJvQzszqZck74S5oC8dEAZ