Speed up assistant span prefix comparisons

[bradhilton]

Long conversation histories repeatedly scan prompt prefixes in Python while locating assistant-generated spans. Replace those two element-by-element loops with a private helper that compares growing blocks using native slice equality, then searches within the first unequal block. On the complete canonical validation corpus, tensorization averages 250.9 → 140.0 seconds (44.2% shorter) with byte-identical outputs.

The helper handles strings and integer-token lists. An empty/first-mismatch fast path avoids copying a long history when the answer is immediately known. Temporary slices grow with the compared prefix; there is no persistent cache. Rendering, span fallbacks, selected tokens, flags, log-probabilities and error paths retain their existing behavior. The runtime diff is 24 added/14 removed lines, with no dependency or public API change.

Repeated profiling at base 077abf6fa7a3a51f9ff255c37f8886c6c8256082 and candidate 45eaed3259e179ca9068dd0851723a643536a85f, in A/B/B/A order:

Run	Source	Tensorization
A1	Base	251.214 s
B1	Candidate	140.745 s
B2	Candidate	139.225 s
A2	Base	250.516 s

This saves 110.880 seconds on average, a 1.792× speedup. Every run retained all 128 sources and 256 real/canonical-generated views, including the 68 recorded model-error returns. All 768 ordered token/flag/log-probability fields match the fresh base byte for byte, including shapes, dtypes and NaN bit patterns. Historical frozen tensor parity also passed separately. Token counts and template/encoding/span call counts are unchanged.

CPU profiling localizes the savings to span finding. These are summed worker-thread CPU seconds, averaged across both runs; each row excludes its instrumented child stages:

Stage	Calls/run	Base CPU	Candidate CPU
Assistant span finding	3,524	152.808 s	42.887 s
Chat template rendering	99,660	58.296 s	56.990 s
Encoding	103,440	84.198 s	82.271 s

Total worker conversion CPU falls from 325.382 to 211.631 seconds. The campaign used fresh sequential processes, the same four concurrent work slots and timing instrumentation, frozen captured inputs and an offline tokenizer. Imports, capture loading and parity-artifact work are excluded from the tensorization timer. All four runs finished within the fixed 20-minute campaign, and all 16 owned process identities were independently verified absent.

Validation on the final revision:

• 515 local trajectory tests pass: 458 existing plus 57 new deterministic tests. Coverage includes differential string/list oracles, Unicode, mismatch/truncation boundaries, both span call sites, suffix/removal fallbacks, overlap/unmappable errors, input preservation and full text/token-ID tokenization flags.
• Independent differential review passed 11,907 final-helper cases. Removing the helper and restoring the two original loops reproduces the original module AST exactly.
• Scoped ty, Ruff, formatting, whitespace and offline lock checks pass. CI is green: 773 Megatron lightweight tests and 1,263 unit tests pass, with 49 unit tests skipped.
• An earlier local suite attempt reached its 240-second external cap; that incomplete outcome is retained. The final 458-test rerun passed in 167.72 seconds. Its observed subprocess/filesystem waits do not prove the earlier timeout's cause.

McCarthy's independent source review is clear: 515 tests pass at this head, 557 pass with the held tokenizer composition, and 37,468 helper differential executions pass. The reviewer also rehashed the complete profiling evidence and tensor artifacts.

The committed standalone microbenchmark loads the exact production helper AST without importing the ML runtime, validates results before timing and records source hashes, iterations and medians. It needs no private corpus:

uv run --no-project --python 3.12 python scripts/benchmark_assistant_prefix.py \
  --sizes 0 16 1024 65536 --repeats 7

These are two runs per variant on one shared host and one captured corpus, not a general latency guarantee or a new end-to-end inference/probe qualification. Microbenchmarks show that very short nonzero prefixes can incur approximately 0.2–0.3 µs additional overhead; long shared prefixes benefit substantially. There are no timing assertions in tests.

Evidence on the shared machine: /home/brad/.local/share/thanos/art-prefix-optimization-20260910/, including profiling/REPORT.md, comparison.json, per-view/stage timings, source/input manifests, raw tensor comparisons and cleanup receipts. Canonical checkpoint: wandb-artifact:///wandb/049-retail49-program/r49-program-cmapping-he8cb0aa602dcc0b9:v1, step 31, sources 0–127; corpus manifest SHA256 e21f304bab8b76bc14442b2839148417f0327f1cdfdc143192e43cb376c25381. Final profiling manifest SHA256 a6dfbcda3c71eab9987825f34a3db5b25f7df1107017cb1a4477949f6bba6429 verifies all 822 evidence files.

Maintained 049 adoption is separate: its reviewed raw G/V helper correctly refuses the changed tokenizer source hash and moved exception line. It needs a separately reviewed helper revision before updating the ART pin; this PR does not bypass that fence.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. Replaces two element-by-element prefix-scanning loops in assistant span finding with a private _common_prefix_length helper that compares doubling slices natively and binary-searches the first unequal block. On the captured 128-source corpus, tensorization drops 250.9 to 140.0 s (44.2%) with byte-identical outputs; no public API or dependency change.

Review. McCarthy and Minsky both cleared 45eaed3 (base 077abf6), CPU/source scope, no blocking findings; Thanos produced the A/B/B/A profiling receipts, which McCarthy rehashed (768 tensor fields byte-identical). Verified: 515 trajectory tests (57 new) pass at the head, 557 on the held ART composition, 37,468 plus 2.5 million helper differential checks against a linear oracle, Ruff/format clean, CI green (773 Megatron lightweight and 1,263 unit tests).

Deferred / follow-ups.

• 049 adoption: the changed _tokenize.py source and moved mask-raise line are correctly refused by the raw G/V helper fence; a separately reviewed helper revision is needed before any ART pin update.