Preserve distinct captures with different prompt tokens

[bradhilton]

Captures with different original prompts can be mistaken for duplicates when their response identifiers, generated tokens, and output metadata match. Include captured prompt token IDs in the existing source fingerprint so both training examples survive. Genuine copies of the same capture still deduplicate.

This version is standalone on ART main 9a16366d79eebc36e00f75670965e2629c012b75. It contains only the source-identity change from the previous #886: one runtime function in art.trajectories._tokenize, its explanatory comment, and tests. _history.py is byte-identical to main; #885's automatic history-attribution change is excluded. Public APIs, automatic history construction, explicit reconciliation, manual-history editing, and loss normalization retain main's behavior. No new edited-history validator or probability recomputation is introduced. The broader history-attribution discussion in #883 remains separate.

Validation on the standalone source:

• 547 trajectory tests pass, including all 16 source-identity tests. On unchanged main, those 16 produce 10 expected failures and six passing controls.
• Exact diff/AST checks prove the only runtime function change is the same fingerprint function as the original Preserve distinct captures with different prompt tokens #886, with captured-prompt extraction precedence preserved for Chat, Completions, Responses and Messages.
• Ruff, formatting and lock checks pass. Local full type checks report the same 11 diagnostics on unchanged main and this candidate; hosted CI is tracked separately.
• Independent standalone review passes nine extra identity controls and 11 manual/edit/reconciliation cases whose native outputs are byte-identical to main. Fresh formal reviews from Minsky and McCarthy are requested for head 3627311444748067482c486909d3642644509efd; earlier stacked-head approvals remain historical evidence.

Prepared at Brad's request. This does not merge #885, relax experiment source fences, launch research, or authorize deployment.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Review text removed on 2026-09-17; the consolidated review record is in the summary comment on this PR.

[bradhilton]

Consolidated review record (automated agent review traffic removed 2026-09-17)

Change. Captures with different original prompts but matching response IDs, generated tokens and output metadata were mistaken for duplicates, dropping a training example. The sampled-source fingerprint in art.trajectories._tokenize now includes captured prompt token IDs for all four protocols, so distinct prompts survive while genuine copies still deduplicate; standalone on main, excluding #885.

Review. Minsky and McCarthy each posted fresh CLEAR verdicts at the merged head 3627311, after earlier clearing the stacked head. Verified: 547 trajectory tests pass plus 21 extra identity controls; the 16 new identity tests give 10 expected failures on unchanged main; ruff/format clean; the two-file patch matches the original delta with _history.py unchanged. Schulman confirmed correct-base CI after reopening: the GPU classifier gate passed (GPU job skipped), Prek still in progress.

Deferred / follow-ups.

• Broader history-attribution and reconcile/manual/repair gaps remain open in Retained sampled suffix can carry logprobs under a different causal prefix #883.
• Preserve captured conditioning when retaining sampled lineage #885 (automatic history attribution) held separately.
• Prompt hashing adds work proportional to prompt length; no performance qualification claimed.