fix(indexer): cap chunk length so an unsplittable line cannot OOM the daemon

[m-bo-one]

Problem

RecursiveSplitter treats chunk_size as a target, not a bound. A line containing no separator it
recognises comes back as one chunk, however long the line is, and process_file hands that chunk
straight to embedder.embed().

That is fatal rather than wasteful. SentenceTransformerEmbedder._embed batches up to 64 texts and
pads the batch to its longest member, and attention costs O(n²) in that length. A 60,000-character
chunk is not 60x a normal one, it is thousands of times the memory.

Where it showed up: a Godot project. .tscn scene files store packed arrays — vertices, polygons,
navigation meshes — as a single line, and a mid-sized scene reaches 60,000 characters on one line
easily. Nothing about the bug is Godot-specific, though: minified JS or CSS, a long JSON literal, a
base64 blob or a generated lookup table all take the same path.

How it surfaced

The report was "indexing hangs, then eats all the RAM", and daemon.log ended mid-session with no
traceback, because the process is killed before anything is written. What made it findable:

1. Sampling the daemon process rather than watching it. Private commit is the number that moves; RSS
   stays modest because most of the growth never becomes resident before the kill.
2. Narrowing by input. The repository held 5.7 MB of indexed text — small — so size was not the
   trigger. Indexing files individually put the whole failure on one 472 KB scene file: 0.8 GB → 106
   GB of commit in three seconds, dead process.
3. Running the chunker alone on that file, no daemon and no GPU. It returned a chunk of 60,942
   characters from a splitter asked for 1,000. Every other chunk in that repository and in a second,
   unrelated one measured at most 1,000, so this was the single input out of contract.

Reproduction, deterministic and instant:

from cocoindex.ops.text import RecursiveSplitter

content = "surfaces/0 = " + ",".join(str(i % 97) for i in range(20_000))
chunks = RecursiveSplitter().split(
    content, chunk_size=1000, min_chunk_size=250, chunk_overlap=150, language="text"
)
print(len(chunks), max(len(c.text) for c in chunks))
# 2 57929   <- a 57,929-character chunk out of a splitter asked for 1000

Write that string into a file inside an indexed project and ccc index reproduces the crash end to
end.

Fix

cap_chunk_size() cuts every chunk down to limit, defaulting to CHUNK_SIZE, and carries line,
column, character and byte offsets forward so the pieces still report where they came from. When
nothing is oversized it returns the input list unchanged, so the ordinary path allocates nothing.

• Applied after the custom-chunker branch too: returning a whole file as one chunk is a documented
  use of CHUNKER_REGISTRY, and that should not be able to kill the daemon either.
• Pieces are evened out rather than cut at the limit with a remainder — 2007 characters become three
  of 669, not 1000, 1000, 7. A seven-character tail would be an embedding of noise, and nothing
  else in the pipeline emits a chunk below MIN_CHUNK_SIZE.
• The ceiling is CHUNK_SIZE and not a multiple of it: at CHUNK_SIZE * 4 the same file still
  peaked at 32 GB, since with max_batch_size=64 four times the sequence length is sixteen times the
  attention matrix.

Validation

Windows 11, RTX 5090, nomic-ai/CodeRankEmbed on cuda:0, 61.6 GB RAM. Peak is private commit.

Workload	Before	After	Peak commit
One 472 KB scene file, 60,942-character line	daemon killed, empty log	539 chunks, 16 s	106 GB → 11 GB
413-file project containing that file	daemon killed mid-run	9,811 chunks, 22 s	76 GB → 14.7 GB
210-file Python corpus, no oversized chunk	3,227 chunks, 9.7 s	3,227 chunks, 9.7 s	unchanged

The third row is the control: with nothing over the ceiling the run is identical, which is what the
early return guarantees.

Tests are at three levels, since a unit test on cap_chunk_size alone would still pass if the call
were dropped from process_file:

• tests/test_chunk_cap.py, 7 unit cases: pass-through, the even cut, the worst case for evening out
  (one character over the ceiling), position carry-over across a newline, byte offsets on multi-byte
  text, a caller-supplied limit, and the real 60,942-character shape.
• 2 cases against the real RecursiveSplitter. One asserts that it does exceed its own
  chunk_size on such a line, so if that is fixed upstream this test fails and reports the cap as
  dead code instead of leaving it in place forever.
• tests/test_e2e.py::test_session_caps_oversized_chunks: a full init + index run, asserting no
  row in the index exceeds CHUNK_SIZE. Checked by disabling the call, where it fails with
  assert 57929 <= 1000. Adds about 8 s to the suite.

uv run pytest — 326 passed, 5 skipped, 8 deselected. uv run prek run --all-files — clean.
(test_file_walk.py::test_max_file_size_keeps_unstattable_files fails on my machine on main as
well: it creates a symlink and Windows refuses without the privilege, WinError 1314.)

Compatibility

No configuration, no migration, no API change. Chunk IDs change only for files that previously
produced an oversized chunk, since those become several rows — such files reindex once and become
searchable at a useful granularity, which they were not before.

Notes for reviewers

• The root cause is one level down: RecursiveSplitter could fall back to a hard character cut when
  it runs out of separators, and then no consumer would need this. I put the fix here because this is
  where the ceiling is known and where the crash lands, but I am happy to move it into the engine
  instead, or to open the issue there — say which you prefer.
• The ceiling is deliberately not a setting: it is CHUNK_SIZE, already the pipeline's chunk
  contract, and a separate knob would invite a value that reintroduces the crash. limit stays a
  parameter only so the tests can drive it.
• The warning fires once per oversized chunk, so a file with many such lines produces a burst. Happy
  to fold it into one line per file if you prefer.

[badmonster0]

thanks @m-bo-one , @georgeh0 can help take a look!