_ShardLease.load holds every shard of a target in memory at once — the forecast leg's #126

[Polichinel]

The defect

_ShardLease.load materialises every shard of a target before decoding any of them:

def load(self):
    """Fetch (by pinned id), verify, curate — return the PRODUCT ``(frame, headers)``."""
    shard_bytes = {
        name: self.store.download(file_id)
        for name, file_id in self.shard_file_ids.items()
    }
    frame, headers = track_a_source.frames_for_target(self.manifest, shard_bytes)

The dict comprehension completes before frames_for_target is called, so all 36 monthly
shards are resident simultaneously, and the decoded frame is built on top of them.

This compounds with the manager: CRAFDPostProcessorManager holds self._historical_frame
— 28,421,738 rows over 64,742 cells for the CRAF'd delivery — from _read through the whole
of _save. Peak is the sum: historical frame + every shard's bytes + the decoded frame.

Measured

First un_crafd delivery attempt, 2026-08-13 (views-crafdapi#44). Same code, same data,
two runs:

Run	Machine state	Outcome
17:43	15 GB already in use, swap exhausted (1.8 of 2.0 GB)	died in the shard download
18:08	memory freed, nothing else changed	completed, staged cleanly

dmesg shows the kernel OOM-killing a 14.6 GB process minutes later. No code changed
between the two runs.

For scale on the consumer side: views-crafdapi's preflight assembles the same run and peaks
at 4.9 GB RSS — and it streams. This delivery path does not.

The repo has met this before, on the other leg

#126 — "un_fao: retire the pandas historical path — deliver frame-native end-to-end
(fixes run-0 OOM)" — and #131, which recorded the result as "OOM gone".

That fixed the historical leg. The forecast shard leg has the same
everything-resident shape and was never revisited. CRAF'd's historical is already
frame-native (the manager logs "Historical is frame-native"), which is precisely why the
remaining pressure is the shard dict rather than the frame.

This is a design change, not a patch

track_a_source.frames_for_target(manifest, shard_bytes) takes the whole mapping, so
streaming means changing that signature or introducing an iterator between them. Worth
scoping deliberately rather than reaching for the smallest edit.

A working precedent for the shape exists on the consumer side —
views-crafdapi/src/views_crafdapi/preflight.py::_assembly_gate walks the same run
month-by-month, releasing each month's state before the next, and completes on the full
production run in 4.9 GB. Not code to import (different repo, different purpose), but the
traversal it uses is the one this path needs.

Acceptance

• A target's shards are decoded incrementally; the whole run's bytes are never resident at once.
• The historical frame is released, or not held, across the forecast delivery phase.
• Measured peak RSS for a full 3-target / 36-month / 64,742-cell delivery is recorded in
  the issue, so the next person has a number rather than an anecdote.
• un_fao delivery still completes — this path is shared, and un_fao: retire the pandas historical path — deliver frame-native end-to-end (fixes run-0 OOM) #126's fix must not regress.

Cross-references

• _ContractStorePort.download fails open where upload refuses — C-79's untreated sibling #268 — _ContractStorePort.download fails open. It is why this failure was unreadable:
  the memory problem surfaced as AttributeError: 'NoneType' object has no attribute 'get'
  rather than a named refusal. Independent defects; they met in the same crash.
• un_fao: retire the pandas historical path — deliver frame-native end-to-end (fixes run-0 OOM) #126 / run-0 DELIVERED: first FAO global-land forecast shipped frame-native (OOM gone) — verify delivery integrity? #131 — the historical leg's OOM and its frame-native fix. Same class of problem,
  the leg that was done.
• views-crafdapi**chore: retire vestigial Git LFS config (resolves C-41) #44** (D4) — the delivery that surfaced it. The run has since staged and
  passed all ten consumer preflight gates, so nothing is blocked on this.

Filed from the views-crafdapi seat. No change was made in this repo. Per this repo's
convention ("The FILE is the claim; the line number is not"), this issue names symbols.