fix(write): bound SVAR2 genotype-writing memory with max_mem

docs/source/faq.md

@@ -76,7 +76,7 @@ GVL's read path (haplotype reconstruction and track re-alignment) is parallelize
 Both are sparse columnar variant archives from [`genoray`](https://github.com/mcvickerlab/genoray) that `gvl.write(variants=...)` accepts alongside BCF/PGEN; see [write.md](write.md) for how to build one. The two differ in their read-time behavior:
 
 - **`.svar`** reconstructs by building an interval search tree over the queried window and a per-read dense union of the overlapping variants.
-- **`.svar2`** reconstructs via a **read-bound** path: `gvl.write` caches small per-`(region, sample, ploid)` variant-key ranges at write time, and `Dataset.__getitem__` gathers directly off that cache and calls all-Rust kernels — it builds **no interval search tree and no dense union per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts.
+- **`.svar2`** reconstructs via a **read-bound** path: `gvl.write` caches per-`(region, sample, ploid)` variant-key ranges at write time — **not small at cohort scale**, see the size formula in `format.md` — and `Dataset.__getitem__` gathers directly off that cache and calls all-Rust kernels — it builds **no interval search tree and no dense union per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts.
 
 `.svar2` is Phase-1 scope: a handful of combinations (`annotated` haplotypes, `min_af`/`max_af`, `VarWindowOpt(ref="allele")`, fixed-length haplotype-realigned tracks, spliced `variant-windows`, and `variants`/`variant-windows` output with jitter) aren't wired yet and raise `NotImplementedError` rather than silently mis-computing. Haplotype and `variants` output support splicing, `var_filter="exonic"`, and negative-strand reverse-complementation. `"variant-windows"` output, `unphased_union` (for both `"variants"` and `"variant-windows"`), and `var_fields`-selected store INFO/FORMAT fields (also for both, when the `.svar2` was written with them) are also supported. See the `genvarloader` skill's `.svar2` section or `docs/source/format.md` for the full list. Everything else — haplotypes, tracks, and variants/variant-windows at any supported jitter/output-length combination — is byte-identical between the two backends.
 

docs/source/format.md

@@ -24,8 +24,9 @@ When the dataset was built from an `.svar`, the heavy per-variant arrays (`varia
 `dosages.npy`, `index.arrow`) are **not duplicated** into the dataset. Instead the dataset
 records a back-reference to the source `.svar` in `metadata.json` (see `svar_link` below).
 Likewise, a dataset built from an `.svar2` records a back-reference (`svar2_link`, below)
-and caches only small per-`(region, sample, ploidy)` range arrays under `genotypes/svar2_ranges/`
-— the bulk variant data stays in the `.svar2` store.
+and caches per-`(region, sample, ploidy)` range arrays under `genotypes/svar2_ranges/`
+— the bulk variant data stays in the `.svar2` store. See "`genotypes/svar2_ranges/` layout"
+below for the on-disk size of this cache; it is not small at cohort scale.
 
 ## `metadata.json` schema
 
@@ -87,10 +88,23 @@ Written only when the dataset's variant source is a `.svar2` store. `R` = number
 | `vk_indel_range.npy` | `(R, S, P, 2)` | Same, for the indel variant-key column. |
 | `dense_snp_range.npy` | `(R, 2)` | Per-region (sample-independent) range into the dense SNP store. |
 | `dense_indel_range.npy` | `(R, 2)` | Per-region (sample-independent) range into the dense indel store. |
-| `region_starts.npy` | `(R,)` | Per-region write-time start coordinate. Retained for parity/debugging; the read path derives per-query starts from the (post-jitter) query regions and does **not** read this array's values. |
 | `sample_cols.npy` | `(S,)` | Maps the dataset's selected-sample slot to the `.svar2` store's original sample index. |
 | `svar2_meta.json` | — | Records each array's `shape`/`dtype` plus `ploidy`. |
 
+`vk_snp_range.npy` and `vk_indel_range.npy` are each
+`(regions, samples, ploidy, 2)` int64, so the two together occupy
+
+```
+2 x regions x samples x ploidy x 2 x 8 bytes
+```
+
+This grows linearly in **both** the number of BED rows and the number of
+selected samples. It is not small at cohort scale: ~4,000 regions over 414,830
+diploid samples is approximately **98 GiB** for a single chromosome/panel.
+`gvl.write` logs the projected size before allocating and warns when it exceeds
+free disk. Budget disk accordingly, or reduce the region count or sample
+selection.
+
 At read time, `Dataset.__getitem__` slices these memmaps (numpy fancy-indexing; no interval
 search) to build the flat per-query inputs for the read-bound Rust kernels — no interval-search
 tree and no dense-union rebuild happen per read, unlike the `.svar` path.

docs/source/write.md

@@ -107,4 +107,4 @@ gvl.write(
 
 Both formats store a back-reference in the dataset's `metadata.json` instead of duplicating per-variant arrays, so the source store must remain accessible when the dataset is later opened with [`gvl.Dataset.open()`](api.md#genvarloader.Dataset.open) (override its location with `svar=`/`svar2=` if it has moved).
 
-`.svar2` additionally produces a small write-time cache under `<path>/genotypes/svar2_ranges/` and reads back through an all-Rust, read-bound path with no interval-search-tree build and no dense-union rebuild per read — see [the FAQ](faq.md) for the read-path and on-disk-size tradeoffs, and [the format reference](format.md) for the on-disk layout. `.svar2` currently has a Phase-1 scope: a handful of output combinations (`annotated` haplotypes, `min_af`/`max_af`, spliced variant-window/track outputs, etc.) aren't wired yet and raise `NotImplementedError` — see the `genvarloader` skill or `format.md` for the full list. Haplotype and `variants` output support splicing and `var_filter="exonic"`.
+`.svar2` additionally produces a write-time cache under `<path>/genotypes/svar2_ranges/` and reads back through an all-Rust, read-bound path with no interval-search-tree build and no dense-union rebuild per read — see [the FAQ](faq.md) for the read-path and on-disk-size tradeoffs, and [the format reference](format.md) for the on-disk layout, including the size formula. This cache is **not small at cohort scale**. `gvl.write` honours `max_mem` when writing `.svar2` genotype ranges, bounding the RAM used while producing the cache; the permanent range cache itself is governed by disk space, not `max_mem` (see the format reference). `.svar2` currently has a Phase-1 scope: a handful of output combinations (`annotated` haplotypes, `min_af`/`max_af`, spliced variant-window/track outputs, etc.) aren't wired yet and raise `NotImplementedError` — see the `genvarloader` skill or the format reference for the full list. Haplotype and `variants` output support splicing and `var_filter="exonic"`.