diff --git a/docs/source/faq.md b/docs/source/faq.md index ae4beb0b..95fd6128 100644 --- a/docs/source/faq.md +++ b/docs/source/faq.md @@ -76,7 +76,7 @@ GVL's read path (haplotype reconstruction and track re-alignment) is parallelize Both are sparse columnar variant archives from [`genoray`](https://github.com/mcvickerlab/genoray) that `gvl.write(variants=...)` accepts alongside BCF/PGEN; see [write.md](write.md) for how to build one. The two differ in their read-time behavior: - **`.svar`** reconstructs by building an interval search tree over the queried window and a per-read dense union of the overlapping variants. -- **`.svar2`** reconstructs via a **read-bound** path: `gvl.write` caches small per-`(region, sample, ploid)` variant-key ranges at write time, and `Dataset.__getitem__` gathers directly off that cache and calls all-Rust kernels — it builds **no interval search tree and no dense union per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts. +- **`.svar2`** reconstructs via a **read-bound** path: `gvl.write` caches per-`(region, sample, ploid)` variant-key ranges at write time — **not small at cohort scale**, see the size formula in `format.md` — and `Dataset.__getitem__` gathers directly off that cache and calls all-Rust kernels — it builds **no interval search tree and no dense union per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts. `.svar2` is Phase-1 scope: a handful of combinations (`annotated` haplotypes, `min_af`/`max_af`, `VarWindowOpt(ref="allele")`, fixed-length haplotype-realigned tracks, spliced `variant-windows`, and `variants`/`variant-windows` output with jitter) aren't wired yet and raise `NotImplementedError` rather than silently mis-computing. Haplotype and `variants` output support splicing, `var_filter="exonic"`, and negative-strand reverse-complementation. `"variant-windows"` output, `unphased_union` (for both `"variants"` and `"variant-windows"`), and `var_fields`-selected store INFO/FORMAT fields (also for both, when the `.svar2` was written with them) are also supported. See the `genvarloader` skill's `.svar2` section or `docs/source/format.md` for the full list. Everything else — haplotypes, tracks, and variants/variant-windows at any supported jitter/output-length combination — is byte-identical between the two backends. diff --git a/docs/source/format.md b/docs/source/format.md index c3457e26..dbf6cdef 100644 --- a/docs/source/format.md +++ b/docs/source/format.md @@ -24,8 +24,9 @@ When the dataset was built from an `.svar`, the heavy per-variant arrays (`varia `dosages.npy`, `index.arrow`) are **not duplicated** into the dataset. Instead the dataset records a back-reference to the source `.svar` in `metadata.json` (see `svar_link` below). Likewise, a dataset built from an `.svar2` records a back-reference (`svar2_link`, below) -and caches only small per-`(region, sample, ploidy)` range arrays under `genotypes/svar2_ranges/` -— the bulk variant data stays in the `.svar2` store. +and caches per-`(region, sample, ploidy)` range arrays under `genotypes/svar2_ranges/` +— the bulk variant data stays in the `.svar2` store. See "`genotypes/svar2_ranges/` layout" +below for the on-disk size of this cache; it is not small at cohort scale. ## `metadata.json` schema @@ -87,10 +88,23 @@ Written only when the dataset's variant source is a `.svar2` store. `R` = number | `vk_indel_range.npy` | `(R, S, P, 2)` | Same, for the indel variant-key column. | | `dense_snp_range.npy` | `(R, 2)` | Per-region (sample-independent) range into the dense SNP store. | | `dense_indel_range.npy` | `(R, 2)` | Per-region (sample-independent) range into the dense indel store. | -| `region_starts.npy` | `(R,)` | Per-region write-time start coordinate. Retained for parity/debugging; the read path derives per-query starts from the (post-jitter) query regions and does **not** read this array's values. | | `sample_cols.npy` | `(S,)` | Maps the dataset's selected-sample slot to the `.svar2` store's original sample index. | | `svar2_meta.json` | — | Records each array's `shape`/`dtype` plus `ploidy`. | +`vk_snp_range.npy` and `vk_indel_range.npy` are each +`(regions, samples, ploidy, 2)` int64, so the two together occupy + +``` +2 x regions x samples x ploidy x 2 x 8 bytes +``` + +This grows linearly in **both** the number of BED rows and the number of +selected samples. It is not small at cohort scale: ~4,000 regions over 414,830 +diploid samples is approximately **98 GiB** for a single chromosome/panel. +`gvl.write` logs the projected size before allocating and warns when it exceeds +free disk. Budget disk accordingly, or reduce the region count or sample +selection. + At read time, `Dataset.__getitem__` slices these memmaps (numpy fancy-indexing; no interval search) to build the flat per-query inputs for the read-bound Rust kernels — no interval-search tree and no dense-union rebuild happen per read, unlike the `.svar` path. diff --git a/docs/source/write.md b/docs/source/write.md index 5f20f90b..a9ab61f6 100644 --- a/docs/source/write.md +++ b/docs/source/write.md @@ -107,4 +107,4 @@ gvl.write( Both formats store a back-reference in the dataset's `metadata.json` instead of duplicating per-variant arrays, so the source store must remain accessible when the dataset is later opened with [`gvl.Dataset.open()`](api.md#genvarloader.Dataset.open) (override its location with `svar=`/`svar2=` if it has moved). -`.svar2` additionally produces a small write-time cache under `/genotypes/svar2_ranges/` and reads back through an all-Rust, read-bound path with no interval-search-tree build and no dense-union rebuild per read — see [the FAQ](faq.md) for the read-path and on-disk-size tradeoffs, and [the format reference](format.md) for the on-disk layout. `.svar2` currently has a Phase-1 scope: a handful of output combinations (`annotated` haplotypes, `min_af`/`max_af`, spliced variant-window/track outputs, etc.) aren't wired yet and raise `NotImplementedError` — see the `genvarloader` skill or `format.md` for the full list. Haplotype and `variants` output support splicing and `var_filter="exonic"`. +`.svar2` additionally produces a write-time cache under `/genotypes/svar2_ranges/` and reads back through an all-Rust, read-bound path with no interval-search-tree build and no dense-union rebuild per read — see [the FAQ](faq.md) for the read-path and on-disk-size tradeoffs, and [the format reference](format.md) for the on-disk layout, including the size formula. This cache is **not small at cohort scale**. `gvl.write` honours `max_mem` when writing `.svar2` genotype ranges, bounding the RAM used while producing the cache; the permanent range cache itself is governed by disk space, not `max_mem` (see the format reference). `.svar2` currently has a Phase-1 scope: a handful of output combinations (`annotated` haplotypes, `min_af`/`max_af`, spliced variant-window/track outputs, etc.) aren't wired yet and raise `NotImplementedError` — see the `genvarloader` skill or the format reference for the full list. Haplotype and `variants` output support splicing and `var_filter="exonic"`. diff --git a/docs/superpowers/plans/2026-07-30-svar2-write-memory.md b/docs/superpowers/plans/2026-07-30-svar2-write-memory.md new file mode 100644 index 00000000..203fde6a --- /dev/null +++ b/docs/superpowers/plans/2026-07-30-svar2-write-memory.md @@ -0,0 +1,1969 @@ +# SVAR2 Write-Path Memory & `find_ranges` Complexity Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** Make `gvl.write(..., variants=SparseVar2(...), max_mem=...)` complete at population scale (414k samples, ~4k regions) with bounded memory and visible progress, by fixing genoray's `find_ranges` from `O(regions x total_variants)` to `O(total_variants)` and adding a memory-bounded chunked API. + +**Architecture:** genoray's `find_ranges` currently rebuilds a `SearchTree` and a `v_ends` vector for every `(region, column)` pair. Hoist that region-independent state into a `VkColumnIndex` built once per column, invert the loop to column-outer, and parallelize with rayon over disjoint output slices. Fold per-region max-end computation into the same sweep so GenVarLoader can stop decoding every sample. Expose a chunked Python API (`_find_ranges_chunked`) that yields hap-slices sized from `max_mem`, and have the GenVarLoader writer consume it with per-chunk memmap writes, a disk preflight, and fractional progress. + +**Tech Stack:** Rust (pyo3 0.29, numpy 0.29, ndarray 0.17, rayon 1.11), Python 3.10+ (numpy, polars, loguru, tqdm), pixi for both repos. + +Design spec: `docs/superpowers/specs/2026-07-30-svar2-write-memory-design.md` +Issue: [gvl#333](https://github.com/mcvickerlab/GenVarLoader/issues/333) + +## Global Constraints + +- **Two repositories.** genoray (`/carter/users/dlaub/projects/genoray`) and GenVarLoader. genoray Tasks 1–4 land and release first; GenVarLoader Task 6 depends on that release. +- **Worktrees.** Work in `.claude/worktrees/` under each repo root. genoray needs its own worktree — GenVarLoader's worktree already exists at `.claude/worktrees/issue-333-svar2-write-mem`. +- **Branch targets.** Both repos: `main`. This is the file-backed `gvl.write` path, **not** StreamingDataset-board work, so it does **not** target the `streaming` branch. +- **Conventional Commits** in both repos (commitizen enforces this via a `commit-msg` hook). +- **Never edit genoray's `CHANGELOG.md`** — commitizen owns and regenerates it. +- **Composite max-end key packing:** `key = (pos << 21) | ext`, where `pos` is the 0-based variant position and `ext = 1 + deletion_len` (so `end = pos + ext`). `SHIFT = 21`. A key of `0` means "no variant in this region". This encoding is fixed by existing GenVarLoader behavior at `python/genvarloader/_dataset/_write.py:1101-1120` and must be reproduced exactly. +- **genoray version floor after release:** `genoray>=3.4.0,<4` (a `feat:` commit yields a minor bump from 3.3.0). +- **genoray Rust tests:** `pixi run -e lint test-rust`. Python tests: `pixi run pytest tests/`. After **any** Rust change, rebuild the editable extension with `pixi run maturin develop --release` before running Python tests, or the stale `.so` is imported silently. +- **GenVarLoader tests:** `pixi run -e dev pytest -q`. Before pushing, run the full tree: `pixi run -e dev pytest tests -q`. +- **Pre-commit hooks:** installed in both repos (`prek install`). In a fresh GenVarLoader worktree the `pyrefly-check` hook shells out to `pixi run -e dev`, which provisions a whole dev environment and can take >10 minutes on first run. Either provision the env first or use `SKIP=pyrefly-check` for commits that touch no Python. + +--- + +## File Structure + +**genoray** + +| File | Responsibility | Change | +|---|---|---| +| `src/query/reader.rs` | Per-column search state | Replace `vk_snp_overlap`/`vk_indel_overlap` with `VkColumnIndex` + `vk_snp_index`/`vk_indel_index`; add `max_deletion_len` | +| `src/query/gather.rs` | Batch search core | Add `find_ranges_haps` (column-outer, rayon); rewire `find_ranges` | +| `src/query/union.rs` | Dense union | Add `dense_max_end_keys`; expose `DenseUnion::max_del` | +| `src/py_query_ranges.rs` | pyo3 bindings | Add `find_ranges_header`, `find_ranges_chunk` | +| `python/genoray/_svar2_batch.py` | Python query surface | Add `RangesChunk`, `RangesStream`, `_find_ranges_chunked`, `MAX_END_SHIFT` | +| `tests/test_ranges_split.rs` | Rust core tests | Add complexity + max-end-key tests | +| `tests/test_svar2_ranges.py` | Python API tests | Add chunk-equivalence + max-end tests | + +**GenVarLoader** + +| File | Responsibility | Change | +|---|---|---| +| `python/genvarloader/_dataset/_write.py` | Write pipeline | Add `_svar2_ranges_cache_bytes`, `_svar2_preflight`; rewrite `_write_from_svar2`; delete `_svar2_region_max_ends` | +| `pyproject.toml`, `pixi.toml` | Dependency floor | Bump genoray pin | +| `tests/dataset/test_write_svar2.py` | Write-path tests | Add preflight + chunking tests | +| `docs/source/format.md`, `docs/source/write.md`, `skills/genvarloader/SKILL.md` | User docs | Correct the "small" claim; document `max_mem` for SVAR2 | + +## Task Dependency Graph + +``` +Task 0 (setup) + | + +-- Task 1 -> Task 2 -> Task 3 -> Task 4 [genoray chain] + | + +-- Task 5 [GenVarLoader, independent] + | + +-- Task 7 [docs, independent] + +Task 4 + Task 5 -> Task 6 [GenVarLoader integration] +``` + +**Parallel execution:** Tasks 1–4 (genoray chain), Task 5, and Task 7 are mutually independent and should be dispatched concurrently using superpowers:dispatching-parallel-agents with superpowers:subagent-driven-development. Task 6 waits for Task 4 and Task 5. Use Sonnet or weaker for implementation agents; reserve stronger models for second-pass fixes. + +--- + +### Task 0: Setup — genoray tracking issue and worktree + +**Files:** +- Create: genoray worktree at `/carter/users/dlaub/projects/genoray/.claude/worktrees/issue-333-find-ranges` + +**Interfaces:** +- Produces: a genoray worktree on branch `worktree-issue-333-find-ranges`, and a genoray issue number to reference in commits. + +- [ ] **Step 1: File the genoray tracking issue** + +```bash +cd /carter/users/dlaub/projects/genoray +gh issue create \ + --title "find_ranges rebuilds a SearchTree per (region, column): O(regions x total_variants)" \ + --body "\`query::find_ranges\` loops region-outer / hap-inner and calls \`vk_snp_overlap\` / \`vk_indel_overlap\` per (region, column). Each call rebuilds region-independent state from scratch (\`src/query/reader.rs:244\`, \`:260\`): + +\`\`\`rust +let v_ends: Vec = positions.iter().map(|&p| p + 1).collect(); +let tree = SearchTree::new(positions); +\`\`\` + +\`SearchTree::new\` is O(n) and allocates two Vecs sized to the column, so a batch of R regions over H columns does R*H*2 full tree builds and sweeps the packed store R times instead of once. + +At the scale reported in mcvickerlab/GenVarLoader#333 (3,964 regions, 414,830 samples, ploidy 2) that is 6.6e9 tree builds. The caller sat at 0% for hours before being OOM-killed. + +Fix: hoist the per-column state into a \`VkColumnIndex\` built once per column, invert to column-outer, parallelize with rayon. Also add a chunked API so the R*S*P payload can be produced under a memory budget, and fold per-region max-end computation into the same sweep. + +Cross-repo: mcvickerlab/GenVarLoader#333" +``` + +Record the issue number; the genoray commits below reference it as `#`. + +- [ ] **Step 2: Create the genoray worktree** + +```bash +cd /carter/users/dlaub/projects/genoray +git fetch origin +git worktree add -b worktree-issue-333-find-ranges \ + .claude/worktrees/issue-333-find-ranges origin/main +``` + +- [ ] **Step 3: Install hooks and verify the baseline is green** + +```bash +cd /carter/users/dlaub/projects/genoray/.claude/worktrees/issue-333-find-ranges +pixi run prek-install +pixi run -e lint test-rust 2>&1 | tail -20 +``` + +Expected: all Rust tests pass. If `cargo test` fails to load `libpython`, prepend +`LD_LIBRARY_PATH=$PWD/.pixi/envs/lint/lib` to the command. + +- [ ] **Step 4: Verify the Python baseline is green** + +```bash +pixi run pytest tests/test_svar2_ranges.py tests/test_svar2_batch.py -q 2>&1 | tail -10 +``` + +Expected: all pass. This is the byte-identity oracle for Task 1. + +--- + +### Task 1: Hoist per-column search state; invert `find_ranges` to column-outer + +**Files:** +- Modify: `src/query/reader.rs:241-284` (replace `vk_snp_overlap` / `vk_indel_overlap`) +- Modify: `src/query/gather.rs:338-393` (`find_ranges`) +- Test: `tests/test_ranges_split.rs` + +**Interfaces:** +- Consumes: existing `ContigReader` fields `vk_snp`, `vk_indel`, `vk_indel_max_del`, `ploidy`, `n_samples`; `search::{SearchTree, overlap_range}`; `rvk::deletion_len`. +- Produces: + - `pub(crate) struct VkColumnIndex` with `pub(crate) o0: usize` and `pub(crate) fn overlap(&self, q_start: u32, q_end: u32) -> Range` + - `ContigReader::vk_snp_index(&self, col: usize) -> VkColumnIndex` + - `ContigReader::vk_indel_index(&self, sample: usize, p: usize) -> VkColumnIndex` + - `pub fn find_ranges_haps(reader: &ContigReader, regions: &[(u32, u32)], sample_cols: &[usize], hap_lo: usize, hap_hi: usize, out_snp: &mut [i64], out_indel: &mut [i64])` + - `pub const PAR_COLUMN_THRESHOLD: usize = 64` + - `find_ranges` keeps its exact existing signature and `RangesBundle` return contract. + +- [ ] **Step 1: Write the failing complexity test** + +Add to `tests/test_ranges_split.rs`: + +```rust +/// `find_ranges` must build a bounded number of search trees regardless of how +/// many regions are queried. Before the column-outer rewrite this was +/// O(regions x columns): each `vk_*_overlap` call rebuilt the column's tree. +/// +/// The fixture is deliberately small (2 samples x 2 ploidy = 4 columns, well +/// under `PAR_COLUMN_THRESHOLD`) so the serial path runs on this thread and +/// `search::search_tree_build_count` — a thread-local — stays observable. +#[test] +fn test_find_ranges_tree_builds_do_not_scale_with_regions() { + let tmp = tempdir().unwrap(); + let out = tmp.path().join("out"); + std::fs::create_dir_all(&out).unwrap(); + let reader = synth_reader(&out); + + let one = vec![(0u32, 1_000_000u32)]; + let many: Vec<(u32, u32)> = (0..16).map(|i| (i * 20, i * 20 + 1_000_000)).collect(); + + let b0 = search::search_tree_build_count(); + let _ = find_ranges(&reader, &one, None); + let cost_one = search::search_tree_build_count() - b0; + + let b1 = search::search_tree_build_count(); + let _ = find_ranges(&reader, &many, None); + let cost_many = search::search_tree_build_count() - b1; + + // Per-region dense-union/dense-snp/dense-indel trees are still built once + // per region (3 per region, cheap and cohort-shared). The var_key channels + // — the R*H term that made this O(regions x total_variants) — must not grow. + let dense_growth = 3 * (many.len() - one.len()); + assert!( + cost_many <= cost_one + dense_growth, + "tree builds grew with region count: {cost_one} -> {cost_many} \ + (allowed growth {dense_growth})" + ); +} +``` + +- [ ] **Step 2: Run it to confirm it fails** + +```bash +cd /carter/users/dlaub/projects/genoray/.claude/worktrees/issue-333-find-ranges +pixi run -e lint cargo test --no-default-features --features conversion \ + --test test_ranges_split test_find_ranges_tree_builds_do_not_scale_with_regions -- --nocapture +``` + +Expected: FAIL — "tree builds grew with region count". With 4 columns and 2 channels the current code builds `R*8` var_key trees, so the count grows by 120 while only 45 is allowed. + +- [ ] **Step 3: Add `VkColumnIndex` and the two constructors** + +In `src/query/reader.rs`, delete `vk_snp_overlap` and `vk_indel_overlap` (lines 241–284) and add in their place: + +```rust +/// Region-independent per-column search state for one var_key channel. +/// +/// Built ONCE per column, then queried per region. Hoisting this out of the old +/// per-`(region, column)` `vk_*_overlap` methods is what turns `find_ranges` +/// from O(regions x columns) tree builds into O(columns): the packed store is +/// swept once instead of once per region. +pub(crate) struct VkColumnIndex { + /// Absolute base offset of this column in the channel's packed arrays. + pub(crate) o0: usize, + /// `None` for an empty column — `SearchTree`/`overlap_range` are not + /// defined over an empty position array, matching the old early return. + inner: Option<(SearchTree, Vec)>, + max_del: u32, +} + +impl VkColumnIndex { + /// Absolute `[start, end)` into the channel's packed positions/keys for one + /// region. Every element of the returned range truly overlaps + /// `[q_start, q_end)` — `overlap_range` does the left-overlap sub-scan. + pub(crate) fn overlap(&self, q_start: u32, q_end: u32) -> Range { + let Some((tree, v_ends)) = &self.inner else { + return self.o0..self.o0; + }; + let (s, e) = overlap_range(tree, v_ends, self.max_del, q_start, q_end); + (self.o0 + s)..(self.o0 + e) + } + + /// Absolute index of the highest-position variant overlapping the region, + /// or `None` when the region is empty for this column. Positions are sorted + /// within a column and the range is contiguous, so that is its last element. + pub(crate) fn last_overlapping(&self, q_start: u32, q_end: u32) -> Option { + let r = self.overlap(q_start, q_end); + (r.end > r.start).then(|| r.end - 1) + } +} + +impl ContigReader { + /// SNP-channel column index. SNP `v_end = pos + 1` and `max_region_length = + /// 0`, since a SNP spans exactly one base. + pub(crate) fn vk_snp_index(&self, col: usize) -> VkColumnIndex { + let vk_range = self.vk_snp.column(col); + let (o0, o1) = (vk_range.start, vk_range.end); + let positions = &self.vk_snp.positions()[o0..o1]; + if positions.is_empty() { + return VkColumnIndex { o0, inner: None, max_del: 0 }; + } + let v_ends: Vec = positions.iter().map(|&p| p + 1).collect(); + VkColumnIndex { + o0, + inner: Some((SearchTree::new(positions), v_ends)), + max_del: 0, + } + } + + /// Indel-channel column index for `(sample, p)`. `v_end = pos + 1 + + /// deletion_len(key)`; the search bound is this column's `max_del`. + pub(crate) fn vk_indel_index(&self, sample: usize, p: usize) -> VkColumnIndex { + let col = sample * self.ploidy + p; + let vk_range = self.vk_indel.column(col); + let (o0, o1) = (vk_range.start, vk_range.end); + let positions = &self.vk_indel.positions()[o0..o1]; + if positions.is_empty() { + return VkColumnIndex { o0, inner: None, max_del: 0 }; + } + let keys = &as_u32(&self.vk_indel.keys)[o0..o1]; + let v_ends: Vec = positions + .iter() + .enumerate() + .map(|(i, &pos)| pos + 1 + rvk::deletion_len(keys[i])) + .collect(); + VkColumnIndex { + o0, + inner: Some((SearchTree::new(positions), v_ends)), + max_del: self.vk_indel_max_del[[sample, p]], + } + } +} +``` + +- [ ] **Step 4: Add `find_ranges_haps` and rewire `find_ranges`** + +In `src/query/gather.rs`, add `use rayon::prelude::*;` to the imports, then add before `find_ranges`: + +```rust +/// Below this many columns the serial path runs instead of rayon's: fork/join +/// overhead dominates for small batches, and staying on the caller's thread +/// keeps `search::search_tree_build_count` (a thread-local) observable in tests. +pub const PAR_COLUMN_THRESHOLD: usize = 64; + +/// Fill hap-major `[hap_lo, hap_hi)` slices of the two var_key range channels. +/// +/// `out_snp` / `out_indel` are `(hap_hi - hap_lo, R, 2)` row-major `i64` — one +/// contiguous `R * 2` run per hap, which is exactly what lets rayon hand each +/// column a disjoint `par_chunks_mut` slice. The hap axis indexes the SELECTED +/// samples: hap `h` is `(sample_cols[h / ploidy], h % ploidy)`, matching the +/// sample-major-then-ploid order `find_ranges` has always produced. +/// +/// Column-outer / region-inner, so each column's `VkColumnIndex` is built +/// exactly once. +pub fn find_ranges_haps( + reader: &ContigReader, + regions: &[(u32, u32)], + sample_cols: &[usize], + hap_lo: usize, + hap_hi: usize, + out_snp: &mut [i64], + out_indel: &mut [i64], +) { + let ploidy = reader.ploidy; + let r = regions.len(); + let n_haps = hap_hi - hap_lo; + assert_eq!(out_snp.len(), n_haps * r * 2, "out_snp must be (n_haps, R, 2)"); + assert_eq!(out_indel.len(), n_haps * r * 2, "out_indel must be (n_haps, R, 2)"); + if n_haps == 0 || r == 0 { + return; + } + + let fill = |h_off: usize, snp_row: &mut [i64], indel_row: &mut [i64]| { + let h = hap_lo + h_off; + let s = sample_cols[h / ploidy]; + let p = h % ploidy; + let snp_ix = reader.vk_snp_index(s * ploidy + p); + let indel_ix = reader.vk_indel_index(s, p); + for (ri, &(qs, qe)) in regions.iter().enumerate() { + let a = snp_ix.overlap(qs, qe); + snp_row[ri * 2] = a.start as i64; + snp_row[ri * 2 + 1] = a.end as i64; + let b = indel_ix.overlap(qs, qe); + indel_row[ri * 2] = b.start as i64; + indel_row[ri * 2 + 1] = b.end as i64; + } + }; + + if n_haps < PAR_COLUMN_THRESHOLD { + for (h_off, (snp_row, indel_row)) in out_snp + .chunks_mut(r * 2) + .zip(out_indel.chunks_mut(r * 2)) + .enumerate() + { + fill(h_off, snp_row, indel_row); + } + } else { + out_snp + .par_chunks_mut(r * 2) + .zip(out_indel.par_chunks_mut(r * 2)) + .enumerate() + .for_each(|(h_off, (snp_row, indel_row))| fill(h_off, snp_row, indel_row)); + } +} +``` + +Then replace the `vk_snp_range` / `vk_indel_range` construction in `find_ranges` +(currently `src/query/gather.rs:369-379`) with: + +```rust + // `find_ranges_haps` fills hap-major because that is the layout rayon can + // split into disjoint slices. `RangesBundle` is region-major and is replayed + // by `gather_ranges` unchanged, so transpose here. This costs one extra copy + // of the payload; `find_ranges` is the small-batch read-path entry point, + // while the population-scale writer uses the chunked API and never builds a + // bundle at all. + let mut snp_flat = vec![0i64; h * n_regions * 2]; + let mut indel_flat = vec![0i64; h * n_regions * 2]; + find_ranges_haps( + reader, regions, &sample_cols, 0, h, &mut snp_flat, &mut indel_flat, + ); + + let mut vk_snp_range: Vec> = Vec::with_capacity(n_regions * h); + let mut vk_indel_range: Vec> = Vec::with_capacity(n_regions * h); + for ri in 0..n_regions { + for hh in 0..h { + let k = (hh * n_regions + ri) * 2; + vk_snp_range.push(snp_flat[k] as usize..snp_flat[k + 1] as usize); + vk_indel_range.push(indel_flat[k] as usize..indel_flat[k + 1] as usize); + } + } +``` + +Export the new items from `src/query/mod.rs` alongside the existing `find_ranges` +export: `pub use gather::{..., find_ranges_haps, PAR_COLUMN_THRESHOLD};` + +- [ ] **Step 5: Run the complexity test — expect PASS** + +```bash +pixi run -e lint cargo test --no-default-features --features conversion \ + --test test_ranges_split test_find_ranges_tree_builds_do_not_scale_with_regions -- --nocapture +``` + +Expected: PASS. + +- [ ] **Step 6: Run the full Rust suite — nothing else may change** + +```bash +pixi run -e lint test-rust 2>&1 | tail -20 +``` + +Expected: all pass, including the pre-existing `test_find_ranges_dense_range_matches_overlap_batch`, `test_gather_ranges_reproduces_overlap_batch_field_for_field`, and the `test_readbound_gather.rs` tree-count assertions. These are the byte-identity oracle: `find_ranges` output must be unchanged. + +- [ ] **Step 7: Rebuild and run the Python suite** + +```bash +pixi run maturin develop --release 2>&1 | tail -5 +pixi run pytest tests/test_svar2_ranges.py tests/test_svar2_batch.py \ + tests/test_py_ranges_readbound.py -q 2>&1 | tail -10 +``` + +Expected: all pass. + +- [ ] **Step 8: Commit** + +```bash +git add src/query/reader.rs src/query/gather.rs src/query/mod.rs tests/test_ranges_split.rs +git commit -m "perf(query): build each var_key column's search tree once + +find_ranges looped region-outer and rebuilt a SearchTree plus a v_ends +vector inside vk_snp_overlap/vk_indel_overlap for every (region, column) +pair, making a batch O(regions x total_variants) and sweeping the packed +store once per region. + +Hoist the region-independent state into VkColumnIndex, invert the loop to +column-outer in the new find_ranges_haps, and parallelize with rayon over +disjoint par_chunks_mut slices above PAR_COLUMN_THRESHOLD columns. +find_ranges keeps its exact RangesBundle contract by transposing the +hap-major fill back to region-major. + +Closes # +Relates to mcvickerlab/GenVarLoader#333" +``` + +--- + +### Task 2: Per-region max-end composite keys in the same sweep + +**Files:** +- Modify: `src/query/gather.rs` (`find_ranges_haps` signature and body) +- Modify: `src/query/reader.rs` (`ContigReader::max_deletion_len`) +- Modify: `src/query/union.rs` (`DenseUnion::max_del` accessor, `dense_max_end_keys`) +- Modify: `src/query/mod.rs` (exports) +- Test: `tests/test_ranges_split.rs` + +**Interfaces:** +- Consumes: `VkColumnIndex::last_overlapping`, `ContigReader::{vk_snp, vk_indel, dense_union, dense_view, ploidy}`, `DenseUnion::{refs, src, v_ends}`, `rvk::deletion_len`. +- Produces: + - `pub const MAX_END_SHIFT: u32 = 21;` + - `find_ranges_haps` now **returns** `Vec` of length `regions.len()` — the per-region max composite key over this hap slice. `0` means no variant. Signature otherwise unchanged. + - `pub fn dense_max_end_keys(reader: &ContigReader, regions: &[(u32, u32)], dense_range: &[Range], sample_cols: &[usize], all_samples: bool) -> Vec` + - `ContigReader::max_deletion_len(&self) -> u32` + +- [ ] **Step 1: Write the failing max-end test** + +Add to `tests/test_ranges_split.rs`. `synth_reader` builds chr1 with SNP@100 (S0 hap0), INS@200 (S0 hap1, S1 both), DEL@300 `AT>A` (S0 both, S1 hap1) — so `deletion_len = 1` and the DEL's end is `300 + 1 + 1 = 302`. + +```rust +use genoray_core::query::{dense_max_end_keys, find_ranges_haps, MAX_END_SHIFT}; + +fn unpack_end(key: u64) -> u32 { + ((key >> MAX_END_SHIFT) + (key & ((1 << MAX_END_SHIFT) - 1))) as u32 +} + +/// The per-region max end must be the end of the HIGHEST-POSITION overlapping +/// variant (ties broken by the larger end), not the largest end overall — this +/// is the SVAR1-parity rule GenVarLoader's `_svar2_region_max_ends` implements. +#[test] +fn test_max_end_keys_pick_highest_position_variant() { + let tmp = tempdir().unwrap(); + let out = tmp.path().join("out"); + std::fs::create_dir_all(&out).unwrap(); + let reader = synth_reader(&out); + + // Region covering all three variants; the DEL at 300 is highest-position. + let regions = vec![(0u32, 1_000u32)]; + let sample_cols: Vec = (0..2).collect(); + let h = 2 * reader.ploidy; + let mut snp = vec![0i64; h * 2]; + let mut indel = vec![0i64; h * 2]; + let vk_keys = find_ranges_haps( + &reader, ®ions, &sample_cols, 0, h, &mut snp, &mut indel, + ); + + let dense = reader.dense_union(); + let dense_range: Vec<_> = regions.iter().map(|&(a, b)| dense.overlap(a, b)).collect(); + let dense_keys = + dense_max_end_keys(&reader, ®ions, &dense_range, &sample_cols, true); + + let key = vk_keys[0].max(dense_keys[0]); + assert_ne!(key, 0, "region has variants, so the key must be non-zero"); + assert_eq!(unpack_end(key), 302, "DEL@300 with deletion_len 1 ends at 302"); +} + +/// A region containing only the SNP must report that SNP's end, and an empty +/// region must report the 0 sentinel so the caller keeps its original chromEnd. +#[test] +fn test_max_end_keys_snp_only_and_empty_region() { + let tmp = tempdir().unwrap(); + let out = tmp.path().join("out"); + std::fs::create_dir_all(&out).unwrap(); + let reader = synth_reader(&out); + + let regions = vec![(90u32, 110u32), (900u32, 950u32)]; + let sample_cols: Vec = (0..2).collect(); + let h = 2 * reader.ploidy; + let mut snp = vec![0i64; h * regions.len() * 2]; + let mut indel = vec![0i64; h * regions.len() * 2]; + let vk_keys = find_ranges_haps( + &reader, ®ions, &sample_cols, 0, h, &mut snp, &mut indel, + ); + + let dense = reader.dense_union(); + let dense_range: Vec<_> = regions.iter().map(|&(a, b)| dense.overlap(a, b)).collect(); + let dense_keys = + dense_max_end_keys(&reader, ®ions, &dense_range, &sample_cols, true); + + let k0 = vk_keys[0].max(dense_keys[0]); + assert_eq!(unpack_end(k0), 101, "SNP@100 ends at 101"); + assert_eq!(vk_keys[1].max(dense_keys[1]), 0, "no variants in [900, 950)"); +} + +/// Splitting the hap axis must not change the reduced result — the writer +/// reduces per-chunk keys with an elementwise max. +#[test] +fn test_max_end_keys_reduce_across_hap_slices() { + let tmp = tempdir().unwrap(); + let out = tmp.path().join("out"); + std::fs::create_dir_all(&out).unwrap(); + let reader = synth_reader(&out); + + let regions = vec![(0u32, 1_000u32)]; + let sample_cols: Vec = (0..2).collect(); + let h = 2 * reader.ploidy; + + let mut snp = vec![0i64; h * 2]; + let mut indel = vec![0i64; h * 2]; + let whole = find_ranges_haps( + &reader, ®ions, &sample_cols, 0, h, &mut snp, &mut indel, + ); + + let mut reduced = vec![0u64; 1]; + for lo in (0..h).step_by(1) { + let mut s = vec![0i64; 2]; + let mut i = vec![0i64; 2]; + let part = find_ranges_haps( + &reader, ®ions, &sample_cols, lo, lo + 1, &mut s, &mut i, + ); + reduced[0] = reduced[0].max(part[0]); + } + assert_eq!(whole, reduced); +} +``` + +- [ ] **Step 2: Run to confirm they fail** + +```bash +pixi run -e lint cargo test --no-default-features --features conversion \ + --test test_ranges_split max_end 2>&1 | tail -20 +``` + +Expected: FAIL to compile — `dense_max_end_keys`, `MAX_END_SHIFT` unresolved, and `find_ranges_haps` returns `()`. + +- [ ] **Step 3: Make `find_ranges_haps` return per-region max keys** + +In `src/query/gather.rs`, add the constant and change `find_ranges_haps`: + +```rust +/// Bit width reserved for `ext` in the packed max-end key `(pos << SHIFT) | ext`, +/// where `ext = 1 + deletion_len` so that `end = pos + ext`. Packing the small, +/// bounded `ext` (rather than the absolute end) makes an integer max over the +/// key order by position first and end second — the SVAR1 tie-break rule. Fixed +/// by GenVarLoader's existing `_svar2_region_max_ends`; do not change. +pub const MAX_END_SHIFT: u32 = 21; +``` + +Change the signature's return type to `-> Vec` and the body's tail. The +`fill` closure gains a `&mut [u64]` accumulator: + +```rust + let fill = |h_off: usize, snp_row: &mut [i64], indel_row: &mut [i64], acc: &mut [u64]| { + let h = hap_lo + h_off; + let s = sample_cols[h / ploidy]; + let p = h % ploidy; + let snp_ix = reader.vk_snp_index(s * ploidy + p); + let indel_ix = reader.vk_indel_index(s, p); + let snp_pos = reader.vk_snp.positions(); + let indel_pos = reader.vk_indel.positions(); + let indel_keys = as_u32(&reader.vk_indel.keys); + for (ri, &(qs, qe)) in regions.iter().enumerate() { + let a = snp_ix.overlap(qs, qe); + snp_row[ri * 2] = a.start as i64; + snp_row[ri * 2 + 1] = a.end as i64; + let b = indel_ix.overlap(qs, qe); + indel_row[ri * 2] = b.start as i64; + indel_row[ri * 2 + 1] = b.end as i64; + + // Positions are sorted within a column and the range is contiguous, + // so the last element is the highest-position overlapping variant. + let mut k = 0u64; + if a.end > a.start { + let pos = snp_pos[a.end - 1] as u64; + k = k.max((pos << MAX_END_SHIFT) | 1); // SNP/INS: ext = 1 + } + if b.end > b.start { + let i = b.end - 1; + let pos = indel_pos[i] as u64; + let ext = 1 + rvk::deletion_len(indel_keys[i]) as u64; + k = k.max((pos << MAX_END_SHIFT) | ext); + } + acc[ri] = acc[ri].max(k); + } + }; + + if n_haps < PAR_COLUMN_THRESHOLD { + let mut acc = vec![0u64; r]; + for (h_off, (snp_row, indel_row)) in out_snp + .chunks_mut(r * 2) + .zip(out_indel.chunks_mut(r * 2)) + .enumerate() + { + fill(h_off, snp_row, indel_row, &mut acc); + } + acc + } else { + out_snp + .par_chunks_mut(r * 2) + .zip(out_indel.par_chunks_mut(r * 2)) + .enumerate() + .fold( + || vec![0u64; r], + |mut acc, (h_off, (snp_row, indel_row))| { + fill(h_off, snp_row, indel_row, &mut acc); + acc + }, + ) + .reduce( + || vec![0u64; r], + |mut a, b| { + for i in 0..r { + a[i] = a[i].max(b[i]); + } + a + }, + ) + } +``` + +Also change the two early returns: `if n_haps == 0 || r == 0 { return vec![0u64; r]; }`. + +Add `use crate::rvk;` and the `as_u32` import to `gather.rs` if not already present. + +- [ ] **Step 4: Add `dense_max_end_keys`** + +In `src/query/union.rs`, add a `max_del` accessor on `DenseUnion`: + +```rust +impl DenseUnion { + /// The per-contig dense deletion bound, for the caller's overflow preflight. + pub(crate) fn max_del(&self) -> u32 { + self.max_del + } +} +``` + +Then add, in the same file: + +```rust +/// Per-region max `(pos << MAX_END_SHIFT) | ext` over the DENSE channel, +/// restricted to variants carried by at least one selected hap. `0` when the +/// region has no such variant. +/// +/// The dense genotype matrix is hap-major (`hap * n_dense_variants + col`), so a +/// "is this variant carried by anyone selected?" probe is strided across haps. +/// Two things keep that cheap: +/// +/// * The walk runs BACKWARD from the end of the region's dense window and stops +/// once it drops below the position of the first carried variant it found. +/// Dense variants are common by construction, so this almost always terminates +/// on the first index. +/// * `all_samples` skips the carriage probe entirely: every dense variant in the +/// store has at least one carrier among all samples, so the last truly +/// overlapping variant is the answer. This is the path `gvl.write` takes. +/// +/// The whole tied run at the winning position is scanned rather than stopping at +/// the first hit: within a class the table's order is not by `ext`, so a later +/// same-position variant can carry a longer deletion. +pub fn dense_max_end_keys( + reader: &ContigReader, + regions: &[(u32, u32)], + dense_range: &[Range], + sample_cols: &[usize], + all_samples: bool, +) -> Vec { + let ploidy = reader.ploidy; + let dense = reader.dense_union(); + let mut out = vec![0u64; regions.len()]; + + for (ri, &(qs, _)) in regions.iter().enumerate() { + let (ds, de) = (dense_range[ri].start, dense_range[ri].end); + let mut best = 0u64; + let mut best_pos: Option = None; + let mut j = de; + while j > ds { + j -= 1; + let pos = dense.refs[j].position; + if let Some(bp) = best_pos { + if pos < bp { + break; // every remaining index has a lower position + } + } + if dense.v_ends[j] <= qs { + continue; // no true left-overlap + } + let carried = all_samples || { + let (class, dcol) = dense.src[j]; + let view = reader + .dense_view(class) + .expect("dense src implies table"); + sample_cols + .iter() + .any(|&s| (0..ploidy).any(|p| view.carried(s * ploidy + p, dcol))) + }; + if !carried { + continue; + } + let ext = (dense.v_ends[j] - pos) as u64; + best = best.max(((pos as u64) << MAX_END_SHIFT) | ext); + best_pos = Some(pos); + } + out[ri] = best; + } + out +} +``` + +Add `use crate::query::gather::MAX_END_SHIFT;` to `union.rs`. + +- [ ] **Step 5: Add the overflow preflight accessor** + +In `src/query/reader.rs`: + +```rust +impl ContigReader { + /// The largest deletion span on this contig across both the per-hap indel + /// channel and the dense union. Callers packing max-end keys must check + /// `1 + max_deletion_len() < (1 << MAX_END_SHIFT)` before doing so — a + /// pathological >~2 Mb deletion footprint would otherwise silently corrupt + /// the packed key. + pub fn max_deletion_len(&self) -> u32 { + let vk = self.vk_indel_max_del.iter().copied().max().unwrap_or(0); + vk.max(self.dense_union().max_del()) + } +} +``` + +Export from `src/query/mod.rs`: `pub use gather::MAX_END_SHIFT;` and +`pub use union::dense_max_end_keys;`. + +- [ ] **Step 6: Run the new tests — expect PASS** + +```bash +pixi run -e lint cargo test --no-default-features --features conversion \ + --test test_ranges_split 2>&1 | tail -20 +``` + +Expected: all pass, including Task 1's tests. + +- [ ] **Step 7: Run the full Rust suite** + +```bash +pixi run -e lint test-rust 2>&1 | tail -20 +``` + +Expected: all pass. + +- [ ] **Step 8: Commit** + +```bash +git add src/query/gather.rs src/query/union.rs src/query/reader.rs \ + src/query/mod.rs tests/test_ranges_split.rs +git commit -m "feat(query): compute per-region max-end keys during the range sweep + +find_ranges_haps now also returns the per-region max +(pos << 21) | (1 + deletion_len) composite key over its hap slice, taken +from the last element of each channel's range -- free, since the range was +just computed. dense_max_end_keys does the same for the cohort-shared dense +channel via a backward walk with an all-samples fast path. + +Packed keys (not unpacked ends) are the reduction unit: the SVAR1 rule is +max by position THEN end, so reducing ends across hap slices would pick the +wrong variant when a lower-position deletion reaches further. + +Lets consumers stop decoding every sample just to extend chromEnd. + +Relates to #, mcvickerlab/GenVarLoader#333" +``` + +--- + +### Task 3: Chunked pyo3 bindings + +**Files:** +- Modify: `src/py_query_ranges.rs` +- Test: `tests/test_svar2_ranges.py` + +**Interfaces:** +- Consumes: `find_ranges_haps`, `dense_max_end_keys`, `MAX_END_SHIFT`, `ContigReader::max_deletion_len`, existing `bundle_to_dict` helpers. +- Produces two new `PyContigReader` methods: + - `find_ranges_header(regions, samples) -> dict` with keys `region_starts` (i32, R), `dense_range` (i32, (R,2)), `dense_snp_range` (i32, (R,2)), `dense_indel_range` (i32, (R,2)), `sample_cols` (i64, S), `dense_max_end_keys` (i64, R), `n_regions`, `n_samples`, `ploidy` + - `find_ranges_chunk(regions, samples, hap_lo, hap_hi) -> dict` with keys `vk_snp_range` (i64, (n_haps*R, 2)), `vk_indel_range` (i64, (n_haps*R, 2)), `max_end_keys` (i64, R), `hap_lo`, `hap_hi` + +- [ ] **Step 1: Write the failing Python test** + +Add to `tests/test_svar2_ranges.py`: + +```python +def test_find_ranges_chunk_matches_find_ranges(svar2_store: Path): + """Chunked hap slices must reassemble into the region-major bundle exactly.""" + sv = SparseVar2(svar2_store) + starts, ends = [0, 5], [40, 20] + reg = list(zip(starts, ends)) + reader = sv._reader("chr1") + bundle = sv._find_ranges("chr1", starts, ends) + + R = len(reg) + P = sv.ploidy + S = sv.n_samples + H = S * P + + header = reader.find_ranges_header(reg, None) + np.testing.assert_array_equal( + np.asarray(header["dense_snp_range"]), np.asarray(bundle["dense_snp_range"]) + ) + np.testing.assert_array_equal( + np.asarray(header["sample_cols"]), np.asarray(bundle["sample_cols"]) + ) + + # One hap per call: the most adversarial chunking. + snp = np.empty((H, R, 2), np.int64) + indel = np.empty((H, R, 2), np.int64) + for h in range(H): + d = reader.find_ranges_chunk(reg, None, h, h + 1) + snp[h] = np.asarray(d["vk_snp_range"]).reshape(1, R, 2) + indel[h] = np.asarray(d["vk_indel_range"]).reshape(1, R, 2) + + # bundle vk ranges are region-major (R*H, 2); ours are hap-major (H, R, 2). + np.testing.assert_array_equal( + snp.transpose(1, 0, 2).reshape(R * H, 2), + np.asarray(bundle["vk_snp_range"]), + ) + np.testing.assert_array_equal( + indel.transpose(1, 0, 2).reshape(R * H, 2), + np.asarray(bundle["vk_indel_range"]), + ) +``` + +- [ ] **Step 2: Run to confirm it fails** + +```bash +pixi run pytest tests/test_svar2_ranges.py::test_find_ranges_chunk_matches_find_ranges -q +``` + +Expected: FAIL — `PyContigReader` has no attribute `find_ranges_header`. + +- [ ] **Step 3: Implement the two bindings** + +In `src/py_query_ranges.rs`, extend the imports and add to the `#[pymethods]` +block: + +```rust +use crate::query::{ + BatchResult, MAX_END_SHIFT, RangesBundle, dense_max_end_keys, find_ranges, + find_ranges_haps, gather_ranges, read_ranges, +}; +use pyo3::exceptions::PyValueError; + +// ... inside impl PyContigReader ... + + /// Region-level half of a chunked `find_ranges`: everything whose size is + /// O(regions) rather than O(regions * samples * ploidy), plus the dense + /// channel's max-end contribution. Cheap enough to compute eagerly. + pub fn find_ranges_header<'py>( + &self, + py: Python<'py>, + regions: Vec<(u32, u32)>, + samples: Option>, + ) -> PyResult> { + // Fail fast rather than silently corrupting a packed key. `ext` is + // 1 + deletion_len and must fit below the position field. + let max_del = self.inner.max_deletion_len(); + if (1u64 + max_del as u64) >= (1u64 << MAX_END_SHIFT) { + return Err(PyValueError::new_err( + "variant footprint exceeds tie-break packing width", + )); + } + + let all_samples = samples.is_none(); + let sample_cols: Vec = match &samples { + Some(s) => s.clone(), + None => (0..self.inner.n_samples).collect(), + }; + + let dense = self.inner.dense_union(); + let dense_range: Vec> = regions + .iter() + .map(|&(qs, qe)| dense.overlap(qs, qe)) + .collect(); + let dense_snp_range: Vec> = regions + .iter() + .map(|&(qs, qe)| self.inner.dense_snp_overlap(qs, qe)) + .collect(); + let dense_indel_range: Vec> = regions + .iter() + .map(|&(qs, qe)| self.inner.dense_indel_overlap(qs, qe)) + .collect(); + let region_starts: Vec = regions.iter().map(|&(qs, _)| qs).collect(); + let dmax = dense_max_end_keys( + &self.inner, ®ions, &dense_range, &sample_cols, all_samples, + ); + + let pairs_i32 = |v: &[Range]| -> Vec { + let mut o = Vec::with_capacity(v.len() * 2); + for r in v { + o.push(r.start as i32); + o.push(r.end as i32); + } + o + }; + let to2d = |v: Vec| { + Array2::from_shape_vec((regions.len(), 2), v) + .expect("region pair shape") + .to_pyarray(py) + }; + + let d = PyDict::new(py); + d.set_item("dense_range", to2d(pairs_i32(&dense_range)))?; + d.set_item("dense_snp_range", to2d(pairs_i32(&dense_snp_range)))?; + d.set_item("dense_indel_range", to2d(pairs_i32(&dense_indel_range)))?; + d.set_item("region_starts", u32_to_i32_pyarray(py, ®ion_starts))?; + let cols: Vec = sample_cols.iter().map(|&x| x as i64).collect(); + d.set_item("sample_cols", PyArray1::from_slice(py, &cols))?; + let dmax_i64: Vec = dmax.iter().map(|&x| x as i64).collect(); + d.set_item("dense_max_end_keys", PyArray1::from_slice(py, &dmax_i64))?; + d.set_item("n_regions", regions.len())?; + d.set_item("n_samples", sample_cols.len())?; + d.set_item("ploidy", self.inner.ploidy)?; + Ok(d) + } + + /// One hap slice `[hap_lo, hap_hi)` of a chunked `find_ranges`. Fills freshly + /// allocated numpy arrays IN PLACE, so the payload exists exactly once — + /// unlike `find_ranges`, whose `Vec>` -> `Vec` -> + /// `ToPyArray` chain holds three copies at peak. Releases the GIL for the + /// search so rayon and the caller's progress bar can both run. + /// + /// `vk_snp_range` / `vk_indel_range` come back hap-major, shape + /// `(n_haps * R, 2)`; reshape to `(n_haps_samples, ploidy, R, 2)` in Python. + pub fn find_ranges_chunk<'py>( + &self, + py: Python<'py>, + regions: Vec<(u32, u32)>, + samples: Option>, + hap_lo: usize, + hap_hi: usize, + ) -> PyResult> { + let sample_cols: Vec = match &samples { + Some(s) => s.clone(), + None => (0..self.inner.n_samples).collect(), + }; + let h_total = sample_cols.len() * self.inner.ploidy; + if hap_lo > hap_hi || hap_hi > h_total { + return Err(PyValueError::new_err(format!( + "hap slice [{hap_lo}, {hap_hi}) out of bounds for {h_total} haps" + ))); + } + let n_haps = hap_hi - hap_lo; + let r = regions.len(); + + let snp = PyArray2::::zeros(py, [n_haps * r, 2], false); + let indel = PyArray2::::zeros(py, [n_haps * r, 2], false); + let max_keys = { + let mut snp_rw = snp.readwrite(); + let mut indel_rw = indel.readwrite(); + let snp_s = snp_rw.as_slice_mut()?; + let indel_s = indel_rw.as_slice_mut()?; + py.detach(|| { + find_ranges_haps( + &self.inner, ®ions, &sample_cols, hap_lo, hap_hi, snp_s, indel_s, + ) + }) + }; + + let keys_i64: Vec = max_keys.iter().map(|&x| x as i64).collect(); + let d = PyDict::new(py); + d.set_item("vk_snp_range", snp)?; + d.set_item("vk_indel_range", indel)?; + d.set_item("max_end_keys", PyArray1::from_slice(py, &keys_i64))?; + d.set_item("hap_lo", hap_lo)?; + d.set_item("hap_hi", hap_hi)?; + Ok(d) + } +``` + +**If `py.detach` fails to compile** because the `&mut [i64]` slices are not +`Ungil`: compute into local `Vec`s inside `py.detach`, then copy into the +numpy arrays afterward. That costs one extra chunk-sized copy — still bounded, +still far better than today's three copies of the whole contig. Do not abandon +the GIL release to keep the in-place fill; the GIL release is the more important +of the two. + +- [ ] **Step 4: Rebuild and run the test — expect PASS** + +```bash +pixi run maturin develop --release 2>&1 | tail -5 +pixi run pytest tests/test_svar2_ranges.py::test_find_ranges_chunk_matches_find_ranges -q +``` + +Expected: PASS. + +- [ ] **Step 5: Run the full genoray suite** + +```bash +pixi run -e lint test-rust 2>&1 | tail -10 +pixi run test 2>&1 | tail -15 +``` + +Expected: all pass. + +- [ ] **Step 6: Commit** + +```bash +git add src/py_query_ranges.rs tests/test_svar2_ranges.py +git commit -m "feat(query): add chunked find_ranges bindings + +find_ranges_header returns the O(regions) arrays plus the dense channel's +max-end keys; find_ranges_chunk returns one hap slice of the var_key +payload. The chunk binding fills freshly allocated numpy arrays in place +and releases the GIL, so peak memory is one copy of the chunk instead of +three copies of the whole contig. + +Also preflights the max-end key packing width against the contig's largest +deletion, raising ValueError rather than silently corrupting a key. + +Relates to #, mcvickerlab/GenVarLoader#333" +``` + +--- + +### Task 4: Python `RangesStream` / `_find_ranges_chunked` + +**Files:** +- Modify: `python/genoray/_svar2_batch.py` +- Test: `tests/test_svar2_ranges.py` + +**Interfaces:** +- Consumes: `PyContigReader.find_ranges_header`, `PyContigReader.find_ranges_chunk`. +- Produces (all importable from `genoray._svar2_batch`): + - `MAX_END_SHIFT: int = 21` + - `RangesChunk` frozen dataclass: `sample_start: int`, `n_samples: int`, `vk_snp_range: NDArray[np.int64]` shape `(n_samples, ploidy, R, 2)`, `vk_indel_range` same shape, `max_end_keys: NDArray[np.int64]` shape `(R,)` + - `RangesStream` frozen dataclass: `n_regions`, `n_samples`, `ploidy`, `samples_per_chunk`, `region_starts`, `dense_range`, `dense_snp_range`, `dense_indel_range`, `sample_cols`, `dense_max_end_keys`, `chunks: Iterator[RangesChunk]` + - `SparseVar2._find_ranges_chunked(contig, starts, ends, samples=None, *, max_mem=None) -> RangesStream` + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/test_svar2_ranges.py`: + +```python +import pytest + +from genoray._svar2_batch import MAX_END_SHIFT + + +def _reassemble(stream) -> tuple[np.ndarray, np.ndarray, np.ndarray]: + R, P, S = stream.n_regions, stream.ploidy, stream.n_samples + snp = np.empty((S, P, R, 2), np.int64) + indel = np.empty((S, P, R, 2), np.int64) + keys = stream.dense_max_end_keys.copy() + for ch in stream.chunks: + s0, s1 = ch.sample_start, ch.sample_start + ch.n_samples + snp[s0:s1] = ch.vk_snp_range + indel[s0:s1] = ch.vk_indel_range + np.maximum(keys, ch.max_end_keys, out=keys) + return snp, indel, keys + + +@pytest.mark.parametrize("max_mem", [None, 1 << 30, 1]) +def test_chunked_matches_find_ranges(svar2_store: Path, max_mem): + """Every chunking, including one sample per chunk, reassembles identically.""" + sv = SparseVar2(svar2_store) + starts, ends = [0, 5], [40, 20] + bundle = sv._find_ranges("chr1", starts, ends) + R, P, S = 2, sv.ploidy, sv.n_samples + + if max_mem == 1: + # 1 byte cannot fit a sample; the API must say so rather than silently + # producing a zero-sized chunk. + with pytest.raises(ValueError, match="max_mem"): + sv._find_ranges_chunked("chr1", starts, ends, max_mem=max_mem) + return + + stream = sv._find_ranges_chunked("chr1", starts, ends, max_mem=max_mem) + snp, indel, _ = _reassemble(stream) + np.testing.assert_array_equal( + snp.reshape(S * P, R, 2).transpose(1, 0, 2).reshape(R * S * P, 2), + np.asarray(bundle["vk_snp_range"]), + ) + np.testing.assert_array_equal( + indel.reshape(S * P, R, 2).transpose(1, 0, 2).reshape(R * S * P, 2), + np.asarray(bundle["vk_indel_range"]), + ) + + +def test_chunked_max_end_keys_unpack_to_variant_ends(svar2_store: Path): + """The reduced key unpacks to the end of the highest-position variant. + + The fixture's chr1 carries SNP@2, INS@6 and DEL@11 (ilen -2, so it ends at + 11 + 1 + 2 = 14). Region [0, 40) therefore ends at 14; region [0, 5) sees + only SNP@2, which ends at 3. + """ + sv = SparseVar2(svar2_store) + stream = sv._find_ranges_chunked("chr1", [0, 0], [40, 5]) + _, _, keys = _reassemble(stream) + mask = (1 << MAX_END_SHIFT) - 1 + ends = (keys >> MAX_END_SHIFT) + (keys & mask) + assert keys[0] != 0 and keys[1] != 0 + assert int(ends[0]) == 14 + assert int(ends[1]) == 3 + + +def test_chunked_sample_subset(svar2_store: Path): + """A sample subset takes the carriage-probing dense path, not the fast path.""" + sub = [SparseVar2(svar2_store).available_samples[1]] + sv = SparseVar2(svar2_store) + bundle = sv._find_ranges("chr1", [0], [40], samples=sub) + stream = sv._find_ranges_chunked("chr1", [0], [40], samples=sub) + assert stream.n_samples == 1 + snp, _, _ = _reassemble(stream) + np.testing.assert_array_equal( + snp.reshape(-1, 2), np.asarray(bundle["vk_snp_range"]) + ) +``` + +- [ ] **Step 2: Run to confirm they fail** + +```bash +pixi run pytest tests/test_svar2_ranges.py -q -k chunked +``` + +Expected: FAIL — `SparseVar2` has no attribute `_find_ranges_chunked`. + +- [ ] **Step 3: Implement the Python layer** + +In `python/genoray/_svar2_batch.py`, add near the top: + +```python +from collections.abc import Iterator +from dataclasses import dataclass + +#: Bit width reserved for ``ext`` in a packed max-end key. Mirrors Rust's +#: ``query::MAX_END_SHIFT``; consumers unpack with +#: ``end = (key >> MAX_END_SHIFT) + (key & ((1 << MAX_END_SHIFT) - 1))``. +MAX_END_SHIFT = 21 + + +@dataclass(frozen=True) +class RangesChunk: + """One hap slice of a chunked ``_find_ranges``. + + Attributes: + sample_start: Offset of this chunk on the SELECTED sample axis. + n_samples: Number of selected samples in this chunk. + vk_snp_range: Shape ``(n_samples, ploidy, n_regions, 2)``, hap-major. + vk_indel_range: Shape ``(n_samples, ploidy, n_regions, 2)``, hap-major. + max_end_keys: Shape ``(n_regions,)``. Packed ``(pos << MAX_END_SHIFT) | + ext`` maxima over this chunk's haps; ``0`` means no variant. Reduce + across chunks with an elementwise maximum BEFORE unpacking -- the + ordering rule is position first, end second, so reducing unpacked + ends would pick the wrong variant. + """ + + sample_start: int + n_samples: int + vk_snp_range: "np.ndarray" + vk_indel_range: "np.ndarray" + max_end_keys: "np.ndarray" + + +@dataclass(frozen=True) +class RangesStream: + """Memory-bounded, chunked form of ``_find_ranges``. + + The ``O(n_regions)`` arrays are computed eagerly; the + ``O(n_regions * n_samples * ploidy)`` payload arrives via ``chunks``. + ``n_samples`` is the progress denominator and each ``RangesChunk`` reports + how many samples it advanced by. + """ + + n_regions: int + n_samples: int + ploidy: int + samples_per_chunk: int + region_starts: "np.ndarray" + dense_range: "np.ndarray" + dense_snp_range: "np.ndarray" + dense_indel_range: "np.ndarray" + sample_cols: "np.ndarray" + dense_max_end_keys: "np.ndarray" + chunks: "Iterator[RangesChunk]" +``` + +Then add the method to `_BatchQueryMixin`: + +```python + def _find_ranges_chunked( + self, + contig: str, + starts: "ArrayLike", + ends: "ArrayLike", + samples: "ArrayLike | None" = None, + *, + max_mem: int | None = None, + ) -> RangesStream: + """Chunked, memory-bounded ``_find_ranges``. + + ``starts``/``ends`` and ``samples`` behave as in :meth:`read_ranges`. + + The var_key payload is ``n_regions * n_samples * ploidy * 2`` int64 + pairs per channel, which is tens of GiB at cohort scale. This splits it + along the SAMPLE axis -- not the region axis -- because the search is + column-outer: chunking regions would re-sweep the whole packed store per + chunk, while chunking samples keeps a single sweep. + + Args: + contig: Contig name. + starts: 0-based start positions of the query regions. + ends: 0-based, exclusive end positions of the query regions. + samples: Sample names selecting (and reordering) a subset. + max_mem: Approximate byte budget for one chunk's payload. ``None`` + yields a single chunk covering every sample. + + Returns: + A :class:`RangesStream` whose ``chunks`` generator yields + :class:`RangesChunk` in ascending ``sample_start`` order. + + Raises: + ValueError: If ``max_mem`` cannot fit a single sample's payload, or + if the contig's largest deletion overflows the max-end key + packing width. + """ + reg = self._regions(starts, ends) + sample_idxs = self._sample_idxs(samples) + reader = self._reader(contig) + header = reader.find_ranges_header(reg, sample_idxs) + + n_regions = int(header["n_regions"]) + n_samples = int(header["n_samples"]) + ploidy = int(header["ploidy"]) + + # Both channels, 2 endpoints, int64. The 2x is slop for the transient + # the binding holds while handing the arrays back. + bytes_per_sample = n_regions * ploidy * 2 * 8 * 2 + if max_mem is None: + per = max(n_samples, 1) + else: + per = int(max_mem) // (2 * bytes_per_sample) if bytes_per_sample else n_samples + if per < 1: + raise ValueError( + f"max_mem ({int(max_mem)} bytes) is too small for even one " + f"sample of {n_regions} regions at ploidy {ploidy}: needs at " + f"least {2 * bytes_per_sample} bytes." + ) + per = min(per, max(n_samples, 1)) + + def _gen() -> "Iterator[RangesChunk]": + for s0 in range(0, n_samples, per): + s1 = min(s0 + per, n_samples) + d = reader.find_ranges_chunk( + reg, sample_idxs, s0 * ploidy, s1 * ploidy + ) + cs = s1 - s0 + shape = (cs, ploidy, n_regions, 2) + yield RangesChunk( + sample_start=s0, + n_samples=cs, + vk_snp_range=np.asarray(d["vk_snp_range"]).reshape(shape), + vk_indel_range=np.asarray(d["vk_indel_range"]).reshape(shape), + max_end_keys=np.asarray(d["max_end_keys"], np.int64), + ) + + return RangesStream( + n_regions=n_regions, + n_samples=n_samples, + ploidy=ploidy, + samples_per_chunk=per, + region_starts=np.asarray(header["region_starts"]), + dense_range=np.asarray(header["dense_range"]), + dense_snp_range=np.asarray(header["dense_snp_range"]), + dense_indel_range=np.asarray(header["dense_indel_range"]), + sample_cols=np.asarray(header["sample_cols"]), + dense_max_end_keys=np.asarray(header["dense_max_end_keys"], np.int64), + chunks=_gen(), + ) +``` + +- [ ] **Step 4: Run the tests — expect PASS** + +```bash +pixi run pytest tests/test_svar2_ranges.py -q +``` + +Expected: all pass. + +- [ ] **Step 5: Lint and typecheck** + +```bash +pixi run -e lint ruff check python/genoray tests +pixi run -e lint ruff format --check python/genoray tests +pixi run typecheck +``` + +Expected: clean. + +- [ ] **Step 6: Run the full genoray suite** + +```bash +pixi run test 2>&1 | tail -15 +pixi run -e lint test-rust 2>&1 | tail -10 +``` + +Expected: all pass. + +- [ ] **Step 7: Commit and push** + +```bash +git add python/genoray/_svar2_batch.py tests/test_svar2_ranges.py +git commit -m "feat(svar2): add _find_ranges_chunked memory-bounded stream API + +Returns a RangesStream: the O(regions) arrays eagerly, plus a generator of +per-sample-slice RangesChunk sized from max_mem. Chunking is along the +sample axis rather than the region axis because the search is column-outer +-- region chunks would re-sweep the packed store once per chunk. + +Callers reduce max_end_keys across chunks with an elementwise maximum and +unpack once at the end. + +Relates to #, mcvickerlab/GenVarLoader#333" +git push -u origin worktree-issue-333-find-ranges +``` + +- [ ] **Step 8: Open the genoray PR** + +```bash +gh pr create --draft --base main \ + --title "perf(query): make find_ranges O(total_variants) and add a chunked API" \ + --body "Fixes the O(regions x total_variants) tree rebuild in \`find_ranges\`, adds a memory-bounded chunked API, and folds per-region max-end computation into the same sweep. + +- Hoist per-column search state into \`VkColumnIndex\`; invert \`find_ranges\` to column-outer; parallelize with rayon. Tree builds drop from R*H*2 to H*2. +- \`find_ranges_haps\` also returns per-region packed max-end keys, so consumers no longer decode every sample to extend chromEnd. +- \`_find_ranges_chunked\` yields hap slices sized from \`max_mem\`; the chunk binding fills numpy in place and releases the GIL. + +Closes # +Unblocks mcvickerlab/GenVarLoader#333 + +🤖 Generated with [Claude Code](https://claude.com/claude-code)" +``` + +After review and merge, confirm the released version is >= 3.4.0 before Task 6. + +--- + +### Task 5: GenVarLoader preflight and `max_mem` plumbing + +**Files:** +- Modify: `python/genvarloader/_dataset/_write.py:324-328` (call site), `:1124-1135` (signature) +- Test: `tests/dataset/test_write_svar2.py` + +**Interfaces:** +- Consumes: `write()`'s existing `effective_max_mem` local; `loguru.logger`; `shutil` (already imported at `_write.py:4`). +- Produces: + - `_svar2_ranges_cache_bytes(n_regions: int, n_samples: int, ploidy: int) -> int` + - `_svar2_preflight(out_dir: Path, n_regions: int, n_samples: int, ploidy: int) -> int` — logs and warns, returns the byte count + - `_write_from_svar2(path, bed, svar2, samples, extend_to_length, max_mem)` — new trailing `max_mem: int` parameter + +This task does **not** depend on the genoray release; it only changes GenVarLoader-internal plumbing. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/dataset/test_write_svar2.py`: + +```python +def test_svar2_ranges_cache_bytes(): + """Both var-key channels: 2 * R * S * P * 2 endpoints * 8 bytes.""" + from genvarloader._dataset._write import _svar2_ranges_cache_bytes + + assert _svar2_ranges_cache_bytes(1, 1, 2) == 2 * 1 * 1 * 2 * 2 * 8 + # The scale from gvl#333: ~98 GiB for one chromosome/panel. + big = _svar2_ranges_cache_bytes(3964, 414830, 2) + assert 90 * 1024**3 < big < 110 * 1024**3 + + +def test_svar2_preflight_warns_when_disk_is_short(tmp_path, monkeypatch): + """A projected cache larger than free space must warn, not silently proceed.""" + from collections import namedtuple + + from loguru import logger + + from genvarloader._dataset import _write + + Usage = namedtuple("Usage", "total used free") + msgs: list[str] = [] + sink = logger.add(lambda m: msgs.append(str(m)), level="WARNING") + try: + monkeypatch.setattr( + _write.shutil, "disk_usage", lambda p: Usage(total=1000, used=999, free=1) + ) + n = _write._svar2_preflight(tmp_path, 3964, 414830, 2) + finally: + logger.remove(sink) + + assert n == _write._svar2_ranges_cache_bytes(3964, 414830, 2) + assert any("free" in m for m in msgs), msgs +``` + +- [ ] **Step 2: Run to confirm they fail** + +```bash +cd /carter/users/dlaub/projects/GenVarLoader/.claude/worktrees/issue-333-svar2-write-mem +pixi run -e dev pytest tests/dataset/test_write_svar2.py -q -k "cache_bytes or preflight" +``` + +Expected: FAIL — `cannot import name '_svar2_ranges_cache_bytes'`. + +- [ ] **Step 3: Add the two helpers** + +In `python/genvarloader/_dataset/_write.py`, immediately above `_write_from_svar2`: + +```python +def _svar2_ranges_cache_bytes(n_regions: int, n_samples: int, ploidy: int) -> int: + """Permanent on-disk size of the two ``svar2_ranges`` var-key caches. + + Each of ``vk_snp_range`` and ``vk_indel_range`` is a + ``(regions, samples, ploidy, 2)`` int64 array. These are NOT small: one + chromosome of a 414k-sample cohort over ~4k regions is ~98 GiB. + + Args: + n_regions: Number of BED rows in the dataset. + n_samples: Number of selected samples. + ploidy: Ploidy of the variant source. + + Returns: + Total bytes both channels will occupy on disk. + """ + return 2 * n_regions * n_samples * ploidy * 2 * 8 + + +def _svar2_preflight( + out_dir: Path, n_regions: int, n_samples: int, ploidy: int +) -> int: + """Log the projected ``svar2_ranges`` cache size and warn if disk is short. + + Warns rather than raising: free-space reporting is unreliable on some + network filesystems, and a false refusal would block a valid large build. + + Args: + out_dir: Directory the cache will be written to. + n_regions: Number of BED rows in the dataset. + n_samples: Number of selected samples. + ploidy: Ploidy of the variant source. + + Returns: + Projected total bytes of the two var-key caches. + """ + n_bytes = _svar2_ranges_cache_bytes(n_regions, n_samples, ploidy) + logger.info( + f"svar2 range cache: {format_memory(n_bytes)} for {n_regions} regions " + f"x {n_samples} samples x ploidy {ploidy}." + ) + try: + free = shutil.disk_usage(out_dir).free + except OSError: + return n_bytes + if n_bytes > free: + logger.warning( + f"svar2 range cache needs {format_memory(n_bytes)} but only " + f"{format_memory(free)} is free at {out_dir}. The write will likely " + f"fail with ENOSPC." + ) + return n_bytes +``` + +- [ ] **Step 4: Run the tests — expect PASS** + +```bash +pixi run -e dev pytest tests/dataset/test_write_svar2.py -q -k "cache_bytes or preflight" +``` + +Expected: PASS. + +- [ ] **Step 5: Plumb `max_mem` and call the preflight** + +Change the signature at `_write.py:1124`: + +```python +def _write_from_svar2( + path: Path, + bed: pl.DataFrame, + svar2: SparseVar2, + samples: list[str], + extend_to_length: bool, + max_mem: int, +) -> tuple[pl.DataFrame, Svar2Link]: +``` + +Change the call site at `_write.py:324-327`: + +```python + elif isinstance(variants, SparseVar2): + gvl_bed, _svar2_link = _write_from_svar2( + path, + gvl_bed, + variants, + samples, + extend_to_length, + effective_max_mem, + ) +``` + +Immediately after the `R, S, P = bed.height, len(samples), svar2.ploidy` line in +`_write_from_svar2`, and before the first `np.memmap(...)` call, insert: + +```python + _svar2_preflight(out_dir, R, S, P) +``` + +- [ ] **Step 6: Run the SVAR2 write tests** + +```bash +pixi run -e dev pytest tests/dataset/test_write_svar2.py -q +``` + +Expected: all pass — behavior is unchanged, `max_mem` is accepted but not yet consumed. + +- [ ] **Step 7: Lint** + +```bash +pixi run -e dev ruff check python/ tests/ +pixi run -e dev ruff format --check python/ tests/ +``` + +Expected: clean. + +- [ ] **Step 8: Commit** + +```bash +git add python/genvarloader/_dataset/_write.py tests/dataset/test_write_svar2.py +git commit -m "feat(write): preflight the svar2 range cache and accept max_mem + +_write_from_svar2 now receives write()'s effective_max_mem (it previously +got no memory budget at all) and logs the projected on-disk size of the two +var-key range caches before allocating them, warning when it exceeds free +space. At 414k samples over ~4k regions that projection is ~98 GiB. + +max_mem is accepted but not yet consumed; the chunked consumption lands +with the genoray 3.4 API. + +Relates to #333" +``` + +--- + +### Task 6: Consume the chunked stream; delete `_svar2_region_max_ends` + +**Files:** +- Modify: `python/genvarloader/_dataset/_write.py:1067-1121` (delete `_svar2_region_max_ends`), `:1178-1200` (contig loop) +- Modify: `pyproject.toml:15`, `pixi.toml:107` (genoray pin) +- Test: `tests/dataset/test_write_svar2.py` + +**Interfaces:** +- Consumes: `SparseVar2._find_ranges_chunked` and `genoray._svar2_batch.MAX_END_SHIFT` from Task 4; `_svar2_preflight` from Task 5. +- Produces: no new public symbols. `_svar2_region_max_ends` is removed. + +**Prerequisite:** genoray >= 3.4.0 must be installed in the dev env. + +- [ ] **Step 1: Write the failing tests** + +Add to `tests/dataset/test_write_svar2.py`: + +```python +def test_write_svar2_chunked_matches_unchunked( + svar2_store: Path, vcf_and_ref, tmp_path +): + """A tiny max_mem must force multiple chunks and produce identical output.""" + from genoray import SparseVar2 + + _, ref = vcf_and_ref + bed = pl.DataFrame( + {"chrom": ["chr1", "chr1"], "chromStart": [0, 5], "chromEnd": [20, 30]} + ) + + calls: list[int] = [] + real = SparseVar2._find_ranges_chunked + + def spy(self, *args, **kwargs): + stream = real(self, *args, **kwargs) + calls.append(stream.samples_per_chunk) + return stream + + big = tmp_path / "big.gvl" + gvl.write(big, bed, SparseVar2(svar2_store), reference=ref, max_mem="4g") + + SparseVar2._find_ranges_chunked = spy + try: + small = tmp_path / "small.gvl" + # 1 region-sample of payload is R*P*2*8*2 = 64 bytes; 2x slop -> 128. + gvl.write(small, bed, SparseVar2(svar2_store), reference=ref, max_mem=128) + finally: + SparseVar2._find_ranges_chunked = real + + assert calls and all(c == 1 for c in calls), ( + f"expected one sample per chunk under a 128-byte budget, got {calls}" + ) + + for name in ( + "vk_snp_range.npy", + "vk_indel_range.npy", + "dense_snp_range.npy", + "dense_indel_range.npy", + "sample_cols.npy", + ): + a = (big / "genotypes" / "svar2_ranges" / name).read_bytes() + b = (small / "genotypes" / "svar2_ranges" / name).read_bytes() + assert a == b, name + + ra = pl.read_ipc(big / "input_regions.arrow") + rb = pl.read_ipc(small / "input_regions.arrow") + assert ra["chromEnd"].to_list() == rb["chromEnd"].to_list() + + +def test_write_svar2_max_ends_extend_chromend(svar2_store: Path, vcf_and_ref, tmp_path): + """chromEnd must extend past a deletion that starts inside the region. + + The fixture's DEL is at 0-based POS 11 with ilen -2, so it ends at 14. A + region of [0, 12) must be extended to 14. + """ + from genoray import SparseVar2 + + _, ref = vcf_and_ref + bed = pl.DataFrame({"chrom": ["chr1"], "chromStart": [0], "chromEnd": [12]}) + out = tmp_path / "ext.gvl" + gvl.write(out, bed, SparseVar2(svar2_store), reference=ref, max_mem="1g") + regions = pl.read_ipc(out / "input_regions.arrow") + assert regions["chromEnd"].to_list() == [14] +``` + +- [ ] **Step 2: Bump the genoray pin and install** + +In `pyproject.toml:15`: `"genoray>=3.4.0,<4",` +In `pixi.toml:107`: `genoray = ">=3.4.0,<4"` + +```bash +pixi install -e dev 2>&1 | tail -5 +pixi run -e dev python -c "import genoray; print(genoray.__version__ if hasattr(genoray,'__version__') else 'ok'); from genoray import SparseVar2; assert hasattr(SparseVar2, '_find_ranges_chunked')" +``` + +Expected: no AssertionError. + +- [ ] **Step 3: Run to confirm the tests fail** + +```bash +pixi run -e dev pytest tests/dataset/test_write_svar2.py -q -k "chunked or max_ends_extend" +``` + +Expected: FAIL — `test_write_svar2_chunked_matches_unchunked` fails on `assert calls`, because `_write_from_svar2` still calls `_find_ranges`, not `_find_ranges_chunked`. + +- [ ] **Step 4: Delete `_svar2_region_max_ends`** + +Remove the entire function at `python/genvarloader/_dataset/_write.py:1067-1121`. + +- [ ] **Step 5: Rewrite the contig loop** + +Replace the loop body in `_write_from_svar2` (currently `_write.py:1178-1200`) with: + +```python + max_ends = np.empty(R, np.int32) + contig_offset = 0 + pbar = tqdm(total=R, unit=" region") + for (c,), df in bed.partition_by( + "chrom", as_dict=True, maintain_order=True + ).items(): + c = cast(str, c) + pbar.set_description(f"Processing svar2 ranges for {df.height} regions on {c}") + lo, hi = contig_offset, contig_offset + df.height + rc = df.height + starts = df["chromStart"].to_numpy() + ends = df["chromEnd"].to_numpy() + # extend_to_length is validated at function entry (False raises); the + # read-bound kernel sizes haplotype output at read time. + stream = svar2._find_ranges_chunked( + c, starts, ends, samples=samples, max_mem=max_mem + ) + dense_snp[lo:hi] = np.asarray(stream.dense_snp_range, np.int64).reshape(rc, 2) + dense_indel[lo:hi] = np.asarray(stream.dense_indel_range, np.int64).reshape( + rc, 2 + ) + + # Packed (pos << SHIFT) | ext keys, NOT unpacked ends: SVAR1 parity picks + # the highest-POSITION variant (ties by end), so a lower-position variant + # with a longer deletion must not win the cross-chunk reduction. + keys = stream.dense_max_end_keys.copy() + for ch in stream.chunks: + s0, s1 = ch.sample_start, ch.sample_start + ch.n_samples + # Chunks are hap-major (samples, ploidy, regions, 2); the cache is + # region-major. transpose() is a view -- numpy copies straight into + # the memmap with no intermediate array. + vk_snp[lo:hi, s0:s1] = ch.vk_snp_range.transpose(2, 0, 1, 3) + vk_indel[lo:hi, s0:s1] = ch.vk_indel_range.transpose(2, 0, 1, 3) + np.maximum(keys, ch.max_end_keys, out=keys) + # Bound the dirty page cache: at cohort scale these memmaps are tens + # of GiB and the kernel would otherwise reclaim at unpredictable times. + vk_snp.flush() + vk_indel.flush() + pbar.update(rc * ch.n_samples / S) + + mask = (1 << MAX_END_SHIFT) - 1 + region_ends = np.asarray(ends, np.int64).copy() + has = keys > 0 # 0 is the "no variant in this region" sentinel + region_ends[has] = (keys[has] >> MAX_END_SHIFT) + (keys[has] & mask) + max_ends[lo:hi] = region_ends.astype(np.int32) + + contig_offset += df.height + pbar.close() +``` + +Add the import near the other genoray imports at the top of `_write.py`: + +```python +from genoray._svar2_batch import MAX_END_SHIFT +``` + +- [ ] **Step 6: Run the SVAR2 write tests — expect PASS** + +```bash +pixi run -e dev pytest tests/dataset/test_write_svar2.py -q +``` + +Expected: all pass. + +- [ ] **Step 7: Verify no stale references to the deleted helper** + +```bash +rg -n "_svar2_region_max_ends" python/ tests/ docs/ +``` + +Expected: no matches. + +- [ ] **Step 8: Run the full tree** + +```bash +pixi run -e dev ruff check python/ tests/ +pixi run -e dev ruff format --check python/ tests/ +pixi run -e dev typecheck +pixi run -e dev pytest tests -q 2>&1 | tail -20 +``` + +Expected: all clean and passing. A scoped run would miss `tests/unit/`, which this change's deleted symbol could reach. + +- [ ] **Step 9: Commit** + +```bash +git add python/genvarloader/_dataset/_write.py tests/dataset/test_write_svar2.py \ + pyproject.toml pixi.toml +git commit -m "fix(write): bound svar2 genotype-writing memory with max_mem + +_write_from_svar2 called _find_ranges once per contig, materializing two +O(regions x samples x ploidy) int64 arrays plus Rust and numpy transients -- +~245 GiB at 414k samples -- and then called _svar2_region_max_ends, which +decoded ALL samples for the whole contig regardless of the caller's +selection. Together those OOM-killed a real All of Us chr22 build. + +Consume genoray 3.4's _find_ranges_chunked instead: per-sample-slice chunks +sized from max_mem, written straight into the memmaps via a transposed +view, flushed per chunk. max_ends now comes from the same sweep as packed +composite keys, reduced across chunks and unpacked once, so no decode pass +happens at all. + +Progress is now fractional within a contig, so a single-contig BED no longer +sits at 0% for the entire run. + +Closes #333" +``` + +- [ ] **Step 10: Push and open the PR** + +```bash +git push -u origin worktree-issue-333-svar2-write-mem +gh pr create --draft --base main \ + --title "fix(write): bound SVAR2 genotype-writing memory with max_mem" \ + --body "Closes #333. + +Depends on genoray >= 3.4.0 (d-laub/genoray#). + +Three defects, only one named in the issue: + +1. genoray \`find_ranges\` was O(regions x total_variants) -- a \`SearchTree\` was rebuilt per (region, column). Fixed in genoray 3.4. +2. \`_svar2_region_max_ends\` decoded ALL samples for the whole contig, ignoring the sample selection. Deleted; genoray now returns packed max-end keys from the range sweep. +3. \`_write_from_svar2\` received no \`max_mem\` and materialized a whole contig's ranges at once. Now consumes the chunked stream with per-chunk memmap writes and flushes. + +Also preflights and logs the ~98 GiB permanent range cache, and makes progress fractional within a contig. + +🤖 Generated with [Claude Code](https://claude.com/claude-code)" +``` + +--- + +### Task 7: Documentation + +**Files:** +- Modify: `docs/source/format.md` (the `genotypes/svar2_ranges` section) +- Modify: `docs/source/write.md` and the `gvl.write` docstring at `python/genvarloader/_dataset/_write.py:141-148` +- Modify: `skills/genvarloader/SKILL.md` + +This task is independent of Tasks 1–6 and can run concurrently. + +- [ ] **Step 1: Find the false "small" claim** + +```bash +cd /carter/users/dlaub/projects/GenVarLoader/.claude/worktrees/issue-333-svar2-write-mem +rg -n "small" docs/source/format.md +rg -n "svar2_ranges" -A 25 docs/source/format.md +``` + +- [ ] **Step 2: Correct `format.md`** + +Replace the sentence describing the per-`(region, sample, ploidy)` arrays as +"small" with: + +```markdown +`vk_snp_range.npy` and `vk_indel_range.npy` are each +`(regions, samples, ploidy, 2)` int64, so the two together occupy + +``` +2 x regions x samples x ploidy x 2 x 8 bytes +``` + +This grows linearly in **both** the number of BED rows and the number of +selected samples. It is not small at cohort scale: ~4,000 regions over 414,830 +diploid samples is approximately **98 GiB** for a single chromosome/panel. +`gvl.write` logs the projected size before allocating and warns when it exceeds +free disk. Budget disk accordingly, or reduce the region count or sample +selection. +``` + +- [ ] **Step 3: Update the `max_mem` docstring** + +At `python/genvarloader/_dataset/_write.py:141-148`, extend the `max_mem` +description with: + +``` + For a ``.svar2`` variant source this also bounds the genotype + range-cache write: ranges are produced in per-sample chunks sized to + fit the budget rather than a whole contig at once. +``` + +- [ ] **Step 4: Update `write.md`** + +```bash +rg -n "max_mem" docs/source/write.md +``` + +Add, in the `max_mem` discussion, a sentence noting that the SVAR2 genotype +branch honours it as of this release, and that the permanent range cache is +governed by disk (see `format.md`), not `max_mem`. + +- [ ] **Step 5: Update the skill** + +In `skills/genvarloader/SKILL.md`, under the `gvl.write` section add a `max_mem` +note, and under "Common gotchas" add: + +```markdown +- **SVAR2 range caches scale with `regions x samples x ploidy`.** `gvl.write` + with a `.svar2` source writes a permanent + `2 x regions x samples x ploidy x 2 x 8` byte cache under + `genotypes/svar2_ranges/`. That is ~98 GiB for ~4,000 regions over 414,830 + diploid samples. `max_mem` bounds RAM during the write; it does not bound this + on-disk cache. +``` + +- [ ] **Step 6: Verify the api.md/`__all__` invariant still holds** + +```bash +pixi run -e dev python -c "import re,genvarloader as g; api=open('docs/source/api.md').read(); print('MISSING:', [n for n in g.__all__ if n not in api] or 'none')" +``` + +Expected: `MISSING: none`. This change adds no public symbols, so this is a +regression check rather than an edit. + +- [ ] **Step 7: Build the docs** + +```bash +pixi run -e docs doc 2>&1 | tail -20 +``` + +Expected: builds without new warnings. + +- [ ] **Step 8: Commit** + +```bash +git add docs/source/format.md docs/source/write.md skills/genvarloader/SKILL.md \ + python/genvarloader/_dataset/_write.py +git commit -m "docs: correct the svar2 range-cache size claim + +format.md described the per-(region, sample, ploidy) range arrays as +'small'. They are 2 * R * S * P * 2 * 8 bytes -- ~98 GiB for one chromosome +of a 414k-sample cohort. Give the formula, a worked population-scale +example, and the disk-vs-max_mem distinction, and note that max_mem now +governs the SVAR2 genotype write. + +Relates to #333" +``` + +--- + +## Follow-up issues to file after merge + +- [ ] **genoray/GenVarLoader:** shrink the on-disk SVAR2 range cache below 16 bytes/entry (e.g. `start: int64` + `len: int32`). Format change, needs read-path and version-compat work. +- [ ] **GenVarLoader:** `_write_from_svar` (SVAR1) also ignores `max_mem`. Different mechanism (`_find_starts_ends_with_length(..., out=)`), no transient amplification, so lower priority. + +--- + +## Self-Review + +**Spec coverage** + +| Spec section | Task | +|---|---| +| genoray Rust core — hoist per-column index | Task 1 | +| genoray Rust core — column-outer `find_ranges_haps` + rayon | Task 1 | +| genoray Rust core — output order / region-major bundle preserved | Task 1, Step 4 | +| genoray chunked Python API — `RangesStream`/`RangesChunk` | Task 4 | +| genoray chunked Python API — chunk sizing from `max_mem` | Task 4, Step 3 | +| genoray chunked Python API — in-place numpy fill, GIL release | Task 3, Step 3 | +| genoray `max_ends` — vk channels | Task 2, Step 3 | +| genoray `max_ends` — dense channel backward walk + `samples=None` fast path | Task 2, Step 4 | +| genoray `max_ends` — packing-width overflow guard | Task 2 Step 5 + Task 3 Step 3 | +| GenVarLoader — preflight | Task 5 | +| GenVarLoader — `max_mem` plumbing | Task 5, Step 5 | +| GenVarLoader — per-contig chunked loop, flush, fractional progress | Task 6, Step 5 | +| GenVarLoader — delete `_svar2_region_max_ends` | Task 6, Step 4 | +| Testing — loop-inversion byte-identity guard | Task 1, Steps 6–7 | +| Testing — complexity regression guard via `TREE_BUILDS` | Task 1, Step 1 | +| Testing — chunk equivalence property | Task 4, Step 1 | +| Testing — `max_ends` parity | Task 2 Step 1, Task 4 Step 1, Task 6 Step 1 | +| Testing — `samples` subset (non-fast-path dense walk) | Task 4, Step 1 | +| Testing — GenVarLoader scale guard (#333 §4) | Task 6, Step 1 | +| Docs — `format.md`, `write.md`, SKILL.md, CHANGELOG | Task 7 | +| Follow-up issues | end of plan | + +No spec requirement is unassigned. + +**Deviations from the spec, deliberate** + +1. **`max_ends` is carried as packed composite keys, not unpacked ends.** The spec's field names were `max_ends: NDArray[np.int32]` / `dense_max_ends`. Reducing unpacked ends across hap chunks is *wrong*: the SVAR1 rule orders by position first and end second, so a lower-position variant with a longer deletion would win a naive `np.maximum` over ends. The fields are therefore `max_end_keys` / `dense_max_end_keys` (int64 packed), reduced with `np.maximum` and unpacked once by the consumer. The spec has been amended to match. +2. **`find_ranges` gains one transposed copy of its payload.** The spec claimed the region-major reorder would be free inside `bundle_to_dict`. Keeping the rayon-safe hap-major fill means `find_ranges` transposes into its region-major `Vec>`. This only affects the small-batch read path; the population-scale writer uses the chunked API and never builds a bundle. Noted in the Task 1 code comment and amended in the spec. +3. **`PAR_COLUMN_THRESHOLD` was not in the spec.** Small batches stay serial so rayon's fork/join overhead is avoided *and* so `search::search_tree_build_count` — a thread-local that existing tests already depend on — stays observable from the caller's thread. Without this the complexity guard test could not be written. + +**Placeholder scan:** no TBD/TODO. `#` is a deliberate, explained substitution produced by Task 0, Step 1. + +**Type consistency:** `find_ranges_haps` returns `Vec` after Task 2 and every later call site (Task 3's binding, Task 2's tests) uses that return. `MAX_END_SHIFT` is `u32` in Rust and `int` in Python, both 21. `RangesChunk.vk_snp_range` is `(n_samples, ploidy, R, 2)` at its definition (Task 4) and is transposed with `(2, 0, 1, 3)` at its only consumer (Task 6) into `(R, n_samples, ploidy, 2)`, matching the `vk_snp[lo:hi, s0:s1]` memmap slice. `_svar2_preflight` returns `int` and Task 5's test asserts against `_svar2_ranges_cache_bytes`. diff --git a/docs/superpowers/specs/2026-07-30-svar2-write-memory-design.md b/docs/superpowers/specs/2026-07-30-svar2-write-memory-design.md new file mode 100644 index 00000000..7c3cb413 --- /dev/null +++ b/docs/superpowers/specs/2026-07-30-svar2-write-memory-design.md @@ -0,0 +1,367 @@ +# SVAR2 write path: bound memory, fix `find_ranges` complexity, report progress + +Design for [gvl#333](https://github.com/mcvickerlab/GenVarLoader/issues/333). +Cross-cutting: genoray and GenVarLoader each get a PR. + +## Problem + +`gvl.write(..., variants=SparseVar2(...), max_mem=...)` was SIGKILLed (exit 137) +after hours at 0% progress on an All of Us chr22 build: 414,830 samples, ~3,964 +MANE Select CDS regions. `max_mem` never reaches the SVAR2 genotype branch. + +Investigation found three independent defects, only one of which the issue names. + +### 1. `find_ranges` is `O(regions x total_variants)` (genoray, dominant) + +`query::find_ranges` (`src/query/gather.rs`) loops region-outer, hap-inner and +calls `reader.vk_snp_overlap(col, qs, qe)` / `vk_indel_overlap(...)` per +`(region, column)`. Each call rebuilds region-independent state from scratch +(`src/query/reader.rs:244`, `:260`): + +```rust +let v_ends: Vec = positions.iter().map(|&p| p + 1).collect(); +let tree = SearchTree::new(positions); +``` + +`SearchTree::new` is `O(n)` and allocates two `Vec`s sized to the column. At +3,964 regions x 414,830 samples x 2 ploidy x 2 channels that is **6.6 billion +full tree builds**, and the vk store is swept 3,964 times instead of once. + +This is why the job sat at 0% for hours. Bounding memory alone would convert an +OOM into a job that never finishes. + +### 2. `_svar2_region_max_ends` decodes every sample (GenVarLoader) + +`python/genvarloader/_dataset/_write.py:1067` calls +`svar2.decode(contig, all_regions)`, which materializes shape +`(R, S_all, P, None)` — **all 414,830 samples**, ignoring the caller's `samples` +selection, filtering only afterward. That is the full genotype content of chr22 +in RAM plus a `R*S_all*P+1` int64 offsets array (~26 GB). An independent, and +arguably larger, OOM than the one reported. + +### 3. `find_ranges` triple-copies its payload (genoray) + +`bundle_to_dict` (`src/py_query_ranges.rs:78`) flattens `Vec>` into +a `Vec` and then `ToPyArray`s it, so each 49 GiB channel exists roughly 3x +at peak (Rust ranges + flattened vec + numpy destination). The binding also +never releases the GIL. + +Compounding all three: `max_mem` is parsed in `write()` but the SVAR2 branch is +called without it, and the progress bar advances only once per contig, so a +single-contig BED shows 0% for the entire run. + +## Non-goals + +- Shrinking the permanent on-disk range cache below 16 bytes/entry. It is + ~98 GiB for the reported input (`2 channels x R x S x P x 2 x 8`). This design + preflights and reports it; narrowing the format is a separate issue. +- The SVAR1 `_write_from_svar` branch, which also ignores `max_mem` but uses a + different mechanism (`_find_starts_ends_with_length(..., out=)`) with no + transient amplification. Separate follow-up issue. +- Making the write-time range cache optional or lazily computed at read time. + +## Scope and sequencing + +Two PRs, both targeting `main`. This is the file-backed `gvl.write` path, not +StreamingDataset-board work. + +1. **genoray** (`main`, 3.3.0 -> 3.4.0) — loop inversion, chunked private API, + `max_ends`. Land and release first. +2. **GenVarLoader** (`main`) — plumb `max_mem`, consume the chunked API, + preflight, delete `_svar2_region_max_ends`, bump the pin to + `genoray>=3.4,<4`. + +## genoray: Rust core + +### Hoist the per-column index + +`src/query/reader.rs` gains a struct holding the region-independent state: + +```rust +pub(crate) struct VkColumnIndex { + o0: usize, // absolute base offset of this column + tree: SearchTree, // over positions[o0..o1] + v_ends: Vec, + max_del: u32, // indel channel only; 0 for snp +} + +impl VkColumnIndex { + fn overlap(&self, qs: u32, qe: u32) -> Range; +} +``` + +`vk_snp_overlap` / `vk_indel_overlap` are **replaced** by +`vk_snp_index(col)` / `vk_indel_index(sample, p)` returning a `VkColumnIndex`, +plus the cheap `.overlap()`. The old per-call methods are deleted rather than +kept alongside, so there is one way to do this. + +### Column-outer core + +`src/query/gather.rs` gains: + +```rust +pub fn find_ranges_haps( + reader: &ContigReader, + regions: &[(u32, u32)], + sample_cols: &[usize], + hap_lo: usize, hap_hi: usize, // half-open, into the selected hap axis + out_snp: &mut [i64], // (hap_hi - hap_lo, R, 2), hap-major + out_indel: &mut [i64], + // returns Vec: (R,) partial max of the packed (pos << 21) | ext + // composite key over this hap slice. 0 = no variant. +) +``` + +The loop is column-outer / region-inner, so each column's tree is built exactly +once and the store is swept once. Parallelized with +`out_snp.par_chunks_mut(R * 2)` zipped over columns — disjoint mutable slices, +no `unsafe`. + +Tree builds drop from `R x H x 2` to `H x 2`: 3,964x fewer for the reported +input, with the store read once instead of `R` times. + +### Output order + +`find_ranges_haps` produces **hap-major** `(H, R, 2)` — what the column-outer +loop naturally produces and what `par_chunks_mut` can write safely. + +The existing `find_ranges` binding keeps its region-major `(R*H, 2)` contract by +transposing the hap-major fill back into its `Vec>`. That is one +extra copy of the payload, paid only on the un-chunked path: `find_ranges` is the +small-batch read-path entry point, while the population-scale writer uses the +chunked API and never builds a bundle at all. `_gather_ranges` and the read path +(`_svar2_haps.py`) are untouched. + +`find_ranges` becomes a thin wrapper over `find_ranges_haps` covering all haps. + +## genoray: chunked Python API + +New private method on `_BatchQueryMixin` (`python/genoray/_svar2_batch.py`), +alongside the existing `_find_ranges`: + +```python +@dataclass(frozen=True) +class RangesChunk: + sample_start: int # into the SELECTED sample axis + n_samples: int + vk_snp_range: NDArray[np.int64] # (n_samples, ploidy, R, 2), hap-major + vk_indel_range: NDArray[np.int64] + max_end_keys: NDArray[np.int64] # (R,), packed key; 0 = no variant + + +@dataclass(frozen=True) +class RangesStream: + n_regions: int + n_samples: int # progress denominator + ploidy: int + samples_per_chunk: int # derived; exposed for observability + region_starts: NDArray[np.int32] # eager, R-sized + dense_range: NDArray[np.int32] # (R, 2) + dense_snp_range: NDArray[np.int32] # (R, 2) + dense_indel_range: NDArray[np.int32] # (R, 2) + sample_cols: NDArray[np.int64] # (S,) + dense_max_end_keys: NDArray[np.int64] # (R,), dense-channel contribution + chunks: Iterator[RangesChunk] + + +def _find_ranges_chunked( + self, contig, starts, ends, samples=None, *, max_mem: int | None = None +) -> RangesStream: ... +``` + +The R-sized region-level arrays are cheap, so they are computed eagerly and +returned in the header; the `O(R x S x P)` payload arrives in chunks. This is +the "(progress denominator, generator)" contract, typed. `max_mem=None` yields a +single chunk. + +Each chunk is one Rust call — `find_ranges_chunk(regions, sample_idxs, hap_lo, +hap_hi)` — with the generator driving the loop in Python. No callbacks cross the +FFI boundary. The call releases the GIL (`py.detach`), which the current +`find_ranges` does not; without it the new rayon parallelism would serialize +against the consumer's progress bar. + +### Chunk sizing + +The binding allocates each destination numpy array first and fills it in place +(`PyArray2::zeros` -> `&mut [i64]`), eliminating the +`Vec>` -> `Vec` -> `ToPyArray` triple. Peak per chunk is then +exactly one copy of the payload: + +``` +bytes_per_sample = R * ploidy * 2 endpoints * 8 bytes * 2 channels # R * P * 32 +samples_per_chunk = max(1, max_mem // (2 * bytes_per_sample)) # 2x slop +``` + +At `R=3,964`, `P=2` that is ~248 KiB/sample; a 2 GiB budget gives ~4,200 +samples/chunk, ~99 chunks for 414,830 samples. Chunks are whole samples so both +ploids stay together and the consumer's destination slice is clean. + +If `max_mem` cannot fit a single sample, raise with the required minimum, +mirroring the existing gvl error at `_write.py:300`. + +### Rejected: `find_ranges_into(memmap_slice)` + +Once chunks are bounded, the remaining copy is one chunk-sized memcpy (~0.2 s +per 2 GiB) against seconds of chunk compute, and it would couple genoray to +gvl's `(R, S, P, 2)` destination layout. The in-place numpy fill above already +removes the two copies that mattered. + +## genoray: `max_ends` + +Computed inside the same column sweep. No second pass, no decode. + +**vk channels.** Positions are sorted within a column, so the max-position +variant for a `(region, hap)` pair is the last element of the range +`find_ranges_haps` just computed. Read its `pos`/`key`, apply +`end = pos + 1 - min(ilen, 0)`, reduce per region. Free. + +**dense channel.** `DenseView::carried(hap, col)` indexes a memmapped hap-major +bit matrix at `hap * n_dense_variants + col` (`src/query/sidecar.rs:87`), and +there is no per-variant carrier-count sidecar. So "is dense variant `j` carried +by any selected hap" is a strided probe across haps. The algorithm is a backward +walk from `de - 1` over the region's dense range, stopping at the first variant +carried by any selected hap: + +- Dense variants are common by construction, so the walk almost always + terminates on the first variant after a handful of probes. +- When `samples is None` (gvl's `write` default), every dense variant in the + store has at least one carrier among all samples, so the last dense variant in + the region is exact with **zero** bitmap access. This is the fast path. +- Worst case — a selected subset carrying none of a region's dense variants — + degrades to `dense_in_region x H_selected` bit probes for that region. Bounded + and documented, not guarded against. + +The dense contribution is eager (`RangesStream.dense_max_end_keys`); the +consumer reduces `np.maximum` over it and each chunk's `max_end_keys`, then +unpacks once. + +**The reduction unit is the packed key, not an unpacked end.** The SVAR1 rule +orders by position first and end second, so reducing unpacked ends across hap +chunks would let a lower-position variant with a longer deletion win. Packing +`ext` (bounded) rather than the absolute end is what makes an integer `max` over +the key reproduce that ordering. + +This replaces gvl's `_svar2_region_max_ends`. + +### Parity risk + +Two details must be confirmed against the current gvl implementation before this +is considered equivalent: + +1. Whether `SparseVar2.decode` (and therefore today's `max_ends`) includes + dense-channel variants. +2. Exact reproduction of the `(pos << 21) | ext` composite-key tie-break and the + 0-based-to-1-based `pos` conversion at `_write.py:1100-1120`. + +If genoray's implementation diverges from gvl's, determine which is correct +rather than treating gvl as the oracle. If today's gvl behavior is the buggy +one, that becomes its own issue and PR, and the divergent case is excluded from +the parity test. + +## GenVarLoader: writer + +`_write_from_svar2` gains a `max_mem: int` parameter, plumbed from `write()`'s +`effective_max_mem` — the same value the VCF/PGEN branch already receives. + +### Preflight + +Before creating any memmap: + +```python +cache_bytes = 2 * R * S * P * 2 * 8 # both vk channels +``` + +Log it through the writer's logger and compare against +`shutil.disk_usage(path).free`; warn when it exceeds free space. Warn, do not +hard-error: free-space reporting is unreliable on some filesystems and a false +refusal would block valid large builds. + +### Per-contig loop + +```python +stream = svar2._find_ranges_chunked(c, starts, ends, samples=samples, max_mem=max_mem) +dense_snp[lo:hi] = stream.dense_snp_range +dense_indel[lo:hi] = stream.dense_indel_range +keys = stream.dense_max_end_keys.copy() +for ch in stream.chunks: + s0, s1 = ch.sample_start, ch.sample_start + ch.n_samples + vk_snp[lo:hi, s0:s1] = ch.vk_snp_range.transpose(2, 0, 1, 3) + vk_indel[lo:hi, s0:s1] = ch.vk_indel_range.transpose(2, 0, 1, 3) + np.maximum(keys, ch.max_end_keys, out=keys) + vk_snp.flush() + vk_indel.flush() + pbar.update(rc * ch.n_samples / S) +mask = (1 << 21) - 1 +region_ends = np.asarray(ends, np.int64).copy() +has = keys > 0 # 0 = no variant; keep the original chromEnd +region_ends[has] = (keys[has] >> 21) + (keys[has] & mask) +max_ends[lo:hi] = region_ends.astype(np.int32) +``` + +`transpose(2, 0, 1, 3)` turns `(n_samples, ploidy, R, 2)` into +`(R, n_samples, ploidy, 2)` as a view; numpy performs the strided copy directly +into the memmap slice, with no intermediate array. + +The per-chunk `flush()` matters at this scale. Without it ~98 GiB of dirty pages +accumulate and the kernel reclaims them at unpredictable times. + +### Progress + +The bar stays region-denominated (`total=R, unit=" region"`) but takes +fractional updates, so a single-contig BED advances smoothly instead of sitting +at 0%. This is the issue's second reported symptom. + +### Deletions + +`_svar2_region_max_ends` is removed. + +## Testing + +### genoray + +- **Loop-inversion guard.** `find_ranges` returns a byte-identical bundle before + and after the refactor, on existing fixtures. This covers the whole + `VkColumnIndex` extraction. +- **Complexity regression guard.** A Rust unit test asserting the existing + `TREE_BUILDS` counter (`src/search.rs:48`) scales as `O(H)`, not `O(R x H)`, + across two region counts. A wall-clock test would be too noisy on a shared + node; the counter is deterministic. +- **Chunk equivalence.** Property test over `max_mem` values (including one + sample per chunk) asserting the reassembled chunked result equals the + unchunked bundle, so chunk boundaries land in different places. +- **`max_ends` parity.** Against a Python oracle reproducing the current gvl + semantics, including the tie-break packing. +- **`samples` subset.** `_find_ranges_chunked(samples=subset)` agrees with + `_find_ranges(samples=subset)`, exercising the non-fast-path dense walk. + +### GenVarLoader + +Covering issue #333 section 4: + +- Monkeypatch `_find_ranges_chunked` to record chunk sizes: assert more than one + chunk under a small `max_mem`, and that no chunk exceeds the budget. +- Written cache byte-identical to a dataset produced by the current code path, + on an existing fixture (`phased_svar_gvl` / `build_case` session fixtures). +- Progress advances per chunk, and each chunk is released before the next. +- Preflight logs the expected byte count. + +Run `pixi run -e dev pytest tests -q` (full tree) before pushing: this touches +shared write-path code and renames a private symbol. + +## Documentation + +- `docs/source/format.md` — the `genotypes/svar2_ranges` section calls these + arrays "small". Replace with the `R x S x P` scaling formula and a worked + population-scale example. +- `docs/source/write.md` and the `gvl.write` docstring — `max_mem` now governs + the SVAR2 branch. +- `skills/genvarloader/SKILL.md` — `max_mem` behavior note under `gvl.write`; + add to "Common gotchas" that SVAR2 range caches scale with + `regions x samples x ploidy`. +- genoray `CHANGELOG.md`. The `genoray-api` skill documents public surface only; + `_find_ranges_chunked` is private, so no change expected there. + +## Follow-up issues to file + +1. genoray/gvl: shrink the on-disk SVAR2 range cache below 16 bytes/entry. +2. gvl: `_write_from_svar` (SVAR1) also ignores `max_mem`. diff --git a/pixi.lock b/pixi.lock index 6b2f270a..9787be87 100644 --- a/pixi.lock +++ b/pixi.lock @@ -204,10 +204,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -357,6 +357,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/55/5e/f8f4f7cc9b24b35103eb9e19f0f69935d16b878b4c5e4511ecd2261403fb/vcfixture-0.6.0-py3-none-any.whl @@ -378,7 +379,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl @@ -605,10 +605,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8b/4d/5740c27110b83634d8491c3b5facf0111b3e554c3164f4fb953be9bddaf6/pytorch_lightning-2.6.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl @@ -774,6 +774,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/55/5e/f8f4f7cc9b24b35103eb9e19f0f69935d16b878b4c5e4511ecd2261403fb/vcfixture-0.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/56/c6/65f646c7ff09bd257f660434adb45c4dfcbbcebcc030562fecf6f5bf887d/pydantic_core-2.46.4-cp310-cp310-macosx_11_0_arm64.whl @@ -793,7 +794,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl @@ -1060,13 +1060,13 @@ environments: - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7a/f6/51d8a97116de23c9280c1fa3b813bc088f8571ce5936ba84af1ecf13ed45/pybigwig-0.3.25-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/b3/6b4067be973ae96ba0d615946e314c5ae35f9f993eca561b356540bb0c2b/alabaster-1.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/b9/e73d5d9f405cba7706c539aa8b311b49d4c2f3d698d9c12f815231169c71/ipykernel-7.2.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/11/00d3c3dfc25ad54e731d91449895a79e4bf2384dc3ac01809010ba88f6d5/seaborn-0.13.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/a0/5b0c2f11142ed1dddec842457d3f65eaf71a0080894eb6f018755b319c3a/nbclient-0.10.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8d/3f/95338030883d8c8b91223b4e21744b04d11b161a3ef117295d8241f50ab4/accessible_pygments-0.0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl @@ -1259,6 +1259,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/7f/a946aa4f8752b37102b41e64dca18a1976ac705c3a0d1dfe74d820a02552/mistune-3.2.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2d/0b/ceb7694d864abc0a047649aec263878acb9f792e1fec3e676f22dc9015e3/jupyter_client-8.8.0-py3-none-any.whl @@ -1313,7 +1314,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/92/e7/038aab64a946d535901103da16b953c8c9cc9c961dadcbf3609ed6428d23/pyzmq-27.1.0-cp312-abi3-macosx_10_15_universal2.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/98/c1/37f2fcccce3f1494147e46ccc04996226defe9ccae8251a9ce61296fa599/pysam-0.24.0-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/99/5d/8268b644392ee874ee82a635cd0df1773de230bde356c38de28e298392cc/parso-0.8.7-py2.py3-none-any.whl @@ -1607,13 +1607,13 @@ environments: - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7a/f6/51d8a97116de23c9280c1fa3b813bc088f8571ce5936ba84af1ecf13ed45/pybigwig-0.3.25-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/b3/6b4067be973ae96ba0d615946e314c5ae35f9f993eca561b356540bb0c2b/alabaster-1.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/b9/e73d5d9f405cba7706c539aa8b311b49d4c2f3d698d9c12f815231169c71/ipykernel-7.2.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/11/00d3c3dfc25ad54e731d91449895a79e4bf2384dc3ac01809010ba88f6d5/seaborn-0.13.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/a0/5b0c2f11142ed1dddec842457d3f65eaf71a0080894eb6f018755b319c3a/nbclient-0.10.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8d/3f/95338030883d8c8b91223b4e21744b04d11b161a3ef117295d8241f50ab4/accessible_pygments-0.0.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl @@ -1821,6 +1821,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/7f/a946aa4f8752b37102b41e64dca18a1976ac705c3a0d1dfe74d820a02552/mistune-3.2.1-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2c/58/ca301544e1fa93ed4f80d724bf5b194f6e4b945841c5bfd555878eea9fcb/referencing-0.37.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2d/0b/ceb7694d864abc0a047649aec263878acb9f792e1fec3e676f22dc9015e3/jupyter_client-8.8.0-py3-none-any.whl @@ -1874,7 +1875,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/8e/37/efad0257dc6e593a18957422533ff0f87ede7c9c6ea010a2177d738fb82f/pure_eval-0.2.3-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/92/e7/038aab64a946d535901103da16b953c8c9cc9c961dadcbf3609ed6428d23/pyzmq-27.1.0-cp312-abi3-macosx_10_15_universal2.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/98/c1/37f2fcccce3f1494147e46ccc04996226defe9ccae8251a9ce61296fa599/pysam-0.24.0-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/99/5d/8268b644392ee874ee82a635cd0df1773de230bde356c38de28e298392cc/parso-0.8.7-py2.py3-none-any.whl @@ -2019,12 +2019,12 @@ environments: - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/61/cceae43728b7de99d9b847560c262873a1f6c98202171fd5ed62640b494b/tomli-2.4.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/81/e6/cd9575ac904136b3cbf7aa7ee819ef86eedb7274e46f230e94ea4342e729/platformdirs-4.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/82/3b/64d4899d73f91ba49a8c18a8ff3f0ea8f1c1d75481760df8c68ef5235bf5/rich-15.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -2098,6 +2098,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/32/46/9cb0e58b2deb7f82b84065f37f3bffeb12413f947f9388e4cac22c4621ce/sortedcontainers-2.4.0-py2.py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/38/3d/2d244233ac4f76e38533cfcb2991c9eb4c7bf688ae0a036d30725b8faafe/importlib_metadata-9.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3a/13/547360d81e6d88d58492968ffda9f9542854f11310ee556fef14260cc886/zipp-4.1.0-py3-none-any.whl @@ -2128,7 +2129,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/8a/0e/97c33bf5009bdbac74fd2beace167cab3f978feb69cc36f1ef79360d6c4e/exceptiongroup-1.3.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8f/fe/a4867bc2b8b81d9b1648992fb7e4a732b3db480ff2d02df2c7b59189c812/hypothesis-6.156.6-cp310-cp310-macosx_11_0_arm64.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl @@ -2464,10 +2464,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/9d/66/47cf0a44c768792154c665eedeb6a33201d89d75e5fba62c7b4b585d08c4/awkward_cpp-54-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl @@ -2681,6 +2681,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/55/5e/f8f4f7cc9b24b35103eb9e19f0f69935d16b878b4c5e4511ecd2261403fb/vcfixture-0.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/56/c6/65f646c7ff09bd257f660434adb45c4dfcbbcebcc030562fecf6f5bf887d/pydantic_core-2.46.4-cp310-cp310-macosx_11_0_arm64.whl @@ -2700,7 +2701,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl @@ -2920,10 +2920,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8e/6d/41991e503e51fc1134502694c5fa7a1671501a17ffa12716a4a9151af3df/scipy-1.15.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -3073,6 +3073,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/55/5e/f8f4f7cc9b24b35103eb9e19f0f69935d16b878b4c5e4511ecd2261403fb/vcfixture-0.6.0-py3-none-any.whl @@ -3094,7 +3095,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/6e/0c3bf90fae0e910c274db43304ebe25a6b391327f3f10b5dcc638c090795/scipy-1.15.3-cp310-cp310-macosx_12_0_arm64.whl @@ -3325,10 +3325,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/80/50/540cd3aeefc041beb111125c4bff779831a2111fc6b15a9138cda277d32c/pydantic_core-2.46.4-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -3470,6 +3470,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/28/53/21f7b97e82772caa61541348427f42435120b32961c92d16f9c8ce9757d6/cslug-1.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/34/0b/b9d1911cfefa61399821dfb37f486d83e0f42630a8d12f7194270c417002/llvmlite-0.47.0-cp311-cp311-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl @@ -3493,7 +3494,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/94/dc/80564a3071a57c20b7c32575e4a0120e8a330ef487c319b122942d665960/pyarrow-21.0.0-cp311-cp311-macosx_12_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/b3/650500c2eab4534d98e9166f4298e0f3c69c742afdf24e6eabccd1f16ad8/numba-0.65.1-cp311-cp311-macosx_12_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -3723,9 +3723,9 @@ environments: - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7a/f6/51d8a97116de23c9280c1fa3b813bc088f8571ce5936ba84af1ecf13ed45/pybigwig-0.3.25-cp312-cp312-manylinux_2_17_x86_64.manylinux2014_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/c9/65f89382870c8cf8277b06680f81ea7621324de4c49fbce7276e0fd17ce5/cyvcf2-0.32.1-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.whl @@ -3870,6 +3870,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/28/53/21f7b97e82772caa61541348427f42435120b32961c92d16f9c8ce9757d6/cslug-1.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2f/97/9214bd9b860e680a281232e218d10b718a7280b593f4ab56240a558dc975/pgenlib-0.94.0-cp312-cp312-macosx_10_13_universal2.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl @@ -3892,7 +3893,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/72/1e6442a00cd2924d361aa1b642ab6373ec35c6fabf311a760be9f76e0f13/scipy-1.18.0-cp312-cp312-macosx_12_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/98/c1/37f2fcccce3f1494147e46ccc04996226defe9ccae8251a9ce61296fa599/pysam-0.24.0-cp312-cp312-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -4122,9 +4122,9 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/89/4b/7782438b551dbb0468892a276b8c789b8bbdb25ea5c5eb27faadd753e037/pyarrow-21.0.0-cp313-cp313-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl @@ -4271,6 +4271,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/28/53/21f7b97e82772caa61541348427f42435120b32961c92d16f9c8ce9757d6/cslug-1.0.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3e/fe/1624eb5024e897bf4074bfc31f9e5e823160aed1ac14e7720e849a3d1109/selectolax-0.4.8-cp313-cp313-macosx_11_0_arm64.whl @@ -4298,7 +4299,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/91/88/b55b3117287a8540b76dbdd87733808d4d01c8067a3b339408c250bb3600/typeguard-4.5.1-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/9e/c9/b2622292ea83fbb4ec318f5b9ab867d0a28ab43c5717bb85b0a5f6b3b0a4/networkx-3.6.1-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a2/09/77d55d46fd61b4a135c444fc97158ef34a095e5681d0a6c10b75bf356191/sympy-1.14.0-py3-none-any.whl @@ -4649,10 +4649,10 @@ environments: - pypi: https://files.pythonhosted.org/packages/78/b6/6307fbef88d9b5ee7421e68d78a9f162e0da4900bc5f5793f6d3d0e34fb8/annotated_types-0.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/79/7b/2c79738432f5c924bef5071f933bcc9efd0473bac3b4aa584a6f7c1c8df8/mypy_extensions-1.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7b/91/984aca2ec129e2757d1e4e3c81c3fcda9d0f85b74670a094cc443d9ee949/joblib-1.5.3-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/7e/85/a5bfaebfd305ac18b57b0854d74e37e586809061a91fda62f0bd50c8518e/narwhals-2.24.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/7f/3e/5db95bcf282c52709639744ca2a8b149baccf648e39c8cc87553df9eae0c/urllib3-2.7.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/8b/4d/5740c27110b83634d8491c3b5facf0111b3e554c3164f4fb953be9bddaf6/pytorch_lightning-2.6.5-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl @@ -4881,6 +4881,7 @@ environments: - pypi: https://files.pythonhosted.org/packages/2a/09/f8d8f8f31e4483c10a906437b4ce31bdf3d6d417b73fe33f1a8b59e34228/einops-0.8.2-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/2d/d4bf65e47cea8ff2c794a600c4fd1273a7902f268757c531e0ee9f18aa58/pooch-1.9.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/2a/68/1fc93dd759605b5d00fc98b50200739e41ed32bd22d6ba35ca6c3932371b/rich_rst-2.1.0-py3-none-any.whl + - pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/3b/92/73ab995881de743ddfebe51e6ed35aa6bd709ae7f09eea2cc8e9716bfa36/seqpro-0.22.0-cp39-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/55/5e/f8f4f7cc9b24b35103eb9e19f0f69935d16b878b4c5e4511ecd2261403fb/vcfixture-0.6.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/56/c6/65f646c7ff09bd257f660434adb45c4dfcbbcebcc030562fecf6f5bf887d/pydantic_core-2.46.4-cp310-cp310-macosx_11_0_arm64.whl @@ -4900,7 +4901,6 @@ environments: - pypi: https://files.pythonhosted.org/packages/83/89/35ea267fb12e608529f0df315aff200171e555623cb38b2e4444592ce872/pyranges-0.1.4-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/86/b2/04438111b57e3591c09dfa9f220609ae1afacf436fba124a328dbdb9b7b2/genvarloader_cli-0.1.0-py3-none-any.whl - - pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/96/34/ef34ef77f1ee38fc8e4f9775217a613b452916e633c4f1d98f31db52c4a5/zstandard-0.25.0-cp310-cp310-macosx_11_0_arm64.whl - pypi: https://files.pythonhosted.org/packages/99/55/db07de81b5c630da5cbf5c7df646580ca26dfaefa593667fc6f2fe016d2e/tabulate-0.10.0-py3-none-any.whl - pypi: https://files.pythonhosted.org/packages/a0/f4/c67b0b3f1b9245e8d266f0f112c500d50e5b4e83cb6f3b71b6528104182a/requests-2.34.2-py3-none-any.whl @@ -8891,7 +8891,7 @@ packages: license: Apache-2.0 license_family: APACHE purls: - - pkg:pypi/importlib-metadata?source=compressed-mapping + - pkg:pypi/importlib-metadata?source=hash-mapping size: 34766 timestamp: 1779714582554 - conda: https://conda.anaconda.org/conda-forge/noarch/iniconfig-2.3.0-pyhd8ed1ab_0.conda @@ -11616,7 +11616,7 @@ packages: name: genvarloader requires_dist: - seqpro>=0.22 - - genoray>=3.3.1,<4 + - genoray>=3.4.0,<4 - numpy - loguru - natsort @@ -12647,6 +12647,41 @@ packages: - brotlicffi>=1.2 ; platform_python_implementation != 'CPython' and extra == 'speedups' - backports-zstd ; python_full_version < '3.14' and platform_python_implementation == 'CPython' and sys_platform != 'android' and sys_platform != 'ios' and extra == 'speedups' requires_python: '>=3.10' +- pypi: https://files.pythonhosted.org/packages/2b/db/6e9a4b3792a0ef2d9b8309fe03bc5b4059f690b8bca5467cd37b957e9dad/genoray-3.4.0-cp310-abi3-macosx_11_0_arm64.whl + name: genoray + version: 3.4.0 + sha256: 294d5cfcea5524a714734097733ce8402ffc571cba679557f5b0a3bd43c1c79e + requires_dist: + - seqpro>=0.21.1,<0.23 + - numpy>=1.26 + - pandas>=2.2.3 + - hirola>=0.3.0 + - pgenlib>=0.91.0 + - cyvcf2>=0.31.1 + - pysam>=0.22 + - polars>=1.37.1 + - polars-bio>=0.20.1,<0.34 + - pyranges>=0.1.3 + - rich>=13 + - typing-extensions>=4.14 + - pyarrow>=21 + - tqdm>=4.65 + - phantom-types>=3 + - more-itertools>=10 + - loguru>=0.7.0 + - attrs + - awkward + - numba + - cyclopts + - zstandard + - pydantic + - oxbow>=0.5.1,<0.6 + - joblib>=1.4.2,<2 + - joblib-progress>=1.0.6,<2 + - filelock>3,<4 + - scipy>=1.10 + - pooch>=1.7 + requires_python: '>=3.10,<3.15' - pypi: https://files.pythonhosted.org/packages/2c/2d/6ea7cad2c2f0625c4120bef5353ab7cf749141bf1d070011cebb72f68189/pandera-0.31.1-py3-none-any.whl name: pandera version: 0.31.1 @@ -14367,6 +14402,41 @@ packages: version: 1.5.3 sha256: 5fc3c5039fc5ca8c0276333a188bbd59d6b7ab37fe6632daa76bc7f9ec18e713 requires_python: '>=3.9' +- pypi: https://files.pythonhosted.org/packages/7e/36/5097be182ab84db61dab91f9143c1097f2801e2c6997e52c1f4fe6cfa8b6/genoray-3.4.0-cp310-abi3-manylinux_2_28_x86_64.whl + name: genoray + version: 3.4.0 + sha256: c93c0c8ecd198df87d7d0aa2fd42eb5254927f86376849a8c122b24fa0ce831a + requires_dist: + - seqpro>=0.21.1,<0.23 + - numpy>=1.26 + - pandas>=2.2.3 + - hirola>=0.3.0 + - pgenlib>=0.91.0 + - cyvcf2>=0.31.1 + - pysam>=0.22 + - polars>=1.37.1 + - polars-bio>=0.20.1,<0.34 + - pyranges>=0.1.3 + - rich>=13 + - typing-extensions>=4.14 + - pyarrow>=21 + - tqdm>=4.65 + - phantom-types>=3 + - more-itertools>=10 + - loguru>=0.7.0 + - attrs + - awkward + - numba + - cyclopts + - zstandard + - pydantic + - oxbow>=0.5.1,<0.6 + - joblib>=1.4.2,<2 + - joblib-progress>=1.0.6,<2 + - filelock>3,<4 + - scipy>=1.10 + - pooch>=1.7 + requires_python: '>=3.10,<3.15' - pypi: https://files.pythonhosted.org/packages/7e/46/81b71b7aa9e3703ee6e4ef1f69a87e40f58ea7c99212bf49a95071e99c8c/polars_runtime_32-1.37.1-cp310-abi3-macosx_11_0_arm64.whl name: polars-runtime-32 version: 1.37.1 @@ -14698,41 +14768,6 @@ packages: - testpath ; extra == 'test' - xmltodict ; extra == 'test' requires_python: '>=3.10.0' -- pypi: https://files.pythonhosted.org/packages/84/02/a6253af89cf532e8029c7508423848704e8362d9e3080bdc5e27f34b7b12/genoray-3.3.1-cp310-abi3-manylinux_2_28_x86_64.whl - name: genoray - version: 3.3.1 - sha256: fb7c314e013835db6bef9ab6a940219908c24888a4dd574239f1987381022ed1 - requires_dist: - - seqpro>=0.21.1,<0.23 - - numpy>=1.26 - - pandas>=2.2.3 - - hirola>=0.3.0 - - pgenlib>=0.91.0 - - cyvcf2>=0.31.1 - - pysam>=0.22 - - polars>=1.37.1 - - polars-bio>=0.20.1,<0.34 - - pyranges>=0.1.3 - - rich>=13 - - typing-extensions>=4.14 - - pyarrow>=21 - - tqdm>=4.65 - - phantom-types>=3 - - more-itertools>=10 - - loguru>=0.7.0 - - attrs - - awkward - - numba - - cyclopts - - zstandard - - pydantic - - oxbow>=0.5.1,<0.6 - - joblib>=1.4.2,<2 - - joblib-progress>=1.0.6,<2 - - filelock>3,<4 - - scipy>=1.10 - - pooch>=1.7 - requires_python: '>=3.10,<3.15' - pypi: https://files.pythonhosted.org/packages/85/dc/bf8a9b7e289dd9b0b550b9964786231fe48264583eecd733f7ab77b374b7/ncls-0.0.70-cp310-cp310-macosx_11_0_arm64.whl name: ncls version: 0.0.70 @@ -15006,41 +15041,6 @@ packages: requires_dist: - cffi ; implementation_name == 'pypy' requires_python: '>=3.8' -- pypi: https://files.pythonhosted.org/packages/94/07/0884ea45f06ace0f1ee423d9184f40d598303ba746ff85a6a9a35b56cb94/genoray-3.3.1-cp310-abi3-macosx_11_0_arm64.whl - name: genoray - version: 3.3.1 - sha256: 95c013e328b3deee12a7a5cd2aeb340469ca0f12f668cbcbe1740aca94229473 - requires_dist: - - seqpro>=0.21.1,<0.23 - - numpy>=1.26 - - pandas>=2.2.3 - - hirola>=0.3.0 - - pgenlib>=0.91.0 - - cyvcf2>=0.31.1 - - pysam>=0.22 - - polars>=1.37.1 - - polars-bio>=0.20.1,<0.34 - - pyranges>=0.1.3 - - rich>=13 - - typing-extensions>=4.14 - - pyarrow>=21 - - tqdm>=4.65 - - phantom-types>=3 - - more-itertools>=10 - - loguru>=0.7.0 - - attrs - - awkward - - numba - - cyclopts - - zstandard - - pydantic - - oxbow>=0.5.1,<0.6 - - joblib>=1.4.2,<2 - - joblib-progress>=1.0.6,<2 - - filelock>3,<4 - - scipy>=1.10 - - pooch>=1.7 - requires_python: '>=3.10,<3.15' - pypi: https://files.pythonhosted.org/packages/94/dc/80564a3071a57c20b7c32575e4a0120e8a330ef487c319b122942d665960/pyarrow-21.0.0-cp311-cp311-macosx_12_0_arm64.whl name: pyarrow version: 21.0.0 diff --git a/pixi.toml b/pixi.toml index 872476e8..23676bd2 100644 --- a/pixi.toml +++ b/pixi.toml @@ -102,11 +102,11 @@ numba = "==0.59.1" pyarrow = ">=21" hirola = "==0.3" seqpro = "==0.22.0" -# genoray >=3.3.1 as the prebuilt abi3 wheel from PyPI — one cp310-abi3 wheel covers -# py310-313 on both platforms. 3.0.0 was the first release with the INFO/FORMAT field-read -# API; 3.3.1 is the first whose seqpro cap (<0.23) admits seqpro>=0.22. Mirrors the -# pyproject floor. -genoray = ">=3.3.1,<4" +# genoray >=3.4.0 as the prebuilt abi3 wheel from PyPI — one cp310-abi3 wheel covers +# py310-313 on both platforms. 3.4.0 carries SparseVar2._find_ranges_chunked, the +# memory-bounded chunked range API _write_from_svar2 consumes (gvl#333). Mirrors +# the pyproject floor. +genoray = ">=3.4.0,<4" polars = "==1.37.1" loguru = "*" natsort = "*" diff --git a/pyproject.toml b/pyproject.toml index 507e3ee0..47ab1f73 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -11,11 +11,10 @@ license = { file = "LICENSE.txt" } requires-python = ">=3.10,<3.14" # >= 3.14 blocked by pyarrow/genoray dependencies = [ "seqpro>=0.22", - # >=3.0.0 carried the INFO/FORMAT field-read API gvl relies on; >=3.3.1 is the first - # release whose seqpro cap (<0.23) admits the seqpro>=0.22 above. Earlier 3.x pin - # seqpro<0.22, so they are unsatisfiable here — state that instead of making the - # resolver discover it. - "genoray>=3.3.1,<4", + # >=3.4.0 carries SparseVar2._find_ranges_chunked, the memory-bounded chunked + # range API _write_from_svar2 consumes to avoid materializing a whole + # contig's ranges at once (gvl#333). + "genoray>=3.4.0,<4", "numpy", "loguru", "natsort", diff --git a/python/genvarloader/_dataset/_write.py b/python/genvarloader/_dataset/_write.py index a94eeb2e..9309c112 100644 --- a/python/genvarloader/_dataset/_write.py +++ b/python/genvarloader/_dataset/_write.py @@ -22,6 +22,7 @@ from genoray import exprs as _gexprs from genoray._svar import dense2sparse from genoray._svar._convert import _dense2sparse_with_length +from genoray._svar2_batch import MAX_END_SHIFT from genoray._types import V_IDX_TYPE from genoray._contigs import ContigNormalizer from genoray._utils import format_memory, parse_memory @@ -147,6 +148,9 @@ def write( budget is too small to fit even a single variant chunk. Otherwise ``max_mem`` is a soft limit on overall usage and may be exceeded by a small amount. + For a ``.svar2`` variant source this also bounds the genotype + range-cache write: ranges are produced in per-sample chunks sized to + fit the budget rather than a whole contig at once. extend_to_length: Whether to continue reading/writing variants until all haplotypes have a length at least as long as the intervals in `bed`. Otherwise, deletions can cause the length of haplotypes to be less than the intervals in `bed`. This can be disabled if having haplotypes shorter than the intervals is acceptable, in which case they will be padded with reference bases when appropriate. @@ -323,7 +327,12 @@ def write( metadata["svar_link"] = _svar_link elif isinstance(variants, SparseVar2): gvl_bed, _svar2_link = _write_from_svar2( - path, gvl_bed, variants, samples, extend_to_length + path, + gvl_bed, + variants, + samples, + extend_to_length, + effective_max_mem, ) metadata["svar2_link"] = _svar2_link metadata["ploidy"] = variants.ploidy @@ -1064,61 +1073,55 @@ def _write_from_svar( ), svar_link -def _svar2_region_max_ends( - svar2: SparseVar2, - contig: str, - starts: NDArray[np.integer], - ends: NDArray[np.integer], - samples: list[str], -) -> NDArray[np.int32]: - """SVAR1 parity: per region, the end (``pos - min(ilen, 0)``) of the highest-position variant over the SELECTED samples' haplotypes. Regions with no variants keep their original ``chromEnd``. - - ``SparseVar2.decode`` reports 0-based ``pos`` (unlike ``SparseVar.index``'s - 1-based VCF ``POS``, which SVAR1's ``v_ends`` formula is written against), so - ``pos`` is converted to 1-based here before applying the same formula -- - otherwise every extension would be off by one (masked in most regions - because the un-extended ``chromEnd`` already dominates the max). - - Vectorized as a per-region scatter-max over a ``(pos << 21) | end`` composite - key, which reproduces the pos-then-end tie-break exactly (a single haplotype - never carries two variants at the same position, so a global per-region max - over the selected samples' variants equals the original per-hap-argmax loop). +def _svar2_ranges_cache_bytes(n_regions: int, n_samples: int, ploidy: int) -> int: + """Permanent on-disk size of the two ``svar2_ranges`` var-key caches. + + Each of ``vk_snp_range`` and ``vk_indel_range`` is a + ``(regions, samples, ploidy, 2)`` int64 array. These are NOT small: one + chromosome of a 414k-sample cohort over ~4k regions is ~98 GiB. + + Args: + n_regions: Number of BED rows in the dataset. + n_samples: Number of selected samples. + ploidy: Ploidy of the variant source. + + Returns: + Total bytes both channels will occupy on disk. + """ + return 2 * n_regions * n_samples * ploidy * 2 * 8 + + +def _svar2_preflight(out_dir: Path, n_regions: int, n_samples: int, ploidy: int) -> int: + """Log the projected ``svar2_ranges`` cache size and warn if disk is short. + + Warns rather than raising: free-space reporting is unreliable on some + network filesystems, and a false refusal would block a valid large build. + + Args: + out_dir: Directory the cache will be written to. + n_regions: Number of BED rows in the dataset. + n_samples: Number of selected samples. + ploidy: Ploidy of the variant source. + + Returns: + Projected total bytes of the two var-key caches. """ - R, S_all, P = len(starts), svar2.n_samples, svar2.ploidy - sel = np.asarray([svar2.available_samples.index(s) for s in samples], np.int64) - dec = svar2.decode(contig, list(zip(starts.tolist(), ends.tolist()))) - pos_arr = np.asarray(dec.data["pos"], np.int64) - ilen_arr = np.asarray(dec.data["ilen"], np.int64) - off = np.asarray(dec.offsets, np.int64) # length R*S_all*P + 1 - out = np.asarray(ends, np.int64).copy() # default = chromEnd - if pos_arr.size: - n_hap = R * S_all * P - counts = np.diff(off) # variants per hap - hap_of_var = np.repeat(np.arange(n_hap), counts) # region-major hap per variant - s_of_hap = (np.arange(n_hap) // P) % S_all - keep = np.isin(s_of_hap[hap_of_var], sel) # only selected samples - region_of_var = hap_of_var // (S_all * P) - # end = pos + ext, where ext = 1 - min(ilen, 0) (1-based bump plus the - # deletion extension: SNP/INS -> 1, DEL -> 1 + |ilen|). Pack the BOUNDED - # `ext` into the low bits, NOT the absolute `end` (which is ~pos-sized and - # would overflow past ~2 Mb into any contig), so the composite key orders - # by pos then by end; recover end = pos + ext on unpack. - ext_var = 1 - np.minimum(ilen_arr, 0) # small: 1 + deletion length - SHIFT = 21 - # raise (not assert) so it still fails fast under `python -O`: a pathological - # >~2 Mb deletion footprint would otherwise silently corrupt the packed key. - if int(ext_var.max(initial=0)) >= (1 << SHIFT): - raise ValueError("variant footprint exceeds tie-break packing width") - key = (pos_arr << SHIFT) | ext_var - key_k = key[keep] - region_k = region_of_var[keep] - if key_k.size: - best = np.full(R, -1, np.int64) - np.maximum.at(best, region_k, key_k) # per-region max composite key - has = best >= 0 - # end = pos + ext = (key >> SHIFT) + (key & mask) - out[has] = (best[has] >> SHIFT) + (best[has] & ((1 << SHIFT) - 1)) - return out.astype(np.int32) + n_bytes = _svar2_ranges_cache_bytes(n_regions, n_samples, ploidy) + logger.info( + f"svar2 range cache: {format_memory(n_bytes)} for {n_regions} regions " + f"x {n_samples} samples x ploidy {ploidy}." + ) + try: + free = shutil.disk_usage(out_dir).free + except OSError: + return n_bytes + if n_bytes > free: + logger.warning( + f"svar2 range cache needs {format_memory(n_bytes)} but only " + f"{format_memory(free)} is free at {out_dir}. The write will likely " + f"fail with ENOSPC." + ) + return n_bytes def _write_from_svar2( @@ -1127,6 +1130,7 @@ def _write_from_svar2( svar2: SparseVar2, samples: list[str], extend_to_length: bool, + max_mem: int, ) -> tuple[pl.DataFrame, Svar2Link]: # symbolic/breakend variants are rejected upstream at .svar2 conversion; the # store cannot represent them, and SparseVar2 exposes no index to re-check. @@ -1143,6 +1147,7 @@ def _write_from_svar2( out_dir.mkdir(parents=True, exist_ok=True) R, S, P = bed.height, len(samples), svar2.ploidy + _svar2_preflight(out_dir, R, S, P) vk_snp = np.memmap(out_dir / "vk_snp_range.npy", np.int64, "w+", shape=(R, S, P, 2)) vk_indel = np.memmap( out_dir / "vk_indel_range.npy", np.int64, "w+", shape=(R, S, P, 2) @@ -1184,20 +1189,39 @@ def _write_from_svar2( ends = df["chromEnd"].to_numpy() # extend_to_length is validated at function entry (False raises); the # read-bound kernel sizes haplotype output at read time. - d = svar2._find_ranges(c, starts, ends, samples=samples) - - # _find_ranges returns row-major (R*S*P, 2) for vk ranges; reshape into (R,S,P,2). - vk_snp[lo:hi] = np.asarray(d["vk_snp_range"], np.int64).reshape(rc, S, P, 2) - vk_indel[lo:hi] = np.asarray(d["vk_indel_range"], np.int64).reshape(rc, S, P, 2) - dense_snp[lo:hi] = np.asarray(d["dense_snp_range"], np.int64).reshape(rc, 2) - dense_indel[lo:hi] = np.asarray(d["dense_indel_range"], np.int64).reshape(rc, 2) + stream = svar2._find_ranges_chunked( + c, starts, ends, samples=samples, max_mem=max_mem + ) + dense_snp[lo:hi] = np.asarray(stream.dense_snp_range, np.int64).reshape(rc, 2) + dense_indel[lo:hi] = np.asarray(stream.dense_indel_range, np.int64).reshape( + rc, 2 + ) - # max_ends: SVAR1 parity, per region end of the max-position variant - # over the selected samples' haplotypes (see _svar2_region_max_ends). - max_ends[lo:hi] = _svar2_region_max_ends(svar2, c, starts, ends, samples) + # Packed (pos << SHIFT) | ext keys, NOT unpacked ends: SVAR1 parity picks + # the highest-POSITION variant (ties by end), so a lower-position variant + # with a longer deletion must not win the cross-chunk reduction. + keys = stream.dense_max_end_keys.copy() + for ch in stream.chunks: + s0, s1 = ch.sample_start, ch.sample_start + ch.n_samples + # Chunks are hap-major (samples, ploidy, regions, 2); the cache is + # region-major. transpose() is a view -- numpy copies straight into + # the memmap with no intermediate array. + vk_snp[lo:hi, s0:s1] = ch.vk_snp_range.transpose(2, 0, 1, 3) + vk_indel[lo:hi, s0:s1] = ch.vk_indel_range.transpose(2, 0, 1, 3) + np.maximum(keys, ch.max_end_keys, out=keys) + # Bound the dirty page cache: at cohort scale these memmaps are tens + # of GiB and the kernel would otherwise reclaim at unpredictable times. + vk_snp.flush() + vk_indel.flush() + pbar.update(rc * ch.n_samples / S) + + mask = (1 << MAX_END_SHIFT) - 1 + region_ends = np.asarray(ends, np.int64).copy() + has = keys > 0 # 0 is the "no variant in this region" sentinel + region_ends[has] = (keys[has] >> MAX_END_SHIFT) + (keys[has] & mask) + max_ends[lo:hi] = region_ends.astype(np.int32) contig_offset += df.height - pbar.update(df.height) pbar.close() for mm in (vk_snp, vk_indel, dense_snp, dense_indel): mm.flush() diff --git a/skills/genvarloader/SKILL.md b/skills/genvarloader/SKILL.md index 0b6e58e5..4dc62020 100644 --- a/skills/genvarloader/SKILL.md +++ b/skills/genvarloader/SKILL.md @@ -83,7 +83,7 @@ SVARs are resolved at `Dataset.open` time via `metadata.json` → caller `svar=` `.svar2` is genoray's newer sparse columnar variant store. Pass it to `gvl.write` exactly like a `.svar`, BCF, or PGEN — `gvl.write(path, bed, variants="cohort.svar2")` or `variants=SparseVar2("cohort.svar2")`. Like `.svar`, the dataset stores a back-reference (`metadata.json` → `svar2_link`) instead of duplicating per-variant arrays, so the `.svar2` store must remain accessible at read time. -Unlike `.svar` (whose read path builds an interval search tree + a per-read dense-union over the queried window), a `.svar2`-backed dataset reconstructs via a **read-bound** path: `gvl.write` caches small per-`(region, sample, ploid)` variant-key ranges under `/genotypes/svar2_ranges/` (sized to the dataset's *selected* samples, not the full `.svar2` cohort), and at read time gvl gathers directly off that cache and calls all-Rust kernels — **no interval-search-tree build and no dense-union rebuild per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts. See `docs/source/faq.md`. +Unlike `.svar` (whose read path builds an interval search tree + a per-read dense-union over the queried window), a `.svar2`-backed dataset reconstructs via a **read-bound** path: `gvl.write` caches per-`(region, sample, ploid)` variant-key ranges under `/genotypes/svar2_ranges/` (sized to the dataset's *selected* samples, not the full `.svar2` cohort) — **not small at cohort scale**, see the "Common gotchas" bullet below — and at read time gvl gathers directly off that cache and calls all-Rust kernels — **no interval-search-tree build and no dense-union rebuild per read**. `.svar2` stores are also typically smaller on disk than `.svar`, especially for large cohorts. See `docs/source/faq.md`. `.svar2` is resolved at `Dataset.open` time in the same order as `.svar`: caller `svar2=` arg → recorded relative path → recorded absolute path → sibling `*.svar2`. `Dataset.open(path, svar2=)` mirrors `svar=`. See `docs/source/format.md` ("`.svar2` resolution at open time"). @@ -131,6 +131,8 @@ Notable: **Parallelism:** `gvl.write` now parallelizes over write categories. Variants are processed first (serially). Then per-sample `tracks` and `annot_tracks` run concurrently (joblib loky backend). The `max_mem` budget is divided across the concurrently-running categories. +**`max_mem` and `.svar2`:** for a `.svar2` variant source, `max_mem` also bounds the genotype range-cache write — ranges are produced in per-sample chunks sized to fit the budget rather than a whole contig at once. It does not bound the permanent `genotypes/svar2_ranges/` cache's on-disk size; that scales with `regions x samples x ploidy` and is governed by disk space (see "Common gotchas" below and `format.md`). + Source: `python/genvarloader/_dataset/_write.py`. **Atomic creation:** `gvl.write` builds into a private sibling temp directory and publishes via an atomic `os.replace`. A best-effort `filelock` avoids redundant rebuilds when parallel jobs share the same destination, but correctness relies on the rename — the lock is advisory only. **Datasets do not auto-rebuild**; if the on-disk artifact is missing or corrupt, re-run `gvl.write`. @@ -457,6 +459,12 @@ See `docs/source/format.md` for the full schema, versioning, and SVAR-link detai - `dummy_variant` padding applies to **both `"variants"` and `"variant-windows"`** outputs. Setting `dummy_variant=` and then indexing with any other kind (`"haplotypes"`, `"annotated"`, `"reference"`, or no seqs) raises `ValueError`. For token fields (`flank_tokens`, `ref_window`/`alt_window`, bare `ref`/`alt`), the dummy fill is all-`unknown_token` — the `DummyVariant.ref`/`.alt` bytes only set the dummy allele's byte-length, not the token value. `dummy_variant=False` with an unsupported output kind is silently ignored. - A non-`b"N"` `DummyVariant.alt` (or `.ref`) **is reverse-complemented** on negative-strand regions, exactly like a real variant allele. The default `b"N"` is rc-invariant; use it if you want a strand-neutral sentinel. - `unphased_union=True` + `with_seqs("haplotypes")` / `with_seqs("annotated")` raises — `unphased_union` only applies to `"variants"` / `"variant-windows"` output. +- **SVAR2 range caches scale with `regions x samples x ploidy`.** `gvl.write` + with a `.svar2` source writes a permanent + `2 x regions x samples x ploidy x 2 x 8` byte cache under + `genotypes/svar2_ranges/`. That is ~98 GiB for ~4,000 regions over 414,830 + diploid samples. `max_mem` bounds RAM during the write; it does not bound this + on-disk cache. ## Maintaining this skill diff --git a/tests/dataset/test_write_svar2.py b/tests/dataset/test_write_svar2.py index bb759195..c9f94849 100644 --- a/tests/dataset/test_write_svar2.py +++ b/tests/dataset/test_write_svar2.py @@ -153,7 +153,10 @@ def mm(name: str) -> np.ndarray: samples=sorted_samples, ) # vk ranges: reshape (rc, S, P, 2) -> (rc*S*P, 2) must equal _find_ranges' - # row-major (R*S*P, 2). This pins the reshape done in _write_from_svar2. + # row-major (R*S*P, 2). This is the layout oracle: it pins the transposed, + # chunked write in _write_from_svar2 (hap-major chunks reordered via + # `.transpose(2, 0, 1, 3)` into the region-major memmap) against genoray's + # unchunked, row-major `_find_ranges` bundle. np.testing.assert_array_equal( vk_snp[lo:hi].reshape(rc * S * P, 2), np.asarray(d["vk_snp_range"], np.int64), @@ -328,97 +331,127 @@ def test_svar2_extend_to_length_false_raises(svar2_store: Path, tmp_path: Path): ) -def _reference_region_max_ends(svar2, contig, starts, ends, samples): - """Byte-for-byte copy of the ORIGINAL _svar2_region_max_ends triple-loop, - kept here as the oracle that pins the vectorized rewrite byte-identical.""" - import numpy as np - - R, S_all, P = len(starts), svar2.n_samples, svar2.ploidy - sel = [svar2.available_samples.index(s) for s in samples] - dec = svar2.decode(contig, list(zip(starts.tolist(), ends.tolist()))) - pos_arr = dec.data["pos"] - ilen_arr = dec.data["ilen"] - off = np.asarray(dec.offsets) - out = np.asarray(ends, np.int64).copy() - for r in range(R): - best_pos, best_end = -1, -1 - for s in sel: - for p in range(P): - h = (r * S_all + s) * P + p - a, b = int(off[h]), int(off[h + 1]) - if a == b: - continue - seg_pos = pos_arr[a:b] - seg_ilen = ilen_arr[a:b] - j = int(np.argmax(seg_pos)) - p_pos = int(seg_pos[j]) - p_end = (p_pos + 1) - min(int(seg_ilen[j]), 0) - if p_pos > best_pos or (p_pos == best_pos and p_end > best_end): - best_pos, best_end = p_pos, p_end - if best_pos >= 0: - out[r] = best_end - return out.astype(np.int32) - - -def test_svar2_region_max_ends_matches_reference(svar2_store: Path): - """Vectorized _svar2_region_max_ends must equal the original per-hap loop, - including the pos-then-end tie-break and the empty-region default = chromEnd.""" +def test_write_svar2_chunked_matches_unchunked(svar2_store: Path, tmp_path): + """A tiny max_mem must force multiple chunks and produce identical output.""" from genoray import SparseVar2 - from genvarloader._dataset._write import _svar2_region_max_ends + bed = pl.DataFrame( + {"chrom": ["chr1", "chr1"], "chromStart": [0, 5], "chromEnd": [20, 30]} + ) - svar2 = SparseVar2(svar2_store) - # Overlaps the DEL at 0-based POS 11 with varying windows + a no-variant - # region ([20,30]) so both the extension and keep-chromEnd branches run. - starts = np.array([0, 0, 5, 12, 20], dtype=np.int64) - ends = np.array([15, 20, 10, 13, 30], dtype=np.int64) - samples = list(svar2.available_samples) - - got = _svar2_region_max_ends(svar2, "chr1", starts, ends, samples) - ref = _reference_region_max_ends(svar2, "chr1", starts, ends, samples) - np.testing.assert_array_equal(got, ref) - - # Anti-vacuity: at least one region must be EXTENDED past its chromEnd (the - # DEL at POS 11 extends windows that overlap it), else the test only checks - # the trivial default path. - assert (got != ends.astype(np.int32)).any(), ( - f"test is vacuous: no region extended (got={got.tolist()}, ends={ends.tolist()})" + calls: list[int] = [] + real = SparseVar2._find_ranges_chunked + + def spy(self, *args, **kwargs): + stream = real(self, *args, **kwargs) + calls.append(stream.samples_per_chunk) + return stream + + big = tmp_path / "big.gvl" + gvl.write( + big, + bed, + variants=SparseVar2(svar2_store), + samples=None, + max_mem="4g", + overwrite=True, ) + SparseVar2._find_ranges_chunked = spy + try: + small = tmp_path / "small.gvl" + # 2 regions x ploidy 2 x 2 channels x 2 endpoints x 8 bytes = 128 bytes + # per sample; the chunker's own 2x safety margin needs 256 bytes for + # even one sample, so 256 is the smallest budget that both succeeds + # and forces one-sample-per-chunk (this store has S=2, so that's 2 + # chunks). + gvl.write( + small, + bed, + variants=SparseVar2(svar2_store), + samples=None, + max_mem=256, + overwrite=True, + ) + finally: + SparseVar2._find_ranges_chunked = real + + assert calls and all(c == 1 for c in calls), ( + f"expected one sample per chunk under a 256-byte budget, got {calls}" + ) -def test_svar2_region_max_ends_large_positions(): - """Regression: the composite key must pack a BOUNDED tie-break, not the - absolute end. A variant past ~2 Mb (real chromosomes are hundreds of Mb) - must not overflow the packing / assert-fail. Uses a stub whose decode returns - large positions so we can exercise realistic coordinates without a huge store. + for name in ( + "vk_snp_range.npy", + "vk_indel_range.npy", + "dense_snp_range.npy", + "dense_indel_range.npy", + "sample_cols.npy", + ): + a = (big / "genotypes" / "svar2_ranges" / name).read_bytes() + b = (small / "genotypes" / "svar2_ranges" / name).read_bytes() + assert a == b, name + + # regions.npy (not input_regions.arrow, which holds the pre-extension bed + # verbatim) carries the write-time-extended chromEnd; columns are + # chrom_idx, chromStart, chromEnd, strand. + ra = np.load(big / "regions.npy") + rb = np.load(small / "regions.npy") + assert ra[:, 2].tolist() == rb[:, 2].tolist() + + +def test_write_svar2_max_ends_extend_chromend(svar2_store: Path, tmp_path): + """chromEnd must extend past a deletion that starts inside the region. + + The fixture's DEL is at 0-based POS 11 with ilen -2, so it ends at 14. A + region of [0, 12) must be extended to 14. """ - from types import SimpleNamespace - - import numpy as np - - from genvarloader._dataset._write import _svar2_region_max_ends - - # 2 regions x 1 sample x ploidy 1 = 2 haps, 1 variant each: - # region 0: SNP at pos 3_000_000 (ilen 0) -> end 3_000_001 - # region 1: DEL at pos 5_000_000 (ilen -2) -> end 5_000_003 - class _StubSvar2: - n_samples = 1 - ploidy = 1 - available_samples = ["S0"] - - def decode(self, contig, regions): - return SimpleNamespace( - data={ - "pos": np.array([3_000_000, 5_000_000], np.int64), - "ilen": np.array([0, -2], np.int64), - }, - offsets=np.array([0, 1, 2], np.int64), - ) - - svar2 = _StubSvar2() - starts = np.array([0, 0], np.int64) - ends = np.array([10, 10], np.int64) # small chromEnd so both variants extend - got = _svar2_region_max_ends(svar2, "chrBig", starts, ends, ["S0"]) - ref = _reference_region_max_ends(svar2, "chrBig", starts, ends, ["S0"]) - np.testing.assert_array_equal(got, ref) - np.testing.assert_array_equal(got, np.array([3_000_001, 5_000_003], np.int32)) + from genoray import SparseVar2 + + bed = pl.DataFrame({"chrom": ["chr1"], "chromStart": [0], "chromEnd": [12]}) + out = tmp_path / "ext.gvl" + gvl.write( + out, + bed, + variants=SparseVar2(svar2_store), + samples=None, + max_mem="1g", + overwrite=True, + ) + # regions.npy carries the write-time-extended chromEnd (input_regions.arrow + # holds the pre-extension bed verbatim); columns are chrom_idx, chromStart, + # chromEnd, strand. + regions = np.load(out / "regions.npy") + assert regions[:, 2].tolist() == [14] + + +def test_svar2_ranges_cache_bytes(): + """Both var-key channels: 2 * R * S * P * 2 endpoints * 8 bytes.""" + from genvarloader._dataset._write import _svar2_ranges_cache_bytes + + assert _svar2_ranges_cache_bytes(1, 1, 2) == 2 * 1 * 1 * 2 * 2 * 8 + # The scale from gvl#333: ~98 GiB for one chromosome/panel. + big = _svar2_ranges_cache_bytes(3964, 414830, 2) + assert 90 * 1024**3 < big < 110 * 1024**3 + + +def test_svar2_preflight_warns_when_disk_is_short(tmp_path, monkeypatch): + """A projected cache larger than free space must warn, not silently proceed.""" + from collections import namedtuple + + from loguru import logger + + from genvarloader._dataset import _write + + Usage = namedtuple("Usage", "total used free") + msgs: list[str] = [] + sink = logger.add(lambda m: msgs.append(str(m)), level="WARNING") + try: + monkeypatch.setattr( + _write.shutil, "disk_usage", lambda p: Usage(total=1000, used=999, free=1) + ) + n = _write._svar2_preflight(tmp_path, 3964, 414830, 2) + finally: + logger.remove(sink) + + assert n == _write._svar2_ranges_cache_bytes(3964, 414830, 2) + assert any("free" in m for m in msgs), msgs