perf: bulk dataset IO should use buffered read/write, not np.memmap — ~5-6x slower on NFSv3

[d-laub]

Found while benchmarking copy strategies for dataset concatenation (#334).

gvl.write writes all bulk output through np.memmap(..., mode="w+"):

• _write_phased_variants_chunk — variant_idxs.npy, offsets.npy, dosages.npy
• _write_ragged_intervals — starts.npy, ends.npy, values.npy, offsets.npy
• _write_from_svar — offsets.npy (2,R,S,P)
• _write_from_svar2 — svar2_ranges/*.npy

On NFSv3 this is roughly 5-6x slower than buffered file.write(). Page faults go out as 4 KiB RPCs instead of using the mount's 1 MiB rsize/wsize.

Measurement

512 MiB, 16 MiB chunks, 3 reps. Sources evicted with posix_fadvise(POSIX_FADV_DONTNEED) before every read, so reads are genuinely cold on both filesystems. carter-cn-03, Slurm job 13336789.

NFSv3 (carter-storage:/carter, vers=3,rsize=1048576,wsize=1048576,hard,proto=tcp):

pattern	rep1	rep2	rep3
buffered read → buffered write	85.6	102.0	93.0 MB/s
memmap read → buffered write	16.9	19.7	14.4 MB/s
buffered read → memmap write	13.8	15.7	12.6 MB/s
memmap read → memmap write	13.6	14.1	14.4 MB/s

Local XFS (/dev/md0 on /tmp, node-local):

pattern	rep1	rep2	rep3
buffered read → buffered write	101.2	106.7	98.8 MB/s
memmap read → buffered write	103.1	103.1	100.1 MB/s
buffered read → memmap write	101.2	98.7	98.8 MB/s
memmap read → memmap write	107.9	105.1	108.2 MB/s

Conclusions

1. On local XFS the pattern does not matter — all four are within noise of ~100 MB/s, the device limit. memmap is not harmful there.
2. On NFSv3 any memmap in the path costs ~5-6x, read side or write side or both. Buffered IO on NFS matches local-disk throughput.

Correction to the first version of this issue

The original text reported memmap reads at 316 MB/s and concluded "memmap reads are in fact the fastest option and should be kept." That was wrong. That run did not evict the page cache and the sources had just been written, so it measured RAM, not NFS. Re-measured with fadvise(DONTNEED), memmap reads are slow on NFS too. The recommendation below is correspondingly broader than it first was.

Proposal

Use buffered read()/write() in ~16 MiB chunks for bulk sequential array IO instead of assigning into / streaming out of a memmap. Buffered IO is not slower on local disk, so this can be unconditional rather than an NFS special case.

Important scope limit

These are sequential-streaming numbers. gvl's read path (_haps.py, _tracks.py) does random, fancy-indexed access into memmapped arrays — a different workload, and one where memmap's page cache is a genuine asset across epochs. Nothing here says the read path is slow, and it should not be changed on the basis of this issue. Someone should measure the random-access case separately before touching it.

Out of scope for the #334 concat PR, which uses buffered IO on both sides from the start.

[d-laub]

Re-measured with posix_fadvise(POSIX_FADV_DONTNEED) on both NFSv3 and node-local XFS. Two changes to the original finding, so I have rewritten the issue body:

1. The original memmap-read number (316 MB/s) was a page-cache artifact — sources had just been written and were never evicted. The claim that memmap reads are fastest and should be kept was wrong. Cold, they run at ~17 MB/s on NFS, i.e. as slow as memmap writes. The fix is therefore buffered IO on both sides, not just the write side.

2. Local XFS now measured (the original open question): no penalty for any pattern, all four within noise of ~100 MB/s. So this is an NFS-specific effect, and buffered IO is a safe unconditional choice rather than a network-storage special case.

Also added an explicit scope limit: these are sequential-streaming numbers and say nothing about the random fancy-indexed access in the read path, which should not be changed on this basis.