Skip to content

fix(redundancy): copy buffered shards to prevent parity corruption - #5543

Closed
gacevicljubisa wants to merge 2 commits into
masterfrom
fix/feeder-buffer-aliasing
Closed

fix(redundancy): copy buffered shards to prevent parity corruption#5543
gacevicljubisa wants to merge 2 commits into
masterfrom
fix/feeder-buffer-aliasing

Conversation

@gacevicljubisa

@gacevicljubisa gacevicljubisa commented Jul 23, 2026

Copy link
Copy Markdown
Member

Checklist

  • I have read the coding guide.
  • My change requires a documentation update, and I have done it.
  • I have added tests to cover my changes.
  • I have filled out the description and linked the related issues.

Description

redundancy.chunkWrite stored the caller's slice directly in the erasure coding shard buffer, but PipeWriteArgs slices are only valid for the duration of the ChainWrite call, the chunk feeder reuses one scratch buffer for every chunk a single Write produces. A multi-chunk Write therefore left every shard aliasing the same memory, so the parities encoded duplicated data.

Reproduced at MEDIUM: the root chunk held 119 distinct references out of 128.

Invisible until it matters. The file reads back fine while every chunk survives; the damage only surfaces when a chunk is lost and recovery returns garbage. Not reachable from FeedPipeline (one chunk per call), but pipeline.Interface is exported and embeds io.Writer with no documented size constraint.

Fix

Copy at the retention site, reusing the slot's backing array, slots refill from index 0 after each encode, so this allocates once per slot rather than once per chunk. Parity slots are excluded: encode hands those downstream, so they must stay owned by whoever stores them.

Also documents the ownership contract on PipeWriteArgs and the reuse at the feeder, so the next retainer doesn't reintroduce it.

Cost

16 MB upload at MEDIUM, versus the buggy baseline: +0.8 MB, +202 allocs, −2.9 ms. A naive copy-per-chunk would have cost +20.1 MB and +4,148 allocs, slot reuse makes the fix effectively free.

Open API Spec Version Changes (if applicable)

Motivation and Context (Optional)

Related Issue (Optional)

Screenshots (if appropriate):

AI Disclosure

  • This PR contains code that has been generated by an LLM.
  • I have reviewed the AI generated code thoroughly.
  • I possess the technical expertise to responsibly review the code generated in this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant