fix(redundancy): copy buffered shards to prevent parity corruption - #5543
Closed
gacevicljubisa wants to merge 2 commits into
Closed
fix(redundancy): copy buffered shards to prevent parity corruption#5543gacevicljubisa wants to merge 2 commits into
gacevicljubisa wants to merge 2 commits into
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Checklist
Description
redundancy.chunkWritestored the caller's slice directly in the erasure coding shard buffer, butPipeWriteArgsslices are only valid for the duration of theChainWritecall, the chunk feeder reuses one scratch buffer for every chunk a singleWriteproduces. A multi-chunkWritetherefore left every shard aliasing the same memory, so the parities encoded duplicated data.Reproduced at MEDIUM: the root chunk held 119 distinct references out of 128.
Invisible until it matters. The file reads back fine while every chunk survives; the damage only surfaces when a chunk is lost and recovery returns garbage. Not reachable from
FeedPipeline(one chunk per call), butpipeline.Interfaceis exported and embedsio.Writerwith no documented size constraint.Fix
Copy at the retention site, reusing the slot's backing array, slots refill from index 0 after each encode, so this allocates once per slot rather than once per chunk. Parity slots are excluded:
encodehands those downstream, so they must stay owned by whoever stores them.Also documents the ownership contract on
PipeWriteArgsand the reuse at the feeder, so the next retainer doesn't reintroduce it.Cost
16 MB upload at MEDIUM, versus the buggy baseline: +0.8 MB, +202 allocs, −2.9 ms. A naive copy-per-chunk would have cost +20.1 MB and +4,148 allocs, slot reuse makes the fix effectively free.
Open API Spec Version Changes (if applicable)
Motivation and Context (Optional)
Related Issue (Optional)
Screenshots (if appropriate):
AI Disclosure