SDSTOR-23030 fix backward compatibility#450
Merged
Hooper9973 merged 1 commit intoJul 9, 2026
Merged
Conversation
|
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## stable/v4.x #450 +/- ##
==============================================
Coverage ? 55.08%
==============================================
Files ? 36
Lines ? 5310
Branches ? 671
==============================================
Hits ? 2925
Misses ? 2087
Partials ? 298 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
xiaoxichen
approved these changes
Jul 9, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Background
Commit
4b25833madeCREATE_SHARD_MSGandSEAL_SHARD_MSGlog-only: the leader now callsasync_alloc_writewith empty data (HS_DATA_INLINED), so no data blocks are allocated forshard creation. Correspondingly,
HeapChunkSelector::select_chunkwas changed toRELEASE_ASSERT(false, ...)since it should never be called for shard messages in the newdesign.
During a rolling upgrade, a pre-upgrade SM (old code) still creates shards in the old
format:
it allocates one block, serializes the
BlkIdinto the Raft journal value, and stores theentry
as
HS_DATA_LINKED(value_size = sizeof(MultiBlkId) > 0,blk_count = 1). Apost-upgrade SM (new code) must handle these old-format entries when acting as a
follower.
Root Cause
The crash is a two-phase cascade caused by
localize_journal_entry_prepareoverwritingthe
Raft journal value with the follower's localized blkid.
Phase 1 — Post-upgrade follower processes old-format entry from pre-upgrade leader
Pre-upgrade leader log:
HS_DATA_LINKED | value_size > 0 | blkid = {X, nblks=1, Y}
Post-upgrade follower, localize_journal_entry_prepare:
entry_blkid.blk_count() = 1 → data_size = 1 × blk_size = 4096
→ get_blk_alloc_hints(data_size=4096)
data_size > 0 branch → returns tombstone_pbas ✓
→ alloc_local_blks: m_local_blkids = [tombstone_pbas{0,0,0}] (blk_count = 0)
→ memcpy(journal_value, local_blkid.serialize(), sizeof(MultiBlkId))
overwrites journal value with tombstone bytes (all zeros)
The post-upgrade follower's local logstore now holds:
HS_DATA_LINKED | value_size = sizeof(MultiBlkId) > 0 | value = tombstone bytes (blk_count =
0)
codeandvalue_sizeare unchanged, but the value bytes now encode a zero blkid.Phase 2 — Any subsequent post-upgrade follower receives the already-corrupted entry
Once the corrupted entry exists in any SM's local log, any new post-upgrade follower that
catches
up from that SM receives:
HS_DATA_LINKED | value_size > 0 | value = tombstone bytes
localize_journal_entry_prepare:
entry_blkid.deserialize(...) → tombstone_pbas, blk_count() = 0
data_size = 0 × blk_size = 0
Condition: (code == HS_DATA_LINKED) && (value_size > 0) → TRUE (both hold)
→ takes HS_DATA_LINKED branch
→ applier_create_req(op_code = HS_DATA_LINKED, data_size = 0)
→ init(): has_linked_data() = true, data_size = 0
→ alloc_local_blks(listener, data_size = 0)
→ get_blk_alloc_hints(data_size = 0)
data_size == 0 branch → returns empty blk_alloc_hints{} ← BUG
→ committed_blk_id.has_value() = false
→ falls through to data_service().alloc_blks()
→ VirtualDev::alloc_blks() → HeapChunkSelector::select_chunk()
→ RELEASE_ASSERT(false, "create shard is log only...") → SIGABRT