Skip to content

SDSTOR-23030 fix backward compatibility#450

Merged
Hooper9973 merged 1 commit into
eBay:stable/v4.xfrom
Hooper9973:SDSTOR-23030-fix-compatibility
Jul 9, 2026
Merged

SDSTOR-23030 fix backward compatibility#450
Hooper9973 merged 1 commit into
eBay:stable/v4.xfrom
Hooper9973:SDSTOR-23030-fix-compatibility

Conversation

@Hooper9973

Copy link
Copy Markdown
Contributor

Background

Commit 4b25833 made CREATE_SHARD_MSG and SEAL_SHARD_MSG log-only: the leader now calls
async_alloc_write with empty data (HS_DATA_INLINED), so no data blocks are allocated for
shard creation. Correspondingly, HeapChunkSelector::select_chunk was changed to
RELEASE_ASSERT(false, ...) since it should never be called for shard messages in the new
design.

During a rolling upgrade, a pre-upgrade SM (old code) still creates shards in the old
format:
it allocates one block, serializes the BlkId into the Raft journal value, and stores the
entry
as HS_DATA_LINKED (value_size = sizeof(MultiBlkId) > 0, blk_count = 1). A
post-upgrade SM (new code) must handle these old-format entries when acting as a
follower.

Root Cause

The crash is a two-phase cascade caused by localize_journal_entry_prepare overwriting
the
Raft journal value with the follower's localized blkid.

Phase 1 — Post-upgrade follower processes old-format entry from pre-upgrade leader

Pre-upgrade leader log:
HS_DATA_LINKED | value_size > 0 | blkid = {X, nblks=1, Y}

Post-upgrade follower, localize_journal_entry_prepare:
entry_blkid.blk_count() = 1 → data_size = 1 × blk_size = 4096
→ get_blk_alloc_hints(data_size=4096)
data_size > 0 branch → returns tombstone_pbas ✓
→ alloc_local_blks: m_local_blkids = [tombstone_pbas{0,0,0}] (blk_count = 0)
→ memcpy(journal_value, local_blkid.serialize(), sizeof(MultiBlkId))
overwrites journal value with tombstone bytes (all zeros)

The post-upgrade follower's local logstore now holds:

HS_DATA_LINKED | value_size = sizeof(MultiBlkId) > 0 | value = tombstone bytes (blk_count =
0)

code and value_size are unchanged, but the value bytes now encode a zero blkid.

Phase 2 — Any subsequent post-upgrade follower receives the already-corrupted entry

Once the corrupted entry exists in any SM's local log, any new post-upgrade follower that
catches
up from that SM receives:

HS_DATA_LINKED | value_size > 0 | value = tombstone bytes

localize_journal_entry_prepare:
entry_blkid.deserialize(...) → tombstone_pbas, blk_count() = 0
data_size = 0 × blk_size = 0

Condition: (code == HS_DATA_LINKED) && (value_size > 0) → TRUE (both hold)
→ takes HS_DATA_LINKED branch
→ applier_create_req(op_code = HS_DATA_LINKED, data_size = 0)
→ init(): has_linked_data() = true, data_size = 0
→ alloc_local_blks(listener, data_size = 0)
→ get_blk_alloc_hints(data_size = 0)
data_size == 0 branch → returns empty blk_alloc_hints{} ← BUG
→ committed_blk_id.has_value() = false
→ falls through to data_service().alloc_blks()
→ VirtualDev::alloc_blks() → HeapChunkSelector::select_chunk()
→ RELEASE_ASSERT(false, "create shard is log only...") → SIGABRT

@codecov-commenter

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 0% with 4 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (stable/v4.x@4b25833). Learn more about missing BASE report.

Files with missing lines Patch % Lines
...ib/homestore_backend/replication_state_machine.cpp 0.00% 4 Missing ⚠️
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.
Additional details and impacted files
@@              Coverage Diff               @@
##             stable/v4.x     #450   +/-   ##
==============================================
  Coverage               ?   55.08%           
==============================================
  Files                  ?       36           
  Lines                  ?     5310           
  Branches               ?      671           
==============================================
  Hits                   ?     2925           
  Misses                 ?     2087           
  Partials               ?      298           

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@Hooper9973 Hooper9973 merged commit 924e403 into eBay:stable/v4.x Jul 9, 2026
48 of 49 checks passed
@Hooper9973 Hooper9973 deleted the SDSTOR-23030-fix-compatibility branch July 9, 2026 00:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants