Soundness: OOB heap reads via unchecked tape indexing in public safe deserialization APIs

[Manishearth]

Note

This finding was identified during an agentic unsafe Rust code review performed by Gemini AI, followed by human review and verification.

The Issue

In src/lib.rs, simd_json::Deserializer defines an internal helper method next_ that retrieves intermediate tape nodes. In release builds (not(debug_assertions)), next_ retrieves nodes from self.tape using get_kinda_unchecked, which executes slice::get_unchecked without bounds verification

simd-json/src/lib.rs

Lines 875 to 879 in f8e5d67

	pub unsafe fn next_(&mut self) -> Node<'de> {
	let r = *unsafe { self.tape.get_kinda_unchecked(self.idx) };
	self.idx += 1;
	r
	}

This internal helper is unconditionally invoked by primitive integer and floating-point parsing helpers (such as parse_u8()) used in the public safe serde::Deserializer trait implementation

simd-json/src/serde.rs

Lines 220 to 227 in f8e5d67

	fn parse_u8(&mut self) -> Result<u8> {
	match unsafe { self.next_() } {
	Node::Static(s) => s
	.as_u8()
	.ok_or_else(|| Self::error(ErrorType::ExpectedUnsigned)),
	_ => Err(Self::error(ErrorType::ExpectedUnsigned)),
	}
	}

Additionally, next_() is called without bounds checks by public safe DOM tree parsing wrappers OwnedDeserializer::parse and BorrowDeserializer::parse

simd-json/src/value/owned.rs

Lines 307 to 314 in f8e5d67

	pub fn parse(&mut self) -> Value {
	match unsafe { self.de.next_() } {
	Node::Static(s) => Value::Static(s),
	Node::String(s) => Value::from(s),
	Node::Array { len, count: _ } => self.parse_array(len),
	Node::Object { len, count: _ } => self.parse_map(len),
	}
	}

simd-json/src/value/borrowed.rs

Lines 394 to 401 in f8e5d67

	pub fn parse(&mut self) -> Value<'de> {
	match unsafe { self.0.next_() } {
	Node::Static(s) => Value::Static(s),
	Node::String(s) => Value::from(s),
	Node::Array { len, count: _ } => self.parse_array(len),
	Node::Object { len, count: _ } => self.parse_map(len),
	}
	}

serde::Deserializer trait methods and DOM .parse() methods are public safe interfaces. External safe Rust code can call deserialization methods arbitrarily many times on any valid Deserializer instance. When invoked on an empty tape or after all valid tape nodes have been consumed (self.idx >= self.tape.len()), these APIs execute out-of-bounds reads on heap memory backing self.tape, resulting in immediate Undefined Behavior.

Minimal Reproduction (Miri)

use serde::Deserialize;

fn main() {
    let mut data = b"123".to_vec();
    let mut de = simd_json::Deserializer::from_slice(&mut data)
        .expect("failed to initialize deserializer");

    // First deserialization consumes the valid tape node.
    let _val1 = u8::deserialize(&mut de).expect("first deserialization");

    // Second deserialization call in safe code on the same deserializer instance.
    // The tape index is now exhausted (idx == tape.len()), but internal helper next_()
    // performs unchecked slice indexing (get_unchecked) on the tape buffer,
    // triggering an out-of-bounds heap read and Undefined Behavior.
    let _val2 = u8::deserialize(&mut de);
}

error: Undefined Behavior: `assume` called with `false`
  --> /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/simd-json-0.17.0/src/safer_unchecked.rs:25:26
   |
25 |         let r = unsafe { self.get_unchecked(index) };
   |                          ^^^^^^^^^^^^^^^^^^^^^^^^^ Undefined Behavior occurred here
   |
   = help: this indicates a bug program: it performed an invalid operation, and caused Undefined Behavior
   = help: see https://doc.rust-lang.org/nightly/reference/behavior-considered-undefined.html for further information
   = note: stack backtrace:
           0: <[simd_json::Node<'_>] as simd_json::safer_unchecked::GetSaferUnchecked<simd_json::Node<'_>>>::get_kinda_unchecked::<usize>
               at /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/simd-json-0.17.0/src/safer_unchecked.rs:25:26: 25:51
           1: simd_json::Deserializer::<'_>::next_
               at /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/simd-json-0.17.0/src/lib.rs:876:27: 876:66
           2: simd_json::serde::<impl simd_json::Deserializer<'_>>::parse_u8
               at /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/simd-json-0.17.0/src/serde.rs:221:24: 221:36
           3: simd_json::serde::de::<impl serde::Deserializer<'_> for &mut simd_json::Deserializer<'_>>::deserialize_u8::<serde::de::impls::<impl serde::Deserialize<'de> for u8>::deserialize::PrimitiveVisitor>
               at /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/simd-json-0.17.0/src/serde/de.rs:140:32: 140:47
           4: serde::de::impls::<impl serde::Deserialize<'_> for u8>::deserialize::<&mut simd_json::Deserializer<'_>>
               at /usr/local/google/home/manishearth/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/serde_core-1.0.228/src/de/impls.rs:148:17: 148:60
           5: main
               at src/bin/repro1.rs:15:17: 15:41

note: some details are omitted, run with `MIRIFLAGS=-Zmiri-backtrace=full` for a verbose backtrace

error: aborting due to 1 previous error

Suggested Fix

Either perform safe bounds-checked indexing (self.tape.get(self.idx)) inside Deserializer::next_, or ensure that all public safe wrappers and trait implementations verify self.idx < self.tape.len() before calling unsafe { self.next_() }.

Note

The full audit report below also contains additional minor findings (such as missing safety comments or undocumented FFI assumptions) that are probably worth fixing as well but not the primary goal of this issue. The audit report has not been human-reviewed, it may contain misleading claims.

Full Gemini Codebase Audit Report Appendix

Unsafe Rust Review: simd_json (v0_17)

Overall Safety Assessment

simd_json is a high-performance, SIMD-accelerated JSON parser for Rust inspired by the simdjson C++ library. The crate parses raw JSON byte buffers in two stages: Stage 1 utilizes architecture-specific SIMD intrinsics (SSE4.2, AVX2, NEON, SIMD128) to build index bitmasks of structural characters ({, }, [, ], :, ,) and strings while validating UTF-8 encoding. Stage 2 walks these index bitmasks to construct a JSON Tape (an intermediate flattened representation of the document syntax tree) or directly populate Serde data models.

Unsafe Surface & Density

The crate exhibits a very high density of unsafe code (~176 distinct unsafe fn, unsafe trait, or unsafe {} blocks distributed across core SIMD parsing algorithms, string unescaping buffers, Serde trait wrappers, and DOM sequence builders). Key functional groupings of this unsafe surface include:

• SIMD Intrinsics & Stage 1 Parsing (src/impls/*/stage1.rs, src/lib.rs): Direct invocations of platform vector intrinsics (_mm256_loadu_si256, vld1q_u8, etc.), pointer casting, and unaligned bit manipulation to identify quotes, backslashes, and whitespace.
• Unchecked Slice Indexing (src/safer_unchecked.rs, src/stage2.rs, src/numberparse/): Extensive use of internal wrappers (GetSaferUnchecked) delegating to slice::get_unchecked in release builds (not(debug_assertions)) to eliminate bounds check branching on hot paths.
• In-Place Buffer Mutability & String Unescaping (src/impls/*/deser.rs, src/stringparse.rs, src/charutils.rs): Pointer arithmetic and unchecked UTF-8 construction (from_utf8_unchecked) when unescaping JSON strings directly within temporary vector buffers (string_buffer) or in-place within input buffers.
• Serde Deserialization & DOM Tree Construction (src/serde.rs, src/serde/de.rs, src/value/owned.rs, src/value/borrowed.rs): Unchecked tape indexing (self.next_()) and raw pointer sequence initialization (res_ptr.add(i).write(...)).

Architectural Soundness & Audit Proofs

Our audit reveals that simd_json version v0_17 is Unsound due to severe encapsulation failures in its public Serde and DOM tree deserialization APIs.

While internal SIMD algorithms rely on a strict invariant that all input buffers are padded with at least 64 bytes of initialized zero bytes (SIMDINPUT_LENGTH)—ensuring that SIMD unaligned vector reads and fixed-width lookahead window accesses remain inbounds—the public Rust API fails to uphold Rust's safety guarantees:

1. Public Safe API Soundness: The crate exports standard Serde deserialization traits (serde::Deserializer) and DOM sequence parsers (OwnedDeserializer::parse) as unconditionally safe interfaces while internally performing unvalidated out-of-bounds indexing (get_unchecked) on tape structures.
2. Buffer Initialization Hygiene: The codebase invokes Vec::set_len on uninitialized byte buffers (Vec<u8>), creating instant Undefined Behavior under Rust Reference rules.
3. Safety Documentation: The crate exhibits almost zero safety documentation hygiene (~176 missing // SAFETY: comments or # Safety docstrings), frequently relying on implicit assumptions regarding data layout and call tree synchrony.

Critical Findings

1. Out-of-Bounds Heap Reads via Unchecked Deserializer::next_ in Public Safe Serde APIs (src/serde.rs:221-332, src/serde/de.rs:491) 🔴 🚨

• Priority: 🔴 High
• Threat Vector: 🚨 Untrusted Input
• Bug Type: Out-of-Bounds Read
• Locations: src/serde.rs:221, 232, 244, 255, 266, 277, 288, 299, 310, 321, 332, src/serde/de.rs:491.
• Observation: simd_json::serde::Deserializer implements standard Serde deserialization traits (serde::Deserializer for &mut Deserializer<'de>). For all primitive integer and floating-point types (deserialize_u8, deserialize_u16, deserialize_u32, deserialize_u64, deserialize_u128, deserialize_i8, deserialize_i16, deserialize_i32, deserialize_i64, deserialize_i128, deserialize_f32, deserialize_f64), as well as map keys (MapKey::deserialize_any integer macro), parsing delegates to internal helpers (parse_u8(), etc.). These helpers unconditionally invoke unsafe { self.next_() }. .next_() retrieves nodes via self.tape.get_kinda_unchecked(self.idx) and increments self.idx. In release builds (not(debug_assertions)), get_kinda_unchecked executes slice::get_unchecked without verifying self.idx < self.tape.len().
• Soundness Violation: In Rust safety contracts, serde::Deserializer is a public safe trait. External safe Rust code (such as custom Deserialize implementations or direct trait method invocations) can call e.g. u8::deserialize(&mut deserializer) arbitrarily many times on any valid deserializer instance. If invoked on an empty tape or more times than there are tape nodes, this executes out-of-bounds reads on heap memory backing self.tape. Under Rust Reference § Behavior considered undefined, out-of-bounds memory accesses constitute immediate Undefined Behavior (UB) and represent an exploitable memory safety vulnerability.

2. Out-of-Bounds Heap Reads via Public Safe OwnedDeserializer::parse and BorrowedDeserializer::parse (src/value/owned.rs:307, src/value/borrowed.rs:394) 🔴 ⚠️

• Priority: 🔴 High

• Threat Vector: ⚠️ Accidental Misuse

• Bug Type: Out-of-Bounds Read

• Locations: src/value/owned.rs:307, src/value/borrowed.rs:394.

• Observation: Both OwnedDeserializer::parse(&mut self) and BorrowedDeserializer::parse(&mut self) are exported as unconditionally safe public methods (pub fn parse). They can be instantiated from any Deserializer via the safe public constructor from_deserializer.

• Soundness Violation: Internally, .parse() calls unsafe { self.de.next_() } or unsafe { self.0.next_() } without bounds verification. Calling .parse() from safe code on a deserializer whose tape index is exhausted (idx >= tape.len()) invokes Vec::get_unchecked out of bounds in release builds, triggering immediate Undefined Behavior.

Fishy Findings

1. UB via Vec::set_len Exposing Uninitialized Memory in Buffers::string_buffer (src/lib.rs:810) 🟠 ⚠️

• Priority: 🟠 Medium

• Threat Vector: ⚠️ Accidental Misuse

• Bug Type: Uninitialized Memory Exposure

• Location: src/lib.rs:810.

• Observation: In Deserializer::from_slice_with_buffers, string parsing scratch space is prepared via: buffer.string_buffer.clear(); buffer.string_buffer.reserve(len + SIMDJSON_PADDING); unsafe { buffer.string_buffer.set_len(len + SIMDJSON_PADDING); };

• Analysis: According to authoritative standard library documentation (Vec::set_len), the caller must guarantee that all elements up to new_len are fully initialized. Exposing uninitialized memory as integers (u8) constitutes immediate Undefined Behavior under Rust Reference § Behavior considered undefined ("producing an integer... that is uninitialized"). While string_buffer is a private field of Buffers and is overwritten during unescaping, leaving uninitialized integers in a vector risks severe compiler miscompilation during vector resizing, copying, or drop.

2. Public Safe Functions codepoint_to_utf8 and hex_to_u32_nocheck Lacking Bounds Checks (src/charutils.rs:54, 86) 🟠 ⚠️

• Priority: 🟠 Medium

• Threat Vector: ⚠️ Accidental Misuse

• Bug Type: Missing Bounds Check

• Locations: src/charutils.rs:54, 86.

• Observation: Both hex_to_u32_nocheck(src: &[u8]) and codepoint_to_utf8(cp: u32, c: &mut [u8]) are exported as safe public functions (pub fn). Internally, they index slices via src.get_kinda_unchecked(0..4) and c.get_kinda_unchecked_mut(0).

• Analysis: In release builds (not(debug_assertions)), get_kinda_unchecked executes slice::get_unchecked. Calling either function from safe external code with an empty or short slice (e.g., hex_to_u32_nocheck(&[]) or codepoint_to_utf8(0x20, &mut [])) triggers out-of-bounds slice reads/writes, breaking safe encapsulation.

3. Memory Leaks of DOM Trees During Panic Unwinding in Sequence Construction (src/value/owned.rs:323, src/value/borrowed.rs:410, src/stage2.rs:151) 🟡 ⚠️

• Priority: 🟡 Low

• Threat Vector: ⚠️ Accidental Misuse

• Bug Type: Memory Leak

• Locations: src/value/owned.rs:323, src/value/borrowed.rs:410, src/stage2.rs:151.

• Observation: In parse_array and build_tape, uninitialized vector buffers are populated via raw pointer writes (res_ptr.add(i).write(...)) prior to updating the vector length (res.set_len()).

• Analysis: If recursive parsing (self.parse()) panics midway through sequence construction, stack unwinding drops res while its length is recorded as 0. Previously initialized elements containing allocated heap pointers (such as Box<Vec<Value>> or Object) are bypassed during RAII teardown, permanently leaking memory. While memory leaks during panic are safe under general Rust semantics, bypassing RAII cleanup on complex recursive DOM structures reflects fragile unsafe state management.

4. Undocumented Proof Obligations on Unsafe Trait Declarations (src/lib.rs:153-163, src/safer_unchecked.rs:4-8) 🟡 ⚠️

• Priority: 🟡 Low

• Threat Vector: ⚠️ Accidental Misuse

• Bug Type: Missing Safety Documentation

• Locations: src/lib.rs:153-163, src/safer_unchecked.rs:4-8.

• Observation: Across Stage1Parse and GetSaferUnchecked, multiple trait methods are declared as unsafe fn (e.g., new, compute_quote_mask, get_kinda_unchecked).

• Analysis: None of these trait declarations provide # Safety docstrings defining the exact caller proof obligations required to prevent UB (such as SIMD register alignment, zero padding guarantees, or slice index validity).

Missing Safety Comments

Across the crate, ~176 locations lack required // SAFETY: proof comments or # Safety docstrings. We enumerate every location below with formal proof obligations.

Module src/lib.rs (14 locations) 🔴

• src/lib.rs:153, 155, 157, 159, 161, 163, 288 (Stage1Parse trait methods new, compute_quote_mask, cmp_mask_against_input, unsigned_lteq_against_input, find_whitespace_and_structurals, flatten_bits, fill_s8): Missing # Safety docstrings.
• Proposed Theorem: Precondition: ptr / self must point to an input byte slice padded with at least SIMDINPUT_LENGTH initialized zero bytes.
• src/lib.rs:184 (unsafe { *quote_bits = ... } in find_quote_mask_and_bits): Missing // SAFETY: comment.
• Proposed Proof: SIMD lookahead bounds are guaranteed by Stage1Parse caller padding precondition.
• src/lib.rs:219 (unsafe { self.cmp_mask_against_input(b'\\') } in find_odd_backslash_sequences): Missing // SAFETY: comment.
• Proposed Proof: Discharged by trait caller padding precondition.
• src/lib.rs:459 (pub(crate) unsafe fn parse_str_): Missing # Safety docstring.
• Proposed Theorem: Precondition: data must be padded with SIMDINPUT_LENGTH bytes; idx must be a valid quoted string start index within data.
• src/lib.rs:468 (unsafe { ... } in parse_str_): Missing // SAFETY: comment.
• Proposed Proof: Discharged by parse_str_ caller bounds and architecture detection preconditions.
• src/lib.rs:809 (unsafe { buffer.string_buffer.set_len(...) }): Missing // SAFETY: comment.
• Proposed Proof: (Unsound / Fishy Finding Serde #1 - requires initialization prior to setting length).
• src/lib.rs:818 (unsafe { input_buffer.copy_from_nonoverlapping(...) }): Missing // SAFETY: comment.
• Proposed Proof: input_buffer capacity verified >= len + SIMDINPUT_LENGTH; non-overlapping copy of len bytes followed by zero padding is inbounds.
• src/lib.rs:876 (let r = *unsafe { self.tape.get_kinda_unchecked(self.idx) } in next_): Missing // SAFETY: comment.
• Proposed Proof: Discharged by function precondition (self.idx < self.tape.len()).

Module src/serde.rs (13 locations) 🔴

• src/serde.rs:101 (unsafe { s.as_bytes_mut() } in from_str): Missing // SAFETY: comment.
• Proposed Proof: Caller contract permits temporary UTF-8 invalidation during mutable byte parsing.
• src/serde.rs:129 (unsafe { s.as_bytes_mut() } in from_str_with_buffers): Missing // SAFETY: comment.
• Proposed Proof: Discharged by function caller contract.
• src/serde.rs:221, 232, 244, 255, 266, 277, 288, 299, 310, 321, 332 (unsafe { self.next_() } in primitive integer/float parse methods): Missing // SAFETY: comments.
• Proposed Proof: (Unsound / Critical Finding Serde #1 - unvalidated tape index access on public Deserializer).

Module src/serde/de.rs (1 location) 🔴

• src/serde/de.rs:491 (unsafe { self.de.next_() } in deserialize_integer_key! macro): Missing // SAFETY: comment.
• Proposed Proof: (Unsound / Critical Finding Serde #1).

Module src/safer_unchecked.rs (6 locations) 🔴

• src/safer_unchecked.rs:4, 8 (GetSaferUnchecked trait declarations): Missing # Safety docstrings.
• Proposed Theorem: Precondition: index must be strictly inbounds of the slice (index < self.len()).
• src/safer_unchecked.rs:18, 30 (unsafe fn get_kinda_unchecked, get_kinda_unchecked_mut impls): Missing # Safety docstrings.
• src/safer_unchecked.rs:25, 37 (unsafe { self.get_unchecked(index) }): Missing // SAFETY: comments.
• Proposed Proof: Discharged by trait method caller bounds preconditions.

Module src/stage2.rs (10 locations) 🔴

• src/stage2.rs:15 (unsafe { ... } in is_valid_true_atom): Missing // SAFETY: comment.
• Proposed Proof: loc has length >= 8 due to 64-byte zero padding on input2. Reading 8 unaligned bytes is inbounds.
• src/stage2.rs:29 (unsafe { $a.get_kinda_unchecked($i) } in get! macro): Missing // SAFETY: comment.
• Proposed Proof: Structural index traversal strictly bounded by structural_indexes.len().
• src/stage2.rs:45, 70 (unsafe { loc.as_ptr().cast::<u64>().read_unaligned() } in is_valid_false_atom, null_atom): Missing // SAFETY: comments.
• Proposed Proof: Discharged by 64-byte input padding guarantee.
• src/stage2.rs:140, 159 (unsafe { res.set_len(r_i) } in s2try!, success!): Missing // SAFETY: comments.
• Proposed Proof: r_i elements were initialized via res_ptr.add(r_i).write(...) within reserved capacity.
• src/stage2.rs:151 (unsafe { res_ptr.add(r_i).write($t) } in insert_res!): Missing // SAFETY: comment.
• Proposed Proof: res reserved capacity matching structural_indexes.len(); r_i < capacity is maintained.
• src/stage2.rs:276 (pub(crate) unsafe fn parse_str_): Missing # Safety docstring.
• src/stage2.rs:289, 300 (unsafe slice accesses in parse_str_): Missing // SAFETY: comments.

Module src/stringparse.rs (5 locations) 🔴

• src/stringparse.rs:52, 53, 58, 59, 64 (src_ptr.get_kinda_unchecked in get_unicode_codepoint): Missing // SAFETY: comments.
• Proposed Proof: src_ptr is backed by input2 with 64-byte zero padding; lookahead up to 12 bytes remains strictly inbounds.

Module src/charutils.rs (4 locations) 🔴

• src/charutils.rs:27, 32 (get_kinda_unchecked in structural lookup): Missing // SAFETY: comments.
• Proposed Proof: c is u8 (0..255); lookup tables have 256 elements. Access is unconditionally inbounds.
• src/charutils.rs:59 (DIGITTOVAL.get_kinda_unchecked in hex_to_u32_nocheck): Missing // SAFETY: comment.
• Proposed Proof: (Unsound / Fishy Finding Investigate slow float parsing #2 - exported safe API missing slice length precondition check).
• src/charutils.rs:87 (c.get_kinda_unchecked_mut(0) in codepoint_to_utf8): Missing // SAFETY: comment.
• Proposed Proof: (Unsound / Fishy Finding Investigate slow float parsing #2).

Modules src/numberparse/correct.rs and src/numberparse/approx.rs (18 locations) 🔴

• src/numberparse/correct.rs:37, 107, 110, 175, 187, 196, 387, 410: Missing // SAFETY: comments.
• Proposed Proof: buf is backed by input2 with 64-byte zero padding; 8-byte SWAR integer lookahead is strictly inbounds.
• src/numberparse/approx.rs:86, 91, 94, 95, 100, 105, 118, 121, 130, 131: Missing // SAFETY: comments.
• Proposed Proof: Discharged by 64-byte input padding guarantee on float lookahead windows.

Modules src/value/borrowed.rs and src/value/owned.rs (15 locations) 🔴

• src/value/borrowed.rs:113, 143 (unsafe { std::mem::transmute(...) } in static lifetime coercion): Missing // SAFETY: comments.
• Proposed Proof: Owned String / Vec heap allocations are valid for any lifetime parameter 'a, including 'static.
• src/value/borrowed.rs:395, src/value/owned.rs:308 (unsafe { next_() } in .parse()): Missing // SAFETY: comments.
• Proposed Proof: (Unsound / Critical Finding Investigate slow float parsing #2 - public safe method lacking tape bounds verification).
• src/value/borrowed.rs:410, 426, 428, 449, 450, 457, 472, src/value/owned.rs:323, 337, 339, 836: Missing // SAFETY: comments justifying DOM vector population and key deduplication lookups.

SIMD Architecture Backends (src/impls/*/deser.rs and stage1.rs) (~90 locations) 🔴

• src/impls/*/deser.rs (native, sse42, avx2, neon, simd128) (36 locations): Every backend unsafe fn parse_str lacks # Safety docstrings. Inside string scanning and unescaping loops, invocations of _mm_loadu_si128, vld1q_u8, from_utf8_unchecked, and copy_from_nonoverlapping lack // SAFETY: comments.
• Proposed Proof: Vector loads are unaligned and bounded by 32-byte SIMDJSON_PADDING on string buffers and 64-byte SIMDINPUT_LENGTH on input buffers. UTF-8 validity verified during Stage 1 structural scan.
• src/impls/*/stage1.rs (native, sse42, avx2, neon, simd128, portable) (54 locations): Missing // SAFETY: comments on unaligned SIMD block loads, pointer offsets (.add), and bitmask manipulation (_mm256_movemask_epi8).
• Proposed Proof: Discharged by Stage1Parse caller guarantee of 64-byte zero-padded alignment buffers.