Skip to content

fix(value): [OBE-10735] consolidate and raise the array-index cap - #14

Open
JuanMantica45 wants to merge 1 commit into
Sentinel-One:mainfrom
JuanMantica45:obe-10735-array-index-cap
Open

fix(value): [OBE-10735] consolidate and raise the array-index cap#14
JuanMantica45 wants to merge 1 commit into
Sentinel-One:mainfrom
JuanMantica45:obe-10735-array-index-cap

Conversation

@JuanMantica45

Copy link
Copy Markdown
Contributor

Why

Supersedes #12, which is conflicting with main. Same substance, rebased and self-contained.

main declares the bound twice — MAX_ARRAY_INDEX = 32_768 (crud/mod.rs) enforces it, MAX_ARRAY_CAPACITY = 32_769 (crud/insert.rs) preallocates for it. Two constants for one invariant means they can drift. This collapses them into one pub(super) constant and raises it to 2^20, per @ajayshekar-s1's feedback on #7 that 32_768 could reject legitimate large-array use.

The cap stays, and it cannot be optimised away

The review question on #7 was whether the cap should exist at all. It has to, and not for the reason the original fix implied.

Assigning to index N materialises N + 1 elements. The null padding is observable VRL semantics, not a preallocation hint — length() sees it, and test_insert_array already asserts c[2] = 10 yields [5, null, 10]. So removing Vec::with_capacity would only trade one large allocation for amortised doubling; the array still ends up holding N + 1 elements. There is no version of this where an event-controlled index does not commit memory proportional to the index. Bounding it is the only fix that does not change the language.

2^20 bounds a single indexed write to ~42 MB at today's 40-byte Value — 32x more headroom than the original cap, while staying bounded rather than unbounded.

What changed

  • One pub(super) const MAX_ARRAY_INDEX = 1_048_576 in crud/mod.rs; insert.rs derives its capacity bound as MAX_ARRAY_INDEX + 1 instead of redeclaring it. grep MAX_ARRAY_CAPACITY src/ now returns nothing.
  • index.unsigned_abs() replaces (-index) as usize in the capacity calculation, which overflows on isize::MIN (no positive isize counterpart). insert_value already used unsigned_abs; this was the remaining call site. Found by @jsbalis1 in review of fix(security): prevent 9 panic/OOM vectors in VRL runtime (batch J) #7.
  • test_value_size_is_pinned asserts size_of::<Value>() is 40. It is a drift detector, not a correctness assertion: the cap is justified in terms of memory (cap x size), so a new Value variant should force a human to re-check the budget rather than silently changing it.

Test plan

  • cargo test --lib: 1761 passed, 0 failed.
  • New/updated in crud::insert: paired accept-at-2^20 / reject-at-2^20+1 (both signs), isize::MIN no-panic, and the size_of pin. The accept-at-cap and isize::MIN tests both fail against main — the latter with attempt to negate with overflow at insert.rs:33.
  • lib/tests fixture runner: new obe 10735 array index cap passes. Suite goes 762 -> 763 passed; the 2 parse_etld custom-PSL failures and 3 emit_metric clippy errors are present on unmodified main and are untouched here.

🤖 Generated with Claude Code

`MAX_ARRAY_INDEX` (32_768) and `MAX_ARRAY_CAPACITY` (32_769) were declared
separately in `crud/mod.rs` and `crud/insert.rs`, so the enforcement bound and
the preallocation bound could drift apart. Collapse them into one `pub(super)`
constant and raise it to 2^20, per review feedback that 32_768 could reject
legitimate large-array use.

The cap stays because the amplification is real and cannot be optimised away:
assigning to index N materialises N + 1 elements, and that null padding is
observable VRL semantics (`length` sees it, `test_insert_array` asserts it), not
merely a `with_capacity` hint. Dropping the preallocation would only trade one
large allocation for amortised doubling. 2^20 bounds a single indexed write to
~42 MB at today's 40-byte `Value`; `test_value_size_is_pinned` fails if that
size changes so the budget gets re-reviewed rather than silently drifting.

Also replaces `(-index) as usize` with `index.unsigned_abs()` in the capacity
calculation, which overflowed on `isize::MIN` (no positive `isize` counterpart).
`insert_value` already used `unsigned_abs`; this was the remaining call site.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant