fix(value): [OBE-10735] consolidate and raise the array-index cap - #14
Open
JuanMantica45 wants to merge 1 commit into
Open
fix(value): [OBE-10735] consolidate and raise the array-index cap#14JuanMantica45 wants to merge 1 commit into
JuanMantica45 wants to merge 1 commit into
Conversation
`MAX_ARRAY_INDEX` (32_768) and `MAX_ARRAY_CAPACITY` (32_769) were declared separately in `crud/mod.rs` and `crud/insert.rs`, so the enforcement bound and the preallocation bound could drift apart. Collapse them into one `pub(super)` constant and raise it to 2^20, per review feedback that 32_768 could reject legitimate large-array use. The cap stays because the amplification is real and cannot be optimised away: assigning to index N materialises N + 1 elements, and that null padding is observable VRL semantics (`length` sees it, `test_insert_array` asserts it), not merely a `with_capacity` hint. Dropping the preallocation would only trade one large allocation for amortised doubling. 2^20 bounds a single indexed write to ~42 MB at today's 40-byte `Value`; `test_value_size_is_pinned` fails if that size changes so the budget gets re-reviewed rather than silently drifting. Also replaces `(-index) as usize` with `index.unsigned_abs()` in the capacity calculation, which overflowed on `isize::MIN` (no positive `isize` counterpart). `insert_value` already used `unsigned_abs`; this was the remaining call site. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
Supersedes #12, which is conflicting with
main. Same substance, rebased and self-contained.maindeclares the bound twice —MAX_ARRAY_INDEX = 32_768(crud/mod.rs) enforces it,MAX_ARRAY_CAPACITY = 32_769(crud/insert.rs) preallocates for it. Two constants for one invariant means they can drift. This collapses them into onepub(super)constant and raises it to 2^20, per @ajayshekar-s1's feedback on #7 that 32_768 could reject legitimate large-array use.The cap stays, and it cannot be optimised away
The review question on #7 was whether the cap should exist at all. It has to, and not for the reason the original fix implied.
Assigning to index
NmaterialisesN + 1elements. The null padding is observable VRL semantics, not a preallocation hint —length()sees it, andtest_insert_arrayalready assertsc[2] = 10yields[5, null, 10]. So removingVec::with_capacitywould only trade one large allocation for amortised doubling; the array still ends up holdingN + 1elements. There is no version of this where an event-controlled index does not commit memory proportional to the index. Bounding it is the only fix that does not change the language.2^20 bounds a single indexed write to ~42 MB at today's 40-byte
Value— 32x more headroom than the original cap, while staying bounded rather than unbounded.What changed
pub(super) const MAX_ARRAY_INDEX = 1_048_576incrud/mod.rs;insert.rsderives its capacity bound asMAX_ARRAY_INDEX + 1instead of redeclaring it.grep MAX_ARRAY_CAPACITY src/now returns nothing.index.unsigned_abs()replaces(-index) as usizein the capacity calculation, which overflows onisize::MIN(no positiveisizecounterpart).insert_valuealready usedunsigned_abs; this was the remaining call site. Found by @jsbalis1 in review of fix(security): prevent 9 panic/OOM vectors in VRL runtime (batch J) #7.test_value_size_is_pinnedassertssize_of::<Value>()is 40. It is a drift detector, not a correctness assertion: the cap is justified in terms of memory (cap x size), so a newValuevariant should force a human to re-check the budget rather than silently changing it.Test plan
cargo test --lib: 1761 passed, 0 failed.crud::insert: paired accept-at-2^20 / reject-at-2^20+1 (both signs),isize::MINno-panic, and thesize_ofpin. The accept-at-cap andisize::MINtests both fail againstmain— the latter withattempt to negate with overflowatinsert.rs:33.lib/testsfixture runner: newobe 10735 array index cappasses. Suite goes 762 -> 763 passed; the 2parse_etldcustom-PSL failures and 3emit_metricclippy errors are present on unmodifiedmainand are untouched here.🤖 Generated with Claude Code