API: Rewrite string truncate equality predicates onto the source column - #16362
API: Rewrite string truncate equality predicates onto the source column#16362wombatu-kun wants to merge 1 commit into
Conversation
e78b03e to
fc66672
Compare
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
No-no-no, it's not stale! |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Generated-by: Claude Code (claude-opus-5)
fc66672 to
bb9d8c9
Compare
What
Resolves the long-standing
TODO: translate truncate(col) == value to startsWith(value)inUnboundPredicate.bindLiteralOperation. When the term of anEQ/NOT_EQpredicate is a stringtruncate[W]transform, binding now produces an exactly-equivalent predicate on the untransformed source column.The equivalence depends on the literal length vs. the truncate width
W:truncate[W](col) == vtruncate[W](col) != vlen(v) > WalwaysFalse()alwaysTrue()len(v) == Wcol STARTS_WITH vcol NOT_STARTS_WITH vlen(v) < Wcol == vcol != vInteger/long/decimal/binary truncate and all other operators are intentionally left unchanged — they have no exact source-column equivalence.
Why
This rewrite is already assumed by the rest of the engine.
InclusiveMetricsEvaluator.startsWith()returnsROWS_MIGHT_MATCHfor non-identity transform terms with the explicit comment "truncate must be rewritten in binding". Until now the binder never performed that rewrite for equality, sotruncate(col) == vkept an opaqueBoundTransformterm and metrics/dictionary/partition pruning could not use the column. After this change such predicates prune correctly (e.g.equal(truncate("str",3),"xyz")against bounds["abc","abe"]now skips the file instead of reading it).Implementation notes
< / == / > widthdecision is centralized in a singleTruncate.lengthRewritehelper, shared by predicate binding and byTruncateString.project/projectStrict, so the two paths cannot diverge.BoundTransformterm would otherwise defeatProjectionUtil.projectTransformPredicate(which matches partition transforms bytoString()) and collapse strict projection toFalse. To preserve the previous precision,TruncateStringnow projects EXACT-length (len(v) < W)EQ/NOT_EQpredicates directly onto the partition value — provably the same result the old transform-term path produced, sincetruncate[W](x) == v ⟺ x == vwhenlen(v) < W.Transforms.StringTruncateRewriteenum +Transforms.stringTruncateRewrite);revapipasses with no accepted-breaks entry.Testing
TestPredicateBinding: all three length classes forEQ/NOT_EQ, empty-string literal, plus negatives (non-string truncate, non-truncate transforms, other operators unchanged).TestStartsWith: runtime-equivalence check viaEvaluator, including the EXACT-vs-prefix distinction.TestInclusiveMetricsEvaluatorWithTransforms: a pruning case that now prunes where it previously could not.:iceberg-api:testand:iceberg-core:test, all transform/projection/residual regression suites,spotlessCheck, and:iceberg-api:revapipass.🤖 Generated with Claude Code
AI Disclosure