Skip to content

[VL] Add native BROUND with captured ANSI and compatibility guards - #13047

Draft
luis4a0 wants to merge 1 commit into
apache:mainfrom
luis4a0:feature/spark-bround-delivery
Draft

luis4a0 wants to merge 1 commit into
apache:mainfrom
luis4a0:feature/spark-bround-delivery

Conversation

@luis4a0

@luis4a0 luis4a0 commented Sep 17, 2026

Copy link
Copy Markdown
Contributor

What changes are proposed in this pull request?

Wire Spark BROUND to the corresponding Velox functions, including routing
decimal inputs to decimal_bround.

Integral calls carry the Spark expression's captured ANSI flag as a native
constant. This preserves overflow behavior even when a DataFrame is analyzed
or physically planned before the session mode changes.

Add planning-time compatibility checks:

  • Scales outside the native [-400, 400] interval fall back to Spark. Checking
    before native execution is necessary because primitive initialization errors
    can be deferred.
  • Nonzero-scale FLOAT/DOUBLE BROUND falls back on JVMs older than Java 21 to
    preserve their decimal-conversion behavior. Scale-zero floating rounding and
    integral/decimal calls remain eligible for native execution.
  • NULL INTEGER scales are accepted. Scales not folded to INTEGER literals
    retain safe fallback.

Existing ROUND behavior is unchanged.

Dependency: this draft requires the companion upstream Velox BROUND change
to be included in Gluten's pinned Velox dependency before merge.
facebookincubator/velox#19072

The integration was built and exercised against public IBM Velox
dft-2026_09_14 with that same production patch applied:
https://github.com/IBM/velox/tree/dft-2026_09_14

The stock tag does not yet contain BROUND. This PR does not claim otherwise or
replace the dependency pin with an unrelated upstream build.

How was this patch tested?

  • Fresh native backend and public-dependency Spark 4.1 / Scala 2.13 builds.
  • Five focused native ROUND/BROUND execution and mapping tests passed, in
    addition to the earlier parser regressions. The new tests first failed
    against the baseline: missing BROUND registration and incorrect decimal
    function mapping.
  • Eight validator unit tests and two mode-serialization unit tests passed.
  • Nine Spark 4.1 integration tests passed on both Java 17 and Java 21. They
    assert the actual native BROUND projection or the intentional fallback,
    compare schemas and floating-point bits, and exercise integral overflow,
    decimal precision/overflow, folded and NULL scales (including native FLOAT
    and DOUBLE NULL-scale calls on both JVMs and execution modes), unsupported scales,
    and mode changes after analysis and after cached native physical planning.
    A remaining Spark BROUND in a partially native projection fails the check.
  • The companion native implementation matched registered public Spark 4.1.1
    interpreted/generated expressions on 5,535,698 distinct typed inputs
    (11,071,396 LEGACY/ANSI mode-cases per flat and dictionary/partial-selection
    shape
    ) with zero mismatches. Another 586,332 mode-cases per shape covered
    every supported scale.
  • The expression-mode overload also passed 5,452,400 integral evaluations with
    the native query mode deliberately set opposite to the captured mode.

The multi-million floating campaign uses JDK 21. A separate actual-JVM
comparison established the older-JDK differences that motivate the fallback;
461,532 native-eligible dense-scale cases matched between JDK 17 and JDK 21.
The evidence does not claim universal cross-JDK floating equivalence, exact
error-message equivalence, exhaustive correctness, or performance gains.
Six sequential OpenAI code-review rounds completed; findings were corrected
and rechecked before publication.

Was this patch authored or co-authored using generative AI tooling?

Generated-by: GitHub Copilot CLI 1.0.86-2; OpenAI models.

## Summary
Route decimal BROUND to decimal_bround and pass the Spark expression's
captured ANSI flag for integral calls. Add scale/JVM planning guards and
Spark 4.1 integration coverage that distinguishes complete native execution
from partial or full Spark fallback.

## Prompting Intent
Integrate the independently designed upstream BROUND implementation in
Apache Gluten, preserve Spark 4.1 correctness across execution modes and
supported JVMs, add and run regression tests, and prepare a draft companion
PR using only public implementation material and qualification evidence.

## Linked Sources
- Companion upstream Velox implementation:
  facebookincubator/velox#19072
- Spark 4.1.1 BRound and captured ANSI semantics:
  https://github.com/apache/spark/blob/v4.1.1/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala
- Apache Gluten architecture baseline:
  https://github.com/apache/gluten/tree/48b5fca64d72b863960bca8129d11cbfdfae03d8
- Public pinned Velox dependency used with the companion BROUND patch:
  https://github.com/IBM/velox/tree/dft-2026_09_14

## Rationale
Serialize expression-level ANSI state rather than reading a later session
setting; even an already materialized native physical plan must retain its
overflow policy. The backend hook keeps other backends' existing argument
shape unchanged. Check supported scales before native initialization can
defer errors, and retain Spark execution for pre-JDK21 nonzero floating
scales whose decimal-conversion semantics differ.

The tests cover both mode-change directions after analysis and physical
planning, native floating null scales, overflow, output types, and exact
floating bits. Existing ROUND execution remains covered. The companion
draft depends on the Velox change reaching the pinned dependency before
merge; the stock dependency is not claimed to contain the new function.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d412b830-d346-4c14-b070-3bc5a6cacba1
@github-actions

Copy link
Copy Markdown

Run Gluten Clickhouse CI on x86

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CORE works for Gluten Core DOCS VELOX

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant