[DONT MERGE][GLUTEN-12569][VL] Add Spark 4.2 build profile and shim layer - #13052
Open
akshaytayal wants to merge 14 commits into
Open
akshaytayal wants to merge 14 commits into
akshaytayal wants to merge 14 commits into
Conversation
Spark 4.2 added parameters to three case classes, which silently breaks positional extractor patterns: - CharType gained a collation parameter. - AppendDataExec / OverwriteByExpressionExec gained tableName and transaction. Match on type (and on the named field where a value is needed) instead, which is stable across all supported Spark versions. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit b2a23c0)
Spark 4.2 hoisted checkAnswer, checkDataset, assertCached and friends out of QueryTest into a new QueryTestBase trait, which SharedSparkSession now mixes in. Because GlutenQueryTest declared its own copies of those members, every suite mixing both inherited conflicting definitions. Extending QueryTest (an abstract class up to 4.1, a trait in 4.2) makes Gluten's versions genuine overrides on every supported Spark version, and removes the duplicated-member conflict. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit a0c9f51)
- Spark 4.2 bumps Netty to 4.2.13, which removed PlatformDependent.allocateDirectNoCleaner. Use ByteBuffer.allocateDirect, which works on every supported version. - Spark 4.2's spark-catalyst ships a patched copy of datasketches ResourceImpl, tripping the ban-duplicate-classes enforcer. Both artifacts are 'provided', so nothing extra is packaged; ignore it. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: a4b4451c-d099-4965-871d-12781176d082 (cherry picked from commit 5d4853d)
…rk 4.2
Spark 4.2 adds `getBinaryView` to `SpecializedGetters`, which makes
`PlaceholderRow` (and any other `InternalRowSparkCompatible` subclass) fail to
compile as non-abstract:
BatchCarrierRow.scala:110: error: class PlaceholderRow needs to be abstract.
Missing implementation for member of trait SpecializedGetters:
def getBinaryView(x$1: Int): org.apache.spark.unsafe.types.BinaryView
Handled with the existing cross-version seam: `SpecializedGettersSparkCompatible`
gains a `getBinaryView` stub and `InternalRowSparkCompatible` overrides it. The
`Nothing` return type conforms to whichever concrete type the running Spark
version declares, so one definition covers the whole 3.x/4.x matrix -- exactly
how getVariant/getGeography/getGeometry are already handled.
Verified with clean builds (stale target/ classes previously masked this):
spark-3.4, spark-3.5, spark-4.0, spark-4.1 and spark-4.2 all BUILD SUCCESS.
(cherry picked from commit 7210d79)
Spark 4.2 moved postDriverMetrics to SupportsCustomDriverMetrics and made the reported task metrics an explicit argument. Move doPostDriverMetrics out of the version-agnostic BatchScanExecTransformer into each BatchScanExecShim so the call site can differ per version. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5
Two Spark 4.2 renames that leak into version-agnostic code: - SampleExec.seed became Option[Long], resolved lazily via resolvedSeed. - KeyGroupedPartitioning was renamed to KeyedPartitioning. Both are now hidden behind SparkShims (getSampleSeed, isKeyGroupedPartitioning) so gluten-substrait and backends-velox stay version-agnostic. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5
Adds the `spark-4.2` Maven profile and a `shims/spark42` module cloned from `shims/spark41` with identifiers renamed. No behavioural change to existing Spark versions: `spark-4.2` is opt-in and no other profile is touched. - root pom.xml: new `spark-4.2` profile (spark.version 4.2.0, arrow 19.0.0 to track Spark 4.2's own Arrow, JDK 17+ / Scala 2.13 enforcers); `spark-4.2` added to the spark-version requireActiveProfile list. - shims/pom.xml: `spark-4.2` profile activates module `spark42`. - shims/spark42: 37 files, renamed spark41 -> spark42 / Spark41 -> Spark42. Shim selection needs no hardcoded version: SparkShimDescriptor.DESCRIPTOR is derived from the SPARK_COMPILE_VERSION build property. This commit does not compile on its own; the Spark 4.2 source-incompatibilities are fixed in the following commits so that each can be reverted independently. (cherry picked from commit c4e06e1) (cherry picked from commit f57b33e)
Implements the Spark 4.2 side of the shim contracts introduced earlier: doPostDriverMetrics on BatchScanExecShim, plus getSampleSeed and isKeyGroupedPartitioning on Spark42Shims. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5 (cherry picked from commit 464cfea)
Spark 4.2 replaced the storage-partitioned-join (SPJ) plumbing on `BatchScanExec`: - `StoragePartitionJoinParams` and the `KeyGroupedPartitionedScan` trait were removed; the constructor now takes `keyGroupedPartitioning: Option[Seq[Expression]]`. - `KeyGroupedPartitioning` was replaced by `KeyedPartitioning`, which exposes `partitionKeys: Seq[InternalRowComparableWrapper]`, `keyOrdering`, `keyRowOrdering` and `toGrouped` instead of `partitionValues`/`expressions`. - `filteredPartitions` became `Seq[Option[InputPartition]]`, runtime-filter pushdown moved to `PushDownUtils.pushRuntimeFilters`, and `postDriverMetrics` now requires the task-metrics argument. Changes are confined to `shims/spark42`; `gluten-core`, `gluten-substrait` and the backends are untouched. `Spark42Shims` and `BatchScanExecShim` keep the exact signatures of their spark41 counterparts, so the `keyGroupedPartitioning` propagation added in apache#12567 (ScanTransformerFactory) needs no change. KNOWN LIMITATION (Spark 4.2 only, marked DEGRADED in the code): `getCommonPartitionValues` returns `None`, and `orderPartitions` no longer applies `joinKeyPositions`, `reducers` or partially-clustered replication. Spark 4.2 moved these refinements off the scan node into `EnsureRequirements`/`GroupPartitionsExec`, so there is no scan-level equivalent to reproduce. The base fully-clustered SPJ path is unaffected. This is currently unreachable: the only callers of `getCommonPartitionValues` are `IcebergScanTransformer` and `PaimonScanTransformer`, and neither Iceberg nor Paimon publishes Spark 4.2 artifacts yet (both 404 on Maven Central, vs 200 for their 4.1 builds); both profiles are also activeByDefault=false. Gluten never populates `joinKeyPositions`/`reducers` anywhere. To be revisited when Iceberg/Paimon ship Spark 4.2 support. (cherry picked from commit 687d477) (cherry picked from commit 15a9182)
Spark 4.2 reworked the columnar vector API: - `WritableColumnVector` gains three abstract members — `putBytes(int,int,ByteBuffer,int)`, `putShortsFromIntsLittleEndian(...)` and `getBytesAsBinaryView(...)`. Added stub overrides to `WritableColumnVectorShim` following the file's existing style. - `GeographyVal` and `GeometryVal` were removed in 4.2 and `ColumnVector` instead exposes `getBinaryView` (`ColumnVector.java:297`). `ArrowColumnarArray` drops the two geo overrides and delegates `getBinaryView` to `data`. Verified against the 4.1.1 and 4.2.0 source trees: GeographyVal/GeometryVal exist only in 4.1.1, BinaryView only in 4.2.0. These are independent of the SPJ port and were masked until Scala compiled, since scala-compile-first runs before javac. (cherry picked from commit 4d14a83) (cherry picked from commit 1e94dfc)
Spark 4.2 reworked `MemoryStream` factory overloads: the `apply[A](numPartitions: Int)(encoder, sparkSession)` form no longer exists. The remaining explicit-session overload is `apply[A: Encoder](sparkSession: SparkSession, numPartitions: Int)`. Switch the Spark 4.2 shim to that overload; the encoder is supplied by the existing `A: Encoder` context bound. (cherry picked from commit 252505b) (cherry picked from commit 434b980)
…ark42 The CI license-header check inspects newly added files. Replace the short Apache form carried over from shims/spark41 with the canonical ASF header. Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com> Copilot-Session: 8ae7d6bd-f561-417a-a635-40fe24dc67a5 (cherry picked from commit c6eb1cc)
…lType shim PromotePrecision and the decimal-rescale path (incl. SparkShims.widerDecimalType) were removed from Gluten in GLUTEN-12991 (apache#13002). On current main the spark42 shim copies are unused, so remove the PromotePrecision shim class and the Spark42Shims.widerDecimalType override (which now overrides nothing).
|
Run Gluten Clickhouse CI on x86 |
baibaichen
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Adds the Apache Spark 4.2 build profile (
-Pspark-4.2) and theshims/spark42layer, so Gluten can compile and run against Spark 4.2.0.
Part of the Spark 4.2 support effort (#12569). This is the shim layer; the
gluten-ut/spark42unit-test module follows in separate PRs.Details
shims/spark42module (Spark42Shims+SparkShimProvider) implementingthe shim contract for Spark 4.2, including the SampleExec seed (SPARK-53564) and
KeyGroupedPartitioning->KeyedPartitioning(SPARK-53401) changes.postDriverMetrics,SpecializedGetters.getBinaryView, columnar-vector,MemoryStream, PythonUDFRunner and BatchScan SPJ shims for Spark 4.2.
main: the stalePromotePrecisionshim andwiderDecimalTypeoverride are dropped (removed upstream in GLUTEN-12991 / [GLUTEN-12991][CORE] Remove the leftover PromotePrecision shim and its decimal rescale path #13002).How was this patch tested
shims/common+shims/spark42compile under-Pspark-4.2 -Pscala-2.13 -Pjava-17 -Pbackends-velox(BUILD SUCCESS). Full Spark 4.2 UT validation lands with thegluten-ut/spark42 PRs.