Spark: Make the view stored-schema coercion configurable by bmorck · Pull Request #17499 · apache/iceberg

bmorck · 2026-08-03T18:35:09Z

This change adds a new spark.sql.iceberg.view.schema-binding-mode property, used in ResolveViews in Spark 4.1. Prior to this property, ResolveViews always wraps each output column in an UpCast that only does widening type coercion. So if the stored schema of the view has a narrower type than what the SQL actually produces, we get an error like the following:

[CANNOT_UP_CAST_DATATYPE] Cannot up cast id from "DOUBLE" to "BIGINT"

The new spark.sql.iceberg.view.schema-binding-mode introduces 3 modes to relax this behavior, taking its mode names from Spark's ViewSchemaMode:

value	coercion
`BINDING`	`UpCast(col, storedType)` — the current behaviour, and the default
`COMPENSATION`	`Cast(col, storedType, ansiEnabled = true)` — narrowing allowed
`TYPE_EVOLUTION`	no cast, so the view reports the types its SQL produces

Spark's has thespark.sql.legacy.viewSchemaBindingMode and spark.sql.legacy.viewSchemaCompensation confs to relax this behavior on the v1 SessionCatalog view path, but this doesn't apply to v2 views.

This change was adapted from: #17453

ResolveViews rebuilds a view's output from the stored schema, by position, wrapping each column in an UpCast. UpCast only widens, so a view whose stored type is narrower than what its SQL produces cannot be read at all, and there is no way to relax it. Add spark.sql.iceberg.view.schema-binding-mode, taking its mode names and coercions from Spark's ViewSchemaMode: BINDING (UpCast, the default and current behaviour), COMPENSATION (an ANSI cast, allowing narrowing) and TYPE_EVOLUTION (no cast, so the view reports the types its SQL produces). All three keep the stored column name and metadata. When the conf is unset, Spark's spark.sql.legacy.viewSchemaBindingMode and viewSchemaCompensation are honored instead, reproducing how SessionCatalog.castColToType treats SchemaUnsupported.

bryanck · 2026-08-11T00:42:10Z

+  // COMPENSATION permits any ANSI cast, which can truncate values or fail at runtime.
+  // TYPE_EVOLUTION applies no cast, so the view reports the types its SQL produces.
+  // When unset, Spark's spark.sql.legacy.viewSchemaBindingMode and
+  // spark.sql.legacy.viewSchemaCompensation are honored instead; neither can select TYPE_EVOLUTION.


Consider breaking this comment up and moving the mode comments next to the constant below

bryanck · 2026-08-11T00:49:49Z

+  // VIEW_SCHEMA_COMPENSATION. Referenced by name because they were added in Spark 4.0 and this
+  // rule is also compiled against Spark 3.5. Both default to true.
+  private val sparkViewSchemaBindingMode = "spark.sql.legacy.viewSchemaBindingMode"
+  private val sparkViewSchemaCompensation = "spark.sql.legacy.viewSchemaCompensation"


These are available in Spark 4.0 and 4.1, can we remove them here and only add them for 3.5?

Ended up removing these from the diff, see my reply below

bryanck · 2026-08-11T00:54:59Z

+        // Mirror SessionCatalog.castColToType: turning binding mode off selects SchemaUnsupported,
+        // which compensates with an ANSI cast unless compensation is turned off as well. Neither conf
+        // can select TYPE_EVOLUTION: in Spark that mode is requested per view, with
+        // CREATE or ALTER VIEW ... WITH SCHEMA TYPE EVOLUTION, and stored on the view itself.


Some of the comments are pretty verbose, are there opportunities to make them more concise?

bryanck · 2026-08-11T00:56:12Z

+  // When unset, Spark's spark.sql.legacy.viewSchemaBindingMode and
+  // spark.sql.legacy.viewSchemaCompensation are honored instead; neither can select TYPE_EVOLUTION.
+  public static final String VIEW_SCHEMA_BINDING_MODE =
+      "spark.sql.iceberg.view.schema-binding-mode";


This property should be added to the docs.

Also, it seems this feature is global to the session, is there value in having it per view?

Also, it seems this feature is global to the session, is there value in having it per view?

Do you mean something like spark.sql.iceberg.view.<qualifiedVIewName>.schema-binding-mode? If so I'm thinking perhaps we do that in a follow-up PR.

I think it is useful to have a conf that applies to the session globally. This allows for different default behavior, for example without knowing which views are being used by a particular job.

bryanck · 2026-08-11T01:00:17Z

+        // CREATE or ALTER VIEW ... WITH SCHEMA TYPE EVOLUTION, and stored on the view itself.
+        if (isExplicitlyFalse(sparkViewSchemaBindingMode) &&
+          !isExplicitlyFalse(sparkViewSchemaCompensation)) {
+          SparkSQLProperties.VIEW_SCHEMA_MODE_COMPENSATION


Will this be backwards compatible if someone had previously set spark.sql.legacy.viewSchemaBindingMode=false?

Thanks for catching this. This wouldn't be backwards compatible since this would override the default spark.sql.iceberg.view.schema-binding-mode=BINDING, and spark.sql.legacy.viewSchemaBindingMode currently has no effect on V2 views. So if someone had been using spark.sql.legacy.viewSchemaBindingMode for their v1 session catalog views in spark and was expecting it to have no impact on their v2 views, this would break that assumption.

After thinking a bit more about this, I think we should remove spark.sql.legacy.viewSchemaBindingMode and spark.sql.legacy.viewSchemaCompensation since they were originally intended only for Spark's V1 session catalog views and this would overload their original intent.

Went ahead and removed them

bryanck · 2026-08-11T01:05:37Z

+        if (mode == SparkSQLProperties.VIEW_SCHEMA_MODE_COMPENSATION) {
+          Cast(attr, expected.dataType, ansiEnabled = true)
+        } else if (mode == SparkSQLProperties.VIEW_SCHEMA_MODE_TYPE_EVOLUTION) {
+          attr


If we remove UpCast will we still be checking Iceberg field IDs? e.g. if a field was dropped and added back with the same name.

Another thing, will the type being reported (say with describe view) mismatch what is returned?

I did a deep dive with claude and it doesn't look like field IDs are consulted on this path as is. So this shouldn't change that. If a column is dropped and re-added under the same name, the view's SQL resolves against the table by name so it would bind to the new column before and after this change.

Looks like the relevant path is:

spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/source/SparkView.java#L80-L86
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/SparkSchemaUtil.java#L96-L98
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L76-L77
spark/v4.1/spark/src/main/java/org/apache/iceberg/spark/TypeToSparkType.java#L197-L202

SparkView.schema() (80) calls SparkSchemaUtil.convert (96), which visits with TypeToSparkType, where each field's metadata comes from fieldMetadata(field.fieldId()) (76) — and that returns Metadata.empty() for anything that isn't a metadata column

On your other question, when we enable TYPE_EVOLUTION mode that will create a mismatch between what is returned and what the describe view has. For the default BINDING mode this wouldn't be the case

Let me know what you think. I'm thinking that since TYPE_EVOLUTION is not the default and is up to the user's discretion to set, it may not be an issue if the type reported from the schema mismatches. The user in this case would set TYPE_EVOLUTION likely understanding that it will create drift here. And that this mode was the primary motivation for this change

The existing TYPE EVOLUTION feature in Spark's V1 Session catalog views also can create the same drift between stored schema and output type, so there is precedent at least. But open to dissent here

bryanck · 2026-08-11T01:12:56Z

+   * Read on every resolution rather than cached, so that SET takes effect within a session.
+   */
+  private def viewSchemaMode: String = {
+    spark.conf.getOption(SparkSQLProperties.VIEW_SCHEMA_BINDING_MODE) match {


I think we want to use the rule's conf instead of reading the session conf?

Falling back to spark.sql.legacy.viewSchemaBindingMode and viewSchemaCompensation would change behaviour on upgrade for anyone who had set them. Iceberg views are strict today regardless of those confs, and the fallback would silently switch them to an ANSI cast. Those confs are also internal and are documented as controlling the WITH SCHEMA clause for view DDL, which Iceberg views do not implement; relaxing the cast is a side effect of disabling that feature rather than the confs' stated purpose. The Iceberg conf now defaults to BINDING when unset, matching current behaviour exactly. This removes the two conf names, the fallback branch, the isExplicitlyFalse helper and the tests that covered them.

Drops the SessionCatalog.castColToType cross-reference, which is in the PR description, and a scaladoc line that restated the method name. Also drops the note about Preconditions and Scala 2.12: that applies to spark/v3.5, which is cross-built against 2.12, not to this file.

Rule extends SQLConfHelper, so conf is the settings the surrounding analysis is running under. getConfString also takes the default, which removes the separate fallback branch.

…figuration table

github-actions Bot added the spark label Aug 3, 2026

bmorck marked this pull request as ready for review August 3, 2026 18:51

bmorck changed the title ~~Spark 4.1: Make the view stored-schema coercion configurable~~ Spark: Make the view stored-schema coercion configurable Aug 3, 2026

bryanck reviewed Aug 11, 2026

View reviewed changes

Bobby Morck added 5 commits August 10, 2026 23:35

Spark 4.1: Read the schema-binding mode from the rule's conf

7fdebb8

Rule extends SQLConfHelper, so conf is the settings the surrounding analysis is running under. getConfString also takes the default, which removes the separate fallback branch.

Spark 4.1: Move the mode comments next to their constants

361996b

Docs: Add spark.sql.iceberg.view.schema-binding-mode to the Spark con…

2388e74

…figuration table

github-actions Bot added the docs label Aug 11, 2026

bmorck requested a review from bryanck August 11, 2026 05:04

Uh oh!

Conversation

bmorck commented Aug 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

bmorck Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bmorck commented Aug 3, 2026 •

edited

Loading

bmorck Aug 11, 2026 •

edited

Loading