Skip to content

[SPARK-58876][SQL][FOLLOWUP] Map sub-microsecond Oracle TIMESTAMP to nanos NTZ #58809

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
aleksandar-trajkovic-db wants to merge 1 commit into
base: master
Choose a base branch
from
+49 −21
Open

[SPARK-58876][SQL][FOLLOWUP] Map sub-microsecond Oracle TIMESTAMP to nanos NTZ #58809

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
23 changes: 14 additions & 9 deletions sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -193,6 +193,19 @@ object JdbcUtils extends Logging with SQLConfHelper {
if (isTimestampNTZ) TimestampNTZNanosType(precision) else TimestampLTZNanosType(precision)
}

// When nanosecond timestamps are requested (and the preview feature is enabled), a driver
// TIMESTAMP that reports a sub-microsecond fractional-second scale (7-9) is mapped to the
// nanosecond-capable type. Otherwise the historical microsecond mapping is preserved.
def resolveTimestampType(
isTimestampNTZ: Boolean, scale: Int, preferTimestampNanos: Boolean): DataType = {
if (preferTimestampNanos &&
scale >= TimestampNTZNanosType.MIN_PRECISION &&
scale <= TimestampNTZNanosType.MAX_PRECISION &&
conf.timestampNanosTypesEnabled) {
getTimestampNanosType(isTimestampNTZ, scale)
} else getTimestampType(isTimestampNTZ)
}

/**
* Maps a JDBC type to a Catalyst type. This function is called only when
* the JdbcDialect class corresponding to your database driver returns null.
Expand Down Expand Up @@ -251,15 +264,7 @@ object JdbcUtils extends Logging with SQLConfHelper {
TimeType(timePrecision)
} else getTimestampType(isTimestampNTZ)
case java.sql.Types.TIMESTAMP =>
// When nanosecond timestamps are requested (and the preview feature is enabled), a driver
// TIMESTAMP that reports a sub-microsecond fractional-second scale (7-9) is mapped to the
// nanosecond-capable type. Otherwise the historical microsecond mapping is preserved.
if (preferTimestampNanos &&
scale >= TimestampNTZNanosType.MIN_PRECISION &&
scale <= TimestampNTZNanosType.MAX_PRECISION &&
conf.timestampNanosTypesEnabled) {
getTimestampNanosType(isTimestampNTZ, scale)
} else getTimestampType(isTimestampNTZ)
resolveTimestampType(isTimestampNTZ, scale, preferTimestampNanos)
case java.sql.Types.TINYINT => IntegerType
case java.sql.Types.VARBINARY => BinaryType
case java.sql.Types.VARCHAR
Expand Down
10 changes: 7 additions & 3 deletions sql/core/src/main/scala/org/apache/spark/sql/jdbc/OracleDialect.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -187,9 +187,13 @@ private case class OracleDialect() extends JdbcDialect with SQLConfHelper with N
// Oracle DATE and TIMESTAMP are zoneless; map to NTZ and mark it wall-clock so a later flag
// flip can't desync the read. TZ/LTZ variants are handled above.
if (md != null) md.putBoolean(JdbcUtils.READ_TIMESTAMP_NTZ_WALL_CLOCK, value = true)
// TODO: map sub-microsecond TIMESTAMP(7-9) to TimestampNTZNanosType when the nanosecond
// timestamp preview is enabled, instead of truncating to microsecond TimestampNTZType.
Some(TimestampNTZType)
val metadata = if (md != null) md.build() else Metadata.empty
// Absent scale metadata: Oracle TIMESTAMP defaults to TIMESTAMP(6).
val scale = if (metadata.contains("scale")) metadata.getLong("scale").toInt else 6
val preferNanos = metadata.contains("preferTimestampNanos") &&
metadata.getBoolean("preferTimestampNanos")
Some(JdbcUtils.resolveTimestampType(
isTimestampNTZ = true, scale = scale, preferTimestampNanos = preferNanos))
case _ => None
}
}
Expand Down
37 changes: 28 additions & 9 deletions sql/core/src/test/scala/org/apache/spark/sql/jdbc/JDBCSuite.scala
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1901,16 +1901,35 @@ class JDBCSuite extends SharedSparkSession {
"{ts '2018-07-06 06:00:00.0'}")
}

test("SPARK-58876: Oracle TIMESTAMP stays microsecond TimestampNTZType under the nanos preview") {
val oracleDialect = JdbcDialects.get("jdbc:oracle")
// Even with the nanosecond timestamp preview enabled, the Oracle mapping is microsecond
// TimestampNTZType and does not engage that preview (no nanosecond type, no deferral).
withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> "true") {
val md = new MetadataBuilder()
.putBoolean("preferTimestampNanos", value = true).putLong("scale", 9)
assert(oracleDialect.getCatalystType(java.sql.Types.TIMESTAMP, "TIMESTAMP", 0, md) ===
Some(TimestampNTZType))
test("SPARK-58876: Oracle TIMESTAMP(7-9) resolves to nanosecond NTZ under the nanos preview") {
// scale/preferTimestampNanos reach the dialect only as metadata getSchema stamps, so resolve a
// mocked Oracle TIMESTAMP column via getSchema for each (scale, option, preview) combination.
def resolve(scale: Int, preferNanos: Boolean, nanosEnabled: Boolean): DataType = {
val rsmd = mock(classOf[java.sql.ResultSetMetaData])
when(rsmd.getColumnCount).thenReturn(1)
when(rsmd.getColumnLabel(anyInt())).thenReturn("T")
when(rsmd.getColumnType(anyInt())).thenReturn(java.sql.Types.TIMESTAMP)
when(rsmd.getColumnTypeName(anyInt())).thenReturn("TIMESTAMP")
when(rsmd.getPrecision(anyInt())).thenReturn(0)
when(rsmd.getScale(anyInt())).thenReturn(scale)
when(rsmd.isSigned(anyInt())).thenReturn(false)
when(rsmd.isNullable(anyInt())).thenReturn(java.sql.ResultSetMetaData.columnNullable)
val rs = mock(classOf[ResultSet])
when(rs.getMetaData).thenReturn(rsmd)
withSQLConf(SQLConf.TIMESTAMP_NANOS_TYPES_ENABLED.key -> nanosEnabled.toString) {
JdbcUtils.getSchema(mock(classOf[Connection]), rs, OracleDialect(),
preferTimestampNanos = preferNanos).fields.head.dataType
}
}

@uros-b uros-b Sep 15, 2026

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nit: resolve only returns the data type. SPARK-58876 also stamps READ_TIMESTAMP_NTZ_WALL_CLOCK so a later flag flip cannot desync the microsecond NTZ read. That marker is redundant for TimestampNTZNanosType (the nanos getter already uses getObject(LocalDateTime)), but it is still part of the Oracle contract.

Consider returning the StructField (as the nearby preferTimestampNTZ test does) and asserting the marker is still present for both the nanos and microsecond outcomes, e.g. scale=9 with both flags on and scale=6 / flags off.

// Sub-microsecond scales (7-9) widen to the nanosecond NTZ type only when both the read option
// and the preview are on; every coarser scale and either flag off stays microsecond NTZ.
(TimestampNTZNanosType.MIN_PRECISION to TimestampNTZNanosType.MAX_PRECISION).foreach { s =>
assert(resolve(s, preferNanos = true, nanosEnabled = true) === TimestampNTZNanosType(s),
s"scale=$s")
}
assert(resolve(6, preferNanos = true, nanosEnabled = true) === TimestampNTZType)
assert(resolve(9, preferNanos = false, nanosEnabled = true) === TimestampNTZType)
assert(resolve(9, preferNanos = true, nanosEnabled = false) === TimestampNTZType)
}

test("SPARK-42469: OracleDialect Limit query test") {
Expand Down