[SPARK-58876][SQL][FOLLOWUP] Map sub-microsecond Oracle TIMESTAMP to nanos NTZ - #58809
aleksandar-trajkovic-db wants to merge 1 commit into
Conversation
…nanos NTZ Oracle DATE/TIMESTAMP mapping short-circuited the shared TIMESTAMP resolver, regressing TIMESTAMP(7-9) to microsecond TimestampNTZType under the nanos preview. Extract the micro-vs-nanos decision into JdbcUtils.resolveTimestampType and route Oracle's NTZ mapping through it (forcing NTZ). Co-authored-by: Isaac <no-reply@databricks.com>
This doesn't look like a valid test invocation. Perhaps something like |
| JdbcUtils.getSchema(mock(classOf[Connection]), rs, OracleDialect(), | ||
| preferTimestampNanos = preferNanos).fields.head.dataType | ||
| } | ||
| } |
There was a problem hiding this comment.
Nit: resolve only returns the data type. SPARK-58876 also stamps READ_TIMESTAMP_NTZ_WALL_CLOCK so a later flag flip cannot desync the microsecond NTZ read. That marker is redundant for TimestampNTZNanosType (the nanos getter already uses getObject(LocalDateTime)), but it is still part of the Oracle contract.
Consider returning the StructField (as the nearby preferTimestampNTZ test does) and asserting the marker is still present for both the nanos and microsecond outcomes, e.g. scale=9 with both flags on and scale=6 / flags off.
2 participants
What changes were proposed in this pull request?
Follow-up to SPARK-58876. That change mapped Oracle
DATE/TIMESTAMPtoTimestampNTZTypedirectly inOracleDialect.getCatalystType, which short-circuitedthe shared
Types.TIMESTAMPresolution inJdbcUtilsthat widens a sub-microsecondcolumn to a nanosecond-capable type.
This PR extracts that micro-vs-nanos decision into
JdbcUtils.resolveTimestampType(isTimestampNTZ, scale, preferTimestampNanos)(thegeneric
Types.TIMESTAMPbranch now calls it) and routes Oracle's mapping through itwith
isTimestampNTZ = true. OracleDATE/TIMESTAMPtherefore stays NTZ, but aTIMESTAMP(7-9)now maps toTimestampNTZNanosType(scale)when the nanos preview isengaged.
Why are the changes needed?
Before SPARK-58876, Oracle
DATE/TIMESTAMPreturnedNonefrom the dialect and fellthrough to the shared resolver, so
TIMESTAMP(7-9)was mapped to a nanosecond type whenthe
preferTimestampNanosread option andspark.sql.timestampNanosTypes.enabledwereboth on. SPARK-58876 accidentally regressed this, truncating such columns to microsecond
TimestampNTZType. This restores the nanosecond precision while keeping the newzoneless-NTZ behavior.
Does this PR introduce any user-facing change?
Yes — relative to the unreleased SPARK-58876 change, not to any released version. With
preferTimestampNanos=trueandspark.sql.timestampNanosTypes.enabled=true, an OracleTIMESTAMP(7-9)column now reads asTimestampNTZNanos(scale)instead of microsecondTimestampNTZType.DATE,TIMESTAMP(0-6), and the default (preview off) areunchanged. The value is read time-zone-independently (wall-clock).
How was this patch tested?
Added a
getSchema-level unit test inJDBCSuiteasserting that an OracleTIMESTAMP(7-9)resolves toTimestampNTZNanosType(scale)only when both thepreferTimestampNanosoption and the preview flag are on, and stays microsecondTimestampNTZTypefor coarser scales or with either flag off. This replaces the priortest that pinned tehavior.
Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Anthropic Claude Opus 4.8)
This is my original work and I license it to the project under the project's open source license.