[feature](datetimev2) Support nanoseconds part1: Add nanosecond storage and basic SQL support - #66333
Draft
jacktengg wants to merge 6 commits into
Draft
Conversation
…ge and basic SQL support Issue Number: None Related PR: None Problem Summary: This is part 1 of the feature of supporting nanoseconds of datetimev2 type. DATETIMEV2 was limited to six fractional digits and used the packed civil-time representation. Add the signed Int64 epoch-nanosecond representation for scales 7 through 9, integrate it with FE literals and type metadata, and support basic table models, partition and bucket keys, insert and stream-load input, string conversion, MySQL result serialization, predicates, hashing, grouping, joins, and ordering. Support DATETIMEV2(7), DATETIMEV2(8), and DATETIMEV2(9) in core storage and basic SQL execution paths. Complete features will be implemented in following PRs. - Behavior changed: Yes, DATETIMEV2 accepts scales 7 through 9 in core SQL and storage paths - Does this need documentation: Yes, DATETIMEV2 precision and range documentation should be updated
Contributor
|
Thank you for your contribution to Apache Doris. Please clearly describe your PR:
|
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28911 ms |
Contributor
TPC-DS: Total hot run time: 170643 ms |
Contributor
ClickBench: Total hot run time: 23.98 s |
Contributor
FE Regression Coverage ReportIncrement line coverage |
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: STR_TO_DATE returns DATETIMEV2(9) for formats containing fractional seconds after nanosecond precision was enabled. The BE implementation only dispatched DATEV2 and microsecond DATETIMEV2 result columns, so a nanosecond result hit a DCHECK during scanner projection. Add a DATETIMEV2_NANO result path and carry fractional digits seven through nine through format parsing into the epoch-nanosecond value.
### Release note
Fix STR_TO_DATE with nanosecond DATETIMEV2 results.
### Check List (For Author)
- Test:
- BE ASAN unit test: VTimestampFunctionsTest.str_to_date_nanosecond_test
- BE ASAN unit tests: VDateTimeValueTest.date_v2_from_date_format_str_test and VDateTimeValueTest.datetime_v2_from_date_format_str_test
- Behavior changed: Yes. STR_TO_DATE now returns scale-9 values without aborting and preserves nanosecond digits.
- Does this need documentation: No
This reverts commit 451d79b.
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: Raising the DATETIMEV2 maximum scale from 6 to 9 changed DateTimeV2Type.MAX from microsecond to nanosecond precision. STR_TO_DATE used MAX for the %f format and therefore planned a DATETIMEV2(9) result, while its BE implementation and documented semantics remain microsecond-based, causing a TYPE_DATETIMEV2 DCHECK. The same symbolic MAX misuse affected twelve microsecond-granularity date arithmetic functions and could unnecessarily promote low-scale values to nanosecond storage. Introduce explicit microsecond precision constants, make STR_TO_DATE return DATETIMEV2(6), and infer arithmetic results with a minimum scale of 6 while preserving scale 7-9 inputs and TIMESTAMPTZ types.
### Release note
Fix STR_TO_DATE %f and microsecond date arithmetic return-type inference after DATETIMEV2 nanosecond support.
### Check List (For Author)
- Test:
- FE unit tests: StrToDateTest and DateTimeMicrosecondFunctionTest
- Behavior changed: Yes. %f returns DATETIMEV2(6); microsecond arithmetic uses at least scale 6 while preserving nanosecond input scale and TIMESTAMPTZ.
- Does this need documentation: No
### What problem does this PR solve?
Issue Number: None
Related PR: None
Problem Summary: DATETIMEV2 nanosecond support raises its maximum scale from 6 to 9. TypeCoercionUtils correctly uses DateTimeV2Type.MAX when combining date-like values with types whose fractional precision is unknown or exceeds the old microsecond limit, but TypeCoercionMatrixTest still expected DATETIMEV2(6). Update only the maximum-precision matrix cases to use DateTimeV2Type.MAX while preserving explicit scale-6 expectations where coercion retains the input precision.
### Release note
None
### Check List (For Author)
- Test:
- FE unit test: TypeCoercionMatrixTest
- Behavior changed: No. Test expectations now match DATETIMEV2 nanosecond type coercion.
- Does this need documentation: No
### What problem does this PR solve? Issue Number: None Related PR: apache#66333 Problem Summary: DATETIMEV2 nanosecond support made scales 7-9 and the new primitive/storage enum values valid, but stale unit tests still classified them as unsupported. Raising the global maximum scale to 9 also moved legacy implicit coercion, MySQL binary timestamp parameters, numeric date parsing, and DATETIMEV2(0-6) ORC reads onto nanosecond paths. This caused far-future literals and ORC timestamps to exceed the Int64 epoch-nanosecond range and broke prepared statements, point queries, DATE joins, and coercion tests. Keep compatibility paths at scale 6 unless an operand is explicitly DATETIMEV2(7-9), retain wide-range ORC decoding for DATETIMEV2(0-6), and use the epoch-nanosecond decoder only for DATETIMEV2(7-9). Update enum and scale tests, and restore ForwardToMasterTest assertions to the forwarded REST response shape. ### Release note Preserve the existing DATETIMEV2(0-6) range and implicit-coercion behavior while supporting explicit DATETIMEV2(7-9). ### Check List (For Author) - Test: - Unit Test: BE StorageFieldTypeTest, DataTypeDateTimeV2Test, and OrcReaderFillDataTest targeted tests - Unit Test: 105 targeted FE tests, including FE checkstyle validation - Behavior changed: Yes. Implicit datetime coercion remains at microsecond precision unless an input explicitly uses nanosecond precision. - Does this need documentation: No
Contributor
Author
|
run buildall |
Contributor
TPC-H: Total hot run time: 28664 ms |
Contributor
TPC-DS: Total hot run time: 169560 ms |
Contributor
ClickBench: Total hot run time: 23.61 s |
Contributor
BE UT Coverage ReportIncrement line coverage Increment coverage report
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Issue Number: None
Related PR: None
Problem Summary:
This is part 1 of the feature of supporting nanoseconds of datetimev2 type.
DATETIMEV2 was limited to six fractional digits and used the packed civil-time representation. Add the signed Int64 epoch-nanosecond representation for scales 7 through 9, integrate it with FE literals and type metadata, and support basic table models, partition and bucket keys, insert and stream-load input, string conversion, MySQL result serialization, predicates, hashing, grouping, joins, and ordering.
Support DATETIMEV2(7), DATETIMEV2(8), and DATETIMEV2(9) in core storage and basic SQL execution paths.
Complete features will be implemented in following PRs.
What problem does this PR solve?
Issue Number: close #xxx
Related PR: #xxx
Problem Summary:
Release note
None
Check List (For Author)
Test
Behavior changed:
Does this need documentation?
Check List (For Reviewer who merge this PR)