Skip to content

Hybrid table engine fails to merge data in some cases between MergeTree and Iceberg sections #2102

Description

@hodgesrm

Describe the bug
Certain queries on hybrid tables fail to merge data between MergeTree and Iceberg segments. Here's an example from the antalya demo server. It returns two rows for each group by key, not just one.

SELECT Carrier, count() AS flights, min(FlightDate), max(FlightDate)
FROM ontime.fact_ontime_hybrid
WHERE Carrier IN ('WN') GROUP BY Carrier ORDER BY Carrier
SETTINGS object_storage_cluster = 'swarm'

   ┌─Carrier─┬──flights─┬─min(FlightDate)─┬─max(FlightDate)─┐
1. │ WN      │ 29441617 │      1987-10-01 │      2019-12-31 │
2. │ WN      │  7582834 │      2020-01-01 │      2025-12-31 │
   └─────────┴──────────┴─────────────────┴─────────────────┘

To Reproduce
Steps to reproduce the behavior:

  1. Login to Antalya demo server: clickhouse-client -h antalya.demo.altinity.cloud --port 9440 -s --user=demo --password=demo
  2. Issue the query show above. It should return two rows.
  3. Remove SETTINGS object_storage_cluster = 'swarm' and run the query again.
  4. You should just see one row. (Though this is not consistent.)

Expected behavior
In both cases each group by key should return a single row.

Key information
Provide relevant runtime details.

  • Project Antalya Build Version: 26.3.10.20001.altinityantalya
  • Cloud provider: AWS
  • Kubernetes provider: EKS
  • Object storage: AWS S3
  • Iceberg catalog: Ice

Additional context
To see the hybrid table definition run show create table ontime.fact_ontime_hybrid. It should show the following:

CREATE TABLE ontime.fact_ontime_hybrid
(
    `FlightDate` Date COMMENT 'Flight Date (yyyymmdd)' CODEC(DoubleDelta, ZSTD(1)),
...
    `Div5TailNum` LowCardinality(String) COMMENT 'Aircraft Tail Number for Diverted Airport Code5'
)
ENGINE = Hybrid(cluster('{cluster}', 'ontime.fact_ontime'), toYear(FlightDate) >= 2020, ice.`ontime.fact_ontime`, toYear(FlightDate) < 2020)
COMMENT 'Flights on-time data, 231M rows from US BTS'

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions