Describe the bug
Certain queries on hybrid tables fail to merge data between MergeTree and Iceberg segments. Here's an example from the antalya demo server. It returns two rows for each group by key, not just one.
SELECT Carrier, count() AS flights, min(FlightDate), max(FlightDate)
FROM ontime.fact_ontime_hybrid
WHERE Carrier IN ('WN') GROUP BY Carrier ORDER BY Carrier
SETTINGS object_storage_cluster = 'swarm'
┌─Carrier─┬──flights─┬─min(FlightDate)─┬─max(FlightDate)─┐
1. │ WN │ 29441617 │ 1987-10-01 │ 2019-12-31 │
2. │ WN │ 7582834 │ 2020-01-01 │ 2025-12-31 │
└─────────┴──────────┴─────────────────┴─────────────────┘
To Reproduce
Steps to reproduce the behavior:
- Login to Antalya demo server:
clickhouse-client -h antalya.demo.altinity.cloud --port 9440 -s --user=demo --password=demo
- Issue the query show above. It should return two rows.
- Remove
SETTINGS object_storage_cluster = 'swarm' and run the query again.
- You should just see one row. (Though this is not consistent.)
Expected behavior
In both cases each group by key should return a single row.
Key information
Provide relevant runtime details.
- Project Antalya Build Version: 26.3.10.20001.altinityantalya
- Cloud provider: AWS
- Kubernetes provider: EKS
- Object storage: AWS S3
- Iceberg catalog: Ice
Additional context
To see the hybrid table definition run show create table ontime.fact_ontime_hybrid. It should show the following:
CREATE TABLE ontime.fact_ontime_hybrid
(
`FlightDate` Date COMMENT 'Flight Date (yyyymmdd)' CODEC(DoubleDelta, ZSTD(1)),
...
`Div5TailNum` LowCardinality(String) COMMENT 'Aircraft Tail Number for Diverted Airport Code5'
)
ENGINE = Hybrid(cluster('{cluster}', 'ontime.fact_ontime'), toYear(FlightDate) >= 2020, ice.`ontime.fact_ontime`, toYear(FlightDate) < 2020)
COMMENT 'Flights on-time data, 231M rows from US BTS'
Describe the bug
Certain queries on hybrid tables fail to merge data between MergeTree and Iceberg segments. Here's an example from the antalya demo server. It returns two rows for each group by key, not just one.
To Reproduce
Steps to reproduce the behavior:
clickhouse-client -h antalya.demo.altinity.cloud --port 9440 -s --user=demo --password=demoSETTINGS object_storage_cluster = 'swarm'and run the query again.Expected behavior
In both cases each group by key should return a single row.
Key information
Provide relevant runtime details.
Additional context
To see the hybrid table definition run
show create table ontime.fact_ontime_hybrid. It should show the following: