Skip to content

Add ClickHouse limit_by and deduplicate dataset methods - #59

Merged
tycooon merged 7 commits into
mainfrom
usafe-9620-limit-by
Aug 14, 2026
Merged

Add ClickHouse limit_by and deduplicate dataset methods#59
tycooon merged 7 commits into
mainfrom
usafe-9620-limit-by

Conversation

@tycooon

@tycooon tycooon commented Aug 13, 2026

Copy link
Copy Markdown
Member

Why

On a ReplacingMergeTree table, the final setting is usually a session-wide default. That turns every query into a full-row merge across all parts — even a point lookup on a single id. In the consumer app that made an admin page's single-order query take 23.8 s to return 3 rows; the same query without table-wide FINAL takes 385 ms.

FINAL can't simply be switched off, because it does real work: it collapses row versions and drops rows flagged is_deleted. So it has to be replaced, per query, with an explicit dedup over an already-small row set.

What

Dataset#limit_by(*exprs, rows: 1) — ClickHouse's LIMIT n BY, emitted ahead of the regular LIMIT/OFFSET:

CH.from(:events).order(Sequel.desc(:version)).limit_by(:user_id, rows: 3)
#=> SELECT * FROM "events" ORDER BY "version" DESC LIMIT 3 BY "user_id"

Dataset#deduplicate — a boundary that collapses row versions by hand. Everything chained before it goes inside the dedup subquery, everything after applies outside:

CH.from(:external_operations_distributed)
  .where(order_id: 42)   # inside
  .deduplicate           # boundary
  .order(:created_at)    # outside

Sorting key, version column and is_deleted are read from system.tables and cached per process. Distributed tables carry no sorting key of their own, so they are resolved through to the local table they wrap.

Which side of the boundary a filter lands on matters, and the README says so: only immutable selectors belong before it, because filtering a mutable column first can match a superseded version and resurrect a row FINAL would have dropped. is_deleted is applied after the boundary for the same reason.

Automatic final: 0 — a deduplicated dataset would otherwise still run FINAL inside its own subquery, which is both slow and redundant. Deduplicated datasets are now sent with final: 0; an explicit final: from the caller still wins.

Base#count now forwards settings (it previously swallowed them), so a count on a deduplicated dataset behaves like the query it counts.

ClickHouseStringEscaping is renamed to ClickHouseDatasetMethods, since it is no longer only escaping.

Testing

New specs run against a real ClickHouse: limit_by SQL emission and row collapsing, engine_full parsing across all six Replacing forms plus a non-Replacing engine raising, Distributed resolution, dedup SQL shape and filter placement, final: 0 injection and override, and count passthrough. One asserts the deduplicated result is identical to what FINAL returns.

Two test tables were added to the spec fixtures (a ReplacingMergeTree with version + is_deleted, one without is_deleted) plus a Distributed wrapper over the first.

Full suite: 214 examples, 0 failures. RuboCop clean.

@tycooon-review-bot tycooon-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by Claude Code (Opus 5).

Comment thread lib/umbrellio_utils/click_house/backends/base.rb Outdated
Comment thread lib/umbrellio_utils/click_house/backends/base.rb
Comment thread lib/umbrellio_utils/click_house/backends/base.rb
Comment thread lib/umbrellio_utils/click_house/backends/base.rb Outdated
Comment thread lib/umbrellio_utils/click_house/table_metadata.rb
Comment thread lib/umbrellio_utils/click_house/backends/native.rb Outdated

@tycooon-review-bot tycooon-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review by Claude Code (Opus 5).

Comment thread lib/umbrellio_utils/click_house/table_metadata.rb Outdated

@tycooon-review-bot tycooon-review-bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved by Claude Code (Opus 5) — re-review of 188b0e0 is clean, and the last open thread is fixed and resolved.

@tycooon
tycooon merged commit 8f47c1e into main Aug 14, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants