Add ClickHouse limit_by and deduplicate dataset methods - #59
Merged
Conversation
tycooon-review-bot
left a comment
There was a problem hiding this comment.
Review by Claude Code (Opus 5).
tycooon-review-bot
left a comment
There was a problem hiding this comment.
Review by Claude Code (Opus 5).
tycooon-review-bot
approved these changes
Aug 14, 2026
tycooon-review-bot
left a comment
There was a problem hiding this comment.
Approved by Claude Code (Opus 5) — re-review of 188b0e0 is clean, and the last open thread is fixed and resolved.
KirIgor
approved these changes
Aug 14, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a
ReplacingMergeTreetable, thefinalsetting is usually a session-wide default. That turns every query into a full-row merge across all parts — even a point lookup on a single id. In the consumer app that made an admin page's single-order query take 23.8 s to return 3 rows; the same query without table-wide FINAL takes 385 ms.FINAL can't simply be switched off, because it does real work: it collapses row versions and drops rows flagged
is_deleted. So it has to be replaced, per query, with an explicit dedup over an already-small row set.What
Dataset#limit_by(*exprs, rows: 1)— ClickHouse'sLIMIT n BY, emitted ahead of the regularLIMIT/OFFSET:Dataset#deduplicate— a boundary that collapses row versions by hand. Everything chained before it goes inside the dedup subquery, everything after applies outside:Sorting key, version column and
is_deletedare read fromsystem.tablesand cached per process.Distributedtables carry no sorting key of their own, so they are resolved through to the local table they wrap.Which side of the boundary a filter lands on matters, and the README says so: only immutable selectors belong before it, because filtering a mutable column first can match a superseded version and resurrect a row FINAL would have dropped.
is_deletedis applied after the boundary for the same reason.Automatic
final: 0— a deduplicated dataset would otherwise still run FINAL inside its own subquery, which is both slow and redundant. Deduplicated datasets are now sent withfinal: 0; an explicitfinal:from the caller still wins.Base#countnow forwards settings (it previously swallowed them), so a count on a deduplicated dataset behaves like the query it counts.ClickHouseStringEscapingis renamed toClickHouseDatasetMethods, since it is no longer only escaping.Testing
New specs run against a real ClickHouse:
limit_bySQL emission and row collapsing,engine_fullparsing across all six Replacing forms plus a non-Replacing engine raising, Distributed resolution, dedup SQL shape and filter placement,final: 0injection and override, andcountpassthrough. One asserts the deduplicated result is identical to what FINAL returns.Two test tables were added to the spec fixtures (a
ReplacingMergeTreewith version +is_deleted, one withoutis_deleted) plus aDistributedwrapper over the first.Full suite: 214 examples, 0 failures. RuboCop clean.