diff --git a/README.md b/README.md index 0f42d8c..ce7a11d 100644 --- a/README.md +++ b/README.md @@ -1,229 +1 @@ -# ClickHouse SQL Parser -![GitHub CI](https://github.com/AfterShip/clickhouse-sql-parser/actions/workflows/ci.yaml/badge.svg) [![LICENSE](https://img.shields.io/github/license/AfterShip/clickhouse-sql-parser.svg)](https://github.com/AfterShip/clickhouse-sql-parser/blob/master/LICENSE) [![GoDoc](https://img.shields.io/badge/Godoc-reference-blue.svg)](https://godoc.org/github.com/AfterShip/clickhouse-sql-parser) [![Coverage Status](https://coveralls.io/repos/github/AfterShip/clickhouse-sql-parser/badge.svg?branch=master)](https://coveralls.io/github/AfterShip/clickhouse-sql-parser?branch=master) Featured｜HelloGitHub - -The goal of this project is to build a ClickHouse SQL parser in Go with the following key features: - -- Parse ClickHouse SQL into AST -- Beautify ClickHouse SQL format - -This project is inspired by [memefish](https://github.com/cloudspannerecosystem/memefish) which is a SQL parser for Spanner in Go. -## How to use - -You can use it as your Go library or CLI tool, see the following examples: - -- Use clickhouse-sql-parser as a Go library - -```Go -package main - -import ( - clickhouse "github.com/AfterShip/clickhouse-sql-parser/parser" -) - -query := "SELECT * FROM clickhouse" -parser := clickhouse.NewParser(query) -// Parse query into AST -statements, err := parser.ParseStmts() -if err != nil { - return nil, err -} -``` - -- Install clickhouse-sql-parser as a CLI tool - - -On Linux: - -```bash -$ go install github.com/AfterShip/clickhouse-sql-parser@latest -``` - -On macOS: - -```bash -$ brew install clickhouse-sql-parser -``` - -Parse ClickHouse SQL into AST or format ClickHouse SQL: - -```bash -## Parse query into AST -$ clickhouse-sql-parser "SELECT * FROM clickhouse WHERE a=100" - -## Format query (compact, single-line output) -$ clickhouse-sql-parser -format "SELECT * FROM clickhouse WHERE a=100" - -## Beautify query (formatted with proper indentation and line breaks) -$ clickhouse-sql-parser -beautify "SELECT * FROM clickhouse WHERE a=100" - -## Parse query from file -$ clickhouse-sql-parser -f ./test.sql -``` - -### Beautify SQL Example - -The `-beautify` flag formats SQL with proper indentation and line breaks, making complex queries more readable: - -```SQL -# Input (compact, hard to read) -$ clickhouse-sql-parser -beautify "SELECT user_id, COUNT(*) AS total, AVG(amount) AS avg_amount FROM orders WHERE status='completed' AND created_at>'2024-01-01' GROUP BY user_id HAVING COUNT(*)>5 ORDER BY total DESC LIMIT 10" - -# Output (beautified, easy to read) -SELECT - user_id, - COUNT(*) AS total, - AVG(amount) AS avg_amount -FROM orders -WHERE - status = 'completed' -AND - created_at > '2024-01-01' -GROUP BY - user_id -HAVING COUNT(*) > 5 -ORDER BY - total DESC -LIMIT 10 -``` - -- Parsed tree(AST) back into a SQL statement - -```Go -parser := clickhouse.NewParser("SELECT * FROM clickhouse") -// Parse query into AST -statements, err := parser.ParseStmts() -if err != nil { - return nil, err -} - -// Format AST back into a SQL string -for _, stmt := range statements { - fmt.Println(clickhouse.Format(stmt)) -} -``` - -## AST Traversal - -### Walk Pattern (Recommended) - -The Walk pattern provides a simple and efficient way to traverse AST nodes. Use the `Walk` function to visit all nodes in the AST: - -```Go -import ( - clickhouse "github.com/AfterShip/clickhouse-sql-parser/parser" -) - -parser := clickhouse.NewParser("SELECT * FROM table WHERE id = 1") -statements, err := parser.ParseStmts() -if err != nil { - return err -} - -// Walk through all nodes in the AST -clickhouse.Walk(statements[0], func(node clickhouse.Expr) bool { - fmt.Printf("Node type: %T\n", node) - return true // return false to stop traversal for this subtree -}) -``` - -#### Walk Pattern Functions - -- **`Walk(node Expr, fn WalkFunc)`** - Traverses all nodes in depth-first order -- **`WalkWithBreak(node Expr, fn WalkFunc)`** - Allows early termination of traversal -- **`Find(root Expr, predicate func(Expr) bool)`** - Finds the first node matching a condition -- **`FindAll(root Expr, predicate func(Expr) bool)`** - Finds all nodes matching a condition -- **`Transform(root Expr, transformer func(Expr) Expr)`** - Applies transformations to nodes - -#### Examples - -Find all table identifiers: -```Go -tables := clickhouse.FindAll(stmt, func(node clickhouse.Expr) bool { - _, ok := node.(*clickhouse.TableIdentifier) - return ok -}) -``` - -Find the first WHERE clause: -```Go -whereClause, found := clickhouse.Find(stmt, func(node clickhouse.Expr) bool { - _, ok := node.(*clickhouse.WhereClause) - return ok -}) -``` - -## Update test assets - -For the files inside `output` and `format` dir are generated by the test cases, - -if you want to update them, you can run the following command: - -```bash -$ make update_test -``` - -## Benchmarks - -```sh -go test -bench=. -benchmem ./parser -``` - -Results - -``` -$ go test -bench=. -benchmem ./parser -goos: linux -goarch: amd64 -pkg: github.com/AfterShip/clickhouse-sql-parser/parser -cpu: Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz -BenchmarkParseSQLFiles/access_tuple_with_dot.sql-28 23294 58467 ns/op 13448 B/op 293 allocs/op -BenchmarkParseSQLFiles/query_with_expr_compare.sql-28 43560 25704 ns/op 6240 B/op 132 allocs/op -BenchmarkParseSQLFiles/select_cast.sql-28 75055 16518 ns/op 4648 B/op 92 allocs/op -BenchmarkParseSQLFiles/select_column_alias_string.sql-28 499798 2785 ns/op 704 B/op 13 allocs/op -BenchmarkParseSQLFiles/select_expr.sql-28 488187 2448 ns/op 696 B/op 12 allocs/op -BenchmarkParseSQLFiles/select_item_with_modifiers.sql-28 54124 23305 ns/op 6232 B/op 136 allocs/op -BenchmarkParseSQLFiles/select_order_by_timestamp.sql-28 232302 5809 ns/op 1368 B/op 28 allocs/op -BenchmarkParseSQLFiles/select_simple.sql-28 30602 50022 ns/op 9920 B/op 216 allocs/op -BenchmarkParseSQLFiles/select_simple_field_alias.sql-28 178126 6316 ns/op 1712 B/op 39 allocs/op -BenchmarkParseSQLFiles/select_simple_with_bracket.sql-28 71902 16172 ns/op 3824 B/op 85 allocs/op -BenchmarkParseSQLFiles/select_simple_with_cte_with_column_aliases.sql-28 67050 19916 ns/op 4648 B/op 101 allocs/op -BenchmarkParseSQLFiles/select_simple_with_group_by_with_cube_totals.sql-28 107047 10348 ns/op 2768 B/op 58 allocs/op -BenchmarkParseSQLFiles/select_simple_with_is_not_null.sql-28 55285 21957 ns/op 5224 B/op 111 allocs/op -BenchmarkParseSQLFiles/select_simple_with_is_null.sql-28 66648 22412 ns/op 4728 B/op 102 allocs/op -BenchmarkParseSQLFiles/select_simple_with_top_clause.sql-28 269910 4166 ns/op 1088 B/op 22 allocs/op -BenchmarkParseSQLFiles/select_simple_with_with_clause.sql-28 58494 18417 ns/op 5144 B/op 109 allocs/op -BenchmarkParseSQLFiles/select_table_alias_without_keyword.sql-28 121261 10011 ns/op 2896 B/op 65 allocs/op -BenchmarkParseSQLFiles/select_table_function_with_query.sql-28 98017 14929 ns/op 4168 B/op 81 allocs/op -BenchmarkParseSQLFiles/select_when_condition.sql-28 220394 5457 ns/op 1304 B/op 28 allocs/op -BenchmarkParseSQLFiles/select_with_distinct.sql-28 172948 6531 ns/op 1560 B/op 33 allocs/op -BenchmarkParseSQLFiles/select_with_join_only.sql-28 286346 5594 ns/op 1520 B/op 32 allocs/op -BenchmarkParseSQLFiles/select_with_left_join.sql-28 88200 13627 ns/op 3880 B/op 75 allocs/op -BenchmarkParseSQLFiles/select_with_literal_table_name.sql-28 241094 5099 ns/op 1304 B/op 27 allocs/op -BenchmarkParseSQLFiles/select_with_multi_join.sql-28 44700 31964 ns/op 8240 B/op 188 allocs/op -BenchmarkParseSQLFiles/select_with_multi_line_comment.sql-28 363499 4460 ns/op 824 B/op 18 allocs/op -BenchmarkParseSQLFiles/select_with_multi_union.sql-28 146233 7827 ns/op 2176 B/op 36 allocs/op -BenchmarkParseSQLFiles/select_with_number_field.sql-28 129945 8746 ns/op 2352 B/op 51 allocs/op -BenchmarkParseSQLFiles/select_with_query_parameter.sql-28 33850 36346 ns/op 9936 B/op 209 allocs/op -BenchmarkParseSQLFiles/select_with_string_expr.sql-28 142882 7530 ns/op 1880 B/op 34 allocs/op -BenchmarkParseSQLFiles/select_with_union_distinct.sql-28 147031 9601 ns/op 2352 B/op 47 allocs/op -BenchmarkParseSQLFiles/select_with_variable.sql-28 179158 6669 ns/op 1880 B/op 36 allocs/op -BenchmarkParseSQLFiles/select_with_window_function.sql-28 54925 31320 ns/op 6720 B/op 136 allocs/op -BenchmarkParseSQLFiles/select_with_placeholder.sql-28 196771 5145 ns/op 1272 B/op 26 allocs/op -BenchmarkParseSQLFiles/set_simple.sql-28 172419 7062 ns/op 2480 B/op 49 allocs/op -BenchmarkParseComplexQueries/testdata/query/select_with_multi_join.sql-28 39056 36897 ns/op 8240 B/op 188 allocs/op -BenchmarkParseComplexQueries/testdata/query/select_with_window_function.sql-28 47629 29916 ns/op 6720 B/op 136 allocs/op -BenchmarkParseComplexQueries/testdata/query/select_simple_with_with_clause.sql-28 69210 19731 ns/op 5144 B/op 109 allocs/op -BenchmarkParseComplexQueries/testdata/query/select_with_left_join.sql-28 74576 15338 ns/op 3880 B/op 75 allocs/op -BenchmarkParseComplexQueries/testdata/benchdata/posthog_huge_0.sql-28 235 6231253 ns/op 1236189 B/op 26696 allocs/op -BenchmarkParseComplexQueries/testdata/benchdata/posthog_huge_1.sql-28 279 4438280 ns/op 1043374 B/op 22717 allocs/op -PASS -ok github.com/AfterShip/clickhouse-sql-parser/parser 66.547s -``` - -## For AI coding agents - -If you are an AI coding agent (or are using one) to contribute to this project, read [AGENTS.md](./AGENTS.md) first and use it as your working context. It covers the project structure, coding style, testing workflow, and contribution practices — including validating SQL against `clickhouse-local` before adding new syntax or fixing parsing issues. - -## Contact us - -Feel free to open an issue or discussion if you have any issues or questions. +# TEST PR by Calvin