diff --git a/docs/design/google-sheets/README.md b/docs/design/google-sheets/README.md new file mode 100644 index 0000000..ddaea58 --- /dev/null +++ b/docs/design/google-sheets/README.md @@ -0,0 +1,85 @@ +# Google Sheets integration design catalog + +Status: **Proposed** + +This catalog is the source of truth for the Google Sheets integration in Altinity SQL Browser. GitHub issues define executable implementation slices and link back to these documents. + +## Product boundary + +The integration connects a saved ClickHouse query to a Google spreadsheet without adding an application backend. + +```text +Google Sheet hyperlink + ↓ +ClickHouse-hosted SQL Browser SPA + ↓ +ClickHouse query execution + ↓ +Google Sheets API +``` + +SQL Browser owns query execution, typed parameters, schema validation, refresh orchestration, managed worksheet contents, and synchronization checkpoints. Google Sheets owns formulas, native pivots, charts, comments, sharing, and presentation sheets. + +## Supported project scope + +- One-time snapshot export. +- Create or attach a linked spreadsheet. +- Manual deep-link refresh. +- Failure-safe full refresh through staging. +- Large-result streaming and quota management. +- Append-only refresh for eligible ClickHouse 26.8+ MergeTree-family sources using `_block_number` and `_block_offset`. +- Refresh history, repair, and unlink workflows. + +## Explicit non-goals + +- Scheduled or background refresh. +- Refresh while SQL Browser is closed. +- `SELECT ... STREAM` integration. +- Updates or deletes in append mode. +- Google Docs integration. +- Apps Script or a dedicated application backend. +- Service-account impersonation. +- Strict distributed locking between spreadsheet collaborators. + +## Documents + +- [Product scope](product-scope.md) +- [Architecture and data flow](architecture.md) +- [Resource model](resource-model.md) +- [Authentication and authorization](authentication.md) +- [Spreadsheet layout](spreadsheet-layout.md) +- [Snapshot export](snapshot-export.md) +- [Full refresh](full-refresh.md) +- [Append refresh](append-refresh.md) +- [Large results and quotas](large-results.md) +- [Failure recovery](failure-recovery.md) +- [Sharing and security](sharing-security.md) +- [Deployment configuration](configuration.md) +- [Testing strategy](testing.md) +- [Architecture decisions](decisions.md) + +## Interactive companion diagrams + +- [Architecture](diagrams/architecture.html) +- [Refresh flow](diagrams/refresh-flow.html) +- [Append checkpoint](diagrams/append-checkpoint.html) + +The Markdown documents and Mermaid diagrams are canonical. HTML diagrams are explanatory companions and must be updated in the same pull request when their corresponding design changes. + +## Implementation tracking + +The umbrella GitHub issue owns project sequencing. Child issues should contain only their implementation slice, acceptance criteria, dependencies, tests, and links to the relevant catalog sections. They must not duplicate the complete architecture. + +## Change-management rule + +Design precedence is: + +```text +repository design catalog + ↓ +implementation issue + ↓ +pull request +``` + +A pull request that changes an agreed behavior must update the corresponding design document and, where appropriate, the decision log. diff --git a/docs/design/google-sheets/append-refresh.md b/docs/design/google-sheets/append-refresh.md new file mode 100644 index 0000000..4db4601 --- /dev/null +++ b/docs/design/google-sheets/append-refresh.md @@ -0,0 +1,78 @@ +# Append refresh + +Append refresh is an opt-in mode for immutable inserted rows. Updates and deletes are out of scope; run a full refresh when they must be reflected. + +## Availability + +Append mode is shown only when all checks pass: + +- configured feature enabled; +- ClickHouse version is at least the configured minimum, initially `26.8`; +- direct `MergeTree` or `ReplicatedMergeTree` source; +- stable table UUID; +- persistent `_block_number` and `_block_offset` available; +- finite ordinary SELECT; +- validated append-compatible query shape; +- destination checkpoint valid. + +Version gating is necessary but not sufficient. Runtime capability probes are authoritative. + +`SELECT ... STREAM` is explicitly out of scope. + +## Eligible query shape + +Allow deterministic projection, aliases, typed parameters, optional filters, and WHERE predicates over one direct source table. + +Reject joins, views, `FINAL`, aggregation, `GROUP BY`, `DISTINCT`, window functions, `UNION`, table functions, business `LIMIT`, changing top-N semantics, and arbitrary query rewriting. + +The binding pins table UUID, query snapshot, query hash, schema hash, and filter parameter values. Changing the source subset requires a full refresh. + +## Cursor + +Commit position is per partition: + +```json +{ + "partitions": { + "202607": { + "blockNumber": "18401", + "blockOffset": "982" + } + } +} +``` + +All values are decimal strings. Rows are complete but not globally ordered across partitions. Process partitions in deterministic `partition_id` order and rows within a partition in `(_block_number, _block_offset)` order. + +## Refresh procedure + +```mermaid +flowchart TD + A[Read Google checkpoint] --> B[Revalidate server, table UUID, query, schema] + B --> C[Capture per-partition upper block boundaries] + C --> D[Fetch finite bounded page] + D --> E[Upload page to hidden delta staging] + E --> F[Atomically copy rows and advance partition checkpoint] + F --> G{Upper boundaries reached?} + G -- no --> D + G -- yes --> H[Record completion] +``` + +Each page query filters after the stored `(blockNumber, blockOffset)`, stops at the captured upper block, orders by commit position, and applies a bounded limit. Cursor columns are technical and need not appear in the visible sheet. + +## Google commit invariant + +For each page, copying rows to the exact live destination and advancing that partition checkpoint occur in the same final Google structural batch. Never advance the checkpoint independently of the corresponding rows. + +## Initial population + +- **Export existing rows, then append**: default. Use the cursor-aware bounded path for history. +- **Append future rows only**: capture current upper boundaries without exporting history. + +## Invalidation + +Require an explicit full refresh when server capability, table UUID, query hash, schema, parameters defining the source subset, managed row count, or checkpoint validity changes. Never silently fall back. + +## Concurrency + +Use an advisory lease and generation check. Re-read generation immediately before each commit and abort when another refresh advanced it. Strict distributed locking between collaborators is not promised. diff --git a/docs/design/google-sheets/architecture.md b/docs/design/google-sheets/architecture.md new file mode 100644 index 0000000..e254290 --- /dev/null +++ b/docs/design/google-sheets/architecture.md @@ -0,0 +1,74 @@ +# Architecture and data flow + +## Runtime architecture + +Altinity SQL Browser remains a static single-page application served by ClickHouse. It does not listen on a port, receive Google callbacks, or add an application server. + +```mermaid +flowchart LR + U[User in Google Sheets] -->|clicks Refresh data| B[Browser navigation] + B -->|GET /sql| C[ClickHouse HTTP server] + C -->|serves static SPA| A[SQL Browser SPA] + A -->|authenticated SQL over HTTP| C + A -->|OAuth access token and REST| G[Google Sheets API] + C -->|query result stream| A + A -->|staged values and publish requests| G +``` + +The refresh URL uses an opaque binding identifier in the fragment: + +```text +https://clickhouse.example.com/sql#sheet-refresh= +``` + +The fragment is interpreted by the SPA and is not sent to ClickHouse by the browser. It contains no SQL, parameters, credentials, or spreadsheet contents. + +## Binding resolution + +The SPA resolves the opaque identifier through a deployment-configured ClickHouse binding table using the current authenticated ClickHouse identity. The latest append-only binding version contains the spreadsheet identity, managed sheet IDs, pinned query snapshot, parameter policy, limits, and refresh mode. + +## Authentication boundaries + +```mermaid +sequenceDiagram + participant U as User + participant A as SQL Browser SPA + participant C as ClickHouse + participant G as Google Identity / Sheets + + U->>A: Open refresh deep link + A->>C: Authenticate and resolve binding + C-->>A: Authorized binding definition + U->>A: Continue with Google + A->>G: Request short-lived browser access token + G-->>A: Access token callback + A->>C: Execute pinned query + C-->>A: Stream rows + A->>G: Upload staging values + A->>G: Publish and update checkpoint +``` + +ClickHouse authentication authorizes query execution and binding access. Google authentication authorizes spreadsheet modification. Neither identity implies the other. + +## Data path + +```text +ClickHouse response stream + ↓ +SQL Browser JavaScript memory + ↓ +Google Sheets REST API +``` + +Data does not pass through an Altinity application backend. + +## Source-of-truth hierarchy + +- ClickHouse binding table: durable binding definition and versions. +- Google hidden state sheet: destination commit state and append checkpoint. +- Managed data sheet: last committed exported rows. +- Local workspace: navigation convenience and editing source, not the durable refresh authority. + +## Refresh entry and completion + +When SQL Browser is closed, the spreadsheet remains usable with its last committed data. Nothing refreshes in the background. The control worksheet hyperlink launches a new SPA instance, which resolves, authenticates, executes, uploads, publishes, and terminates the refresh workflow. The completion screen links back to the spreadsheet and source query. diff --git a/docs/design/google-sheets/authentication.md b/docs/design/google-sheets/authentication.md new file mode 100644 index 0000000..5a34ddd --- /dev/null +++ b/docs/design/google-sheets/authentication.md @@ -0,0 +1,48 @@ +# Authentication and authorization + +## Independent sessions + +ClickHouse and Google authorization are independent: + +```text +ClickHouse identity → execute query and access binding +Google identity → edit destination spreadsheet +``` + +A Google Sheet collaborator does not automatically receive ClickHouse access. A ClickHouse user does not automatically receive permission to edit the spreadsheet. + +## ClickHouse authentication + +The refresh deep link opens the normal SQL Browser SPA. If the ClickHouse session is absent or expired, the existing authentication flow runs. The pending refresh intent must survive the redirect through the URL fragment or tab-scoped session state. + +After authentication, the SPA resolves the binding and executes the pinned query using the current user’s ClickHouse credentials. Binding visibility and query access are enforced by ClickHouse grants and row policies. + +## Google authentication + +Use the Google Identity Services browser token model. The user explicitly invokes a Continue with Google action, receives a short-lived access token in JavaScript, and the SPA calls Google APIs directly. + +Rules: + +- keep Google access tokens in browser memory only; +- never put tokens in URLs, logs, IndexedDB, workspace exports, or spreadsheet metadata; +- request the narrowest scopes that support Picker and editing files selected or created by the application; +- handle popup blocking and multiple signed-in Google accounts explicitly; +- request a new token after expiry through a user gesture; +- do not store a Google refresh token; +- do not use service accounts or impersonation in this project. + +## Account mismatch + +Before modification, show the active Google account and target spreadsheet identity. When the selected Google account lacks edit access, report the account and permission problem without changing the spreadsheet. + +## Secrets and sensitive parameters + +Spreadsheet developer metadata and hidden worksheets are not secret stores. Never write: + +- ClickHouse credentials or JWTs; +- Google tokens; +- complete private SQL unless the design explicitly exposes it; +- password-like parameter values; +- secrets embedded in query parameters. + +Sensitive parameters must use prompt-on-refresh mode. Stored non-sensitive parameters remain part of the versioned binding and are validated as typed ClickHouse parameters before execution. diff --git a/docs/design/google-sheets/configuration.md b/docs/design/google-sheets/configuration.md new file mode 100644 index 0000000..3362bc5 --- /dev/null +++ b/docs/design/google-sheets/configuration.md @@ -0,0 +1,43 @@ +# Deployment configuration + +Google Sheets integration is enabled only when a valid configuration object is present. + +```json +{ + "google_sheets": { + "oauth_client_id": "...apps.googleusercontent.com", + "binding_table": "asb.google_sheet_bindings", + "refresh_log_table": "asb.google_sheet_refresh_log", + "append_refresh": { + "enabled": true, + "min_clickhouse_version": "26.8" + }, + "limits": { + "max_rows_full_refresh": 100000, + "max_rows_per_append_run": 250000, + "max_managed_cells": 2000000, + "max_serialized_bytes": 200000000, + "google_request_target_bytes": 1000000, + "commit_batch_rows": 20000 + } + } +} +``` + +## Rules + +- Absence of `google_sheets` hides all Google actions and preserves existing behavior. +- Parse ClickHouse table names strictly as qualified identifiers and quote database/table components independently. +- The SPA does not create, alter, or migrate configured tables. +- Validate binding and refresh-log table conformance before first use per connection and cache successful checks for the session. +- Host-specific configuration may override top-level configuration for configured alternate ClickHouse hosts. +- Manual or unlisted connections have no Google Sheets capability unless explicitly configured. +- Limits are deployment policy and must be enforced before or during transfer without silent truncation. + +## Required ClickHouse privileges + +Binding operations require the deployment-selected combination of `SELECT` and `INSERT` on configured tables. Query execution requires the ordinary privileges of the pinned saved query. No application-specific `CREATE`, `ALTER`, `UPDATE`, or `DELETE` privilege is required. + +## Feature compatibility + +Full refresh does not require the append minimum version. Append mode additionally requires the configured minimum version and successful runtime probes for the source table and persistent commit-position columns. diff --git a/docs/design/google-sheets/decisions.md b/docs/design/google-sheets/decisions.md new file mode 100644 index 0000000..9d62fce --- /dev/null +++ b/docs/design/google-sheets/decisions.md @@ -0,0 +1,73 @@ +# Architecture decisions + +## GS-001 — Google Sheets is the integration target + +Status: accepted. + +Google Sheets is the collaborative exploration and presentation surface. Google Docs is outside this project. + +## GS-002 — Manual refresh launches the SPA + +Status: accepted. + +The spreadsheet contains a hyperlink to the ClickHouse-hosted SQL Browser SPA. Google does not call SQL Browser, and SQL Browser does not listen for inbound requests. + +## GS-003 — Binding definitions are stored in ClickHouse + +Status: accepted. + +Bindings use append-only versions in a deployment-configured ClickHouse table. This enables durable cross-browser resolution while preserving ClickHouse authorization as the security boundary. + +## GS-004 — Bindings pin saved-query snapshots + +Status: accepted. + +Editing a workspace query does not silently change an existing spreadsheet. Updating the binding is explicit and versioned. + +## GS-005 — Full refresh uses staging + +Status: accepted. + +The previous managed dataset remains live until the complete replacement is uploaded and published into the stable managed sheet ID. + +## GS-006 — Append mode requires newer ClickHouse + +Status: accepted. + +Append mode begins with a configurable minimum of ClickHouse 26.8 and remains subject to runtime capability probes. Older or incompatible servers use full refresh only. + +## GS-007 — Append cursor uses commit position + +Status: accepted. + +The checkpoint is a per-partition map of `_block_number` and `_block_offset`. Values are stored as decimal strings. Updates and deletes are not represented. + +## GS-008 — Streaming SELECT is out of scope + +Status: accepted. + +`SELECT ... STREAM` is not used. Manual refresh consists of finite bounded queries followed by a Google commit. + +## GS-009 — Google state is the destination commit authority + +Status: accepted. + +The hidden `_ASB_STATE` worksheet records the committed row count, generation, hashes, and append checkpoint. Rows and checkpoint are advanced together. + +## GS-010 — Managed and user-owned sheets are separate + +Status: accepted. + +SQL Browser modifies only control, state, staging, and managed data worksheets. User analysis, pivot, and chart worksheets are preserved. + +## GS-011 — No silent fallback or truncation + +Status: accepted. + +Append incompatibility requires explicit full refresh approval. Capacity and limit failures stop with diagnostics rather than truncating or weakening publication safety. + +## GS-012 — Concurrency is advisory in V1 + +Status: accepted. + +Use refresh leases and generation checks, but do not promise strict distributed locking between collaborators without a coordinating backend or Apps Script. diff --git a/docs/design/google-sheets/diagrams/append-checkpoint.html b/docs/design/google-sheets/diagrams/append-checkpoint.html new file mode 100644 index 0000000..e08fc67 --- /dev/null +++ b/docs/design/google-sheets/diagrams/append-checkpoint.html @@ -0,0 +1,9 @@ +Append checkpoint protocol

Append checkpoint protocol

Committed Google state

partition 202607 +blockNumber = 18401 +blockOffset = 982 +committedRows = 143221 +generation = 17
→

Finite ClickHouse page

cursor > stored cursor +cursor <= captured upper bound +ORDER BY _block_number, + _block_offset +LIMIT 20000

Hidden delta staging

Upload the complete page using explicit, idempotent Google ranges.

→

One Google commit

Copy rows to the exact live destination and advance the partition cursor, row count, refresh ID, and generation together.

Invariant: never advance a cursor unless the corresponding rows become live in the same publication operation. If the browser closes before commit, the previous checkpoint remains authoritative.
\ No newline at end of file diff --git a/docs/design/google-sheets/diagrams/architecture.html b/docs/design/google-sheets/diagrams/architecture.html new file mode 100644 index 0000000..99b7712 --- /dev/null +++ b/docs/design/google-sheets/diagrams/architecture.html @@ -0,0 +1 @@ +Google Sheets integration architecture

Google Sheets integration architecture

The browser is the only orchestrator. Google never sends an HTTP request to SQL Browser.

Google Sheet

User clicks a refresh hyperlink.

→
ClickHouse HTTP

Serves the static SQL Browser SPA.

→
SQL Browser SPA

Resolves binding, authenticates, executes, and uploads.

ClickHouse tables

Binding versions, refresh history, and source query data.

↔
SQL Browser SPA

Streams rows through browser memory.

↔
Google Sheets API

Stages and publishes managed worksheet data.

Security boundary: ClickHouse identity authorizes source data; Google identity authorizes destination editing. The refresh URL contains only an opaque binding ID.
\ No newline at end of file diff --git a/docs/design/google-sheets/diagrams/refresh-flow.html b/docs/design/google-sheets/diagrams/refresh-flow.html new file mode 100644 index 0000000..5e3838d --- /dev/null +++ b/docs/design/google-sheets/diagrams/refresh-flow.html @@ -0,0 +1 @@ +Google Sheets refresh flow

Manual linked-sheet refresh

1User clicks Refresh data in Google Sheets.
2ClickHouse serves the static SQL Browser SPA.
3The SPA authenticates and resolves the binding from ClickHouse.
4The user authorizes Google; the SPA verifies spreadsheet identity and edit access.
5The SPA validates query, parameters, schema, limits, and refresh mode.
6ClickHouse streams rows; the SPA uploads byte-bounded chunks to hidden staging.
7A final Google batch publishes into the stable managed sheet and updates state.
8The SPA records refresh history and reports completion.
Failure guarantee

Any failure before publication leaves the previous live dataset and checkpoint unchanged. Ambiguous publication is reconciled by reading the committed generation and refresh ID.

What happens while SQL Browser is closed?

Nothing refreshes. The spreadsheet remains usable with its last committed data, formulas, pivots, charts, comments, and sharing.

\ No newline at end of file diff --git a/docs/design/google-sheets/failure-recovery.md b/docs/design/google-sheets/failure-recovery.md new file mode 100644 index 0000000..803b555 --- /dev/null +++ b/docs/design/google-sheets/failure-recovery.md @@ -0,0 +1,62 @@ +# Failure recovery + +## Refresh states + +```text +idle +→ resolving +→ authenticating +→ validating +→ executing +→ uploading +→ publishing +→ recording +→ complete +``` + +Every refresh has a unique refresh ID, generation, start time, mode, and binding ID. + +## Safety rules + +- No live data mutation before publication. +- No checkpoint advancement without corresponding live rows. +- Previous committed data remains readable after every pre-publication failure. +- Ambiguous network outcomes are resolved by reading committed generation and refresh ID before retrying. + +## Recovery cases + +### Browser tab closed + +Abort in-flight ClickHouse and Google requests where possible. The live sheet remains unchanged. The next refresh removes orphan staging worksheets and restarts full refresh, or resumes append from the last committed page checkpoint. + +### Authentication expired + +Preserve the pending refresh intent through the normal ClickHouse redirect. Google authorization is requested again through an explicit user gesture. + +### Google permission removed + +Stop before execution or publication when possible. Report the active Google account and required edit permission. Do not change state. + +### Query or schema failure + +Keep the previous dataset. Append incompatibilities require an explicit full refresh; full-refresh query failures remain non-destructive. + +### Ambiguous publication + +Read `_ASB_STATE`. Matching refresh ID and expected generation means success; the old generation means no commit; an unexpected generation means another refresh intervened. Never blindly repeat an append publication. + +### Orphan staging + +Staging sheets carry binding and refresh metadata. A later refresh may delete stale staging belonging to the same binding after verifying it is not the live managed sheet. + +### Concurrent refresh + +Use an advisory lease with owner, started-at, and expiry. Before publication, re-read generation and abort if it changed. The product does not promise strict distributed locking. + +### Deleted or copied spreadsheet resources + +Offer explicit repair: recreate a managed worksheet, relink a copied spreadsheet, remove stale metadata, reset checkpoint through full refresh, or unlink. Never identify resources by worksheet name alone. + +## Diagnostics + +Persist a normalized error category, phase, message, ClickHouse query ID when available, Google status code when safe, rows/cells processed, and whether destination state changed. Never record credentials or access tokens. diff --git a/docs/design/google-sheets/full-refresh.md b/docs/design/google-sheets/full-refresh.md new file mode 100644 index 0000000..4fe65de --- /dev/null +++ b/docs/design/google-sheets/full-refresh.md @@ -0,0 +1,36 @@ +# Full refresh + +Full refresh replaces all managed rows while preserving the previous committed dataset until publication succeeds. + +## Procedure + +1. Resolve and authorize the binding. +2. Obtain Google authorization and verify spreadsheet edit access. +3. Validate spreadsheet, managed sheet IDs, metadata, query hash, parameters, limits, and schema policy. +4. Execute the pinned query against ClickHouse. +5. Stream converted rows into a hidden staging worksheet in byte-bounded chunks. +6. Verify row, column, cell, and byte counts. +7. Publish with one structural `spreadsheets.batchUpdate` that clears the managed range, copies staged cells into the stable managed sheet, clears stale trailing cells, applies formats and metadata, updates `_ASB_STATE`, and deletes staging. +8. Record refresh history and update the control worksheet. + +## Publication invariant + +The user observes either: + +```text +old complete dataset +or +new complete dataset +``` + +Never expose a partially refreshed live managed sheet. + +## Append reset + +When the binding supports append mode, a successful full refresh captures and stores a compatible per-partition commit checkpoint. Append mode resumes only after this checkpoint is committed with the full result. + +## Failure behavior + +Before publication, every failure leaves the previous managed data and checkpoint unchanged. Orphan staging worksheets are discovered and removed on the next refresh. An ambiguous publication response is resolved by reading the committed generation and refresh ID before retrying. + +Full refresh is never triggered silently as a fallback from append mode. The user must approve the potentially expensive replacement operation. diff --git a/docs/design/google-sheets/large-results.md b/docs/design/google-sheets/large-results.md new file mode 100644 index 0000000..9614d79 --- /dev/null +++ b/docs/design/google-sheets/large-results.md @@ -0,0 +1,47 @@ +# Large results and quotas + +Large transfers are streamed and chunked. The browser must not retain the complete result in memory or send one giant Google request. + +## Limits + +Enforce configurable row, cell, serialized-byte, duration, and per-cell length limits. Google spreadsheet capacity is evaluated across the entire workbook, including user worksheets and temporary staging. + +Full refresh needs enough capacity for the current managed data and the staged replacement. Append refresh needs capacity for the new rows and temporary delta staging. + +No operation silently truncates output. + +## Pipeline + +```text +ClickHouse response stream +→ incremental result parser +→ canonical type conversion +→ byte-bounded row chunk +→ explicit Google range write +→ release chunk memory +``` + +Chunk boundaries are based primarily on serialized UTF-8 size, with a row-count ceiling. A reasonable default request target is approximately 1 MB, configurable by deployment. + +Use explicit ranges with `values.update` or `values.batchUpdate`, not `values.append`. Explicit ranges make retries idempotent and avoid logical-table inference. + +## Type conversion + +- Boolean and safely representable numbers remain native values. +- UInt64, wider integers, and precision-sensitive decimals are strings. +- Dates and timestamps follow one documented timezone policy. +- NULL uses the configured blank/null representation. +- Arrays, tuples, maps, and JSON use stable serialization. +- Values are written with RAW semantics to prevent formula interpretation. + +## Quota handling + +- Limit concurrent writes to one or two. +- Apply truncated exponential backoff to quota and transient failures. +- Leave headroom for verification, formatting, metadata, publication, and cleanup calls. +- Show separate ClickHouse read and Google upload progress. +- Cancellation stops new writes, aborts ClickHouse where possible, preserves committed live data, and cleans staging when possible. + +## Capacity failure + +When safe staging cannot fit, offer explicit alternatives: reduce rows or columns, aggregate/filter in ClickHouse, export a one-time snapshot to a new spreadsheet, or cancel. Do not degrade a failure-safe refresh into an in-place partial rewrite. diff --git a/docs/design/google-sheets/product-scope.md b/docs/design/google-sheets/product-scope.md new file mode 100644 index 0000000..17525c3 --- /dev/null +++ b/docs/design/google-sheets/product-scope.md @@ -0,0 +1,65 @@ +# Product scope + +## Goal + +Provide a browser-only Google Sheets integration for saved ClickHouse queries while preserving ClickHouse as the data authorization boundary. + +## User workflows + +### Snapshot export + +Run a saved query and copy its current result into a new or existing spreadsheet. No refresh binding or synchronization state is created. + +### Linked spreadsheet + +Create a durable binding between a pinned saved-query snapshot and a spreadsheet. A linked spreadsheet supports manual full refresh and, when eligible, append refresh. + +### Manual refresh + +A hyperlink in the control worksheet opens the ClickHouse-hosted SQL Browser SPA with an opaque binding identifier in the URL fragment. Google never calls SQL Browser. The user authenticates to ClickHouse and Google in the browser, then the SPA performs the refresh. + +## Ownership boundary + +SQL Browser owns: + +- saved-query snapshot and query hash; +- typed parameter policy and values; +- source validation and execution; +- ClickHouse-to-Sheets type conversion; +- managed data worksheet; +- control and hidden state worksheets; +- staging worksheets; +- refresh status, history, and checkpoints. + +Google Sheets owns: + +- formulas outside managed worksheets; +- native pivots and charts; +- comments and collaboration; +- sharing and file organization; +- user-created presentation and analysis worksheets. + +## Refresh modes + +### Full refresh + +Replaces all managed data only after query execution and staging upload succeed. It is supported on all compatible ClickHouse versions. + +### Append refresh + +Appends immutable inserted rows using a ClickHouse commit-order checkpoint. It is available only for validated ClickHouse 26.8+ MergeTree-family sources with persistent `_block_number` and `_block_offset` support. + +Append refresh does not reflect updates or deletes. When those semantics are needed, the user runs a full refresh. + +## Non-goals + +- automatic scheduling; +- background refresh while the SPA is closed; +- inbound callbacks or webhooks to SQL Browser; +- `SELECT ... STREAM`; +- updates, deletes, or generic upsert synchronization; +- arbitrary query rewriting for append eligibility; +- Google Docs; +- Apps Script, add-ons, or an app-specific backend; +- sharing ClickHouse credentials through Google; +- strict collaborative refresh serialization. diff --git a/docs/design/google-sheets/resource-model.md b/docs/design/google-sheets/resource-model.md new file mode 100644 index 0000000..6f88827 --- /dev/null +++ b/docs/design/google-sheets/resource-model.md @@ -0,0 +1,93 @@ +# Resource model + +## Binding + +A linked spreadsheet is a first-class workspace resource. One saved query may have multiple bindings. + +```ts +interface GoogleSheetBindingV1 { + version: 1; + id: string; + name: string; + + spreadsheetId: string; + controlSheetId: number; + managedSheetId: number; + stateSheetId: number; + + sourceQueryId: string | null; + sourceQuerySnapshot: SavedQueryPortableV1; + sourceQueryHash: string; + schemaHash: string | null; + + refreshMode: 'full' | 'append'; + parameterPolicy: 'stored' | 'prompt' | 'defaults'; + parameters: Record; + + appendConfig?: GoogleSheetAppendConfigV1; + limits: GoogleSheetLimitsV1; +} +``` + +The pinned query snapshot is authoritative. Editing the workspace query does not silently change the binding. Updating a binding to the current query is explicit and creates a new binding version. + +## Append configuration + +```ts +interface GoogleSheetAppendConfigV1 { + kind: 'clickhouse-commit-order'; + minimumClickHouseVersion: string; + database: string; + table: string; + tableUuid: string; + engine: 'MergeTree' | 'ReplicatedMergeTree'; + initialization: 'history' | 'future-only'; +} +``` + +## Limits + +```ts +interface GoogleSheetLimitsV1 { + maxRowsFullRefresh: number; + maxRowsPerAppendRun: number; + maxManagedCells: number; + maxSerializedBytes: number; + googleRequestTargetBytes: number; + commitBatchRows: number; +} +``` + +## Google checkpoint + +The hidden state worksheet is authoritative for destination commit state. + +```ts +interface GoogleSheetCheckpointV1 { + version: 1; + bindingId: string; + generation: string; + committedRows: string; + sourceQueryHash: string; + schemaHash: string; + tableUuid?: string; + partitions?: Record; + lastRefreshId: string; + lastRefreshAt: string; +} +``` + +All `UInt64` values are decimal strings and must never round-trip through JavaScript `number`. + +## Server storage + +Bindings are stored as append-only versions in a configured ClickHouse table. Refresh history is stored in a separate configured append-only table. The SPA never creates, alters, or migrates these tables. + +ClickHouse grants and row policies remain the authorization layer. The application does not implement a second ACL system. + +## Portability + +Workspace export preserves binding metadata needed for navigation and relinking, but never exports Google or ClickHouse credentials. Imported bindings must be revalidated against the current server, binding table, spreadsheet identity, and current user permissions before refresh is enabled. diff --git a/docs/design/google-sheets/sharing-security.md b/docs/design/google-sheets/sharing-security.md new file mode 100644 index 0000000..a957816 --- /dev/null +++ b/docs/design/google-sheets/sharing-security.md @@ -0,0 +1,36 @@ +# Sharing and security + +## Sharing model + +Anyone with access to the spreadsheet may see exported data according to Google sharing rules. Refreshing requires both: + +- permission to resolve and execute the binding in ClickHouse; +- permission to edit the destination spreadsheet in Google. + +Spreadsheet sharing never grants ClickHouse access, and ClickHouse access never grants spreadsheet access. + +## Security boundary + +ClickHouse remains the authoritative data-access boundary. Binding rows and refresh-history rows are protected through ClickHouse grants and row policies. SQL Browser does not implement application-managed ACLs or impersonation. + +## Managed metadata + +Developer metadata and hidden worksheets identify resources and store synchronization state, but are untrusted and non-secret. Validate binding ID, spreadsheet ID, numeric sheet IDs, table UUID, query and schema hashes, generations, row counts, and cursor values before use. + +## Formula and content safety + +Write data through RAW value input. Strings beginning with `=`, `+`, `-`, or `@` remain data rather than formulas. Enforce per-cell length and total-byte limits, and normalize unsupported values deterministically. + +## URL and logging safety + +Refresh URLs contain only opaque binding IDs. Do not include SQL, parameters, spreadsheet IDs, tokens, or credentials in fragments, query strings, analytics, console logs, or error reports. + +Spreadsheet IDs may be considered sensitive operational metadata and should be redacted from routine telemetry. + +## Content Security Policy + +Google Identity and Picker resources are loaded only when the integration is invoked. Deployment documentation must list required script, frame, connect, and popup allowances without weakening unrelated CSP protections. + +## Unlinking + +Unlink disables future refresh while preserving exported data by default. Removing managed worksheets is a separate explicit action. SQL Browser does not delete the entire Google spreadsheet in V1. diff --git a/docs/design/google-sheets/snapshot-export.md b/docs/design/google-sheets/snapshot-export.md new file mode 100644 index 0000000..ae948f8 --- /dev/null +++ b/docs/design/google-sheets/snapshot-export.md @@ -0,0 +1,23 @@ +# Snapshot export + +Snapshot export is a one-time transfer. It creates or selects a spreadsheet, writes query results, and creates no durable binding or refresh checkpoint. + +## Flow + +1. Require a saved row-returning query and resolve typed parameters. +2. Ask the user to create or select a spreadsheet. +3. Obtain a short-lived Google access token. +4. Execute the query and stream rows into byte-bounded upload chunks. +5. Write to a new managed data worksheet using RAW values. +6. Apply headers, basic formatting, frozen header, and filter. +7. Report rows, columns, cells, and any conversion warnings. + +## Semantics + +- Snapshot export never modifies an existing linked binding. +- It may write into a new spreadsheet or a newly created worksheet in an existing spreadsheet. +- No refresh hyperlink, `_ASB_STATE`, binding table row, or refresh history row is created. +- No silent truncation is permitted. +- Failure may delete the partially created worksheet when safe; otherwise it is clearly marked incomplete. + +Snapshot export shares the canonical type conversion, quota handling, and streaming upload implementation with linked refreshes. diff --git a/docs/design/google-sheets/spreadsheet-layout.md b/docs/design/google-sheets/spreadsheet-layout.md new file mode 100644 index 0000000..22cd299 --- /dev/null +++ b/docs/design/google-sheets/spreadsheet-layout.md @@ -0,0 +1,44 @@ +# Spreadsheet layout + +A linked spreadsheet contains three application-managed worksheets and any number of user-owned worksheets. + +```text +SQL Browser visible control/status worksheet +_Data · visible managed data worksheet +_ASB_STATE hidden synchronization state +Analysis / Pivots / ... user-owned worksheets +``` + +Temporary refresh worksheets use names such as `_ASB_STAGING_` and remain hidden. + +## Control worksheet + +Display source name, binding name, refresh mode, last successful refresh, row and column counts, current status, source revision, and actions: + +- Refresh data +- Open source query +- Run full refresh +- View refresh history +- Unlink + +The Refresh data action is an ordinary hyperlink to the ClickHouse-hosted SPA with an opaque binding ID in the fragment. + +## Managed data worksheet + +SQL Browser owns the complete used range. It writes values with RAW input semantics, freezes the header, applies stable type-oriented formats, and may create a basic filter. + +Users should not sort, truncate, or edit the managed data worksheet. Analysis requiring sorting or formulas belongs in a separate worksheet. + +The numeric `sheetId` and developer metadata are authoritative. Names are presentation only and may be changed by users. + +## State worksheet + +`_ASB_STATE` stores the committed row count, generation, schema and query hashes, last refresh ID, and append checkpoints. It is hidden and protected where possible, but all content is treated as untrusted input and validated before use. + +## Copies and moves + +Moving a spreadsheet does not break the binding because the spreadsheet ID remains stable. Copying creates a new spreadsheet ID and does not automatically clone the binding. A copied file with stale metadata must offer relink, metadata removal, or cancel. + +## Preservation rule + +Refresh operations must preserve stable managed sheet IDs. Never publish by deleting the old managed sheet and renaming staging, because formulas, charts, and pivots may reference the original sheet identity. diff --git a/docs/design/google-sheets/testing.md b/docs/design/google-sheets/testing.md new file mode 100644 index 0000000..c82ab99 --- /dev/null +++ b/docs/design/google-sheets/testing.md @@ -0,0 +1,57 @@ +# Testing strategy + +## Unit tests + +Cover canonical codecs and validators for bindings, limits, schema descriptors, checkpoints, refresh generations, table identifiers, version comparisons, capability probes, type conversions, append eligibility, query hashes, and error normalization. + +## Google API seam tests + +Use an injected fake Google client to cover: + +- spreadsheet creation and selection; +- developer metadata lookup; +- exact-range chunk writes; +- staging creation and cleanup; +- atomic publication request construction; +- quota backoff and cancellation; +- ambiguous response reconciliation; +- copied, renamed, moved, and deleted spreadsheet resources; +- permission and account mismatch diagnostics. + +## ClickHouse seam tests + +Cover: + +- binding-table conformance and append-only reads/writes; +- pinned query execution with native typed parameters; +- version and runtime capability probes; +- table UUID and engine validation; +- `_block_number`/`_block_offset` cursor query construction; +- per-partition upper-bound capture; +- schema/query/parameter invalidation; +- full-refresh checkpoint reset. + +## Workflow tests + +- snapshot export success, cancellation, and partial cleanup; +- full refresh preserves old data until publication; +- large full refresh streams without complete buffering; +- append commits rows and checkpoint together; +- browser close leaves only staging and committed state; +- append resumes from the last committed page; +- concurrent generation change aborts publication; +- ambiguous append publication is not duplicated; +- full refresh is explicit after append invalidation; +- unlink preserves data and disables refresh. + +## Browser coverage + +Run supported end-to-end workflows in Chromium, Firefox, and WebKit. Include OAuth popup handling, authentication redirects, deep-link restoration, cross-tab behavior, abort controllers, and CSP failures. + +## Real Google integration suite + +Use a dedicated Google Cloud test project and disposable spreadsheets. Never use personal developer files. Tests must clean up created worksheets/files where permitted and leave unique metadata for diagnosing incomplete cleanup. + +## Scale tests + +Include generated results near configured row, cell, byte, and per-cell limits. Assert bounded browser memory, byte-based chunking, progress reporting, no silent truncation, and safe rejection when staging capacity cannot fit.