feat(store): run SQLite on Node by default, mirroring the Cloudflare deploy

.changeset/sqlite-node-store.md

@@ -0,0 +1,24 @@
+---
+"sideshow": minor
+---
+
+The local Node server now stores data in SQLite (via the built-in `node:sqlite`)
+by default — the same `SqlStore` the Cloudflare Durable Object deploy runs, so
+local development mirrors production over one storage code path instead of a
+separate JSON file. On first SQLite boot an existing `sideshow.json` board is
+migrated in once automatically (sessions, surfaces, version history, comment
+ordering, and assets preserved); the JSON file is left untouched as a backup,
+and the import never runs again or overwrites a non-empty database.
+
+This also fixes the JSON store's scaling cliff — it rewrote the entire file
+(assets base64-inlined) on every write — since assets are now per-row BLOBs.
+
+Both stores now strip embedded NUL bytes from stored text (titles, comments,
+trace labels, settings) so they behave identically — SQLite would otherwise
+truncate a value at the first NUL while the JSON file preserved it.
+
+Configuration: `SIDESHOW_STORE=json` keeps the legacy single-file JSON store;
+`SIDESHOW_DB` sets the SQLite file path (default `data/sideshow.db`);
+`SIDESHOW_DATA` still names the JSON file and doubles as the migration source.
+The `sideshow/server` package now also exports `SqlStore` and
+`createSqliteStorage` alongside `JsonFileStore`.

AGENTS.md

@@ -39,9 +39,17 @@ consciously, not as a side effect):
   is the reference-aware LRU policy.
 - `server/public.ts` — the `sideshow/server` package export (`createApp`,
   `JsonFileStore`, types) for embedding the app in a Node process.
-- `server/storage.ts` — `JsonFileStore` (local Node). `workers/sqlStore.ts` —
-  `SqlStore` (Durable Object SQLite). Both must pass `test/storeContract.ts`,
-  and both migrate legacy `snippets`/`snippetId` data to surfaces on load.
+- `server/sqlStore.ts` — `SqlStore`, the SQLite-backed `Store`. It takes a
+  `SqlStorage` (the narrow SQL surface declared in `types.ts`, not the ambient
+  Cloudflare global), so the SAME store runs on the Durable Object
+  (`ctx.storage.sql`) and on Node via `server/sqliteStorage.ts`'s `node:sqlite`
+  adapter — the local default, so dev mirrors the deploy. `server/storage.ts` —
+  `JsonFileStore`, the legacy single-file store, still selectable with
+  `SIDESHOW_STORE=json`. All must pass `test/storeContract.ts`, and all migrate
+  legacy `snippets`/`snippetId` data to surfaces on load. On first SQLite boot
+  `migrateJsonToSqlite` copies an existing JSON board in once (identity, history,
+  and comment `seq` preserved via `JsonFileStore.exportBoard` →
+  `SqlStore.importBoard`); it's idempotent and never imports into a non-empty db.
 - `server/kits.ts` — opt-in style/behavior bundles for html parts (`issues`,
   `slides`). An html part lists kit ids in `kits`; `renderHtmlPage` injects each
   kit's CSS/JS into the sandbox after the base. Runtime-agnostic; allowlisted in
@@ -179,10 +187,13 @@ The first four must pass before committing; pre-commit formats staged files
 
 Testing notes:
 
-- `runStoreContract()` runs the same suite against both stores. SqlStore runs
-  on a `node:sqlite` shim (`test/sqlStorageShim.ts`); the ambient `SqlStorage`
-  types live in `test/workersSqlTypes.d.ts` because the real workers types
-  conflict with `@types/node`.
+- `runStoreContract()` runs the same suite against every store. SqlStore runs
+  on `createSqliteStorage()` (`:memory:`), the same `node:sqlite` adapter the
+  local server uses on disk — so the contract covers the real Node SQLite path.
+  `SqlStorage`/`SqlStorageValue`/`SqlStorageCursor` are plain interfaces in
+  `server/types.ts`; a real DO `SqlStorage` is structurally assignable, so no
+  ambient Cloudflare globals are needed in the node program. `test/migration.
+test.ts` covers the JSON→SQLite import.
 - `JsonFileStore` returns live objects that later mutate — capture fields
   before update calls when asserting against them.
 - The update-notes card is also a `.card`: scope snippet-card e2e selectors

docs/uploads-design.md

@@ -129,9 +129,8 @@ and `JsonFileStore`'s on-disk JSON).
 
 - **`SqlStore`**: a new table with a `BLOB` `data` column — 33% smaller than
   base64 TEXT and no decode-on-serve, which directly helps the shared ~10 GB DO
-  ceiling. DO SQLite stores blobs natively; the `node:sqlite` contract shim must
-  be extended to bind `Uint8Array` (today `sqlStorageShim.ts` only binds
-  `string | number | bigint | null`) — a one-line type widening, since
+  ceiling. DO SQLite stores blobs natively; the `node:sqlite` adapter
+  (`server/sqliteStorage.ts`) binds `Uint8Array` for BLOB columns, since
   `node:sqlite` already round-trips blobs.
 
   ```sql

server/index.ts

@@ -3,7 +3,10 @@ import { readFile } from "node:fs/promises";
 import { basename, dirname, join } from "node:path";
 import { fileURLToPath } from "node:url";
 import { createApp } from "./app.ts";
+import { SqlStore } from "./sqlStore.ts";
+import { createSqliteStorage, migrateJsonToSqlite } from "./sqliteStorage.ts";
 import { JsonFileStore } from "./storage.ts";
+import type { Store } from "./types.ts";
 
 // Source layout puts this file at server/index.ts; the published package runs
 // the compiled copy at dist/server/index.js. viewer/ and guide/ live at the
@@ -25,8 +28,34 @@ const [viewerHtml, guideMarkdown, setupText, agentHowtoText, pkgJson] = await Pr
 const pr = process.env.SIDESHOW_PUBLIC_READ;
 const publicRead = pr === "session" || pr === "full" ? pr : undefined;
 
+// Storage backend. SQLite (via node:sqlite) is the default so the local server
+// mirrors the Cloudflare Durable Object deploy — both run the same SqlStore.
+// SIDESHOW_STORE=json selects the legacy single-file JSON store instead.
+// SIDESHOW_DATA names the JSON file (and the one-time migration source);
+// SIDESHOW_DB names the SQLite file.
+const jsonPath = process.env.SIDESHOW_DATA ?? join(root, "data", "sideshow.json");
+// The SQLite file defaults next to the JSON one (same dir, `.db` suffix) so a
+// deploy that only sets SIDESHOW_DATA still gets an isolated, co-located db —
+// and the migration source sits right beside it.
+const dbPath = process.env.SIDESHOW_DB ?? `${jsonPath.replace(/\.json$/, "")}.db`;
+let store: Store;
+if (process.env.SIDESHOW_STORE === "json") {
+  store = new JsonFileStore(jsonPath);
+  console.log(`sideshow store: JSON file at ${jsonPath}`);
+} else {
+  const sqlite = new SqlStore(createSqliteStorage(dbPath));
+  // First SQLite boot with a legacy JSON file present copies it in once.
+  await migrateJsonToSqlite(sqlite, jsonPath);
+  store = sqlite;
+  // Announce the backend so an existing SIDESHOW_DATA deploy isn't surprised by
+  // the silent switch to SQLite (set SIDESHOW_STORE=json to keep the old store).
+  console.log(
+    `sideshow store: SQLite at ${dbPath} (SIDESHOW_STORE=json for the legacy JSON store)`,
+  );
+}
+
 const app = createApp({
-  store: new JsonFileStore(process.env.SIDESHOW_DATA ?? join(root, "data", "sideshow.json")),
+  store,
   viewerHtml,
   guideMarkdown,
   setupText,

server/public.ts

@@ -2,5 +2,7 @@
 // HTTP/SSE/MCP app without depending on the package's internal dist layout.
 
 export { createApp, type AppOptions, type AuthenticateHook } from "./app.js";
+export { SqlStore } from "./sqlStore.js";
+export { createSqliteStorage, migrateJsonToSqlite } from "./sqliteStorage.js";
 export { JsonFileStore } from "./storage.js";
 export type * from "./types.js";

workers/sqlStore.ts → server/sqlStore.ts

@@ -1,5 +1,6 @@
 import {
   type Asset,
+  type BoardSnapshot,
   collectAssetIds,
   type Comment,
   type CommentQuery,
@@ -14,16 +15,21 @@ import {
   newId,
   selectEvictions,
   type Session,
+  type SqlStorage,
+  type SqlStorageValue,
+  stripNul,
+  stripNulStep,
   type Store,
   type Surface,
   type SurfacePart,
   type SurfaceVersion,
   type TraceStep,
   type UpdateSurfaceInput,
-} from "../server/types.ts";
+} from "./types.ts";
 
-// Store implementation on Durable Object SQLite. One board = one DO = one
-// database, so plain SQL with no tenant columns.
+// Store implementation on SQLite — a Durable Object's `ctx.storage.sql` in the
+// Worker, or node:sqlite via an adapter on Node (see server/sqliteStorage.ts).
+// One board = one database, so plain SQL with no tenant columns.
 export class SqlStore implements Store {
   private sql: SqlStorage;
 
@@ -140,8 +146,8 @@ export class SqlStore implements Store {
     };
   }
 
-  // The BLOB comes back as an ArrayBuffer (real DO) or a Uint8Array (the
-  // node:sqlite test shim); normalize to a fresh Uint8Array either way.
+  // The BLOB comes back as an ArrayBuffer (real DO) or a Uint8Array
+  // (node:sqlite); `new Uint8Array(raw)` copies from either into a fresh array.
   private rowToAsset(r: Record<string, SqlStorageValue>): Asset {
     const raw = r.data as ArrayBuffer | Uint8Array;
     return {
@@ -151,7 +157,7 @@ export class SqlStore implements Store {
       contentType: r.contentType as string,
       byteLength: r.byteLength as number,
       filename: (r.filename as string) ?? null,
-      data: raw instanceof Uint8Array ? new Uint8Array(raw) : new Uint8Array(raw),
+      data: new Uint8Array(raw),
       createdAt: r.createdAt as string,
       lastAccessedAt: r.lastAccessedAt as string,
     };
@@ -188,9 +194,9 @@ export class SqlStore implements Store {
     const now = new Date().toISOString();
     const session: Session = {
       id: newId(),
-      agent: input.agent.trim() || "agent",
-      title: input.title?.trim() || null,
-      cwd: input.cwd ?? null,
+      agent: stripNul(input.agent).trim() || "agent",
+      title: stripNul(input.title)?.trim() || null,
+      cwd: stripNul(input.cwd ?? null),
       createdAt: now,
       lastActiveAt: now,
       agentSeq: 0,
@@ -210,7 +216,7 @@ export class SqlStore implements Store {
   async renameSession(id: string, title: string) {
     const session = await this.getSession(id);
     if (!session) return null;
-    session.title = title.trim() || null;
+    session.title = stripNul(title).trim() || null;
     this.sql.exec("UPDATE sessions SET title = ? WHERE id = ?", session.title, id);
     return session;
   }
@@ -259,8 +265,8 @@ export class SqlStore implements Store {
   async setSetting(key: string, value: string) {
     this.sql.exec(
       "INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
-      key,
-      value,
+      stripNul(key),
+      stripNul(value),
     );
   }
 
@@ -287,7 +293,7 @@ export class SqlStore implements Store {
     const surface: Surface = {
       id: newId(),
       sessionId: input.sessionId,
-      title: input.title?.trim() || "Untitled",
+      title: stripNul(input.title)?.trim() || "Untitled",
       parts: input.parts,
       createdAt: now,
       updatedAt: now,
@@ -328,7 +334,8 @@ export class SqlStore implements Store {
         },
       ];
       if (history.length > HISTORY_LIMIT) history.shift();
-      const title = patch.title !== undefined ? patch.title.trim() || surface.title : surface.title;
+      const title =
+        patch.title !== undefined ? stripNul(patch.title).trim() || surface.title : surface.title;
       const parts = patch.parts !== undefined ? patch.parts : surface.parts;
       const version = surface.version + 1;
       const updatedAt = new Date().toISOString();
@@ -388,15 +395,16 @@ export class SqlStore implements Store {
     const surface = input.surfaceId ? await this.getSurface(input.surfaceId) : null;
     const id = newId();
     const createdAt = new Date().toISOString();
-    const author = input.author.trim() || "user";
+    const author = stripNul(input.author).trim() || "user";
+    const text = stripNul(input.text);
     this.sql.exec(
       "INSERT INTO comments (id, sessionId, surfaceId, surfaceTitle, author, text, createdAt) VALUES (?, ?, ?, ?, ?, ?, ?)",
       id,
       input.sessionId,
       surface?.id ?? null,
       surface?.title ?? null,
       author,
-      input.text,
+      text,
       createdAt,
     );
     const seq = this.sql.exec("SELECT last_insert_rowid() AS seq").one().seq as number;
@@ -408,7 +416,7 @@ export class SqlStore implements Store {
       surfaceId: surface?.id ?? null,
       surfaceTitle: surface?.title ?? null,
       author,
-      text: input.text,
+      text,
       createdAt,
     };
   }
@@ -436,7 +444,8 @@ export class SqlStore implements Store {
   async setTrace(sessionId: string, steps: TraceStep[]) {
     this.sql.exec("DELETE FROM trace_steps WHERE sessionId = ?", sessionId);
     let seq = 0;
-    for (const s of steps) {
+    for (const raw of steps) {
+      const s = stripNulStep(raw);
       this.sql.exec(
         "INSERT INTO trace_steps (sessionId, seq, kind, label, detail, ts) VALUES (?, ?, ?, ?, ?, ?)",
         sessionId,
@@ -490,9 +499,9 @@ export class SqlStore implements Store {
       id,
       sessionId: input.sessionId,
       kind: input.kind,
-      contentType: input.contentType,
+      contentType: stripNul(input.contentType),
       byteLength: input.data.byteLength,
-      filename: input.filename ?? null,
+      filename: stripNul(input.filename ?? null),
       data: input.data,
       createdAt: now,
       lastAccessedAt: now,
@@ -548,4 +557,98 @@ export class SqlStore implements Store {
   async isAssetReferenced(id: string) {
     return this.referencedAssetIds().has(id);
   }
+
+  // One-time bulk import to migrate another backend's data into this database
+  // (see server/sqliteStorage.ts → migrateJsonToSqlite). Every field is written
+  // verbatim — ids, versions, history, the comment `seq` and `agentSeq` the
+  // feedback cursor keys on, asset bytes — so identity survives the copy.
+  // Wrapped in a transaction so a crash mid-copy rolls back to an empty db
+  // rather than a half-migrated board. Intended for an empty database; the
+  // caller gates on that. Only ever runs through the node:sqlite adapter.
+  importBoard(snapshot: BoardSnapshot): void {
+    this.sql.exec("BEGIN");
+    try {
+      for (const s of snapshot.sessions) {
+        this.sql.exec(
+          "INSERT INTO sessions (id, agent, title, cwd, createdAt, lastActiveAt, agentSeq) VALUES (?, ?, ?, ?, ?, ?, ?)",
+          s.id,
+          s.agent,
+          s.title,
+          s.cwd,
+          s.createdAt,
+          s.lastActiveAt,
+          s.agentSeq,
+        );
+      }
+      for (const s of snapshot.surfaces) {
+        this.sql.exec(
+          "INSERT INTO surfaces (id, sessionId, title, parts, createdAt, updatedAt, version, history) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+          s.id,
+          s.sessionId,
+          s.title,
+          JSON.stringify(s.parts),
+          s.createdAt,
+          s.updatedAt,
+          s.version,
+          JSON.stringify(s.history),
+        );
+      }
+      for (const c of snapshot.comments) {
+        this.sql.exec(
+          "INSERT INTO comments (seq, id, sessionId, surfaceId, surfaceTitle, author, text, createdAt) VALUES (?, ?, ?, ?, ?, ?, ?, ?)",
+          c.seq,
+          c.id,
+          c.sessionId,
+          c.surfaceId ?? null,
+          c.surfaceTitle ?? null,
+          c.author,
+          c.text,
+          c.createdAt,
+        );
+      }
+      for (const t of snapshot.traces) {
+        let seq = 0;
+        for (const step of t.steps) {
+          this.sql.exec(
+            "INSERT INTO trace_steps (sessionId, seq, kind, label, detail, ts) VALUES (?, ?, ?, ?, ?, ?)",
+            t.sessionId,
+            seq++,
+            step.kind ?? null,
+            step.label,
+            step.detail ?? null,
+            step.ts ?? null,
+          );
+        }
+      }
+      for (const a of snapshot.assets) {
+        const buf = a.data.buffer.slice(
+          a.data.byteOffset,
+          a.data.byteOffset + a.data.byteLength,
+        ) as ArrayBuffer;
+        this.sql.exec(
+          "INSERT INTO assets (id, sessionId, kind, contentType, byteLength, filename, data, createdAt, lastAccessedAt) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)",
+          a.id,
+          a.sessionId,
+          a.kind,
+          a.contentType,
+          a.byteLength,
+          a.filename,
+          buf,
+          a.createdAt,
+          a.lastAccessedAt,
+        );
+      }
+      for (const { key, value } of snapshot.settings) {
+        this.sql.exec(
+          "INSERT INTO settings (key, value) VALUES (?, ?) ON CONFLICT(key) DO UPDATE SET value = excluded.value",
+          key,
+          value,
+        );
+      }
+      this.sql.exec("COMMIT");
+    } catch (e) {
+      this.sql.exec("ROLLBACK");
+      throw e;
+    }
+  }
 }