feat(compiler): split frontend — keyword unification

docs/agents/compiler-review-openspec-handoff.md

@@ -0,0 +1,115 @@
+# Compiler Deep-Review — OpenSpec Orchestration Handoff
+
+Handed off: 2026-08-21 · Branch: **julia/compiler-review-proposals** (base **main** @ 739150a)
+
+## 1. What this is
+
+A deep review of **packages/compiler** (twelve parallel subagent review reports, consolidated)
+produced twelve remediation changes. All twelve are staged here as planning-complete OpenSpec
+changes (proposal + design + tasks, plus spec deltas for the two behavior-changing ones). All
+twelve pass **openspec validate**. No compiler source code has been implemented. You are
+orchestrating the implementation across multiple worker sessions.
+
+Out of scope: **packages/llvm** (owned by a different agent — do not touch it) and the
+uncommitted .claude/ and packages/llvm/** working-tree edits that belong to that other agent. Do
+not git-add, commit, or modify those.
+
+## 2. The twelve changes
+
+| # | Change (dir under openspec/changes/) | Kind | Spec delta home | Primary files touched |
+| - | --- | --- | --- | --- |
+| 1 | remove-compiler-dead-code | refactor (skip_specs) | — | Analysis, Type, Mir, Hir, SuspensionMir, DeclarationIndex, ModuleTooling, SyntaxCorrespondence/ProjectAnalysis, CallableFieldRealization, OpaqueRealization |
+| 2 | extract-compiler-shared-helpers | refactor (skip_specs) | — | ModuleClosure, DeclarationIndex, OpaqueRealization, ModuleSummary, ImportPlan, NameResolution, Layout, Backend, WasmBackend, Lexer, LiteralForm, StaticText, FloatingPoint, Transcendental, Type, Presentation, PhaseReport, Pipeline, Driver |
+| 3 | reconcile-compiler-duplicated-semantics | behavior | bootstrap-complete-interface-contracts | Type, TypeCompatibility, InterfaceWitnessCompatibility, Ownership, SuspensionOwnership, Lower, Token |
+| 4 | harden-compiler-native-boundary | behavior | bootstrap-native-toolchain, bootstrap-backend | NativeToolchain, Driver, Target, Backend, WasmBackend, BootstrapEvaluation |
+| 5 | split-compiler-frontend | refactor (skip_specs) | — | Parser, Lexer |
+| 6 | split-compiler-type-system | refactor (skip_specs) | — | Type, Instances, Pipeline |
+| 7 | split-compiler-elaboration | refactor (skip_specs) | — | Elaboration |
+| 8 | split-compiler-mid-end | refactor (skip_specs) | — | Lower, Ownership, SuspensionOwnership |
+| 9 | split-compiler-layout | refactor (skip_specs) | — | Layout, OpaqueRealization, CallableFieldRealization, Scalar, Match |
+| 10 | split-compiler-ir-evaluation | refactor (skip_specs) | — | Mir, ProvisionalMir, SuspensionMir, BootstrapEvaluation |
+| 11 | split-compiler-declaration-index | refactor (skip_specs) | — | DeclarationIndex |
+| 12 | split-compiler-backends | refactor (skip_specs) | — | Backend, WasmBackend |
+
+skip_specs: true means the change is a pure refactor with no observable behavior change; its
+.openspec.yaml already carries the marker, so openspec validate accepts it with zero spec deltas.
+
+## 3. Dependency graph (waves)
+
+Edges are hard serialization because two changes edit the same file. Everything not listed is
+parallelizable.
+
+Wave A — foundation (run first; 1 and 2 in parallel):
+
+- remove-compiler-dead-code — deletes symbols that later changes rename or relocate.
+- extract-compiler-shared-helpers — creates the internal helpers every split imports.
+
+Wave B — semantic + boundary (3 and 4 in parallel, both after Wave A):
+
+- reconcile-compiler-duplicated-semantics — after 1 + 2 (edits Type.ts, Ownership, Lower).
+- harden-compiler-native-boundary — after 1 + 2 (edits Backend, WasmBackend, BootstrapEvaluation).
+
+Wave C — the eight splits (parallelize, honoring these edges):
+
+| Change | Blocks on |
+| --- | --- |
+| split-compiler-elaboration | — (only touches Elaboration.ts; can start in Wave B) |
+| split-compiler-frontend | 2 (Lexer byte-classification move) |
+| split-compiler-type-system | 2 + 3 (Type.ts) and 4 (Pipeline/Driver) |
+| split-compiler-mid-end | 3 (Ownership/SuspensionOwnership/Lower) |
+| split-compiler-layout | 1 + 2 (OpaqueRealization/CallableFieldRealization/Layout) |
+| split-compiler-ir-evaluation | 1 + 4 (Mir, BootstrapEvaluation) |
+| split-compiler-declaration-index | 1 + 2 (DeclarationIndex) |
+| split-compiler-backends | 2 + 4 (Backend/WasmBackend) |
+
+Recommended schedule: A → B (+ elaboration in parallel any time) → C in two mini-batches of four,
+picking membership so no two mini-batch members touch the same file.
+
+## 4. Ground rules (non-negotiable, from AGENTS.md)
+
+- Green-field: no compatibility contract. Implement the clean target design, delete superseded
+  code, do not keep shims/aliases/fallbacks. A change is incomplete while an obsolete path remains.
+- Verification order for every change: pnpm typecheck → pnpm exec biome check . → pnpm test.
+  Run pnpm check as the final gate before handing a change back.
+- Do not touch packages/llvm or the other agent's uncommitted working-tree edits.
+- Effect conventions: one module per actor, data-first + dual, Effect.fn / Effect.fnUntraced, wrap
+  external APIs in Effect (no bare throw or Promise across a boundary), @effect/vitest tests.
+- No non-null assertions, no as-casts except truths TypeScript cannot express, no lint suppressions.
+- Tests stay cheap: prove semantics with Analysis.evaluate; add corpus programs to
+  test/support/corpus.ts rather than per-feature native-parity tests; assert diagnostic codes and
+  spans, never message text; no per-feature fresh-process determinism tests.
+- If a proposal needs revision during implementation, use the openspec-update-change skill, never
+  hand-edit the artifacts to force validation.
+
+## 5. Orchestration protocol
+
+1. For each change you dispatch, create a branch agent/<change-name> from
+   julia/compiler-review-proposals (or from the current integration tip).
+2. Give the worker exactly one change, the worker prompt below, and the gating rules in section 4.
+3. Worker applies via the openspec-apply-change skill (tasks.md drives the work; applyRequires is
+   already satisfied since tasks are present).
+4. Worker returns: branch name, pnpm check result, a list of every diagnostic-code/span/golden
+   change with justification, and any unfinished task.
+5. You review the diff scope (the change must not wander outside its listed files), merge the
+   branch back into the integration branch, re-run pnpm check once, then dispatch the next wave.
+6. On a conflict you cannot resolve silently (two changes both need an overlapping hunk), pull the
+   later change into the same session rather than editing two proposals in parallel.
+7. After a change is fully implemented and verified, archive it with openspec archive <change> (or
+   the openspec-archive-change skill) to promote its spec deltas into openspec/specs/.
+
+## 6. Worker prompt template
+
+Load the openspec-apply-change skill, then implement the OpenSpec change openspec/changes/<NAME>
+exactly as specified — proposal/design/tasks are authoritative. Follow AGENTS.md green-field and
+Effect conventions. Touch only the files the proposal lists (plus the callers/tests/docs that must
+move with them). Verify in order: pnpm typecheck, pnpm exec biome check ., pnpm test, then pnpm
+check. Report the final check output, every intentional diagnostic-code/span/golden change, and any
+task you could not complete. Do not touch packages/llvm, do not edit the change's own planning
+artifacts, and do not modify unrelated working-tree state.
+
+## 7. Done criteria
+
+- All twelve changes applied, merged to one integration branch, and archived (spec deltas synced).
+- pnpm check green on the integration branch at the end.
+- Each refactor is byte-determinism clean (golden byte comparisons unchanged except where a
+  behavior change — items 3 and 4 — intentionally alters them and the delta is documented).

openspec/changes/extract-compiler-shared-helpers/.openspec.yaml

@@ -0,0 +1,3 @@
+schema: spec-driven
+skip_specs: true
+created: 2026-08-21

openspec/changes/extract-compiler-shared-helpers/design.md

@@ -0,0 +1,23 @@
+## Context
+
+See proposal.md. Each helper is extracted by taking the most general existing implementation, moving it to a single owner, and routing every other call site through it.
+
+## Decisions
+
+- **`internal/Graph.stronglyConnected(nodes, neighbors)`**: parameterized by key/neighbor projection so the module-name cycle (`ModuleClosure`), inline struct-dependency cycle (`DeclarationIndex`), and opaque-realization cycle all call the same fixed point. Preserve each caller's existing deterministic ordering (sorted canonical roots); keep the per-caller sort in the caller.
+- **`ImportPath.spelling(path)` / `canonicalTarget(path)`**: consolidate the "filter segments, require available syntax, join" step; the `/` source-spelling vs `.` canonical form is decided once.
+- **`alignUp`**: export from one low-level module (e.g. `internal/Align.ts`) and import into `Backend.ts:1101`, `WasmBackend.ts:228`, `CoroutineFrame.ts:6`.
+- **`internal/ByteClass.ts`**: move the classifiers from `Lexer`/`LiteralForm` and `hexValue`/`digitValue` from `StaticText`/`IntegerLiteral`.
+- **`internal/Escape.ts`**: single `escapeExtent` (what `scalarCount` computes) derived from the same escape list `StaticText.decode` consumes.
+- **`FloatingPoint.canonicalNaN(width)`**: returns `0x7fc00000n`/`0x7ff8000000000000n`; `fromNumber`, `squareRoot`, and the transcendental `Plan` records reference it.
+- **`PhaseReport.measure`**: one `measure` accepting optional `{ counter, heapBytes }`; `Pipeline.measured`/`measuredModuleWork` and `Driver.phase`/`phaseWithHeap` and the three inline reports all call it. Heap source injected once, not read from `node:process` inline.
+- **`suspensionPointKey`**: export from `Backend.ts` and delete the copy at `WasmBackend.ts:1596`.
+
+## Risks / Trade-offs
+
+- [Determinism] → each extraction preserves the exact existing key/ordering; golden byte comparisons guard regressions.
+- [Ordering subtlety] → Tarjan component sort order stays per-caller.
+
+## Validation
+
+`pnpm typecheck`, `pnpm exec biome check .`, `pnpm test` (golden/determinism suites are the regression net).

openspec/changes/extract-compiler-shared-helpers/proposal.md

@@ -0,0 +1,29 @@
+## Why
+
+The same low-level algorithm is implemented two to four times across unrelated files, each copy a drift hazard. This change extracts one owner per helper so a fix or determinism change is made exactly once.
+
+## What Changes
+
+- **One Tarjan SCC helper** (`internal/Graph.ts`) replaces `ModuleClosure.cycleFacts`, `DeclarationIndex.stronglyConnected`, and the `OpaqueRealization` copy.
+- **One import-path extractor** (`ImportPath.spelling`/`canonicalTarget`) replaces the four hand-rolled `/` vs `.` joins in `ModuleSummary`, `ImportPlan`, `ModuleClosure`, `NameResolution`.
+- **One `alignUp`** shared by `Layout`, `Backend`, `WasmBackend`, and `CoroutineFrame`.
+- **One ASCII byte-classification module** (`internal/ByteClass.ts`): `isAsciiLetter`/`isDecimalDigit`/`isIdentifierStart`/`isIdentifierContinue`/`hexValue`, used by `Lexer`, `LiteralForm`, `StaticText`, `IntegerLiteral`.
+- **One escape-vocabulary module** (`internal/Escape.ts`) owning "which escapes extend a literal" and "what each escape means", closing the `scalarCount`↔`decode` sync risk.
+- **One canonical-NaN constant** (`FloatingPoint.canonicalNaN(width)`) replacing the four hardcoded bit patterns in `FloatingPoint` and `Transcendental`.
+- **One requirement-member renderer** (`Type.encodeRequirement`) consumed by `Presentation`, deleting five inline copies.
+- **One phase-measurement path** (`PhaseReport.measure`) replacing the four re-implementations in `Pipeline` and `Driver`.
+- **One `suspensionPointKey`** exported from `Backend` and imported by `WasmBackend`.
+
+## Capabilities
+
+### New Capabilities
+
+<!-- none -->
+
+### Modified Capabilities
+
+<!-- none: behavior-preserving refactor (skip_specs) -->
+
+## Impact
+
+Pure dedup; no observable behavior change. Touches `ModuleClosure`, `DeclarationIndex`, `OpaqueRealization`, `ModuleSummary`, `ImportPlan`, `NameResolution`, `Layout`, `Backend`, `WasmBackend`, `CoroutineFrame`, `Lexer`, `LiteralForm`, `StaticText`, `IntegerLiteral`, `FloatingPoint`, `Transcendental`, `Type`, `Presentation`, `PhaseReport`, `Pipeline`, `Driver`. `skip_specs: true`.

openspec/changes/extract-compiler-shared-helpers/tasks.md

@@ -0,0 +1,23 @@
+## 1. Graph and import helpers
+
+- [ ] 1.1 Add `internal/Graph.stronglyConnected` and route `ModuleClosure`, `DeclarationIndex`, `OpaqueRealization` through it
+- [ ] 1.2 Add `ImportPath.spelling`/`canonicalTarget` and replace the four inline extractors
+
+## 2. Numeric and byte helpers
+
+- [ ] 2.1 Extract one `alignUp` and import it in `Layout`, `Backend`, `WasmBackend`, `CoroutineFrame`
+- [ ] 2.2 Add `internal/ByteClass.ts` and route `Lexer`, `LiteralForm`, `StaticText`, `IntegerLiteral` through it
+- [ ] 2.3 Add `internal/Escape.ts` and unify `scalarCount`/escape decoding
+- [ ] 2.4 Add `FloatingPoint.canonicalNaN` and replace the four literals
+
+## 3. Rendering, measurement, ABI
+
+- [ ] 3.1 Route `Presentation` through `Type.encodeRequirement`; delete the five inline copies
+- [ ] 3.2 Unify phase measurement on `PhaseReport.measure`; delete `Pipeline`/`Driver` wrappers and inline reports
+- [ ] 3.3 Export `suspensionPointKey` from `Backend` and delete the `WasmBackend` copy
+
+## 4. Verification
+
+- [ ] 4.1 Run `pnpm typecheck` and verify clean
+- [ ] 4.2 Run `pnpm exec biome check .` and verify clean
+- [ ] 4.3 Run `pnpm test` and verify golden/determinism suites pass

openspec/changes/harden-compiler-native-boundary/.openspec.yaml

@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-08-21

openspec/changes/harden-compiler-native-boundary/design.md

@@ -0,0 +1,49 @@
+## Context
+
+See proposal.md. The boundary sits between `NativeToolchain.ts` (Node, deep import), `Driver.ts` / `Pipeline.ts` (Effect orchestration), `Target.ts` (pure data), `Backend.ts` / `WasmBackend.ts` (Effect backends), and `BootstrapEvaluation.ts` (synchronous leaf).
+
+## Goals / Non-Goals
+
+**Goals:** one owner per external dependency; expected failures in the typed error channel; temp-dir/scope cleanup via `acquireRelease`; a genuinely browser-safe root barrel.
+
+**Non-Goals:** changing build-plan semantics; renaming public subpaths; god-module splits (separate changes).
+
+## Decisions
+
+### NativeToolchain service boundary
+
+Split `NativeToolchain` into a pure planning surface (existing `ToolchainPlan`) and an effectful executor. Add a `ToolchainError extends Data.TaggedError` with `operation`, `message`, and a `reason` union (`SpawnFailed`, `StorageFailed`, `LinkFailed`) where the wrapped branch carries `cause`. `withBuildScope` becomes `Effect.acquireRelease(mkdtemp, rmSync)`. `runPlanned`/`writeArtifact`/`commit`/`commitCachedArtifact`/`ClangLinker.link` all `yield*` typed failures. `Driver.compile` `yield*`s them instead of calling synchronously.
+
+### One atomic-commit seam
+
+Add `atomicCommit(destination, bytes, { mode? })`: mkdir, write temp sibling (`${path}.silk-tmp-${pid}`), `renameSync`, and `rmSync` the temp in a finally-like bracket on any failure. Route `makeDiskArtifactCache.set` (104), `commitCachedArtifact` (192), `commit` (322), and `ClangLinker.link` (458) through it. This fixes the leak where `makeDiskArtifactCache.set`'s catch did not remove its temp.
+
+### Host detection leaves Target
+
+Move `platform()`/`arch()` into `NativeToolchain` (or a small `HostTarget.ts` next to it) and have `Target.fromHost` accept the values instead of reading `node:os`. `Target.ts` becomes node-free. Collapse `Target.triple` into `Target.id` (they are always equal).
+
+### Driver artifact hand-off
+
+Have `finalizeWasm` / `ClangLinker.link` return `{ path, bytes }` so `Driver` seeds the artifact cache without `readFileSync`. Provide `memoryUsage` through an injected `HeapObservation` service (default Node impl at the app edge) instead of importing `node:process` in `Driver`.
+
+### Backend error channels
+
+`WasmBackend.ts:6133` returns `yield* new BackendError({ reason: { _tag: 'InvalidMir', ... } })` instead of throwing. `Backend.emitProgram` becomes `Effect.fnUntraced` pinned to `LlvmError | BackendError`; `LlvmBackend.emit` wraps via `patchErrors`/`catchTag` on `LlvmError` rather than `cause._tag === 'BackendError'` probing. `locate` takes the concrete instruction type (the return of `Value.instruction`) and drops the `unknown` cast.
+
+### BootstrapEvaluation host catches
+
+`HostWrite` (`BootstrapEvaluation.ts:2912`) and the `OsCall` arm (`:3342`) keep the typed-result path for expected failures and reserve the catch for unexpected throws, mapping them with `cause` preserved rather than a bare `'Other'`.
+
+### Remove NativeLinker
+
+Delete the `NativeLinker` interface; `Driver` calls the `ClangLinker` implementation directly (single implementation today).
+
+## Risks / Trade-offs
+
+- [Error channel change is observable] → new spec scenarios pin the typed-failure behavior; test replacements verify `cause`/stage.
+- [Browser-safe barrel] → a bundle smoke check asserts no `node:` module in the root import graph.
+- [Atomic commit behavior] → the determinism/durable-cache tests plus a new "failed rename leaves no temp" case.
+
+## Validation
+
+`pnpm typecheck`, `pnpm exec biome check .`, `pnpm test`; new `@effect/vitest` cases for spawn/storage/link typed failures, temp cleanup, and the wasm invalid-request `BackendError`.

openspec/changes/harden-compiler-native-boundary/proposal.md

@@ -0,0 +1,28 @@
+## Why
+
+The compiler's native-toolchain boundary violates the effect-patterns boundary rule. `NativeToolchain.ts` is a fully synchronous imperative core (raw `node:child_process`/`node:crypto`/`node:fs`/`node:os`/`node:path` with scattered try/catch and a manual try/finally in `withBuildScope`) that returns data-union failures, then gets driven synchronously inside `Driver.compile`'s `Effect.gen`. A spawn/fs throw therefore escapes the generator as a Defect rather than a typed error, and the same fs failure lands in two different channels depending on which line threw. `node:*` ownership is split three ways, `Target.ts` imports `node:os` and is re-exported from the barrel the design claims is browser-safe, `Driver.ts` re-reads artifacts from disk with raw `readFileSync`, and `WasmBackend` throws a `RangeError` inside an `Effect.gen` instead of yielding a typed `BackendError`.
+
+## What Changes
+
+- **`NativeToolchain` becomes a thin Effect boundary**: `Effect.try` around spawn/fs, `Effect.acquireRelease` for build scopes and temp dirs, a `ToolchainError` family in the error channel; `ToolchainPlan` stays pure.
+- **One `atomicCommit` seam** (guaranteed cleanup) replaces four divergent copy-paste copies and fixes the stale-temp leak in `makeDiskArtifactCache.set`.
+- **`node:os` host detection moves out of `Target`** into a Node-only boundary, so importing the package root no longer pulls node built-ins; the redundant `Target.triple` field is collapsed.
+- **`Driver` stops re-reading artifacts** with `readFileSync` (finalizers return bytes); `node:fs`/memory telemetry become single-owner.
+- **`WasmBackend`** routes request-validation failures to `BackendError` instead of throwing; **`Backend.emitProgram`** pins its error channel with `catchTag` rather than `_tag`-string probing and becomes `Effect.fnUntraced` with a concrete debug-location type.
+- **`BootstrapEvaluation`** host-provider catches preserve the cause instead of flattening to `'Other'`.
+- **Remove the never-injected `NativeLinker` interface** (single implementation, never provided).
+
+## Capabilities
+
+### New Capabilities
+
+<!-- none -->
+
+### Modified Capabilities
+
+- `bootstrap-native-toolchain`: typed failure channel, atomic artifact/cache commit with cleanup, browser-safe barrel, in-memory artifact hand-off.
+- `bootstrap-backend`: expected request-validation failures yield typed `BackendError`, never defects.
+
+## Impact
+
+Observable: native-toolchain failures now surface as typed errors (callers can recover on stage and cause), and invalid backend requests no longer bypass `BackendError`. Everything else is internal refactoring. Node-only code stays a deep import; `@silk-effect/compiler` root becomes genuinely browser-safe.