Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,6 @@ public/robots.txt
public/sitemap-0.xml
public/sitemap.xml
public/feed.json

# macOS
.DS_Store
48 changes: 48 additions & 0 deletions app/globals.css
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,54 @@

@custom-variant dark (&:is(html[class~="dark"] *));

/* ===========================================================================
Sandworm accent tokens — site-wide.
Raw HSL triplets (consumed as `hsl(var(--teal-500) / 0.5)`), so any
component can tint without redeclaring a color. LIGHT is the base; DARK
overrides under `html.dark` (next-themes, which also resolves "system").
The light shades are deliberately DARKER than Sandworm's native register
so they stay legible on a light ground — don't "correct" them upward
without re-checking contrast.
Previously scoped under `.docs-home` in components/docs-home.css, which
put them out of reach of everything else on the site. The `--stone-*` ramp
stays scoped there: it's inverted for that page and isn't general-purpose.
======================================================================== */
:root {
--magenta-300: 316 50% 45%;
--magenta-400: 316 48% 47%;
--magenta-500: 316 50% 44%;
--magenta-600: 316 52% 40%;
--magenta-950: 315 33.3% 7.1%;
--red-400: 352 70% 48%;
--teal-300: 178 40% 32%;
--teal-400: 178 38% 35%;
--teal-500: 178 35% 33%;
--teal-700: 178 30% 26%;
--violet-400: 253 60% 52%;
--violet-500: 253 62% 50%;
--violet-700: 255 50% 46%;
--sand-300: 30 75% 42%;
--blue-500: 200 75% 40%;
}

html.dark {
--magenta-300: 318 41.2% 70%;
--magenta-400: 317 40.2% 60%;
--magenta-500: 316 39.2% 51%;
--magenta-600: 316 42% 45%;
--magenta-950: 315 33.3% 7.1%;
--red-400: 355 91% 73.9%;
--teal-300: 175 27.5% 72.9%;
--teal-400: 176 28% 56%;
--teal-500: 177 28.2% 45.9%;
--teal-700: 178 30% 30%;
--violet-400: 253 76.5% 70%;
--violet-500: 253 73.4% 63.1%;
--violet-700: 255 58% 43.9%;
--sand-300: 28 100% 72%;
--blue-500: 195 73.4% 63.1%;
}

@theme {
--breakpoint-*: initial;
--breakpoint-sm: 640px;
Expand Down
1 change: 1 addition & 0 deletions app/materialize/api/_meta.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,5 @@ export default {
"client-sdks": "Client SDKs",
"watch-permission-sets": "WatchPermissionSets",
"lookup-permission-sets": "LookupPermissionSets",
"download-permission-sets": "DownloadPermissionSets",
};
2 changes: 2 additions & 0 deletions app/materialize/api/client-sdks/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: "SDK versions with generated gRPC and protobuf support for the Auth

# Client SDKs

<FeatureBadge />

All SpiceDB SDKs have the generated gRPC and protobuf code

- [authzed-go v0.15.0](https://github.com/authzed/authzed-go/releases/tag/v0.15.0)
Expand Down
103 changes: 103 additions & 0 deletions app/materialize/api/download-permission-sets/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
---
title: "DownloadPermissionSets"
description: "Bulk-download permission sets as Parquet or Avro files for a fast initial backfill, with guidance on parsing normalized and denormalized files and why Parquet is the recommended format."
---

import { Callout } from "nextra/components";

# DownloadPermissionSets

<FeatureBadge />

For large datasets, `DownloadPermissionSets` is a faster alternative to [LookupPermissionSets] for the initial [backfill](../concepts/permission-set-lifecycle): instead of streaming individual events over a single connection, it hands you a manifest of files you can fetch directly from blob storage, in parallel.

## Request

The request maps to [`DownloadPermissionSetsRequest`]:

```json
{
"optional_at_revision": "the_zed_token"
}
```

If `optional_at_revision` is omitted, Materialize returns files for the latest fully-published revision. If provided, Materialize returns files for that specific revision if they're still available (see [Snapshot rotation](../concepts/snapshots#snapshot-lifecycle-events)).

Once you've backfilled from the downloaded files, switch to [WatchPermissionSets] to keep your copy current. See [The Permission Set Lifecycle](../concepts/permission-set-lifecycle) for the full flow.

## Response

The response ([`DownloadPermissionSetsResponse`]) lists the available [`File`]s:

```json
{
"timestamp": "2026-07-01T12:00:00Z",
"files": [
{ "name": "metadata.json", "url": "https://..." },
{ "name": "members.parquet", "url": "https://..." },
{ "name": "sets.parquet", "url": "https://..." },
{ "name": "member_to_set.parquet", "url": "https://..." },
{ "name": "set_to_set.parquet", "url": "https://..." }
]
}
```

Each file's `url` is a **time-limited signed URL**: download promptly, and re-call `DownloadPermissionSets` to get fresh URLs if a batch download runs long. `metadata.json` is always included and describes the revision:

```json
{
"revision": "the_zed_token",
"watched_permissions": ["document#view@user", "document#edit@user"],
"numShards": 4,
"files": ["members-shard-0.parquet", "members-shard-1.parquet", "..."]
}
```

<Callout type="warning">
Large datasets are sharded across multiple files per table (`members-shard-0.parquet`,
`members-shard-1.parquet`, and so on). The internal IDs in each table are only unique *within
their own shard*. Read `metadata.json`'s `files` list to discover the actual shard files for a
revision, and treat each table's shards as one combined multi-file dataset rather than
concatenating them naively.
</Callout>

## File formats

<Callout type="warning">
Avro support is being phased out. New integrations should use Parquet: it's faster to query, works
natively with standard data tools, and is where future development is focused.
</Callout>

Which formats a given revision publishes depends on how your Materialize instance is configured. Work with your AuthZed account team to enable Parquet if you're not seeing it in your manifest yet.

### Parquet (recommended)

Parquet ships in two shapes, covering the same [permission set](../concepts/permission-sets) data at different levels of normalization:

- **Normalized** (`members.parquet`, `sets.parquet`, `member_to_set.parquet`, `set_to_set.parquet`): mirrors the underlying model directly. `members.parquet` and `sets.parquet` assign a compact integer `internal_id` to each subject and each resource#permission set. `member_to_set.parquet` expresses each **member → set** edge as a `member_id` column plus a `sets` column (a list of set IDs). `set_to_set.parquet` expresses each **set → set** edge (see [Permission Sets](../concepts/permission-sets) for what "child" and "parent" mean here) as a `child_set_id` column plus a `parent_sets` column (a list of parent set IDs). Smaller on disk, but you'll need to join back to `members`/`sets` to recover the actual `subject_type`/`subject_id` or `resource_type`/`resource_id` values.
- **Flat** (`flat_member_to_set.parquet`, `flat_set_to_set.parquet`): fully denormalized, every row already carries the resolved string values directly, no lookup tables, no joins. `flat_member_to_set.parquet` has `resource_type`, `resource_rel`, `resource_id`, `subject_type`, `subject_rel`, `subject_id`. `flat_set_to_set.parquet` has the same six fields split into `child_resource_type`/`child_resource_rel`/`child_resource_id` and `parent_resource_type`/`parent_resource_rel`/`parent_resource_id`. Pick this if you're loading straight into a query engine or your own database.

Both shapes are plain Parquet files, so any Parquet reader works: DuckDB, pandas/pyarrow, Spark, and most data warehouses can query them directly without a custom parser. For example, with the flat files and [DuckDB](https://duckdb.org/):

```sql
-- Every subject that can view a given document, no joins required
SELECT subject_type, subject_id
FROM 'flat_member_to_set.parquet'
WHERE resource_type = 'document' AND resource_id = '123' AND resource_rel = 'view';
```

### Avro (legacy)

Avro files (`members.avro`, `sets.avro`, `member_to_set.avro`, `set_to_set.avro`) use the same normalized shape as the Parquet equivalents: integer `internal_id`s in `members`/`sets`, referenced by `member_to_set`/`set_to_set`. They're encoded as Avro [Object Container Format](https://avro.apache.org/docs/current/specification/#object-container-files), which embeds its own schema, so any standard Avro library can decode a file without needing the schema separately.

## Errors

### NotFound

Returned when no download manifest exists for the requested revision (or for the latest revision, if none was specified). For example, if hydration hasn't published a manifest yet, or the requested revision has aged out. Retry against the latest revision, or re-run without `optional_at_revision`.

[WatchPermissionSets]: https://buf.build/authzed/api/docs/v1.35.0:authzed.api.materialize.v0#authzed.api.materialize.v0.WatchPermissionSetsService.WatchPermissionSets
[LookupPermissionSets]: https://buf.build/authzed/api/docs/v1.35.0:authzed.api.materialize.v0#authzed.api.materialize.v0.WatchPermissionSetsService.LookupPermissionSets
[`DownloadPermissionSetsRequest`]: https://buf.build/authzed/api/file/main%3Aauthzed/api/materialize/v0/
[`DownloadPermissionSetsResponse`]: https://buf.build/authzed/api/file/main%3Aauthzed/api/materialize/v0/
[`File`]: https://buf.build/authzed/api/file/main%3Aauthzed/api/materialize/v0/
2 changes: 2 additions & 0 deletions app/materialize/api/lookup-permission-sets/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import { Callout } from "nextra/components";

# LookupPermissionSets

<FeatureBadge />

This API complements [WatchPermissionSets].
When you first bring on a system that needs permissions data, [LookupPermissionSets] lets you create an initial snapshot of the permissions data, and then you can use the [WatchPermissionSets] API to keep the snapshot updated.

Expand Down
2 changes: 2 additions & 0 deletions app/materialize/api/watch-permission-sets/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import { Callout } from "nextra/components";

# WatchPermissionSets

<FeatureBadge />

This is an update stream of all the permissions Materialize is configured to watch.
You can use this to store all permissions tracked in the system closer to your application database to be used in database-native ACL filtering.
Permissions can also be stored in secondary indexes like Elasticsearch.
Expand Down
4 changes: 3 additions & 1 deletion app/materialize/concepts/_meta.ts
Original file line number Diff line number Diff line change
@@ -1,6 +1,8 @@
export default {
"watched-permissions": "Watched Permissions",
snapshots: "Snapshots",
hydration: "Hydration",
"permission-sets": "Permission Sets",
"permission-set-lifecycle": "Permission Set Lifecycle",
snapshots: "Snapshots",
"managing-client-state": "Managing Client State",
};
26 changes: 26 additions & 0 deletions app/materialize/concepts/hydration/page.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,26 @@
---
title: "Hydration"
description: "Hydration is how Materialize turns your SpiceDB schema and relationships into precomputed permission sets, computing only what's reachable from the permissions you've configured it to watch."
---

# Hydration

<FeatureBadge />

**Hydration** is the process Materialize uses to turn your SpiceDB schema and relationship data into the precomputed [permission sets](./permission-sets) that both Materialize features depend on: [Accelerated Queries](../getting-started/overview#accelerated-queries) reads hydrated data directly to answer checks and lookups, and [Event Streams](../getting-started/overview#event-streams) exposes that same hydrated data through `LookupPermissionSets` and `WatchPermissionSets`.

## Only what's reachable

Hydration doesn't compute every permission your schema could produce, only the [permissions you've configured Materialize to watch](./watched-permissions), plus everything those permissions structurally depend on through unions, intersections, exclusions, and arrows. This **reachability** is a pure function of your schema and your watched permissions, so Materialize computes it once per hydration run and reuses it throughout.

This is also why the cost of hydration tracks the size of the reachable graph, not the size of your whole schema: watching a permission with a much wider reach, or adding a new watched permission, can noticeably change how much work hydration has to do.

## Offline hydration vs. online hydration

- **Offline hydration** reads a full [snapshot](./snapshots) of your relationship data and computes every reachable permission set from scratch. This is what backs the initial [backfill](./permission-set-lifecycle).
- **Online hydration** applies as new relationship and schema changes arrive after that initial pass, updating only the permission sets a given change actually affects, in the order the changes occurred.

## When rehydration is required

A [breaking schema change](./snapshots#snapshot-lifecycle-events) invalidates previously hydrated permission sets, since the reachability computed from the old schema no longer holds.
Materialize signals this so you can re-run the [backfill](./permission-set-lifecycle) at the new revision. See [The Permission Set Lifecycle](./permission-set-lifecycle) for the full rebuild flow.
2 changes: 2 additions & 0 deletions app/materialize/concepts/managing-client-state/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,8 @@ description: "State diagram of the transitions a client application moves throug

# Managing Client State

<FeatureBadge />

This diagram shows the various states your client application will need to transition through when calling the [LookupPermissionSets] and the [WatchPermissionSets] APIs.

![authzed-materialize](/images/materialize-client-state-diagram.png)
Expand Down
4 changes: 3 additions & 1 deletion app/materialize/concepts/permission-set-lifecycle/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,8 @@ import { Callout } from "nextra/components";

# The permission set lifecycle

<FeatureBadge />

The **permission set lifecycle** describes how your application's local copy of permission data is populated, kept current, and rebuilt over time — from an empty index to a complete, queryable [Permission Sets](./permission-sets) store, and back to a fresh index when your schema changes.

When you first bring a system online, you have no permission data locally.
Expand All @@ -16,7 +18,7 @@ Once the backfill is complete, you switch to [WatchPermissionSets](../api/watch-
## Stages of the lifecycle

1. **Backfill** — populate your store with the current state of every permission set. Two APIs can produce the initial backfill:
- **`DownloadPermissionSets`** (the Download API) — **recommended**. It ingests significantly faster than `LookupPermissionSets` and is the preferred path for the initial backfill.
- **[DownloadPermissionSets](../api/download-permission-sets)** (the Download API) — **recommended**. It ingests significantly faster than `LookupPermissionSets` and is the preferred path for the initial backfill.
- **[LookupPermissionSets](../api/lookup-permission-sets)** — kept available for clients that find it easier to integrate with. Page through the stream via cursors until an iteration yields zero events. Each event carries the permission-set data to store **and** the cursor to resume from on failure.
2. **Record the snapshot revision** — every backfill event includes the revision (`ZedToken`) the data was computed at. Store it transactionally alongside the data.
3. **Switch to the change stream** — once the backfill completes, open [WatchPermissionSets](../api/watch-permission-sets) using the stored snapshot revision so no changes are missed between the backfill and live updates.
Expand Down
48 changes: 46 additions & 2 deletions app/materialize/concepts/permission-sets/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,13 +7,57 @@ import { Callout } from "nextra/components";

# Permission Sets

<FeatureBadge />

A **permission set** is the unit of precomputed authorization data that Materialize produces.
Where SpiceDB answers a permission question on demand by walking the relationship graph, Materialize continuously **precomputes** the membership of the permissions you configure it to watch and exposes that denormalized data to your application.
Where SpiceDB answers a permission question on demand by walking the relationship graph, Materialize continuously [**hydrates**](./hydration) the membership of the [permissions you configure it to watch](./watched-permissions) and exposes that denormalized data to your application.

Conceptually, a permission set captures two kinds of edges:

- **Member → set** — a subject (e.g. `user:evan`) is a member of a permission set (e.g. `document:123#view`).
- **Set → set** — one set is nested inside another (e.g. `group:shared#member` grants `document:456#view`).
- **Set → set** — one set is nested inside another. The nested set can be a plain relation, like a group membership (`group:engineering#member` grants `document:456#view`), or another permission entirely, like an edit permission that also grants view access (`document:123#edit` grants `document:123#view`).

<details>
<summary>See the schema and relationships that produce the edges above</summary>

```zed
definition user {}

definition group {
relation member: user
}

definition document {
relation viewer: user | group#member
relation editor: user

permission edit = editor
permission view = viewer + edit
}
```

```zed
document:123#viewer@user:evan
document:123#editor@user:alex
document:456#viewer@group:engineering#member
```

- `document:123#viewer@user:evan` makes `user:evan` a **member** of the `document:123#view` permission set.
- `document:123#editor@user:alex` makes `user:alex` a member of `document:123#edit`. Because `view` is defined as `viewer + edit`, `document:123#edit` is nested in `document:123#view` — the **set → set** edge from the permission example above.
- `document:456#viewer@group:engineering#member` makes the `group:engineering#member` relation itself a viewer of `document:456`, nesting `group:engineering#member` inside `document:456#view` — the **set → set** edge from the group example above.

</details>

<Callout type="info">
Materialize (and the files it produces, see
[DownloadPermissionSets](../api/download-permission-sets#file-formats)) calls the nested set the
**child** and the set it grants into the **parent** — but this isn't a containment hierarchy like
a filesystem or an org chart, where a parent owns and cascades down to its children. It's closer
to set membership: being in the child set implies you're also in the parent set, so the grant
flows from child up to parent, not parent down to child. It's also not a strict tree — a single
child set can have many parents at once (e.g., one group's membership might grant view access to
many different documents), so don't assume a child has only one parent.
</Callout>

Together these let your application reconstruct "which resources can this subject access" — and the inverse — without issuing a `LookupResources` or `CheckPermission` call per request.
This is what makes authorization-aware search, sorting, and filtering over large result sets practical.
Expand Down
5 changes: 4 additions & 1 deletion app/materialize/concepts/snapshots/page.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,14 @@ import { Callout } from "nextra/components";

# Snapshots

<FeatureBadge />

A **snapshot** is a point-in-time, internally consistent view of every [Permission Set](./permission-sets) Materialize is tracking, computed at a specific SpiceDB revision.
Every piece of permission data Materialize hands you is anchored to the revision (`ZedToken`) it was computed at — that revision _is_ the snapshot's identity.

Snapshots are what make the two Materialize APIs compose safely:
Snapshots underpin both Materialize features: they're what Materialize [hydrates](./hydration) in order to answer Accelerated Queries, and they're what make the Event Streams APIs compose safely:

- Materialize reads the current snapshot directly to answer Accelerated Queries like `CheckPermission`, `CheckBulkPermissions`, `LookupResources`, and `LookupSubjects`.
- [The initial backfill](./permission-set-lifecycle) reads a snapshot in full.
- [WatchPermissionSets](../api/watch-permission-sets) advances that snapshot forward, delivering the deltas that move it from one revision to the next.

Expand Down
Loading
Loading