Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 9 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@ A repository of `provider` interface documents supporting [stackql](https://stac

StackQL provider interface documents inform the stackql application on how to interact with a given provider (like `aws`, `azure`, `google`, etc), including what methods are available in the provider and how to invoke these using SQL semantics. Provider interface documents are `yaml` formatted, OpenAPI specifications with extensions.

The documents are versioned per provider in this repository, and built as signed and compressed as packaged artifacts. The packaged artifacts are registered and published to the StackQL Provider Registry Artifact Repository in AWS S3. The provider registry API is a [Deno Deploy](https://deno.com/deploy) application that serves the provider interface documents to the stackql application using the `REGISTRY LIST` and `REGISTRY PULL` commands.
The documents are versioned per provider in this repository, and built as signed and compressed packaged artifacts. The packaged artifacts are registered and published to the StackQL Provider Registry Artifact Repository in AWS S3 (the master/archive store). The full docs tree is then mirrored to Cloudflare R2 and served at the edge by a Cloudflare Worker (source in [origin/](origin/)), which provides the provider interface documents to the stackql application using the `REGISTRY LIST` and `REGISTRY PULL` commands.

The following diagram shows the context of the provider registry:

Expand All @@ -20,26 +20,24 @@ C4Context
System_Ext(github_repo, "stackql-provider-registry", "GitHub Repository")
System_Ext(github_actions, "Build and Deploy", "GitHub Actions")
SystemDb(artifact_repo, "Artifact Repository", "AWS S3")
System(deno_registry, "Provider Registry API", "Deno Deploy")
SystemDb(r2_bucket, "Docs Mirror", "Cloudflare R2")
System(cf_worker, "Provider Registry Origin", "Cloudflare Worker")
System(stackql, "StackQL Application", "stackql")

Rel(github_repo, github_actions, "triggers...")
Rel(github_actions, artifact_repo, "registers and pushes to...", "signed tgz package")
Rel(github_actions, deno_registry, "pushes to...", "signed tgz package")
Rel(stackql, deno_registry, "list and pulls registry docs from...", "REGISTRY LIST | REGISTRY PULL")
Rel(github_actions, r2_bucket, "syncs docs tree to...")
Rel(cf_worker, r2_bucket, "reads provider docs from...")
Rel(stackql, cf_worker, "list and pulls registry docs from...", "REGISTRY LIST | REGISTRY PULL")
UpdateLayoutConfig($c4ShapeInRow="3", $c4BoundaryInRow="0")
UpdateRelStyle(github_repo, github_actions, $offsetY="10", $offsetX="-20")
UpdateRelStyle(github_actions, artifact_repo, $offsetY="44", $offsetX="-55")
UpdateRelStyle(github_actions, deno_registry, $offsetY="-18", $offsetX="-130")
UpdateRelStyle(stackql, deno_registry, $offsetY="40", $offsetX="-40")
```

The public StackQL Provider Registry is distributed via [Deno Deploy](https://deno.com/deploy), using the following endpoints:
The public StackQL Provider Registry is served from Cloudflare, using the following endpoints:

| Endpoint | Description |
| --- | --- |
| [registry.stackql.app](https://registry.stackql.app/ping) | Production registry (built from `main`) |
| [registry-dev.stackql.app](https://registry.stackql.app/ping) | Development registry (built from `develop`) |
| [registry-dev.stackql.app](https://registry-dev.stackql.app/ping) | Development registry (built from `dev`) |

## Contributing

Expand All @@ -53,7 +51,7 @@ Once you have an OpenAPI specification, you can use the [openapisaurus](https://

## Build and Deployment Workflow

The provider registry is built and deployed using GitHub Actions. Provider documents are validated and tested in workflow steps and then packaged and stored in the artifact repository. The most recent packaged versions are published to the registry API (a [Deno Deploy](https://deno.com/deploy) application), where they are available from the `stackql` application using `REGISTRY LIST` or `REGISTRY PULL`. See [docs/build-and-deployment.md](docs/build-and-deployment.md) for more information.
The provider registry is built and deployed using GitHub Actions. Provider documents are validated and tested in workflow steps and then packaged and stored in the artifact repository. The reconstructed docs tree is mirrored to Cloudflare R2 and served by the Cloudflare Worker origin, where the provider documents are available from the `stackql` application using `REGISTRY LIST` or `REGISTRY PULL`. See [docs/build-and-deployment.md](docs/build-and-deployment.md) for more information.

A separate workflow guards against providers being deleted from `providers/src` on any push; intentional removals require an explicit override in the commit message. See [provider delete guard](docs/build-and-deployment.md#provider-delete-guard) for details.

Expand Down
18 changes: 9 additions & 9 deletions origin/README.md
Original file line number Diff line number Diff line change
@@ -1,9 +1,7 @@
# StackQL Provider Registry origin (Cloudflare Worker)

Origin server for the public StackQL provider registry, served from Cloudflare
Workers + R2 (docs) + D1 (download analytics). This is the "green" origin in the
blue-green migration away from Deno Deploy. It preserves the existing URL
contract exactly:
Workers + R2 (docs) + D1 (download analytics). The URL contract is:

| Request | Response |
| ------------------------------------------ | ----------------------------------------------------- |
Expand All @@ -16,7 +14,7 @@ contract exactly:
| any non-GET method | 405 |

Docs are read from the R2 binding `REGISTRY_BUCKET` using the request path with
the leading slash stripped (the same layout the Deno origin read from disk).
the leading slash stripped as the object key (`providers/dist/...`).
Analytics are written one row per `.tgz` pull to the D1 binding `ANALYTICS_DB`
inside `ctx.waitUntil`, so logging never adds latency to a pull.

Expand All @@ -26,7 +24,7 @@ inside `ctx.waitUntil`, so logging never adds latency to a pull.
origin/
wrangler.toml two envs: dev (dev branch) and production (main branch)
schema.sql D1 downloads table + index
src/index.ts the Worker (port of deno-deploy-registry/website/index.ts)
src/index.ts the Worker
package.json wrangler + types
```

Expand Down Expand Up @@ -66,10 +64,12 @@ npm install
npx wrangler d1 execute stackql-registry-analytics-dev --local --file=./schema.sql

# seed a known object pair into the dev bucket so the endpoint checks pass
# (<docs-tree> is a local copy of the reconstructed registry docs tree, e.g. a
# `providers/dist` directory pulled from the artifact repository)
npx wrangler r2 object put stackql-provider-registry-dev/providers/dist/providers.yaml \
--file=../tmp/deno-deploy-registry/website/providers/dist/providers.yaml
--file=<docs-tree>/providers/dist/providers.yaml
npx wrangler r2 object put stackql-provider-registry-dev/providers/dist/aws/v0.1.3.tgz \
--file=../tmp/deno-deploy-registry/website/providers/dist/aws/v0.1.3.tgz
--file=<docs-tree>/providers/dist/aws/v0.1.3.tgz

npm run dev
```
Expand All @@ -88,8 +88,8 @@ curl -i http://localhost:8787/analytics/last24hours # 200 applicati
curl -i -X POST http://localhost:8787/ping # 405
```

Note: `localhost` Host headers are intentionally not logged to D1 (matches the
Deno origin). Test analytics writes against a deployed hostname.
Note: `localhost` Host headers are intentionally not logged to D1. Test
analytics writes against a deployed hostname.

## Deploy

Expand Down
9 changes: 4 additions & 5 deletions origin/src/index.ts
Original file line number Diff line number Diff line change
@@ -1,8 +1,7 @@
/**
* StackQL Provider Registry origin - Cloudflare Worker (green).
* StackQL Provider Registry origin - Cloudflare Worker.
*
* Port of the Deno Deploy origin (deno-deploy-registry/website/index.ts).
* The URL contract is preserved exactly:
* The URL contract:
*
* GET (anything).tgz -> 200 application/gzip, log one download event
* GET (anything)providers.yaml -> 200 text/plain, not logged
Expand Down Expand Up @@ -31,7 +30,7 @@ interface RequestMetadata {

function extractRequestMetadata(request: Request): RequestMetadata {
return {
// Deno used conn.remoteAddr.hostname; on Cloudflare the real client IP is here.
// On Cloudflare the real client IP is in the CF-Connecting-IP header.
ipAddr: request.headers.get('CF-Connecting-IP') || '',
ts: new Date().toISOString(),
userAgent: request.headers.get('user-agent') || '',
Expand Down Expand Up @@ -362,7 +361,7 @@ async function handleRequest(request: Request, env: Env, ctx: ExecutionContext):
});
}

// R2 key mirrors the Deno on-disk layout: `.${pathname}` -> strip the leading slash
// R2 key is the request path with the leading slash stripped (`providers/dist/...`)
const key = pathname.replace(/^\//, '');

const obj = await env.REGISTRY_BUCKET.get(key);
Expand Down
8 changes: 4 additions & 4 deletions origin/wrangler.toml
Original file line number Diff line number Diff line change
@@ -1,9 +1,9 @@
#
# StackQL Provider Registry origin Worker (green).
# StackQL Provider Registry origin Worker.
#
# Two environments mirror the existing dev/prod Deno Deploy split:
# - `dev` -> deployed from the `dev` branch (cutover host: registry-dev.stackql.app)
# - `production` -> deployed from the `main` branch (cutover host: registry.stackql.app)
# Two environments:
# - `dev` -> deployed from the `dev` branch (host: registry-dev.stackql.app)
# - `production` -> deployed from the `main` branch (host: registry.stackql.app)
#
# Named environments do NOT inherit top-level bindings, so each environment
# declares its own R2 + D1 bindings explicitly. The top-level block below is
Expand Down
138 changes: 107 additions & 31 deletions providers/src/deno/v00.00.00000/provider.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,52 +2,128 @@ id: deno
name: deno
version: v00.00.00000
providerServices:
database:
id: database:v00.00.00000
name: database
apps:
id: apps:v00.00.00000
name: apps
preferred: true
service:
$ref: deno/v00.00.00000/services/database.yaml
title: database API
$ref: deno/v00.00.00000/services/apps.yaml
title: apps API
version: v00.00.00000
description: Operations about databases
deployment:
id: deployment:v00.00.00000
name: deployment
description: >-
An app is the top-level container for a deployable application.

Apps have configuration (build settings, environment variables), can
reference layers for shared config, and contain revisions (deployments).


**Key characteristics:**


- Identified by UUID or human-readable slug. App slugs must be 3–32
characters long, may contain only lowercase letters, numbers, and hyphens,
cannot contain underscores, must not start or end with a hyphen, must not
have consecutive hyphens in positions 3 and 4, and cannot be a reserved
slug. App IDs are UUIDs.

- Support up to 5 labels for filtering and grouping

- Reference layers via `layers` array for inherited configuration

- Have app-specific `env_vars` that override layer values

- Have a `config` that provides defaults for revisions
databases:
id: databases:v00.00.00000
name: databases
preferred: true
service:
$ref: deno/v00.00.00000/services/deployment.yaml
title: deployment API
$ref: deno/v00.00.00000/services/databases.yaml
title: databases API
version: v00.00.00000
description: Operations about deployments
domain:
id: domain:v00.00.00000
name: domain
description: >-
Create database instances (BYO Postgres, Deno KV, Prisma) and bind the
manual databases hosted on them. Manual databases are bound to one of an
app's timelines (`production`, `preview`, etc.) and coexist with the
per-timeline databases provisioned by default. Use the deploy endpoint's
`databases` parameter to bind a specific manual database to an individual
revision without affecting other revisions on the timeline.
domains:
id: domains:v00.00.00000
name: domains
preferred: true
service:
$ref: deno/v00.00.00000/services/domain.yaml
title: domain API
$ref: deno/v00.00.00000/services/domains.yaml
title: domains API
version: v00.00.00000
description: Operations about domains
organization:
id: organization:v00.00.00000
name: organization
description: >-
A domain is a hostname (apex or wildcard) owned by an organization. Once
registered, a domain must be verified via a DNS-published
`_acme-challenge.<domain>` token, then receives a TLS certificate
(uploaded manually or provisioned via ACME).


**Lifecycle:**


1. `POST /domains` — register the domain and receive the DNS records to
publish.

2. `POST /domains/{domainId}/verify` — re-runs DNS verification once
records propagate.

3. Either `POST /domains/{domainId}/certificates` (manual) or `POST
/domains/{domainId}/certificates/provision` (automatic ACME).

4. Attach to revisions via deploy or per-revision endpoints.
layers:
id: layers:v00.00.00000
name: layers
preferred: true
service:
$ref: deno/v00.00.00000/services/organization.yaml
title: organization API
$ref: deno/v00.00.00000/services/layers.yaml
title: layers API
version: v00.00.00000
description: Operations about organizations
project:
id: project:v00.00.00000
name: project
description: >-
A layer is a mutable configuration object that can be shared across
multiple apps. Layers provide the solution for bulk environment variable
management: instead of updating thousands of apps individually, you create
a layer, attach it to apps, then update the layer once.


**Key characteristics:**


- Organization-scoped and identified by ID or slug

- Contain environment variables

- Can include other layers (base layers) for hierarchical configuration

- Apps reference layers in their `layers` array

- Updating a layer is O(1) regardless of how many apps reference it

- Layer updates cause running isolates to restart but do not require
redeployment
revisions:
id: revisions:v00.00.00000
name: revisions
preferred: true
service:
$ref: deno/v00.00.00000/services/project.yaml
title: project API
$ref: deno/v00.00.00000/services/revisions.yaml
title: revisions API
version: v00.00.00000
description: Operations about projects
description: >-
A revision represents a specific build and deployment of an app. Revisions
are immutable once created — to make changes, you create a new revision.
The only mutable property is `retention` (enterprise opt-in), a
garbage-collection policy of `auto` or `indefinite`.


Status lifecycle: `queued` → `building` → `succeeded` (success), `queued`
→ `failed` (build error, cancelled, or timeout), or `queued` → `skipped`.
config:
auth:
credentialsenvvar: DENO_DEPLOY_TOKEN
type: bearer
credentialsenvvar: DENO_DEPLOY_TOKEN
Loading
Loading