Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 20 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -190,6 +190,26 @@ W3DS_EXTRA_RESERVED_USERNAMES=""
# Minimum eID Wallet version accepted. Temporary - drops out after the rollout.
W3DS_MIN_WALLET_VERSION="0.4.0"

# Forgejo code sync (services/forgejo-code-sync)
# Syncs commits pushed to GitW3 into the pushing author's eVault, resolved via
# the login_name the bridge above writes on sign-in. See
# docs/superpowers/specs/2026-08-14-forgejo-code-sync-design.md.
FORGEJO_SYNC_PUBLIC_URL="http://localhost:4300"
FORGEJO_SYNC_PORT=4300
# HMAC secret configured on the Forgejo system webhook (POST /api/v1/admin/hooks).
# Compared against the raw, unprefixed X-Forgejo-Signature header.
FORGEJO_WEBHOOK_SECRET="replace-with-a-strong-secret"
# GitW3's base URL, for the admin Users API call and diff fetching.
FORGEJO_API_URL="http://localhost:3080"
# PAT on a dedicated site-admin service account created for this service alone
# (never a shared human admin's token) - scopes read:user,read:repository.
# read:user alone is not enough: GitW3 only returns login_name to a caller whose
# account has IsAdmin=true, regardless of token scope. See the spec's Trust model.
FORGEJO_ADMIN_TOKEN=""
# Commit diffs are uploaded to the DO_SPACES_* bucket configured above (the
# same one evault-core's StorageService.ts uses) rather than inlined into the
# eVault write - no service-specific S3 keys needed here, just those.

# --- Deploying GitW3 and the bridge together -------------------------------
# Only needed for docker-compose.gitw3.yml. Local development uses the block
# above and runs the bridge with `pnpm --filter w3ds-oidc-bridge dev`.
Expand Down
42 changes: 42 additions & 0 deletions docker/Dockerfile.forgejo-code-sync
Original file line number Diff line number Diff line change
@@ -0,0 +1,42 @@
FROM node:20-alpine AS base
RUN apk add --no-cache libc6-compat python3 make g++
WORKDIR /app

ENV CI=true
ENV PYTHON=/usr/bin/python3
RUN ln -sf python3 /usr/bin/python

# ---
FROM base AS prepare
RUN npm install -g pnpm@10.25.0 turbo@^2
COPY . .
RUN turbo prune forgejo-code-sync --docker

# ---
FROM base AS builder
RUN npm install -g pnpm@10.25.0
# Dependencies first, since they change far less often than the source.
COPY --from=prepare /app/out/json/ .
RUN pnpm install --frozen-lockfile
COPY --from=prepare /app/out/full/ .
RUN pnpm turbo build --filter=forgejo-code-sync

# ---
FROM base AS runner
COPY --from=builder /app/package.json ./
COPY --from=builder /app/pnpm-workspace.yaml ./
COPY --from=builder /app/pnpm-lock.yaml ./

COPY --from=builder /app/services/forgejo-code-sync/dist ./services/forgejo-code-sync/dist
COPY --from=builder /app/services/forgejo-code-sync/package.json ./services/forgejo-code-sync/
COPY --from=builder /app/services/forgejo-code-sync/node_modules ./services/forgejo-code-sync/node_modules
COPY --from=builder /app/node_modules ./node_modules

WORKDIR /app/services/forgejo-code-sync

# Keep in step with FORGEJO_SYNC_PORT.
EXPOSE 4300
HEALTHCHECK --interval=30s --timeout=3s --start-period=10s --retries=3 \
CMD node -e "require('http').get('http://localhost:4300/healthz', (r) => {process.exit(r.statusCode === 200 ? 0 : 1)}).on('error', () => process.exit(1))"

CMD ["node", "dist/index.js"]
349 changes: 349 additions & 0 deletions docs/superpowers/plans/2026-08-14-forgejo-code-sync-plan.md

Large diffs are not rendered by default.

810 changes: 810 additions & 0 deletions docs/superpowers/specs/2026-08-14-forgejo-code-sync-design.md

Large diffs are not rendered by default.

176 changes: 135 additions & 41 deletions pnpm-lock.yaml

Large diffs are not rendered by default.

3 changes: 3 additions & 0 deletions services/forgejo-code-sync/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
.queue/
.queue-snapshots/
.repo-envelopes/
129 changes: 129 additions & 0 deletions services/forgejo-code-sync/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,129 @@
# forgejo-code-sync

Syncs commits pushed to GitW3 into the pushing author's own eVault, and keeps a full up-to-date copy of the repo in
its **owner's** own eVault - two independent sync paths off the same webhook, see [What gets synced](#what-gets-synced).

**Design:** [docs/superpowers/specs/2026-08-14-forgejo-code-sync-design.md](../../docs/superpowers/specs/2026-08-14-forgejo-code-sync-design.md)
**Plan:** [docs/superpowers/plans/2026-08-14-forgejo-code-sync-plan.md](../../docs/superpowers/plans/2026-08-14-forgejo-code-sync-plan.md)

## The two contracts

**Forgejo side** is a system webhook - configured once, instance-wide, no per-repo setup - delivering every `push`
event. `pusher.login` identifies the authenticated account that ran `git push`; each commit's own `author`/`committer`
fields are free-text `git config`, never validated against any Forgejo account, and this service never resolves
identity from them. The webhook's `X-Forgejo-Signature` header is a raw, unprefixed HMAC-SHA256 hex digest over the
exact request bytes - not the GitHub-style `sha256=`-prefixed `X-Hub-Signature-256` Forgejo sends alongside it for
compatibility.

**eVault side** is the same certify-then-per-eName-GraphQL pattern used by
[`platforms/calendar/api/src/services/EVaultService.ts`](../../platforms/calendar/api/src/services/EVaultService.ts):
this service certifies itself with the Registry once, then writes into *the pusher's own* eVault by presenting their
eName as `X-ENAME` on each write.

The identity link between the two is the [w3ds-oidc-bridge](../w3ds-oidc-bridge/README.md): once someone signs into
GitW3 through it, GitW3's `login_name` for that account holds their full eName (`@` included). This service resolves
`pusher.login -> login_name -> eName` via `GET /api/v1/users/{username}`, which only returns `login_name` to an
admin-authenticated caller - see [Configuration](#configuration) for what that requires of `FORGEJO_ADMIN_TOKEN`.

## What gets synced

One MetaEnvelope per commit (`services/ontology/schemas/codeCommit.json`), written into the pusher's eVault with an
`acl` that mirrors the source repository's visibility at push time: `["*"]` for a public repo, owner-only for a
private one. A commit's diff is inlined when it fits under `FORGEJO_SYNC_DIFF_MAX_BYTES`, and replaced with a
`diffUrl` pointer back to GitW3 otherwise - or on any fetch failure, so an oversized or momentarily unreachable diff
never blocks the commit's own metadata from being synced.

A push from an account with no linked eVault (`login_name` doesn't start with `@`) is skipped silently - the ordinary
case for most GitW3 accounts, not a failure.

**A second, independent path off the same webhook** writes the repo's complete current state - every file and
folder, via GitW3's archive endpoint, uploaded to S3 - into the **owner's** own eVault
(`services/ontology/schemas/repoSnapshot.json`), replaced in place on every push rather than accumulating one
envelope per push. `repository.owner.login`, not `pusher.login`, is resolved the same way (`IdentityResolver` is
generic over any Forgejo username); an org-owned repo or an owner with no linked eVault skips the same way an
unlinked pusher does. See the spec's [Repo-owner full snapshot](../../docs/superpowers/specs/2026-08-14-forgejo-code-sync-design.md#repo-owner-full-snapshot-added-2026-08-15)
section for the full design and live-verification detail.

## Delivery reliability

Forgejo has no automatic retry or redelivery of failed webhook deliveries at all - confirmed against GitW3's own
`services/webhook/deliver.go`. So this service owns its own reliability: every commit is durably queued to disk
(`.queue/` locally - see [Deployment](#deployment)) before the webhook handler responds, and a failed sync is retried
with exponential backoff rather than dropped. A task that exhausts its retry budget is left on disk in an `exhausted`
status - logged distinctly from an ordinary skip - rather than silently removed, since it needs a human.

The repo-owner snapshot sync has its own independent queue (`.queue-snapshots/`), same reliability discipline, same
skip/retry/exhausted semantics - a slow or down eVault delays the owner's snapshot the same way it delays a pusher's
commit, and neither queue's failure affects the other's.

## Configuration

Read from the repository root `.env`, same `required()`-throws-at-startup pattern as the bridge's own `config.ts`.

| Variable | Default | Note |
|---|---|---|
| `FORGEJO_SYNC_PUBLIC_URL` | - | this service's own base URL, used for Registry platform certification and to build the webhook URL registered on GitW3 |
| `FORGEJO_SYNC_PORT` | `4300` | |
| `FORGEJO_WEBHOOK_SECRET` | - | HMAC secret configured on the Forgejo system webhook |
| `FORGEJO_API_URL` | - | GitW3's base URL |
| `FORGEJO_ADMIN_TOKEN` | - | PAT on a **dedicated site-admin service account**, scopes `read:user,read:repository` - see below |
| `FORGEJO_SYNC_DIFF_MAX_BYTES` | `131072` | inline cap before falling back to `diffUrl` |
| `PUBLIC_REGISTRY_URL` | - | already in the root `.env` |
| `PUBLIC_EVAULT_SERVER_URI` | - | already in the root `.env` |

### Why the admin token has to be this big

`GET /api/v1/users/{username}` only returns `login_name` when the caller is the account itself or a site admin -
scope alone doesn't gate it, confirmed against `services/convert/user.go`'s `toUser`. Fetching a private repo's diff
separately requires `read:repository`. So `FORGEJO_ADMIN_TOKEN` has to be a PAT belonging to an actual site-admin
account, not merely one carrying those scopes - and because that token can read every account's `login_name` and
every repo's content, not just what this service needs at a given moment, it should be a service account created for
this purpose alone, never a shared human admin's personal token. See the spec's Trust model for the full reasoning,
including the accepted limitation this implies for the ACL decision below.

## Running locally

```bash
pnpm --filter forgejo-code-sync dev
```

Then register the webhook against a local GitW3 instance (idempotent - safe to re-run):

```bash
pnpm --filter forgejo-code-sync register-webhook
```

`GET /healthz` returns `200` once the service is up. Pushing to a repo whose owner has signed into that GitW3 through
the bridge should produce a `codeCommit` MetaEnvelope in their eVault within one drain cycle (5s).

## Testing

```bash
pnpm --filter forgejo-code-sync test
```

Everything through identity resolution, signature verification, ACL derivation, the retry queue, and the drain loop
is covered without a live GitW3, eVault, or Registry - every external call is stubbed. What isn't covered by
automated tests: a real end-to-end run against a live GitW3 + bridge + eVault, which needs the same manual walkthrough
the bridge's own README describes for testing without a phone, extended by pushing a commit as the final step - see
the plan's Phase 6.

**Live verification, done three times, not just planned.** See the spec's Testing section for all three passes: the
first live push (public/private repo, unlinked-account skip, single-webhook regression, plus two GitW3
webhook-provisioning traps found only by testing); the 2026-08-15 follow-up that closed the one gap the first pass
left open - a real differential proof that a private-repo diff's S3 object is anonymously unreachable while a public
one isn't, plus a fourth webhook trap (`PATCH /admin/hooks/{id}` silently ignores a secret rotation - fixed in
`scripts/register-webhook.ts`, which now deletes and recreates instead of patching); and the same day's repo-owner
snapshot pass - the archive endpoint found and live-verified before any code was written against it, a real two-push
sequence proving update-in-place (same envelope id, S3 object content changed), the same S3 ACL differential proof
applied to a repo archive, and the org/unlinked-owner skip path exercised live.

## Deployment

Inherits the bridge's own unresolved blocker: **no service deployment manifest exists in this repository for a
production or staging host.** See the spec's Deployment section. `docker/Dockerfile.forgejo-code-sync` follows the
`docker/Dockerfile.<name>` convention once a host is known.

Where the retry queue persists in a real deployment is part of that same open question - the local default is a
`.queue/` directory next to the package (gitignored), which is enough for development but not a production storage
decision.
33 changes: 33 additions & 0 deletions services/forgejo-code-sync/package.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
{
"name": "forgejo-code-sync",
"version": "0.1.0",
"description": "Syncs commits pushed to GitW3 into the pushing author's eVault, resolved via the identity the w3ds-oidc-bridge links on sign-in",
"type": "module",
"private": true,
"main": "./dist/index.js",
"scripts": {
"dev": "tsx watch src/index.ts",
"build": "tsc -p tsconfig.build.json",
"start": "node dist/index.js",
"register-webhook": "tsx scripts/register-webhook.ts",
"test": "vitest run",
"test:watch": "vitest",
"check": "npx @biomejs/biome check ./src && tsc --noEmit",
"check-format": "npx @biomejs/biome format ./src",
"check-lint": "npx @biomejs/biome lint ./src",
"check-types": "tsc --noEmit"
},
"dependencies": {
"@aws-sdk/client-s3": "^3.700.0",
"dotenv": "^16.4.5",
"express": "^4.18.2",
"graphql-request": "^6.1.0"
},
"devDependencies": {
"@types/express": "^4.17.21",
"@types/node": "^20.11.24",
"tsx": "^4.7.1",
"typescript": "^5.3.3",
"vitest": "^3.0.9"
}
}
Loading
Loading