Skip to content

feat: add portable archive export and restore - #1561

Open
larry-zy wants to merge 28 commits into
oceanbase:masterfrom
larry-zy:feat/1421-portable-export-backup-restore
Open

larry-zy wants to merge 28 commits into
oceanbase:masterfrom
larry-zy:feat/1421-portable-export-backup-restore

Conversation

@larry-zy

@larry-zy larry-zy commented Sep 10, 2026

Copy link
Copy Markdown
Contributor

Closes #1421

Rationale for this change

PowerContext needs a backend-neutral logical archive for disaster recovery, offline transfer, customer-controlled backup, and SQLite/OceanBase migration without losing immutable history, lineage, review decisions, or Handoff evidence. Database-native backup remains a separate operational recovery mechanism.

What changes are included in this PR?

  • Adds a versioned .pcb contract with deterministic manifests, producer/schema versions, per-record SHA-256 digests, an aggregate checksum, deterministic ZIP output, and optional compression.
  • Streams scope-selective export, inspect, target-aware dry-run validation, and transactional/idempotent restore with content-free progress.
  • Preserves Scope relationships, Sources and journal positions, immutable Artifact Revisions, lineage and cross-Scope publication provenance, Memory versions/heads, Candidate decisions, Work Sources, Handoffs, and receipts.
  • Requires export authorization before database enumeration, rejects credentials/provider secrets/absolute host paths, and excludes access bindings, audit/usage/evaluation data, host-local state, cursors, and projections.
  • Adds immutable conflict detection, rollback-safe restore, durable readiness receipts, and Memory/Experience/Skill projection rebuilds.
  • Supports configured SQLite, SeekDB, and OceanBase deployments through the local CLI; includes an opt-in live SQLite-to-OceanBase acceptance test.
  • Documents format compatibility, recovery workflow, exclusions, and distinct SQLite/OceanBase native backup guidance.

Are there any user-facing changes?

Adds powercontext archive export, powercontext archive inspect, and powercontext archive restore. Restore writes require --yes; --dry-run performs target compatibility and conflict checks without domain writes. This introduces portable bundle format version 1 and the pc_portable_restore_receipts table. No existing API or bundle format is changed.

How was this change tested?

  • prek run -a
  • ty check
  • Focused archive/schema/CLI/E2E suite: 22 passed
  • make docs-test: 792 public pages and internal links verified
  • Full macOS suite: 2471 passed and 101 skipped; the remaining three tests are Linux systemd-only platform guards

AI usage statement

Codex (GPT-5) was used to assist implementation, tests, documentation, and review; all changes were reviewed and validated locally.

…r and Client

The ready-to-run Server and the primary Python Client did not enforce the
network safety policy already applied by the Agent integrations, allowing two
unsafe transports:

- ServerSettings accepted a non-loopback bind (e.g. 0.0.0.0) while bearer
  authentication was disabled, silently exposing an unauthenticated Server.
- ClientSettings / PowerContextClient accepted a non-loopback http:// URL and
  would send a bearer token over plaintext.

Introduce a single shared transport-policy module (powercontext.transport)
and consume it everywhere:

- ClientSettings rejects non-loopback plaintext http:// Server URLs.
- PowerContextClient refuses to send a bearer token over an unencrypted
  non-loopback connection.
- ServerSettings rejects an unauthenticated non-loopback bind unless the
  operator opts in via POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK
  (for TLS-terminated / controlled-network deployments).
- The `server run` CLI applies the same guard, since model_copy bypasses the
  settings validator on --host overrides.

Add a cross-surface contract test suite and update the EN/ZH configuration
reference.

Closes oceanbase#1319
# Conflicts:
#	docs/en/docs/reference/configuration.md
#	docs/zh/docs/reference/configuration.md
#	src/powercontext/server/settings.py
Load the Codex plugin's settings module by path and assert its private
loopback host set and _http_base_url behavior match the shared transport
contract, so the isolated plugin copy cannot silently drift. Tighten the
configuration docs to note the plaintext-URL rule applies to configured
Server URLs.
The loopback check only matched the three named hosts 127.0.0.1, ::1 and
localhost, so plaintext HTTP to any other 127.0.0.0/8 address (e.g.
127.0.0.2) was wrongly rejected as non-loopback. Detect loopback via
ipaddress.ip_address(...).is_loopback, which covers the whole IPv4
loopback block, and route the Client CLI through the shared helper.

Both agent plugins ship isolated and cannot import powercontext, so each
vendors its own copy of the check; parametrised drift-guard tests pin both
copies to the shared contract. Surface ServerSettings validation errors as
a typer.BadParameter instead of a raw traceback.
…ports

The bearer-token-over-plaintext guard is only meaningful for the transport
the client opens itself, where base_url's scheme reflects what crosses the
wire. A caller-supplied http_client owns its own transport (ASGI in-process,
a Unix socket, a TLS-terminating proxy -- all carrying an http:// label), so
the scheme is no longer a reliable signal and enforcing it only produced
false positives, breaking the authenticated in-memory e2e flow. Enforce the
guard only when the client creates the transport.
The Docker image binds POWERCONTEXT_SERVER_HTTP_HOST=0.0.0.0 so the port is
reachable from the harness container, but the unauthenticated-non-loopback
guard now refuses that bind and the Server fails to start. The Compose
network is isolated, so opt in via
POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK to restore startup.
 review

- client: gate the plaintext-token guard on an explicit trust_transport_security
  opt-in; supplying an http_client is no longer treated as evidence the transport
  is safe, so a bearer token is still refused over unencrypted non-loopback HTTP.
- docker: declare POWERCONTEXT_SERVER_ALLOW_UNAUTHENTICATED_NON_LOOPBACK in the
  image itself so the documented `docker run` starts out of the box, document the
  network-exposure trade-off, and drop the masking env from the e2e compose so it
  exercises the real image contract; add a Dockerfile smoke test.
- server cli: merge --host/--port before validation so a safe override can repair
  an unsafe environment bind, and translate the actionable settings failures
  (unauthenticated non-loopback bind, missing bearer token) into friendly CLI
  errors recognised by exception identity rather than raw validation text.
- cli: unpack the ServerSettings kwargs so the partial ``http`` mapping is not
  type-checked against the full HttpConfig annotation (ty invalid-argument-type).
- tests: pin a wide COLUMNS in the friendly-error tests so the rich panel does
  not hard-wrap the asserted --host / env-var tokens on a narrow CI terminal.
…ntract

Pi's isLoopback only accepted localhost, 127.0.0.1 and [::1], so its production
resolveConfig rejected http://127.0.0.2:8000 as "must use HTTPS outside loopback"
while the shared policy and the Python plugins already trust the whole 127.0.0.0/8
block. Rewrite it to mirror is_loopback_host (strip brackets, lowercase, accept
127.0.0.0/8, localhost and ::1).

Add a Pi Vitest drift guard and a JSON fixture of loopback/non-loopback host
vectors that both the Python drift guard and the Pi suite consume, so a future
divergence from the shared contract fails in at least one language.
…error assertions

pretty-format-json rewrites the fixture with one array element per line; apply that
formatting so the quality hook passes in CI.

The friendly-error CLI tests asserted tokens (--host, the opt-in env var) against
typer's rich error panel. Under GitHub Actions typer forces force_terminal=True and
the runner's TERM=dumb pins rich to 80 columns while ignoring COLUMNS, so the panel
hyphen-breaks --host and force-splits the long env var. Replace the COLUMNS override
with a fixture that neutralises the forced terminal and sets an explicit wide width,
making the assertions independent of the terminal environment.
…hes for the transport

The loopback guard only tripped when a bearer token was present, so an
unauthenticated client would still ship Memory content -- carried in the request
body -- over unencrypted non-loopback HTTP. Broaden the guard to reject *any*
plaintext non-loopback request, and let a caller that knows its transport is
secure opt in explicitly with `trust_transport_security=True`; supplying an
`http_client` is not itself evidence of safety.

Thread that opt-in through the LangGraph adapter's `shared_http_client` /
`open_client`: a plain pooling client stays untrusted, while in-process ASGI
harnesses declare trust. Update the in-process test harnesses accordingly.

Also stop the remote-access guides from documenting a bare
`server run --host 0.0.0.0`, which the bind policy now rejects; show the
authenticated bind and the explicit unauthenticated opt-in instead, and add a
command-level contract test so a copy-pasteable command that fails to start
fails in CI rather than only for a reader.
Resolve tests/e2e/test_runtime_server.py: upstream dropped the explicit
handoff-report project/workstream setup in the delegation-loop e2e; keep that
refactor and re-apply this branch's `trust_transport_security=True` on the
in-process ASGI client.
The hardened client guard now refuses plaintext non-loopback URLs
unconditionally, which broke the e2e harness where agent containers
reach the server over the private Compose bridge via host-gateway.

Add an explicit POWERCONTEXT_BUB_TRUST_TRANSPORT_SECURITY opt-in
(default false) that routes through the client's endorsed bypass: a
caller-supplied httpx transport plus trust_transport_security=True.
Enable it only in the e2e Compose harness, mirror the server-side
ALLOW_UNAUTHENTICATED_NON_LOOPBACK opt-in shape, document the switch,
and cover both the refusal default and the trusted construction.
…t-policy

# Conflicts:
#	e2e/bub/uv.lock
#	integrations/bub/pyproject.toml
#	integrations/bub/src/powercontext_bub/plugin.py
…t policy

The LangChain middleware and pydantic-ai tests merged from master were
written before the client started refusing all plaintext non-loopback
URLs, so their in-process ASGI transports now hit the guard.

Port the langgraph-style trust plumbing to the LangChain shared HTTP
client and let the affected tests vouch for their ASGI transports with
trust_transport_security=True.
The plugin hooks honoured POWERCONTEXT_BUB_TRUST_TRANSPORT_SECURITY, but the
powercontext.search/remember/context tools still constructed bare clients, so
the acceptance harness failed on the plaintext host-gateway hop. Hoist the
vouched construction into a module-level open_client helper, carry the flag in
the per-turn tool settings, and cover both tool paths in the trust tests.
…k requests

The reference still described the pre-hardening guard, which only refused to
attach a bearer token. Align both language variants with the shipped behaviour:
every request over unencrypted non-loopback HTTP is refused, and callers whose
http:// base URL merely labels a secure transport must supply their own
http_client with an explicit trust_transport_security=True.
@Teingi

Teingi commented Sep 10, 2026

Copy link
Copy Markdown
Member

resolve conflicts

@larry-zy
larry-zy force-pushed the feat/1421-portable-export-backup-restore branch from ea1688e to fc1521a Compare September 11, 2026 13:26
@larry-zy
larry-zy force-pushed the feat/1421-portable-export-backup-restore branch from fc1521a to 3c0e6fb Compare September 11, 2026 19:27
@larry-zy
larry-zy marked this pull request as ready for review September 11, 2026 19:28
@larry-zy larry-zy changed the title feat: add local portable archive CLI feat: add portable archive export and restore Sep 11, 2026

@Teingi Teingi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed bffe534. The focused archive suite passed (16 tests; the large-bundle memory test was excluded). I reproduced the six issues below in isolated SQLite databases; vector coverage used a deterministic embedding model. I did not run live OceanBase validation.

Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/builtin/runtime/relational.py Outdated
for scope_id in scope_ids:
services = self._services_for(scope_id)
_, catalog = services.sources()
await services.memory(catalog).rebuild_projections()

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Rebuild vectors before reporting readiness

rebuild_projections() defaults its embedding model to None; it does not use the model configured on the Memory service. Restoring into a vector-enabled target therefore reports ready, while explicit vector search raises CapabilityNotSupportedError and auto falls back to FTS. Please pass the target embedding model and verify vector completeness before marking the restore ready.

Comment thread src/powercontext/builtin/portability/bundle.py
Comment thread src/powercontext/cli/archive.py Outdated

@Teingi Teingi left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed 9f10299. The focused suite passed (46 passed, 6 skipped; the large-bundle test was excluded), and Ruff passed. The previous fixes passed their relevant SQLite regressions. I reproduced the two issues below in isolated SQLite and local seekdb 1.3.0; I did not validate a remote OceanBase server.

# AsyncDatabase has already issued START TRANSACTION for MySQL-mode
# engines. Replacing that empty transaction with a consistent snapshot
# is safe because authorization has completed and no query has run yet.
await connection.exec_driver_sql("START TRANSACTION WITH CONSISTENT SNAPSHOT")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Establish a transaction-wide snapshot for MySQL-mode exports

On local seekdb 1.3.0, the default isolation is READ-COMMITTED, and this statement does not keep subsequent reads on one snapshot. Committing a second Source after the journal-head read produced an archive with head 1 and Source positions [1, 2]. Export returned exported, but validation against a clean target failed with source journal snapshot is inconsistent. The final database check sees the newer head and misses the inconsistency already written to the archive. The same probe with REPEATABLE READ produced a valid archive. Please establish the required isolation before reading records and add a concurrent-writer regression for this backend.

configured = (
self._supported_source_types if supported_source_types is None else frozenset(supported_source_types)
)
missing_sources = () if configured is None else tuple(sorted(required_sources - configured))

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Allow portable SourceObservations without a local adapter

This check treats every stored Source type as requiring an entry in the target Python registry, but worker-materialized SourceObservation records are readable without that adapter. I registered a remote note definition, submitted an observation through the public API, and exported it successfully. The same-version target returned compatible=False with unsupported_source_types=("note",) and rejected restore, even after registering the same manifest there. Restoring through the generic bundle service succeeded, and the normal repository read returned the original observation. Please distinguish native Sources from observations during compatibility checks and preserve the required definition metadata.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat: add portable export, backup, and restore

3 participants