Blimp prod-source kit

Blimp

Blimp is an ACID cache and autonomous materialized view — an efficient per-core cache engine that keeps your CPU/GPU utilization high and AI/ML context fresh with delta change queries in 1-2s. Launch a scalable Blimp node on any server or cloud instance to optimize your existing pipeline → blimp.software

run-blimp connects a Blimp node to your application, which can use it for cache or query engine within your environment.

   ┌───── your node (any cloud) ─────┐     ┌─────── Blimp node ───────┐
   │  blimp CLI                      │     │  gateway                 │
   │  REST catalog  :8181  ──────────┼────▶│   S3 :9000               │
   │  your S3 / MinIO data           │ wire│   eblobbers              │
   └───────────────┬─────────────────┘     └───────────┬──────────────┘
                   └────── snapshot_changed webhook ─────┘
                          (your pipeline, on each commit)

If your node is already networked to the Blimp node (e.g. the same private network/account) — vpc mode — --setup reaches the gateway on its private IP using the node's own identity, no keys to type. Otherwise — external mode — it reaches the gateway over public DNS with explicit S3 keys. blimp --setup detects which and wires it accordingly.

Both paths are supported and proven by this kit for demo/test setup — pick whichever matches where your node already lives. For production, run that node on the same private network as the Blimp node (same account/ region, ideally the same zone): that path uses private IPs with no S3 keys, and every S3 call to the origin stays off the public internet. A different network, account, or cloud works too, but adds network hops and (outside the same account/region) real data-transfer egress cost on every read — fine for a demo or a one-off test, but not the recommended production topology.

Install

blimp is a self-contained CLI — install it once, then it bootstraps its own prerequisites on first --setup. Two supported ways to get it:

# 1. one-line installer (open repo, no Docker) — puts `blimp` on your PATH
curl -fsSL https://raw.githubusercontent.com/0chain/run-blimp/main/install.sh | sh

# 2. or just clone and run in place
git clone https://github.com/0chain/run-blimp && cd run-blimp && ./blimp

You need nothing pre-installed — blimp --setup installs what it uses (docker for the catalog, python + pyiceberg, aws CLI, duckdb, unzip). Set BLIMP_SKIP_DEPS=1 on hardened/offline hosts to manage deps yourself.

Docker alternative (optional): a prebuilt image bundles everything, for hosts where you'd rather not install anything on the OS:

docker build -t blimp-kit . && \
docker run --rm --network host -e CLUSTER_ID=… -e WAREHOUSE=s3://… blimp-kit --setup

--network host lets it use the node's own identity and reach the gateway private IP (that's what makes vpc mode key-free). See docker-compose.yml to bring up the catalog + CLI together.

Quick start — the blimp command, beginning to end

blimp                 list the commands
blimp --setup         connect a Blimp node to your data (interactive)
blimp --query         query optimizer + incremental CDC delta-merge
blimp --storage       storage & read-through cache suite
blimp --acid          ACID / linearizability check of both data paths
blimp --bench         author / materialize / delta-merge timing profile
blimp --update        update software on the node: status | all | zs3,eblobber,nessie,gotenberg,rclone
                      (--tag svc=tag, --dry-run; runs on the node via its :9401 helper, one at a time)

Wiring is saved to ~/.blimp_env by --setup; every command reads it. Running a command with no wiring offers to run --setup for you first.

Rough timing: --setup takes about 10-15 minutes end to end, --query about 5 minutes, --storage about 30 minutes (it drives the fullest benchmark suite — warp, mlperf, cache), and --acid about 5 minutes.

Zero-touch / CI: every prompt is skipped when its env var is pre-set — export these (or source a file with set -a) and --setup runs unattended:

REGION CLUSTER_ID ICEBERG_URL WAREHOUSE ORIGIN_BUCKET NAMESPACE \
S3_KEY S3_SECRET        # external mode only; blank/unset in vpc mode
GW GW_AK GW_SK          # optional — auto-derived from CLUSTER_ID when unset

Step 1 — create a Blimp node

blimp.software → Create a Blimp node. Note the node id.

Step 2 — ./blimp --setup on your Iceberg node

Fully interactive — every value is prompted with a default (Enter accepts); any env var already set skips its prompt (that's the zero-touch/CI path):

S3 region [ap-south-1]:
Blimp cluster id (from blimp.software): 1784970467881

It then runs the network assessment: probes the gateway's private IP. Reachable → vpc mode (no keys typed). Not reachable (other network/ account/cloud) → external mode (public DNS zus-<id>-0.zus.network, explicit S3 keys):

network assessment → MODE=vpc (gateway private 10.10.12.249 reachable: yes)
  gateway → 10.10.12.249 · advertise this node as → 10.10.12.168 · S3 creds → blank (IAM role)

Remaining prompts: gateway (auto-filled), namespace, existing Iceberg REST URL (blank = stand one up here via docker), S3 endpoint, data bucket, warehouse (defaults to s3://<data-bucket>/wh), S3 keys (blank in vpc mode).

Guardrails --setup enforces (each is a real failure mode):

1. Warehouse co-located with the data bucket. A warehouse in a different bucket breaks the gateway's catalog-metadata reads → MV author refuses ("grain not sampleable"). Divergence warns and offers to fix.
2. Bucket access grant (vpc/same-account): applies a bucket policy for the gateway's own identity + your account — no silent 403 at author time.
3. Blank keys are the correct vpc answer — real keys are only ever typed in external mode.

What --setup does, in order:

1. Deps bootstrap — installs docker / python venv + pyiceberg / aws CLI / unzip if missing (BLIMP_SKIP_DEPS=1 to manage yourself).

2. Network assessment — probes the gateway's private IP → picks vpc mode (no keys) or external mode (public DNS, explicit keys).

3. Catalog — reuses your Iceberg REST URL, or stands one up here via docker run (iceberg-rest on :8181 over your warehouse).

4. Bucket grant (vpc/same-account) — applies the bucket policy so the gateway role can read; skipped in external mode (bring your own grant).

5. Saves the wiring to ~/.blimp_env (mode 600) for every later command.

6. Wires the Blimp node over its admin API — no SSH, no restart:

   POST http://<gateway>:9000/admin/source/configure
   Authorization: Bearer zus-<CLUSTER_ID>
   {"source":"customer","iceberg_url":"http://<node>:8181","warehouse":"s3://…/wh",
    "namespace":"…","bucket":"…","s3_key":"…","s3_secret":"…","s3_region":"…"}
   

   The gateway applies this to its live source config (ZS3_SRC_CUSTOMER_*) — the very next query reads your data — and persists it so restarts keep the wiring. Keys travel over the authenticated admin API and are stored root-only on the gateway volume. On success --setup prints ✓ cluster wired: source=customer … (live, no restart).

Finally it prints the same values for the Blimp node UI (Query Optimizer → Production, the manual path) and the snapshot_changed webhook for your pipeline.

Step 3 — register your tables (optional)

venv/bin/python3 register_tpcds_tables.py \
  --catalog http://localhost:8181 --warehouse s3://my-bucket/wh \
  --source-bucket my-bucket --region ap-south-1 --namespace myns

(add_files registration — no data copy.)

Step 4 — point the Blimp node at the source (optional)

Automatic (no SSH): --setup wires the cluster itself over the authenticated admin API — POST http://<gw>:9000/admin/source/configure (Authorization: Bearer zus-<CLUSTER_ID>). The gateway applies the source in its live env (effective on the next query, no restart) and persists it across reboots. Requires a gateway image ≥ 2026-07-27; on older images the call fails gracefully and --setup prints the manual steps.

Manual fallback (older gateway image, or the admin-API call failed): paste --setup's printed values into the Blimp node UI (Production tab).

External mode requires S3_KEY/S3_SECRET; --setup sends them in the /admin/source/configure body. In vpc mode leave them unset — the gateway reads via its own identity.

Firewall: the gateway must reach this node on the catalog port. If the Blimp node security group doesn't open 8181, serve the catalog on an open port (e.g. docker run -p 8081:8181 …) and use that URL.

Fleet — one S3 URL for all your nodes

When you run more than one Blimp node, they form a fleet with a single S3 endpoint and one shared key — you never juggle per-node URLs. Point any S3 tool or a mount-s3 (FUSE) client at it and you see your entire namespace, no matter which node stores each object; content is deduplicated fleet-wide (identical data kept once across all nodes), and the URL is round-robin + health-checked, so a node going down moves traffic to a healthy one.

• Endpoint: https://fleet-<account>.blimp.software:9443 (TLS, path-style)
• Credentials: your account's shared fleet S3 key (Access + Secret)
• S3 tools:

  mc alias set fleet https://fleet-<account>.blimp.software:9443 <AK> <SK>
  aws --endpoint-url https://fleet-<account>.blimp.software:9443 s3 ls
  

• FUSE (mp-s3) — same endpoint, same key:

  mount-s3 --force-path-style --endpoint-url https://fleet-<account>.blimp.software:9443 <bucket> /mnt/fleet
  

Any node resolves the full namespace (content-addressed dedup index + cross-node fetch), and a cross-node read that hits a transient break is retried server-side — the client never sees a truncated stream. Prefer the fleet URL over a per-node blimp-<node>-0.blimp.software:9443 for anything user-facing.

One connection lands on one node, so a single client is bounded by that node's link — run several clients (or several mounts) to aggregate across the fleet. The run-blimp test harness does exactly this: it drives warp/mlperf legs from multiple client boxes at the fleet URL so the round-robin spreads them across nodes.

Testing the Blimp node

The --query, --storage, and --bench commands are testing/validation tools — they prove the wiring, measure the node, and gate a rollout. They are not part of production operation (production is your pipeline + the snapshot_changed webhook from Step 4).

A — ./blimp --query (prove authoring + CDC)

Defaults to q9 q88 q14 q64 q4 on store_sales: ① serve/author each → ② append rows to all five facts + fire the webhook → ③ re-run and report how each MV refreshed. CDC is lazy: the webhook only marks the MV stale — the next query pays the merge and reports merge_ms inline (read from the phase-3 query response, with /admin/mv/wave/report as fallback):

q64  store_sales  merge_ms=8856  mode=incremental
q4   store_sales  author_ms=45048  no MV — served from base

There is no PASS/FAIL verdict. The suite reports what each query did and you judge it: mode=incremental is the delta-merge fast path, no-delta is a full re-author, and "no MV" means the query fell back to scanning base. Nothing here asserts a threshold, because the interesting outcomes (a query that authors no MV, a merge that silently measured unchanged data) are not binary.

Add --verify to check the merged rows are correct rather than merely fast — off by default because the gateway runs no verification while serving, so an unflagged run measures the production path.

Wider: SUITES="store_sales:3 19 43 52 55;store_returns:1" ./blimp --query

Join-CTE queries (q64-class, SUITES="catalog_sales:64") only see a delta when the append touches both sides of the join — the seeder therefore appends catalog_sales referentially (matching catalog_returns rows, keys copied). A sales-only append correctly reports no-delta, not a bug.

B — ./blimp --storage (storage & cache suite)

Storage = warp S3 PUT/GET + TTFB, mlperf. The suite is self-contained: it installs its own tools first (warp pinned v1.1.4, mount-s3, dlio) — BLIMP_SKIP_DEPS=1 opts out on hardened hosts, and a section whose tool still can't install is skipped loudly. Cap sizes on small nodes: WARP_BUDGET_MIB=5120 MLPERF_NUM_FILES=35 ./blimp --storage.

C — ./blimp --bench (timing profile)

Bench = min/median/avg/max author / materialize / delta-merge profile (ITERS AUTHOR_ITERS CDC_ROWS, all defaulting to 3/3/50000; BENCH_QNR=64 benches a different TPC-DS query from $Q_DIR/q<N>.sql instead of the built-in q1; BENCH_FACT=catalog_sales appends referentially for join-CTE queries).

Check the n= on each summary line before quoting it — an append that re-materializes instead of merging contributes no delta_merge_ms, so n can be lower than ITERS. See the worked example in the walkthrough below for what the numbers do and don't mean.

D — Testing ACID (blimp --acid)

A Blimp node is not a single disk — a write is erasure-coded across many independent blobbers, and reads are served through several front-ends (the gateway S3 API, a read-through cache, a mounted filesystem). --acid proves that this distributed stack still behaves like one correct store: a value you just wrote is the value everyone reads, and a read that races an overwrite never returns a stale copy or a torn mix of the old and new bytes.

ACID is OFF by default on the gateway, and --acid turns it on for you. As of 2026-08-06 the gateway defaults ZS3_ACID_ALL to off: the warp benchmark that argued strict whole-file verify was free is the same tool that produced the phantom EOF errors, so it cannot be used to justify paying that cost on every deployment. ZS3_ACID_BUCKETS still keeps the MV warehouse buckets strict.

blimp --acid arms it via POST /admin/acid before the run and restores the previous setting afterwards — including on failure or Ctrl-C. That pin is a RUNTIME setting and is not persisted, so a gateway restart reverts to ZS3_ACID_ALL regardless.

If you run the porcupine checker by hand, arm it yourself first — a linearizability test against a gateway with ACID off is measuring the wrong configuration, and any torn read it reports says nothing about the ACID path:

curl -X POST http://$GW:9000/admin/acid \
  -H "Authorization: Bearer zus-$CLUSTER_ID" \
  -H 'Content-Type: application/json' -d '{"enabled":true}'
# ... run the test ...
curl -X POST http://$GW:9000/admin/acid \
  -H "Authorization: Bearer zus-$CLUSTER_ID" \
  -H 'Content-Type: application/json' -d '{"enabled":false}'

Leaving it pinned on makes every LATER benchmark quietly pay the ACID cost — which is exactly how a misleading measurement gets made.

It uses porcupine, the same linearizability model-checker used in Jepsen distributed-systems testing. Many clients hammer the same keys with concurrent writes and reads; porcupine then searches for any ordering of those operations consistent with a single correct register. If none exists, the history is NOT LINEARIZABLE and the offending operation is reported. It runs against both read paths, each under two profiles:

Path	What it is
gateway S3 :9000	the raw S3 API → gosdk → blobbers
mountpoint-s3	the gateway bucket mounted as a POSIX filesystem (the mlperf / customer mount path)

• single-writer — one client writes a key while N clients read it. This is the read-after-write guarantee an object store actually promises; a stale or torn read here is a genuine consistency bug.
• multi-writer — every client both writes and reads the shared keys, a stricter total-order probe. (High "errors" counts on the FUSE leg are just the client refusing two concurrent writers to one key — the verdict is over the operations that completed.)

blimp --acid
# tune with ACID_CLIENTS (8), ACID_KEYS (4), ACID_DURATION (45s)

A clean run prints LINEARIZABLE for every leg — read-after-write is preserved and no torn erasure-decode is ever exposed, whether you reach the node over S3 or as a mounted filesystem. --setup builds the checker (a small Go program under acid/) automatically; it needs no configuration beyond the gateway S3 keys already in your wiring. Run the checker from a box other than the gateway (e.g. the Iceberg node) — co-locating the load generator on a small gateway can starve it and produce spurious Illegal verdicts. (A concurrent-read unexpected EOF from the warp load tool specifically is a warp client artifact, not a consistency failure — verified separately with aws s3 cp md5 checks that pass byte-for-byte with the strict ACID verify on and off.)

---

Reference

• WALKTHROUGH.md — the same flow as a full transcript: every command and its real output, start to finish, on a fresh node.
• TESTING.md — testing the kit itself (./test_kit.sh), not the Blimp node.
• Product docs: docs.zus.network/zus-docs/webapps/blimp — the optimizer, incremental MVs (CDC), and the Prod-Query & MV API.

---

Verified walkthrough (real commands + output)

A fresh Ubuntu 24.04 cloud VM on the cluster's private network, cluster 1784970467881. Every line below is the actual command and its actual output from a live run.

1. Confirm the node's identity (no keys anywhere).

$ whoami; hostname; hostname -I
ubuntu
ip-10-10-12-62
10.10.12.62 172.17.0.1

$ curl -s -H "X-aws-ec2-metadata-token: $TOK" \
    http://169.254.169.254/latest/meta-data/iam/security-credentials/
ec2-ssm-role-1784970467881

2. One command does setup end-to-end (deps → mode → catalog → wiring). blimp --setup on the bare box:

== blimp --setup — connect a Blimp cluster to this node's data ==
== checking prerequisites ==
  installing: docker-compose-v2               # ← installs its own missing deps
  ✓ deps ready (python: /home/ubuntu/.blimp_venv/bin/python3)

network assessment → MODE=vpc (gateway private 10.10.12.249 reachable: yes)
  gateway → 10.10.12.249 · advertise this node as → 10.10.12.62 · S3 creds → blank (IAM instance role)

standing up an Iceberg REST catalog on :8181 over s3://blimp-tpcds-sf1-aps1/wh3
 Container iceberg-rest  Started
  Iceberg catalog up — cluster reaches it at http://10.10.12.62:8181
saved wiring -> /home/ubuntu/.blimp_env

================= POINT THE BLIMP CLUSTER AT THIS SOURCE =================
  Iceberg REST URL : http://10.10.12.62:8181
  Warehouse        : s3://blimp-tpcds-sf1-aps1/wh3
  Namespace        : tpcds_sf1x
setup done — validate with:  blimp --query   (and  blimp --storage )

(The installing: docker-compose-v2 line was later removed — the catalog now starts with a plain docker run, so the client box needs only the docker engine, no compose plugin.)

3. Register your parquet as Iceberg (once):

$ ~/.blimp_venv/bin/python3 register_tpcds_tables.py --catalog http://localhost:8181 \
    --warehouse s3://my-bucket/wh --source-bucket my-bucket --namespace tpcds_sf1x
registered 24/24 tables into http://localhost:8181 ns=tpcds_sf1x

Testing the Blimp node (walkthrough)

The remaining items are the TESTING commands — validation of the wired node, not production operation.

A. blimp --query — author + incremental CDC:

Five queries against a single fact, appending to all five facts each cycle so a multi-fact query exercises a multi-fact merge. Live run, SF1 cluster 1785550395356, 2026-08-01:

== CDC bench: cluster=1785550395356 gw=10.10.114.87 rows/append=5000 suites=[store_sales:9 88 14 64 4] ==
==== fact: store_sales (q9 q88 q14 q64 q4) ====
>> phase 1: serve/author all (force_author=0)
   q9: author=? materialize=? cold_serve=2826ms mv=?x? (mv_h_0e90d0b63216)
   q88: author=? materialize=? cold_serve=2668ms mv=?x? (mv_h_ae8ceee390cc)
   q14: author=12252 materialize=? cold_serve=3134ms mv=?x? (none)
   q64: author=? materialize=? cold_serve=2657ms mv=?x? (mv_h_ad47f860697f)
   q4: author=45048 materialize=? cold_serve=2971ms mv=?x? (none)
>> phase 2: append +5000 to [store_sales store_returns catalog_sales catalog_returns web_sales] + snapshot_changed
tpcds.store_sales: +5000 rows -> snapshot 3827416906052195600
tpcds.store_returns: +5000 rows -> snapshot 686337096249913447
tpcds.catalog_returns: +1666 referential rows -> snapshot 6382376194974615624
tpcds.catalog_returns: +5000 rows -> snapshot 7838483972766842389
tpcds.web_sales: +5000 rows -> snapshot 4945461575749320971
>> phase 3: re-run all (incremental)

============================== CDC CONTRIBUTIONS ==============================
query      fact             mv_rows x cols   author_ms   merge_ms     mode   incr_ms
q9         store_sales                 ?x?           ?       5879 incremental      2781
q88        store_sales                 ?x?           ?       6507 incremental      2693
q14        store_sales                 ?x?       12252          -        -      3031
q64        store_sales                 ?x?           ?       8856 incremental      3875
q4         store_sales                 ?x?       45048          -        -      2959
  q9: MV mv_h_0e90d0b63216 — delta-merged
  q88: MV mv_h_ae8ceee390cc — delta-merged
  q14: no MV — served from base
  q64: MV mv_h_ad47f860697f — delta-merged
  q4: no MV — served from base

How to read it: mode=incremental with a merge_ms means the MV was refreshed by an append-only delta-merge (the O(|MV|+|delta|) fast path). A query with no MV serves from base and reports author_ms instead — q14 and q4 do that here, which is a real gap, not a pass/fail. There is no PASS/FAIL line: the suite reports what happened and you judge it.

An append that fails leaves nothing to merge, and phase 3 then measures UNCHANGED data while still printing plausible-looking no-delta rows. The suite now prints the seeder's full traceback and says so explicitly when that happens — if you see APPEND FAILED, every number below it is meaningless.

B. Raw stop/start (all 4 instances) → self-heal + re-validate:

private IPs unchanged; all 4 public IPs changed
DNS reconciled by the 60s cron (no touch):
  zus-1784970467881-0 -> 13.201.26.189   (gateway)
  zus-1784970467881-1 -> 3.110.120.231   (blobber-1)
  zus-1784970467881-2 -> 13.127.74.12    (blobber-2)
  zus-1784970467881-3 -> 3.111.245.151   (blobber-3)
post-restart q1: merge_ms=5138 mode=incremental   RESULT: PASS

C. External-cloud host (a different network, over public DNS):

network assessment → MODE=external (gateway private unknown reachable: no)
live query over zus-1784970467881-0.zus.network:9000
  {status: ok, rows: 1, author_ms: 575, query_ms: 2447, md5: 7a26dcec…}

D. blimp --storage numbers (2/1 cluster, 5 GB warp set):

warp   S3 PUT 749 MiB/s · GET 1673 MiB/s   (0 errors)

Those are from a 2/1 cluster on larger instances. Size the benchmark set to the cluster — a single 17 MB object measures nothing and reports throughput as slower than expected purely from per-request overhead.

E. blimp --bench — MV lifecycle timing profile:

Phase A cold-authors the same query AUTHOR_ITERS times (evicting the MV between iterations); phase B appends and refreshes ITERS times. Live run, SF1 cluster 1785550395356, 2026-08-01:

== MV lifecycle benchmark v2: cluster 1785550395356 (authors=3, appends=3, upserts=3, rows/cycle=50000) ==
  author[1] q1 COLD status=ok author_ms=24087 materialize_ms=4024 wall_ms=27393.2 mv=mv_h_aff2e89bc41f
  author[2] q1 COLD status=ok author_ms=15188 materialize_ms=3773 wall_ms=18415.7 mv=mv_h_aff2e89bc41f
  author[3] q1 COLD status=ok author_ms=15239 materialize_ms=3705 wall_ms=18522.2 mv=mv_h_aff2e89bc41f
  append[1] refresh delta_merge_ms=? materialize_ms=3763 query_ms=2955 engine=duckdb wall_ms=21891.6
  append[2] refresh delta_merge_ms=8321 materialize_ms=? query_ms=3151 engine=duckdb wall_ms=11930.5
  append[3] refresh delta_merge_ms=6627 materialize_ms=? query_ms=3059 engine=duckdb wall_ms=10051.4

== SUMMARY (q1-scale MV over 50000-row commits) ==
  one-time author_ms (fresh queries):   n=3 min=15188 median=15239 avg=18171 max=24087 (ms)
  one-time materialize_ms:              n=3 min=3705 median=3773 avg=3834 max=4024 (ms)
  one-time wall_ms:                     n=3 min=18416 median=18522 avg=21444 max=27393 (ms)
  append  delta_merge_ms:               n=2 min=6627 median=7474 avg=7474 max=8321 (ms)
  append  refresh materialize_ms:       n=1 min=3763 median=3763 avg=3763 max=3763 (ms)
  append  commit→answer wall_ms:        n=3 min=10051 median=11930 avg=14624 max=21892 (ms)