dfkv — distributed KV cache for LLM inference (SGLang · LMCache · vLLM)

A small, self-contained distributed key-value cache that pools GPU-node NVMe SSDs into a shared, large-capacity KV pool for LLM inference (e.g. GLM-5.1 / MLA, DeepSeek-V4), without any DingoFS / brpc / S3-RADOS dependency — it runs on its own (only its built-in MDS + etcd for dynamic membership). It plugs into three engines through thin adapters over one portable core:

• SGLang HiCache as an L3 external KV store (--hicache-storage-backend dynamic).
• LMCache as a RemoteConnector.
• vLLM directly as a KVConnectorBase_V1 (GPUDirect RDMA, no LMCache).

Origin: extracted from the DingoFS branch feat/kvcache-sglang (src/cache/kvclient). The portable core has zero coupling to DingoFS, so it lives here as an independent repo. To instead fuse these semantics into the production dingo-cache (brpc + MDS), see docs/INTEGRATION.md.

What it is

• dfkv_server — a cache-node daemon. Disk + bounded eviction, cache-only (a miss is a clean NotFound; no object-store fallback), synchronous durable-visible writes. Supports multiple NVMe SSDs per node (--dir d1,d2,d3, intra-node Ketama). With --mds, --group, --id, --advertise, --weight it registers into the MDS tier. With neither a --store-engine option nor DFKV_STORE_ENGINE, every server/store path selects the restart-safe slab backend. file remains an explicit diagnostic/rollback choice; invalid slab capacity/geometry refuses startup rather than falling back. An optional write-through RAM hot tier (DFKV_RAM_TIER=1) serves registered-memory hits without disk I/O — see docs/ARCHITECTURE.md.
• dfkv_mds — stateless Membership Directory Service daemon. Flags: --listen <port> and --etcd <host:port> (default 127.0.0.1:2379). The only etcd client in the system; holds each node's etcd lease on its behalf. Deploy as N replicas — no load-balancer needed; nodes and clients each pick any reachable MDS and fail over automatically.
• libdfkv.so — C ABI client (canonical namespace + object-key hashing, Ketama routing, opaque raw-value Put/Get/Exist).
• integration/hicache/dfkv_hicache.py — SGLang HiCacheStorage plugin loaded via --hicache-storage-backend dynamic (no SGLang fork). MLA: one packed-latent object per page, no tp_rank suffix, backup_skip (only tp_rank 0 writes).

Design in one breath

SGLang HiCache (zero-copy interface_v1) → dfkv_hicache.py (ctypes) → libdfkv v2 client (length-framed namespace/object identity + Ketama route) → TCP/RDMA → dfkv_server → optional RAM hot tier → DiskCacheGroup over N NVMe (production slab extents; explicit file fallback). Distributed = client-side consistent hashing; no replication (regenerable KV → node loss = miss → recompute). Full architecture: docs/ARCHITECTURE.md.

Membership is managed by the MDS tier (dfkv_mds + etcd). Nodes register with the MDS on startup and send periodic heartbeats; etcd leases (TTL 30 s) are the liveness signal. Native clients use the single immutable dfkv_client_options_v2 constructor: set either static members or mds_endpoints + mds_group. MDS mode polls the directory and rebuilds the weighted consistent-hash ring whenever its epoch advances. Two-layer offline detection: layer-2 — etcd lease expiry → MDS view changes → client epoch → ring rebuild (authoritative removal, ≤ 30 s); layer-1 — PeerHealth fast avoidance short-circuits transport failures to misses during a cooldown. There is no post-open membership mutation API in v2.

Client registration (who is using dfkv): cache consumers (inference connector instances — vLLM / LMCache / SGLang HiCache) register themselves with the MDS under a disjoint etcd prefix (/dfkv/v1/groups/<g>/clients/<id>) so they never enter the placement ring. The same lease/heartbeat contract as nodes applies — a dead connector's key expires out of etcd within the TTL, no explicit deregister, no stale keys. Connectors set DFKV_CLIENT_OPT_REGISTER_WITH_MDS, client_id, client_info, and heartbeat fields in dfkv_client_options_v2 when MDS discovery is used (opt out with DFKV_CLIENT_REGISTER=0). Observe with dfkvctl clients --mds <ep,...> --group <g> or the dfkv_mds_group_clients gauge. Only upgraded clients register, so an empty list means "none of the current consumers are registered," not "no one is using dfkv."

Build & test (no GPU / no RDMA needed)

cmake -S . -B build            # add -DDFKV_STATIC_LIBSTDCXX=ON for portable binaries
cmake --build build -j
ctest --test-dir build --output-on-failure   # C++ gtests + the Python plugin test

Artifacts: build/dfkv_server, build/dfkv_mds, build/libdfkv.so.

Run a cluster

# 1. Start etcd (one or three nodes, external)

# 2. Start MDS replicas (stateless, any number)
dfkv_mds --listen 9400 --etcd 127.0.0.1:2379

# 3. On each cache node (--mds requires --id and --advertise)
dfkv_server --dir /mnt/disk1/dfkv,/mnt/disk2/dfkv,/mnt/disk3/dfkv \
            --port 12000 --cap 6597069766656 \
            --mds 10.0.0.1:9400,10.0.0.2:9400 \
            --group default --id n1 --advertise 10.0.0.10:12000

# 4. Client: one immutable v2 construction descriptor (recommended MDS mode)
#    dfkv_client_options_v2 o = { ... .mds_endpoints = "10.0.0.1:9400,10.0.0.2:9400",
#                                 .mds_group = "default" };
#    dfkv_client_t c = dfkv_open_v2(&o);
# Static single-node mode sets o.members instead; no mutable follow-up calls.

Observe the cluster

dfkvctl ring    --mds 10.0.0.1:9400 --group default        # cache nodes + ring share
dfkvctl clients --mds 10.0.0.1:9400 --group default        # inference consumers
dfkvctl stats   --mds 10.0.0.1:9400 --group default        # ring stats + clients=N
dfkvctl stat    --mds 10.0.0.1:9400 --group default --all  # per-node deep-dive

Full dfkv CLUSTER deploy runbook (etcd + MDS + systemd units): docs/DEPLOY.md. Per-engine connect/config + client env/config reference (all connectors): docs/CONNECTORS.md.

Layout

src/        portable C++ core: common/ (shared types) · utils/ (generic helpers) ·
            transport/ (TCP/RDMA + wire protocol) · cache/ (StoreEngine: file
            KVStore | slab SlabAllocator+DiskSlabStore · RamTier · dfkv_server) ·
            client/ (KV client + C ABI) · mds/ (membership service + dfkv_mds) · tools/ (CLIs)
integration/hicache/  dfkv_hicache.py (SGLang dynamic backend plugin) + dfkv_telemetry/
                      (canonical shared telemetry pkg, vendored by the other connectors)
integration/common/   dfkv_common shared identity and C ABI schema package
integration/lmcache/  dfkv_connector  (LMCache RemoteConnector, ctypes over libdfkv.so)
integration/vllm/     dfkv_vllm       (vLLM KVConnectorBase_V1, GPUDirect RDMA, bypass LMCache)
test/       gtest suites + test/python (unittest + no-torch sglang shim)
docs/       ARCHITECTURE.md (layers · storage engines · RAM hot tier · wire protocol) ·
            CONNECTORS.md (engine connectors: HiCache · vLLM · LMCache + client env/config reference) ·
            DEPLOY.md (dfkv CLUSTER deploy: etcd + MDS + server + systemd) · INTEGRATION.md (fuse into dingo-cache)
            CACHELIB_EVALUATION.md (F25 decision: no CacheLib/Navy backend in v2.0.0)

Engine integrations

• SGLang HiCache: integration/hicache/dfkv_hicache.py — see docs/CONNECTORS.md §2 (connect/config/use; cluster deploy is docs/DEPLOY.md).
• LMCache: integration/lmcache/ (dfkv_connector) — see docs/CONNECTORS.md §4 (deploy + design + implementation).
• vLLM (direct): integration/vllm/ (dfkv_vllm) — a KVConnectorBase_V1 connector occupying the same --kv-transfer-config slot as MooncakeStoreConnector, storing/loading KV directly over GPUDirect RDMA (no LMCache, no host bounce). Pure-Python ctypes over libdfkv.so; uses the scatter-gather batch API to coalesce per-chunk keys. Validated on H100 + IB with DeepSeek-V4 (multi kv_cache_group / MLA + SWA), full cross-restart and cross-DP prefix hit. See docs/CONNECTORS.md §3 (config reference + recommended settings) and integration/vllm/README.md.

Operability & performance features

• Slab-first storage (no flag/env required; --store-engine=slab may pin it explicitly): a fixed pool of pre-allocated extent files carved into deterministic size classes. Sparse slots.tbl metadata is rebuilt in bulk after restart, preserving cache warmth without directory walks, per-object opens, or file-per-block inode pressure. CRC-protected records, dirty-epoch cold reset, deferred removal, O_DIRECT payload I/O, and byte-aware eviction keep failures fail-closed. --store-engine=file is an explicit diagnostic fallback, not a silent recovery path or the production default.
• RAM hot tier (DFKV_RAM_TIER=1, off by default): a pre-registered arena fronting the disk — PUT is write-through (synchronously visible, async-flushed to disk) and a warm GET is served straight from the arena over RDMA (zero-copy scatter-send from the arena MR, no open/pread/disk), removing the disk-bound COLD load bottleneck. Send-in-flight slot pinning + flush backpressure keep it correct; dfkv_ram_* metrics expose hit-rate + backpressure. See docs/ARCHITECTURE.md §5–6.
• 128-bit native block identity: SHA-256 of length-framed canonical namespace + object-key bytes, truncated to 128 bits for wire/storage identity. Connector object keys are self-delimiting binary bytes encoding pool, full chunk hash, DP/TP/PCP/DCP/PP coordinates, cache-group/component, and optional scatter geometry; namespace remains separate binary identity. C ABI v2 passes every key as pointer + exact length (parallel arrays for batch/SG), so embedded NUL and non-UTF-8 bytes are preserved. Values remain opaque raw bytes with no hidden header or compatibility unwrap.
• Connection pooling + keep-alive (TCP_NODELAY): ~250× lower latency vs dial-per-call.
• Batch APIs with concurrent fan-out across nodes and local NVMe disks.
• Connect/IO timeouts + stale-connection retry: a hung node fails fast, never hangs.
• Observability (docs/METRICS.md): opt-in embedded Prometheus /metrics on dfkv_server and dfkv_mds (--metrics-port); sampled op-latency histogram, eviction/error/per-disk/RDMA counters server-side; client-side counters (peer health, IO errors) via dfkv_stats_snapshot + a plugin poller. Opt-in and off the datapath — no --metrics-port ⇒ no listener, behavior unchanged. The three connectors (vLLM / LMCache / SGLang HiCache) can also push fleet metrics (ops/keys/bytes, op latency, per-peer latency) over OTLP to a central Collector → Grafana — opt-in via DFKV_METRICS_ENABLED=1, zero-dependency stdlib exporter by default; see deploy/observability/CONNECTOR-USAGE.md and docs/METRICS.md §3.4.
• Dynamic membership: the immutable dfkv_client_options_v2 descriptor selects MDS discovery at construction; the client polls MDS and atomically rebuilds its weighted Ketama ring on etcd-epoch changes. No mutable membership or post-open discovery API is exported.
• CLI tools: dfkv_smoke (roundtrip check), dfkvctl — per-node ops (put/get/exist/stat) plus cluster views: dfkvctl ring (membership + ring vnode share + each node's self-reported version/config — engine, capacity, RAM tier, RDMA dev — carried on register/heartbeat, so fleet-wide version/config audit is one command, no per-node ssh) and dfkvctl stat --all (per-node metrics + aggregate) via MDS.
• RDMA transport v2 (gated -DDFKV_WITH_RDMA=ON, native libibverbs RC): with no device override each host chooses its first ACTIVE local HCA and sends an empty device selector to its peer. An explicit comma whitelist opts into multi-rail round-robin and therefore must name the intended fabric on both hosts. QPs bootstrap over a tiny TCP channel, so the data fabric needs no IP. A v2 capability probe is mandatory; protocol, QP, receive-segment, or registration mismatch rejects the connection. Unset DFKV_RDMA selects TCP; once RDMA is requested it never switches transports.
• One-sided zero-copy data plane: control descriptors/status use small 4-KiB SEND/RECV buffers. PUT payloads RDMA-WRITE into leases from one process-wide, pre-registered receive segment; GET payloads RDMA-WRITE directly into the caller's registered buffer. No connection-sized block buffers or payload copy are used.
• Optional pipelining (DFKV_RDMA_DEPTH=K): K requests in flight per connection. A network-latency hider, not a throughput knob — GET and PUT are both depth-flat (the per-connection serve loop is in-order; benchmarked GET ~1.24 GB/s at depth 1 == 32). The throughput levers are multi-connection fan-out (batch_concurrency) and fewer/larger keys. See docs/datapath-perf-notes.md.
• NUMA-aware rail selection (DFKV_RDMA_NUMA=1): with an explicit multi-rail whitelist, prefers an ACTIVE rail local to the calling thread, then round-robins the listed healthy rails. Server threads follow their QP's rail NUMA node; the single shared receive segment is registered on each selected rail but is not separately NUMA-allocated per rail. Off by default; vendor-neutral (sysfs + sched_getcpu, no libnuma/CUDA).
• HiCache v2 (PoolTransfer) for multi-pool models (Mamba/SWA/DeepSeek-V4).
• Packaging: CPack (deb/rpm/tgz) + Dockerfile; graceful shutdown; leveled logging.

Recommended tuning (v2.0)

Validated on a 5-node ring (8×B200 hosts, 6× Gen4 NVMe + 8×400G IB per node, 128 GiB RAM arena): cold read 97 → 156 GB/s, hot read 48 → 97 GB/s single-pair / 147 GB/s mesh, cold-read p99 0.9–2.2 s → ~50 ms. Defaults are safe for a single-fabric node; production overrides below make fabric selection and capacity explicit.

Server (per cache node):

Knob	Recommended	Why
--rdma-dev	leave unset for one local HCA; list the fabric explicitly for multi-rail	Unset selects the first ACTIVE local HCA (peer names may differ). A comma list opts into multi-rail, anchors only listed active devices, and requires compatible names/fabric on both hosts; inactive entries are rejected.
DFKV_DISK_HASH_WEIGHT	10	Flattens the intra-server disk ring share from ±20 % to ±3 % so the hottest disk stops gating the whole node (+5–6 % cold read, ~2× lower p99). Re-routes existing keys (cache miss + refill) — flip together with a restart/upgrade window.
--rdma-depth	4 (keep)	Deeper is not itself a throughput knob. It consumes more slots from the fixed shared receive segment.
--ram-tier / --ram-tier-bytes / --ram-tier-shards	on / sized to the node / 16 for ≥100 GiB arenas	Large arenas contend on the shard locks under mixed load (+40 % mixed R/W at 16 shards on a 128 GiB arena); small (≤16 GiB) arenas are fine at the default 8.
--store-engine	slab	Index rebuilds on restart; removes file-per-block hazards.

Production client (inference connectors, one process per TP rank):

Knob	Recommended	Why
DFKV_RDMA_DEV	best: rail affinity per rank — inject the matching local/server HCA name per process; simpler on symmetric hosts: the full comma list + DFKV_RDMA_NUMA=1	Explicit values are sent to the peer, so verify those names exist on both hosts/containers. Leave unset to let each host select its own first ACTIVE HCA when names differ.
DFKV_RDMA_DEPTH	4	Pairs with the server's posted depth (window = min of both, negotiated).
DFKV_FANOUT_THREADS	unset (default 32)	Only wide single-process clients (benchmarks, many concurrent Batch* callers) need more.

Read-side convoy collapse (v2.0, opt-in) — for MLA + TP-N inference rings, where every rank is a separate process fetching the SAME page and the NVMe otherwise eats N identical reads per page (measured 1:1 disk:wire on a TP8 replay; with the knobs below: ~2.4 device reads per duplicated page, repeat reads of promoted pages hit zero disk):

Knob	Recommended	Why
DFKV_READ_COALESCE	1	Master switch. Concurrent identical GETs share one disk read (sync + io_uring paths), and a whole-value read with convoy evidence is promoted into the RAM arena as a born-durable resident (zero flush cost, evictable at once). Unset = exact pre-1.35 data path.
DFKV_READ_COALESCE_RECUR_MS	1000 (default)	Recurrence window: a lone whole read leaves a 64-byte key fingerprint; an identical read inside the window promotes on completion even when rank drift missed the in-flight overlap. Fingerprints, not payloads — widening it costs ~nothing (64 Ki-entry bound). 0 restores the overlap-only gate.
DFKV_READ_COALESCE_TIMEOUT_MS	500 (default)	Follower wait bound; a waiter whose leader connection dies falls back to its own read instead of hanging.

Watch dfkv_read_coalesce_{leaders,coalesced,recur,timeouts}_total and dfkv_ram_promoted_total; timeouts staying at 0 and promoted tracking recur are the healthy signature. Requires the RAM tier (--ram-tier) for promotion; coalescing alone works without it. Note for env-file deployments: the file is sourced, so new knobs must be explicitly exported by the start script or a systemd drop-in to reach the server process.

Order matters when rolling this out: upgrade servers to v1.34+ first, then widen clients to multi-rail — pre-1.34 servers have no anchor on non-default rails and will exhibit the idle re-registration storm described above.

Benchmark reproduction (dfkv_bench): DFKV_RDMA=1 (explicit, or it silently measures TCP), 8-rail DFKV_RDMA_DEV, DFKV_RDMA_DEPTH=4, DFKV_FANOUT_THREADS=256; cold-read sweet spot --threads 16 --batch 8 --size 4194304 per client node. Keep --count ≤ the seed's written key count, and let the drives settle ~10 min after bulk writes before cold-read A/Bs (FTL GC depresses cold reads ~25 %). The command prints every requested phase report, then exits 1 if any PUT/GET failed, 2 for invalid CLI usage, and 0 only when every requested operation succeeded.

Status

TDD; 264 C++ ctest entries (default) / 288 (RDMA+io_uring) + Python plugin & connector tests green, 0 warnings, ThreadSanitizer-clean. CI: gcc/clang build+test, TSan, RDMA datapath (Soft-RoCE loopback), RDMA compile-check, static-artifact build. License: Apache-2.0. Architecture & design: docs/ARCHITECTURE.md. Rollout: docs/DEPLOY.md.