Skip to content

Telemetry - #2109

Draft
pblazej wants to merge 13 commits into
mainfrom
blaze/telemetry
Draft

pblazej wants to merge 13 commits into
mainfrom
blaze/telemetry

Conversation

@pblazej

@pblazej pblazej commented Sep 18, 2026

Copy link
Copy Markdown

Telemetry

Draft: client telemetry in livekit-client, implementing the "Client Telemetry" design doc for the JS ecosystem. The cross-SDK wire spec — events, attributes, cadence and upload policy — lives in livekit-telemetry/SPEC.md; the decisions specific to this ecosystem, with the measurements behind them, are in TELEMETRY.md.

The Swift, Kotlin and Dart SDKs get this from a Rust core over UniFFI (livekit/rust-sdks#1396). A browser cannot link it and React Native could not share it (Hermes has no WebAssembly), so this is the second implementation — and the only one the JS ecosystem needs, since @livekit/react-native wraps this package.

Design

  • One pipeline per page, one scope (trace id) per Room connection. lk.connect and lk.reconnect spans with their checkpoints, lk.publish, lk.subscribe ended by the first inbound bytes, lk.rtc.stats.sample windows, lk.room.disconnected, and the device state a page can observe.
  • Windows are folded from the readings the existing per-track monitors already take (monitorFrequency = 2000), so measuring a call costs no second getStats().
  • The only OpenTelemetry code that ships is the OTLP encoder (@opentelemetry/otlp-transformer, 5.2 KB gzipped). The rest of the SDK is 23 KB and gives a batch processor and a fetch transport that do not fit the policy below; TELEMETRY.md §1 has the numbers.
  • Write-ahead: a batch is stored before the network is tried and removed only once the collector takes it. Where it is stored is TelemetryStorage, the one replaceable part — memory in a tab, a directory of files on React Native.
  • Telemetry never wins over media: one request in flight, uploads held while a connect or reconnect span is open (60 s cap), a 429 that keeps its batch instead of losing it, four batches per tick when replaying a backlog, Priority: u=7, and fetch(keepalive) under 64 KiB for the page's last flush.
  • Off unless Telemetry.configure({ endpoint }) is called or the room is on LiveKit Cloud, which derives the route and the token from the connect itself.

Integration

@livekit/react-native is the platform integration: livekit/client-sdk-react-native#458. It supplies the file-backed store and the device state a browser cannot see, and reuses every call site here unchanged.

Size

pnpm size-limit, brotli: ESM { Room } 114.24 kB → 123.19 kB against a 150 kB limit, comfortable. The UMD bundle cannot tree-shake it: 123.58 kB → 132.70 kB against a 130 kB limit, so this PR raises that limit to 135 kB. The alternative is a separate UMD entry point, as the e2ee and frame-metadata workers already have — that is a call for this repo's owners, not something to decide quietly in a diff.

Testing

  • pnpm test — 854 unit tests, including a telemetry suite that reads the JSON bodies a stubbed collector would receive: a window's counters and gauges, a hold that stops uploads and not collection, a 429 that keeps its batch, oldest-first eviction with the record count, a replayed backlog, and a cached batch keeping the encoding it was written with.
  • pnpm vitest run --config vitest.telemetry.config.mts — one full session in a real Chromium with fake media devices, against a real livekit-server --dev and a real collector fanning out to the same Grafana LGTM stack the mobile harness uses. Two Rooms in one page, both reconnect paths. One run delivers 2 × lk.connect with all four checkpoints, 2 × lk.publish, 6 × lk.subscribe (subscribedfirst_media), 2 × lk.reconnect (attempt 1 quickattempt 2 full at signal_disconnected, then a full one), 9 stats windows across both directions and kinds, and 2 × lk.room.disconnected.

Known gaps

  • The SDK's typed stats are a subset of SPEC: no inbound RTT, jitter-buffer counters, video freeze/pause counts, audio level or interruptions. Each is a field to add to getSenderStats/getReceiverStats, which parse the raw report already — packetsReceived/packetsLost on inbound audio were added here as the first.
  • lk.subscribe starts at TrackSubscribed rather than at the intent to subscribe, which is what SPEC asks for and what the Rust core does.
  • OTLP/JSON trace ids do not survive the Cloud ingest: the spec mandates hex, the ingest reads stock protojson base64, and ids land as zeros. Protobuf is unaffected and is the default here, but this is worth fixing server-side — every standard OTLP/JSON client hits it.

Before undrafting

  • Decide the UMD budget: raise the limit, or put telemetry behind its own entry point
  • Fill the missing SPEC stats fields in getSenderStats/getReceiverStats
  • Move lk.subscribe to the intent, not the subscription
  • Review the public surface (Telemetry.configure/emit/setCadenceFactor/deviceState, TelemetryStorage)

pblazej and others added 13 commits September 18, 2026 09:47
…lector

The mobile SDKs get this from the Rust core; the browser and React Native cannot
(Hermes has no WebAssembly), so the question is what to build and what to take.
TELEMETRY.md answers it with measurements: the OTLP encoder is worth importing
(5.2 KB gzipped), the rest of the OpenTelemetry SDK is not (23 KB against 44 KB
of remaining size budget) because our trace is a scope and our batching is the
upload policy.

The PoC is the transport end of that: one lk.ping, both encodings, from a real
Chromium at the same collector and Grafana LGTM stack the mobile harness uses.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
esbuild said 5.2 KB gzipped for the OTLP serializers; the repo's size-limit says
what that means here — 4.4 kB brotli on the UMD bundle, which has 6.4 kB of room.
The ESM path has 31 kB and does not care. Recorded because it is the number that
decides against the full OpenTelemetry SDK, and because the pipeline that goes on
top of the encoder will not fit under the UMD limit as it stands.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The pipeline from TELEMETRY.md, wired into Room: a scope per connect (one trace
id on every record of the call), lk.connect with its checkpoints, one lk.reconnect
per reconnect however many attempts it takes, lk.publish, lk.subscribe ended by
the first inbound bytes, lk.rtc.stats.sample windows folded from the readings the
track monitors already take, and lk.room.disconnected. Uploads hold while connect
or reconnect owns the uplink, and a 429 holds without losing the batch.

RTCEngine gained one field so Resuming/Restarting can say why they fired.
RemoteAudioTrack.getReceiverStats now fills in packetsReceived/packetsLost, which
its own type already declared.

Costs +8.95 kB brotli. The ESM budget absorbs it; the UMD one did not, and is
raised here rather than quietly — see TELEMETRY.md for the alternative.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
app_state from visibilityState, which the SDK already watches, and the connection
from navigator.connection, which SignalClient already reads for the connect
params. Both drive SPEC's cadence factor, which stretches the flush interval and
the stats window together and stops at 4x.

Thermal, low power and memory have no web API at all, so a page simply never
reports them and its factor stays at whatever the rest implies. React Native
answers those from its native module, which is the next commit over.

Also: an attribute nobody set is no longer shipped as an empty value.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
registerGlobals configures the resource and starts reporting device state long
before a connect names the collector, and SPEC says the pipeline may start
without a destination — so the queue now holds those records instead of dropping
them. An SDK nobody asked for telemetry still collects nothing at all.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
React Native is going to bind the Rust core through UniFFI, so that a phone
behaves the same whichever SDK the app used. What it must not do is reimplement
the instrumentation — when a span starts, which checkpoints it carries, how a
subscribe ends at first media, which getStats fields become a window. So that is
now on one side of a seam and the carrying is on the other: backend.ts is SPEC's
typed surface, the same boundary Swift, Kotlin and Dart already cross, expressed
in terms this package owns. Room does not know which backend is installed.

Nothing mobile is left on this side of it. DeviceState carries visibility and the
connection, which a page can answer; thermal, low power and memory pressure are
gone, because this package cannot observe them and the platform that can reports
them to its own backend.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…a choice

A batch is stored before the network is tried and removed only once the collector
has taken it — so a refused upload, a lost connection or a process that dies costs
nothing, which is what the Rust core has always done. Where it is stored is now
TelemetryStorage: the same five synchronous operations as the core's BatchCache,
defaulting to memory bounded by 4 MiB and 512 batches. A platform with a
filesystem supplies its own and this package learns nothing about it.

Requeueing is gone with it: a batch that could not be sent keeps its place in the
cache instead of being unshifted back into an array where the queue cap could eat
it. A backlog replays at four batches a tick beside a live call; a shutdown drains
without the budget. Evictions cost a known number of records, because the batch id
carries the count.

Two more platform-neutral verbs, for the pressure this package cannot see: emit()
names a pipeline-scoped record, setCadenceFactor() slows the cadence. A phone
reports the number, not the reason.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cache stores bytes; the Content-Type was coming from whatever the pipeline
was configured with when the batch finally went out. A React Native app that
cached JSON batches and replayed them as protobuf lost all of them to 4xx — 40
records, which only surfaced because the self-report says dropped.rejected
rather than letting a 4xx look like success.

The batch id now carries the encoding alongside the route and the record count,
so a batch written by an earlier run of the app is sent the way it was written.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…its place

It had one implementation and one hypothetical one. React Native turned out to
need exactly one thing from this package that a browser does not provide —
somewhere on disk to keep batches — and TelemetryStorage already is that seam,
with two real implementations behind it.

The scope and span classes get their names back, and the platform-neutral verbs
React Native does use stay: emit() for a record this package has no vocabulary
for, setCadenceFactor() for pressure it cannot see.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@changeset-bot

changeset-bot Bot commented Sep 18, 2026

Copy link
Copy Markdown

🦋 Changeset detected

Latest commit: 24ffacf

The changes in this PR will be included in the next version bump.

This PR includes changesets to release 1 package
Name Type
livekit-client Minor

Not sure what this means? Click here to learn what changesets are.

Click here if you're a maintainer who wants to add another changeset to this PR

@github-actions

Copy link
Copy Markdown
Contributor

size-limit report 📦

Path Size
dist/livekit-client.esm.mjs 121.41 KB (+8.84% 🔺)
dist/livekit-client.umd.js 130.51 KB (+8.15% 🔺)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant