feat(wasmhv)+dmsg: TinyGo wasm hypervisor (net/http-free) + browser p2p transports - #6
Closed
0pcom wants to merge 54 commits into
Closed
feat(wasmhv)+dmsg: TinyGo wasm hypervisor (net/http-free) + browser p2p transports#60pcom wants to merge 54 commits into
0pcom wants to merge 54 commits into
Conversation
… can't wedge the RPC (skycoin#3166) initAddressResolver runs early and tr (transport) + cli (RPC) depend on ar, so ANY blocking call here stalls the entire visor — including the RPC, leaving the operator unable to access the visor at all. The STUN-fallback path did an UNBOUNDED <-v.stun.ready; when STUN is slow (e.g. during a deployment-side dmsg/AR hiccup) the visor hung for minutes with no RPC. Bound it to 20s and proceed without a STUN-derived public IP on timeout (same as the existing 'could not determine public IP' path). Also nil-guard v.stun.client. NOTE: partial robustness fix — the proper fix is to decouple the RPC + resolver init from ar's network-dependent steps entirely (same anti-pattern as skycoin#3143: a network dependency gating a critical surface) so a flaky deployment can never take operator control of a visor offline. Tracked as a follow-up.
…g/health identity (skycoin#3167) - Colorized /skywire.log was rendering every line one uniform dim color; rewrite colorizeLine to split per element matching logrus's console scheme: dim-gray timestamp, level-colored LEVEL + [module], BRIGHT message (the message is now the brightest element, as in the terminal), dimmed key=value fields. Non-matching lines pass through escaped; ?raw=1 untouched; all html.EscapeString -escaped (XSS test green). - Identity was redundantly shown in both places. Landing page now shows the PUBLIC KEY only; /health carries dmsg_address only (public_key omitted via omitempty). Per operator preference.
…IP) (skycoin#3169) * fix(visor): decouple RPC from AR public-IP discovery — async SetPublicIP The RPC control plane (cli) transitively depends on ar via cli→tr→ar, and the init module runner has no per-module timeout. initAddressResolver determined the visor's public IP synchronously (dmsg LookupIPGeo ≤10s + STUN ≤20s + the Phase-2c family lookup) before returning, so a slow-dmsg boot left ~30s of data-plane IP discovery wedging the entire control plane — the operator could not reach the visor over RPC for the better part of a minute (on top of the 30s dmsg-connect wait in initDmsgHTTP). Construct the AR client immediately with an empty public IP and determine the IP in a background goroutine (resolvePublicIPForAR) that pushes it in via a new APIClient.SetPublicIP. ar now completes in milliseconds, so tr→cli(RPC) bind promptly regardless of dmsg/STUN state. The data plane stays correct: BindSTCPR does a bounded wait (35s) on the asynchronously-set IP before registering, so the first bind still carries the public IP in the common case; if it times out the bind proceeds with what the AR observes from the source IP and the stcpr re-registration loop (90s) carries the real IP once it lands. All clientPublicIP/v6 access is now guarded by an RWMutex (concurrent SetPublicIP vs bind reads). Also bound the previously-unbounded <-v.stun.ready in initDiscovery (same wedge class on the launcher path) and nil-guard v.stun.client there. Validated: build, vet, race tests (addrresolver, transport/network) green. * chore(lint): justify gosec G118 nolint on async AR IP goroutine + spelling
…n#3168) * fix(dmsg): retry entry-publish on failure + bound the attempt updateClientEntryLoop unconditionally rearmed its timer at c.updateInterval (default 1 min, 5 min under dmsg.DefaultConfig) whether the previous updateClientEntry returned success or error, AND called updateClientEntry with the long-lived service ctx — no per-attempt deadline. Two pathologies follow: 1. A transient publish failure leaves the service unregistered for a full updateInterval. The first published entry is the only one visors can dial against, so for the duration of that window the service is effectively offline. 2. If the underlying PUT call itself doesn't error out cleanly (e.g. a dmsg-HTTP RoundTrip whose DialStream retries push the overall call past several internal deadlines without surfacing a terminal error), the loop body BLOCKS in updateClientEntry indefinitely. No log, no retry, no recovery. Both bite hardest on a fresh dmsg-only deployment service when the service and dmsg-discovery cold-start together via autopull: the service's first PutEntry races dmsg-discovery's own outbound dmsg sessions establishing, hits "i/o deadline reached" repeatedly, and then sits silent for 5+ minutes (often forever). Verified on transport-setup-4 with discovery_dmsg-only config after skycoin#3161 + skycoin#3157 landed: the entry-publish path is correct, only the retry cadence and attempt-bounding are missing. Two narrowly-scoped changes: - Wrap each updateClientEntry call with context.WithTimeout (entryUpdateAttemptTimeout = 30s). A genuinely stuck PUT errors out at the call boundary instead of wedging the loop, and the caller's ctx still drives shutdown. - On error from updateClientEntry, rearm the timer with exponential backoff (entryFailureBackoff: 1s, 2s, 4s, 8s, 16s, clamped at entryUpdateMaxBackoff = 30s) instead of c.updateInterval. On success, reset to c.updateInterval as before. consecutiveFailures resets on the first success. No semantic change on the happy path: a successful update still schedules the next attempt at updateInterval. The change is only in how aggressively the loop recovers from transient failures and in that a stuck attempt eventually surfaces an error rather than blocking the goroutine. * fix(lint): gosec G104 on visorlog_html.go Flush() calls Two pre-existing unhandled-error sites on develop block skycoin#3168's CI: both *bufio.Writer.Flush() calls at the body-write and trailer-write positions return an error that was ignored. Silence with `_ =` + //nolint:errcheck; the surrounding logic already drains the source reader to EOF and the response is best-effort streamed, so a flush error has no caller worth surfacing it to. * fix(lint): gosec G104 — use explicit discard for io.WriteString //nolint:errcheck silences errcheck but not gosec G104. The two io.WriteString calls in renderVisorLogHTML need an explicit `_, _ =` discard (or a real error handler); since the surrounding logic streams the response best-effort and a write error means the client is gone — caller has nothing to recover with — explicit discard is the right call. Replaces the //nolint:errcheck directive form so both linters agree. * fix(lint): satisfy both errcheck (check-blank:true) and gosec G104 The repo's .golangci.yml sets errcheck.check-blank: true, which makes errcheck flag even blank-identifier discards (`_, _ = ...`). So the prior `_, _ = io.WriteString(...)` (which silenced gosec G104) is still flagged by errcheck. Conversely, the original `io.WriteString(...) //nolint:errcheck` silenced errcheck but gosec G104 ignored the //nolint directive. The combination that satisfies both: `_, _ =` discard (gosec sees the explicit choice and stops flagging) + `//nolint:errcheck` (suppresses errcheck despite check-blank:true). Inline comment on the multi-line WriteString lives on its own line per the style the linter accepts there.
…ore init (skycoin#3170) * fix(visor): install log broadcaster before module init so --verbose works during startup v.logBcast was set at run() (visor.go:542) only AFTER NewVisor returns, but the RPC 'cli' module inits INSIDE NewVisor (InitConcurrent). A 'cli ... --verbose' log stream connecting during startup found v.logBcast nil → 'log broadcaster not available'. Thread the (already logger-hooked) broadcaster into NewVisor and set v.logBcast before InitConcurrent. Single caller updated; SubscribeLogs is nil-safe so the nil-default (tests) is unchanged. * fix(visor): nil-guard runtime-log RPCs + wire logstore before init (fixes startup segfault) cli visor log / --follow (RuntimeLogs / RuntimeLogsSince) dereferenced v.logstore, which run() wired via SetLogstore only AFTER NewVisor returns — but the RPC binds DURING module init inside NewVisor. A log query in that window hit a nil v.logstore and SIGSEGV'd the whole visor (panic at api_visor.go:432). A long-lived 'cli visor log --follow' reconnects the instant RPC binds on every restart, so it reliably lands in that window and crash-loops a freshly-started visor — and skycoin#3169 (faster RPC bind) widened the window. - Nil-guard RuntimeLogs (return "[]") and RuntimeLogsSince (empty delta): a CLI log query must never crash the visor. - Wire the logstore (and the already-moved broadcaster) into NewVisor BEFORE module init, eliminating the window so startup log queries return real data. - Regression test: nil-logstore RuntimeLogs/RuntimeLogsSince do not panic. Same init-order class as the broadcaster fix in the parent commit.
…ap seed (skycoin#3171) The RSN (pkg/router/setupnode.go) and transport-setup (pkg/transport-setup/api/api.go) post-skycoin#3161 single-client dmsg bootstrap built its direct discovery purely from conf.Dmsg.Servers. With an empty conf.Dmsg.Servers, directDisc held only synthetic client entries for the local PK and the dmsg-disc PK — no server addresses to dial — and the RegisteringFallbackDiscClient's HTTP- over-dmsg path required dmsg sessions to make HTTP calls, which it can't form without knowing a server: bootstrap deadlock, never hits dmsgC.Ready(). Mirror dmsgsrv.buildTransitDmsg's merge shape: build the seed- server set from conf.Dmsg.Servers ∪ dmsg.Prod.DmsgServers, deduped by Static PK, operator-supplied entries first. A deployment may omit conf.Dmsg.Servers entirely and still cold-start from the embedded Prod set, matching the fallback behavior dmsg-server and the visor have always had. Same null/Server-nil guard moves from the SeedEntryCache loops into a shared addServer closure.
…wedge the client (skycoin#3172) The first session's setSessionCallback was calling updateClientEntry SYNCHRONOUSLY on the session-dial goroutine with an UNBOUNDED context and only closing c.ready on success. When the publish stalled — a dmsg-only deployment service (skycoin#3157, skycoin#3168) whose only established session couldn't reach the discovery PK — the entire client wedged: - first setSessionCallback blocks → dialSession never returns → the Serve loop's EnsureSession chain freezes; - subsequent session callbacks (ready never closed) re-enter the same default branch and block on httpClient.updateMux which the first call holds; - updateClientEntryLoop's runUpdate (with the skycoin#3168 30 s attempt timeout) ALSO blocks on updateMux because mutex acquisition does NOT honor context — so the loop's retry never fires either. Production symptom: TPS-4/5/6 + SD logged "Updating entry" once at startup and went silent for hours after the .Discovery http URL was stripped from their configs. No POST ever reached dmsg-discovery. Fix has two layers: 1. setSessionCallback runs the first-session publish on its OWN goroutine with the same per-attempt timeout the entry loop uses, waits up to that bound, then closes c.ready regardless of result. The publish goroutine continues in the background; the updateClientEntryLoop retries on failure with exponential backoff. 2. dmsghttp.HTTPTransport.do installs a ctx watcher that closes the underlying stream when req.Context() is canceled, so req.Write and http.ReadResponse don't outlive their caller's timeout. Without this the dmsg.Stream's own deadlines (write: none after handshake, read: StreamIdleTimeout = 2 min) were the only bound and the skycoin#3168 per-attempt ctx had no effect on a stuck stream. Together the two changes break the cascade: the per-attempt ctx actually fires through the stream, the publish goroutine returns, the mutex releases, and the loop keeps retrying on the standard exponential backoff.
…ser) (skycoin#3173) The colorized /skywire.log view was a one-shot render that stopped at EOF. Make it a never-ending chunked response: write the backlog, then tail the file, flushing each newly-appended line until the client disconnects (detected via the request context). No Content-Length → Go uses chunked transfer-encoding, so each Flush reaches the browser over the dmsg-HTTP tunnel immediately — a live tail -f in the browser. - Flush per-line once caught up to EOF (low latency); batch-flush the initial backlog (every 256 lines) to avoid a chunk per line. - Detect log rotation/truncation (path resolves to a file smaller than our read offset) and re-open the new file, emitting an inline '--- log rotated ---'. - Tiny inline auto-scroll script (progressive enhancement): pins to the newest line while the reader is at the bottom, releases when they scroll up. - ?raw=1 (one-shot plain text) and ?follow=1 (filtered plain-text stream) are unchanged. Test: TestRenderVisorLogHTMLStreams renders backlog + an append-after-start, returns promptly on disconnect, emits no closing document. Race-clean.
…skycoin#3174) Stops route-finder, address-resolver, transport-discovery, service- discovery, uptime-tracker, and config-bootstrapper from making plain- HTTP calls to dmsg-discovery. They now route all discovery RPC over dmsg-HTTP via their bootstrap dmsg client. svcmode.Config gets a new DmsgDiscoveryDmsg field (form `dmsg://<PK>:<port>`). When set, cmdutil.BootstrapDmsg wires the disc- fallback http.Client's Transport to dmsghttp.MakeHTTPTransport(ctx, dmsgC) between dmsg.NewClient and Serve — same single-client wiring post-skycoin#3161 setup-node/transport-setup use to dodge the dual-client self-eviction storm. dmsgDisc (HTTP URL) is ignored entirely in this mode; the background server refresh also uses dmsg-HTTP exclusively with no plain-HTTP fallback — a failed refresh is preferable to a silent fallback once the operator has opted out of HTTP. Each of RF/AR/TPD/SD now reads cfg.Dmsg.DiscoveryDmsg and defaults to dmsg.DiscAddr(false) (the embedded dmsg-PK URL from deployment.Prod.DmsgDiscoveryDmsg) when omitted. The plain-HTTP discovery fallback (`if dmsgDisc == "" { dmsgDisc = dmsg.DiscURL(false) }` and the TPD variant against deployment.Prod.DmsgDiscovery) is removed — cfg.Dmsg.Discovery is now an ignored field on these service configs, preserved only for backward-compat JSON parsing. uptime-tracker and config-bootstrapper get a new --dmsg-disc-dmsg flag (default dmsg.DiscAddr(false)). The existing --dmsg-disc flag no longer defaults to the HTTP URL; operators who still need plain-HTTP discovery must set it explicitly. Both flags are passed through; when --dmsg-disc-dmsg is non-empty (which is the new default) the binary runs dmsg-only. cmdutil.BootstrapDmsg's signature gains a dmsgDiscoveryDmsg parameter. All three existing non-svcmode callers (skywire-cli config gen and the two log commands) pass "" — preserving HTTP behavior for the CLI. After this lands, deployment configs may drop their `discovery` HTTP URLs entirely; the embedded deployment config supplies the dmsg-PK URL default. The HTTP URL fields remain in the embedded config as reference data only — the deployment binaries never call them.
…sions=1 default (skycoin#3175) setupDmsgC was calling dmsg.DefaultConfig() which hardcodes MinSessions=1. The Serve loop's if MinSessions != 0 && SessionCount() >= MinSessions { wait <-errCh } gate fires the moment the FIRST seeded server's session establishes, leaving the other 5 (including the only one definitely connected to the discovery PK in single-host deployments) un-dialed. For the registering variant TPS uses, the entry never publishes because PutEntry's DialStream can't find a session that forwards to the discovery PK — dmsg error 202 forever. RSN's pkg/router/setupnode.go:83 already reads conf.Dmsg.SessionsCount via `&dmsg.Config{MinSessions: conf.Dmsg.SessionsCount}`; this aligns TPS with the same shape. Production symptom: post-skycoin#3171/skycoin#3172 deploy of TPS-4/5/6 against DiscoveryDmsg only — Serve loop took the first random seed server, parked, and the 5-minute publish-then-silent pattern recurred on the new binary even though both the deadlock and ctx-watcher fixes had landed. Concretely: sessions_count: 6 in tps.json was silently ignored.
… clients (skycoin#3177) Picks up the lingering plain-HTTP discovery paths from the post-skycoin#3163 follow-up list and converts each to either dmsg-HTTP or the embedded-prod short-circuit. ### log/log.go + log/dmsghttp_client.go BootstrapDmsg calls now route the per-PK Entry() fallback over dmsg- HTTP by default via deployment.Prod.DmsgDiscoveryDmsg. A custom --dmsg-disc http://… still works (legacy passthrough) but the prod default no longer egresses plain HTTP at bootstrap. dmsgDiscArgs in log/root.go centralizes the resolution. ### route/trace.go Replaces `rfclient.NewHTTP(traceRfURL, ..., &http.Client{Timeout}, nil)` with a POST /routes over the local visor's DmsgHTTP RPC when traceRfURL has a dmsg twin in DmsgURLForHTTP (production default). Custom --rf URLs without a known dmsg twin still fall back to the plain rfclient. Re-uses the visor's already-established dmsg sessions — no per-call dmsg bootstrap. ### svc/health.go resolveServerEntry now consults deployment.Prod.DmsgServers BEFORE the plain-HTTP discovery all_servers fetch. Every prod dmsg-server PK resolves locally; the HTTP path remains for the rare custom case. ### reward/lookup.go, ptyfs/mount_linux.go, dmsg/curl.go Each had its own ad-hoc `dmsg.NewClient(pk, sk, disc.NewHTTP(...))` ephemeral bootstrap with MinSessions=1 and a plain-HTTP discovery fallback. Replaced with cmdutil.BootstrapDmsg using the embedded prod server set + dmsg-only Entry() fallback (post-skycoin#3174 plumbing). Same Ready() / 30 s timeout shape, no behavior change for callers.
… Entry recursion (skycoin#3178) skycoin#3174 wired the dmsg-only disc fallback through dmsg-HTTP but did not add the dmsg-discovery's own PK to the direct.Client synthetic seed map. With dmsg-only routing in steady state: - dmsg.Client.DialStream(<remote-pk>) — for any unfamiliar PK or for the dmsg-discovery itself — calls discClient.Entry(<dmsg-disc-pk>) to discover an entry first. - directDClient doesn't know the dmsg-disc PK, falls through to the dmsg-HTTP fallback. - dmsg-HTTP's request needs to dial the dmsg-disc PK, which loops back through discClient.Entry(<dmsg-disc-pk>) again. The dmsg client's session cache eventually short-circuits the second roundtrip, but every fresh caller-site retriggers the lookup. Observed in prod after skycoin#3174 landed: route-finder/conf-service/uptime-tracker in tight loops at ~300-600 disc lookups/s, host-wide CPU at ~315% spread across these services. Seed the dmsg-disc PK into directDClient as a synthetic client entry delegating to every bootstrap server, so Entry(<dmsg-disc-pk>) short- circuits at the direct layer. Same trick setup-node and transport- setup already use post-skycoin#3161 via dmsgServicePKsFromConf / dmsgServicePKs.
…ot after the inner for-loop completes (skycoin#3179) The Serve loop's per-iteration background-loop start lines — updateClientEntryLoop, pingSessionsLoop, porterReapLoop — sit AFTER the inner for-loop that iterates seed entries. That was correct under the original assumption that EnsureSession returns quickly: iterate all entries, hit MinSessions on the last few, fall through to start the loops, then park on <-errCh. Post-skycoin#3172, setSessionCallback runs the first-session publish synchronously and blocks for up to entryUpdateAttemptTimeout (30 s). Phase 3 of THAT publish's own DialStream recursively dials extra sessions via EnsureAndObtainSession, pushing SessionCount past MinSessions before the original EnsureSession returns. When it finally does, the next inner-for iteration trips if ce.conf.MinSessions != 0 && ce.SessionCount() >= ce.conf.MinSessions { select { case <-ce.done: ... case err := <-ce.errCh: ... } } and parks on <-errCh — for the rest of the process's lifetime if sessions are stable. The for-loop never completes, the lines below never run, updateClientEntryLoop never starts, and the entry-publish retry path never fires. The service stays unregistered until the very first session dies. Production symptom (prod02 TPS-4 today, post-skycoin#3174/skycoin#3175 deploy): - 23:00:28 first 'Updating entry.' debug (setSessionCallback's initial publish goroutine) - 23:00:58 'Initial client entry update did not complete within bound; closing Ready and letting the loop retry.' fires (the skycoin#3172 30 s timeout) - 23:00:58 'client entry update failed for discovery; will retry next tick' debug (PutEntry returned dmsg error 202) - then 12+ minutes of silence — no 'Failed to update discovery entry (tick).' warn ever, because the loop the comment refers to was never started. 9 sessions established, all healthy. Visor health checks succeed (via dialViaConnectedServers fallback). TPS-4 just never publishes its entry. Fix moves the same updateEntryLoopOnce.Do / pingLoopOnce.Do / porterReapLoopOnce.Do block into the inner for-loop's success branch, so they fire on the FIRST successful EnsureSession instead of after the whole iteration completes. sync.Once makes the post-iteration block a no-op when this path fired. The post-iteration block is kept as a defensive net for the case where every entry's EnsureSession failed and we fell through to the outer-select retry path.
…skycoin#3180) The operator 'skywire svc setup-node health <pk>' subcommand bootstrapped its ephemeral dmsg client with disc.NewHTTP against deployment.Prod.DmsgDiscovery (plain HTTP) and dialed dmsg-discovery directly to look up the remote setup-node entry. With Caddy retired from the deploy-services HTTP path that lookup now 404s; the subcommand becomes unusable in prod even though the operator and the target setup-node are both fine. Replace with cmdutil.BootstrapDmsg in dmsg-only mode: pass dmsg.Prod.DmsgServers as the embedded seed, leave dmsgDisc empty, set dmsgDiscoveryDmsg to dmsg.DiscAddr(false). Mirrors the path the deploy services themselves take post-skycoin#3174 + the disc-seed fix in this PR — no plain-HTTP egress. Drops net/http and pkg/dmsg/disc imports (no longer used directly in this file).
…tream must not hang the hypervisor (skycoin#3181) * fix(hypervisor): start tpviz data-population async so it can't block the hypervisor servers NewHypervisor called hv.tpvizServer.Start() SYNCHRONOUSLY. Start() fetches geoip and refreshes its TPD/SD caches over the (prod, plain-HTTP) deployment URLs (tpviz.DefaultConfig: TPDURL=deployment.Prod.TransportDiscovery). When the deployment HTTP front door is closed/slow (e.g. Caddy returning 404 for the deployment services), those blocking fetches stall NewHypervisor → initHypervisor → and the hypervisor never reaches hv.Enable(ctx), so its HTTP (:http_addr) and DMSG-RPC (:dmsg_port) servers never start. Managed visors then fail to connect with 'dmsg error 306 - request has no associated listener', and 'cli visor hv status' shows disabled despite a valid, enabled config. Run tpvizServer.Start() in a goroutine: the hypervisor control plane comes up immediately; tpviz serves empty/stale until its first refresh completes. Latent since the unconditional-tpviz change (skycoin#2474); surfaced when the deployment services' HTTP was retired in favor of dmsg-only. * fix(visor): connect to uptime-tracker async so an unreachable UT can't hang init initUptimeTracker called utclient.NewHTTP synchronously. NewHTTP runs a blocking auth handshake (GET /security/nonces) with INFINITE exponential-backoff retry. When the UT service is unreachable — e.g. it is being deprecated/torn down and its dmsg endpoint returns 'dmsg error 202 - cannot connect to delegated server' — this never returns, so initUptimeTracker hangs forever. That stalls the whole 'visor' module from completing, which blocks every module that depends on it. Most visibly: the 'hypervisor' module (maker(hv, …, &vis)) waits on vis, so it never runs hv.Enable() → the hypervisor's HTTP (:http_addr) and DMSG-RPC (:dmsg_port) servers never start → managed/SBC visors fail to connect with 'dmsg error 306 - request has no associated listener', and 'hv status' shows disabled despite a valid, enabled hypervisor config. Move the entire UT connect + TPD-heartbeat-client + heartbeat loop into a background goroutine (using v.ctx, not the init ctx). initUptimeTracker returns immediately; UT is non-critical (transport re-registration also records uptime) and must never gate startup. Surfaced fleet-wide as the standalone UT is retired and the deployment HTTP front door closed.
…v ls (also fixes tpviz restart panic) (skycoin#3182) * feat(hypervisor): separate web-UI serving from the RPC/managed-visor surface `hv disable` tore down everything: the DMSG-RPC listener (managed visors connect here), the managed-visor tracking, AND the web UI — so it also killed CLI access (`hv ls` gates on IsEnabled). You couldn't shrink the public web attack surface while keeping secure CLI access to connected visors. Split the two concerns: - The hypervisor's DMSG-RPC listener + managed-visor tracking stay tied to enable/disable (`hv enable`/`hv disable`, unchanged) — the secure, whitelist-gated surface that backs `hv ls`. - The web UI (HTTP :http_addr) is now an independent toggle, gated by a new `hypervisor.ui_disable` config field (default false = served; backward-compat). New `hv ui enable` / `hv ui disable` (with -w to persist) flip ONLY the HTTP server; the DMSG-RPC + `hv ls` keep working. Implementation: extract startUI/stopUI from Enable/Disable; add a uiServing flag separate from enabled; Enable starts the UI only when !ui_disable; tpviz (which only backs UI routes) now starts/stops with the UI. RPC: EnableHypervisorUI/ DisableHypervisorUI/IsHypervisorUIServing on the API + RPC server/client + proxy/ mock. Build + vet + lint green. * feat(dmsg): track + expose no-listener port hits (cli dmsg port-hits) Companion to the hv-ui split: when a peer opens a dmsg stream to a local port with no listener bound, the session serve loop already rejected it with ErrReqNoListener (dmsg 306) and logged src_pk/dst_port. Capture those into a bounded (cap 256, evict-oldest), mutex-guarded tracker on EntityCommon keyed by (src PK, dst port) → {count, first_seen, last_seen}. Surfaced via Visor.DmsgPortHits() RPC and 'cli dmsg port-hits' (table or --json). Lets an operator see who's trying to reach a service they aren't serving — e.g. managed visors hitting a disabled hypervisor port (the exact '306 - no associated listener' case from the hv-ui discussion), misconfigured peers, or port scans — even with nothing listening to accept them. Scope: no-listener/error hits only (not all accepted traffic). Unit test covers coalesce/evict/order/nil-safety; build+vet+lint+race green. * fix(hypervisor): clearer error when hv instance isn't ready vs not configured 'hv enable/ui enable/ui disable' returned 'hypervisor not configured — add hypervisor section' whenever v.hvInstance was nil — INCLUDING when the config DOES have a hypervisor section but the instance hasn't initialized yet (the visor is still starting, or its init stalled because a dependency hung — e.g. the uptime-tracker hang fixed in skycoin#3181). Operators with a valid hypervisor config were sent off to add a section that already exists. Disambiguate: only say 'not configured' when conf.Hypervisor == nil; otherwise report 'configured but not initialized yet — the visor may still be starting up (or its init stalled); check the visor log and retry'. * feat(config): stop generating the deprecated uptime_tracker block The standalone uptime-tracker service is deprecated — uptime is now tracked by the discovery services (TPD et al.) and the service was turned off. Generated visor configs no longer carry an `uptime_tracker` block: MakeBaseConfig and the CLI `config gen` paths (dmsg-only, dual, http; unified + legacy dmsghttp) no longer create or populate it, so conf.UptimeTracker stays nil and initUptimeTracker skips it. Removing the block exposed two unguarded conf.UptimeTracker.Addr derefs that would nil-panic a visor booted without it: - pkg/visor/survey.go:84 (system_survey goroutine — crashed on boot) - cmd/skywire-cli/commands/survey/root.go (cli survey) Both are now nil-guarded, matching the other readers (api_services, embedded_dmsgweb, init_dmsg, init_apps) which already guard it. Validated on the local hypervisor: clean boot against a UT-less config, survey generated without panic, managed visors reconnected.
… / N visors (skycoin#3176) * fix(e2e): make all e2e configs dmsg-only — strip every http:// service URL The e2e deployment + visor configs still carried plain-HTTP service URLs (discovery/transport_discovery/address_resolver/route_finder/uptime_tracker/ service_discovery) even though the network and the deploy-service binaries are now dmsg-only (skycoin#3161/skycoin#3171/skycoin#3174). The setup-node/transport-setup, which build their dmsg client from conf.Dmsg.DiscoveryDmsg, couldn't bootstrap with only 'discovery: http://…' set, deadlocking the shared 'skywire svc run' supervisor → 'deployment-services unhealthy' → linux e2e red on develop and every PR. Convert ALL e2e configs to dmsg-only (zero http:// URLs), matching the live network: - Visor configs (visorA/B/C + -internal): drop dmsg.discovery, transport.discovery, transport.address_resolver, routing.route_finder, uptime_tracker.addr, launcher.service_discovery; set the corresponding *_dmsg fields (dmsg://<pk>:80, PKs from services-config.json). geoip → . - setup-node.json / transport-setup.json: discovery_dmsg + servers (the e2e dmsg-server) + drop http discovery; setup-node transport_discovery → _dmsg; transport-setup sessions_count 0→1 (honored after skycoin#3175). - services.json (tpd/rf/sd/ar): discovery_dmsg, drop http discovery (deploy-services dmsg-only via skycoin#3174). - dmsg-server.json: discovery_dmsg (dmsgsrv is strict dmsg-only), drop http. - services-config.json: drop the http_* discovery URLs in test+prod, keep _dmsg. All PKs cross-checked against dmsg-server.json + services-config.json + derived from the service SKs. Zero http:// remain; JSON valid; no new null fields. Depends on skycoin#3171/skycoin#3172/skycoin#3174/skycoin#3175 (all merged except skycoin#3171/skycoin#3172) for the binaries to bootstrap dmsg-only. * fix(e2e): consolidate to one redis + drop deprecated uptime-tracker/postgres/network-monitor Slim the e2e container set to '1 redis + 1 deployment-services + N visors': - Five per-service redis containers (dmsgd/ar/sd/tpd/ut-redis) → a single 'redis'. Each discovery service uses its own logical DB to preserve the keyspace isolation the separate instances gave (go-redis ParseURL honors the redis://host:6379/<N> DB selector): /0 dmsg-discovery /1 transport-discovery + route-finder (shared — RF reads TPD's transport data) /2 service-discovery /3 address-resolver - Drop the standalone uptime-tracker service + its postgres + ut-redis — uptime tracking is now integrated into every discovery service (TPD et al.), so 'skywire svc ut' is deprecated. Removed from compose, the e2e-run Makefile stages, env_test's expected-container list, the visor configs (uptime_tracker block) and services-config.json (uptime_tracker_dmsg). - Drop network-monitor: already disabled (never started by e2e-run, excluded in env_test) and its 'svc nm' entrypoint monitored services over plain HTTP incl. the now-removed uptime-tracker. pkg/network-monitor code untouched. e2e-run now brings up redis + deployment-services + visor-a/b/c (was 11 containers → 5, + the transient e2e-test runner). compose config valid; integration test compiles; the only http:// left is the container-local :6094/debug/service healthcheck.
…nto the router (skycoin#2607 stage-4a) (skycoin#3183) The faithful-UDP stack (DatagramPacket wire type skycoin#2609, DatagramRouteGroup skycoin#2610, per-datagram AEAD skycoin#2612, UDPBridge/appnet API skycoin#2613/skycoin#2614) shipped its components but the router-side dispatch was never wired: a DatagramPacket arriving off a transport fell through handleTransportPacket's switch to ErrUnknownPacketType and was dropped, so the unit-tested DatagramRouteGroup.Handle was unreachable in a running visor. This wires the receive + relay path: - router gains an rgsDatagrams map (keyed like rgsNs) + datagramRouteGroup / setDatagramRouteGroup / removeDatagramRouteGroup accessors. Datagram groups live in their own map so a DatagramPacket is never confused with a reliable RouteGroup on the same descriptor. - handleTransportPacket dispatches DatagramPacket to handleDatagramPacket: intermediary/forward rules relay to the next hop; a consume rule delivers to the registered DatagramRouteGroup.Handle. No parking — a datagram that races route-group registration is dropped (faithful-UDP loss tolerance), unlike the reliable path which parks handshake frames. A closed group de-registers itself. - forwardPacket relays DatagramPacket: re-stamp the next-hop route ID, pass the opaque AEAD-sealed payload through unchanged (the seal is end-to-end; intermediaries never decrypt). Inert until route-setup constructs+registers datagram groups (stage-4b) — nothing emits a DatagramPacket yet, so production behavior is unchanged. New datagram_dispatch_test.go drives real DatagramPackets through handleTransportPacket (deliver / drop-when-unregistered / unknown-route-errors / closed-group-deregisters).
…tup (skycoin#2607 stage-4b) (skycoin#3184) * feat(router): datagram AEAD master-key derivation from Noise ChannelBinding (skycoin#2607 stage-4b foundation) First piece of the datagram route-setup integration, per the chosen key-exchange design: the datagram sibling of a route keys itself off the reliable route's already-completed Noise session instead of running a second handshake. - dmsg/noise: export Noise.ChannelBinding() — the handshake hash, identical on both peers and bound to the full transcript (incl. PQ-hybrid payload). Already used internally for the hybrid binding; now reachable to key a sibling channel. - router: deriveDatagramMasterKey(channelBinding) HKDF-expands the binding into the 32-byte master key NewDatagramCipher consumes, domain-separated ("skywire-datagram-master-v1") from the per-direction subkey labels. Both peers derive an identical key because the binding is identical. Pure + unit-tested (deterministic across peers, distinct per session, empty-binding rejected, and an end-to-end seal/open round-trip through NewDatagramCipher). Not yet wired into route setup — that (threading the binding out of EncryptConn, building + registering the datagram sibling on both ends, app DialPacket + UDPBridge) is the remainder of stage-4b. * feat(router): build the faithful-UDP datagram sibling during route setup (skycoin#2607 stage-4b) With dispatch wired (stage-4a, skycoin#3183), route setup can now construct the DatagramRouteGroup so datagrams actually flow. Design: a datagram route is NOT a separate route — the reliable route is set up as usual (Noise KK handshake → RouteGroup), and when both ends locally intend datagram mode, each additionally builds a datagram sibling over the SAME rules/transport, keyed off the reliable session's Noise ChannelBinding (no second handshake). Key exchange (chosen approach — one-time Noise → HKDF): - dmsg/noise: export Noise.ChannelBinding() + delegate through ReadWriter/Conn, so network.EncryptConn's returned conn exposes the session binding. - router.deriveDatagramMasterKey(binding): HKDF-SHA256 the binding into the 32-byte master both peers feed to NewDatagramCipher (domain-separated from the per-direction subkey labels). Identical binding on both ends → identical key. On-demand by LOCAL intent (no wire negotiation — see the design discussion): - dial side: DialOptions.Datagram (set by the DialPacket / forwarded-UDP dialer); - accept side: RegisterDatagramPort marks the local serving port. saveRouteGroupRules gains a `datagram bool`; on the encrypt path it calls buildDatagramSibling, which derives the master, builds the out/in cipher pair (AAD binds to the receiver's PK; verified by a seal→dispatch→open round-trip test), and registers the sibling. Best-effort: any failure logs + skips, never touching the reliable route. Auto-every-route was rejected (per-route memory + attack-surface tax for a niche feature). Lifetime: the sibling has no keep-alive of its own (a receive-only group never writes), so it's coupled to the reliable route — closeDatagramSibling reaps it from the close-packet path, removeRouteGroupOfRule, and the GC sweep (+ a defensive self-closed sweep). Still inert end to end until stage-4c wires SkywireNetworker.PacketNetworker + the forwarded-ports UDP listener loop (NewUDPBridge) + RegisterDatagramPort. Tests: master-key determinism/interop, port registry, and a full seal→handleTransportPacket→AEAD-open→ReadFrom round-trip. Build/vet/lint green; full router suite passes.
…skycoin#2607 stage-4c) (skycoin#3185) Completes faithful UDP: with dispatch (4a) + route-setup sibling construction (4b) in place, this wires the two endpoints so datagrams flow app-to-app. Client (dial) path: - router.DialRoutesDatagram: dial a normal route with Datagram intent, then return both the reliable conn (kept alive — closing it tears the route down) and the registered DatagramRouteGroup sibling. Added to the Router interface (+ MockRouter). - appnet.SkywireNetworker.DialPacketContext (implements PacketNetworker): dials via DialRoutesDatagram and returns a skywirePacketConn — the sibling wrapped so Close also tears down the reliable route and frees the local port. This makes appnet.DialPacket functional for skynet addrs (was ErrPacketNetworkerNotSupported). Server (accept) path: - router gains an accept channel: buildDatagramSibling offers each accept-side sibling (Initiator=false) to AcceptDatagram along with the route's local port. Register/UnregisterDatagramPort + AcceptDatagram added to the Router interface. - visor serveUDPForwards accept loop (started from initSkynetForwardPorts) drains accepted siblings and, per the forwarded_ports.udp registry, bridges each to its local UDP service via NewUDPBridge (SetLocalPeer = the service; poll IsAlive to reap on route close). - RegisterForwardedPort(UDP) → RegisterDatagramPort; DeregisterTCPPort(UDP) → UnregisterDatagramPort; startup restores datagram intent for persisted UDP ports. Both ends opt in by LOCAL intent (DialOptions.Datagram on the dialer; the server's forwarded UDP port registration) — no wire negotiation. NewUDPBridge is no longer dead code; appnet.DialPacket works. Tests: accept-side-offers-but-dial-side-doesn't; existing seal→dispatch→open round-trip + port registry. Build/vet/lint green; full router + appnet suites pass. Remaining: two-visor UDP-echo live validation.
…stage-4d) (skycoin#3186) Adds the trigger layer so faithful UDP is actually usable + testable end to end (previously nothing registered a forwarded_ports.udp or called DialPacket). Server: `skywire cli skynet port add <port> --udp` registers a UDP datagram service over skynet (sets ForwardedPort.UDP → RegisterDatagramPort → the accept loop bridges inbound datagram routes to the local UDP service). Client: `skywire cli skynet port udp-dial <remote-pk> <remote-port> [--local-port L]` dials a remote forwarded_ports.udp service and bridges it to 127.0.0.1:L — the operator's UDP app talks to L and datagrams reach the remote service (replies come back), loss-tolerant + unordered. Backed by a new visor RPC `DialUDPForward` that ResolveNetworker(skynet)→PacketNetworker→DialPacketContext and runs a client-side UDPBridge, tracked in v.udpClientBridges and torn down by `udp-stop` / visor shutdown. `udp-ls` lists active client forwards. RPC plumbing: DialUDPForward / StopUDPForward / ListUDPForwards on the API interface + gateway (UDPForwardIn) + rpc client + mock + proxy stubs (not routed via hypervisor proxy — use direct --rpc). Build/vet/lint green; CLI smoke-checked (`port --help`, `port add --udp`, `udp-dial --help`). Ready for fleet live-validation once boards auto-update.
…rams (skycoin#2607) (skycoin#3187) * feat(transport): skywire-PK-bound QUIC TLS identity (option A, skycoin#2607 QUIC follow-on) QUIC mandates TLS 1.3 but skywire authenticates with secp256k1 keys, not a CA. Following the libp2p-tls pattern, each side presents a self-signed cert with an ephemeral ed25519 key, and a custom X.509 extension carries the skywire PK plus a signature (by the skywire SK) over the TLS key. VerifyPeerCertificate checks the binding and pins the expected remote PK, so a wrong-PK peer is rejected at the TLS layer before route setup. Self-contained + unit-tested (bind/recover, dialer pinning, listener learns peer PK, tampered-sig + missing-extension rejected). Foundation for the QUIC skynet transport (reliable streams first, then RFC 9221 datagrams to make faithful-UDP wire-real). * test(transport): in-process QUIC + option-A integration proof; disable TLS resumption Proves the QUIC transport core works end to end before the framework/AR wiring: a real quic.Dial/quic.Listen connection secured by the skywire-PK-bound TLS identity mutually authenticates (client pins the server PK, server learns the client PK), rejects a wrong pinned PK at the handshake, and round-trips bytes on a stream. Also hardens quicTLSConfig with SessionTicketsDisabled so the full handshake — and thus the VerifyPeerCertificate identity check — runs on every connection (a resumed session could otherwise skip it). quic-go promoted to a direct dependency. * feat(transport): QUIC skynet transport — reliable streams (skycoin#2607 QUIC follow-on) Adds a QUIC-over-UDP transport type, mirroring sudph/stcpr (AR-resolved) but riding quic-go with the skywire-PK-bound TLS identity (option A). One QUIC stream carries one skywire transport; the existing handshake + noise layers run over the stream exactly as for the other types, so it slots into the transport framework unchanged. - types.QUIC ("quic") + ClientFactory case. - quic.go: quicClient (Dial → AR resolve → quic.Dial on an ephemeral UDP socket → OpenStreamSync → wrap; Start/listen → UDP socket → quic.Listen → per-conn stream-accept → reuse genericClient.initTransport), quicStreamConn (adapts a quic.Stream to net.Conn with Local/RemoteAddr from the conn), quicListener (net.Listener over *quic.Listener, concurrent stream-accept), AR re-register loop mirroring STCPR. - addrresolver.BindQUIC (HTTP POST /bind/quic, mirrors BindSTCPR) + APIClient interface entry — registers the visor's QUIC UDP port for peers to Resolve. - Transport.QUICPort config (opt-in: 0 disables; a fixed port keeps the AR registration + firewall rule stable) + initQuicClient module wired into the visor init graph. Datagrams (quic.Config.EnableDatagrams is already on) ride a follow-up tp.WriteDatagram path that makes faithful-UDP wire-real. The AR SERVER /bind/quic handler (mirror of /bind/stcpr) is a separate change needed for QUIC to be resolvable on the live network; until then a visor listens on QUIC but isn't AR-discoverable. Build/vet/lint green; identity + in-process conn tests pass. * feat(address-resolver): /bind/quic endpoint so QUIC transports are resolvable (skycoin#2607 QUIC follow-on) Refactors the STCPR bind handler into a type-parameterized bindForType (STCPR behavior unchanged — bind() now just calls bindForType(STCPR) and still mirrors to the v6 AR) and adds POST /bind/quic → bindForType(QUIC), storing the visor's QUIC UDP address under the "quic" type for peers to Resolve(quic, pk). Same validation as STCPR (auth, public-IP / declared-PublicIP, hasAddress, v6 declaration). No v6 mirror for QUIC in v1. With this deployed, a visor configured with quic_port registers its QUIC address and becomes dialable over QUIC by other QUIC-enabled visors. (Behind-NAT UDP-port discovery / hole-punching is a follow-up; this targets publicly-reachable nodes.) Also adds BindQUIC to the AR MockAPIClient. AR api tests pass; build/vet/lint green. * feat(transport): faithful UDP over QUIC datagrams — wire-real (skycoin#2607 QUIC step 2) Makes the merged faithful-UDP path actually behave like UDP on the wire (no head-of-line blocking, genuine loss tolerance) when the underlying transport is QUIC, by carrying DatagramPackets over QUIC's RFC 9221 datagram channel instead of the reliable stream. Falls back transparently to the reliable stream on non-QUIC transports (stcp/sudph/dmsg) or oversized datagrams, so it's always correct. - network: DatagramConn interface { WriteDatagram, ReadDatagram }; *quicStreamConn implements it over quic.Conn SendDatagram/ReceiveDatagram. transport.encrypt captures the raw conn's datagram channel BEFORE the (stream-only) noise wrapper replaces it, exposed via the concrete *transport.Datagram() (not the Transport interface, so reliable transports + mocks need no change). The datagram rides the raw QUIC conn (its TLS = hop-by-hop encryption; payload already end-to-end AEAD-sealed by the DatagramRouteGroup). - transport: ManagedTransport.WriteDatagram (native datagram with reliable fallback) + datagramReadLoop, started from Serve for datagram-capable transports, draining QUIC datagrams into the same readCh the router consumes — so the existing DatagramPacket dispatch handles them. Drops on a full readCh (faithful-UDP loss tolerance). - router: DatagramRouteGroup.WriteTo now uses tp.WriteDatagram. Tests: real-QUIC datagram round-trip through quicStreamConn; router + transport suites green. Build/vet/lint clean. With this, two QUIC-enabled visors carry faithful UDP as true unreliable datagrams end to end.
skycoin#2607) (skycoin#3188) * feat(dmsg/disc): advertise optional QUIC (UDP) server endpoints (skycoin#2607 dmsg-over-QUIC foundation) Adds Server.AddressUDP / AddressUDPV6 to the discovery entry so a dmsg server can advertise QUIC (UDP) endpoints alongside its TCP Address(es), paired with Entry.Protocol="quic". QUIC-capable clients will dial these for a session with native QUIC stream multiplexing + an unreliable datagram channel; clients and servers that don't know QUIC ignore the new fields (omitempty) and use the TCP Address exactly as today — fully backward-compatible (a QUIC server dual-listens TCP+UDP). Server.String() surfaces the UDP endpoint for diagnostics. Additive foundation only — the session-layer rework (QUIC SessionManager backend, noise-free object exchange over QUIC streams, dial/accept wiring, datagram API) is the next change. * feat(dmsg): SessionManager QUIC backend + noise-free object exchange (skycoin#2607 dmsg-over-QUIC) Additive session-layer groundwork for dmsg-over-QUIC: - SessionManager gains a quic *quic.Conn backend (exactly one of yamux/smux/quic is set); Close handles the QUIC case (CloseWithError). - writeObject/readObject become noise-optional: when sc.ns == nil (a QUIC session) the signed object is written/read straight off the already-TLS- encrypted QUIC stream, with no per-object Noise encrypt/decrypt or nonce window. The object stays SIGNED, so dmsg-level authentication is unchanged; QUIC's PK-bound TLS provides the transport encryption (option A). Inert until the dial/accept paths construct QUIC sessions (next) — existing TCP+Noise+yamux/smux sessions are unaffected (the new branches only trigger on ns==nil / sm.quic!=nil). Builds clean. * feat(dmsg): Stream carries a native QUIC stream (skycoin#2607 dmsg-over-QUIC) The dmsg Stream now supports a third underlying mux stream — *quic.Stream — alongside yamux/smux, collapsed behind a muxStream() helper + muxStreamConn interface so the stream protocol (writeObject/readObject, deadlines, Close, nsConn) is transport-agnostic. newInitiatingStream/newRespondingStream open/accept a QUIC stream when the session is QUIC (sm.quic). Crucially, the END-TO-END client↔client Noise (s.ns / nsConn) wraps the QUIC stream exactly as it wraps yamux/smux — so client↔client confidentiality through the relay is identical across transports (QUIC TLS only secures the client↔server hop). Additive: the quic branches only trigger for QUIC sessions; existing yamux/smux behavior is unchanged (dmsg unit tests pass). * refactor(skyquic): extract PK-bound QUIC TLS identity to a shared package (skycoin#2607) The skywire-PK-bound QUIC TLS helpers (option A) lived in pkg/transport/network, but that package imports pkg/dmsg/dmsg (the dmsg transport), so dmsg-over-QUIC couldn't reuse them without an import cycle. Moved them to a new pkg/skyquic (depends only on crypto/* + pkg/cipher), exported as NewCertificate / TLSConfig / VerifyCert. network/quic_identity.go is now thin wrappers so the merged skynet QUIC transport + its tests are unchanged; the identity unit tests moved to pkg/skyquic. Both packages build; all QUIC + identity tests pass. * feat(dmsg): QUIC session dial + server-session accept (skycoin#2607 dmsg-over-QUIC) - session_common: initQUIC sets up a session over a *quic.Conn with no Noise handshake (QUIC TLS authenticated the peer PK + encrypts the hop); quicAddrConn satisfies the addr-only netConn accessors. - client: dialSession picks QUIC when the server advertises Protocol "quic" + AddressUDP — dialSessionQUIC does quic.Dial with the PK-bound TLS (pkg/skyquic, pins the server PK), then makeClientSessionQUIC (no Noise). The shared newest-session-wins storing logic is reused for both transports. - server: makeServerSessionQUIC + a QUIC AcceptStream branch in ServerSession.Serve (mirrors yamux/smux); serveStream's read-deadline assertion broadened to cover QUIC streams (which have SetReadDeadline but aren't net.Conns). Additive — QUIC paths only trigger for QUIC sessions; existing TCP+Noise+yamux/smux behavior unchanged (dmsg unit tests pass). Remaining: the server QUIC listener (ServeQUIC/handleQUICConn) + AddressUDP advertisement + server-binary wiring. * feat(dmsg): server QUIC listener + advertisement (skycoin#2607 dmsg-over-QUIC) - Server.ServeQUIC(udpConn, advertisedUDPAddr): QUIC listener over the given UDP socket using the PK-bound TLS (pkg/skyquic, option A), run alongside the TCP Serve. Each accepted conn → handleQUICConn, which authenticates the peer PK from its QUIC TLS certificate (quicPeerPK → skyquic.VerifyCert), builds a QUIC server session (no Noise), and serves it — mirroring handleSession's lifecycle (setSession / Serve / peer cleanup / delSession). - EntityCommon.advertisedUDPAddr (set by ServeQUIC) → updateServerEntryOnEndpoint publishes Server.AddressUDP + Protocol "quic" so QUIC-capable clients dial QUIC; TCP-only servers/clients are unchanged (omitempty + dual-advertise). dmsg-over-QUIC is now complete at the library level: discovery, the shared skyquic identity, the QUIC SessionManager backend, native-QUIC-stream dmsg streams (end-to-end client↔client Noise preserved), client dial, and server accept. Full build green; pkg/dmsg/dmsg + disc tests pass (the only failure in ./pkg/dmsg/... is the pre-existing Redis-needs-localhost store test). Remaining: wire the dmsg-server binary to call ServeQUIC + optional session datagram API. * feat(dmsg): dual-listen TCP+QUIC in the server binary + client TCP fallback (skycoin#2607) - dmsgserver.ListenAndServe now also binds a UDP listener (same port number as TCP) and runs Server.ServeQUIC alongside the TCP Serve — best-effort + additive (a QUIC bind failure leaves TCP serving). Deployed servers that update to this code dual-listen and advertise their QUIC endpoint automatically. - Client: when a QUIC dial fails (e.g. UDP blocked by a firewall) the client falls back to the server's TCP endpoint, which a QUIC-advertising server also listens on. Robust for UDP-hostile networks; behaves exactly like a normal TCP dial on fallback. dmsg-over-QUIC is now COMPLETE end to end: a QUIC-capable visor dials a QUIC-advertising dmsg server over native QUIC streams (no Noise handshake; PK-bound QUIC TLS + preserved end-to-end client↔client Noise), with full backward compatibility (omitempty discovery fields, dual-listen, TCP fallback). Build/lint green; dmsg unit tests pass. Follow-up: expose a session-level datagram API (QUIC SendDatagram/ReceiveDatagram) + a two-server live test. * feat(dmsg): session-level datagram channel over QUIC (skycoin#2607 dmsg-over-QUIC) Exposes the payoff of dmsg-over-QUIC: SessionCommon.WriteDatagram / ReadDatagram / SupportsDatagrams give dmsg a genuine unreliable datagram channel (QUIC RFC 9221 SendDatagram/ReceiveDatagram) — UDP-over-dmsg — riding the same PK-bound-TLS- encrypted QUIC connection as the session's streams. Non-QUIC (TCP+yamux/smux) sessions return ErrDatagramsUnsupported (they carry reliable streams only), resolving the coupling: dmsg can faithfully carry UDP only when it runs over a datagram-capable transport, which is now QUIC. Test covers the non-QUIC contract; the round-trip is exercised by the two-server live test. This completes dmsg-over-QUIC end to end. * fix(skyquic): actually commit the extracted package + quic_identity wrappers (skycoin#2607) The earlier skyquic-extraction commit silently dropped its payload: the 'git add ... quic_identity_test.go' errored on the already-removed test, which aborted the whole git add, so pkg/skyquic/ and the quic_identity.go thin wrappers were never staged. HEAD kept the old full quic_identity.go and had no skyquic package — but the dmsg-over-QUIC commits import pkg/skyquic, so the pushed branch didn't compile and CI failed at the lint/build step. (Local builds passed only because the files existed untracked in the working tree.) This commits pkg/skyquic (identity.go + identity_test.go) and the quic_identity.go wrappers. Whole-repo build + CI-config golangci-lint on the dmsg/skyquic/transport packages are clean.
…e URLs (skycoin#3189) * feat(dmsg): dmsg-over-WebSocket transport Add WebSocket as a third dmsg session transport alongside TCP and QUIC. A WebSocket is a bidirectional, ordered, reliable byte pipe over HTTP(S), so coder/websocket's NetConn adapts it to a net.Conn and the EXISTING Noise+yamux session stack runs over it unchanged — no new session semantics, mux, or crypto. The per-hop Noise handshake and end-to-end per-stream noise are both preserved; WS only changes how the bytes are carried. Why: a browser tab (JS or Go js/wasm) cannot open a raw TCP or UDP socket, so neither the legacy TCP transport nor dmsg-over-QUIC can reach the mesh from a browser. WebSocket is the one transport the sandbox allows, making a future WASM dmsg client — a real PK-authenticated leaf node in a web page — possible. It is also the universal fallback for restrictive networks that only permit HTTP(S)/443 and for CDN fronting. coder/websocket is used because it also compiles to js/wasm, so the same dial path serves the browser client. - disc.Server.AddressWS advertises a ws:// or wss:// URL; orthogonal to Protocol (WS reuses yamux), so a server can advertise TCP + QUIC + WS at once, each dialed by the clients that can use it. - Server.ServeWS serves WS on a plaintext listener (TLS terminated by a front proxy); the WS handler hands the NetConn to the shared handleSession path. - Client dials WS when Config.PreferWS is set and the server advertises AddressWS, with TCP fallback; forced on in the js/wasm build later. - dmsg-server service gains optional ws_address + public_address_ws config (off by default, additive). - ws_test.go: full end-to-end test — a server listening ONLY over WS, two PreferWS clients, a stream bridged between them. * fix(tpviz,config): stop fetching clearnet http; centralize service URLs on services-config.json The tp-viz UI was auto-refreshing transport/uptime data over hardcoded clearnet http (tpd.skywire.skycoin.com/all-transports, ut.skywire.skycoin.com/uptimes) — both now 404 (TPD http frontend deprecated, uptime tracker decommissioned), spamming "Auto-refresh failed" warnings. tp-viz: - When embedded in a visor, source all-transports from the visor's dmsg/CXO-backed transport-discovery feed (new VisorAPI.AllTransports → Visor.FetchAllTransportsCXO) — never clearnet. On a CXO cache miss, surface 503 and let the UI retry rather than hitting a dead endpoint. - Drop the standalone uptime tracker entirely (decommissioned): UTURL defaults empty, handleUptimes serves an empty map, refreshCache and the --ut-url CLI default no longer point at a dead endpoint. - index.html: drop the hardcoded https://tpd.skywire.skycoin.com file:// fallback; always use the serving origin's relative /api. Hardcoded-URL audit — services-config.json is the single source of truth for deployment URLs; remove dead duplicates from code: - Delete pkg/skyenv/values.go entirely. Every URL constant (prod+test) was already dead (consumers use deployment.Prod / gen.go services.* / dmsg.DiscAddr), and the PK lists (route_setup_nodes, transport_setup, survey_whitelist, reward_system) + stun_servers + dns_server all duplicate services-config.json. The one live ref (visor ip → STUN) is repointed to deployment.Prod.StunServers. - rewards: drop the hardcoded tpd.skywire.skycoin.com fallback. - network-monitor: --sd-url defaults to deployment.Prod.ServiceDiscovery. * fix(cli): visor ip reports the visor's already-known public IP, not a fresh STUN round The visor resolves its public IP + NAT type via STUN once at startup (for SUDPH/STCPR + the address resolver) and holds it in its Overview. `cli visor ip` was running its OWN STUN round — redundant work that could even disagree with the visor's own view. Now it just asks the running visor for what it already knows. - Add NATType to Overview (the visor's STUN-classified NAT type string), alongside the existing PublicIP. - `visor ip` calls rpcClient.Overview() and prints PublicIP + NATType. - Standalone fallback (no visor reachable) still runs a local STUN round with the embedded default servers — the only option without a visor.
…in#3190) * feat(dmsg): compile the dmsg client to js/wasm + a browser WASM client Make the dmsg client buildable for the browser (GOOS=js GOARCH=wasm) and add a WASM entrypoint that exposes a dmsg API to JS. This is the foundation for a browser-native dmsg client — chat, WebRTC signaling, and (the real target) a zero-install hypervisor UI that talks to the fleet over dmsg. Build-tag retargeting (js/!js -> tinygo/!tinygo): - The js stubs in pkg/logging, pkg/dmsg/disc and deployment existed because TinyGo lacks the reflect/encoding-json runtime helpers — NOT because the js platform can't run the real code. Standard Go js/wasm has the full stdlib and runs logrus/encoding-json/gob fine. Retargeting the constraints to tinygo/!tinygo gives standard-Go-js the full implementations (so the dmsg client, which uses logrus.FieldLogger + disc.Entry.Sign + the HTTP discovery client, compiles) while the TinyGo install-page build keeps its minimal stubs unchanged. Standard Go is also required regardless because the hypervisor management channel is gob-RPC (reflection-heavy). - deployment/cmd/gen emits //go:build tinygo for the generated data file. cmd/dmsg-wasm: - main.go: standard-Go js/wasm client. Forces Config.PreferWS (the browser sandbox has no raw TCP/UDP — WebSocket is the only transport). Inbound still works: once connected to a dmsg server over WSS, peers dial this client BY PK and the server bridges the streams back down the same connection, so a tab is a first-class inbound-reachable peer. Exposes globalThis.skywireDmsg with connect / dial / listen and per-stream send / onMessage / close. - index.html: dev harness (connect, dial, exchange messages). - Makefile: `dmsg-wasm` (dev build) and `dmsg-wasm-inline` (single self-contained .html with wasm_exec.js + base64-inlined wasm — runs from file://, no static host, no fetch). Native + existing wasm builds unaffected; native logging/disc/deployment/dmsg tests pass. (Standard-Go wasm is ~19MB / ~5MB gzipped — fine for a hypervisor app; size optimization is a follow-up.) * feat(dmsg): seeded WS bootstrap for the browser client (dmsg-only discovery) A browser can't reach a dmsg-only discovery until it has a server (the HTTP discovery frontend is gone). StartDmsgSeeded solves the chicken-and-egg: 1. Seed one server's {PK, AddressWS} into a direct.Client so the dmsg client connects straight to it over WebSocket (PreferWS), no HTTP discovery needed. 2. Once the session is live, upgrade discovery to a registering fallback over dmsg (dmsghttp) so the client registers its own entry in the real discovery (→ inbound-reachable by PK) and resolves arbitrary peers. The discovery's own client entry is preloaded too (delegated to the seed servers) — without it, the first reach to the discovery over dmsg recurses resolving the discovery's own location until the stack overflows. Validated against the live WS-enabled prod02 server (0371ab4b) with a WS-ONLY seed (no TCP fallback): connects over WebSocket and "Initial post entry succeeded" — registered in the real dmsg-discovery over dmsg. Wired into cmd/dmsg-wasm connect(sk, seedPK, seedWsURL, discDmsgAddr); harness pre-filled with the live server. * feat(dmsg-wasm): HTTP-over-dmsg fetch primitive (browser hypervisor-UI transport) Add skywireDmsg.fetch(pkHost, method, path, body) — an HTTP request over dmsg to dmsg://<pk>:80<path>, built on dmsghttp.MakeHTTPTransport over the seeded session. This is the transport a browser hypervisor UI uses to reach any visor/hypervisor BY PUBLIC KEY (no clearnet, no exposed port). Validated against the live network: GET /health over dmsg returned 200 + real build_info. Harness gains a fetch demo. * feat(hypervisor): optionally serve the UI/API over dmsg (DmsgUIPort) Serve the hypervisor's HTTP handler (hv.HTTPHandler() — full web UI + REST API) over a dmsg listener when HypervisorConfig.DmsgUIPort is set, mirroring how the tpviz UI is already served over dmsg. This lets a browser dmsg client (the WASM hypervisor UI) reach the fleet BY PUBLIC KEY with no exposed HTTP port, end-to-end over dmsg. Opt-in (0 = off) since it exposes fleet control over dmsg; the handler's own login/CSRF auth applies and the dmsg Noise layer authenticates the caller PK. Completes the server side of the browser-hypervisor chain: browser wasm client → HTTP-over-dmsg → this listener → hv API. * feat(dmsg-wasm): Service-Worker hypervisor UI over dmsg (the browser port) Make the UNMODIFIED hypervisor Angular UI load in a browser entirely over dmsg, reached BY PUBLIC KEY. A Service Worker (sw.js) runs the WASM dmsg client and transparently proxies every in-scope navigation + fetch over dmsg to the hypervisor's DmsgUIPort listener via skywireDmsg.fetch — index.html, the JS/CSS bundles, /api/*, /assets/* all arrive over dmsg; the page origin only serves the four local assets. hv.html is the bootstrap (registers the SW, then navigates to "/"). jsFetch is now binary-safe (Uint8Array body) + forwards request/response headers (cookies/CSRF/content-type) so arbitrary assets + authenticated requests proxy correctly. Makefile: `dmsg-wasm-hv` builds the bundle. PROVEN in a real headless browser against live infrastructure: wasm connects to the live prod02 server over ws://, registers in dmsg-discovery, fetches the hypervisor /api/ping over dmsg (200 "PONG!"), and the SW loads the full hypervisor Angular index (app-root present) over dmsg. End-to-end: browser → wasm dmsg client → ws:// → dmsg → hypervisor UI+API by PK. Note: a production browser (https/file:// page) needs the seed server on wss:// (secure context can't open ws:// to a remote); the headless test used --disable-web-security to allow plain ws://. * fix(tpviz): source deployment discovery over dmsg, gate dead clearnet fetches The standalone tp-viz (reward UI) auto-refreshed TPD/SD/DMSG discovery over the clearnet HTTP frontends, which are gone on the dmsg-only deployment → continuous 404 spam ("[tp_viz] Auto-refresh failed ... http://dmsgd.skywire.skycoin.com/..."). - tp-viz gains SetDmsgHTTPClient + dmsg:// discovery bases (deployment.Prod.*Dmsg). When a dmsg client is set, TPD/SD/DMSG discovery is fetched over dmsg. - When neither a dmsg client nor a visor API is available (dmsg-only, no source), the clearnet deployment fetches are GATED (skipped) instead of hammering dead URLs. AllowClearnetDisc forces the legacy behavior. - The reward UI starts an embedded dmsg client (StartDmsgEmbedded — seeded from the deployment's dmsg servers, discovery over dmsg, no clearnet) and hands tp-viz an HTTP-over-dmsg client. Validated: connects over dmsg, registers, logs "deployment discovery now sourced over dmsg"; the [tp_viz] 404 spam is gone. - StartDmsgSeeded gained a preferWS flag (wasm=WS, native=TCP); StartDmsgEmbedded is the clearnet-free native bootstrap from the embedded server set. * fix(dmsg,rewards): drop dead clearnet-http discovery flags + bootstrap The deployment is dmsg-only; the clearnet dmsg-discovery HTTP frontend is gone (404), so flags/paths that connect to discovery over plain http no longer work. - dmsg CLI (dmsg curl/dial/web/probe/http via dmsgclient.InitFlags): remove the -Z/--http ("use regular http to connect to DMSG Discovery") flag + its branch in InitDmsgWithFlags (it Fatal'd on the 404), and the -U/--disc-url clearnet URL flag. Discovery is reached over dmsg via -A/--disc-addr (dmsg://, the default); -B/--direct uses the embedded server set with no discovery. Drop the now-dead UseHTTP var + stale doc. - reward-post dmsg server (rewards ui / serveStandalone): bootstrap via dmsgclient.StartDmsgEmbedded (embedded servers, discovery over dmsg) instead of disc.NewHTTP(clearnet) which couldn't connect. Validated: the reward UI now starts with ZERO clearnet 404s (was continuous spam); both its dmsg clients register + connect over dmsg. * fix(tpviz): fetchAny clearnet branch must call fetchURL, not recurse A sed during the dmsg-fetch refactor rewrote fetchAny's clearnet fallback to call s.fetchAny(url) instead of fetchURL(url) — infinite recursion on any http:// URL (and left fetchURL unused). golangci-lint caught it via the unused-func + unused-var (errGatedClearnet, also removed). Restore the fetchURL call. * feat(wasmhv): HTTP-over-dmsg override for the single-file (no-SW) browser hypervisor The Service Worker can't run from file://, so the self-contained single-file UI uses this fetch/XHR override instead: installed before the inlined Angular scripts, it routes every same-origin request (the app's /api calls) over dmsg via skywireDmsg.fetch. Also handles WebCrypto AES-GCM/PBKDF2 password-decryption of an embedded secret key (key never touches disk). Validated separately: an inlined Angular dist boots fully from file:// (lazy webpack chunks self-register, fonts/CSS inlined), leaving only /api — which this override supplies over dmsg. The Go inliner + generator command that bakes the dist + this override + the wasm client into one hypervisor.html is the next step. * feat(wasmhv): standalone wasm hypervisor core — visors dial the browser tab A browser tab can now BE a hypervisor (not just a remote viewer of one): it accepts visor dials on the hypervisor dmsg port (46), runs the gob RPC client against each dialed-in visor, and serves the hypervisor /api surface — so visors that list the tab's PK "just show up", with no binary hypervisor behind it. - pkg/wasmhv: a wasm-compilable hypervisor core that does NOT import pkg/visor (which doesn't compile to wasm). It declares MINIMAL MIRROR TYPES of the visor API responses (About/Overview/HealthInfo/Summary) with matching gob field names + json tags; gob decodes the real reply and ignores the rest. Core.Serve accepts visor dials → rpc.NewClient per visor; Core.ServeHTTP routes the read endpoints the node dashboard needs (/about, /ping, /visors, /visors-summary, /visors/{pk}[/summary|/health], no-auth user/csrf stubs). (Overview omits Apps/Transports from the gob struct — a mismatched element type makes gob fail the whole decode — and re-adds empty arrays via MarshalJSON.) - cmd/dmsg-wasm: serveHypervisor() starts the core; hvApi(method,path,body) serves a request from it (the standalone-mode equivalent of fetch() to a remote hypervisor). PROVEN end-to-end in a real headless browser against the live network: the tab connects over ws://, serves on dmsg :46, the local visor dials in (cli visor hv add <tabPK>), and /api/visors returns the visor's REAL Overview (version, IPs, NAT type, route count) via in-wasm gob-RPC.
…ck + autoconnect wedge (skycoin#3191) * fix(autoconnect): bounded SD fetch + prefer dmsg URL — no infinite retry, no clearnet A v1.3.77 visor stopped creating transports via public autoconnect after running a while. pprof showed its autoconnect goroutine wedged at autoconnect.go:446 — inside fetchPubAddresses' retrier on the service-discovery fetch. Two issues: 1. Infinite retry wedge. NewDefaultRetrier uses DefaultTries=0 (retry FOREVER). When the SD fetch fails repeatedly — the SD unreachable over its transport (the default deployment is dmsg-only, so this is the dmsg SD; a stale/broken dmsg path to it loops indefinitely) — retrier.Do never returns and the whole autoconnect Run loop wedges on a single tick: no public visors fetched, no transports created, and the CXO snapshot (re-triggered at the top of fetchPubAddresses each tick) never gets another chance to populate. Fix: a bounded retrier (3 tries) + 20s per-fetch timeout. The Run loop's ticker already provides the retry cadence, so a failing SD degrades to "no public visors this tick" and the loop self-heals next cycle. 2. HTTP-first SD URL selection. startPublicAutoconnectInternal preferred the clearnet HTTP SD URL over the dmsg one. The autoconnect's discovery client is the dmsg-HTTP transport, so a clearnet URL can't work here anyway (it gets dialed as a dmsg PK → "invalid host address"). The default config is dmsg-only (no clearnet URLs at all — verified), so prefer the dmsg URL and never a clearnet HTTP SD: no plain-HTTP service-discovery fallback. * refactor(visor): rip out plain-HTTP to deployment services — dmsg-only Plain HTTP to deployment services (AR/TPD/RF/SD/dmsg-disc/...) is no longer supported. getHTTPClient is the central transport builder for every deployment-service client; it now returns an ERROR for any non-dmsg:// URL instead of silently building a clearnet http.Client. A clearnet service URL is a misconfiguration and fails loud. - getHTTPClient: drop the plain-http branch; non-dmsg URL → error. - initAddressResolver: drop the "DMSG-HTTP failed, using plain HTTP" fallback + the v6-forced clearnet client (only meaningful for a plain-HTTP AR); take the dmsg URL (AddressResolverDmsg, or AddressResolver which holds the dmsg:// URL in the dmsg-only default) and route it over dmsg. Validated: local visor boots clean on this change, connects to the address resolver over dmsg and binds STCPR/SUDPH — no clearnet, no "not a dmsg" errors. * refactor(visor): SD service fetch + service list are dmsg-only Continue the plain-HTTP-to-deployment-services rip-out in api_services: - serviceFetchOrder: only dmsg:// endpoints; clearnet URLs ignored. Also fixes a latent bug — the dmsg-only default config stores the dmsg URL in the 'http' field (service_discovery = dmsg://...), which the old order mis-classified as a plain-HTTP hop and would GET over clearnet. Now any dmsg:// URL (either field) is routed over dmsg. Removed fetchServiceDataHTTP + serviceHop.dmsg. - servicesFromHTTP → servicesFromDmsg: the VPN/proxy/visor service lists (the CXO-snapshot fallback) used a plain clearnet http.Client; now they resolve the dmsg SD URL and fetch over the dmsg-HTTP client (getHTTPClient). * refactor(visor): health probes + hypervisor UI proxies are dmsg-only Finish the plain-HTTP rip-out in the visor's auxiliary/UI paths: - ServiceHealth: probe every deployment service over dmsg only (drop the clearnet http.Client fallback). The dmsg URL is taken from whichever config field holds the dmsg:// one (the dmsg-only default stores it in the 'http' field); services with no dmsg URL report N/A. - hypervisor TPD-metrics proxy (getNetworkTransports): drop the 'Step 3 plain HTTP' fallback + the clearnet deployment.Prod.TransportDiscovery default; dmsg-only. - hypervisor reward-system proxy (proxyRewardSystem): drop the plain-HTTP fallback; dmsg-only. Remaining (separate, dmsg-package layer): the dmsg client's own discovery upgrade (dmsgfirst) still carries an HTTP fallback — inert on a dmsg-only deployment (the disc URL is dmsg://), but it's the dmsg-bootstrap path so it warrants its own careful change. * test(visor): update TestServiceFetchOrder for dmsg-only fetch serviceFetchOrder is now dmsg-only (clearnet hops dropped, serviceHop.dmsg field removed). Update the test: only dmsg:// URLs are returned, the dmsg URL is taken from whichever field holds it (the dmsg-only default stores it in the http field), and clearnet-only configs return nil.
…kycoin#3193) Adds WebTransport (HTTP/3 over QUIC) as a fourth dmsg transport alongside TCP, QUIC and WebSocket. WebTransport is the one browser-reachable transport that needs NO CA-issued certificate: the browser's WebTransport constructor accepts a self-signed cert via serverCertificateHashes (a pinned SHA-256), so a dmsg server can present a short-lived self-signed ECDSA cert, publish its hash in discovery, and a browser connects by bare IP — no Caddy, no domain, no Let's Encrypt. Like WS, a WT session carries the EXISTING Noise+yamux session over a single bidirectional stream, so the per-hop Noise handshake (which authenticates the client PK, since a browser can't present a client cert) and the session layer are unchanged. WT's native stream multiplexing is intentionally unused — yamux multiplexes inside the one WT stream, keeping a single session implementation. - pkg/skyquic/webtransport.go: NewWebTransportCertificate (ECDSA P-256, 13-day validity, no PK-binding extension) + sha256 hash; WebTransportTLSConfig. - pkg/dmsg/dmsg/wt.go: Server.ServeWebTransport (caller owns cert for deterministic rotation), handleWTSession, wtStreamConn net.Conn adapter, and Client.dialSessionWT (native path; pins CertHashWT, disables resumption so a resumed session can't bypass the pin). - disc.Server.AddressWT + CertHashWT advertised from EntityCommon; PreferWT client config + dial-switch wiring with TCP fallback. - dmsgserver config wt_address/public_address_wt; dmsgsrv serves WT best-effort. - wt_test.go: full bridged round-trip over WT-only server + cert-hash-mismatch rejection test. Vendors github.com/quic-go/webtransport-go v0.11.0 (+ dunglas/httpsfv).
…hypervisor (skycoin#3194) Extends the wasm hypervisor core (skycoin#3190) from a read-only dashboard to a controllable one: the apps (list / start / stop / autostart) and transports (list / types / add / remove) endpoints the hypervisor UI drives daily. As with the read path, the core does NOT import pkg/visor (which doesn't compile to js/wasm) — it declares MINIMAL MIRROR TYPES of the visor RPC request/reply structs (AppState, TransportSummary + its custom-gob LogEntry, and the StartApp/SetAutoStart/Transports/AddTransport argument structs). gob matches struct fields by name and skips the rest, so a partial mirror decodes the fields the UI reads and encodes the fields a control call needs. - control.go: mirror types + control calls (apps/app/startApp/stopApp/ setAutoStart/transports/transportTypes/addTransport/removeTransport). TransportLogEntry reproduces transport.LogEntry's custom Gob/JSON codec so per-transport bandwidth shows. - router.go: method-aware /visors/<pk> routing — GET apps/transports, PUT apps/<app> (status + autostart), POST transports, DELETE transports/<id>. - override.js: standalone mode (CFG.standalone) — route /api to the in-wasm core (hvApi) and call serveHypervisor() on connect, instead of proxying to a remote hypervisor over dmsg. One shared dispatch() picks the path. - gob_mirror_test.go (host-only !js): gob round-trips the REAL visor types through the mirror types in BOTH directions, so a field rename on either side fails a test instead of silently zeroing a value in the browser.
… dmsg-only (skycoin#3195) The deployment discovery services (dmsg-disc / TPD / SD / AR) are no longer served over clear HTTP, so the dmsgfirst "try dmsg, retry on plain HTTP" disc client's fallback leg only added a doomed roundtrip + a WARN per refresh when a dmsg dial transiently failed (e.g. dmsg-202 session-overlap). Worse, it could mask a transient dmsg blip with a dead HTTP endpoint. upgradeDmsgDiscToDmsgfirst → upgradeDmsgDiscToDmsgOnly: each per-deployment disc client now registers + resolves STRICTLY over dmsg (dmsg-HTTP through the direct-client dmsgDC), mirroring dmsgsrv's newDmsgOnly. A transient dmsg error just lets the caller retry over dmsg on its next periodic refresh / re-registration, which self-heals once sessions realign. The pk-absent branch (no discovery_dmsg configured) still uses the explicit plain-HTTP Discovery URL — that is the operator's "no dmsg discovery" config choice, not the runtime fallback being removed. Deletes pkg/dmsg/disc/dmsgfirst (the visor was its only consumer) and the now -stale references in pkg/router/map.go and init_dmsg.go comments. Validated on a live visor: clean boot, 9 dmsg servers, TPD transports list + AR reward resolve over dmsg with no HTTP fallback path.
…ride.js (skycoin#3196) Design doc (docs/design/unified-wasm-hypervisor-ui.md) for collapsing the two hypervisor front-ends (visor-served Angular UI + the WASM hypervisor) into ONE HTML+wasm artifact that detects its context and picks a role: served (dormant, ride the serving visor's session — no key), viewer (dial a remote hypervisor PK), or standalone (this tab IS the hypervisor, visors dial in). Captures the two server→serverless enable-flows, the AddHypervisor RPC affordances Mode 2 needs, and the precise security boundary: never serve the KEY-ENTRY variant from a domain; the served variant is safe anywhere because it takes no key. Lands Phase 1 — the context-detection primitive in override.js: it now only installs the fetch/XHR shims (and boots the wasm) when a mode is configured (CFG.pk or CFG.standalone, and not CFG.served). With neither set it returns early and native fetch/XHR pass through to the serving backend — so the same bundle can be served by a visor (behaves as today's UI) or opened from file:// (serverless), which is the prerequisite for serving the wasm UI in place of the Angular build. No current consumer (the single-file generator is Phase 2).
…nyGo dmsg port plan (skycoin#3197) The lightweight TinyGo logging stub was pinned to GOOS=js by its filename (logging_js.go — Go's implicit _js.go GOOS rule overrides the //go:build tinygo tag), so under wasip1/bare-metal TinyGo pkg/logging had no files at all ("build constraints exclude all Go files"). Renamed to logging_tinygo.go so the //go:build tinygo tag covers js, wasip1, and bare-metal alike. Production (!tinygo) and standard-Go js/wasm builds are unaffected — they never used the stub (verified: go build ./..., GOOS=js GOARCH=wasm build, and the logging tests all stay green). Adds docs/design/tinygo-dmsg-client.md with the empirically-measured blocker map (TinyGo 0.41: gob/json/logrus all compile; net/http works on wasip1 but is broken on the js target; quic-go won't compile) and the phased plan. The only remaining blocker for a TinyGo IoT (wasip1) dmsg client is quic-go woven into the session core, to be extracted behind an interface into !tinygo files next.
…HitTracker flake (skycoin#3192) PortHit ordering (snapshot + eviction) used LastSeen, a wall-clock time. On Windows' ~15ms coarse timer, several hits recorded in quick succession share an identical LastSeen, so the most-recent-first sort was non-deterministic and TestPortHitTracker flaked on the windows CI runner (snapshot[0] not the last-recorded hit). Add a monotonic seq counter, bumped on every record, and order by it instead of LastSeen (eviction picks the lowest seq, snapshot sorts seq-descending). seq is unexported so the JSON shape is unchanged. Test passes 50x.
…UIC stream bugs (skycoin#3198) * feat(dmsg)+tinygo: extract QUIC behind an interface; fix two latent QUIC stream bugs TinyGo Phase 2: decouple quic-go from the dmsg session core so a TinyGo (IoT) dmsg client compiles. quic-go does not build under TinyGo (needs crypto/tls.QUICEncryptionLevel), and it was woven into six session-core files as *quic.Conn / *quic.Stream struct fields. Now the core refers to QUIC only via the quicConn / quicStream interfaces (quic_iface.go); the concrete adapters, session constructors, client dial and server listener live in quic_native.go (//go:build !tinygo), with TinyGo stubs in quic_stub.go that fall back to TCP/WS. wt.go (WebTransport, also quic-go) gets //go:build !tinygo. Result: pkg/dmsg/dmsg no longer pulls quic-go/webtransport-go under the tinygo tag. Writing the first dmsg-over-QUIC round-trip test (quic_test.go — a QUIC-only server, two clients, a bridged A→B stream) surfaced TWO pre-existing bugs in the dmsg-over-QUIC stream path that no test had ever exercised: - server forwardRequest() opened the destination stream with a smux/else-yamux branch — for a QUIC destination it dereferenced a nil *yamux.Session and panicked serveStream. Added the QUIC case (OpenStreamSync). - the client accept loop never treated a QUIC AcceptStream error as terminal (no quic case in the session-closed switch), so it span forever on a closed QUIC session and hung ClientSession.Close(). quic-go's AcceptStream only errors on a done connection, so any error there is now terminal. Supporting TinyGo proofing (logrus/json compile under TinyGo 0.41, unlike the skycoin#2836-era assumption): pkg/logging now builds its real logrus implementation on all targets (dropped the //go:build !tinygo tags + the no-op logging_tinygo.go stub, which couldn't satisfy logrus.FieldLogger for the full client anyway); the VictoriaMetrics-backed Metrics impl is tagged !tinygo (server-only; the client uses the no-op Empty). Regular go build ./... + the full pkg/dmsg/dmsg suite (incl. the new QUIC round-trip) stay green; the QUIC extraction is verified quic-free under `go list -tags tinygo`. The remaining TinyGo blocker is pkg/netutil (interface-enumeration), tracked as Phase 3 in docs/design/tinygo-dmsg-client.md. * docs(tinygo): mark QUIC extraction (Phase 2) done + note the two QUIC bug fixes; netutil is the Phase 3 blocker
skycoin#3199) The dmsg client now builds under TinyGo for the wasip1 (IoT) target: `make tinygo-dmsg` (tinygo build -target wasip1 -no-debug -opt=z) produces a ~2.2 MB wasm binary. Build-check main in cmd/dmsg-tinygo-probe. Clears the remaining peripheral blockers, each with the same !tinygo-split-plus-stub (or de-tag-the-obsolete-!tinygo) pattern established in Phase 2: - pkg/netutil: net.go used net.Interface.Addrs() (absent in TinyGo's net) + a GOOS-split DefaultNetworkInterface with no wasip1 variant. Pure helpers (IsPublicIP/ExtractPort/IsVirtualInterface) stay in net.go; the interface-enumeration + ipinfo HTTP probe move to net_native.go (!tinygo) with stubs in net_tinygo.go. - pkg/dmsg/disc: Entry.Sign / VerifySignature were !tinygo on the stale "encoding/json can't run under TinyGo" assumption (false since 0.41). De-tagged (entry_native.go → entry_sign.go) with a stdlib encoding/json codec shim (interface_tinygo.go) standing in for jsoniter under TinyGo. - pkg/dmsg/dmsg: *net.TCPConn.SetNoDelay (absent in TinyGo) wrapped in a build-tagged setTCPNoDelay helper (no-op on TinyGo). - deployment: EnvServices moved from config_native.go (!tinygo) to the untagged config.go so pkg/dmsg/dmsg's InitConfig can decode the embedded config on TinyGo too. Adds the `tinygo-dmsg` Makefile target and updates docs/design/tinygo-dmsg-client.md (Phase 3 done). Normal go build ./... + pkg/dmsg/dmsg, disc, netutil, logging, deployment test suites stay green.
…assembler) (skycoin#3200) GenerateStandalone assembles a self-contained, serverless hypervisor.html from the built Angular UI + the WASM dmsg client + override.js — a single file with NO external references that boots the in-browser dmsg client and runs the hypervisor UI from file://. This is the keystone of the unified-UI plan (docs/design/unified-wasm-hypervisor-ui.md): "save the hypervisor UI to an HTML file and open it serverless." What it does: - inlines every Angular chunk <script src> (preserving type="module" so the deferred/ordered execution is kept) and every <link rel=stylesheet> as a <style> (deduped — Angular emits the sheet twice); - embeds the wasm gzip+base64 and decompresses it in-browser via DecompressionStream, then instantiates + runs it; - injects override.js + window.__SKYWIRE_HV__ (dmsg seed/disc + identity) as classic <head> scripts BEFORE the Angular module scripts, so the XHR/fetch shim is installed before HttpClient is constructed; - bakes the secret key in password-encrypted (PBKDF2-SHA256/200000 + AES-GCM, base64[salt|iv|ct] — the exact format override.js resolveSK decrypts); the plaintext key never touches the file. Without a password it's ephemeral or (testing only) a bare sk. override.js is embedded (embed.go) so consumers only supply the UI FS + wasm + wasm_exec.js. Tests cover structure (no external refs, dedup, ordering, </script> neutralization), viewer vs standalone CFG, and an encsk encrypt→decrypt round-trip matching the browser scheme. Follow-ups (scoped in the design doc): a CLI/visor consumer that supplies the UI FS + built wasm; inlining the runtime Angular assets (i18n JSON + flag/logo PNGs) into a virtual FS override.js serves (today a generated file shows untranslated strings + broken images); and headless-browser validation of a real bundle.
…ith derived key (skycoin#3201) Wires the standalone generator (skycoin#3200) into a usable command and adds one-way key derivation. `skywire cli hv gen` produces a self-contained, serverless hypervisor.html: the embedded Angular UI + the (--wasm) js/wasm dmsg client + override.js + config, all inlined, openable from file://. Default = standalone (visors dial in); --viewer-pk = dial a remote hypervisor. Identity: - DeriveStandaloneKey(parentSK, index): HMAC-SHA256(parentSK, label||index) → GenerateDeterministicKeyPair. ONE-WAY (a cracked standalone key can't expose the parent), deterministic (regenerable from the one root key — no separate backup), and indexed (distinct standalone identities from one root). `-c <config> [--index N]` derives from the visor's sk and prints the derived PK (the PK to set as the visors' remote hypervisor). `--sk` overrides; neither = ephemeral. - --password encrypts the baked-in key (PBKDF2/AES-GCM, the format override.js decrypts); the plaintext key never touches the file. A bare (unencrypted) key warns. Plumbing: pkg/wasmhv embeds wasm_exec.js (WasmExecJS) so a generated file is self-contained (must match the Go toolchain that built the wasm); pkg/visor exports HypervisorUIFS() for the embedded Angular assets. Validated end-to-end: `hv gen --wasm dmsg.wasm -c config --password ...` derives the key and writes a 9.6 MB fully-inlined hypervisor.html (21.6 MB wasm gzipped to ~6 MB) — no external script/style refs, encrypted key only (no plaintext leak). Tests cover the derivation (deterministic/indexed/parent-bound/one-way). Follow-ups: inline the runtime Angular assets (i18n JSON + flag/logo PNGs) so a generated file isn't untranslated + missing images; headless-browser validation; and the optional tagged 6 MB wasm embed + --extract-wasm for an out-of-box / in-UI flow.
…ml (skycoin#3202) A generated standalone file previously showed untranslated strings + broken images: the Angular app fetches its runtime assets (i18n JSON, flag/logo images, fonts under assets/) at load, and from file:// there's no backend to serve them. The generator now inlines everything under assets/ into a JS map (self.__SKYWIRE_ASSETS__ = {"/assets/...":{ct,b:base64}, ...}), and override.js serves them locally: a new serve() checks assetResponse() FIRST — before ensure() — so assets resolve instantly with no dmsg round-trip, and the UI renders translations + images even before the dmsg client is up. Non-asset paths (/api/*) fall through to the dmsg/hvApi dispatch as before. Works in both standalone and viewer modes (inlined assets beat a remote fetch either way). Cost: base64 adds ~33% over the raw asset size (~3.5 MB assets → ~4.7 MB), so a generated file grows accordingly; gzip + excluding the country-flag PNGs are noted size levers. Tests assert the asset map is inlined with correct path + content-type + base64. Remaining for a fully-working file: headless-browser validation of a real bundle (this covers structure).
…keys (skycoin#3203) Gives the visor config a small wallet-like keyring so the standalone hypervisor uses a deterministically-derived child of the visor key instead of a random, unrelated one — and so derived keys are tracked (no index reuse), in the spirit of a skycoin deterministic wallet. - cipher.DeriveChildKey(parentSK, label, index): the shared one-way derivation primitive — seed = HMAC-SHA256(parentSK, label||index) → GenerateDeterministic KeyPair. One-way (a leaked child can't expose the parent), deterministic (regenerable, no separate backup), label+index namespaced. wasmhv.DeriveStandaloneKey now delegates to it (output-preserving) and exports StandaloneKeyLabel. - visorconfig.KeyRing / KeyEntry (config field "keyring"): wallet-shaped — type + next_index + entries{index,label,address,public_key,secret_key}. The skycoin address is derived from the public key. V1.MintKey(label) derives the next entry and advances next_index (the "derive a new address" op); DeriveKeyEntry re-derives a known entry read-only. Added KeyRing to the custom-UnmarshalJSON mirror (config_compat) so it round-trips. - cli hv gen: with -c <config> it mints the next keyring key, records it, and flushes the config (prints the derived address + PK); --index N re-derives a known entry read-only (no write); --sk still overrides. Validated end-to-end on a config copy: minting advances 0→1 with distinct keys/addresses, the keyring round-trips through load→mint→flush, and the rest of the config is preserved (sk/pk/launcher/dmsg/... intact). Tests cover the derivation determinism + keyring mint/derive consistency with the shared primitive. Deliberately minimal — fleet provisioning, xpub watch-only, and any ledger ideas are explicitly out of scope pending a clear problem to solve.
…kycoin#3204) The KeyRing's only purpose is the standalone-hypervisor identity keypair, not a payment wallet, so the derived skycoin address was vestigial (a leftover of the deferred wallet/ledger idea). Remove KeyEntry.Address + its derivation (and the skycoin/src/cipher import); cli hv gen now prints just the derived PK.
…nt; split RPCClientDialer) TinyGo HV groundwork (1/N): net/rpc transitively pulls net/http, which is broken on the TinyGo js target. Two changes remove net/rpc from the cmd/dmsg-wasm graph: - pkg/wasmhv/gobrpc.go: a minimal client speaking the EXACT net/rpc gob wire protocol (Request+args, Response+reply), so it talks to the visors' net/rpc server unchanged but imports no net/rpc. The core now uses it instead of rpc.NewClient. gobrpc_test.go drives a REAL net/rpc server over a pipe (incl. the error path + stream re-alignment) to prove wire compatibility. - pkg/dmsg/noise: RPCClientDialer (the only net/rpc user) split into rpcdialer.go (//go:build !tinygo); net.go keeps the core Conn/Listener. Normal build + tests unaffected; net/rpc verified gone from the tinygo graph.
…r TinyGo (6.5MB vs 21.6MB)
TinyGo HV groundwork (2/2): the only remaining blocker to a TinyGo-built wasm
hypervisor was net/http, broken on the TinyGo js target (roundtrip_js.go). This
removes net/http from the cmd/dmsg-wasm graph entirely, so it builds with TinyGo:
3.4x smaller (21.6MB -> 6.5MB; 5.6MB -> 2.4MB gzipped), small enough to embed by
default.
net/http was pulled by three things; each gets a build-tag split (native keeps
the existing net/http path; tinygo gets a net/http-free equivalent):
- DISCOVERY over dmsg (the keystone): dmsgclient.dmsgDiscClient implements
disc.APIClient by speaking HTTP/1.1 straight over a dmsg stream (httpdmsg.go:
a minimal, wire-correct HTTP/1.1 client — Content-Length, chunked, and
Connection:close framing; httpdmsg_test.go validates it against a REAL
net/http server). seeded.go's discovery upgrade is split into
seeded_upgrade_{native,tinygo}.go. The net/http-free fallbackDiscClient is
extracted to fallback_disc.go (untagged); the net/http-using bootstrap helpers
+ cobra flags move behind //go:build !tinygo (cli.go, cli_fallback.go,
flags.go).
- WEBSOCKET client dial: coder/websocket pulls net/http via its Dial signature
(*http.Response). ws_js_tinygo.go dials the browser's native WebSocket through
syscall/js and adapts it to a net.Conn (full deadline-aware Read/Write/Close),
flowing through the SAME makeClientSession+yamux path. ws_server.go splits the
server-side ServeWS (http.Server) behind !tinygo. Only the tinygo+js+wasm combo
is affected — native, std-Go wasm, and tinygo+wasip1/IoT keep coder/websocket
(net/http compiles there).
- jsFetch viewer transport: split into fetch_{native,tinygo}.go; the tinygo path
uses the new dmsgclient.FetchOverDmsg (net/http-free HTTP-over-dmsg round trip).
Builds verified: native (go build ./...), std-Go js/wasm, and
tinygo -target wasm. Tests + lint clean. No behavior change on existing builds.
…p (TinyGo) Two more browser transports, both compile-verified under std-Go wasm AND TinyGo wasm (final artifact 6.73MB, +0.2MB over the WS-only baseline): - WebTransport dmsg carrier (wt_js_tinygo.go): dials a dmsg server's WT endpoint via the browser WebTransport API (syscall/js), pinning Server.CertHashWT in serverCertificateHashes — CA-free reach by bare IP, mirroring the native wt.go cert-hash model. One bidirectional WT stream → net.Conn → the same makeClientSession+yamux path as WS/TCP. The TinyGo WT stub splits: quic_stub.go keeps QUIC; wt_stub_tinygo.go covers non-browser (IoT) TinyGo; the browser gets the real dial. awaitJS() bridges JS promises to blocking Go. - WebRTC DataChannel p2p (cmd/dmsg-wasm/webrtc_js.go): the first TRUE peer-to-peer browser transport — a direct DTLS+SCTP DataChannel between two leaves, no relay in the data path. dmsg is the signaling plane: offer/answer/ICE ride a dmsg stream (SignalChannel over *dmsg.Stream, port 47); the opened DataChannel is adapted to net.Conn. JS API webrtcDial(pk)/webrtcListen(cb). This realizes a browser p2p endpoint reachable BY PK through the mesh — a tab need not listen. Design: docs/design/wasm-visor-p2p.md formalizes 'reachability != listening' — a tab with one outbound transport + a TPD edge is a routable node serving skynet, the dmsg model generalized to the routing layer. The WebRTC-as-mesh-transport + wasm-visor integration (TPD registration, route-setup responder) is designed, not built. None of the browser transports is runtime-validated in a browser yet (compile-only); that is the next gate.
cmd/wasm-visor-probe is a build-only frontier check: what tinygo build accepts is
what's browser-portable. Measured frontier (go list -deps -tags tinygo):
✅ compile under TinyGo today: pkg/routing, pkg/visor/visorconfig
⛔ blocked by a small recurring set:
quic-go — the raw-socket networks (stcp/sudph/quic) in pkg/transport/network
net/http — RF/TPD/AR discovery clients (the disc-over-dmsg pattern solves it)
net/rpc — appevent channel + router/cascade_source.go RSN oracle
os/exec — pkg/visor only: the app-SUBPROCESS model (the deepest gap)
docs/design/wasm-visor-p2p.md §7 adds the measured table + a 5-phase plan:
(1) dmsg-only network build, (2) net/http-free RF/TPD/AR, (3) tag out net/rpc →
routing+transport core in the browser; (4) in-process apps (the real fork, cf.
skycoin#2775); (5) browser persistence + cmd/wasm-visor assembly.
No production code touched — analysis + probe only.
…s framing - Correct phase 4: in-process apps are NOT a fork. The launcher already has RunModeInternal (startInProcess, run-func in a goroutine) vs RunModeExternal (startExternal, the only os/exec). A wasm visor uses the internal launcher and tags out startExternal. - New §8: which apps fit, by the rule 'pure mesh I/O OR a browser-exposed capability'. Fit: skychat, content/site hosting, file share, wallet-over-dmsg, CXO pubsub, capability bridge. Don't: skysocks/skysocks-client, VPN, port forwarding (raw sockets / TUN / local listen). Frame: a wasm visor is a personal present-when-you-are node — a CONSUMER of infrastructure apps via routes, not a provider. Start with skychat + content hosting.
The browser transports + in-wasm HV compile but need RUNTIME validation in a tab (the gate compile checks can't cover). This adds the harness to do it: - make tinygo-dmsg-wasm: build dmsg.wasm with TinyGo (~6.5MB) + stage the page. - cmd/dmsg-wasm/serve.go (//go:build ignore): tiny no-Python static server that sets Content-Type: application/wasm so instantiateStreaming works. - cmd/dmsg-wasm/index.html: extend the existing harness with sections for the in-wasm hypervisor (serveHypervisor / hvApi) and WebRTC p2p (webrtcListen / webrtcDial / send); fix the fetch body to TextDecode the binary Uint8Array. Run: make tinygo-dmsg-wasm && go run cmd/dmsg-wasm/serve.go → http://localhost:8085/ Validated: build + serve + correct wasm MIME. Browser-click validation against a live dmsg server is the manual gate this enables.
…browser
First runtime finding from the harness: the TinyGo wasm failed to instantiate
('gojs runtime.getRandomData: function import requires a callable'). TinyGo's
wasm_exec.js provides random via WASI random_get but OMITS the gojs
runtime.getRandomData import the Go runtime calls to seed itself on the crypto
path — so any crypto-using TinyGo wasm (ours generates keys) won't load. tpviz
works only because it uses no crypto/rand.
Fix in the loader: inject runtime.getRandomData ONLY when absent (TinyGo), so the
standard-Go build — whose wasm_exec.js already defines it via a different
sp-based ABI — is untouched. TinyGo passes a []byte as (ptr,len,cap) over
exports.memory, so the shim is crypto.getRandomValues(new Uint8Array(mem,ptr,len)).
Also switch to fetch+arrayBuffer instantiate (MIME-independent) and add a .catch.
Verified headless via node + wasm_exec.js: main() runs, all 8 skywireDmsg methods
register. (Candidate for an upstream TinyGo wasm_exec.js fix.)
Owner
Author
Browser runtime validation — core path ✅Validated the TinyGo wasm in a real browser against a live dmsg server (after fixing one runtime gap, see below). Both
Runtime gap found + fixed (4b06136)TinyGo's Still foundation-only (not yet validated, needs two tabs / a peer)
These are the transport foundations the PR labels experimental; the core net/http-free path is browser-proven and the PR is mergeable on that basis. |
…for 2-tab orchestration
serve.go grows a control bridge so an external operator (or the assistant, via
curl on the same host) can drive the browser tab(s) headlessly — enough to
orchestrate two wasm visors against each other (the inbound-reachability + WebRTC
paths that need two endpoints):
- GET /ctl/events?tab=ID : SSE; the tab subscribes to receive commands
- POST /ctl/result : the tab returns a command's result (correlated by id)
- POST /ctl/log?tab=ID : the tab streams its log lines; GET reads them back
- GET /ctl/tabs : list connected tabs
- POST /ctl/cmd?tab=ID : queue {action,args}, push to the tab, await its result
index.html grows a control client (EventSource + a dispatch over the skywireDmsg
API: connect/dial/listen/send/fetch/serveHypervisor/hvApi/webrtcDial/Listen/rtcsend),
mirroring logs to the server and keeping named stream/datachannel handles. No-ops
if no control server is present, so the manual harness still works. Stdlib-only
(SSE), no new deps.
Validated headless: a node fake-tab + curl drove connect/webrtcDial and read the
result + log buffer back.
…ployment STUN Browser validation of the WebRTC path surfaced + fixed real bugs and proved the signaling end-to-end: - FIX: dmsgSignalChannel.recv did `return m, json.Unmarshal(b, &m)` — Go leaves the result-read vs call order unspecified, and TinyGo reads m (empty) BEFORE Unmarshal mutates it, so every signaling message decoded blank. Decode then return. (std-Go happened to work; TinyGo exposed it. Confirmed by probe.) - FIX: trickle-ICE candidate buffering — addIceCandidate before the remote description is set silently drops candidates. Buffer until remoteSet, then flush. - Instrumentation: WebRTC signaling/ICE progress routed to window.__wrtclog so the control bridge surfaces offer/answer/candidate/ICE-state/datachannel milestones. - WebRTC ICE now uses the deployment's own STUN servers (deployment.Prod.StunServers, exposed to JS as skywireDmsg.stunServers) — no third-party STUN. TODO: a visor-generated UI must inject THAT visor's config, not the embedded default. Harness: control bridge gains stable per-tab ids (sessionStorage), a `reload` command, a /ctl/clear endpoint, and no-cache headers, so reloads can be driven remotely. serve.go silences the favicon 404. Validated in-browser between two tabs: connect, dmsg dial/listen + bidirectional data, HTTP-over-dmsg, in-wasm hypervisor, and WebRTC signaling + ICE negotiation (offer/answer/candidates exchange; ICE reaches 'checking'). WebRTC connection COMPLETION fails in the same-machine/same-NAT case (mDNS host candidates + srflx hairpin) — a topology limitation, the documented ICE open question, not a code bug.
…y (reachable via fallback only)
… from it) The seeded dmsg client (browser/standalone) connected to a server but never registered its entry in dmsg-discovery — peers could only reach it via the shared-server fallback. Root cause: the registering-fallback resolved the client's OWN pk from a synthetic DIRECT-client entry, so both initilizeClientEntry and updateClientEntry saw a fake 'already registered' entry and skipped the POST. Fix (per operator guidance — don't use a direct client for discovery queries): - pkg/dmsg/dmsgclient/seeded_disc.go: seededDiscClient talks to the REAL discovery for register/resolve/AvailableServers, keeping a shortcut ONLY for the discovery PK + seed servers (recursion-avoidance + bootstrap). Wired into the TinyGo upgrade (native keeps the proven fallback). Own pk now resolves against real discovery (404 when absent) → the register-fresh-entry path fires. - pkg/dmsg/dmsg/entity_common.go: setDiscoveryClients clears lastPushedSrvPKs and nudges a re-publish, so swapping the bootstrap disc client for the real one actually (re)registers instead of being short-circuited by the SamePubKeys guard. - discdmsg.go: map the 404 body to disc.ErrKeyNotFound so isEntryNotFound matches. - MinSessions 1→2 (resilience; browser is still capped by WS-capable server count). - jslog hook (window.__skylog) + harness control-bridge reconnect fix, /ctl/clear, no-cache headers, and a READY signal — so reloads + tests can be driven remotely. Validated in-browser (autonomous control bridge): the TinyGo wasm client registers (PostEntry 200 'wrote a new entry'); a discovery lookup of its own PK returns the full entry (200).
…ntained to quic.go+quic_identity.go)
…wants a network registry, own PR
…) behind !tinygo Phase 1 of the visor wasm port: drop the browser-irrelevant raw-socket transport carriers from the TinyGo graph so dmsg (and the future webrtc) remain. - quic.go + quic_identity.go: //go:build !tinygo (quic-go + pkg/skyquic). - sudph.go + stun_client.go: //go:build !tinygo (kcp-go + pfilter, which pull quic-go transitively; go-stun). - The QUIC/SUDPH constructors are routed through build-tagged makeQuicClient / makeSudphClient helpers (real on native, unsupported-error stubs on TinyGo), so client.go's MakeClient stays untagged and the type-switch compiles on both. - stcp/stcpr stay untagged (raw net compiles under TinyGo). Native unchanged (the tagged files stay in native builds) — go build ./... green. Remaining TinyGo blockers in pkg/transport/network: addrresolver (net/http + packetfilter via the AR machinery in client.go's ClientFactory.ARClient) and appevent (net/rpc) — phases 2-3. See docs/design/wasm-visor-p2p.md §7.
…twork→transport→visor (needs a per-layer restructure)
Owner
Author
|
Retargeting to upstream skycoin/skywire. |
0pcom
added a commit
that referenced
this pull request
Jul 7, 2026
…2398) * docs(dht): update 26-DHT.md to current implementation; add SKYWIRE_DHT.md 26-DHT.md was missing several merged PR skycoin#2397 changes: - MaxValueSize 16 KiB → 64 KiB and the rationale - methodGetItems (Tag 5) and methodPutBatch (Tag 6) RPCs - GetItems pagination semantics with Seq cursor - Full-node reconcile loop (pull + push) and FullNodeAdvert - 4 MiB rpcCall message cap - Persistence backends (memory / bbolt / Redis) - CLI surface (`dht status/get/put/list/sync/full-node`) and which commands are useful for full vs non-full nodes - Default constant values from pkg/dht/config.go (PublicPoolSize=5000, RateLimitPerPK=50, ItemTTL=2h, RefreshInterval=30s, MaxItems=10000) Adds SKYWIRE_DHT.md, a descriptive companion to the normative spec. Compares Skywire's DHT side-by-side with BitTorrent BEP5/BEP44, libp2p Kademlia, Ethereum discv5, and Tor v3 onion descriptors across identity, routing table, item model, transport, eclipse/Sybil resistance, and bootstrap/reconciliation. Frames the design choices that diverge from textbook Kademlia (secp256k1 over ed25519, trust tiers, full-node mode, active reconcile, hybrid HTTP fallback) in the context of *why* Skywire needs them rather than treating Kademlia as the ceiling. Also documents what the implementation does NOT do (no CAS, no republish-on-receive, no IP-level rate limiting, single 64 KiB cap) and an honest list of where the implementation is still rough — hourly reconcile cadence, PutMirror admission gating, missing metrics, debugging gaps. * cli: dht peers and dht reconcile subcommands Two CLI gaps surfaced in the DHT audit: 1. 'skywire cli visor dht status' shows the routing-table peer count but no way to actually inspect WHICH peers are in the K-buckets. Debugging a "DHT seems connected but lookups miss" problem required log-grepping. New `dht peers` dumps the full table sorted by bucket index then last-seen, with --json for machine consumption. 2. The hourly fullNodePullLoop reconciles automatically, but there was no way to force a reconcile against a specific peer for testing or to push fresh local state to a known-stale peer immediately. New `dht reconcile <full-node-pk>` runs one pull+push pass synchronously and reports (pulled, pushed) counts. Both wire through the same RPC layer: - pkg/visor/api.go: DHTPeers, DHTReconcile in API interface - pkg/visor/api_dht.go: DHTPeerInfo struct, DHTPeers/DHTReconcile visor methods - pkg/visor/rpc_dht.go: DHTPeers/DHTReconcile RPC handlers, with DHTReconcileResult carrying the pulled+pushed counts on the wire - pkg/visor/rpc_client.go: matching client methods - pkg/visor/rpc_client_mock.go: not-supported stubs Reconcile safety: the receiver-side PutMirror handler stores any signed item without distance/admission gating, so pushing to a non-full-node would silently overflow its store. New dht.Node.IsTrustedFullNode reports whether a PK is in BootstrapPKs ∪ FindAdvertisedFullNodes (signed self-attestation). DHTReconcile rejects untrusted PKs with a clear error before opening the dial. dht.Node.Reconcile is now the exported wrapper around reconcile (still lowercase) with the safety contract spelled out in its doc comment. Verified live on the dev visor: - 'dht peers' returned 18 entries across multiple bucket indexes - 'dht reconcile <bootstrap-pk> --salt tp' pulled 1120 tp items in one pass - 'dht reconcile <random-visor-pk>' rejected with "not a known full node (not in bootstrap config and no fresh fullnode advert)" Updates 26-DHT.md's CLI table to reflect the two new commands. * cli: route calc --source tpd|dht|auto to compare DHT vs TPD coverage The audit identified 'route calc' as a place where the local DHT could substitute for the centralized TPD fetch, useful for full nodes that already mirror the network's transport graph. Adds a --source flag (default tpd, preserving current behavior): --source tpd : fetch /all-transports from the deployment TPD (existing chain: visor RPC → DMSG direct → HTTP) --source dht : build the graph from the local DHT's "tp" salt via DHTGetAll. Only bare-entry format contributes (the deployment-pushed compact [{r,t,l,b}] format omits the source PK in its value — only the storage target hash carries it, and SHA256 isn't invertible — so those entries are skipped) --source auto : try DHT first, fall back to TPD if DHT yielded fewer than 10 entries Tested live on the dev visor (full node, recently reconciled): route calc <pk> --source dht --max 4 -c 0 → 878 routes route calc <pk> --source tpd --max 4 -c 0 → 4828 routes The 18% DHT coverage reflects the fraction of network visors that have published their transports as TPDAdapter bare entries (not yet fully rolled out). When more visors dual-publish — or when the compact-format publishers start emitting an explicit subject PK — DHT-source coverage will catch up. Updates 26-DHT.md to document the flag, including the compact-format limitation and why auto's threshold is 10. * cli: route calc --source dht now recovers compact-format tp entries The compact tp format ([{r, t, l}]) emitted by deployment-side mirror publishers was previously skipped by --source dht because it omits the source PK in its value (only the storage-target hash encodes it, and SHA256 is one-way). Recovery is possible via cross-reference: every visor that publishes to both the dmsg and tp salts has its PK in the dmsg entry's `static` field. Since target = SHA256(pk || salt), the dmsg salt index gives us pk → tp-target. We hash every dmsg entry's PK with the tp salt to build a target → PK map, then resolve compact tp entries through that map. Synthetic transport IDs are generated from the (srcPK, rPK, type) tuple via transport.MakeTransportID — deterministic, so re-runs against the same data produce stable IDs. Also handles []transport.SignedEntry format (previously failed to unmarshal silently). Verified live on the dev visor: Before: --source dht returned 878 routes (max=4, count=0) After: --source dht returned 2674 routes — 3× coverage For comparison, --source tpd returns 4828 routes against the same network state. DHT coverage is now ~55% of TPD (up from ~18%) without changing any publisher; the residual gap is entries from visors that publish only one of dmsg/tp salts. Updates 26-DHT.md to describe all three supported formats and the recovery mechanism. * dht: chunk hub-edge tp publishes; consume compact-envelope format Two of the three follow-up fixes from the audit (Fix 2b — visor dual-publish of dmsg entries — turned out to already be implemented via HybridDiscClient.PostEntry/PutEntry mirroring to both DHT and HTTP, plus the dhtPublishLoop's 60s self-publish; no change needed). == Fix 1: compact-envelope consumer support The deployment-side mirror's compact tp format is [{r, t, l}] — single-letter tags omit the source PK in the value (it's only in the storage-target hash, and SHA256 isn't invertible). The 86d6d74 recovery uses the dmsg-salt's `static` field as a PK→target index; that helps but only resolves visors that publish to both salts. Adds support for an envelope shape that any future publisher can emit instead: {"s": "<source pk hex>", "ts": [{"r","t","l"}, ...]} The source PK is explicit (one PK per visor, not per transport, so hub edges don't blow the size budget) and consumers can decode without cross-referencing anything. cmd/skywire-cli/commands/route/calc.go's fetchAllTransportsFromDHT now tries four shapes in order: bare []transport.Entry, signed []transport.SignedEntry, compact-envelope {s, ts}, compact-array [{r,t,l}] (with dmsg cross-reference recovery). Common compactToEntry helper synthesizes Entry from a compact row plus a known source PK. == Fix 3: chunk hub-edge tp publishes TPDAdapter.putEntries was a single Put with an early "list too large" error if marshaling exceeded MaxValueSize (64 KiB). Hub edges with 250+ transports hit this and dropped silently after the log. Now chunks at tpChunkSize=200 entries per item; chunk 0 keeps the bare "tp" salt for back-compat, chunks 1+ go to "tp:1", "tp:2", … GetTransportsByEdge iterates chunks until missing or empty. prevChunkCount (atomic) on TPDAdapter remembers the last publish's chunk count so a shrunk transport list publishes "[]" tombstones to now-unused chunk salts; readers stop at the first empty chunk for i>0 so they never see stale tail. decodeTpItem extracted from GetTransportsByEdge handles per-chunk shape decoding (bare/signed) — no behavior change for chunk 0, just factored for reuse across chunks. == Spec docs 26-DHT.md: documents all four tp shapes (consumer side) and the chunking scheme (publisher side). * docs: move SKYWIRE_DHT.md to project root + refresh post-merge SKYWIRE_DHT.md is a descriptive comparison document, not a normative spec — it doesn't belong in skywire-specs/specifications/. Moved to the project root next to README.md, AUTO_UPDATE.md, NETWORK_DATA.md and the other top-level reference docs. The normative spec stays at skywire-specs/specifications/26-DHT.md. Refreshed for commits that landed after the original write: - Capsule summary notes hub-edge tp chunking past the value cap - Rough-edge #3 updated: 'dht peers' and 'dht reconcile' CLI now ship, gated to IsTrustedFullNode peers - Rough-edge #6: chunking is implemented for tp salt, scoped to publishers that need it; other salts may need the same later - Rough-edge #7 expands the data-format-coexistence note from svc to the four tp shapes (bare, signed, compact-array, compact-envelope) that route calc --source dht now decodes See-also links updated to reach into skywire-specs/specifications/ since SKYWIRE_DHT.md is now a directory level above them. * dht: TPDAdapter & SvcAdapter accept all wire shapes; unify via DecodeTpItem Pushes multi-shape tolerance from CLI / autoconnect down into the adapter layer so every reader benefits, not just the diagnostic paths that previously knew about it. == TPDAdapter decodeTpItem (called by GetTransportsByEdge for every per-PK lookup through the visor's transport manager) was bare + signed only: 1. Bare: []transport.Entry ✓ 2. Signed: []transport.SignedEntry ✓ 3. Compact-array: [{r, t, l}] ✗ silent miss 4. Compact-envelope: {s, ts: [{r, t, l}]} ✗ silent miss Compact entries on the network (~5% of tp salt today, plus whatever external publishers emit) were invisible to the visor's routing layer even though the CLI's `route calc --source dht` happily decoded them. That meant the visor's HybridTPDClient.GetTransportsByEdge fell through to TPD HTTP unnecessarily. decodeTpItem now accepts all four shapes. Compact-array uses the caller's pk argument as the source PK (which GetTransportsByEdge always knows — it IS the lookup key). Compact-envelope reads `s` from the value. compactTpEntry, compactEnvelope, and compactToEntry moved into pkg/dht/tpd_adapter.go. Public DecodeTpItem(v, srcPK) wrapper exposed for cmd/skywire-cli to reuse — calc.go's fetchAllTransportsFromDHT is now a tight ~25-line loop over rows that just delegates the per-shape decoding to the adapter package. -110 lines of duplication. == SvcAdapter LookupAll only handled []servicedisc.Service and errored on a single Service object. autoconnect.fetchPubAddresses had been inlining a dual-shape parse to work around this; LookupAll now does the same so any other caller through the adapter sees the legacy entries too. == Verification `route calc <pk> --source dht --max 4 -c 0` against the network: Before unification: 2674 routes (55% of TPD's 4828) After unification: 4175 routes (74% of TPD's 5636) TPD count fluctuates between runs (network state). The improvement ratio is consistent — same code now succeeds for the per-PK DHT path that the visor uses, not just the CLI scan. * docs: add CXO_VS_DHT.md retrospective Captures the architectural retrospective: could the work the DHT does today have been done with the CXO TreeStore primitive that's already in the tree? Mostly yes. Should it have been? Probably yes. Should we rip out the DHT? No. The doc walks through: - Side-by-side mapping of every property the DHT provides for the discovery use case to its CXO equivalent - Where the DHT genuinely differs (cross-publisher lookup without prior subscription) and why we don't actually use that primitive in production today (every visor either runs a full node or falls through to HTTP) - Where CXO is the better fit (history, dedup, no-cap merkle DAG, schema registry vs silent shape coexistence) - What we actually built on this PR — every gap-fixing mechanism (full-node reconcile, advertised full-node discovery, hub-edge chunking, multi-shape tolerance, DHT→HTTP mirror) brings the DHT closer to CXO's model, not further from it - Forward path: existing DHT salts stay, new decentralized state defaults to CXO, the framing of "DHT is the future" softens Lands at the project root next to README.md / SKYWIRE_DHT.md / NETWORK_DATA.md / etc. Not a normative spec — a retrospective.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Makes the wasm hypervisor (
cmd/dmsg-wasm) compile under TinyGo, removingnet/httpfrom its graph entirely, and adds two more browser transports (WebTransport, WebRTC). Result: a 6.5 MB wasm (vs 21.6 MB standard-Go; 2.4 MB gzipped) — small enough to embed by default.Everything new is build-tag-gated to
wasm/tinygo; no production (native) build is affected. Verified across nativego build ./..., std-Go js/wasm, andtinygo build -target wasm.Why
net/httpis broken on the TinyGo js target (roundtrip_js.go) and can't be forked (stdlib paths aren't redirectable), so it's removed from thecmd/dmsg-wasmgraph. It was pulled by three things, each given a native/tinygo split (native keepsnet/http; tinygo gets a net/http-free equivalent):pkg/wasmhv/gobrpc.go, a client speaking the exact net/rpc gob wire protocol (tested against a real net/rpc server); noiseRPCClientDialersplit torpcdialer.go(!tinygo).dmsgclient.dmsgDiscClientspeaks HTTP/1.1 straight over a dmsg stream (httpdmsg.go— Content-Length/chunked/Connection:close framing, tested vs a realnet/httpserver).seeded.go's upgrade splits native/tinygo; the net/http-freefallbackDiscClientis extracted tofallback_disc.go; net/http+cobra CLI files move behind!tinygo.Dial's*http.Response) →ws_js_tinygo.godials the browser-native WebSocket viasyscall/jsas a deadline-awarenet.Conn.ServeWSsplit tows_server.go(!tinygo). Onlytinygo && js && wasmis affected — native, std-Go wasm, tinygo+wasip1/IoT keep coder/websocket.Browser transports
ws_js_tinygo.go.wt_js_tinygo.go, CA-free viaserverCertificateHashescert-hash pin; mirrors nativewt.go.cmd/dmsg-wasm/webrtc_js.go; dmsg is the signaling plane (offer/answer/ICE over a dmsg stream), DataChannel →net.Conn. JS APIwebrtcDial/webrtcListen.Validation
make tinygo-dmsg-wasm && go run cmd/dmsg-wasm/serve.gobuilds + serves the harness (cmd/dmsg-wasm/index.html) for manual validation against a live dmsg server — the gate before building on the WebRTC/WebTransport paths.Direction (design only, not implemented here)
docs/design/wasm-visor-p2p.md+cmd/wasm-visor-probemap how much of the visor can port to wasm:pkg/routing+pkg/visor/visorconfigalready compile under TinyGo; the rest is blocked by a small recurring set (quic-go raw-socket networks, net/http discovery clients, net/rpc, os/exec). Includes a phased plan + an app-feasibility analysis ("reachability ≠ listening"; a tab is a routing/publishing participant — a consumer of infra apps, not a provider).🤖 Generated with Claude Code