Mesh implementation for TactilityOS #619
Replies: 6 comments
|
The Heltec is a bit hard to read on-screen, but it averages around 98-100 KB free in the logs. |
|
Cardputer (non-ADV) has 86 KB free with WiFi on and 132 KB free with it off. |
|
This is on my list to process, but I've been busy with other things, so I'll get to this in the coming week. |
|
Title 12 answers:
My own notes: Before this is implemented, we need a change on the The problem with callbacks is that they use the callstack from the subsystem that emits them. This is problematic in terms of callstack size, because all these subsystems have to create massive callstack sizes to facilitate a subscriber's calls to file system, LVGL, network, etc. For example: A task for a service or an app creates an event group. It then uses the event group to register for events with other subsytems such as wifi. The task can then use the event group to await for events. At this point the task is idle and not using CPU cycles. Once an event is triggered, the event group is notified and the task awakens. The task then polls which events are available and process them. |
|
I already started working on refactoring |
|
The |
Uh oh!
There was an error while loading. Please reload this page.
RFC: a mesh service for Tactility
Status: draft for discussion. Nothing here is built yet, and this is deliberately
being circulated before it is, so the design can be argued with while changing it is still
cheap.
What this asks of the project: three small additions to the kernel LoRa API
(section 9), and a decision on packaging (section 7). Everything else lives outside the
Tactility tree.
((This is what i've been falling down protocol and radio rabbit holes and understanding meshtastic and meshcore better since getting the initial radio driver working... when i had the time and energy. I might have gone slightly mad at one point, but i think this is doable feature, but I want more perspectives and eyes on it.))
1. What this is
The SX1262 driver landed and works. A Tactility device can transmit and receive LoRa
packets today. What does not exist is anything that keeps a radio useful: something that
owns the radio, stays running when the UI closes, understands a mesh protocol, remembers
what it heard, and lets applications talk to it.
This proposes a resident mesh service with pluggable protocol backends, supporting
both Meshtastic and MeshCore, exposing an API that other people can write user interfaces
against.
Two protocols, because both have real users and both have been asked for. One service,
because as section 2 shows, the hard part of the work is identical for both and neither
upstream project solves it.
For readers who do not follow the mesh world
Meshtastic and MeshCore are both open source off-grid mesh messaging protocols
running over LoRa radio. They solve the same problem and are completely incompatible with
each other: different packet formats, different routing, different cryptography, and
different LoRa sync words, which means a radio tuned for one literally cannot hear the
other. They are not variants. They are two separate networks that happen to use the same
chip.
Decisions already taken
These were settled before writing, and the reasoning is given where it matters. They are
open to challenge, but the document argues from them rather than relitigating them.
2. Why one service, and not two applications
The instinct is that two incompatible protocols mean two separate apps. The evidence says
otherwise, and it comes from the upstream projects themselves.
Neither Meshtastic nor MeshCore solves the problem this service exists for. Both are
built on the assumption that a phone is attached most of the time. When the client goes
away, here is what each one keeps:
MAX_RX_TOPHONE); 16 or 32 on the ESP32-S3 parts Tactility runs onOFFLINE_QUEUE_SIZE)MSG_WAITING, thenSYNC_NEXT_MESSAGEuntilNO_MORE_MESSAGESA few dozen packets in volatile RAM. Close the UI for an afternoon and the traffic is
gone. MeshCore's model is the better of the two, since it at least has a sync cursor and
an eviction policy that favours direct messages, but neither is a message store.
Note the second row in particular. Neither project survives a reboot, so on both, a
flat battery loses everything a user had not yet read. Section 5 proposes writing every
message to durable storage on arrival instead, and that single difference is what makes
the rest of the design worth building.
So durable storage, per-subscriber cursors, persistent channel configuration and radio
arbitration are new work regardless of which protocol you support. All of it is
protocol independent. That is the majority of the engineering and all of the value over
just running stock firmware.
Building two separate applications means building that expensive layer twice and exposing
it to two upstream projects' churn twice. One service with thin backends builds it once.
3. Architecture
Three properties constrain everything else.
One backend is active at a time. Meshtastic uses LoRa sync word
0x2b; MeshCore uses0x12. Two radios with different sync words cannot hear each other even on an identicalfrequency and spreading factor. This is not a limitation of the design, it is a fact of
the protocols, and no amount of architecture works around it. Backend selection is
configuration; switching retunes the radio. Simultaneous dual-protocol operation would
need two radios.
Switching raises a state question the design has to answer rather than discover: what
happens to outbound messages still queued for the backend being switched away from. They
cannot be delivered, and silently dropping them is the worst option. The proposal is that
queued sends survive the switch, stay visible through the API with a blocked status, and
resume if the user switches back, with an explicit way to cancel them.
One work queue, one thread. LoRa RX callbacks, timers, and inbound API calls all post
to a single queue, and all state mutation happens on the thread that drains it. No locks
inside the protocol core. The driver's RX callback copies its buffer and posts, never
blocking, because it runs on the driver's radio thread and stalling it stalls reception.
The store is a peer of the backends, not a part of them. Backends write into it, they
do not own it. This is what makes a neutral API possible and what keeps two protocols from
bleeding into each other. It gets its own section because it is easy to get wrong.
4. The API
The headline: existing phone apps work unmodified
The Meshtastic backend speaks the stock
ToRadio/FromRadioprotocol. Carried overBLE, that is exactly what the official Meshtastic phone applications already talk to.
A Tactility device running this service should appear to an unmodified Meshtastic phone
app as a Meshtastic node.
That is the single strongest argument for this design. It means the project does not have
to build and maintain a phone app to be useful, and it gives users a reason to try a
Tactility device without giving anything up.
One thing has to be resolved before that claim is fully true. Meshtastic identifies
nodes by a
HardwareModelenum, and there is no value for Tactility devices. A phone appshows users whatever the device claims to be, and picking some other vendor's identifier
would misreport the hardware to everyone on the mesh. The options are to request an
identifier from the Meshtastic project, to use the existing private or unset range and
accept a degraded display, or to ship with the device reporting itself generically. This is
the only open question in this document that cannot be settled inside Tactility, and it
argues for talking to the Meshtastic maintainers early rather than after shipping.
Neutral core
For the majority of what a UI needs, and identical across backends:
A UI written once against this works against either protocol. This is what makes a
third-party message viewer, or a notification widget, or a status bar item, achievable
without the author learning either protocol.
Raw escape hatch
The neutral API deliberately does not try to cover everything, because the protocols
genuinely diverge. Anything protocol-specific goes through a raw byte channel:
This is how repeater CLI commands, admin messages, traceroute, telemetry queries and
anything either project adds next stay reachable, without the service needing a change
every time an upstream adds a feature. Subscribers opt in to raw frames.
Bytes, not structs, at the boundary
The API passes encoded bytes, not protobuf structures. Applications loaded as ELF binaries
link their own nanopb build, so a struct boundary would turn every protobuf regeneration
into an ABI break. This was settled by an earlier prototype and is not reopened here.
One trap worth knowing about
Setting a channel looks like it should be the same operation on both protocols. It is not.
xorHash(name) XOR xorHash(psk). The name iscryptographically load-bearing. Get the name wrong and the packet never decrypts, even
with the correct key.
SHA256(secret)[0]. The name is a purely locallabel, never transmitted and not part of the hash.
So
setChannel(name, key)means two different things. The API has to expose thatdifference rather than paper over it, or users will silently fail to join channels and
have no idea why.
5. The message store
The requirements below are each easy to get wrong and expensive to fix later.
The scenario that drives all of them: a user runs Meshtastic for a while, switches to
MeshCore without having read their Meshtastic messages, and then the device reboots. Those
messages should still be there, still readable, and still clearly labelled as Meshtastic.
Neither upstream project can do this, and it is the single clearest thing the service adds.
Durable, not cached
Every received message is written to durable storage as it arrives. Not buffered in RAM
and flushed later, because a reboot or a flat battery can happen between any two packets.
Messages are retained until storage pressure forces the oldest out, and nothing else evicts
them.
Write path. The radio thread copies its buffer and posts, as it must. The commit to
storage happens on the service thread, one record per message. Flash writes stall, so they
never happen on the radio thread, and the existing single-work-queue design already puts
them in the right place.
Layout: a segmented append-only log. Records are appended to the newest segment.
When a segment fills, a new one starts. When the store reaches its configured budget, the
oldest segment is deleted whole. Coarse eviction is deliberate: no rewriting, no
compaction, no fragmentation, and an interrupted delete leaves the log consistent.
Flash wear is the first objection anyone will raise. The arithmetic answers it. LoRa is slow. A very busy mesh might produce a few
hundred KB of message records per day. Against a 1 MB partition with ESP-IDF wear
levelling, that cycles the partition every few days, and ESP32 SPI flash is rated in the
tens of thousands of cycles. That is decades. The data rate that makes LoRa frustrating
to use is the same data rate that makes durable logging free.
Boot cost. Cursors and per-conversation queries need an index. Persisting one on every
write would double the write traffic, so the index is held in RAM and rebuilt at boot by
scanning record headers. If records are encrypted the scan decrypts as it goes, which AES
acceleration makes cheap at these sizes; if they are not, it reads them directly. Either
way the scan is linear in log size, which bounds how large the log should be allowed to
grow.
What actually gets stored. Messages, meaning user-visible content, are durable.
Telemetry, position and node-info traffic updates node state rather than accumulating,
or a chatty sensor node would evict a user's actual conversations. Raw packet logging is
useful for research and debugging but is separately bounded and off by default. This split
matters and is easy to get wrong by storing everything uniformly.
Independent of the backends
Backends normalise what they receive into a common record: conversation, sender,
timestamp, body, delivery state, and the backend that produced it. Backends write. They do
not own. Applications read one store through one API regardless of which protocol is
currently running.
Namespaced per protocol
Every record is tagged with the backend that wrote it, and every conversation, contact and
channel identifier is a
(backend, native_id)pair, never a bare id.This is not defensive tidiness. The identifier spaces genuinely collide:
Meshtastic channel
0x42and MeshCore channel0x42are unrelated. Storing them underthe same key would silently interleave two different conversations.
NodeNumplus a channel index on Meshtastic, and anEd25519 public key prefix plus a channel hash on MeshCore.
Consequences: queries are backend-scoped by default, cross-backend listing is explicit and
returns namespaced ids, and switching protocols never migrates or reinterprets existing
records. If a user runs Meshtastic for six months and then switches to MeshCore, the old
messages stay readable and stay clearly labelled as what they are.
Stored as plaintext
There are three things the store could hold. The first is ruled out on engineering grounds
rather than security ones.
Storing messages as received, still in protocol ciphertext, and decrypting on demand.
This looks appealing because the encryption comes free. It does not work:
key, and every stored message on that channel becomes permanently unreadable. An archive
should not be hostage to a key the user has good reason to change.
require the Meshtastic backend to be loaded, even while MeshCore is the active protocol.
That is exactly the cross-protocol entanglement the rest of this section exists to
prevent.
inside the encrypted payload, so listing a conversation would mean decrypting the whole
range on every read, including an X25519 agreement per contact for direct messages.
storage format preserves its weaknesses.
So messages are decrypted once on arrival and normalised. That leaves plaintext or
re-encryption under a device key, and this proposal stores plaintext.
The reason is not per-message cost. On an ESP32-S3 with an AES accelerator, encrypting a
few hundred bytes a handful of times per second is far below measurable. The reasons that
do hold:
encryption depends on flash encryption, which Tactility does not enable and which would
mean burning eFuses and giving up plain reflashing. That is not a reasonable default for
a hobbyist operating system. So anyone who can dump the flash recovers the channel
pre-shared keys and the identity key regardless, and can then read the live network
rather than merely the archive. Encrypting the message log while the keys sit beside it
in recoverable storage buys very little.
during that scan is the one place the cost is real, and it grows with exactly the thing
durable storage is meant to make large.
corrupted key makes all history unreadable.
What this means, and it should be documented for users rather than hidden: anyone who
takes the device can read the message history. Meaningful at-rest protection needs flash
encryption at the platform level, which is a Tactility-wide decision and not this service's
to make. If that ever lands, the store should gain an encrypted record codec at the same
time, so the layout keeps a codec seam rather than assuming plaintext throughout.
Ordering without a reliable clock
Durable storage makes an existing problem permanent. Cursors and history need timestamps,
but T-Deck Max, the only board that can run this today, has no RTC in its devicetree.
Time is lost on every reboot, and off-grid there is no NTP. So this is not an edge case to
handle later; it is the normal condition on the target hardware.
Time sources, best to worst. The kernel already provides the seam:
RTC_TYPEexposesrtc_get_timeandrtc_set_time, so whatever source wins can persist time on boards thathave a clock to persist it to.
accurate source available off-grid.
MeshCore adverts and text messages carry a 4-byte Unix timestamp, and Meshtastic
distributes time through position payloads. Adopting one gets a device from "no idea" to
"roughly right" within the first few packets.
than a fabricated one.
On an RTC-less board with no GPS, source 4 is what the device actually runs on, so it
deserves to be designed rather than bolted on.
Adopting mesh time needs guards. These timestamps are self-asserted by the sender.
Meshtastic's are unsigned; MeshCore's adverts are Ed25519-signed, which makes them
attributable to a keyholder but not correct. A node with a broken clock, or a malicious
one, can push time anywhere.
further ahead than a fixed horizon. This is cheap and it stops the worst case.
2038 and every subsequent real message sorts before it, which breaks "messages since"
semantics permanently. Monotonic-forward-only adoption makes this worse, not better,
because it is exactly what makes a bad future value stick.
let corroboration from several nodes firm it up, and let any higher-ranked source
override it outright.
What records carry. Every record stores a monotonic sequence number, the best available
wall-clock time, and a tag naming which source that time came from. Ordering always
uses the sequence number, so history sorts correctly even when the clock was wrong or
absent. The wall clock is a display value, and the source tag lets a UI say "approximate"
rather than presenting a guess as fact.
History is never rewritten. Learning true time later must not trigger a pass over
stored records: it is a write amplification problem on flash and it invalidates any cursor
a client is holding. Recording device uptime alongside each record is enough to compute a
correction after the fact without touching what was already written.
This is a record layout decision, so it should be settled before the store format is
fixed rather than after.
6. Designing against the ESP32
The ESP32 is capable but small, and several parts of this design exist specifically
because of that.
The radio is not the bottleneck. At SF11 / BW250 a full packet occupies roughly one to
two seconds of airtime. A busy mesh delivers a few packets per second. This is not a
throughput problem and the protocol core is not CPU-bound by reception.
RAM and crypto latency are the real constraints. Measured on the existing prototype:
meshtastic_FromRadiostructmeshtastic_ToRadiostructmeshtastic_NodeInfoLite768 bytes is too large to sit in a callback stack frame, so the design uses one shared
scratch buffer rather than a buffer per call.
X25519 key agreement costs tens of milliseconds per peer on this hardware, so shared
secrets are cached per contact, exactly as stock firmware does. Doing it per packet
would be visible as latency.
Storage is internal flash, not SD, which is counterintuitive. T-Deck Max is currently the only board in the tree with an SX1262 in its
devicetree, and it is also the board where SD support is not yet working
(
storage.userDataLocation=Internal). So the durable log in section 5 has to live in abudgeted internal-flash partition from day one. SD raises the budget on boards that have a
working one, but it is not the baseline and the design cannot assume it.
The LoRa-capable boards, as they actually are today
lilygo-tdeck-maxlilygo-tdeck-prolilygo-tlora-pagerheltec-wifi-lora-32-v3m5stack-cardputerm5stack-cardputer-advOnly one board can run this today. The others need devicetree work before any of this
matters to them, which is a good early contribution opportunity for anyone who owns one.
7. Packaging: an open question, not a decision
The intent is an installable service, not something baked into every firmware build.
Whether that is achievable on every board is genuinely unresolved, and the project's input
is wanted.
Correcting an assumption that earlier drafts of this project's own documents got wrong: ELF loading is not gated on PSRAM.
CONFIG_ELF_LOADER_LOAD_PSRAMis optional,and with it disabled
esp_elf_mallocfalls back toMALLOC_CAP_EXECin internal SRAM(
Libraries/elf_loader/src/esp_elf_adapter.c).So the real constraint on a board without PSRAM, such as
heltec-wifi-lora-32-v3, isinternal SRAM budget after the OS, LVGL, and the BLE and WiFi stacks have taken their
share. Nobody has measured whether a mesh service fits there. Three options:
fits.
it does not.
The honest position is that this should be measured before it is decided. A concrete
early task: build the service, and compare its text and data size against free internal
SRAM on a Heltec V3. Opinions from anyone who knows the SRAM headroom on these boards
would short-circuit a lot of guesswork.
8. Full participant, and what it costs
A device running this service relays traffic for other nodes. That is what the mesh
expects of every node, and a network of listeners that never repeat degrades for everyone.
The costs:
a duty cycle, in the radio manager, shared across backends.
accounting is achievable now, purely in software. Carrier sensing is not, because the
driver API for it does not exist. That bounds what can be claimed for European regions:
duty-cycle limited, yes; listen-before-talk, not until the kernel supports it. This is
section 9's first ask.
transmit. This matches stock Meshtastic, which refuses both transmit and receive when
the region is unset, and it is the right default for a device that may be carried across
borders.
Proposed default: relay enabled where the configured region permits it, with the budget
enforced centrally and visible through the API.
9. What this needs from Tactility
These are the only asks of the Tactility tree, they are all in the kernel LoRa API, and
they are all Apache-2.0 with no licence entanglement from either mesh project. Each would
be a separate issue and pull request, hardware-tested as usual.
1. Radio ownership and arbitration.
lora_find_first_registered_device()hands thesame
Device*to every caller, and radio parameters are global and apply on next enable.Two consumers silently fight today. With a resident service plus any other radio user,
that stops being hypothetical.
2. Channel activity detection, noise floor, and a "currently receiving" query. Both
mesh protocols want these. MeshCore's radio abstraction declares all three
(
setCADEnabled,getNoiseFloor,isReceiving); Meshtastic uses carrier sensing in itsRadioLib layer. Tactility's
LoraApihas none of them. This is the gap that limitsEuropean regulatory compliance in section 8.
3. Region-aware transmit power and current limit enforcement. Both are settable
today, but nothing ties them to a region's legal limits.
All three benefit any future radio consumer, not just this service.
10. Both protocols move fast, in different ways
Anyone reviewing this should know what maintaining it commits to. The two projects churn
very differently, and the architecture is shaped around that.
Meshtastic churns constantly, but mostly additively. The 16-byte packet header has not
changed shape in two years. Version 2.6's next-hop routing was added by repurposing
existing header bits. Protobuf absorbs new fields, and old parsers skip what they do not
know. Where it does break is above the wire: node database refactors, phone API replay
semantics, and proto3 explicit-presence changes that are wire-compatible but shift
generated struct layouts. Note that breaking work lands continuously into alpha
releases, not at version boundaries, so there is no stable tag to track. The service
therefore declares a supported protocol version rather than chasing the development
head.
MeshCore churns less often, but breaks the wire when it does. It is hand-rolled binary
with no extensibility mechanism, so new capabilities move the format. In 19 months: four
new payload types, a header change adding optional transport codes, a redesign of the
trace payload one month after it shipped, a repacking of the path length byte to allow
multi-byte path hashes (older firmware drops those packets outright), a change to group
and channel framing, and wider ACKs. The companion protocol version went from 5 to 8 in
six months.
The practical difference: Meshtastic churn costs you a rebuild and a retest. MeshCore
churn costs you interoperability with already-deployed nodes.
The mitigation is the architecture itself. Backends stay thin and sit behind a byte-level
boundary, and the expensive durable layer is insulated from both.
11. Roadmap, and where help is wanted
Phase 0, prerequisites. The three kernel additions in section 9, and the SRAM
measurement from section 7. Devicetree entries for the other LoRa boards.
Phase 1, the service. Skeleton, store, neutral API, and the Meshtastic backend, which
builds on an existing protocol core that already runs on hardware.
Phase 2, BLE. Phone API over BLE, at which point unmodified Meshtastic phone apps
start working.
Phase 3, MeshCore backend. Explicitly help-wanted. The backend interface will be
documented well enough to implement against without touching the core. MeshCore's protocol
core is around 2,400 lines, is genuinely radio-agnostic, and its abstraction seams are
cleaner than Meshtastic's, so this is a tractable piece of work for a contributor who
cares about that network.
Alongside all of it, the things that make a project rather than a code drop: tests running
on the POSIX simulator (the protocol core already builds and tests there), wire-format
regression tests against captured reference payloads, CI, a contribution guide, and this
document maintained as the design record.
12. Questions for reviewers
Genuinely open. Answers will change the design.
Licensing versus the goal. The service is GPL-3.0, because Meshtastic's firmware
and protobufs are, and MeshCore is MIT which combines in. But the stated aim is for
other people to write UIs against it. Proposal: publish the API contract and a client
shim permissively, keeping the implementation GPL-3.0, similar to how kernel headers
are treated. Is the project comfortable with that split? This needs a real legal
opinion, not mine.
Packaging on boards without PSRAM (section 7). Which of the three options, and does
anyone already know the SRAM headroom?
How should a Tactility device identify itself on the Meshtastic network? See
section 4. This needs a decision about engaging the Meshtastic project, and it is the one
open question here that cannot be settled inside Tactility.
How much flash should the message log get? The policy is settled in section 5
(durable, append-only, oldest segment evicted under pressure); the budget is not. What
is a reasonable default partition size on a 16 MB board, and is a partition table change
acceptable, or should the log live inside the existing user data area?
Should this eventually live in-tree, or remain a separate installable indefinitely?
Is there real appetite for the MeshCore backend, or should the first release be
Meshtastic-only with the interface left as a documented extension point?
Appendix: where the claims come from
Every factual claim above was checked against source rather than recalled.
0x2bfirmware/src/mesh/RadioLibInterface.h:840x12)MeshCore/src/helpers/radiolib/Custom*.hMAX_RX_TOPHONE8 / 16 / 32 by platformfirmware/src/mesh/mesh-pb-constants.h:21-37OFFLINE_QUEUE_SIZE16MeshCore/examples/companion_radio/MyMesh.h:63MeshCore/examples/companion_radio/MyMesh.cpp:220-239SHA256(secret)[0]MeshCore/src/helpers/BaseChatMesh.cpp:887Tactility/Libraries/elf_loader/src/esp_elf_adapter.c:34-44Tactility/Devices/*/device.propertiesmeshtac-development-notes.mdReference checkouts used: Meshtastic firmware
f18a8b0(2026-07-16), MeshCored929643(v1.17.1, 2026-08-14).
All reactions