Local inference cannot use the Apple Silicon GPU: mps is refused by the device vocabulary

[JArmandoAnaya]

A local inference connection names the device it runs on, and the vocabulary of legal
device names is closed. Today that vocabulary is cpu, cuda, and cuda:N for the second
GPU on a machine that has one. Apple Silicon's GPU, which PyTorch addresses as mps, is not
in it, so a Mac with an M-series chip runs SAM 2 and Grounding DINO on its CPU cores while the
GPU sits idle. The interactive click-to-suggest loop is where this is felt: a warm suggest
click is dominated by the model's own decode (#571), and that decode is exactly the work a
GPU does faster.

This is a deliberate exclusion, not an oversight

mps is refused on purpose. src/visionset/kernel/domain/inference.py closes the vocabulary
with a regular expression and states the reason beside it:

DEVICE_PATTERN: Final = re.compile(r"^(?:cpu|cuda(?::\d+)?)$")
"""Every device string this build can honestly run on.

A pattern rather than an enum because of the one member that is not a fixed
word. What is *not* here is the point: ``gpu``, ``mps``, ``auto`` and every
typo were accepted before and then quietly fell back to the CPU in full
precision — a connection that names a runtime it never gets.
"""

The rule that docstring states is the right one: a device belongs in the vocabulary when this
build can honour it, and a name that silently degrades to something else is worse than a
refusal. gpu and auto still cannot be honoured and stay out. What changes is that mps
can be honoured, once the adapters know how — which is what this issue asks for.

What has to change

The device vocabulary is written down in more than one place, and all of them move together
or the halves disagree.

• The kernel domain owns the vocabulary: the pattern above, OFFERED_DEVICES, and
  precisions_for, which conditions the legal precisions on the device. mps is
  full-precision only — MPS has no float64 and its bfloat16 support is inconsistent — so
  precisions_for("mps") answers fp32 alone, and the existing cross-field validator then
  refuses an mps connection asking for fp16 with the sentence it already writes for
  cpu + fp16.
• Both local adapters decide at load time where a model actually goes. LocalSamProvider
  and LocalTransformersProvider each carry a private _resolved_device, and the two are
  identical: a connection asking for a device this machine does not offer gets the CPU and a
  warning naming the connection, and half precision is dropped on the way. Adding a second
  device to two identical copies is the moment to promote them into one shared resolver
  rather than to write the new branch twice. Half precision stays CUDA-only, so it must not
  reach mps by the widening.
• PYTORCH_ENABLE_MPS_FALLBACK has to be set, because an operator MPS has not implemented
  otherwise raises instead of running on the CPU for that one operation. It is read when the
  library initialises, so it has to be set before torch is imported rather than before the
  first tensor operation.
• The surfaces that publish the vocabulary: the connection form's device list in
  frontend/ui-core/src/screens/inferenceCatalog.ts, the --device help text on
  visionset inference create and update, and the closed-vocabulary prose in
  docs/inference.md, docs/cli.md and docs/install.md.

What does not change

The installation does not. The local-inference extra declares version floors and pins
nothing platform-specific, and the macOS arm64 PyTorch wheels carry MPS already, so there is
no install-time branch to add and no second index to configure. A user on Apple Silicon
installs the same extra everybody else installs.

The default does not either. A new connection still opens on cpu; mps is offered in the
picker rather than chosen for the user.

Acceptance

• A local connection may name mps, and one that does runs on the GPU of a machine that has
  one.
• mps accepts fp32 and refuses fp16, with the refusal naming the device and the
  precisions it does run in.
• An mps connection on a machine without MPS falls back to the CPU and says so at WARNING,
  which is the behaviour cuda already has on a machine with no GPU.
• gpu and auto are still refused.
• The device-resolution rule exists once, not once per adapter, and has tests — it has none
  today in either copy.
• docs/inference.md has a section covering which devices are supported, what a connection
  names, what happens when the named device is absent, and that Apple Silicon needs nothing
  installed or configured beyond picking the device.

Continuous integration has neither a GPU nor an Apple Silicon runner, so the resolution rule
has to be testable with the availability answers injected, the way CONTRIBUTING.md already
requires of anything that would otherwise need real hardware.

[JArmandoAnaya]

Decision (Armando, 2026-08-12): mps becomes a member of the local device vocabulary rather than a new automatic mechanism. A local connection already names its device explicitly, and the adapters already fall back to the CPU with a warning when that device turns out to be absent at run time — those two together are the override and the automatic path, and a second source of truth for the same question, whether an environment variable or an auto member, would only give them a way to disagree. So DEVICE_PATTERN, OFFERED_DEVICES and precisions_for admit mps; mps is full-precision only, because MPS has no float64 and its bfloat16 support is inconsistent, and half precision stays CUDA-only in both adapters; and an mps connection on a machine without MPS falls back to the CPU with the same warning cuda already gets, rather than refusing, because a workspace configured on one machine has to stay usable on another. There is no install-time platform logic: the macOS arm64 PyTorch wheels ship MPS by default and the local-inference extra pins nothing platform-specific. The adapters set PYTORCH_ENABLE_MPS_FALLBACK=1 themselves, so the user configures nothing beyond choosing the device, and the connection form and --device keep their cpu default. PyTorch remains the sole inference runtime for 0.1.0; ONNX Runtime is out of scope for this distribution, and were it ever taken up it would land as an additional driver under the closed capability vocabulary being designed in the provider-contract epic (#498), never as a replacement. — supersedes any prior direction