A warm suggest click costs about 4.8 s, and almost all of it is the model's own decode

[JArmandoAnaya]

Interactive suggest is slow on every click, not only the first one. An instrumented run on
2026-08-12 measured where a click's time actually goes, and it is not where the previous round of
work looked. The measurements below were taken on the CPU-inference container
(torch 2.13.0+cpu, 16 cores visible, torch.get_num_threads() reporting 8), on an 854×480
asset, through a class that admits polygons, with clicks issued one at a time so that nothing
overlapped. Every absolute number here is a CPU figure and does not transfer to a machine with a
GPU; the ratios and the counters do.

Where a click's time goes

stage	click 1, cold process	click 2, same asset
connection resolve	2.1	2.8
provider pool get, including the first torch and transformers import	4,564.9	0.2
model load	735.2	0 — resident
asset and blob read	4.3	4.4
image encode	34,935.0	0 — cache hit
point decode	4,663.2	4,550.3
post_process_masks	95.3	95.6
mask.tolist()	24.3	2.1
shapes_from, the post-processing pipeline	105.1	126.8
total	45,130.8	4,783.1

A third click on the same asset, also a cache hit, came in at 4,931.2 ms with a 4,671.0 ms decode.

The embedding cache works, and is not what to change

The dogfooding round that reported every click costing the same amount (#557, item B) named the
embedding cache as the suspect, and the pull request that answered it (#562) already argued from
the code that the cache was present and untouched. Instrumentation now confirms that from the
other side: the provider's encodes counter reaches 1 and stays at 1 across the second and third
clicks on an asset, the provider pool reports the same held provider with builds stuck at 1, and
the model logs itself as already resident. The 34.9 s encode is genuinely paid once per asset.

Suggest also runs in-process in the FastAPI application. Nothing in visionset.jobs participates,
and no process is spawned per request, so the encode-once-decode-per-click shape the design asks
for is not precluded by the topology. It is already happening.

The decode is the cost

model(...) in LocalSamProvider._one costs about 4.55 s on every click, whether or not the
image was already encoded. Six measurements across three separate processes: 4,548.5, 4,548.6,
4,544.2, 4,564.5, 4,661.9 and 4,671.0 ms. That is roughly 95% of a warm click's 4,783 ms, so a
person clicking repeatedly on one frame sees a constant several-second wait that reads exactly
like a cache that is not working.

The full-resolution mask conversion tracked separately (#561) is real but is not this: at this
resolution mask.tolist() costs between 2.1 and 24.9 ms, and the whole post-processing block
comes to about 225 ms, under 5% of a warm click. Its weight on a 4K asset was not measured.

One observation is unexplained and worth recording before somebody re-derives it. The single
fastest decode measured, 72.7 ms, came from the one request that had performed its own encode
immediately beforehand; every decode reading an embedding out of the cache cost about 4.55 s. That
is a sixtyfold difference on the same model, the same asset and the same process.

Concurrent clicks defeat the cache and multiply loaded models

Two clicks in flight at once on an asset that has not been encoded yet both miss the cache and
both encode. The instrumented lines read MISS held=0 encodes=0 and then, for the second request
on the same asset, MISS held=0 encodes=1: the first request had incremented its counter but had
not yet reached the point where it stores the result, so the second found nothing.

The pool races the same way. Two concurrent first clicks each logged a provider build from a
builds count of 0, and one process ended up loading the model four times. This is reachable
rather than theoretical: _embedding has no in-flight guard, BoundedCache documents itself as
deliberately not thread safe, ProviderPool likewise, and the route handler is a plain def, so
FastAPI runs concurrent suggests in parallel threadpool threads.

The consequence is not only latency. During one burst of concurrent suggests the whole development
compose stack went down, with nginx exiting 137. Container memory at rest is 113 MiB against
15.69 GiB, so this is what several simultaneous encodes and several loaded model copies transiently
allocate, not steady-state footprint.

What this asks for

1. Establish whether cached embeddings carry a live autograd graph that every later decode pays
   for, and act on the answer. The encode at LocalSamProvider._embedding runs outside the guard
   the decode uses, so the question is worth settling with evidence rather than reasoning.
2. Make the embedding cache and the provider pool single-flight. For one asset, concurrent
   suggests perform exactly one encode; for one connection, exactly one provider build. Different
   assets and different connections must not serialise behind each other.
3. Settle the checkpoint warning. transformers reports on every load that a sam2_video-typed
   checkpoint is being instantiated as Sam2Model. Either load it as the class transformers
   considers correct, preserving image point-suggest and the pinned-revision discipline, or record
   why the mismatch is benign at the load site. Suppressing the warning cosmetically is not one of
   the options.