Skip to content

fix(downloads): resolve a client's path mappings once per batch, not per item - #871

Open
m4bard wants to merge 2 commits into
Listenarrs:canaryfrom
m4bard:fix/gateway-path-mapping-concurrency
Open

fix(downloads): resolve a client's path mappings once per batch, not per item#871
m4bard wants to merge 2 commits into
Listenarrs:canaryfrom
m4bard:fix/gateway-path-mapping-concurrency

Conversation

@m4bard

@m4bard m4bard commented Aug 21, 2026

Copy link
Copy Markdown

Summary

DownloadClientGateway.GetQueueAsync fans out over every queue item, and each item translated its paths by calling TranslatePathAsync, which queries the repository for that client's mappings on every call. The service, the repository and the ListenArrDbContext behind them are all scoped, so a queue of N items issued up to 2N concurrent queries against a context that permits one at a time. DownloadClientQueuePoller runs that inside its own Task.WhenAll across every client, so the fan-out is nested.

Full write-up in #870.

Changes

Fixed

  • IRemotePathMappingService gains a TranslatePath overload taking mappings the caller has already resolved. It does no I/O. TranslatePathAsync keeps its behaviour by fetching and delegating to it, so existing callers are unaffected.
  • DownloadClientGateway resolves a client's mappings once before each of its two item fan-outs and passes them down.
  • The single-item import path resolves once as well, where it is one query either way.

Fewer queries as well as non-overlapping ones: a queue resolves the client's mappings once per batch rather than once per translated path. Corrected 2026-08-24: this line previously said a thirty-item queue goes from up to sixty queries to one. That held only for items carrying no source files, because the first commit left the SourceFiles loop querying per file. The follow-up commit fixes that; see the comment below.

On #783

That issue reports this exception class with a trace through DownloadClientQueuePoller.FetchAsync. That class holds no repository and no context, and did not at 4555ad21 either, which is the commit it reports against. The trace surfaces there because that is where the await unwinds; the query is further down this chain.

I have not reproduced #783's original scenario and this does not claim to close it.

Testing

DownloadClientGatewayPathMappingConcurrencyTests counts the overlap directly rather than racing the EF exception, which would make for a flaky test and a weak claim. A stub records the highest number of lookups in flight, with a small delay so the overlap is observable rather than a scheduling accident.

Ten items carrying two translatable paths each report 10 concurrent lookups without the change and 1 with it. The lookup count is also pinned at 1, so a change that restores per-item querying fails even if it somehow avoids overlapping.

Full suite: 3,031 passed, 0 failed, 125 skipped, against a 3,029 baseline on 03958c15.

Note

This is the second instance of the same pattern, after #862 and its PR #863, which fixes a smaller one in search-result scoring. They are separate files and independently revertable, so they are separate PRs. I originally reported that scoring site as the only instance; that was wrong and is corrected on that issue.

…per item

DownloadClientGateway.GetQueueAsync fans out over every queue item, and each item
translated its paths by calling IRemotePathMappingService.TranslatePathAsync, which
queries the repository for that client's mappings on every call. The service, the
repository and the ListenArrDbContext behind them are all scoped, so a queue of N
items issued up to 2N concurrent queries against a context that permits one at a
time. DownloadClientQueuePoller then runs that whole thing inside its own
Task.WhenAll across every enabled client, so the fan-out is nested.

This is the origin of the trace in upstream Listenarrs#783. The exception surfaces at
DownloadClientQueuePoller.FetchAsync because that is where the await unwinds; the
class itself holds no repository and no context and never did.

Split the lookup from the translation. IRemotePathMappingService gains a
TranslatePath overload that takes mappings the caller has already resolved and does
no I/O, and TranslatePathAsync keeps its behaviour by fetching and delegating to it.
The gateway resolves once per client before each fan-out and passes the result down.
The single-item import path also resolves once, where it is one query either way.

Asserting on the EF exception would mean racing it, so the test counts overlap
directly: a stub records the highest number of lookups in flight. Ten items carrying
two translatable paths each report 10 concurrent lookups without the change and 1
with it, and the lookup count is pinned at 1 so a regression fails even if it
somehow avoids overlapping.
…ed mappings too

The batch lookup added alongside this only covered RemotePath and ContentPath. The
SourceFiles loop still called TranslatePathAsync, which queries the repository for the
client's mappings on every file. That loop is sequential within an item, but it runs
inside GetQueueAsync's Task.WhenAll over items, so the overlap the batch lookup was
meant to remove is still there whenever items carry source files.

qBittorrent's queue mapper populates SourceFiles from the torrent's file list
(QbittorrentResponseMapper.MapQueueItem), and Transmission's does the same, so this is
the normal case on a torrent client rather than an edge case.

The new test covers items carrying source files, which the existing one did not: it
reports 10 concurrent lookups without this change and 1 with it.

DownloadClientGatewayTests' mapping mock only stubbed TranslatePathAsync, so it went
stale when the source-file loop moved onto TranslatePath and returned null for every
path. Stub both.
@m4bard

m4bard commented Aug 24, 2026

Copy link
Copy Markdown
Author

Pushed a follow-up commit. The batch lookup in the first commit does not cover the whole fan-out, and I would rather say that here than have it reviewed as though it did.

TranslateQueueItemPathsAsync translates three things: RemotePath, ContentPath, and every entry in SourceFiles. The first commit moved the first two onto the pre-resolved mappings and left the SourceFiles loop calling TranslatePathAsync, which still queries per file. That loop is sequential within one item, but it runs inside GetQueueAsync's Task.WhenAll over items, so the overlap I was trying to remove is still there whenever items carry source files.

On a torrent client they usually do. QbittorrentResponseMapper.MapQueueItem sets SourceFiles from the torrent's file list, and Transmission's resolver does the same, so most queue items from those clients arrive with the list already populated.

My own test missed it because the items it builds carry no source files. The new one builds ten items with three files each. It reports 10 concurrent lookups without the follow-up commit and 1 with it, and I ran it with the production line reverted to confirm it fails rather than assuming it would.

One test change came with it. DownloadClientGatewayTests's mapping mock only stubbed TranslatePathAsync. Once the source-file loop moved onto TranslatePath the mock returned null for every path, the two case-only filenames deduped to one, and the case-sensitive row of GetQueueItemAsync_DedupesCaseOnlySourceFilesUsingResolvedSemantics failed. Stubbing both restores it. The same staleness was already there after the first commit, for ContentPath. It just did not happen to move that assertion.

Having missed one inside a method I had already read, I swept the rest rather than leave it at that. #870 says I had not checked the other gateway methods, and I can close that now.

Three places resolve mappings, and all three are right: GetQueueAsync at :99 and FetchDownloadsAsync at :174 each resolve once and then fan out, and GetQueueItemAsync at :145 resolves once for a single item. Both fan-outs go through TranslateQueueItemPathsAsync, so the one line above is the whole of it rather than the first of several.

Outside the gateway there are only two callers of either method, both in RemotePathMappingsController, both serving one request with no fan-out.

Full suite: 3,031 passed, 0 failed, 125 skipped.

The line in the description saying a thirty-item queue goes from up to sixty queries to one held only for items with no source files. I have corrected it to say the lookups per batch, which is the claim the tests actually support.

krejko added a commit to nexalapp/Listenarr that referenced this pull request Aug 26, 2026
…lient's path mappings once per batch, not per item"

This reverts commit 55c18f8, reversing
changes made to d64de28.
krejko added a commit to nexalapp/Listenarr that referenced this pull request Aug 26, 2026
…client's path mappings once per batch, not per item"

This reverts commit f3ca823.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant