fix(downloads): resolve a client's path mappings once per batch, not per item - #871
fix(downloads): resolve a client's path mappings once per batch, not per item#871m4bard wants to merge 2 commits into
Conversation
…per item DownloadClientGateway.GetQueueAsync fans out over every queue item, and each item translated its paths by calling IRemotePathMappingService.TranslatePathAsync, which queries the repository for that client's mappings on every call. The service, the repository and the ListenArrDbContext behind them are all scoped, so a queue of N items issued up to 2N concurrent queries against a context that permits one at a time. DownloadClientQueuePoller then runs that whole thing inside its own Task.WhenAll across every enabled client, so the fan-out is nested. This is the origin of the trace in upstream Listenarrs#783. The exception surfaces at DownloadClientQueuePoller.FetchAsync because that is where the await unwinds; the class itself holds no repository and no context and never did. Split the lookup from the translation. IRemotePathMappingService gains a TranslatePath overload that takes mappings the caller has already resolved and does no I/O, and TranslatePathAsync keeps its behaviour by fetching and delegating to it. The gateway resolves once per client before each fan-out and passes the result down. The single-item import path also resolves once, where it is one query either way. Asserting on the EF exception would mean racing it, so the test counts overlap directly: a stub records the highest number of lookups in flight. Ten items carrying two translatable paths each report 10 concurrent lookups without the change and 1 with it, and the lookup count is pinned at 1 so a regression fails even if it somehow avoids overlapping.
…ed mappings too The batch lookup added alongside this only covered RemotePath and ContentPath. The SourceFiles loop still called TranslatePathAsync, which queries the repository for the client's mappings on every file. That loop is sequential within an item, but it runs inside GetQueueAsync's Task.WhenAll over items, so the overlap the batch lookup was meant to remove is still there whenever items carry source files. qBittorrent's queue mapper populates SourceFiles from the torrent's file list (QbittorrentResponseMapper.MapQueueItem), and Transmission's does the same, so this is the normal case on a torrent client rather than an edge case. The new test covers items carrying source files, which the existing one did not: it reports 10 concurrent lookups without this change and 1 with it. DownloadClientGatewayTests' mapping mock only stubbed TranslatePathAsync, so it went stale when the source-file loop moved onto TranslatePath and returned null for every path. Stub both.
|
Pushed a follow-up commit. The batch lookup in the first commit does not cover the whole fan-out, and I would rather say that here than have it reviewed as though it did.
On a torrent client they usually do. My own test missed it because the items it builds carry no source files. The new one builds ten items with three files each. It reports 10 concurrent lookups without the follow-up commit and 1 with it, and I ran it with the production line reverted to confirm it fails rather than assuming it would. One test change came with it. Having missed one inside a method I had already read, I swept the rest rather than leave it at that. #870 says I had not checked the other gateway methods, and I can close that now. Three places resolve mappings, and all three are right: Outside the gateway there are only two callers of either method, both in Full suite: 3,031 passed, 0 failed, 125 skipped. The line in the description saying a thirty-item queue goes from up to sixty queries to one held only for items with no source files. I have corrected it to say the lookups per batch, which is the claim the tests actually support. |
…client's path mappings once per batch, not per item" This reverts commit f3ca823.
Summary
DownloadClientGateway.GetQueueAsyncfans out over every queue item, and each item translated its paths by callingTranslatePathAsync, which queries the repository for that client's mappings on every call. The service, the repository and theListenArrDbContextbehind them are all scoped, so a queue of N items issued up to 2N concurrent queries against a context that permits one at a time.DownloadClientQueuePollerruns that inside its ownTask.WhenAllacross every client, so the fan-out is nested.Full write-up in #870.
Changes
Fixed
IRemotePathMappingServicegains aTranslatePathoverload taking mappings the caller has already resolved. It does no I/O.TranslatePathAsynckeeps its behaviour by fetching and delegating to it, so existing callers are unaffected.DownloadClientGatewayresolves a client's mappings once before each of its two item fan-outs and passes them down.Fewer queries as well as non-overlapping ones: a queue resolves the client's mappings once per batch rather than once per translated path. Corrected 2026-08-24: this line previously said a thirty-item queue goes from up to sixty queries to one. That held only for items carrying no source files, because the first commit left the
SourceFilesloop querying per file. The follow-up commit fixes that; see the comment below.On #783
That issue reports this exception class with a trace through
DownloadClientQueuePoller.FetchAsync. That class holds no repository and no context, and did not at4555ad21either, which is the commit it reports against. The trace surfaces there because that is where the await unwinds; the query is further down this chain.I have not reproduced #783's original scenario and this does not claim to close it.
Testing
DownloadClientGatewayPathMappingConcurrencyTestscounts the overlap directly rather than racing the EF exception, which would make for a flaky test and a weak claim. A stub records the highest number of lookups in flight, with a small delay so the overlap is observable rather than a scheduling accident.Ten items carrying two translatable paths each report 10 concurrent lookups without the change and 1 with it. The lookup count is also pinned at 1, so a change that restores per-item querying fails even if it somehow avoids overlapping.
Full suite: 3,031 passed, 0 failed, 125 skipped, against a 3,029 baseline on
03958c15.Note
This is the second instance of the same pattern, after #862 and its PR #863, which fixes a smaller one in search-result scoring. They are separate files and independently revertable, so they are separate PRs. I originally reported that scoring site as the only instance; that was wrong and is corrected on that issue.