fix(parsing): pin machine-format number parses to the invariant culture - #893
Open
m4bard wants to merge 1 commit into
Open
fix(parsing): pin machine-format number parses to the invariant culture#893m4bard wants to merge 1 commit into
m4bard wants to merge 1 commit into
Conversation
ffprobe, MyAnonamouse, SABnzbd and Torznab all emit numbers with '.' as the decimal separator whatever locale the emitting process runs under. Six parses read those strings with the ambient culture, so on a server whose culture treats '.' as the group separator the value was read as a different number, and on a server whose decimal separator is ',' it did not parse at all. Measured under de-DE and fr-FR: FfprobeMetadataMapper.cs:50 "43200.250" -> 43200250 s / not parsed MyAnonamouseSizeParser.cs:36 "1.5 GB" -> 15 GB / 0 MyAnonamouseSizeParser.cs:49 "1.5 GB" -> 15 GB / 0 SabnzbdResponseMapper.cs:201 "1.5 M" -> 15 MB/s / 0 SabnzbdResponseMapper.cs:320 "1.5" -> 15 MB / 0 TorznabNewznabValueParser.cs:40 "1.5 GB" -> 15 GB / 0 Each site now passes NumberStyles and CultureInfo.InvariantCulture, matching the parse that was already correct in the same file where there is one (SabnzbdResponseMapper.cs:224 uses NumberStyles.Any, MyAnonamouseSizeParser.cs:73 uses NumberStyles.Float) and NzbgetHistoryReader.cs:157 otherwise. Fixes Listenarrs#796.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #796.
ffprobe, MyAnonamouse, SABnzbd and Torznab all emit numbers in machine format:
.is the decimal separator whatever locale the emitting process runs under. Six parses read those strings with the ambient culture, so what the server records depends on the server's locale.Measured under de-DE and fr-FR, on the values from the issue:
FfprobeMetadataMapper.cs:5043200.250secondsMyAnonamouseSizeParser.cs:361.5 GBMyAnonamouseSizeParser.cs:491.5 GBSabnzbdResponseMapper.cs:2011.5 MSabnzbdResponseMapper.cs:3201.5mbTorznabNewznabValueParser.cs:401.5 GBA default container runs under the invariant culture and is unaffected. It takes a real culture reaching the process, which happens when
LANGorLC_ALLis set, and on a desktop install that inherits the operating system's locale. I have not tested the Windows or macOS builds.The duration one gets written down: it becomes
DurationSecondson the file record, so a twelve hour book is stored as 500 days.The shape
I asked on the issue which of three shapes you wanted. Rather than leave it sitting, I picked the smallest one, and it is easy to redo as either of the others if you would rather.
Each site now passes
NumberStylesandCultureInfo.InvariantCulture. Where the same file already contained a correct parse of the same kind, the fixed site copies it rather than introducing a second convention:SabnzbdResponseMapper.cs:224(ParseJsonDouble) already pinsNumberStyles.Any.GetDoubleat :320 is its near-duplicate and now matches it, as doesParseSpeedat :201.MyAnonamouseSizeParser.cs:73already pinsNumberStyles.Float. :36 and :49 now match it.FfprobeMetadataMapperandTorznabNewznabValueParserhave no correct parse in-file, so they followNzbgetHistoryReader.cs:157, which usesNumberStyles.Float.Two of the four files already held one pinned parse and one bare parse of the same kind of value, so this reads as drift rather than as a decision anyone made.
What I did not do:
A shared helper, Readarr's shape.
TryParseExtensions.ParseDoubledoessource.Replace(',', '.')before parsing. That is deliberate, and it has a cost:"1,234.5"becomes"1.234.5"and stops parsing, and a thousands-separated"1,234"becomes1.234. I would not want that on a size field. The helper would also have to live inlistenarr.domainto be visible to bothlistenarr.applicationandlistenarr.infrastructure, since that is the only project both of them reference, and the six sites do not agree onNumberStylesanyway. Say the word and I will write it.A typed ffprobe reader, Sonarr's shape. That is a dependency decision and it touches #791, so not here.
CA1305. It lights up the
DateTimeandintsites too, so it belongs in its own change.Not touched
Audiobook.csandDownloadImportService.Naming.csbelong to #763, andAudibleSeriesWorkflow.cs:342to #892. Both of those pin the same way. Leaving them alone keeps this from conflicting with them.The
DateTimeparses. ISO 8601 and RFC822 both parse correctly under every culture I tried, and the only failure I could produce needed an ambiguous numeric date, where the right answer depends on what the source emits.MyAnonamousePublishDateParser.cs:72,:77and:89are the same shape, readingageHours,ageMinutesandagewhen MyAnonamouse sends them as strings rather than numbers. I left them out because I have not seen MAM emit a decimal there, so unlike the six above I have no measured damage to show. They are three more one-line changes if you want them in.Tests
MachineFormatCultureParsingTests, 28 cases. Each of the six sites, plus the already-correctSabnzbdResponseMapper.cs:224as a reference, asserted under the invariant culture, en-US, de-DE and fr-FR.Control, with only the four production files reverted to canary and the tests left in place, 12 of the 28 fail:
With the fix in place all 28 pass. The invariant and en-US cases pass on both sides, so a stock container never saw any of this.
Full suite 3067 passed, 0 failed, 127 skipped, on canary
d25b3e11(1.3.3).One note on the line numbers. The issue cites v1.2.2, three releases back, but all six sites are unmoved: those four files are byte identical to
4555ad21. The two sites the issue set aside have moved.Audiobook.cs:108is now:101, andDownloadImportService.cs:371is nowDownloadImportService.Naming.cs:54since that file was split.