Handle empty anomaly lists in the point metrics - #672
Open
DMZ22 wants to merge 1 commit into
Open
Conversation
_point_partition derived the partition range with min()/max() over the union of the expected and observed anomalies, before the start/end overrides were applied. With no anomalies on either side that union is empty, so every point metric raised "ValueError: min() arg is an empty sequence" - including when a range was supplied through data/start/end. Take start/end into account first and only fall back to min()/max() when they are not given, so a known range is enough to build the partition. When there are no anomalies and no range to fall back on, return an empty confusion matrix instead: there are no true or false positives and no false negatives, but the number of true negatives is unknown, which is the same (None, fp, fn, tp) shape the contextual metrics already use. Precision, recall and f1 then return nan through the existing zero-division handling, and accuracy raises the existing error for a missing true-negative count unless a range makes it computable. Closes sintel-dev#527.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #527.
Problem
_point_partitionderives the partition range withmin()/max()over the union of the expected and observed anomalies, before thestart/endoverrides are applied:When neither side has any anomalies that union is empty, so every point metric raises
ValueError: min() arg is an empty sequence— and because themin()runs first, supplying a range throughdata/start/enddoesn't help either:Fix
Take
start/endinto account first and only fall back tomin()/max()when they aren't given, so a known range is enough to build the partition. If there are no anomalies and no range to fall back on, return an empty confusion matrix as you specified on the issue — there are no true or false positives and no false negatives, but the number of true negatives is unknown. That is the same(None, fp, fn, tp)shapecontextual.pyalready returns when the true negatives can't be determined, so it keeps the 4-tuple contract the metrics unpack.Resulting behaviour
point_confusion_matrix([], [])ValueError(None, 0, 0, 0)point_confusion_matrix([], [], start=0, end=9)ValueError[10, 0, 0, 0](all true negatives)point_precision/recall/f1_score([], [])ValueErrornan, via the existing zero-division handlingpoint_accuracy([], [])ValueError: min() arg…point_accuracy([], [], start=0, end=9)ValueError1.0Everything with at least one anomaly on either side is untouched —
point_confusion_matrix([1,2,3], [2,3,4])still returns[0, 1, 1, 2], and[2, 1, 1, 2]withstart=0, end=5.Tests
Added
test_point_confusion_matrix_empty,test_point_confusion_matrix_empty_with_rangeandtest_point_scores_empty; all three fail onmasterand pass here.tests/unit/evaluationpasses (36 tests), andflake8/isortare clean on the changed files.