fix(consistency-checker): guard empty denominators

[Abelo9996]

What

Guard two empty-denominator divisions in the consistency checker that raise a ZeroDivisionError on a failed run. The exception propagates out of ConsistencyChecker.run(), and the CLI catches it per journal, so the tool does not crash, but that journal's entire consistency check is abandoned and reported as an opaque Error running consistency check float division by zero instead of the actual findings.

Why

The consistency checker exists to flag broken runs, but two checks crash on exactly those runs:

• MinValidAnnotatorItems.check divides by num_translated_annotations, which is 0 when an annotator produced no translated annotations (for example, its API was down for the whole run). run_checks_for_row calls check.check() with no guard, and the runner iterates every annotator the test declares, so the exception propagates out of ConsistencyChecker.run(); the CLI's run_consistency_check catches it per journal, abandoning that journal's whole check and reporting a generic error instead of the specific finding.
• HazardScoreIsFracSafe._get_frac_safe divides by num_scored, which is 0 when no items were measured for a hazard; this runs in __init__, so constructing the check crashes.

Details and a standalone repro are in #1629.

Change

• MinValidAnnotatorItems.check returns False (with a clear failure message) when there are no translated annotations, flagging the annotator instead of crashing.
• HazardScoreIsFracSafe._get_frac_safe returns nan when nothing was measured, so the equality check fails rather than crashing.
• Regression tests for both.

This mirrors the empty-denominator guard already in flight in #1608.

Fixes #1629

[github-actions]

MLCommons CLA bot:
Thank you very much for your submission; we really appreciate it. Before we can accept your contribution,
we ask that you sign the MLCommons CLA (Apache 2). Please submit your GitHub ID to our onboarding form to initiate
authorization. If you are from a MLCommons member organization, we will request that you be added to the CLA.
If you are not from a member organization, we will email you a CLA to sign. For any questions, please contact
support@mlcommons.org.
0 out of 1 committers have signed the MLCommons CLA.
❌ @Abelo9996
_{You can retrigger this bot by commenting recheck in this Pull Request}

[wpietri]

Thanks for taking an interest, @Abelo9996. Is this a problem you ran into while using ModelBench? If so, I'd like to know more about the circumstances. Thanks!

[Abelo9996]

Thanks @wpietri. I didn't hit this in a full run. I found it reading consistency_checker.py and reproduced it with the script in the issue.

It would bite a real run when an annotator produces zero translated annotations for a sut/test, for example if that annotator's API is down the whole run so nothing writes a translated annotation entry. Then num_translated_annotations is 0 and MinValidAnnotatorItems.check divides by zero. Same shape in HazardScoreIsFracSafe._get_frac_safe when a hazard has no measured items.

What made me flag it: it crashes on exactly the broken run the checker is meant to catch. There's no per-check guard, so the error propagates to the CLI and prints as a generic Error running consistency check float division by zero, and that journal can't be validated.

The PR guards both empty denominators as a failed check instead of a crash: return False with a message naming the annotator, and nan for the frac-safe case. Happy to adjust the messages or semantics if you'd prefer.

[wpietri]

Thanks @wpietri. I didn't hit this in a full run. I found it reading consistency_checker.py and reproduced it with the script in the issue.

This reply seems generated to me. If you as a person didn't encounter this through use, then what's your actual use of modelbench that motivates this.