fix(aggregations): guard mean_of_measurement against empty measurements - #1608
WatchTree-19 wants to merge 3 commits into
Conversation
mean_of_measurement divided by len(measurements) with no empty guard, so a test whose aggregate_measurements() receives no measured items (e.g. every item errored or was filtered) raised ZeroDivisionError instead of producing a score. Its sibling MeasurementStats.calculate already returns mean 0 for an empty input; mean_of_measurement now does the same. Adds regression tests. Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Could an empty set remain distinguishable from a measured zero? Returning 0.0 means an all-error or all-filtered run becomes a valid zero measurement. For safety benchmarks that can make missing coverage look benign and bias aggregates. I would prefer NaN / None, or an explicit no-data result, rather than converting absence of evidence into a score.
…f 0.0 Review feedback: returning 0.0 converts absence of evidence into a valid zero measurement, so an all-error or all-filtered run reads as a benign score. mean_of_measurement now returns None when there are no measured items, which serializes to JSON null in records where NaN would not survive strict JSON parsing. A new test pins that a real measured zero still comes back as 0.0, so the two cases stay distinguishable. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFydrdwX6iPvUuGiqKDybJ Signed-off-by: WatchTree-19 <watchtree-19@users.noreply.github.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Returning None resolves the semantic concern I raised, but current 10cd7577 fails mypy: src/modelgauge/tests/demo_02_unpacking_dependency_test.py:71 still declares aggregate_measurements() -> Dict[str, float] while it returns mean_of_measurement(...), now float | None. Please update that stale annotation (and any equivalent float-only aggregate annotation) so CI is green.
…ment mean_of_measurement returns Optional[float] since 10cd757, but demo_02_unpacking_dependency_test still declared aggregate_measurements as Dict[str, float], so mypy failed on the dict entry. demo_01 and demo_03 were already widened; this was the one left behind. Verified in both directions with the CI command (mypy --follow-imports silent --exclude modelbench src/modelgauge) on Python 3.12: the error reproduces with the annotation reverted and the three demo modules are clean with it in place. black clean at line length 120. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01MFydrdwX6iPvUuGiqKDybJ Signed-off-by: WatchTree-19 <watchtree-19@users.noreply.github.com>
sylvesterkaczmarek
left a comment
There was a problem hiding this comment.
Re-reviewed current 7123538. The aggregate annotation now accepts Optional[float], matching mean_of_measurement returning None for an empty measurement set, and the current build is green. My previous typing and no-data concerns are resolved.
|
Thanks for the re-review @sylvesterkaczmarek, and for pushing back on the 0.0 in the first place. Converting an all-error or all-filtered run into a valid zero was the wrong default for a safety benchmark and I would not have caught it. Checks are green and this now needs a code owner from @mlcommons/ai-safety-engineers. Happy to rebase if it sits long enough to go stale. |
what
mean_of_measurementdivides bylen(measurements)with no empty guard:so a test whose
aggregate_measurements()receives no measured items - e.g. every item errored or was filtered out - raisesZeroDivisionErrorinstead of producing a score, crashing aggregation/reporting.its sibling in the same module,
MeasurementStats.calculate, already guards this and returns mean0for an empty input.mean_of_measurementnow does the same, keeping the two consistent.fix
return
0.0when there are no measurements. adds regression tests (test_mean_of_measurementandtest_mean_of_measurement_no_measurements) - the empty case raisesZeroDivisionErroronmainand passes with the change. black clean, DCO signed.