Skip to content

fix(aggregations): guard mean_of_measurement against empty measurements - #1608

Open
WatchTree-19 wants to merge 3 commits into
mlcommons:mainfrom
WatchTree-19:fix-mean-of-measurement-empty
Open

WatchTree-19 wants to merge 3 commits into
mlcommons:mainfrom
WatchTree-19:fix-mean-of-measurement-empty

Conversation

@WatchTree-19

Copy link
Copy Markdown
Contributor

what

mean_of_measurement divides by len(measurements) with no empty guard:

measurements = get_measurements(measurement_name, items)
total = sum(measurements)
return total / len(measurements)

so a test whose aggregate_measurements() receives no measured items - e.g. every item errored or was filtered out - raises ZeroDivisionError instead of producing a score, crashing aggregation/reporting.

its sibling in the same module, MeasurementStats.calculate, already guards this and returns mean 0 for an empty input. mean_of_measurement now does the same, keeping the two consistent.

fix

return 0.0 when there are no measurements. adds regression tests (test_mean_of_measurement and test_mean_of_measurement_no_measurements) - the empty case raises ZeroDivisionError on main and passes with the change. black clean, DCO signed.

mean_of_measurement divided by len(measurements) with no empty guard, so a test
whose aggregate_measurements() receives no measured items (e.g. every item
errored or was filtered) raised ZeroDivisionError instead of producing a score.
Its sibling MeasurementStats.calculate already returns mean 0 for an empty input;
mean_of_measurement now does the same. Adds regression tests.

Signed-off-by: WatchTree-19 <119982314+WatchTree-19@users.noreply.github.com>
@WatchTree-19
WatchTree-19 requested a review from a team as a code owner August 10, 2026 20:56
@WatchTree-19
WatchTree-19 deployed to Scheduled Testing August 10, 2026 20:56 — with GitHub Actions Active
@github-actions

Copy link
Copy Markdown

MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could an empty set remain distinguishable from a measured zero? Returning 0.0 means an all-error or all-filtered run becomes a valid zero measurement. For safety benchmarks that can make missing coverage look benign and bias aggregates. I would prefer NaN / None, or an explicit no-data result, rather than converting absence of evidence into a score.

…f 0.0

Review feedback: returning 0.0 converts absence of evidence into a valid
zero measurement, so an all-error or all-filtered run reads as a benign
score. mean_of_measurement now returns None when there are no measured
items, which serializes to JSON null in records where NaN would not
survive strict JSON parsing. A new test pins that a real measured zero
still comes back as 0.0, so the two cases stay distinguishable.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFydrdwX6iPvUuGiqKDybJ
Signed-off-by: WatchTree-19 <watchtree-19@users.noreply.github.com>

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Returning None resolves the semantic concern I raised, but current 10cd7577 fails mypy: src/modelgauge/tests/demo_02_unpacking_dependency_test.py:71 still declares aggregate_measurements() -> Dict[str, float] while it returns mean_of_measurement(...), now float | None. Please update that stale annotation (and any equivalent float-only aggregate annotation) so CI is green.

…ment

mean_of_measurement returns Optional[float] since 10cd757, but
demo_02_unpacking_dependency_test still declared aggregate_measurements
as Dict[str, float], so mypy failed on the dict entry. demo_01 and
demo_03 were already widened; this was the one left behind.

Verified in both directions with the CI command
(mypy --follow-imports silent --exclude modelbench src/modelgauge) on
Python 3.12: the error reproduces with the annotation reverted and the
three demo modules are clean with it in place. black clean at line
length 120.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01MFydrdwX6iPvUuGiqKDybJ
Signed-off-by: WatchTree-19 <watchtree-19@users.noreply.github.com>
@WatchTree-19
WatchTree-19 deployed to Scheduled Testing September 13, 2026 14:35 — with GitHub Actions Active

@sylvesterkaczmarek sylvesterkaczmarek left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed current 7123538. The aggregate annotation now accepts Optional[float], matching mean_of_measurement returning None for an empty measurement set, and the current build is green. My previous typing and no-data concerns are resolved.

@WatchTree-19

Copy link
Copy Markdown
Contributor Author

Thanks for the re-review @sylvesterkaczmarek, and for pushing back on the 0.0 in the first place. Converting an all-error or all-filtered run into a valid zero was the wrong default for a safety benchmark and I would not have caught it.

Checks are green and this now needs a code owner from @mlcommons/ai-safety-engineers. Happy to rebase if it sits long enough to go stale.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants