feat(workflow-operator): score the rows an empty cell leaves usable - #8070
feat(workflow-operator): score the rows an empty cell leaves usable#8070kz930 wants to merge 1 commit into
Conversation
The Machine Learning Scorer handed both configured columns straight to scikit-learn's metric functions, so one empty cell reached them as a None and ended the run with "Classification metrics can't handle a mix of unknown and binary targets". The message named neither the column nor a way out, and no configuration could exclude the row. The operator now drops the rows missing either value before scoring, the way the visualization operators drop missing values before plotting. One subset naming both columns rather than two separate drops: dropping on either alone would leave y_true and y_pred misaligned. Closes apache#8058 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
/request-review @aglinxinyuan @carloea2 |
Automated Reviewer SuggestionsBased on the
|
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## main #8070 +/- ##
============================================
- Coverage 93.43% 93.42% -0.02%
+ Complexity 4710 4707 -3
============================================
Files 1181 1181
Lines 47801 47810 +9
Branches 5326 5327 +1
============================================
+ Hits 44663 44666 +3
- Misses 1698 1700 +2
- Partials 1440 1444 +4
*This pull request uses carry forward flags. Click here to find out more. ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
| config | throughput | MB/s | latency | max Δ latest / 7d | |
|---|---|---|---|---|---|
| 🟢 | bs=10 sw=10 sl=64 | 407 | 0.249 | 24,642/30,836/30,836 us | 🟢 -8.0% / 🔴 +120.5% |
| 🔴 | bs=100 sw=10 sl=64 | 895 | 0.546 | 109,836/133,462/133,462 us | 🔴 +6.4% / 🔴 +40.3% |
| ⚪ | bs=1000 sw=10 sl=64 | 1,053 | 0.643 | 946,312/1,006,072/1,006,072 us | ⚪ within ±5% / 🔴 -11.3% |
Baseline details
Latest main bdc6d2a from same runner
| config | metric | PR | latest main | 7d avg | Δ latest | Δ 7d |
|---|---|---|---|---|---|---|
| bs=10 sw=10 sl=64 | throughput | 407 tuples/sec | 414 tuples/sec | 893.99 tuples/sec | -1.7% | -54.5% |
| bs=10 sw=10 sl=64 | MB/s | 0.249 MB/s | 0.252 MB/s | 0.546 MB/s | -1.2% | -54.4% |
| bs=10 sw=10 sl=64 | p50 | 24,642 us | 23,599 us | 11,176 us | +4.4% | +120.5% |
| bs=10 sw=10 sl=64 | p95 | 30,836 us | 33,521 us | 14,009 us | -8.0% | +120.1% |
| bs=10 sw=10 sl=64 | p99 | 30,836 us | 33,521 us | 16,898 us | -8.0% | +82.5% |
| bs=100 sw=10 sl=64 | throughput | 895 tuples/sec | 925 tuples/sec | 1,157 tuples/sec | -3.2% | -22.7% |
| bs=100 sw=10 sl=64 | MB/s | 0.546 MB/s | 0.565 MB/s | 0.706 MB/s | -3.4% | -22.7% |
| bs=100 sw=10 sl=64 | p50 | 109,836 us | 103,218 us | 88,763 us | +6.4% | +23.7% |
| bs=100 sw=10 sl=64 | p95 | 133,462 us | 141,293 us | 95,096 us | -5.5% | +40.3% |
| bs=100 sw=10 sl=64 | p99 | 133,462 us | 141,293 us | 102,854 us | -5.5% | +29.8% |
| bs=1000 sw=10 sl=64 | throughput | 1,053 tuples/sec | 1,040 tuples/sec | 1,187 tuples/sec | +1.3% | -11.3% |
| bs=1000 sw=10 sl=64 | MB/s | 0.643 MB/s | 0.635 MB/s | 0.724 MB/s | +1.3% | -11.2% |
| bs=1000 sw=10 sl=64 | p50 | 946,312 us | 957,307 us | 874,968 us | -1.1% | +8.2% |
| bs=1000 sw=10 sl=64 | p95 | 1,006,072 us | 1,026,390 us | 917,815 us | -2.0% | +9.6% |
| bs=1000 sw=10 sl=64 | p99 | 1,006,072 us | 1,026,390 us | 949,868 us | -2.0% | +5.9% |
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,490.84,200,128000,407,0.249,24641.77,30836.10,30836.10
1,100,10,64,20,2235.25,2000,1280000,895,0.546,109835.96,133461.96,133461.96
2,1000,10,64,20,18994.71,20000,12800000,1053,0.643,946311.67,1006072.10,1006072.10
What changes were proposed in this PR?
The Machine Learning Scorer handed both configured columns straight to scikit-learn's metric functions, so one empty cell reached them as a None and ended the run with "Classification metrics can't handle a mix of unknown and binary targets". The message named neither the column nor a way out, and no configuration could exclude the row.
The operator now drops the rows missing either value before scoring, the way the visualization operators drop missing values before plotting. The subset names both columns rather than dropping on each separately: dropping on either alone would leave y_true and y_pred misaligned.
Any related issues, documentation, discussions?
Closes #8058
How was this PR tested?
A new case in MachineLearningScorerOpDescSpec pins the drop in the generated code. Run by hand on a table with a blank in the actual-value column and a blank in the predicted-value column: before, the run ended with the ValueError above; after, it reports Accuracy 0.6667 over the three scorable rows.
Was this PR authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)