Bug triage results: 2026-07-06

[andygrove]

Triage pass over the open requires-triage queue, per the project Bug Triage Guide.

• Date: 2026-07-06
• Total issues processed: 26 (25 triaged, 1 skipped, 0 failed)
• Type counts: 7 bugs, 18 enhancements
• Priority counts applied: priority:critical 5, priority:high 2
• Guide: docs/source/contributor-guide/bug_triage.md

Labels have already been applied and requires-triage removed from each issue listed under "Bugs" and "Enhancements". A reviewer should spot-check the calls and close this issue when satisfied. To correct a label, edit the affected issue directly.

Bugs

priority:critical

• array_filter: array_compact fast path misfires when IsNotNull operand is not the lambda variable (#4830)
  • Area labels: area:expressions
  • Rationale: CometArrayFilter.convert silently rewrites filter(arr, x -> captured_col IS NOT NULL) as array_compact(arr), producing wrong results whenever arr has null elements or captured_col is null — decision-tree step 1 (silent wrong results).
• Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results (#4813)
  • Area labels: area:aggregation
  • Rationale: incompatible partial buffers cross the Comet/Spark boundary in the 4-stage distinct rewrite; hard crash for percentile_approx but the reporter notes silent wrong results are likely for collect_set, collect_list, exact percentile, avg, decimal sum, and variance — escalated to critical under decision-tree step 1.
• bug: flatten drops nulls and returns wrong results when a sub-array is null (#4788)
  • Area labels: area:expressions
  • Rationale: reporter demonstrates Comet returns a non-null misaligned array where Spark returns null; silent data corruption over a plain Parquet scan — decision-tree step 1.
• bug: null IN () returns false instead of null under legacy null-in-empty behavior (#4786)
  • Area labels: area:expressions
  • Rationale: Comet ignores spark.sql.legacy.nullInEmptyListBehavior=true and silently returns false where Spark returns null — decision-tree step 1.
• [EPIC] Consistent handling of invalid UTF-8 in native StringType (ingress policy) (#4764)
  • Area labels: none (existing correctness, area:shuffle, area:scan, area:ffi preserved by the reporter)
  • Rationale: reporter documents a latent-UB / silent-wrong-result hazard (arrow-rs StringArray::value on FFI-imported non-UTF-8 bytes is unchecked) plus the scan gap that Spark accepts and Comet rejects — decision-tree step 1 for the UB hazard; kept as an EPIC bug.

priority:high

• bug: array/map kernels crash on non-null child field via CometLocalTableScanExec (#4789)
  • Area labels: area:expressions, area:scan
  • Rationale: native assertion failure / InvalidArgumentError panic on slice, map_entries, and array_insert when the local scan carries non-nullable child fields; visible crash rather than wrong results — decision-tree step 2.
• bug: scalar subquery in RepartitionByExpression crashes with "Subquery N not found" (#4787)
  • Area labels: area:expressions
  • Rationale: CometNativeException: Subquery 75 not found for plan 12 when a scalar subquery appears in DISTRIBUTE BY; reproduces over plain Parquet — decision-tree step 2 (native crash).

Enhancements

• Support WindowGroupLimitExec (window-based limit pushdown) (#4837)
  • Area labels: area:expressions
  • Rationale: request for a new native operator so plans containing WindowGroupLimitExec no longer fall back to Spark.
• Support GROUPS window frame units (#4836)
  • Area labels: area:expressions
  • Rationale: adds a currently unsupported window frame type; the existing behavior is a documented, clean fallback to Spark, not a defect.
• Support FIRST_VALUE/LAST_VALUE on a RANGE window frame with a literal offset (#4835)
  • Area labels: area:expressions
  • Rationale: extends native RANGE-frame coverage; today Comet correctly falls back to Spark, so this is a coverage gap.
• Support RANGE window frames with an explicit offset on DATE or DECIMAL ORDER BY (#4834)
  • Area labels: area:expressions
  • Rationale: documented current limitation with clean Spark fallback; request to broaden native RANGE-frame ORDER BY type support.
• Partial project fallback: keep a projection/filter native by evaluating an unsupported subexpression in the JVM (#4825)
  • Area labels: area:expressions
  • Rationale: proposes a new splitter hook so unsupported subexpressions no longer collapse the whole native island — a new feature behind an opt-in config.
• [Enhancement] Enable mixed partial/final execution for approx_count_distinct (HyperLogLogPlusPlus) (#4820)
  • Area labels: area:aggregation
  • Rationale: request to broaden native coverage by flipping supportsMixedPartialFinal for approx_count_distinct now that the buffer format matches Spark; not a defect.
• Support grouping() and grouping_id() indicator functions (#4814)
  • Area labels: area:expressions
  • Rationale: queries already run natively; the ask is to add test coverage and mark the functions as supported in the docs — a coverage/documentation enhancement.
• feat: IO-level dynamic filter pushdown into the native Delta scan (delta-kernel-rs predicate reuse) (#4809)
  • Area labels: area:scan
  • Rationale: performance follow-up to feat: Runtime dynamic filter pushdown for native hash joins (build side -> probe side) #4807 that adds IO-level pruning to the native Delta scan; new functionality.
• feat: IO-level dynamic filter pushdown into IcebergScanExec (iceberg-rust predicate conversion) (#4808)
  • Area labels: area:scan
  • Rationale: performance follow-up to feat: Runtime dynamic filter pushdown for native hash joins (build side -> probe side) #4807 that adds IO-level pruning inside iceberg-rust; new functionality.
• feat: Runtime dynamic filter pushdown for native hash joins (build side -> probe side) (#4807)
  • Area labels: none
  • Rationale: request to wire up DataFusion's dynamic filter machinery through Comet's native hash joins; new optimization behind an opt-in config.
• Add optional fully-native deployment mode (Ballista-backed) (#4796)
  • Area labels: none
  • Rationale: proposes a new deployment mode built on Ballista; discussion-phase feature request.
• Support native batch session_window grouping (MergingSessionsExec / UpdatingSessionsExec) (#4785)
  • Area labels: area:aggregation
  • Rationale: requests new native operators for batch session-window merging; today Comet cleanly falls back to Spark for this.
• Performance optimizations for native in-memory cache (follow-on to feat: Add Native Support for In-Memory Cache #4591) (#4781)
  • Area labels: area:scan
  • Rationale: follow-on performance work tracked from PR feat: Add Native Support for In-Memory Cache #4591 review; no correctness defect.
• feat: Refactor shuffle partition writing behind a writer interface (#4780)
  • Area labels: area:shuffle
  • Rationale: refactor to introduce a PartitionWriter abstraction so alternative shuffle backends can be added later; behavior-preserving.
• Move CometCollationSuite into the spark-4.1+ test shim to avoid per-version duplication (#4770)
  • Area labels: area:ci
  • Rationale: test/build-system cleanup to add ${shims.minorPlusVerSrc} to add-test-source and move the suite; no behavior change.
• Report unsupported metadata column names in fallback reasons (#4757)
  • Area labels: area:scan
  • Rationale: improves fallback message to include the metadata column name; user-experience enhancement.
• Math expressions: revisit interval / input-type gating and add regression tests once interval support lands (#4756)
  • Area labels: area:expressions
  • Rationale: reporter explicitly states "no live defect today"; deferred work to add regression tests and revisit gating once native interval support lands.
• Evaluate moving from chrono to Jiff (#4754)
  • Area labels: area:expressions
  • Rationale: investigation into swapping the chrono dependency for Jiff since chrono-tz is being deprecated; enhancement / exploration.

Escalations to consider

• Distinct-aggregate rewrite can split an incompatible-buffer aggregate across Comet and Spark, causing crash or wrong results (#4813)
  • Reporter observed a hard crash for percentile_approx but explicitly warns that other affected aggregates (collect_set, collect_list, exact percentile, decimal sum, avg, variance) risk silent wrong results — matches the guide's "priority:high crash also produces wrong results" escalation trigger; escalated to priority:critical.

Skipped — needs more info

• Bug triage results: 2026-06-29 (#4751)
  • Prior triage summary issue (auto-labeled requires-triage); meta, awaiting human review and closure, not a bug.