fix!: apply ReadRel projection during schema inference

[alexandrefimov]

infer_rel_schema currently returns the full base_schema for a masked read. A mask selecting one field can therefore report all input fields, and emit mappings are evaluated against the wrong columns.

Apply the read projection before emit, following the Read Operator in spec v0.99.0. Recursively project struct fields, list elements, and map values while preserving their nullability and container types. Keep explicit RelRoot.names and the original input schema unchanged. For projected reads, leave correlations inside filter and best_effort_filter offset-based: Read Filtering explicitly places those predicates before projection.

Mask unwrapping is a compatibility choice. The unwrapping section describes singleton unwrapping by default but leaves its serialization as TBD. This change follows Java's type projector: it preserves structs and containers at every level regardless of maintain_singular_struct. For example, selecting only the string from a nested struct<i64, string> keeps a nested struct<string>, even with the flag false. The validator instead requires the flag for a single-field read projection.

Struct fields also follow Java's mask order, including reordered and repeated selections. The same spec page still lists column reordering as an open question, so preserving mask order and duplicates remains a compatibility choice.

Closes #264

BREAKING CHANGE: ReadRel projection masks now change inferred output schemas and the indices seen by emit and parent relations. Invalid mask field indices and selector/type mismatches now raise errors instead of being ignored.

Summary by CodeRabbit

• Bug Fixes

  • Read projections now preserve field ordering and nested structure metadata.
  • Nested structs, lists, and maps are projected correctly while retaining their surrounding types.
  • Invalid field indexes and incompatible nested selections now produce clear errors.
  • Read filters continue to reference the correct underlying schema when projections are applied.
  • Downstream expressions now use projected field indexes consistently.

• Tests

  • Added comprehensive coverage for nested projections, filters, invalid selections, and plan immutability.

[tokoko]

this sounds like a bug in java impl, right? why should we ignore maintain_singular_struct?

[alexandrefimov]

Right, MaskExpressionTypeProjector never consults it. The flag switches unwrapping on top of the field selection this PR applies: with true we already do what it asks, and false would only matter below the top level, since infer_rel_schema has to return the read's row as a Type.Struct. Reworded that comment in 95a2e9a to rest on the schema rather than on Java, and to say that nested levels keep the struct too.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: e07de236-07ee-40cc-8624-273b9a9ee9f6

📥 Commits

Reviewing files that changed from the base of the PR and between 2f138c1 and 95a2e9a.

📒 Files selected for processing (4)

• src/substrait/type_inference.py
• src/substrait/utils/__init__.py
• tests/test_read_projection.py
• tests/test_utils.py

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

Read schema inference now applies projection masks before emit mappings and recursively projects nested types. Projected-read filter references remain offset-based during relation conversion. Tests cover ordering, validation, nested collections, downstream indices, correlations, and plan immutability.

Changes

Read projection handling

Layer / File(s)	Summary
Projected read schema inference src/substrait/type_inference.py, tests/test_read_projection.py	Read projections select fields in mask order, preserve struct and collection metadata, recurse through nested types, validate selections, and run before emit mappings. Tests cover projected indices and serialized-plan stability.
Projected-read reference conversion src/substrait/utils/__init__.py, tests/test_utils.py	Projected-read filter and best_effort_filter references remain steps_out; unprojected reads still bind them to the read anchor. Correlated subqueries above projected reads use the host read anchor.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Bug fix · Severity of issue fixed: Medium

Suggested reviewers: tokoko

🚥 Pre-merge checks | ✅ 4 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 17.86% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 28 functions across 4 files.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (4 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title clearly and concisely describes the main change: applying ReadRel projection during schema inference.
Description check	✅ Passed	The description explains the problem, implementation scope, compatibility choices, affected behavior, linked issue, and breaking change. It provides the required rationale and is mostly complete.
Linked Issues check	✅ Passed	Issue #264 requires infer_rel_schema to infer the schema after ReadRel.projection, including reordered and dropped fields and nested struct_items.child selections. The PR applies projection mask…
Out of Scope Changes check	✅ Passed	The changes remain within issue #264. The to_id_based_outer_references update preserves offset-based read-filter correlations when a projection changes the read output schema. Its tests cover projec…

• Fix all pre-merge checks with AI

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}