DuckDB 1.5.5 (substrait extension 4ea63dd) and DataFusion 54.0.0 put a projection mask on every table read in the plans I got from them (42 of 42 each), so this reaches their plans whenever the mask drops or reorders columns. For DuckDB's plan of SELECT c2, c0 FROM t, whose read carries the mask [2, 0], infer_rel_schema returns three types for two root names.
infer_rel_schema's read branch takes rel.read.base_schema.struct and never reads rel.read.projection (type_inference.py:698). The word projection does not appear anywhere in that file, on main either, so a masked read infers the columns the mask left out.
Read Operator, Direct Output Order: "Defaults to the schema of the data read after the optional projection (masked complex expression) is applied."
Reproduced with 0.31.0:
import substrait.algebra_pb2 as stalg
import substrait.type_pb2 as stt
from substrait.type_inference import infer_rel_schema
i64 = stt.Type(i64=stt.Type.I64(nullability=stt.Type.NULLABILITY_REQUIRED))
rel = stalg.Rel(read=stalg.ReadRel(
base_schema=stt.NamedStruct(names=["a", "b", "c"],
struct=stt.Type.Struct(types=[i64, i64, i64],
nullability=stt.Type.NULLABILITY_REQUIRED)),
projection=stalg.Expression.MaskExpression(
select=stalg.Expression.MaskExpression.StructSelect(struct_items=[
stalg.Expression.MaskExpression.StructItem(field=2)]))))
print(len(infer_rel_schema(rel).types))
# actual: 3
# expected: 1, the one column the mask selectsForeign plans only, as far as I can see: nothing in builders/ or dataframe/ sets ReadRel.projection, so this is reachable the same way #217 was rather than through the library's own API.
The narrowing would apply to base_schema.names as well as to struct.types, and it belongs before common/emit, whose indices are defined over the post-projection order. One scope question worth settling first: the repro uses a flat StructSelect, and masks also nest through struct_items.child, so it is worth deciding whether the fix covers the nested form now or later. Happy to send the PR.
DuckDB 1.5.5 (substrait extension 4ea63dd) and DataFusion 54.0.0 put a projection mask on every table read in the plans I got from them (42 of 42 each), so this reaches their plans whenever the mask drops or reorders columns. For DuckDB's plan of
SELECT c2, c0 FROM t, whose read carries the mask[2, 0],infer_rel_schemareturns three types for two root names.infer_rel_schema's read branch takesrel.read.base_schema.structand never readsrel.read.projection(type_inference.py:698). The wordprojectiondoes not appear anywhere in that file, onmaineither, so a masked read infers the columns the mask left out.Read Operator, Direct Output Order: "Defaults to the schema of the data read after the optional projection (masked complex expression) is applied."
Reproduced with 0.31.0:
Foreign plans only, as far as I can see: nothing in
builders/ordataframe/setsReadRel.projection, so this is reachable the same way #217 was rather than through the library's own API.The narrowing would apply to
base_schema.namesas well as tostruct.types, and it belongs beforecommon/emit, whose indices are defined over the post-projection order. One scope question worth settling first: the repro uses a flatStructSelect, and masks also nest throughstruct_items.child, so it is worth deciding whether the fix covers the nested form now or later. Happy to send the PR.