fix: require pyarrow>=18.0.0 for native UUID type support#3606
Open
anxkhn wants to merge 1 commit into
Open
Conversation
PyIceberg's PyArrow UUID handling relies on APIs that only exist in Apache Arrow 18.0.0 and later, but the pyarrow, pandas, duckdb, and ray extras still pin pyarrow>=17.0.0. On write, PyArrowSchemaVisitor.visit_uuid returns pa.uuid(); on read, _ConvertToIceberg branches on isinstance(primitive, pa.UuidType). Both pyarrow.uuid() and pyarrow.UuidType were introduced in pyarrow 18.0.0 and do not exist on 17.x, so an install that resolves pyarrow==17.x raises AttributeError as soon as a UUID-typed column is written or an Arrow schema containing a UUID is converted back to Iceberg. Bump the floor to pyarrow>=18.0.0 for the affected extras so the declared dependency matches what the code actually requires, and keep uv.lock in sync.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Rationale for this change
PyIceberg's PyArrow UUID handling depends on the native UUID APIs
pyarrow.uuid()and
pyarrow.UuidType, but thepyarrow,pandas,duckdb, andrayextrasstill pin
pyarrow>=17.0.0.PyArrowSchemaVisitor.visit_uuidreturnspa.uuid()(
pyiceberg/io/pyarrow.py:806)._ConvertToIceberg.primitivebranches onisinstance(primitive, pa.UuidType)(pyiceberg/io/pyarrow.py:1497).Both
pyarrow.uuid()andpyarrow.UuidTypewere introduced in Apache Arrow18.0.0 (released 2024-10-28) and do not exist in 17.x. So an environment that
resolves
pyarrow==17.xunder the current floor raisesAttributeError: module 'pyarrow' has no attribute 'uuid'(or'UuidType') assoon as a UUID-typed column is written, or an Arrow schema containing a UUID is
converted back to Iceberg.
This usage was added in #2007 ("Fix UUID support") without raising the floor, and
the earlier
Bump PyArrow to 18.0.0change only moved the tested/locked version,not the declared lower bound. This PR bumps the floor to
pyarrow>=18.0.0for thefour affected extras so the declared dependency matches what the code actually
requires. See also #1174 (Minimum required pyarrow version).
Are these changes tested?
This is a dependency-floor bump with no code change, so the real guard is CI
resolving the raised floor across the supported Python matrix;
uv lock --checkconfirms
uv.lockstays in sync withpyproject.toml.Locally:
uv lock --checkpasses (lock in sync).make lint(ruff, ruff-format, mypy, license, uv-lock, ...) all pass.tests/io/test_pyarrow_visitor.py(60 passed),tests/utils/test_schema_conversion.py -k UUID(2 passed), and the UUID cases intests/test_conversions.py(2 passed). A manual round-tripUUIDType -> pa.uuid() -> pa.UuidType -> UUIDTyperuns clean.No new test is added because there is no new code path to cover: the existing UUID
conversion tests already exercise
visit_uuidand_ConvertToIceberg, and thefloor bump only makes the declared dependency honest. A heavier alternative would
be to version-gate
pa.uuid()/pa.UuidTypeat runtime (as done forAzureFileSystemviaMIN_PYARROW_VERSION_SUPPORTING_AZURE_FSinpyiceberg/io/pyarrow.py), but that keeps UUID unsupported on 17.x rather thanrequiring the version the feature needs; happy to go that route instead if
preferred.
Are there any user-facing changes?
Yes, indirectly: the minimum supported
pyarrowfor thepyarrow,pandas,duckdb, andrayextras rises from 17.0.0 to 18.0.0. This only affectsenvironments currently resolving
pyarrow17.x, where UUID columns are alreadybroken; it prevents that unsupported install rather than changing any working
behavior. (Adding the
changeloglabel if this warrants a changelog entry - Idon't have permission to set labels as an external contributor.)