Skip to content

PEP 835: Address Discourse feedback from encukou and tjreedy #4998

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
till-varoquaux wants to merge 1 commit into
base: main
Choose a base branch
from
+59 −41
Open

PEP 835: Address Discourse feedback from encukou and tjreedy #4998

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
100 changes: 59 additions & 41 deletions peps/pep-0835.rst
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -228,46 +228,11 @@ origin:
Supported Left-Hand Operands
-----------------------------

The ``@`` operator is implemented by adding ``nb_matrix_multiply`` to the
metatype (``type``) and to several typing-related types. The operator is
supported for any left-hand operand that currently supports the ``|``
operator for making a union.
The ``@`` operator is implemented by adding ``nb_matrix_multiply`` to the metatype (``type``) and to several typing-related types. The operator is supported for instances of ``type`` and other typing constructs that currently support the ``|`` operator for unions. The specific handling of ``None`` is currently unsettled (see Open Issues).

For all other left-hand operands, the operator returns ``NotImplemented``,
allowing normal ``__matmul__`` dispatch to proceed.

Parsing and Grammar
===================

This proposal requires no changes to the Python grammar. Because ``@`` is
already a valid operator, it is natively parsed as a binary operation. The
shorthand is resolved during semantic analysis, entirely bypassing the need
to patch grammar files or update the parser.

How to Teach This
=================

In Python, the ``@`` symbol already has an established association with
metadata through decorators. The annotation shorthand extends this
intuition to the type system: ``int @ Field(gt=0)`` reads as "``int``,
decorated with ``Field(gt=0)``."

For beginners, the key rule is simple: **in a type annotation, ``@`` means
"with this metadata."** The full ``Annotated[int, Field(gt=0)]`` syntax
remains available and is entirely equivalent for those who find it clearer.

For experienced developers, the precedence rules follow standard Python
operator precedence (``@`` binds tighter than ``|``), and chaining
``T @ m1 @ m2`` flattens exactly as nested ``Annotated`` does.

Documentation and teaching materials should introduce the shorthand alongside
``Annotated``, not as a replacement. The longhand form is still preferred in
contexts where multiple metadata items are passed as a group and chaining
would be unwieldy.

Backwards Compatibility
=======================

Forward References and Deferred Evaluation
-------------------------------------------

Expand All @@ -277,8 +242,9 @@ module provides several formats for retrieving annotations:
- ``Format.VALUE``: Fully evaluates the annotation. Raises ``NameError``
if any name is unresolvable.
- ``Format.FORWARDREF``: Wraps unresolvable names in ``ForwardRef`` objects.
However, compound expressions using operators (``@``, ``|``) produce an
opaque ``ForwardRef`` string containing the entire expression.
However, when operators like ``@`` or ``|`` fail to evaluate because of an
unresolvable name, this format falls back to returning the entire expression
as an opaque string wrapped in a single ``ForwardRef`` object.
- ``Format.STRING``: Returns the raw source text with no evaluation.

For the ``@`` operator, ``Format.FORWARDREF`` is insufficient. Consider::
Expand All @@ -293,8 +259,8 @@ forward reference is resolved. This is a blocking issue for libraries like
Pydantic and FastAPI, which inspect metadata at class-definition time.

This proposal introduces a new format, ``Format.FORWARDREF_STRUCTURAL``.
This format assumes typing semantics and evaluates compound type expressions
**structurally**. It always returns an ``AnnotatedType`` for ``@``, a union
This format assumes typing semantics and evaluates type expressions involving
operators (like ``@`` and ``|``) **structurally**. It always returns an ``AnnotatedType`` for ``@``, a union
for ``|``, and a ``GenericAlias`` for subscripting. When a name cannot be
resolved, only that name is wrapped in ``ForwardRef``; the surrounding
operators are still evaluated. The example above produces::
Expand All @@ -321,6 +287,38 @@ coarser than from :pep:`749` thunks. When a name is unresolvable, the
rather than just a name. :pep:`749` provides a strictly better experience and
is the recommended path forward.

Parsing and Grammar
===================

This proposal requires no changes to the Python grammar. Because ``@`` is
already a valid operator, it is natively parsed as a binary operation. The
shorthand is resolved during semantic analysis, entirely bypassing the need
to patch grammar files or update the parser.

How to Teach This
=================

In Python, the ``@`` symbol already has an established association with
metadata through decorators. The annotation shorthand extends this
intuition to the type system: ``int @ Field(gt=0)`` reads as "``int``,
decorated with ``Field(gt=0)``."

For beginners, the key rule is simple: **in a type annotation, ``@`` means
"with this metadata."** The full ``Annotated[int, Field(gt=0)]`` syntax
remains available and is entirely equivalent for those who find it clearer.

For experienced developers, the precedence rules follow standard Python
operator precedence (``@`` binds tighter than ``|``), and chaining
``T @ m1 @ m2`` flattens exactly as nested ``Annotated`` does.

Documentation and teaching materials should introduce the shorthand alongside
``Annotated``, not as a replacement. The longhand form is still preferred in
contexts where multiple metadata items are passed as a group and chaining
would be unwieldy.

Backwards Compatibility
=======================

Operator Overloading
--------------------

Expand Down Expand Up @@ -354,7 +352,7 @@ metaclass.
The private ``typing._AnnotatedAlias`` class is retained as a deprecated
compatibility shim. Code using ``isinstance(x, typing._AnnotatedAlias)``
will continue to work but emit a ``DeprecationWarning``. The shim is
scheduled for removal in Python 3.18.
scheduled for removal in Python 3.21.

Code that should be updated:

Expand Down Expand Up @@ -481,6 +479,26 @@ References
- :pep:`727` -- Documentation metadata in typing (Withdrawn)
- :pep:`749` -- Implementing PEP 649

Open Issues
===========

Supporting ``None``
-------------------

While the ``@`` operator naturally applies to instances of ``type``, it is currently unsettled how it should handle ``None``.

There are two primary options, both with notable downsides:

1. **Implement ``__matmul__`` on ``NoneType``:** This provides the best ergonomics, allowing ``None @ Metadata``. However, it introduces a runtime risk. If a user accidentally uses matrix multiplication on a ``None`` value (e.g., ``a @ b`` where ``a`` is ``None`` instead of a numpy array), it will silently return an ``Annotated`` object instead of raising a ``TypeError``.
2. **Do not support ``None @ Metadata``:** This avoids the runtime risk, but creates an ergonomic cliff. Users would be forced to write ``type(None) @ Metadata`` or fall back to ``typing.Annotated[None, Metadata]``.

The authors of this PEP lean slightly in favor of implementing ``__matmul__`` on ``NoneType`` for its ergonomic benefits, but ultimately defer to the Typing Council and community discussion to decide the best path forward.

Deprecation Timeline
--------------------

This PEP schedules the removal of the ``typing._AnnotatedAlias`` compatibility shim for Python 3.21, following the standard 5-year deprecation policy (:pep:`387`). However, given the widespread use of ``typing.Annotated``, the exact timeline for removal—or whether the shim should be retained indefinitely to prevent churn in legacy code—remains open for community discussion.

Copyright
=========

Expand Down
Loading