opentelemetry-process-context: implement Resource publishing (OTEP-4719)

[herin049]

Description

External profilers and observability tools (such as the OpenTelemetry eBPF Profiler) operate outside the instrumented process and have no access to resource attributes configured inside OpenTelemetry SDKs. OTEP-4719 (Process Context Sharing) introduces a standard mechanism for OpenTelemetry SDKs to publish resource attributes for access by out-of-process readers

This PR introduces the opentelemetry-process-context package as a Rust backed Python extension (built with maturin/pyo3) that implements the process outlined in OTEP-4719. It publishes the SDK's resource attributes to
a memory region that any external reader with access to /proc/<pid>/maps and /proc/<pid>/mem can discover.

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource) -> None: ...
def unpublish_context() -> None: ...

Compatability

The original OTEP explicitly mentions that this functionality is Linux only. However, I have tried to implement this package in such a way that limited testing/development can be performed natively on any Posix compatible system (e.g. MacOS). However, the full functionality of OTEP-4719 is only available on Linux and Windows is explicitly not supported.

Feature	Linux	macOS
Backing region	memfd_create("OTEL_CTX") -> mmap(MAP_PRIVATE)	Anonymous mmap
Reader discoverability	/proc/<pid>/maps shows memfd:OTEL_CTX	Not discoverable (no /proc)
Fork protection	madvise(MADV_DONTFORK)	Skipped
Mapping name	prctl(PR_SET_VMA_ANON_NAME, "OTEL_CTX")	Skipped
Publish timestamp clock	CLOCK_BOOTTIME	CLOCK_MONOTONIC
Production use	Fully supported	Development/testing only

Fixes #5291

Type of change

Please delete options that are not relevant.

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

How Has This Been Tested?

uv run tox -e py314-test-opentelemetry-process-context

Does This PR Require a Contrib Repo Change?

• Yes. - Link to PR:
• No.

Checklist:

• Followed the style guidelines of this project
• Changelogs have been updated
• Unit tests have been added
• Documentation has been updated

[ocelotl]

Key components of this PR are written in Rust, reviewers of this repo may not be experts in this language. I suggest adding comments to all the rust files for ease of understanding:

publish.rs.txt
proto.rs.txt
lib.rs.txt
convert.rs.txt
build.rs.txt

[ocelotl]

I found a few things that may be required in the OTEP but missing here. If so, they are all very small changes to this PR.

A few other things are minor but also worth of considering.

[ocelotl]

Some more automatic review comments ✌️

[herin049]

@ocelotl I've switched from pulling the proto files in the build.rs to creating a bash script in scripts/ which generates the Rust files, so they're included in the sdist. This should no longer require network access for building.

Also, I've combined publish_context/update_context into a single Python function, so the API now just contains publish_context and unpublish_context methods.

[ocelotl]

Looks good!

Just a question

[Copilot]

Pull request overview

This PR adds a new opentelemetry-process-context package to the OpenTelemetry Python monorepo. The package is a Rust-backed (PyO3/maturin) Python extension that publishes the SDK Resource into a discoverable process memory region for out-of-process readers, following the mechanism described in OTEP-4719.

Changes:

• Introduces the opentelemetry-process-context package (Python API + Rust implementation + tests).
• Adds CI coverage (tox envs + GitHub Actions) and release workflow steps to build/upload native wheels and sdist.
• Adds a maintainer script to regenerate the embedded Rust prost code for the process-context protobufs.

Reviewed changes

Copilot reviewed 23 out of 30 changed files in this pull request and generated 6 comments.

Show a summary per file

File	Description
uv.lock	Adds the new workspace member and editable dependency entry.
tox.ini	Adds test/lint envs for the new package and constrains tests to Linux.
scripts/proto_codegen_process_context.sh	New script to regenerate Rust prost code from upstream protos.
pyproject.toml	Adds the new package to the workspace dependencies/members and tool config.
opentelemetry-process-context/tests/test_process_context.py	Adds lifecycle + cross-process /proc validation tests.
opentelemetry-process-context/tests/init.py	Test package marker.
opentelemetry-process-context/test-requirements.in	Adds test dependency input for uv compilation.
opentelemetry-process-context/test-requirements.txt	Compiled test dependency lock for the package tests.
opentelemetry-process-context/src/opentelemetry/process_context/init.py	Public API re-export of the extension functions.
opentelemetry-process-context/src/opentelemetry/process_context/version.py	Package version constant.
opentelemetry-process-context/src/opentelemetry/process_context/py.typed	Marks the package as typed.
opentelemetry-process-context/src/opentelemetry/process_context/_rs/init.pyi	Type stubs for the Rust extension module.
opentelemetry-process-context/rust/src/lib.rs	PyO3 module entrypoint exposing publish/unpublish.
opentelemetry-process-context/rust/src/context.rs	Core shared-memory publication logic.
opentelemetry-process-context/rust/src/convert.rs	Converts Python Resource attributes into protobuf AnyValue/Resource.
opentelemetry-process-context/rust/src/proto.rs	Rust module wiring for checked-in generated prost code.
opentelemetry-process-context/rust/src/generated/opentelemetry.proto.common.v1.rs	Generated protobuf types used for encoding attributes.
opentelemetry-process-context/rust/src/generated/opentelemetry.proto.resource.v1.rs	Generated protobuf types for Resource.
opentelemetry-process-context/rust/src/generated/opentelemetry.proto.processcontext.v1development.rs	Generated protobuf types for ProcessContext.
opentelemetry-process-context/rust/Cargo.toml	Rust crate configuration for the extension.
opentelemetry-process-context/rust/Cargo.lock	Rust dependency lockfile for reproducible builds.
opentelemetry-process-context/pyproject.toml	Defines the Python package (maturin backend) and build metadata.
opentelemetry-process-context/README.rst	Package README scaffold.
opentelemetry-process-context/LICENSE	Package license file.
.github/workflows/test.yml	Adds Ubuntu test jobs for the new tox envs.
.github/workflows/lint.yml	Adds lint job for the new package.
.github/workflows/release.yml	Adds wheel/sdist build jobs for the native extension and downloads artifacts into dist/.
.github/workflows/generate_workflows.py	Updates workflow generator to respect tox platform constraints.
.codespellrc	Skips Rust target artifacts and ignores “crate”.
.changelog/5337.added	Adds changelog entry for the new feature/package.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[aabmass]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.

I'm not super up to date on this though

[herin049]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.

I'm not super up to date on this though

There is a PR for this package in the Rust repo: https://github.com/open-telemetry/opentelemetry-rust/pull/3460/changes#diff-a475efbb673d2bddd008746034e46ce4acd67d12f3629d7a2fd57120659276d7

But, it introduces a lot of heavy dependencies that we should try to avoid. Maybe there is a discussion to be had about making this lighter weight? I'm also unsure of whether that implementation is complete since it doesn't appear to handle forking properly from what I can tell.

[aabmass]

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource) -> None: ...
def unpublish_context() -> None: ...

Are there use cases where it would need to be called from python code?

[herin049]

Very cool! My hunch is a lot of this rust code which implements the exposition format from the OTEP is not python specific, and maybe the OTEP authors have already written it or would be open to maintaining it elsewhere since it could apply across languages.
I'm not super up to date on this though

There is a PR for this package in the Rust repo: https://github.com/open-telemetry/opentelemetry-rust/pull/3460/changes#diff-a475efbb673d2bddd008746034e46ce4acd67d12f3629d7a2fd57120659276d7

But, it introduces a lot of heavy dependencies that we should try to avoid. Maybe there is a discussion to be had about making this lighter weight? I'm also unsure of whether that implementation is complete since it doesn't appear to handle forking properly from what I can tell.

Python API

The introduced Python API has a very light surface area, consisting of two methods to publish and unpublish the process context.

def publish_context(resource: Resource) -> None: ...
def unpublish_context() -> None: ...

Are there use cases where it would need to be called from python code?

This PR only contains the logic to publish the Resource (not the thread context information yet). From my understanding, the idea is that on application startup (e.g. via auto-instrumentation or manual), publish_context() will be called with the newly created Resource object created by the SDK. Likewise, we'll want to register a post-fork handler to publish the newly created Resource since the behavior is that the process context is not inherited by forked processes. Ultimately, there needs to be a way for the Python SDK to associate its Resource object with the one that is published which is why these methods exist.

There will be a follow-up PR which will call publish_context() during auto-instrumentation if the opentelemetry-process-context package is available.

[aabmass]

This PR only contains the logic to publish the Resource (not the thread context information yet).

Ahh thanks I missed that makes sense. Do you mind updating the PR title with "Sharing Resource Attributes". If it's OK, I'd like to wait for @lzchen to take a pass at the Rust code and approve the approach.