feat(core): add request hook to enrich T4 network spans with GCP resource attributes (E) - #18272
Conversation
There was a problem hiding this comment.
Code Review
This pull request introduces OpenTelemetry request hooks to extract and inject Google Cloud semantic and resource attributes (such as resource name, parent, and project ID) from gRPC request objects into OpenTelemetry spans. Specifically, it adds _extract_t4_attributes and _client_request_hook helper functions in _observability.py, registers the hook in the gRPC client interceptors, and includes comprehensive unit tests to verify this behavior. There are no review comments, so I have no feedback to provide.
a7be0e4 to
9efe757
Compare
b5a883a to
83963a7
Compare
9efe757 to
40071ff
Compare
83963a7 to
bf82825
Compare
40071ff to
8c9d21a
Compare
bf82825 to
bc94977
Compare
8c9d21a to
558f8fd
Compare
bc94977 to
67e3879
Compare
558f8fd to
a6f3b3a
Compare
67e3879 to
6b9bfab
Compare
a6f3b3a to
7e1e498
Compare
fb16695 to
9b46ef7
Compare
7e1e498 to
131a80a
Compare
8c5d285 to
62a8261
Compare
131a80a to
0da0216
Compare
007f6b0 to
5b39cff
Compare
0da0216 to
1c3834e
Compare
5b39cff to
c15d70a
Compare
de053b3 to
051b3d7
Compare
| endpoint = getattr(client_options, "api_endpoint", None) | ||
|
|
||
| if endpoint and isinstance(endpoint, str): | ||
| clean = endpoint.replace("http://", "").replace("https://", "").strip("/") |
This comment was marked as resolved.
This comment was marked as resolved.
Sorry, something went wrong.
There was a problem hiding this comment.
Replaced the manual approach with urllib.parse. Great catch. Thnaks!
| attrs["gcp.grpc.resend_count"] = resend_count | ||
|
|
||
| name = getattr(request, "name", None) | ||
| if isinstance(name, str) and name: |
There was a problem hiding this comment.
When would you expect this to be non-string? Are you sure we should fall back to the parent in that case?
| else: | ||
| parent = getattr(request, "parent", None) | ||
| if isinstance(parent, str) and parent: | ||
| attrs["gcp.resource.destination.id"] = parent |
There was a problem hiding this comment.
nit: we could reduce some duplication here:
resource_id = getattr(request, "name", None) or getattr(request, "parent", None)
if isinstance(resource_id, str) and resource_id:
attrs["gcp.resource.destination.id"] = resource_id
There was a problem hiding this comment.
This is OBE. We no longer process "gcp.resource.destination.id"
| _make_grpc_client_request_hook(endpoint_attrs) | ||
| if endpoint_attrs | ||
| else _grpc_client_request_hook | ||
| ) |
There was a problem hiding this comment.
nit: can't this just be _make_grpc_client_request_hook(endpoint_attrs)? It seems like the implementation already handles empty endpoint_attrs, so I'm not sure we need to handle both cases here
There was a problem hiding this comment.
Simplified. Good catch. Appreciate it.
| return False | ||
|
|
||
|
|
||
| _STATUS_CODE_NAMES = { |
There was a problem hiding this comment.
In the PR description, you say " Without request- and response-level enrichment, spans cannot identify ... normalized string status codes".
Can you expand on that? Does the default instrumenter add code numbers, but not strings? It feels strange that we would have to add the cost of an extra callback layer to do that kind of transformation
Could we add this somewhere else in the stack? Or just stick with error numbers instead of names?
There was a problem hiding this comment.
the PR description was outdated and described an earlier prototype where the response hook was trying to do too much.
To answer your questions directly:
-
Does the default instrumenter add code numbers, but not strings?
Yes. Upstreamopentelemetry-instrumentation-grpcsetsrpc.grpc.status_codeas an integer (0,5, etc.) per OTel gRPC conventions. It does not setrpc.response.status_codeat all. -
Can we do this elsewhere / avoid callback translation overhead?
Yes, we moved error resolution out of this hook. All error status mapping ("NOT_FOUND","RESOURCE_EXHAUSTED", etc.) andgcp.errors.*attributes are handled directly on the client method span ingoogle.api_core.gapic_v1.method(PR feat(gapic): add OpenTelemetry T3 client method span wrapping in gapic_v1.method (D) #18274) where exceptions are caught naturally. -
What
_grpc_client_response_hookactually does in this PR:
The response hook here does zero error parsing, zero translation, and zero dictionary lookups. It only stampsrpc.response.status_code = "OK"on successful wire spans (checkingspan.statusfirst to bail out if an error occurred).
I've updated the PR description to reflect the current, simplified implementation.
daniel-sanche
left a comment
There was a problem hiding this comment.
My main comment is around the response callback. Are you sure it can do what we need? And do we really need a callback for status names?
051b3d7 to
2460284
Compare
d6307d6 to
b3931e9
Compare
| if request is None: | ||
| return attrs | ||
|
|
||
| resend_count = getattr(request, "resend_count", None) |
There was a problem hiding this comment.
My understanding is that request will be a protobuf message, which wouldn't have this field. What do you expect it to be? Can we have any clearer typing here?
There was a problem hiding this comment.
DONE:
processing of the resend_count attribute has been removed.
it is in the deferred category so not required for the MVP.
Handling of:
attrs: dict[str, Any] = {
"rpc.system.name": "grpc",
}
is now moved to client_request_hook() within _make_grpc_client_request_hook()
b1d7f5d to
67c8ae7
Compare
b3931e9 to
99a4d3d
Compare
… hook - Add rpc.system.name: 'grpc' - Extract server.address and server.port from client options endpoint - Extract gcp.grpc.resend_count from request resend count - Extract gcp.resource.destination.id from request name or parent - Add _client_response_hook for status code, error.type, and status.message - Plumb response_hook into get_otel_interceptor and get_otel_async_interceptor
- Test endpoint attribute parsing across host/port variations - Test destination id and resend count extraction - Test client request and response hooks covering all status and error cases - Test interceptor creation and custom endpoint attribute propagation - Achieve 100% statement and branch coverage on _observability.py
…and hooks - Rename _extract_t4_attributes to _extract_grpc_request_attributes - Rename _make_client_request_hook to _make_grpc_client_request_hook - Rename _client_request_hook to _grpc_client_request_hook - Rename _client_response_hook to _grpc_client_response_hook - Preserve generic _extract_endpoint_attributes for shared transport usage
…ntion - Rename test_extract_t4_attributes to test_extract_grpc_request_attributes - Rename test_client_request_hook to test_grpc_client_request_hook - Rename test_client_response_hook to test_grpc_client_response_hook - Update interceptor hook references to _grpc_client_* hooks
- Add url.domain extraction from universe_domain or default to googleapis.com - Add _extract_error_attributes helper to extract gcp.errors.domain and gcp.errors.metadata.<key> - Omit server.port when port matches scheme defaults (443 for https/grpc, 80 for http) - Remove redundant _grpc_client_response_hook and _STATUS_CODE_NAMES - Deduplicate name and parent resource lookup for gcp.resource.destination.id - Add comprehensive parametrized unit tests and update interceptor test suites
…tem attribute - Strip leading slash from gRPC attempt span names via span.update_name - Set rpc.method to the fully qualified method name per PRD specification - Retain rpc.system.name: 'grpc' and remove legacy rpc.system attribute to avoid duplication - Update unit tests to verify span name normalization and attribute deduplication
- Remove gcp.resource.destination.id extraction from _extract_grpc_request_attributes - Update unit tests to reflect attribute removal per July Strategy Update
…ments without grpc
…parsing, and attribute handling
…nv version in response hook
0fb354a to
c0fadc4
Compare
Problem
Low-level gRPC transport spans generated by
opentelemetry-instrumentation-grpccapture standard RPC metadata but lack Google Cloud semantic context. Without request- and response-level enrichment, spans cannot identify configured endpoint attributes (server.address,server.port,url.domain), resend count, or the Cloud Observability success status (rpc.response.status_code = "OK").Solution
This PR enriches wire-level gRPC client spans in
google-api-coreby attaching lightweight request and response hooks to both synchronous and asynchronous OpenTelemetry gRPC interceptors:Request Hook (
_make_grpc_client_request_hook):rpc.system.name: "grpc"per OpenTelemetry semantic conventions./, if present)rpc.methodResponse Hook (
_grpc_client_response_hook):rpc.response.status_code = "OK"on successful sync or asycn RPC completions.Transport Integration:
get_otel_interceptor(sync) andget_otel_async_interceptor(async) withingoogle.api_core._observability.Extracts Endpoint Attributes:
server.address,server.port(omitted for standard 443/80 ports), andurl.domainfromclient_options.Notes for Reviewers
gcp.errors.*attributes are handled at the client method span level). Those are covered in a previous PR #18274