Skip to content

Bound OTel traces emitted by long-running subscription handlers - #63

Merged
danielorbach merged 5 commits into
mainfrom
issue-56
Aug 26, 2026
Merged

Bound OTel traces emitted by long-running subscription handlers#63
danielorbach merged 5 commits into
mainfrom
issue-56

Conversation

@danielorbach

@danielorbach danielorbach commented May 31, 2026

Copy link
Copy Markdown
Owner

Long-lived component procedures can process unrelated operations for hours, but the framework kept one OpenTelemetry span open across Procedure.Exec. Handler spans inherited that lifecycle span through L.Context, so trace size grew with component uptime and eventually exhausted downstream live-trace limits. This revision makes the lifecycle a control-flow boundary rather than a trace operation: work started by each handler owns a bounded trace that can complete independently.

WithContext still derives cancellation, deadlines, values, and baggage from its input, but detaches any active span before the procedure begins. The opt-in NewSpanProcessor restores navigability without restoring accidental parentage: it stamps component.name on lifecycle-derived spans and links the detached incoming span only to parentless operation roots. Ordinary child spans retain their normal parent, an explicitly supplied equivalent link is not duplicated, and a component.name supplied when the span starts takes precedence over processor enrichment.

The processor intentionally does not own OpenTelemetry Resource or sampling policy. Process identity such as service.instance.id remains an application-bootstrap concern shared by traces, metrics, and logs. Processor attributes and links are available to exporters and Collector-side policies, but they run after in-process head sampling and therefore cannot influence that decision.

Operation owners now start spans where bounded work begins and handle logging, error recording, span status, and return behavior at the same call site. L.Error no longer attempts to mutate the removed lifecycle span. L.Fatal and L.Fatalf are deprecated while retaining their v1 behavior; new code returns normally for leaf failures and calls L.Terminate before returning only when managed children need cancellation. ProcE follows that supervising form by cancelling with the returned error and then returning normally.

This deliberately omits the older branch's broad trace-attribute options, context-bearing Fatal helpers, and generated process.nonce. It also leaves a possible call-site convenience such as component.Exit for later evidence rather than hiding logging, tracing, and cancellation policy in the core fix. The earlier detachment-only approach remains available for comparison in #64.

The reference loaders and message-loop examples instrument bounded bootstrap, producer, and consumer spans.

Proof: #63 (exported trace topology)

Resolves #56.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes the long-lived per-lifecycle OTel span (issue #56) so that spans started from L.Context() are roots of independent traces, and propagates lifecycle identity via context-stored attributes that a new NewSpanProcessor stamps onto every span. Adds bounded framework spans for runCleanup and Component.Bootstrap, a process-stable process.nonce, and Context-suffixed variants of Error/Fatal so per-handler spans can be the recording target.

Changes:

  • New tracing.go (tracer, processNonce, WithAttributes/WithForkAttributes, NewSpanProcessor) plus package doc; lifecycle.go no longer opens a lifecycle span and wraps cleanup in a bounded <name>.cleanup span.
  • Error/Fatal/WithSpan/WithForkSpanName deprecated; new ErrorContext/FatalContext (+f variants) record onto caller-supplied span; in-repo callers migrated.
  • loader back-links Claim.Footprint, forks components with identity attributes via claimIdentity, emits a bounded <component>.bootstrap span, and warns at Load with the process nonce; kafkaloader/fileloader updated accordingly.

Reviewed changes

Copilot reviewed 17 out of 17 changed files in this pull request and generated 4 comments.

Show a summary per file
File Description
tracing.go New tracer, process nonce, identity-attribute context plumbing, and NewSpanProcessor.
tracing_test.go Coverage for attribute propagation/replacement, nonce stamping, cleanup span, root spans, and context error/fatal recording.
lifecycle.go Drops lifecycle span, stashes identity attrs on ctx, wraps cleanup in bounded span, deprecates Error/Fatal, adds *Context variants.
procedure.go Removes span field and SpanName, deprecates WithSpan/WithForkSpanName, adds attrs to lifecycleOptions.
doc.go New package doc describing the tracing contract, identity propagation, and operator anchor.
loader/loader.go Back-links Claim.Footprint, logs process nonce, forks with WithForkAttributes, bounded <component>.bootstrap span, FatalfContext for bootstrap errors.
loader/example_test.go Migrates ErrorfErrorfContext.
fileloader/jsonloader.go FatalfFatalfContext.
fileloader/yamlloader.go FatalFatalfContext.
fileloader/fileloader.go WithSpan("loader")WithName("loader").
kafkaloader/kafkaloader.go WithSpanWithName, error/fatal migrated to *Context variants, span ctx threaded into handler.
examples/direct/{main,ping,pong}.go Migrate to *Context variants; pong.go adds bounded pong.echo consumer span.
examples/embed/{ping,pong,probe}.go Migrate to ErrorfContext.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread loader/loader.go Outdated
Comment thread procedure.go Outdated
Comment thread procedure.go Outdated
Comment thread loader/loader.go Outdated

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Comment thread lifecycle.go Outdated
Comment thread examples/direct/pong.go Outdated
Comment thread loader/example_test.go Outdated
Comment thread loader/loader.go Outdated
Comment thread examples/direct/main.go Outdated
Comment thread tracing_test.go Outdated
Comment thread loader/loader.go Outdated
Comment thread tracing.go Outdated
Comment thread tracing.go Outdated
Comment thread lifecycle.go Outdated
Comment thread tracing_test.go Outdated
Comment thread tracing_test.go Outdated
@danielorbach

Copy link
Copy Markdown
Owner Author

Could not attach inline (the target line is outside the diff hunk).

procedure.go:53[Low] ProcE.Exec and CleanupError still call the deprecated log-only methods.

ProcE.Exec here calls l.Fatal(err), and CleanupError (lifecycle.go:619) calls l.Error(err) — both now log-only. These are the framework's own internal error funnels (every ForkE/ProcE failure; every CleanupError/Background/Context failure), so the PR's "all in-repo callers migrate" is inaccurate and these paths record onto no span.

ProcE has no ctx to thread; CleanupError could record if the cleanup span's ctx were plumbed into runCleanup (see the lifecycle.go:221 finding). Worth a note either way so the deprecation isn't read as complete.

@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open 14 days with no activity.

Remove the stale label or comment to keep it open, otherwise it will be closed in 7 days.

@github-actions github-actions Bot added the Stale Will be closed soon due to inactivity label Jun 15, 2026
@github-actions

Copy link
Copy Markdown

This PR was closed because it has been inactive for 21 days.

@github-actions github-actions Bot closed this Jun 23, 2026
@danielorbach danielorbach reopened this Jun 23, 2026
@danielorbach danielorbach removed the Stale Will be closed soon due to inactivity label Jun 23, 2026
@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown

This PR is stale because it has been open 14 days with no activity.

Remove the stale label or comment to keep it open, otherwise it will be closed in 7 days.

@github-actions github-actions Bot added the Stale Will be closed soon due to inactivity label Jul 8, 2026
@danielorbach danielorbach removed the Stale Will be closed soon due to inactivity label Jul 8, 2026
@github-actions

Copy link
Copy Markdown

This PR is stale because it has been open 14 days with no activity.

Remove the stale label or comment to keep it open, otherwise it will be closed in 7 days.

@github-actions github-actions Bot added the Stale Will be closed soon due to inactivity label Jul 23, 2026
@danielorbach danielorbach removed the Stale Will be closed soon due to inactivity label Jul 24, 2026
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown

This PR is stale because it has been open 14 days with no activity.

Remove the stale label or comment to keep it open, otherwise it will be closed in 7 days.

@github-actions github-actions Bot added the Stale Will be closed soon due to inactivity label Aug 8, 2026
@danielorbach danielorbach removed the Stale Will be closed soon due to inactivity label Aug 10, 2026
@danielorbach
danielorbach force-pushed the issue-56 branch 2 times, most recently from 3a63a06 to 1ef2f21 Compare August 19, 2026 20:57

@magal10 magal10 left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment thread lifecycle.go Outdated
Comment thread loader/loader.go
Comment thread examples/direct/main.go
Long-lived procedures can handle unrelated operations for the lifetime of
a process. Keeping their span open made every operation part of one
ever-growing trace, exhausting downstream trace limits.

Leaving operation ownership with call sites also establishes the boundary
the following component-attribution and span-linking work can enrich.
Comment thread tracing.go Outdated
Comment thread loader/loader_internal_test.go Outdated
Comment thread tracing.go
Comment thread tracing.go Outdated
Comment thread tracing.go
Comment thread tracing_test.go Outdated
Removing lifecycle parent spans avoids unbounded traces, but would also
remove the navigable relationship between an incoming operation and work
started by a component. An opt-in processor preserves that relationship
without making component own a TracerProvider or its sampling policy.

The attribute is applied in time for exporter and Collector policies.
Application-side component-aware head sampling remains a separate concern.
With lifecycle spans gone, recording errors through L.Context would be a
silent no-op and Fatal would keep termination ownership hidden in the
framework. The in-tree callers now own spans for the bounded work they
perform and choose explicitly between returning and canceling managed
children.

The footprint consumer span ends before Load because loaded components
may run for the lifetime of the process. Its context crosses the
lifecycle boundary only so later operation roots can link back to it.
Call sites now own their bounded spans and error records, so Fatal's
runtime.Goexit is no longer needed as the normal error path. ProcE can
translate its returned error into lifecycle cancellation and then follow
ordinary Go control flow, allowing callers and deferred work to continue
predictably.

Fatal and Fatalf retain their v1 behavior for compatibility while their
deprecations direct new code to choose explicitly between return and
Terminate followed by return.
Continue reports graceful-stop state only. Terminate cancels the lifecycle
context without closing Stopping, so retrying canceled I/O under a
Continue loop can spin forever and prevent the supervisor from completing.

Treating context cancellation as the terminal branch lets managed workers
release the lifecycle wait for both graceful and abrupt shutdown paths.
@danielorbach

Copy link
Copy Markdown
Owner Author

I ran this disposable program against PR head 92bb060 with a real OpenTelemetry SDK provider, component.NewSpanProcessor, and tracetest.SpanRecorder.

main.go
package main

import (
	"context"
	"errors"
	"fmt"
	"sort"

	"github.com/danielorbach/go-component"
	"github.com/danielorbach/go-component/loader"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/sdk/trace/tracetest"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	recorder := tracetest.NewSpanRecorder()
	provider := sdktrace.NewTracerProvider(
		sdktrace.WithSpanProcessor(component.NewSpanProcessor()),
		sdktrace.WithSpanProcessor(recorder),
	)
	previousProvider := otel.GetTracerProvider()
	otel.SetTracerProvider(provider)
	defer otel.SetTracerProvider(previousProvider)

	tracer := provider.Tracer("trace-proof")
	upstreamCtx, upstream := tracer.Start(context.Background(), "upstream request")

	component.RunProc(func(l *component.L) {
		fmt.Printf("L.Context has active span: %t\n", trace.SpanContextFromContext(l.Context()).IsValid())

		operationCtx, operation := tracer.Start(l.Context(), "operation")
		_, child := tracer.Start(operationCtx, "child")
		child.End()
		operation.End()

		l.Go("worker", func(l *component.L) {
			_, operation := tracer.Start(l.Context(), "worker operation")
			operation.End()
		})
	}, component.WithContext(upstreamCtx), component.WithName("service"))

	failure := errors.New("bootstrap failed")
	claim := &loader.Claim{Component: &component.Descriptor{
		Name: "database",
		Bootstrap: func(*component.L, component.Linker, any) error {
			return failure
		},
	}}
	component.Run(claim, component.WithContext(upstreamCtx), component.WithName("loader/database"))
	upstream.End()

	if err := provider.Shutdown(context.Background()); err != nil {
		panic(err)
	}
	printSpans(recorder.Ended())
}

func printSpans(spans []sdktrace.ReadOnlySpan) {
	sort.Slice(spans, func(i, j int) bool { return spans[i].Name() < spans[j].Name() })
	for _, span := range spans {
		parent := "root"
		if span.Parent().IsValid() {
			parent = span.Parent().SpanID().String()
		}

		name := "-"
		attrs := attribute.NewSet(span.Attributes()...)
		if value, ok := attrs.Value(component.TraceKey); ok {
			name = value.AsString()
		}

		links := "-"
		if got := span.Links(); len(got) != 0 {
			links = got[0].SpanContext.SpanID().String()
		}

		fmt.Printf("%-20s trace=%s span=%s parent=%s link=%s component=%s status=%s errors=%d\n",
			span.Name(),
			span.SpanContext().TraceID().String(),
			span.SpanContext().SpanID().String(),
			parent,
			links,
			name,
			span.Status().Code,
			len(span.Events()),
		)
	}
}

Output from the run (trace and span IDs are generated per run):

L.Context has active span: false
time=2026-08-26T15:11:13.414+03:00 level=ERROR msg="bootstrap component" component.name=loader/database err="bootstrap failed"
child                trace=321773964a62b75b07ba3ec13d2b80f8 span=ee3e1d157ded2831 parent=ef03f259fceda731 link=- component=service status=Unset errors=0
component.bootstrap  trace=7c24b88cdb9e4e146d89a71f7160cb0a span=56c0ac06dea1d17a parent=root link=040be46de28fb706 component=database status=Error errors=1
operation            trace=321773964a62b75b07ba3ec13d2b80f8 span=ef03f259fceda731 parent=root link=040be46de28fb706 component=service status=Unset errors=0
upstream request     trace=0c6c3d61bcd84c8d0b09885915636267 span=040be46de28fb706 parent=root link=- component=- status=Unset errors=0
worker operation     trace=da5503aa294ec73cfdd9e0238916ca28 span=37aa62b63d853587 parent=root link=040be46de28fb706 component=service/worker status=Unset errors=0

This demonstrates the intended boundaries:

  • L.Context() does not expose the inherited span as active.
  • Component operations are independent roots linked to the upstream span.
  • An operation's child retains normal parentage and receives no redundant link.
  • Forked work receives the derived service/worker component name.
  • The bounded bootstrap span records the failure and has Error status.

@danielorbach
danielorbach merged commit a7d3545 into main Aug 26, 2026
9 checks passed
@danielorbach
danielorbach deleted the issue-56 branch August 26, 2026 12:33
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Bounding OTel traces emitted by long-running subscription handlers

3 participants