feat: Structured Agent Execution Event Log with ContextItem System#3379
Draft
JasonW404 wants to merge 53 commits into
Draft
feat: Structured Agent Execution Event Log with ContextItem System#3379JasonW404 wants to merge 53 commits into
JasonW404 wants to merge 53 commits into
Conversation
…PR-0) Add foundational context management module under sdk/nexent/core/agents/context/ to support W8/W12/W13 workstreams for progressive component reduction, history projections, and unified context policy. Core components: - ContextItem: Fine-grained context unit with type, authority tier, and fidelity levels - ContextItemHandler: Pluggable handler interface for scoring and reducing context items - ItemHandlerRegistry: Registry mapping context item types to their handlers - 10 built-in handlers (system_prompt, tool, skill, memory, knowledge_base, etc.) - ContextProjector: Converts ContextComponent to ContextItem with proper authority/fidelity - Policy models: SelectionDecision, MemoryDecision for traceable policy decisions - ReductionResult: Immutable record of context reduction operations - Reason codes: 11 standardized codes for decision traceability Testing: - 77 unit tests covering all data models, handlers, and registry - All tests pass with 0.59s execution time This is infrastructure-only (no existing code modified) and provides the foundation for subsequent PRs implementing context projection, policy engines, and handlers.
…PR-1) - Add use_context_items config field (default False for backward compatibility) - Add context_items field to ContextEvidence for traceability - Implement project_context_items() method in ContextManager - Integrate projection logic into assemble_final_context() - Store _source_component reference in metadata for semantic equivalence - Use component.to_messages() to produce formatted text (not JSON dumps) - Add 15 projection tests covering all 7 component types - Add 3 ManagedContextRuntime integration tests - Remove test __init__.py files to fix namespace collision All 112 tests pass. Oracle verified complete and correct.
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…e (PR-2) Implement W12 DB History Projection to persist and reconstruct ReAct execution details from conversation history. Database layer: - Add 5 nullable columns: run_id, step_id, tool_call_id, event_time on conversation_message_t and conversation_message_unit_t - Add indexes on run_id and tool_call_id for query performance - Extend conversation_db.py with optional history projection params - Add get_message_units_by_run() and get_max_run_id_for_conversation() SDK layer: - Create HistoryProjector with dependency injection for DB queries - Support 3 projection modes: model_context, resume, chat - Produce HISTORY_TURN, TOOL_CALL_RESULT, WORKING_MEMORY ContextItems - Wire into ContextManager.use_context_items=True path via config Service layer: - Track run_id/step_id/tool_call_id/event_time in _stream_agent_chunks() - Pass tracking fields through save_message/save_message_unit wrappers - Compute run_id before message creation for proper persistence Tests: 21 new tests, 114 total context tests passing, zero regressions.
…ration tests Production wiring: - Thread conversation_id through execution chain: agent_service → create_agent_run_info → AgentConfig → ManagedContextRuntime → assemble_final_context → HistoryProjector - Inject HistoryProjector into ContextManagerConfig before get_or_create_context_manager() so both run-scoped and conversation-scoped ContextManager paths receive it - use_context_items remains False by default (opt-in) Integration tests (7 new, 28 total): - TestChatProjectionCompleteness: unit type coverage, ordering, metadata, source_refs completeness - TestEndToEndIntegration: component + history projection → FinalContext, graceful failure handling, conversation_id gating 121 context tests + 28 HistoryProjector tests passing, zero regressions.
…eline Blocker 0: Call register_all() in ContextManager.__init__ so ItemHandlerRegistry is populated before any projection occurs. Blocker 1: Add to_messages() method to ContextItemHandler base class with default implementation, plus type-specific overrides in HistoryTurnHandler, ToolCallResultHandler, and WorkingMemoryHandler. Blocker 2: Remove mock of ItemHandlerRegistry.get in integration test so the end-to-end path exercises real handlers. 198 tests passing (121 context agent + 77 context module), zero regressions.
…ort regression Move register_all() call from ContextManager.__init__ to lazy initialization in project_context_items() to avoid breaking isolated test loading. Changes: - Remove register_all() from __init__ (line 301-302) - Add _ensure_handlers_registered() helper with idempotency flag - Call helper at start of project_context_items() - Handlers now register only when actually needed This fixes the 124-test regression caused by eager import in __init__ breaking test_agent_context/test_pure_functions.py isolated loading. All 254 context tests now pass (121 context + 77 module + 56 agent_context).
Add comprehensive OTel tracing to provide visibility into the context assembly pipeline: - ManagedContextRuntime: trace prepare_step() and prepare_final_answer() - ContextManager: trace assemble_final_context(), project_context_items(), compress_if_needed(), and _do_generate_summary() with nested LLM spans - HistoryProjector: trace project() with nested DB query spans All spans include relevant attributes (conversation_id, purpose, counts) and span events for key milestones. Backward compatible - spans are no-ops when monitoring is disabled. Testing: All 254 existing tests pass, no syntax errors.
…ct_items attributes Critical fix: - Move 'with self._lock:' block inside trace_operation context manager - Ensures all compression work (including LLM calls) is properly traced - Fixes broken parent-child span hierarchy for compression path Minor improvements: - Add component_count as span attribute to project_context_items - Remove unused 'as span' from assemble_final_context All 254 tests pass.
Previously _last_uncompressed_est was set from input_messages which are already compressed by prepare_step, making est_raw_i always equal est_i and save% structurally ~0%. Now pull the truly-uncompressed token count from ContextManager.get_token_counts()['last_uncompressed'], recorded in compress_if_needed from the raw memory before compression. Co-Authored-By: Claude <noreply@anthropic.com>
Add test scripts and fixtures used during context-manager refactor development. Only .py and .md files are tracked; .out run artifacts remain ignored via .git/info/exclude. Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
…step.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…_renderer.py Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The token_estimation module lives at core/utils/, not agents/utils/. From agent_context/ sub-package, three dots (...utils) are needed instead of two (..utils). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…remove monolith Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
…ent_context package Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The v2 PreviousCompressor and CurrentCompressor do not fall through from incremental to fresh when LLM returns None - they return PreviousCompressResult/CurrentCompressResult(summary_text=None) immediately. Updated tests P3, C4 to match this behavior. Updated P4 and C6_asymmetry to patch _summarize_pairs with PreviousCompressResult instead of raw tuples. Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
The context_runtime package lives at core/context_runtime/, not agents/context_runtime/. From agent_context/ sub-package, three dots (...context_runtime) are needed instead of two (..context_runtime). Co-Authored-By: Claude Haiku 4.5 <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude <noreply@anthropic.com>
- New test_stats_export.py: 21 tests covering all pure functions (100%) - New test_step_renderer.py: 24 tests covering truncation, rendering, compress_history_offline with mock LLM (49%→86%) - Extended test_pure_functions.py: tests for _is_context_length_error, has_invoked_tools, message_role, trim_pairs_to_budget (92%→97%) Overall agent_context package coverage: 71% → 81% Co-Authored-By: Claude <noreply@anthropic.com>
Merge origin/develop and liudfgoo/feat/opt-agent-context-refactor-v2 into
feat/W5_structured_agent_execution_event_log.
The refactor branch split agent_context.py into a package structure:
- agent_context/__init__.py (re-exports)
- agent_context/manager.py (ContextManager orchestrator)
- agent_context/summary_step.py (SummaryTaskStep, ManagedRunContext)
- agent_context/budget.py (pure budget/cache/fingerprint helpers)
- agent_context/step_renderer.py (StepRenderer, compress_history_offline)
- agent_context/llm_summary.py (LLMSummary class)
- agent_context/previous_compression.py (PreviousCompressor)
- agent_context/current_compression.py (CurrentCompressor)
- agent_context/stats_export.py (stats export functions)
Our changes (OTel instrumentation + strategy filtering fix) were applied
to the new package structure:
1. summary_step.py: Added selected_components field to ManagedRunContext
2. manager.py:
- Added OTel imports and _ensure_handlers_registered() helper
- Wrapped compress_if_needed() with OTel tracing (input/output at all return points)
- Modified prepare_run_context() to perform strategy selection and store selected_components
- Modified assemble_final_context() with use_context_items logic, OTel tracing,
conversation_id parameter, and selected_components for fingerprints
- Added project_context_items() method with OTel tracing
3. loader.py: Added nexent.monitor stub for test isolation
All 240 agent_context tests pass.
All 36 context_runtime/history_projector/integration tests pass.
All 113 monitoring tests pass.
All 5 strategy filtering tests pass.
All 5 PR-0/1/2 comprehensive tests pass.
Moved to test/sdk/core/agents/: - test_pr0_1_2_comprehensive.py → test_context_pr0_1_2.py - test_strategy_filtering_fix.py → test_strategy_filtering.py Deleted (redundant or not suitable for CI): - test_otel_smoke.py (manual smoke test, not a proper unit test) - test_budget_equivalence.py (covered by test_strategy_filtering.py) - test_assemble_equivalence.py (covered by test_strategy_filtering.py) - test_agent_context_items.py (integration test requiring API keys)
- test_pr0_handlers -> test_handlers - test_pr0_projector -> test_projector - test_pr1_equivalence -> test_equivalence - test_pr2_history_projector -> test_history_projector - Updated all test output labels to remove PR-0/1/2 prefixes
Merged origin/develop which includes the agent_context refactor PR (#3383). Resolved conflicts by keeping our OTel instrumentation and strategy filtering fixes on top of the refactored package structure. Conflicts resolved: - manager.py: Kept OTel tracing, strategy selection, use_context_items logic - summary_step.py: Kept selected_components field - loader.py: Kept nexent.monitor stub for test isolation All 240 agent_context tests pass. All 5 strategy filtering tests pass. All 5 context item pipeline tests pass.
- Add conversation_id parameter to create_agent_config and create_agent_run_info assertions - Add run_id, step_id, tool_call_id, event_time parameters to create_message_unit assertions - Update fake_save_message to accept **kwargs for additional parameters - All tests now pass individually (177 + 327 + 68 = 572 tests) - Note: test__stream_agent_chunks_captures_final_answer_and_adds_memory has a pre-existing issue with background task execution
The test was failing because MockProcessType was missing STEP_COUNT, TOOL, and EXECUTION_LOGS attributes that are used in agent_service.py. This caused an exception in the chunk processing loop that prevented captured_final_answer from being set, which in turn prevented the background memory task from running. Added the missing attributes to MockProcessType and added a small sleep to ensure the background task has time to complete before assertions.
Resolved .gitignore conflict by keeping .tokensave entry (used for token storage)
Fixed TypeError when agent_request.history is explicitly None instead of absent. Changed getattr(agent_request, 'history', []) to handle None case properly. This bug prevented message persistence when is_debug=false, causing all event log fields (run_id, step_id, tool_call_id, event_time) to remain NULL.
…loud floating point comparison - Add 5 tests for get_message_units_by_run (with/without run_id, empty result, string coercion, dict mapping) - Add 4 tests for get_max_run_id_for_conversation (found, none, string coercion, zero edge case) - Update stubs to include run_id and step_id attributes - Fix floating point comparison in test_handlers.py using pytest.approx (SonarCloud S1244) This improves patch coverage for conversation_db.py and fixes SonarCloud reliability rating.
Add comprehensive integration test that verifies: - Real LLM model execution through the SDK - OpenTelemetry instrumentation captures spans correctly - Context management with use_context_items=True works end-to-end - LangFuse receives and displays traces properly Test includes: - test_context_items_with_real_model: Basic context management with real model - test_context_compression_with_real_model: Compression with low token threshold - test_history_projector_integration: History projection with real model Requires OPENAI_API_KEY environment variable to run.
- Fix OpenAIModel initialization (add model_id parameter) - Fix ActionStep initialization (use correct parameters) - Fix HistoryProjector initialization (use query_units_fn) - Fix ManagedContextRuntime.prepare_step call (remove conversation_id) - Add local_only marker to pytest.ini - Mark all SDK integration tests as local_only to skip in CI These tests require OPENAI_API_KEY and network access, so they should only run locally, not in CI environments.
JasonW404
changed the title
- Remove CONTEXT_MANAGEMENT_DEV_PLAN.md from git (keep locally) - Add to .gitignore to prevent future commits This is a temporary development document that should not be tracked in version control.
Jasonxia007
reviewed
Jul 8, 2026
| override_model_id: int | None = None, | ||
| requested_output_tokens: int | None = None, | ||
| tool_params: Optional[ToolParamsRequest | Dict[str, Any]] = None, | ||
| conversation_id: Optional[int] = None, |
Contributor
There was a problem hiding this comment.
conversation_id 声明可以移动到 tool_params 之前
Jasonxia007
reviewed
Jul 8, 2026
| create_config_kwargs["request_requested_output_tokens"] = requested_output_tokens | ||
|
|
||
| agent_config = await create_agent_config(**create_config_kwargs, tool_params=tool_params) | ||
| agent_config = await create_agent_config(**create_config_kwargs, tool_params=tool_params, conversation_id=conversation_id) |
Contributor
There was a problem hiding this comment.
前移conversation_id位置后，信息应纳入 create_config_kwargs 中，不需要显式声明
Jasonxia007
reviewed
Jul 8, 2026
| row_data["run_id"] = run_id | ||
| if step_id is not None: | ||
| row_data["step_id"] = step_id | ||
| if tool_call_id is not None: |
Contributor
There was a problem hiding this comment.
tool_call_id 不建议单独成列，为确保未来可扩展性建议以json形式压缩在unit_content字段中
Jasonxia007
reviewed
Jul 8, 2026
| } | ||
| if run_id is not None: | ||
| row_data["run_id"] = run_id | ||
| if step_id is not None: |
| "unit_status": unit_status, | ||
| "delete_flag": 'N', | ||
| } | ||
| if run_id is not None: |
| row_data["step_id"] = step_id | ||
| if tool_call_id is not None: | ||
| row_data["tool_call_id"] = tool_call_id | ||
| if event_time is not None: |
Contributor
There was a problem hiding this comment.
与 memory_hanlder.py 作用重复，建议删除
| dynamic_messages: Tuple[dict, ...] = () | ||
| selected_component_types: Tuple[str, ...] = () | ||
| components: Tuple[Any, ...] = () | ||
| selected_components: Tuple[Any, ...] = () |
Contributor
There was a problem hiding this comment.
此处component相关参数较多，是否存在重复
| EXTERNAL_AGENT = "external_agent" | ||
| HISTORY_TURN = "history_turn" | ||
| TOOL_CALL_RESULT = "tool_call_result" | ||
| WORKING_MEMORY = "working_memory" |
| deploy/k8s/helm/nexent/Chart.lock | ||
|
|
||
| MAC_DEVELOPMENT_GUIDE.md | ||
| CONTEXT_MANAGEMENT_DEV_PLAN.md |
…emove WorkingMemoryHandler - Merge tool+execution_logs into single tool_call row with JSON unit_content - Remove run_id from conversation_message_t and conversation_message_unit_t - Remove tool_call_id and event_time from conversation_message_unit_t - Rename step_id to step_index for clarity - Replace run_id grouping with message_id-based algorithm in HistoryProjector - Remove WorkingMemoryHandler and WORKING_MEMORY context item type - Remove resume projection from HistoryProjector - Remove dead components field from ManagedRunContext - Fix parameter ordering in create_agent_run_info() - Add TOOL_CALL handling in frontend for both streaming and history - Rewrite migration to be idempotent with proper DROP/ADD COLUMN - Update all affected tests (backend + SDK)
- Add 3 tests for tool_call merge logic in _stream_agent_chunks - Fix string-to-int type mismatch in test_conversation_db.py
…p/nexent into feat/W5_structured_agent_execution_event_log
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| }) | ||
| resp.append(combined) | ||
|
|
||
| return {"result": resp} |
| num=topk, | ||
| return_score=False | ||
| ) | ||
| return {"result": results} |
Comment on lines
+415
to
+418
| return { | ||
| "result": [], | ||
| "error": str(e) | ||
| } |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Structured Agent Execution Event Log
Overview
This PR implements structured event logging for agent execution, enabling comprehensive tracking of ReAct process persistence with detailed event logs including
run_id,step_id,tool_call_id, andevent_timefields. The implementation introduces a new ContextItem system for fine-grained context management and history projection capabilities.ContextItem System
Core Data Model
ContextItem - Unified representation for all context elements:
Policy Models:
SelectionDecision- Tracks context item selection decisions with reason codesMemoryDecision- Tracks memory inclusion decisionsReductionResult- Represents content reduction outcomesHandler System
ItemHandlerRegistry - Central registry for type-specific handlers:
to_messages()for type-specific message formatting:_ensure_handlers_registered()for performanceContextProjector
ContextProjector - Converts ContextComponent instances into ContextItem candidates:
_source_componentmetadataFeature Flag:
use_context_itemsfield inContextManagerConfig(default: False)Strategy Filtering:
selected_componentsfield inManagedRunContextHistory Projection
HistoryProjector
New class for database history projection into ContextItem format:
Projected Types:
HISTORY_TURN- User/assistant conversation turnsTOOL_CALL_RESULT- Tool execution results with UUID pairingWORKING_MEMORY- Active goals and pending tool callsProjection Modes:
model_context- For model context assemblyresume- For conversation resumptionchat- For chat history displayTool Call Pairing:
tool_call_idfor each tool invocationDatabase Schema Changes
New Migration:
v2.3.0_0703_history_projection_fields.sqlNew Columns:
conversation_message_t.run_id- Agent run sequence number (increments per agent invocation)conversation_message_unit_t.run_id- Run ID for each unitconversation_message_unit_t.step_id- ReAct step sequence number (increments on STEP_COUNT chunks)conversation_message_unit_t.tool_call_id- UUID for tool→execution_logs pairingconversation_message_unit_t.event_time- Precise event timestamp (not batch insert time)New Indexes:
idx_message_unit_conversation_run- Optimizes conversation+run queriesidx_message_unit_tool_call- Optimizes tool call pairing lookupsBackend Integration
Agent Service (
backend/services/agent_service.py)Event Log Computation:
run_idbefore streaming starts (increments per agent invocation within conversation)step_idprogression across ReAct steps (increments on STEP_COUNT chunks)tool_call_idUUIDs for tool→execution_logs pairingsave_message()andsave_message_unit()Bug Fix:
Nonehistory in agent request (TypeError: 'NoneType' object is not iterable)Conversation Management Service (
backend/services/conversation_management_service.py)New Functions:
get_message_units_by_run_id()- Retrieve message units by run IDget_max_run_id()- Get maximum run ID for a conversationExtended Signatures:
save_message()- Now acceptsrun_idparametersave_message_unit()- Now acceptsrun_id,step_id,tool_call_id,event_timeparametersDatabase Layer (
backend/database/conversation_db.py)New Query Functions:
create_conversation_message()to acceptrun_idcreate_message_unit()to accept event log fieldsSDK Changes
New Package:
sdk/nexent/core/agents/context/(17 files)Core Components:
context_item.py- ContextItem data modelprojector.py- ContextProjector implementationhistory_projector.py- HistoryProjector implementationitem_handler_registry.py- Handler registrypolicy_models.py- Policy decision modelsreason_codes.py- Reason code constantsreducer_models.py- Reduction result modelHandlers (10 files):
handlers/system_prompt_handler.pyhandlers/tool_handler.pyhandlers/skill_handler.pyhandlers/memory_handler.pyhandlers/knowledge_base_handler.pyhandlers/managed_agent_handler.pyhandlers/external_agent_handler.pyhandlers/history_turn_handler.pyhandlers/tool_call_result_handler.pyhandlers/working_memory_handler.pyModified SDK Modules (8 files)
agent_context/manager.py- OpenTelemetry instrumentation oncompress_if_needed()context_runtime/managed/runtime.py- OpenTelemetry instrumentation onprepare_step()andprepare_final_answer()agent_model.py- Addedconversation_idfield toAgentConfigsummary_config.py- Addeduse_context_itemsandhistory_projectorfieldscontext_runtime/contracts.py- Addedcontext_itemsfield toContextEvidencemonitor/monitoring.py- Extendedtrace_operation()to process input/output valuesagents/nexent_agent.py- Passesconversation_idto agent executionagents/agent_context.py- Integration with ContextItem systemOpenTelemetry Instrumentation
Added comprehensive tracing to context management operations:
New Spans:
context.compress_if_needed- Compression decision and executioncontext.prepare_step- Step preparation with context assemblycontext.prepare_final_answer- Final answer preparationcontext.history_project- History projection from databasecontext.project_items- Context item projectionSpan Attributes:
Testing
Unit Tests
New Tests:
conversation_db.pynew functions (get_message_units_by_run_id,get_max_run_id)test_handlers.py)Integration Tests
Created
test/sdk/core/agents/test_sdk_langfuse_integration.pywith real model execution:test_context_items_with_real_model- Context management with real OpenAI modeltest_context_compression_with_real_model- Compression with token threshold exceededtest_history_projector_integration- History projection with real modelNote: Integration tests require
OPENAI_API_KEYand are marked with@pytest.mark.local_onlyto skip in CI environments.Test Results
Files Changed
Total: 48 files changed, +6,178 lines, -242 lines
Backward Compatibility
use_context_itemsfeature is opt-in via configuration (default: False)