Skip to content

fix: recover empty completions and export long responses - #207

Draft
vijaymick67-rgb wants to merge 4 commits into
grinev:mainfrom
vijaymick67-rgb:feat/empty-completion-markdown-export
Draft

fix: recover empty completions and export long responses#207
vijaymick67-rgb wants to merge 4 commits into
grinev:mainfrom
vijaymick67-rgb:feat/empty-completion-markdown-export

Conversation

@vijaymick67-rgb

Copy link
Copy Markdown

Summary

  • Detect empty assistant completions at the session-idle lifecycle boundary instead of presenting them as successful runs.
  • Safely retry the original prompt once only when final text is empty, all available token/cost usage is zero, finish is unknown/invalid, and no tool or reasoning activity occurred.
  • Refuse automatic replay when work may have occurred, and stop after one retry.
  • Attach long final assistant responses as UTF-8 Markdown documents above the configurable ASSISTANT_RESPONSE_FILE_THRESHOLD (default 5000 characters).
  • Add /lastfile for the latest delivered response scoped by Telegram chat and OpenCode session, with bounded in-memory storage.

Validation

pm run build

pm run lint

pm run typecheck

pm test -- --pool=forks --maxWorkers=1 (145 files, 1558 tests)

Deployment

The currently running Telegram bot and OpenCode server were not restarted. Manual restart/deployment is required after merge.

This PR is intentionally draft and must not be auto-merged.

- Aggregate zero-work evidence across every assistant turn of an attempt,
  so an earlier tool or reasoning call or any token/cost usage blocks the
  automatic replay instead of judging only the final empty message.
- Replace the loose retryIdlePending flag with retryDispatched + idleGuard
  lifecycle state: exactly one retry, stale or duplicate session.idle events
  are consumed while the retry is in flight, and the retry completion always
  clears the state so its own idle produces the normal footer.
- Invalidate the retry state on abort, detach, session error, bot-context
  loss, session mismatch, runtime cleanup, original/retry API failure, and
  replacement by a newer prompt; retry API failure restores the idle state
  and resumes the prompt queue.
- Store only the terminal successfully delivered response for /lastfile and
  the automatic Markdown export, committing at session idle; failed, empty,
  or intermediate responses never replace the previous last good one.
- Bind retry and export ownership to the originating Telegram chat.
- Add an assistant-message-started signal to the aggregator and only treat a
  completed message as a terminal final-response candidate when it survived
  with no upstream error, performed no tool activity, and did not end on a
  known non-terminal finish reason. Intermediate commentary and truncated or
  aborted output are still streamed to the user but never committed.
- Invalidate a pending final-response candidate whenever a newer assistant
  message starts, so commentary followed by more tool work never becomes
  /lastfile, an automatic Markdown export, or evidence of a successful task.
- Gate the assistant-run footer on an actually committed terminal response
  instead of any completed message, so a run that ends without a deliverable
  is never announced as complete.
- Harden stale-idle ordering across the automatic retry with an explicit
  response-delivered phase and a settle timer: every session.idle, including a
  stale duplicate of the original attempt's idle, is consumed while the retry
  state is alive, and the retry is finalized exactly once by its own settle
  rather than by the first idle that happens to arrive.
- Keep the retry idle guard off until the retry is actually dispatched so a
  registered-but-unreplayed attempt never swallows a normal run's idle.
- Bind the committed /lastfile, automatic Markdown export, success footer, and
  TTS reply to the originating Telegram chat preserved for the prompt instead
  of the mutable service-wide chat context.
@vijaymick67-rgb

Copy link
Copy Markdown
Author

Reliability amendment: terminality, intermediate invalidation, stale-idle hardening, chat ownership

1. Previous SHA

ba5e2dcc88580a5efae37654cea18a2b39fe0201

2. New SHA

bc798d460c2e662f6ee7989b5a22dc661c6e76cc (new commit fix: never report intermediate or truncated runs as complete)

3. Exact lifecycle fixes

  • Terminality gate (isTerminalAssistantResponse): a completed non-empty assistant message is a final-response candidate only if it carried no upstream error, performed no tool activity (a message that called a tool is never a closing answer), and did not end on a known non-terminal finish reason (max_tokens, length, error, content_filter, tool_use, function_call, aborted, incomplete). Unknown/missing finish stays fail-open.
  • Candidate invalidation: a new assistant-message-started signal (routed through the serialized completion-task queue for correct ordering) invalidates a pending candidate whenever a newer message starts, proving the earlier one was intermediate.
  • Footer gating: the assistant-run footer is emitted only when a terminal response was actually committed (commit returns the chatId), never just because hasCompletedResponse was set by an intermediate message.
  • Stale-idle hardening: the retry state gained an explicit retryResponseDelivered phase plus a settle timer. Every session.idle is consumed while the retry lifecycle is alive, and the retry is finalized exactly once by its own settle window, so a stale/duplicate idle from the original attempt cannot finish the retry early, emit a footer, commit /lastfile, export Markdown, mark the foreground idle, or dispatch queued prompts.
  • Guard scope fix: the idle guard only activates once the retry is actually dispatched, so a registered-but-unreplayed attempt never swallows a normal run's idle.
  • Chat ownership: the committed /lastfile, Markdown export, success footer, and TTS reply are bound to the originating Telegram chat captured for the prompt (getPromptRetryChatId ?? chatIdInstance), not the mutable service-wide context.

4. How terminal success is now established

At session idle the pending final-response candidate is committed only when it (a) completed, (b) is non-empty, (c) has no upstream error, (d) performed no tool activity, (e) was not superseded by a newer assistant message, and (f) the run's hasCompletedResponse is set. Only then are /lastfile, the Markdown export, and the success footer produced. Everything else fails closed and preserves the previous last good response.

5. How intermediate candidates are invalidated

Any new assistant message start invalidates the pending candidate of a different messageId, and an errored/tool-bearing completion never establishes a candidate. The invalidation is serialized through enqueueSessionCompletionTask so event ordering is preserved.

6. How stale original idle is distinguished from retry idle

The original attempt's idle that triggered the retry is consumed by the guard; while the retry state is alive all idles are consumed. Once the retry delivers its terminal response, retryResponseDelivered arms a 300ms settle timer that is reset by every consumed idle and every new message start. The settle timer — not the idle — is the single finalizer, so a stale duplicate idle arriving after the response is inert and the real retry finalize happens exactly once.

7. Chat ownership behavior

Origin chat preserved per prompt (registerPromptRetry.chatId). Retry notifications already used it; now the committed response, export, footer, and TTS use it too. Non-prompt (tracked) sessions fall back to the current service context.

8. Tests added/updated

  • empty-completion-policy.test.ts: terminal-response detection suite.
  • event-subscription-service.test.ts new final response terminality describe: intermediate commentary + later tool activity (no footer/export, /lastfile preserved); truncated/errored completion (no footer, /lastfile preserved); normal short terminal response (one footer, /lastfile updated, no document); normal long terminal response (one footer, /lastfile updated, one document); retry ordering original empty -> retry response -> stale original idle -> real retry idle (stale idle inert, finalize exactly once); originating-chat ownership when the mutable context changes mid-run.
  • Updated the existing successful-retry footer test to await the settle finalize (assertion unchanged).

9. Validation

  • npm run build
  • npm run lint
  • npm run typecheck
  • npm test -- --pool=forks --maxWorkers=1 ✅ 145 files / 1590 tests passed

10. Files changed

src/app/managers/summary-aggregation-manager.ts, src/bot/handlers/prompt.ts, src/bot/services/empty-completion-policy.ts, src/bot/services/event-subscription-service.ts, tests/bot/services/empty-completion-policy.test.ts, tests/bot/services/event-subscription-service.test.ts (6 files, +613/-84).

11. Remaining uncertainty

  • Session idle events carry no attempt identity, so the retry finalize uses a short settle window (300ms) as the robust discriminator; a multi-message retry whose first clean message is followed by more output defers via the message-start reset.
  • Non-prompt (TUI-tracked) sessions fall back to the current chat context because no originating chat is preserved for them.

12. Explicit confirmation

No merge performed. No Telegram bot or OpenCode server restarted. No .env/secrets touched. Existing PR #207 updated only (head branch feat/empty-completion-markdown-export).

- Terminality now fails closed: only the OpenCode FinishReason value
  "stop" (the sole successful terminal value in the @opencode-ai/llm
  literal schema) qualifies a non-empty assistant message as a final
  response. Missing, unknown, or arbitrary finish reasons are never
  treated as terminal success, and the per-step step-finish reason is no
  longer substituted for the authoritative message-level info.finish.
- Remove the 300ms retry settle timer heuristic. A retry is finalized
  only when a session.idle is confirmed against the authoritative OpenCode
  session status: missing/"idle" means genuinely finished, while
  "busy"/"retry" (or a failed or ambiguous lookup) treats the idle as
  stale and consumes it without finalizing anything. Exactly one automatic
  retry remains the maximum.
@vijaymick67-rgb

Copy link
Copy Markdown
Author

Surgical fix: explicit terminal finish allowlist + authoritative retry idle

1. Previous SHA

bc798d460c2e662f6ee7989b5a22dc661c6e76cc

2. New SHA

35ac9559ee3fd33afe8d46b0cadd045dd9979be8 (commit fix: require explicit terminal finish and authoritative retry idle)

3. Successful terminal finish reasons now accepted, and where they came from

Only "stop". Source: the @opencode-ai/llm FinishReason literal schema (packages/llm/src/schema/ids.ts) is ["stop", "length", "tool-calls", "content-filter", "error", "unknown"]. OpenCode's session/llm/ai-sdk.ts also maps any unrecognized AI SDK reason to "unknown", and message-v2.ts treats an assistant message as finished only when info.finish is set and !info.error. "stop" is the sole value meaning the model ended its turn with a successfully delivered answer.

4. How ambiguous finish reasons now fail closed

isTerminalAssistantResponse uses a positive allowlist. undefined, "unknown", and any arbitrary/unexpected string return false, as do all known non-terminal values (length, tool-calls, content-filter, error, and reviewer-flagged names like max_tokens, aborted, tool_use, function_call, incomplete). The aggregator no longer substitutes a per-step step-finish reason for the authoritative message-level info.finish; a missing info.finish now yields finishReason: undefined and fails closed.

5. What replaced the 300ms timer

The RETRY_SETTLE_MS settle timer, armRetrySettle, setOnRetrySettle, and resetPromptRetrySettle are removed. Retry finalization is now driven by decidePromptRetryIdle, which queries the authoritative OpenCode session state (session.status) when an idle arrives during a retry after a terminal response. Positive proof of idleness is required; no arbitrary delay is used.

6. How stale original idle is distinguished from genuine retry completion

OpenCode keeps a session in its active status map while busy and deletes it when it goes idle (publishing session.idle at the same time); SessionStatus.get() defaults a missing session to "idle". So: session.status returns the retry as "busy"/"retry" → the idle is stale and consumed (nothing finalized); the session is missing or "idle" → genuine retry idle → finalize exactly once.

7. Behavior when authoritative status lookup fails

A failed or ambiguous lookup (error, !data, or an unexpected status type) returns "unknown" and the idle is consumed: no footer, no /lastfile, no Markdown export, foreground stays busy, queue not dispatched. Success is never finalized without positive evidence. A later idle with authoritative "idle" status still finalizes exactly once.

8. Tests added/changed

  • empty-completion-policy.test.ts: terminality now asserts "stop" is terminal; undefined, "unknown", arbitrary, empty, and every known non-terminal reason (including max_tokens, aborted, tool_use, function_call, incomplete) are non-terminal.
  • event-subscription-service.test.ts: session.status is now a controllable mock; the emitAssistantCompleted helpers emit the authoritative finish: "stop"; the stale-idle retry test configures busy/idle status per idle. New tests: non-empty response with missing/unknown finish → no Task complete, no /lastfile replacement, no export; stale original idle while status is busy, then a wait longer than the old 300ms window → still inert, then genuine idle → finalize exactly once (one footer, correct /lastfile, one export, queue resumed once); status-lookup error and unexpected status type → fail closed, then genuine idle finalizes.
  • event-subscription-service.lifecycle.test.ts: completion helper emits finish: "stop".

9. Validation

  • npm run build
  • npm run lint
  • npm run typecheck
  • npm test -- --pool=forks --maxWorkers=1 ✅ 145 files / 1595 tests passed

10. Files changed

src/app/managers/summary-aggregation-manager.ts, src/bot/handlers/prompt.ts, src/bot/services/empty-completion-policy.ts, src/bot/services/event-subscription-service.ts, tests/bot/services/empty-completion-policy.test.ts, tests/bot/services/event-subscription-service.lifecycle.test.ts, tests/bot/services/event-subscription-service.test.ts (7 files, +333/-173).

11. Remaining uncertainty

  • A genuine retry idle whose session.status lookup transiently fails is consumed (fail closed); it only finalizes on a later idle with a successful idle-status lookup. If OpenCode emits no further idle and the status API stays down, the retry stays busy rather than reporting false success.
  • session.status reflects OpenCode's in-memory active-status map; it is queried with the retry's original directory.

12. Explicit confirmation

No merge performed. No Telegram bot or OpenCode server restarted. No .env/secrets touched. Existing PR #207 updated only (head branch feat/empty-completion-markdown-export).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant