Skip to content

fix(sandbox): keep multi-line SKILL.md frontmatter descriptions in the skill index - #5028

Open
ChiFungHillmanChan wants to merge 2 commits into
openai:mainfrom
ChiFungHillmanChan:fix/skills-frontmatter-multi-line-description
Open

ChiFungHillmanChan wants to merge 2 commits into
openai:mainfrom
ChiFungHillmanChan:fix/skills-frontmatter-multi-line-description

Conversation

@ChiFungHillmanChan

Copy link
Copy Markdown

Summary

_parse_frontmatter in src/agents/sandbox/capabilities/skills.py splits every frontmatter line on the first : and keeps the rest as the value. YAML block scalars and wrapped plain scalars span more than one line, so description: > returned >, description: | returned |, and a wrapped plain scalar returned only its first line. Each continuation line was then read as a key of its own.

LocalDirLazySkillSource.list_skill_metadata and Skills._resolve_runtime_metadata both build SkillMetadata.description from this parser, and the skills section renders one line per skill as - <name>: <description> (file: <path>). A folded description therefore reached the model as - triage: >, leaving nothing to match a task against, and nothing warned. A continuation line spelled like a key also overwrote the key it collided with: a name: line inside a description replaced the skill's own name in that index line.

The diagnosis and the comparison against yaml.safe_load are the reporter's, in #5026.

This teaches the parser folded (>) and literal (|) block scalars, including their - and + chomping indicators, and wrapped plain scalars. One rule covers all three: a line indented past its key belongs to that key's value and is never read as a new key. That also keeps a nested block's keys from leaking to the top level. PyYAML stays out of the runtime dependencies.

Single-line values are unchanged, quoted or not, including leading and trailing spaces inside the quotes.

Test plan

  • Five regression cases in tests/sandbox/capabilities/test_skills_capability.py, asserting the rendered skills instructions rather than the parser: folded, literal, wrapped plain, folded with a blank line, and a folded description whose continuation line reads name:. All five fail on fbf59a40 and pass on this branch.
  • Before, on fbf59a40, the index line the model receives:
folded >                     - discovered-skill: > (file: .agents/dynamic-skill)
literal |                    - discovered-skill: | (file: .agents/dynamic-skill)
wrapped plain                - discovered-skill: Use for GitHub issue (file: .agents/dynamic-skill)
continuation holds 'name:'   - not-the-skill-name: > (file: .agents/dynamic-skill)
  • After, on this branch, same probe:
folded >                     - discovered-skill: Use for GitHub issue triage. Triggers: /triage, bug report (file: .agents/dynamic-skill)
literal |                    - discovered-skill: Use for GitHub issue triage. (file: .agents/dynamic-skill)
wrapped plain                - discovered-skill: Use for GitHub issue triage, not for PR review. (file: .agents/dynamic-skill)
continuation holds 'name:'   - discovered-skill: Use for GitHub issue triage. name: not-the-skill-name (file: .agents/dynamic-skill)
  • Compared against yaml.safe_load on the same frontmatter for the five shapes in the issue, trailing whitespace removed on both sides because this parser strips the values it returns: 3 mismatches on fbf59a40, 0 on this branch.
  • Each new case was checked against six deliberate breakages of the fix (treating | as folded, taking continuation lines at the key's own indent, folding blank lines to a space, not consuming continuation lines, dropping the wrapped plain continuation, and not removing the block indent for a literal scalar). Every one is caught.
  • .agents/skills/code-change-verification/scripts/run.sh passed: format, lint, typecheck, then 9641 passed and 29 skipped, plus 77 passed and 4 skipped serial.

What this does not change

  • A more-indented line inside a folded block is folded rather than kept literally. That YAML rule is not implemented.
  • Values stay plain strings. A key that opens a nested mapping keeps its empty value as before; its child lines are now consumed instead of leaking to the top level. Only name and description are read.
  • Anchors, aliases, flow collections, and multiple documents are still not parsed, as before.

Issue number

Closes #5026

Checks

  • I've added new tests, if relevant
  • I've run .agents/skills/code-change-verification/scripts/run.sh
  • I've confirmed all verification steps pass
  • If using Codex, I've run /review before submitting this PR

_parse_frontmatter split every frontmatter line on the first colon, so a folded
or literal block scalar returned the header character alone, a wrapped plain
scalar returned only its first line, and continuation lines became keys of
their own. Skill descriptions built from those values reached the model
truncated, and a continuation line spelled like a key could replace the skill
name in the index.

Parse block scalars and wrapped plain scalars, and treat any line indented past
its key as part of that key's value rather than as a new key.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fd4da157f8

ℹ️ About Codex in GitHub

Codex has been enabled to automatically review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

When you sign up for Codex through ChatGPT, Codex can also answer questions or update the PR, like "@codex address that feedback".

"""

index = start
while index < end and (not lines[index].strip() or _indent_of(lines[index]) > key_indent):

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Ignore indented comments when folding plain values

When a valid frontmatter comment is indented beneath a single-line field, such as name: foo followed by # explanation, this loop classifies the comment as a continuation and _fold_lines appends it to the value. The previous parser skipped that comment, but this change makes the indexed and loadable skill name foo # explanation; quoted descriptions similarly regain their surrounding quotes. Comment-only lines should be excluded when collecting plain-scalar continuations, while remaining content inside actual block scalars.

Useful? React with 👍 / 👎.

Comment on lines +467 to +468
if parsed_value in _BLOCK_SCALAR_HEADERS:
parsed_value = _join_block_lines(continuation, literal=parsed_value[0] == "|").strip()

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Treat indented --- as block-scalar content

When a folded or literal description contains an indented line whose content is ---, valid YAML treats that line as part of the scalar, but the earlier delimiter scan uses line.strip() and ends the frontmatter there. Consequently this new block-scalar branch receives only the preceding text, and subsequent metadata such as name is omitted and replaced by the directory-name fallback. Only a document delimiter at the frontmatter's delimiter indentation should terminate parsing.

Useful? React with 👍 / 👎.

A line indented past its key is part of that key's value, but a comment line is
only content inside a block scalar. Folding it into a wrapped plain value made
the skill name 'foo # explanation', and it also kept a quoted value from being
unwrapped. Drop comment lines when collecting a continuation outside a block
scalar, where YAML treats them as comments.
@ChiFungHillmanChan

Copy link
Copy Markdown
Author

Thanks — the first one is a real regression in this PR and is fixed in b16c759. The second one I do not think belongs here; evidence below.

Indented comments folded into plain values. Correct, and this PR caused it. name: foo followed by an indented # explanation returned foo # explanation, and an indented comment under a quoted value kept the quotes. Both are new: fbf59a40 returned foo and hello.

A comment line is content inside a block scalar but a comment anywhere else, which is what yaml.safe_load does:

description: |\n  text\n  # not a comment here   ->  'text\n# not a comment here\n'
name: foo\n  # explanation                       ->  {'name': 'foo'}

So comment lines are now dropped when collecting a continuation outside a block scalar, and kept inside one. Three cases added, and they separate the two rules: the two comment cases pass on fbf59a40, fail on fd4da157f8, and pass now, while the block-scalar case fails on fbf59a40 and passes on both later commits.

Indented --- ending the frontmatter. This one is real but predates the PR and is not reached through the code it touches. The delimiter scan is untouched here, and the outcome is the same before and after:

frontmatter:  description: |\n  some text\n  ---\n  more\nname: foo

fbf59a40      {'description': '|'}
fd4da157f8    {'description': 'some text'}
yaml          {'description': 'some text\n---\nmore\n', 'name': 'foo'}

name is dropped on both, so the patch does not introduce or worsen it. Fixing it properly means deciding what an indented --- means, and the obvious one-line version, requiring the closing delimiter at the opening delimiter's indentation, silently drops all metadata for any file whose closing --- is indented, which parses today. That is a separate behavioural decision, so I would rather not fold it into this PR. Happy to open it as its own issue, or to include it here if you would prefer that.

.agents/skills/code-change-verification/scripts/run.sh is green on b16c759: 9645 passed, 29 skipped, plus 77 serial.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skills frontmatter parser mangles multi-line descriptions (folded >, literal |, wrapped lines) in the skill index

1 participant