feat: add agent scorer authoring @W-23997055 - #317
Conversation
|
Nice work getting this out — the shape is right and the XML matches what the org expects. A few things I'd like to see addressed before we approve; the top three are library-level correctness issues that let Blocking
validateScorerSpec({ dataType: 'Number', ...no specification... }) → no throw buildScorerXml then produces a Number scorer with zero entries and no block. Add: if (spec.dataType === 'Number' && !spec.specification) {
validateScorerSpec({ dataType: 'LightningType', lightningType: 'bogus__type', ... }) → no throw The CLI's JSON schema catches this today, but library-only callers get no protection. Please cross-check against the const you already export.
The Number branch always regenerates enum values from specification and ignores anything the caller passed in outputEnumValues. Either reject the combination in validateScorerSpec (mutually exclusive) or honor Should fix
Nice-to-have
|
|
Thanks for the contribution! It looks like @shanile50 is an internal user so signing the CLA is not required. However, we need to confirm this. |
nabilnaffar-sf
changed the title
Route OpenEnded scorers to the NGT (core) scorer-open-* prompt bodies so
the default prompt content matches what the NGT UI shows:
- text / number: labeled variant when the scorer has predefined values
(outputEnumValues), plain variant otherwise
- boolean / url / date: their dedicated resource bodies
The transcript is referenced via {!$Input:Session} (core pipes the same
Session input through a getSession data action; same meaning, and it keeps
the prompt consistent with the inputs the template declares). Shared JSON
result block + transcript footer are factored into a helper to avoid
duplication across the seven templates. The legacy Predefined
measurement/multilabel branches are left untouched since the authoring
flow only ever creates OpenEnded scorers.
Update unit tests to cover the new per-type routing.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Replace the seven per-type OpenEnded prompt constants (and
resolveOpenEndedCategory) with a single generic SCORER_PROMPT template
whose output shape is driven by the type's JSON Schema, substituted into
{!$OutputSchema}, plus per-scorer guidance in {!$Instructions}.
schemaFor() is now the only place that knows about a given type, so
supporting new/custom Lightning Types is a follow-up that touches only
schemaFor() with no prompt changes. This intentionally diverges from the
NGT UI's per-type prose in favour of extensibility.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
generateNumberEnumValues was exported and unit-tested but had no caller: Number scorers emit a <specification> (min/max/step/threshold) that the backend applies at runtime, and validateScorerSpec forbids outputEnumValues for Number, so the helper was never invoked. Its threshold-agnostic all-NotApplicable output was also misleading. Remove the function, its export, and its tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Keep the branch lint-clean ahead of review. No behavior change: - split validateScorerSpec into validateNumberScorer / validateLightningTypeScorer to drop its cyclomatic complexity under the limit (was 30); - add an explicit `case undefined` so the lightningType switch is exhaustive over `string | undefined`; - demote two narrative block comments from /** to /* (they carry no JSDoc tags) so jsdoc/check-indentation stops flagging their aligned bullet lists.
…upport as default
…upport as default
…upport as default
5795033 to
2b794de
Compare
# Conflicts: # yarn.lock
Remove the scorer mutation/update surface from the lib (version-append, status, and agent-association mutators). Once an AiAgentScorerDefinition XML is written it is the source of truth; any further change is authored directly in the XML. Version parsing and read-time version selection are unchanged. Update doc comments and the not-found error to reference the renamed CLI command 'sf agent scorer generate-metadata-file' (was 'create').
… validation Documentation (single source of truth: ScorerSpec/AgentAssociation JSDoc → generated spec schema): - Explain what each status means and how to author it: Draft = still developing (inner loop), runnable ad-hoc but not activatable; Available = validated/ready, runnable ad-hoc and eligible for automatic production scoring. - Clarify isActive (automatic production scoring, sampled by samplingRate; requires Available) vs ad-hoc runs, and that samplingRate governs automatic sampling only. - STDM session view: require every timestamp to be UTC ending in a literal '+0000' offset (reject 'Z', colon offsets, and non-UTC offsets) via a shared IsoTimestamp type carrying the description and @pattern. Behavior: - Fail a run on a blank/score-less generation (empty completion, or a parseable envelope with empty/all-null outputs and no legacy output) instead of reporting a passing empty score. - Validate apiName presence, engineType, and agentAssociation presence with actionable messages instead of raw TypeErrors. - Ad-hoc run selection falls back to the highest non-Archived (Draft) version when no Available version exists, so a freshly-scaffolded scorer runs without --scorer-version; only throws when every version is Archived. Activation still requires Available.
5 participants
What does this PR do?
Add Agent Scorer authoring & run support
What issues does this PR fix or reference?
Introduces a client-side API for authoring, versioning, and running Agent Scorers — the evaluators that grade Agentforce agent sessions. Consumers (plugin-agent CLI, vscode-agents) get a single, typed entry point that turns a
ScorerSpecinto deployable metadata and runs a scorer against an STDM session, without each repo re-implementing the XML/prompt plumbing.What's included
createScorerDefinition(spec, { outputDir, write })generatesaiAgentScorerDefinitionand (when needed)genAiPromptTemplatemetadata from a typedScorerSpec. Default prompt content is derived from the scorer's output JSON Schema and aligned with the NGT templates.addScorerVersion,setScorerVersionStatus, and activation toggling (setVersionAssociationActiveInScorerXml) let callers add versions and edit status/activation on existing scorer XML.runScorer(spec, session, connection)runs a scorer against a normalized STDMSessionViewand returns aScorerResult, dispatched through a pluggable engine registry (registerEngine/getEngine), withGenerationsandPromptTemplateengines provided.validateScorerSpec, API-name helpers (isValidScorerApiName,labelToApiName), plus exported JSON Schemas (SCORER_SPEC_JSON_SCHEMA,SESSION_VIEW_JSON_SCHEMA) and ascripts/gen-scorer-schema.mjsgenerator.ScorerSpec, engine/status/outcome/input-scope enums, session view model) re-exported from the package root.