DOC Document attack outcomes and scenario resiliency

doc/blog/2026_04_14_scoring_scorers.md

@@ -47,6 +47,9 @@ The `eval_hash` allows us to easily look up metrics associated with a specific s
 
 Here is a diagram of the full end-to-end process of our scorer evaluation framework.
 
+:::{div}
+:class: col-page-right
+
 ```{mermaid}
 flowchart TB
  subgraph INPUT["📁 Input: Human-Labeled CSV Datasets"]
@@ -97,6 +100,7 @@ flowchart TB
     FIND --> PRINTER
     FIND --> BEST
 ```
+:::
 
 1. It begins by loading human-labeled datasets from saved `.csv` files, parsing version metadata, and creating a `HumanLabeledDataset` object that can be ingested by our evaluation methods.
 2. The second step builds the scorer evaluation identifier from the scoring configuration; we can optionally check whether an entry already exists in our JSONL-formatted metrics registry by checking the eval hash.

doc/code/executor/2_multi_turn.ipynb

@@ -19,7 +19,19 @@
     "The adaptive attacks below take two targets: the `objective_target` (the system under test) and an\n",
     "`AttackAdversarialConfig` naming the **adversarial target**. The adversarial target works best\n",
     "**without** content moderation, so it doesn't refuse to generate adversarial prompts. The fixed-script\n",
-    "and streaming attacks (Multi-Prompt Sending, Chunked Request, Barge-In) use only the objective target.\n",
+    "and streaming attacks (Multi-Prompt Sending, Chunked Request, Barge-In) use only the objective target."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "1",
+   "metadata": {
+    "class": "col-page-right"
+   },
+   "source": [
+    "\n",
+    ":::{div}\n",
+    ":class: col-page-right\n",
     "\n",
     "```{mermaid}\n",
     "flowchart LR\n",
@@ -30,6 +42,14 @@
     "    decision -- Yes --> done(\"Done\")\n",
     "    decision -- No --> getPrompt\n",
     "```\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "2",
+   "metadata": {},
+   "source": [
     "\n",
     "| Attack | What it does |\n",
     "|---|---|\n",
@@ -48,7 +68,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "1",
+   "id": "3",
    "metadata": {},
    "outputs": [
     {
@@ -92,7 +112,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "2",
+   "id": "4",
    "metadata": {},
    "source": [
     "## Red Teaming\n",
@@ -104,7 +124,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "3",
+   "id": "5",
    "metadata": {},
    "outputs": [
     {
@@ -258,7 +278,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "4",
+   "id": "6",
    "metadata": {},
    "source": [
     "## Crescendo\n",
@@ -270,7 +290,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "5",
+   "id": "7",
    "metadata": {},
    "outputs": [
     {
@@ -577,7 +597,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "6",
+   "id": "8",
    "metadata": {},
    "source": [
     "## Tree of Attacks with Pruning (TAP)\n",
@@ -590,7 +610,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "7",
+   "id": "9",
    "metadata": {},
    "outputs": [
     {
@@ -720,7 +740,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "8",
+   "id": "10",
    "metadata": {},
    "source": [
     "## Multi-Prompt Sending\n",
@@ -733,7 +753,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "9",
+   "id": "11",
    "metadata": {},
    "outputs": [
     {
@@ -809,7 +829,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "10",
+   "id": "12",
    "metadata": {},
    "source": [
     "## Chunked Request\n",
@@ -821,7 +841,7 @@
   {
    "cell_type": "code",
    "execution_count": null,
-   "id": "11",
+   "id": "13",
    "metadata": {},
    "outputs": [
     {
@@ -929,7 +949,7 @@
   },
   {
    "cell_type": "markdown",
-   "id": "12",
+   "id": "14",
    "metadata": {},
    "source": [
     "## Barge-In (streaming)\n",
@@ -960,7 +980,7 @@
  ],
  "metadata": {
   "jupytext": {
-   "cell_metadata_filter": "-all",
+   "cell_metadata_filter": "class,-all",
    "main_language": "python"
   },
   "language_info": {

doc/code/executor/2_multi_turn.py

@@ -25,6 +25,11 @@
 # `AttackAdversarialConfig` naming the **adversarial target**. The adversarial target works best
 # **without** content moderation, so it doesn't refuse to generate adversarial prompts. The fixed-script
 # and streaming attacks (Multi-Prompt Sending, Chunked Request, Barge-In) use only the objective target.
+
+# %% [markdown] class="col-page-right"
+#
+# :::{div}
+# :class: col-page-right
 #
 # ```{mermaid}
 # flowchart LR
@@ -35,6 +40,9 @@
 #     decision -- Yes --> done("Done")
 #     decision -- No --> getPrompt
 # ```
+# :::
+
+# %% [markdown]
 #
 # | Attack | What it does |
 # |---|---|

doc/code/framework.md

@@ -52,6 +52,9 @@ The main components of PyRIT are seeds, scenarios, attack techniques, executors
 
 The diagram below shows how the pieces fit together: entry points run **scenarios**, which package **datasets** with **attack techniques**; each technique drives an **attack/executor** that orchestrates **converters**, **targets**, and **scorers**; and a shared library layer (**memory**, **registry**, **models**, **output**, and more) supports all of them.
 
+:::{div}
+:class: col-page-right
+
 ```mermaid
 flowchart TB
     subgraph entry [Entry points]
@@ -105,6 +108,7 @@ flowchart TB
     class SCEN,TECH,ATK flow;
     class MEM,REG,MODEL,OUT libnode;
 ```
+:::
 
 The orchestration layers **nest from broadest to narrowest** — each owns less than the layer above it:
 

doc/code/memory/9_schema_diagram.md

@@ -2,6 +2,9 @@
 
 Our memory contains multiple components. This diagram  shows a mapping of our database schema and how our components map together! The arrows indicate the values that map one database to another.
 
+:::{div}
+:class: col-page-right
+
 ```{mermaid}
 flowchart LR
  subgraph EmbeddingData["EmbeddingData"]
@@ -74,3 +77,4 @@ flowchart LR
     linkStyle 0 stroke:#ff8800ff,fill:none
     linkStyle 1 stroke:#14a519ff
 ```
+:::

doc/code/scenarios/0_attack_techniques.ipynb

@@ -611,7 +611,19 @@
     "On the command line this is the `--technique` flag of\n",
     "[`pyrit_scan`](../../scanner/1_pyrit_scan.ipynb); programmatically it's the `scenario_techniques`\n",
     "argument to `initialize_async`. The grouping is what lets `--technique single_turn` or\n",
-    "`--technique light` fan out to a whole family of techniques without naming each one.\n",
+    "`--technique light` fan out to a whole family of techniques without naming each one."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6",
+   "metadata": {
+    "class": "col-page-right"
+   },
+   "source": [
+    "\n",
+    ":::{div}\n",
+    ":class: col-page-right\n",
     "\n",
     "```mermaid\n",
     "flowchart LR\n",
@@ -620,12 +632,13 @@
     "    S -->|name / tag / composite| Sc[\"Scenario\"]\n",
     "    R -->|create with target + scorer| T[\"AttackTechnique<br/>(attack + seeds)\"]\n",
     "    Sc --> T\n",
-    "```"
+    "```\n",
+    ":::"
    ]
   },
   {
    "cell_type": "markdown",
-   "id": "6",
+   "id": "7",
    "metadata": {},
    "source": [
     "## Relationship to single-turn attacks\n",

doc/code/scenarios/0_attack_techniques.py

@@ -187,6 +187,11 @@
 # [`pyrit_scan`](../../scanner/1_pyrit_scan.ipynb); programmatically it's the `scenario_techniques`
 # argument to `initialize_async`. The grouping is what lets `--technique single_turn` or
 # `--technique light` fan out to a whole family of techniques without naming each one.
+
+# %% [markdown] class="col-page-right"
+#
+# :::{div}
+# :class: col-page-right
 #
 # ```mermaid
 # flowchart LR
@@ -196,6 +201,7 @@
 #     R -->|create with target + scorer| T["AttackTechnique<br/>(attack + seeds)"]
 #     Sc --> T
 # ```
+# :::
 
 # %% [markdown]
 # ## Relationship to single-turn attacks

doc/code/scenarios/0_scenarios.ipynb

@@ -465,6 +465,114 @@
     "\n",
     "Scenarios can run for a long time, and because of that, things can go wrong. Network issues, rate limits, or other transient failures can interrupt execution. PyRIT provides built-in resiliency features to handle these situations gracefully.\n",
     "\n",
+    "### Attack Outcomes and Execution Health\n",
+    "\n",
+    "A Scenario tracks two independent axes for every objective:\n",
+    "\n",
+    "| Axis | Values | Meaning |\n",
+    "| --- | --- | --- |\n",
+    "| **Execution health** | completed or incomplete | A completed objective returned an `AttackResult`. An incomplete objective raised an exception before it could return one. |\n",
+    "| **Objective outcome** | `AttackOutcome.SUCCESS`, `FAILURE`, or `UNDETERMINED` | Whether a completed attack achieved its objective. `FAILURE` is a valid security result, not an execution error. |\n",
+    "\n",
+    "A model refusal therefore does not make an objective incomplete. PyRIT persists handled structured\n",
+    "refusals and content-filter responses as blocked model responses, applies the configured scoring\n",
+    "policy, and returns a completed `AttackResult`. A refusal that does not achieve the objective normally\n",
+    "produces `AttackOutcome.FAILURE`; a response that achieves it produces `AttackOutcome.SUCCESS`."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "6",
+   "metadata": {
+    "class": "col-page-right"
+   },
+   "source": [
+    "\n",
+    ":::{div}\n",
+    ":class: col-page-right\n",
+    "\n",
+    "```{mermaid}\n",
+    "%%{init: {\"flowchart\": {\"subGraphTitleMargin\": {\"bottom\": 40}, \"wrappingWidth\": 260}}}%%\n",
+    "flowchart TB\n",
+    "    subgraph objective[\"One objective in<br/>an AtomicAttack\"]\n",
+    "        START[\"Execute attack objective\"] --> TARGET[\"Send or continue conversation\"]\n",
+    "        TARGET --> TARGET_RESULT{\"Target result\"}\n",
+    "\n",
+    "        TARGET_RESULT -->|Normal model output| RESPONSE[\"Persistable model response\"]\n",
+    "        TARGET_RESULT -->|Handled refusal or<br/>content-filter response| REFUSAL[\"Persistable blocked model response<br/>not an execution failure\"]\n",
+    "        TARGET_RESULT --> RUNTIME_ERROR[\"Non-retryable runtime error\"]\n",
+    "        TARGET_RESULT -->|Retryable target error| TARGET_RETRY{\"Target retry budget remains?\"}\n",
+    "        TARGET_RETRY -->|No / exhausted| EXEC_ERROR[\"Execution exception propagates\"]\n",
+    "        TARGET_RETRY -->|Yes| RETRY_TARGET[\"Repeat from<br/>Send or continue conversation\"]\n",
+    "        RUNTIME_ERROR --> EXEC_ERROR\n",
+    "\n",
+    "        RESPONSE --> SCORE[\"Apply configured scorer policy\"]\n",
+    "        REFUSAL --> SCORE\n",
+    "        SCORE --> SCORE_RESULT{\"Scoring result\"}\n",
+    "        SCORE_RESULT -->|Objective not achieved| MORE{\"Attack-specific attempt or turn remains?\"}\n",
+    "        SCORE_RESULT -->|Objective achieved| SUCCESS[\"AttackResult<br/>AttackOutcome.SUCCESS\"]\n",
+    "        SCORE_RESULT -->|No objective scorer| UNDETERMINED[\"AttackResult<br/>AttackOutcome.UNDETERMINED\"]\n",
+    "        SCORE_RESULT -->|Invalid JSON;<br/>retry remains| RETRY_SCORE[\"Repeat from<br/>Apply configured scorer policy\"]\n",
+    "        SCORE_RESULT -->|Scorer error or<br/>out of retries| EXEC_ERROR\n",
+    "        MORE -->|Yes| RETRY_ATTACK[\"Repeat from<br/>Send or continue conversation\"]\n",
+    "        MORE -->|No| FAILURE[\"AttackResult<br/>AttackOutcome.FAILURE\"]\n",
+    "\n",
+    "        FAILURE --> COMPLETE[\"Completed objective\"]\n",
+    "        SUCCESS --> COMPLETE\n",
+    "        UNDETERMINED --> COMPLETE\n",
+    "        EXEC_ERROR --> ERROR_ROW[\"Error handler may persist<br/>AttackOutcome.ERROR for diagnostics\"]\n",
+    "        ERROR_ROW --> INCOMPLETE[\"Incomplete objective<br/>exception retained\"]\n",
+    "    end\n",
+    "\n",
+    "    subgraph aggregation[\"Scenario aggregation<br/>and resiliency\"]\n",
+    "        COMPLETE --> EXECUTOR_RESULT[\"AttackExecutorResult\"]\n",
+    "        INCOMPLETE --> EXECUTOR_RESULT\n",
+    "        EXECUTOR_RESULT --> HAS_INCOMPLETE{\"Any incomplete objectives?\"}\n",
+    "\n",
+    "        HAS_INCOMPLETE -->|Yes| SCENARIO_RETRY{\"Scenario retry budget remains?\"}\n",
+    "        SCENARIO_RETRY -->|Yes; resume only<br/>incomplete objectives| RESUME[\"Repeat objective flow<br/>for incomplete objectives\"]\n",
+    "        SCENARIO_RETRY -->|No / exhausted| PARTIAL[\"Raise ScenarioPartialFailureException<br/>structured counts, incomplete objectives, preserved cause<br/>completed_count may be zero\"]\n",
+    "        PARTIAL --> SCENARIO_FAILED[\"Persist ScenarioRunState.FAILED\"]\n",
+    "\n",
+    "        HAS_INCOMPLETE -->|No| KEEP[\"Keep every completed AttackResult<br/>SUCCESS, FAILURE, and UNDETERMINED\"]\n",
+    "        KEEP --> ALL_DONE{\"All atomic attacks complete?\"}\n",
+    "        ALL_DONE -->|No| NEXT_ATTACK[\"Repeat objective flow<br/>for next atomic attack\"]\n",
+    "        ALL_DONE -->|Yes| SCENARIO_COMPLETE[\"ScenarioResult<br/>ScenarioRunState.COMPLETED\"]\n",
+    "    end\n",
+    "\n",
+    "    classDef model fill:#e8f0fe,stroke:#4285f4,color:#15233a;\n",
+    "    classDef complete fill:#e6f4ea,stroke:#34a853,color:#15233a;\n",
+    "    classDef incomplete fill:#fce8e6,stroke:#d93025,color:#15233a;\n",
+    "    classDef retry fill:#fff4e5,stroke:#f9ab00,color:#15233a;\n",
+    "    class RESPONSE,REFUSAL model;\n",
+    "    class RETRY_TARGET,RETRY_SCORE,RETRY_ATTACK,RESUME,NEXT_ATTACK retry;\n",
+    "    class SUCCESS,FAILURE,UNDETERMINED,COMPLETE,SCENARIO_COMPLETE complete;\n",
+    "    class RUNTIME_ERROR,EXEC_ERROR,ERROR_ROW,INCOMPLETE,PARTIAL,SCENARIO_FAILED incomplete;\n",
+    "```\n",
+    ":::"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "id": "7",
+   "metadata": {},
+   "source": [
+    "\n",
+    "To keep retry paths readable, **Repeat from** nodes name the earlier step where execution resumes\n",
+    "instead of drawing long return arrows across unrelated branches.\n",
+    "\n",
+    "A Scenario reaches `ScenarioRunState.COMPLETED` when every objective execution completes, regardless\n",
+    "of the mix of successful and unsuccessful attack outcomes. Scenario retries resume only objectives\n",
+    "that have not completed; already-persisted results are preserved.\n",
+    "\n",
+    "If retry exhaustion leaves any incomplete objectives, `ScenarioPartialFailureException` reports\n",
+    "`completed_count`, `incomplete_count`, and `incomplete_objectives`, and keeps the first objective\n",
+    "exception as its cause. This typed exception is also used when **none** of the objectives in the\n",
+    "returned `AttackExecutorResult` completed (`completed_count == 0`). If an `AtomicAttack` raises before\n",
+    "it can return an `AttackExecutorResult`, the Scenario instead retries and ultimately re-raises that\n",
+    "exception; multiple concurrent atomic-attack failures are surfaced as an `ExceptionGroup`. In every\n",
+    "terminal execution-failure case, the persisted Scenario state is `ScenarioRunState.FAILED`.\n",
+    "\n",
     "### Automatic Resume\n",
     "\n",
     "If you re-run a `scenario`, it will automatically start where it left off. The framework tracks completed attacks and objectives in memory, so you won't lose progress if something interrupts your scenario execution. This means you can safely stop and restart scenarios without duplicating work.\n",