FIX Bind doc notebook execution to the invoking checkout's Jupyter kernel

.github/instructions/docs.instructions.md

@@ -51,15 +51,23 @@ When making changes:
 
 Before running `jupytext --execute`, make sure the kernel will exercise *the code in this checkout*, not some stale install:
 
-1. **Use a kernel bound to a Python env that has this worktree installed editable.**
-   Reusing an existing `pyrit` kernel is fine *only if* it points at the current
-   checkout — otherwise it will resolve imports against an unrelated copy and
-   either pass on stale code or fail on missing new symbols.
-   - Quick check: `python -c "import pyrit, pathlib; print(pathlib.Path(pyrit.__file__).resolve())"`
-   - If it doesn't match this worktree, install editable here: `pip install -e .`
-     (this rebinds the existing kernel to this checkout, no new kernel needed).
-   - Only create a new kernel (`python -m ipykernel install --user --name <name>`)
-     if you actually need an isolated env.
+1. **Run jupytext through this checkout's environment and pin the venv-local kernel.**
+   Use `uv run` so the command inherits this worktree's `.venv`, and pass `--set-kernel python3`,
+   the kernel that `uv sync` installs inside `.venv`:
+   ```bash
+   uv run jupytext --to ipynb --execute --set-kernel python3 doc/path/to/your_notebook.py
+   ```
+   - Quick check: `uv run python -c "import pyrit, pathlib; print(pathlib.Path(pyrit.__file__).resolve())"`
+     should print a path inside this checkout.
+   - Do **not** select a fixed machine-wide kernel name such as `pyrit-dev`. Those are installed
+     with `--user`, so a single name is shared by every clone and worktree on the machine and
+     resolves to whichever one registered it last. The notebook then runs against unrelated
+     code and silently produces wrong output.
+   - Do **not** use `--set-kernel -`. It matches kernels by comparing `argv[0]` to the running
+     interpreter, which fails against the correct relocatable kernelspec (`argv[0] == "python"`)
+     that `uv sync` installs.
+   - If you genuinely need an isolated named kernel, scope it to the virtual environment:
+     `uv run python -m ipykernel install --sys-prefix --name <name>`.
 2. **Credentials must be pre-configured.** Most notebooks call live targets
    (OpenAI, Azure, etc.) and load creds from `~/.pyrit/.env`. Make sure the
    required keys are present before executing.
@@ -76,12 +84,12 @@ to surface in review; don't paper over it by committing an output-less notebook.
 
 Generate .ipynb from .py (with execution — if it fails it means there are errors):
 ```bash
-jupytext --to ipynb --execute doc/path/to/your_notebook.py
+uv run jupytext --to ipynb --execute --set-kernel python3 doc/path/to/your_notebook.py
 ```
 
 Generate .py from .ipynb:
 ```bash
-jupytext --to py:percent doc/path/to/notebook.ipynb
+uv run jupytext --to py:percent doc/path/to/notebook.ipynb
 ```
 
 If a `doc/**/*.py` notebook fails during `jupytext --execute` with errors that look like uninitialized state (missing env vars, undefined names, `initialize_pyrit_async` apparently not run, failing cell shows `Cell In[1]` despite earlier code), **check the `# %%` cell separators in the .py file first**.

doc/generate_docs/pct_to_ipynb.py

@@ -36,7 +36,14 @@
 file_type = ".py"
 included_dirs = {"code"}
 cache_dir = os.path.join(pyrit_root, "dbdata")
-kernel_name = "pyrit-dev"
+# "python3" is the kernel that `uv sync` installs into .venv. It is resolved through the
+# invoking environment's sys.prefix and its argv[0] is the relocatable string "python", which
+# Jupyter rewrites to the running interpreter -- so running this script via `uv run` from any
+# checkout or worktree executes against that checkout's .venv. A fixed machine-wide name like
+# "pyrit-dev" is installed with --user and points at whichever environment registered it last,
+# which silently executes notebooks against unrelated code. Pass --kernel_name to override
+# (e.g. "pyrit-dev" in the devcontainer).
+kernel_name = "python3"
 
 
 def main():

doc/getting_started/install_local_dev.md

@@ -77,18 +77,30 @@ VS Code should automatically detect the `.venv` virtual environment. If not:
 3. Choose `.venv\Scripts\python.exe`
 
 #### Running Jupyter Notebooks
-You can create a Jupyter kernel using:
-```bash
-uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=pyrit-dev
-```
+
+`uv sync` already installs a `python3` kernel inside `.venv`, and Jupyter binds it to whichever
+interpreter is running it, so notebooks work out of the box with no kernel registration step.
+
 Start the server using
 ```bash
 uv run jupyter lab
 ```
-or using VS Code, open a Jupyter Notebook (.ipynb file) window, in the top search bar of VS Code, type `>Notebook: Select Notebook Kernel` > `Python Environments...` to choose the `pyrit-dev` kernel when executing code in the notebooks, like those in `examples`. You can also choose a kernel with the "Select Kernel" button on the top-right corner of a Notebook.
+or using VS Code, open a Jupyter Notebook (.ipynb file) window, in the top search bar of VS Code, type `>Notebook: Select Notebook Kernel` > `Python Environments...` to choose the `.venv` interpreter for this checkout. You can also choose a kernel with the "Select Kernel" button on the top-right corner of a Notebook.
 
 This will be the kernel that runs all code examples in Python Notebooks.
 
+If you do want a separately named kernel, scope it to the virtual environment:
+
+```bash
+uv run python -m ipykernel install --sys-prefix --name=pyrit-dev
+```
+
+Avoid `--user` with a fixed name if you work in more than one clone or git worktree. A `--user`
+kernel is installed machine-wide, so every checkout that registers the same name overwrites the
+others, and the survivor points at a single interpreter. Notebooks then execute against an
+unrelated checkout, or fail with `FileNotFoundError: [WinError 2]` once that checkout is deleted.
+See [Jupyter setup](troubleshooting/jupyter_setup.md) if you hit this.
+
 
 #### Running Python Scripts
 

doc/getting_started/troubleshooting/jupyter_setup.md

@@ -25,17 +25,62 @@ Note: Jupyter and ipykernel are no longer installed by default with the base pac
 2. Install with all optional dependencies: `uv sync --extra all`
 3. Install just the notebook dependencies manually: `uv pip install jupyter ipykernel`
 
-After installing these dependencies, you can proceed with the kernel setup steps below.
+`uv sync` already installs a `python3` kernel into `.venv/share/jupyter/kernels/`, so in most
+cases no extra step is needed. If you want a separate, clearly named kernel, register one that is
+scoped to this virtual environment:
 
 ```bash
-uv python -m ipykernel install --user --name=pyrit_kernel
+uv run python -m ipykernel install --sys-prefix --name=pyrit_kernel
 ```
 
+Prefer `--sys-prefix` over `--user`. `--sys-prefix` installs into `.venv/share/jupyter/kernels/`,
+so the kernel is tied to this checkout and disappears when the environment does. `--user`
+installs into your machine-wide Jupyter data directory, where kernels accumulate across
+checkouts and keep pointing at interpreters that may later be deleted.
+
 Now you can start Jupyter Notebook:
 
 ```bash
-uv jupyter notebook
+uv run jupyter notebook
 ```
 
 Once the notebook is open, you can select the kernel that matches the name you gave earlier.
 To do this, go to `Kernel > Change kernel > pyrit_kernel`.
+
+## Kernel fails to start with `FileNotFoundError: [WinError 2]`
+
+If launching a kernel (or running the notebook integration tests) fails before any cell executes,
+the kernelspec is likely pointing at an interpreter that no longer exists. Inspect it:
+
+```bash
+uv run jupyter kernelspec list
+```
+
+and open the `kernel.json` of the kernel being used. A healthy `python3` kernelspec that ships with
+`ipykernel` has a relocatable first argument:
+
+```json
+{"argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}"]}
+```
+
+Jupyter rewrites that `"python"` to the interpreter running Jupyter, so the same file works in any
+environment. If `argv[0]` is instead an absolute path into a different (or deleted) checkout, the
+kernelspec has been rewritten in place. Re-install a correct one:
+
+```bash
+uv sync --reinstall-package ipykernel
+```
+
+This is easy to hit when you use several checkouts or git worktrees. On Windows, `uv` installs
+package files by hardlinking them from its shared cache, so a single `kernel.json` inode can be
+shared by the cache and every virtual environment on the machine. Any tool that edits that file
+**in place** therefore corrupts all of them at once. If the problem keeps coming back in freshly
+created environments, the shared cache itself is corrupted; clear just that package and re-sync:
+
+```bash
+uv cache clean ipykernel
+uv sync --reinstall-package ipykernel
+```
+
+Note that `python -m ipykernel install` is safe here: it replaces the kernelspec directory rather
+than editing files in place, so it does not corrupt the shared cache.