FIX Bind doc notebook execution to the invoking checkout's Jupyter kernel - #2300
Merged
romanlutz merged 2 commits intoJul 31, 2026
Merged
Conversation
…ooting The setup commands did not work: `uv python -m ipykernel install ...` and `uv jupyter notebook` are not valid uv invocations (`uv python` is the Python install manager, and `jupyter` is not a uv subcommand). Use `uv run` for both. Also recommend `--sys-prefix` over `--user` so kernels are scoped to the project virtual environment instead of accumulating machine-wide and outliving the interpreters they point at. Adds a troubleshooting section for kernels that fail to launch with `FileNotFoundError: [WinError 2]`. The `python3` kernelspec shipped by ipykernel is deliberately relocatable (`argv[0] == "python"`, which jupyter_client rewrites to the running interpreter). On Windows uv hardlinks that file from its shared cache into every virtual environment, so a single in-place rewrite baking an absolute interpreter path corrupts the cache and every environment linked to it at once. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a008124c-220e-491c-9ea1-2144867ba53d
`pct_to_ipynb.py` defaulted to the `pyrit-dev` kernel, and the local dev setup guide told contributors to register it with `ipykernel install --user --name=pyrit-dev`. `--user` kernels are machine-wide, so every clone and git worktree that follows that step overwrites the same kernelspec and the survivor points at a single interpreter. Regenerating docs from any other checkout then executes every notebook against that unrelated environment. It does not fail -- it produces plausible output from the wrong code, which is a silent correctness problem on the release-time notebook regeneration path. Default to `python3` instead, the kernel `uv sync` installs into `.venv`. It resolves through the invoking environment's sys.prefix and its argv[0] is the relocatable string "python", which Jupyter rewrites to the running interpreter, so `uv run` binds it to whichever checkout invoked it. `--kernel_name` still overrides it for the devcontainer, which installs `pyrit-dev` legitimately. Note `jupytext --set-kernel -` is not a usable alternative: it matches kernels by comparing argv[0] against the running interpreter, so it fails against the correct relocatable kernelspec and only works with an absolute-path kernelspec. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com> Copilot-Session: a008124c-220e-491c-9ea1-2144867ba53d
jbolor21
approved these changes
Jul 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
doc/generate_docs/pct_to_ipynb.pydefaulted to thepyrit-devkernel, andinstall_local_dev.mdtold contributors to register it withipykernel install --user --name=pyrit-dev.--userkernels are machine-wide, so every clone and git worktree that follows that setup step overwrites the same kernelspec, and the survivor points at one interpreter.The result is that regenerating docs from any other checkout executes every notebook against that unrelated environment. It does not fail. It produces plausible output from the wrong code, which is a silent correctness problem on the release-time notebook regeneration path described in
doc/contributing/7_notebooks.md.This was reproduced rather than inferred. Running the same probe notebook from one worktree:
--set-kernel pyrit-dev) executed inPyRIT-wt-ffmpeg-warnings/.venv--set-kernel python3) executed in the invoking worktree's own.venvBoth exited successfully.
The fix defaults
kernel_nametopython3, the kerneluv syncalready installs into.venv. It resolves through the invoking environment'ssys.prefix, and itsargv[0]is the relocatable string"python"that Jupyter rewrites to the running interpreter, souv runbinds it to whichever checkout invoked it.--kernel_namestill overrides, so the devcontainer's legitimatepyrit-devkernel keeps working unchanged.Two things worth flagging for review:
jupytext --set-kernel -looks like the natural fix ("match the current environment") but is not usable here. It matches kernels by comparingargv[0]against the running interpreter, so it fails against the correct relocatable kernelspec and only works when a kernelspec has an absolute interpreter path baked in, which is the anti-pattern this PR moves away from. That trap is documented inline so it does not get re-derived.--userkernels toward--sys-prefix. On a machine with many worktrees,--userkernels accumulate and outlive the interpreters they point at, which shows up later asFileNotFoundError: [WinError 2].Separately,
doc/getting_started/troubleshooting/jupyter_setup.mdcontained two commands that did not work at all:uv python -m ipykernel install ...(uv pythonis the Python install manager and rejects-m) anduv jupyter notebook(jupyteris not a uv subcommand). Both now useuv run. That file also gains a troubleshooting section for kernels failing withFileNotFoundError: [WinError 2], including the case where uv's hardlinking on Windows lets a single in-place rewrite ofkernel.jsoncorrupt the shared cache and every virtual environment linked to it.No functional PyRIT library code is touched. The only behavior change is the default kernel used by the doc generation script.
Tests and Documentation
Verified end to end with JupyText rather than by inspection:
uv run jupytext --to ipynb --execute --set-kernel python3on a probe notebook printingsys.executableandsys.prefix, and confirmed the executed output pointed at the invoking worktree's.venv.--set-kernel pyrit-devfrom that same worktree and confirmed it executed against an unrelated worktree's.venv, which is the bug this PR fixes.python3resolves to the venv-local kernelspec with a relocatableargv[0], and thatKernelManager(kernel_name="python3").format_kernel_cmd()[0]resolves to the invoking checkout's own interpreter.uv sync --reinstall-package ipykernelrestores a corrupted kernelspec to the correct relocatable form.uv sync, and confirmedtests/integration/memory/test_notebooks_memory.py::test_execute_notebooks[1_sqlite_memory.ipynb]passes with no kernel registration step, which is the behavior the updated docs now describe.Documentation updated in the same change:
doc/getting_started/install_local_dev.md,doc/getting_started/troubleshooting/jupyter_setup.md, and the agent-facing.github/instructions/docs.instructions.md, whose pre-execution checklist previously told agents to verify a shared kernel by hand and to runpip install -e .to rebind it.