diff --git a/.github/instructions/docs.instructions.md b/.github/instructions/docs.instructions.md index 7bcc470491..117ee395a3 100644 --- a/.github/instructions/docs.instructions.md +++ b/.github/instructions/docs.instructions.md @@ -51,15 +51,23 @@ When making changes: Before running `jupytext --execute`, make sure the kernel will exercise *the code in this checkout*, not some stale install: -1. **Use a kernel bound to a Python env that has this worktree installed editable.** - Reusing an existing `pyrit` kernel is fine *only if* it points at the current - checkout — otherwise it will resolve imports against an unrelated copy and - either pass on stale code or fail on missing new symbols. - - Quick check: `python -c "import pyrit, pathlib; print(pathlib.Path(pyrit.__file__).resolve())"` - - If it doesn't match this worktree, install editable here: `pip install -e .` - (this rebinds the existing kernel to this checkout, no new kernel needed). - - Only create a new kernel (`python -m ipykernel install --user --name `) - if you actually need an isolated env. +1. **Run jupytext through this checkout's environment and pin the venv-local kernel.** + Use `uv run` so the command inherits this worktree's `.venv`, and pass `--set-kernel python3`, + the kernel that `uv sync` installs inside `.venv`: + ```bash + uv run jupytext --to ipynb --execute --set-kernel python3 doc/path/to/your_notebook.py + ``` + - Quick check: `uv run python -c "import pyrit, pathlib; print(pathlib.Path(pyrit.__file__).resolve())"` + should print a path inside this checkout. + - Do **not** select a fixed machine-wide kernel name such as `pyrit-dev`. Those are installed + with `--user`, so a single name is shared by every clone and worktree on the machine and + resolves to whichever one registered it last. The notebook then runs against unrelated + code and silently produces wrong output. + - Do **not** use `--set-kernel -`. It matches kernels by comparing `argv[0]` to the running + interpreter, which fails against the correct relocatable kernelspec (`argv[0] == "python"`) + that `uv sync` installs. + - If you genuinely need an isolated named kernel, scope it to the virtual environment: + `uv run python -m ipykernel install --sys-prefix --name `. 2. **Credentials must be pre-configured.** Most notebooks call live targets (OpenAI, Azure, etc.) and load creds from `~/.pyrit/.env`. Make sure the required keys are present before executing. @@ -76,12 +84,12 @@ to surface in review; don't paper over it by committing an output-less notebook. Generate .ipynb from .py (with execution — if it fails it means there are errors): ```bash -jupytext --to ipynb --execute doc/path/to/your_notebook.py +uv run jupytext --to ipynb --execute --set-kernel python3 doc/path/to/your_notebook.py ``` Generate .py from .ipynb: ```bash -jupytext --to py:percent doc/path/to/notebook.ipynb +uv run jupytext --to py:percent doc/path/to/notebook.ipynb ``` If a `doc/**/*.py` notebook fails during `jupytext --execute` with errors that look like uninitialized state (missing env vars, undefined names, `initialize_pyrit_async` apparently not run, failing cell shows `Cell In[1]` despite earlier code), **check the `# %%` cell separators in the .py file first**. diff --git a/doc/generate_docs/pct_to_ipynb.py b/doc/generate_docs/pct_to_ipynb.py index 66a38a04a7..225375c2b7 100644 --- a/doc/generate_docs/pct_to_ipynb.py +++ b/doc/generate_docs/pct_to_ipynb.py @@ -36,7 +36,14 @@ file_type = ".py" included_dirs = {"code"} cache_dir = os.path.join(pyrit_root, "dbdata") -kernel_name = "pyrit-dev" +# "python3" is the kernel that `uv sync` installs into .venv. It is resolved through the +# invoking environment's sys.prefix and its argv[0] is the relocatable string "python", which +# Jupyter rewrites to the running interpreter -- so running this script via `uv run` from any +# checkout or worktree executes against that checkout's .venv. A fixed machine-wide name like +# "pyrit-dev" is installed with --user and points at whichever environment registered it last, +# which silently executes notebooks against unrelated code. Pass --kernel_name to override +# (e.g. "pyrit-dev" in the devcontainer). +kernel_name = "python3" def main(): diff --git a/doc/getting_started/install_local_dev.md b/doc/getting_started/install_local_dev.md index efdb5f2c8a..0a1929e364 100644 --- a/doc/getting_started/install_local_dev.md +++ b/doc/getting_started/install_local_dev.md @@ -77,18 +77,30 @@ VS Code should automatically detect the `.venv` virtual environment. If not: 3. Choose `.venv\Scripts\python.exe` #### Running Jupyter Notebooks -You can create a Jupyter kernel using: -```bash -uv run ipython kernel install --user --env VIRTUAL_ENV $(pwd)/.venv --name=pyrit-dev -``` + +`uv sync` already installs a `python3` kernel inside `.venv`, and Jupyter binds it to whichever +interpreter is running it, so notebooks work out of the box with no kernel registration step. + Start the server using ```bash uv run jupyter lab ``` -or using VS Code, open a Jupyter Notebook (.ipynb file) window, in the top search bar of VS Code, type `>Notebook: Select Notebook Kernel` > `Python Environments...` to choose the `pyrit-dev` kernel when executing code in the notebooks, like those in `examples`. You can also choose a kernel with the "Select Kernel" button on the top-right corner of a Notebook. +or using VS Code, open a Jupyter Notebook (.ipynb file) window, in the top search bar of VS Code, type `>Notebook: Select Notebook Kernel` > `Python Environments...` to choose the `.venv` interpreter for this checkout. You can also choose a kernel with the "Select Kernel" button on the top-right corner of a Notebook. This will be the kernel that runs all code examples in Python Notebooks. +If you do want a separately named kernel, scope it to the virtual environment: + +```bash +uv run python -m ipykernel install --sys-prefix --name=pyrit-dev +``` + +Avoid `--user` with a fixed name if you work in more than one clone or git worktree. A `--user` +kernel is installed machine-wide, so every checkout that registers the same name overwrites the +others, and the survivor points at a single interpreter. Notebooks then execute against an +unrelated checkout, or fail with `FileNotFoundError: [WinError 2]` once that checkout is deleted. +See [Jupyter setup](troubleshooting/jupyter_setup.md) if you hit this. + #### Running Python Scripts diff --git a/doc/getting_started/troubleshooting/jupyter_setup.md b/doc/getting_started/troubleshooting/jupyter_setup.md index 84f40f37b9..4ebe6af297 100644 --- a/doc/getting_started/troubleshooting/jupyter_setup.md +++ b/doc/getting_started/troubleshooting/jupyter_setup.md @@ -25,17 +25,62 @@ Note: Jupyter and ipykernel are no longer installed by default with the base pac 2. Install with all optional dependencies: `uv sync --extra all` 3. Install just the notebook dependencies manually: `uv pip install jupyter ipykernel` -After installing these dependencies, you can proceed with the kernel setup steps below. +`uv sync` already installs a `python3` kernel into `.venv/share/jupyter/kernels/`, so in most +cases no extra step is needed. If you want a separate, clearly named kernel, register one that is +scoped to this virtual environment: ```bash -uv python -m ipykernel install --user --name=pyrit_kernel +uv run python -m ipykernel install --sys-prefix --name=pyrit_kernel ``` +Prefer `--sys-prefix` over `--user`. `--sys-prefix` installs into `.venv/share/jupyter/kernels/`, +so the kernel is tied to this checkout and disappears when the environment does. `--user` +installs into your machine-wide Jupyter data directory, where kernels accumulate across +checkouts and keep pointing at interpreters that may later be deleted. + Now you can start Jupyter Notebook: ```bash -uv jupyter notebook +uv run jupyter notebook ``` Once the notebook is open, you can select the kernel that matches the name you gave earlier. To do this, go to `Kernel > Change kernel > pyrit_kernel`. + +## Kernel fails to start with `FileNotFoundError: [WinError 2]` + +If launching a kernel (or running the notebook integration tests) fails before any cell executes, +the kernelspec is likely pointing at an interpreter that no longer exists. Inspect it: + +```bash +uv run jupyter kernelspec list +``` + +and open the `kernel.json` of the kernel being used. A healthy `python3` kernelspec that ships with +`ipykernel` has a relocatable first argument: + +```json +{"argv": ["python", "-m", "ipykernel_launcher", "-f", "{connection_file}"]} +``` + +Jupyter rewrites that `"python"` to the interpreter running Jupyter, so the same file works in any +environment. If `argv[0]` is instead an absolute path into a different (or deleted) checkout, the +kernelspec has been rewritten in place. Re-install a correct one: + +```bash +uv sync --reinstall-package ipykernel +``` + +This is easy to hit when you use several checkouts or git worktrees. On Windows, `uv` installs +package files by hardlinking them from its shared cache, so a single `kernel.json` inode can be +shared by the cache and every virtual environment on the machine. Any tool that edits that file +**in place** therefore corrupts all of them at once. If the problem keeps coming back in freshly +created environments, the shared cache itself is corrupted; clear just that package and re-sync: + +```bash +uv cache clean ipykernel +uv sync --reinstall-package ipykernel +``` + +Note that `python -m ipykernel install` is safe here: it replaces the kernelspec directory rather +than editing files in place, so it does not corrupt the shared cache.