perf: defer timezonefinder & pandas imports to cut startup time#482
Open
mrosseel wants to merge 293 commits into
Open
perf: defer timezonefinder & pandas imports to cut startup time#482mrosseel wants to merge 293 commits into
mrosseel wants to merge 293 commits into
Conversation
- Add pinctrl to PWM overlay to route PWM0_1 to GPIO 13 - Change boot-splash to static mode (no animation) - Fix boot-splash color from blue to red (RGB565) - Fix NFS deploy script /etc symlinks for PAM Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Primary build on self-hosted aarch64 (Pi5) - fast native - Fallback to ubuntu-latest with QEMU if Pi5 unavailable - Also trigger on nixos branch pushes Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
pifinder-dev doesn't exist, use pifinder-netboot Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The closure's /etc is a symlink, so direct paths don't work. Use /etc/static which resolves through the NixOS etc derivation. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Type=idle waits for ALL other services to finish before starting, causing ~2 minute delay. Type=simple starts immediately. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
ln -sfT can't overwrite a directory, need rm -rf first Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Update bootstrap.nix to fetch from correct repo/branch - Update NIXOS_STATUS.md: WiFi warning is cosmetic, not a real issue Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add pifinder.repoUrl module option to both services.nix and bootstrap.nix so switching between mrosseel (testing) and brickbots (production) repos requires only changing the default value in one place. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Builds bootstrap SD image on Pi5 self-hosted runner, extracts boot/rootfs partitions, and creates tarball for RPi OS -> NixOS migration. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Adds push trigger for nixos branch when bootstrap.nix, flake.nix, or the workflow itself changes. Uses VERSION env var for flexibility. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Add -type f to find command to match actual image files inside the sd-image subdirectory. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
The rootfs contains root-owned files and lost+found directories that require elevated permissions to access. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Remove any existing tarball, staging dir, and mount points before starting to avoid permission issues from previous failed runs. Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Port manifest/channel infrastructure from migration branch: - _parse_version with pre-release support - _fetch_manifest / _fetch_version_txt with fallback - _filter_upgrades, _available_channels - Multi-phase menu: channel → version → action - UIReleaseNotes with _strip_markdown - Full test suite for version parsing, filtering, markdown Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
shell.nix provides Python 3.9 + native build deps (linuxHeaders, gcc, SDL2, etc.) with LD_LIBRARY_PATH for scipy/numpy. .envrc auto-creates venv and installs requirements via direnv. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace runtime camera selection (config.txt dtoverlay) with build-time device tree overlays and NixOS specialisations. All three camera variants (imx462, imx296, imx477) are pre-built; switching is instant via `switch-to-configuration boot` + reboot. - Remove runtimeCameraSelection option, always bake overlay into DT - Change default camera from imx296 to imx462 - Add specialisations to SD image (imx296, imx477) and netboot (imx296, imx462) - Add pifinder-switch-camera wrapper script and sudoers entry - Preserve camera choice across upgrades via /var/lib/pifinder/camera-type - Delete Debian sys_utils.py, rename sys_utils_nixos.py to sys_utils.py - Simplify utils.py get_sys_utils() (no more /etc/NIXOS check) - Delegate callbacks.py get_camera_type() to sys_utils - Deploy script: use FDTDIR in extlinux.conf, remove dtoverlay from config.txt Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
The pifinder systemd service has a restricted PATH that doesn't include sudo. Use /run/current-system/sw/bin/sudo for all calls. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
These commands are called outside of sudo (which has secure_path), so they need full paths in the restricted pifinder service PATH. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Bump nixpkgs input to nixos-25.11 - Update Python 3.12 → 3.13 across all references - Switch simplejpeg wheel to cp313 - Add explicit pyproject/build-system to 13 packages (required by 25.11) - Add dontCheckRuntimeDeps for packages with env-level deps - Rename hardware.pulseaudio → services.pulseaudio Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot SD image: use custom ubootSD with CONFIG_CMD_SYSBOOT, hardcoded load address, correct partition (0:2) - RPi.GPIO: patch cpuinfo.c for aarch64 where /proc/cpuinfo lacks Hardware line and u-boot replaces firmware DTB - I2C1: add overlay to fdtoverlay chain (nixos-hardware was bypassed) - sudo: substituteInPlace pifinder-src to use /run/wrappers/bin/sudo - polkit: add reboot/shutdown D-Bus rules for pifinder user - /boot: mount firmware at /boot/firmware so extlinux.conf works Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- u-boot: disable PCI/USB/NET probing (CONFIG_PREBOOT="", CONFIG_PCI=n, CONFIG_USB=n, etc.) — eliminates ~3-4s of unnecessary hardware scanning - boot-splash: move earlier with DefaultDependencies=false, poll for SPI device instead of relying on systemd restart - Disable NetworkManager-wait-online (7.1s savings, NM still works async) - pwm-permissions: no longer blocks pifinder startup (runs in parallel) - pifinder-src: pre-compile .pyc bytecode at build time Net result: splash ~7s (was 21s), menu ~45s (was 65s). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add bashInteractive to systemPackages (mkForce strips default bash) - Add download-buffer-size 512MB for large PiFinder repo tarball - Fix migration_progress OLED binary args to match expected format (<percent> <stage_num> <stage_total> <stage_name>) - Fix nixos-rebuild progress: count copying/building lines instead of parsing non-existent (N/M) format Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Unified Python dev shell: extract python-packages.nix as single source of truth for both Pi production and x86 dev environments, add flake devShell, replace old shell.nix + pip venv with `use flake` - Pygame keyboard capture: monkey-patch luma's _abort to intercept pygame KEYDOWN events, fixing arrow keys on Wayland where PyHotKey/pynput cannot access keyboard - Cedar detect: build from source instead of shipping prebuilt binaries, add Cargo.lock for reproducible builds - Software update UI: channel-based browsing (stable/beta/unstable), scrolling text for long labels, 7x square secret to unlock unstable channel, always show channels even when empty - Fix pre-commit hooks: replace nox/virtualenv with direct devShell tool invocation, fix mypy config for nix environment - Fix numpy.math removal in tetra3 (Python 3.13 compat) - Various Python 3.13 compatibility fixes across codebase Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Fix test mode images: drain debug commands before blocking capture() so mode switch is processed even when camera hardware is unresponsive - Fix PAM errors: create dedicated pifinder PAM service (auth-only, no session/account management that requires setuid) - Fix service PATH: add /run/wrappers/bin and /run/current-system/sw/bin to pifinder systemd service path (camera switch, sudo, systemctl) - Fix AP networking: enable firewall ports 53/67 UDP (DNS/DHCP) and 80 TCP (web UI), add dnsmasq for NM shared mode, AP autoconnect - Fix tetra3 rev pin: update to cded265 (np.math.factorial fix) - Remove bootstrap workflow (no longer used) - Remove absolute paths from sys_utils.py (service PATH handles resolution) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Fetch catalog_images.tar.zst from files.miker.be during SD image build, unpack at build time, and copy writable into the root filesystem. Not included in the regular system closure (no Cachix bloat). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Split pifinder-src into stable astro-data (~193MB) derivation + code (~57MB) for smaller OTA updates - Add bootstrapMode option: minimal NixOS without pifinder-src, runs nixos-rebuild on first boot for migration from Raspbian - Custom tarball builder (replaces make-system-tarball.nix) to avoid nix sandbox symlink conflict during closure copy - Bootstrap tarball: 648MB with boot firmware, nix closure, extlinux - Conditionally disable pifinder/cedar/watchdog services in bootstrap Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Migration tarball is now a complete v2.5.0 system — no two-stage first-boot rebuild needed. Users get a working PiFinder immediately after migration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
images/ was fully stripped but console.py and splash.py load images/welcome.png on startup. Keep that file, strip the rest. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
After migration from Raspbian, the tarball includes /nix-path-registration with Nix store path data. This oneshot service loads it into the Nix DB on first boot so nix-store and nixos-rebuild work correctly. Uses ConditionPathExists so it's a no-op on normal boots. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Replace nixos-rebuild with direct nix store path deployment: - CI builds push to Cachix and stamp pifinder-build.json per branch - Upgrade service uses nix copy + switch-to-configuration (no OOM) - Software UI resolves store paths from per-branch build metadata - Delete versions.json (replaced by pifinder-build.json) Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…rop the baked-target existence condition (ADR 0003)
…y the update UI keys on
…ng shifted, gpsd pointed at a nonexistent device
The repeated block in UBX-NAV-SVINFO is chn, svid, flags, quality, cno, elev, azim — but the parser started reading at chn, so every field was shifted one byte: satellite IDs were channel numbers, the used flag was svid bit 0, and the C/N0 was the 0-7 quality indicator. Because "cno" was really the quality indicator (>=1 even for idle SBAS/QZSS channels still searching), the sats-seen count was inflated to near the full channel count, and the per-satellite used flags were random — which is why uSat was previously treated as stale and ignored. With the offsets fixed: - nSat only counts satellites with an actual signal (cno > 0) - uSat from SVINFO is valid, so report it as the used count instead of relying solely on NAV-SOL - elevation/azimuth are decoded as signed values per the spec Also update the used count from NAV-PVT numSV, which was parsed but never surfaced — on protVer >= 15 receivers gpsd enables NAV-PVT instead of NAV-SOL, so the used count stayed 0 forever despite a valid fix. And actually set got_sat_update when NAV-SAT arrives so the SVINFO fallback defers to it as intended. Verified against a live MAX-M8 (PROTVER 14) stream: SVINFO uSat now matches NAV-SOL numSV, and IDs/C-N0 match gpsd's own SKY decode. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
Newer u-blox receivers (later M8 firmware, M9, M10) get NAV-PVT + NAV-SAT from gpsd instead of NAV-SOL + NAV-SVINFO, so this path is what recent PiFinder GPS units actually exercise. - take "used" from the dedicated svUsed flag (bit 3) instead of inferring it from the quality indicator - only count satellites with an actual signal: NAV-SAT lists every known satellite, so nSat was inflated by idle entries - decode elevation/azimuth as signed values Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
During cold start the receiver reports an estimated C/N0 for almanac-predicted satellites it is still trying to confirm, so a cno > 0 filter makes the seen count start around 20+ and sink to the real value as candidates fail to confirm. Gate on the quality indicator instead: only code-locked signals (quality >= 4) count as seen, in both NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
mrosseel
force-pushed
the
lazy-imports
branch
4 times, most recently
from
51b0bde to
9dfd21e
Compare
Gating the seen count on quality >= 4 (code locked) made the display flap to zero during marginal re-acquisition — walking out to sky view showed 0-3 satellites bouncing instead of the steady climb to a lock. Satellites at quality 2-3 (signal acquired) have a real signal and belong in the count; only quality-1 search candidates, whose C/N0 is an estimate, still don't. Same threshold for NAV-SVINFO and NAV-SAT. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NrdGJ1s3jmA9qzV8HAsdZs
Integrates brickbots/main (through brickbots#523) into the nixos branch. Conflict resolutions: - gps_ubx_parser.py / test_gps_ubx_parser.py: both branches carry the UBX NAV-SVINFO/NAV-SAT decoding fix (upstream brickbots#524). Kept nixos's later refinement that counts satellites at quality >= 2 (signal acquired) instead of >= 4 (code locked), which stopped the seen count flapping to zero during marginal re-acquisition. - ui/software.py: both branches independently built manifest-driven update logic. Kept nixos's _fetch_update_manifest (channels dict via _entry_from_manifest) as the single source and adapted upstream's _migration_version_info_from_manifest to consume it; threaded migration_url/migration_sha256_url through _entry_from_manifest so the migration-tarball selection still works. Both features retained. - test_software.py: unioned both suites; dropped upstream's obsolete TestFetchUpdateManifest (tested the removed raw-dict signature) and reshaped the migration fixtures to mirror real manifest entries. - nixos_migration.sh / nixos_migration_init.sh: took upstream's version. nixos deliberately dropped its Raspbian-side copies (brickbots#521); the canonical pre-migration implementation lives on main.
The update manifest marked a build available: true whenever store_path merely looked like a valid Nix store path, never checking the binary cache. A build whose attic push failed (or was GC'd) was still advertised, so a device selecting it hit 'no substituter can build it' and stranded the upgrade. set_available now probes each cache's narinfo and only advertises a build when one actually serves the closure; otherwise available: false with reason 'not in cache' / 'cache unreachable' (the UI already blocks these). Defaults to the dev + release caches, so the gate is active without any workflow change. --skip-cache-check retained for offline/tests. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
PiFinder's imx462 module has a 74.25 MHz oscillator; on Raspbian the
imx290 overlay parameter clock-frequency=74250000 sets this, but
fdtoverlay cannot apply overlay parameters, so the NixOS DTB kept the
37.125 MHz default — wrong INCKSEL/PLL programming, sensor enumerates
but never streams ("Camera frontend has timed out"). Apply a compiled
override overlay (cam1_clk + sensor node) after the camera overlay.
No kernel change: nixos-hardware's raspberry-pi-4 module already pins
the RPi vendor kernel (stable_20250916), whose imx290 driver carries
imx462 support. ADR 0008 rewritten accordingly.
The nixos branch replaced nox + requirements*.txt with uv, but an
upstream merge kept .github/workflows/nox.yml while dropping
python/noxfile.py, so the nox check failed on every nixos-line PR
("Failed to load Noxfile"). Rewrite the workflow to run the same
checks through uv (ruff pinned to 0.4.8 like upstream's noxfile, mypy,
pytest smoke/unit/ui), keeping the "nox" check name.
Fix what crept in while CI was dead:
- ObservationsDatabase resolved utils.observations_db as an import-time
default argument, so the UI-harness sandbox patch never applied — on a
runner without ~/PiFinder_data every UI test died with "unable to
open database file" (and locally the suite wrote to the real data
dir). Resolve the path at call time.
- UIMigrationConfirm/UIMigrationProgress documented in _COVERAGE_SKIP:
smoke-driving them risks starting a real migration download.
- UBXParser.reader also accepts the aiofiles reader from_file passes;
software.py declares the gettext-installed _ for mypy.
Update CLAUDE.md dev commands (uv, no tetra3 submodule).
state.py imported timezonefinder and built a TimezoneFinder() in SharedState.__init__, and plot.py imported pandas at module level. Both load during boot — SharedState is constructed at startup, and plot.py is pulled in via menu_structure -> UIChart. Defer the TimezoneFinder construction to the first set_location(), and pandas to the plot functions that use it, so neither blocks startup. comets.py is intentionally left unchanged: its module-level 'from skyfield.data import mpc' imports pandas regardless, so deferring pandas there has no effect without also deferring skyfield. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The nixos branch replaced nox + requirements*.txt with uv, but an
upstream merge kept .github/workflows/nox.yml while dropping
python/noxfile.py, so the nox check failed on every nixos-line PR
("Failed to load Noxfile"). Rewrite the workflow to run the same
checks through uv (ruff pinned to 0.4.8 like upstream's noxfile, mypy,
pytest smoke/unit/ui), keeping the "nox" check name.
Fix what crept in while CI was dead:
- ObservationsDatabase resolved utils.observations_db as an import-time
default argument, so the UI-harness sandbox patch never applied — on a
runner without ~/PiFinder_data every UI test died with "unable to
open database file". Resolve the path at call time.
- UIMigrationConfirm/UIMigrationProgress documented in _COVERAGE_SKIP:
smoke-driving them risks starting a real migration download.
- UBXParser.reader also accepts the aiofiles reader from_file passes;
software.py declares the gettext-installed _ for mypy.
Update CLAUDE.md dev commands (uv, no tetra3 submodule).
…doers rule matches The PiFinder-Type (screen_direction) menu and other restart paths call sys_utils.restart_pifinder(), which ran "sudo systemctl restart pifinder". The NixOS sudoers rule allows exactly "systemctl restart pifinder.service" — sudo matches arguments verbatim, so the restart was refused and silently never happened: the UI showed "Restarting..." but the old process kept running. Net effect: switching PiFinder Type right→left left the integrator dead-reckoning with the old IMU geometry (reversed left/right motion) until a manual reboot. Use the full unit name (matches every other systemctl call in the codebase), pass -n so a denied sudo fails instead of hanging on a password prompt, and log a restart failure instead of swallowing it.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Re-implements the salvageable part of #378 (boot-speed), cleanly on top of the
nixosbranch.#378's perf work was lost during that branch's own internal merges — its HEAD no longer contains the lazy-import changes, and they were never merged upstream either. This brings back the part that still matters for startup time.
What this defers
state.py):SharedState.__init__eagerly importedtimezonefinderand constructedTimezoneFinder()(which loads its dataset) at startup. It's now constructed lazily on the firstset_location(), i.e. after boot.plot.py): imported at module level.plot.pysits on the startup path (menu_structure→UIChart→plot), so this loaded pandas during boot. Deferred into the four functions that actually use it.comets.pyis intentionally left unchanged: its module-levelfrom skyfield.data import mpcimports pandas regardless, so deferring pandas there is a no-op without also deferring skyfield (out of scope here).Verification
state.pyno longer loadstimezonefinder(confirmed at runtime).plot.py's only module-level pandas reference is removed;skyfield.data.hipparcosalready imports pandas lazily, and nothing else inplot.py's import chain pulls pandas.py_compileclean on all touched files.🤖 Generated with Claude Code