diff --git a/CHANGELOG.md b/CHANGELOG.md index 24179cc..0fe6998 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -42,6 +42,7 @@ All notable changes to this project will be documented in this file. ### Fixed +- Cargo-installed and standalone binaries now fall back to embedded detector rules when no external `detectors.toml` is available - CRITICAL severity was silently downgraded to LOW at runtime - All clippy warnings resolved (`Default` impl, redundant closures, identity maps) - Public API unit tests moved to `tests/` directory (only private API tests remain in `src/`) @@ -61,7 +62,7 @@ All notable changes to this project will be documented in this file. ### Documentation -- README architecture documentation now uses two inline GitHub-compatible Mermaid diagrams for the system overview and detection pipeline, alongside the layer-by-layer overview and core data type reference +- README architecture documentation now uses three source-controlled D2 diagrams with generated SVGs for CLI modules and adapters, the scan pipeline, and detector/configuration trust boundaries ## [1.1.0] - 2026-05-05 diff --git a/README.md b/README.md index 521d77e..4899d82 100644 --- a/README.md +++ b/README.md @@ -196,109 +196,25 @@ password = 'known-test-password' # keywatch:ignore ## Architecture -### System Overview - -```mermaid -flowchart TD - CLI["key-watch CLI"] - Scan["scan command"] - Hooks["hook install / uninstall"] - Setup["init / verify-integrity"] - - Sources["Scan sources
files, directories, stdin, or git history"] - BuiltIns["detectors.toml
built-in rules"] - UserConfig[".keywatch.toml or --config
custom rules, overrides, excludes"] - Detectors["Merged detector set"] - Pipeline["Detection pipeline"] - Findings["Findings + ScanMetadata"] - BaselineAction{"Baseline action"} - BaselineFilter["Filter known findings
--baseline"] - BaselineUpdate["Write updated baseline
--update-baseline"] - BaselineFile["Baseline JSON + exit"] - Report["JSON or SARIF 2.1.0 report"] - Destination["stdout or --output
summary + exit code"] - - HookTargets["Git hook targets
local or global"] - PreCommit["pre-commit
scan staged files"] - PrePush["pre-push
check policy, then scan repository"] - - CLI --> Scan - CLI --> Hooks - CLI --> Setup - - Hooks --> HookTargets - HookTargets --> PreCommit - HookTargets --> PrePush - PreCommit --> Scan - PrePush --> Scan - - Scan --> Sources - Scan --> BuiltIns - Scan --> UserConfig - BuiltIns --> Detectors - UserConfig --> Detectors - Sources --> Pipeline - Detectors --> Pipeline - Pipeline --> Findings - Findings --> BaselineAction - BaselineAction -->|none| Report - BaselineAction -->|filter| BaselineFilter - BaselineAction -->|update| BaselineUpdate - BaselineFilter --> Report - BaselineUpdate --> BaselineFile - Report --> Destination -``` +KeyWatch is a single Rust CLI organized as a modular monolith. `main.rs` owns startup and maps validation, configuration, or runtime failures to exit code `2`. `run_cli()` validates and routes commands, while the scan coordinator currently terminates successful scan execution with code `0` or `1`. Focused modules own detector loading, repository policy, scanning, baselines, reports, hooks, and filesystem or process adapters. -### Architecture Overview - -KeyWatch is organized into five layers. Data flows top to bottom: input sources and configuration feed the detection pipeline, findings pass through post-processing, and results are serialized to stdout or a file. - -1. **Input** — the CLI accepts files, directories, stdin, or git history (`--git-history`). Flags control exclusion (`--exclude`), baselineing (`--baseline`, `--update-baseline`), output format (`--format`), config path (`--config`), and exit behavior (`--exit-mode`). -2. **Configuration** — `detectors.toml` ships with the binary and holds the built-in rules. An optional `.keywatch.toml` adds custom rules, per-detector severity/enable overrides, and exclude patterns. Configuration merges — it never replaces defaults. -3. **Detection pipeline** — six stages run per file: collect files (recursive walk, skipping `.git` and binary files), apply exclude globs, pre-filter by keyword (fast path that avoids regex on irrelevant files), match regexes (single-line and multiline `(?s)`), gate on Shannon entropy, and apply allowlists plus inline `keywatch:ignore` suppression. Files are scanned in parallel with rayon. -4. **Post-processing** — an optional baseline filter suppresses findings already recorded in the baseline file, keyed by a salted SHA-256 fingerprint of the matched content. -5. **Output** — findings serialize as JSON or SARIF 2.1.0 and are written to stdout or an output file, followed by a severity summary and an exit code derived from the exit mode. - -### Detection Pipeline - -```mermaid -flowchart TD - Start["scan command"] - Config["Load optional configuration"] - Detectors["Initialize built-in and custom detectors"] - Mode{"Input mode"} - - Paths["Files or directories"] - Stdin["stdin stream"] - History["git log patch stream"] - - Collect["Collect targets
recursive walk, skip symlinks and .git"] - Dedupe["Sort and deduplicate targets"] - Exclude["Apply CLI and config exclude globs"] - Read["Read text files
skip binary and non-UTF-8 content"] - Parallel["Scan files in parallel with rayon"] - Stream["Scan stream in overlapping chunks"] - - Keyword["Keyword pre-filter"] - Regex["Line and multiline regex matching"] - Entropy["Entropy threshold"] - Suppress["Detector allowlist + inline suppression"] - Emit["Emit Finding"] - Result["Return findings + metadata"] - - Start --> Config --> Detectors --> Mode - Mode -->|paths| Paths - Mode -->|stdin| Stdin - Mode -->|git history| History - - Paths --> Collect --> Dedupe --> Exclude --> Read --> Parallel - Stdin --> Stream - History --> Stream - - Parallel --> Keyword - Stream --> Keyword - Keyword --> Regex --> Entropy --> Suppress --> Emit --> Result -``` +### CLI Modules and Adapters + +![KeyWatch CLI module and adapter architecture](docs/architecture/cli-modules.svg) + +The green boxes are internal modules, blue boxes mark entry or output boundaries, and yellow boxes are external runtime or distribution adapters. Rust hook management renders and installs scripts; the shell templates are separate runtime adapters that invoke `key-watch scan`. + +### Scan Pipeline + +![KeyWatch scan pipeline](docs/architecture/scan-pipeline.svg) + +Path scans collect and process files in parallel, while stdin and git history use overlapping stream chunks. Baseline updates short-circuit normal report generation. Scan results exit with code `0` or `1`; validation, configuration, and runtime failures are mapped to code `2` at the process boundary. + +### Detector and Configuration Trust Boundaries + +![KeyWatch detector and configuration trust boundaries](docs/architecture/detector-config-trust.svg) + +Detector definitions and repository policy are separate configuration systems. External detector sources retain precedence, with compiled-in rules as the final fallback. Trusted scans skip repository-owned discovery but still honor explicit configuration and non-repository detector sources. ### Core Data Types @@ -309,6 +225,8 @@ flowchart TD - **Baseline** — versioned collection of fingerprint entries; filters out already-known findings. - **ScanMetadata** — files scanned, total lines, and excluded files, reported alongside findings. +The canonical diagram sources are in `docs/architecture/*.d2`. Run `scripts/render-diagrams.sh render` with D2 v0.7.1 after editing them, or `scripts/render-diagrams.sh check` to detect stale SVGs. + ## Development ```sh @@ -317,3 +235,7 @@ cargo test cargo fmt cargo clippy ``` + +# LICENSE - GPLv3 + +[LICENSE](LICENSE) diff --git a/docs/architecture/cli-modules.d2 b/docs/architecture/cli-modules.d2 new file mode 100644 index 0000000..a83d6cc --- /dev/null +++ b/docs/architecture/cli-modules.d2 @@ -0,0 +1,49 @@ +direction: right + +classes: { + boundary: { + style: { + fill: "#FBFBFA" + stroke: "#C8C6C1" + stroke-width: 2 + border-radius: 10 + } + } + entry: { + style: { + fill: "#E1F3FE" + stroke: "#4A84A8" + border-radius: 8 + } + } + core: { + style: { + fill: "#EDF3EC" + stroke: "#5C805F" + border-radius: 8 + } + } + support: { + style: { + fill: "#F7F6F3" + stroke: "#9A9892" + border-radius: 8 + } + } + external: { + style: { + fill: "#FBF3DB" + stroke: "#A9873E" + border-radius: 8 + } + } +} + +distribution: "DISTRIBUTION\nRelease · Action · Docker" {class: support} +entry: "ENTRY\nTerminal · hooks · automation" {class: external} +facade: "CLI FACADE\nparse · validate · dispatch" {class: entry} +commands: "COMMANDS\nscan · hooks · init · integrity" {class: core} +engine: "SCAN ENGINE\nconfig → detector → scanner\n→ baseline → report" {class: core} +runtime: "RUNTIME\nfiles · stdin · git · output" {class: external} + +distribution -> entry -> facade -> commands -> engine -> runtime diff --git a/docs/architecture/cli-modules.svg b/docs/architecture/cli-modules.svg new file mode 100644 index 0000000..15ec675 --- /dev/null +++ b/docs/architecture/cli-modules.svg @@ -0,0 +1,95 @@ +DISTRIBUTIONRelease · Action · DockerENTRYTerminal · hooks · automationCLI FACADEparse · validate · dispatchCOMMANDSscan · hooks · init · integritySCAN ENGINEconfig → detector → scanner→ baseline → reportRUNTIMEfiles · stdin · git · output + + + diff --git a/docs/architecture/detector-config-trust.d2 b/docs/architecture/detector-config-trust.d2 new file mode 100644 index 0000000..1437348 --- /dev/null +++ b/docs/architecture/detector-config-trust.d2 @@ -0,0 +1,53 @@ +direction: right + +classes: { + boundary: { + style: { + fill: "#FBFBFA" + stroke: "#C8C6C1" + stroke-width: 2 + border-radius: 10 + } + } + trusted: { + style: { + fill: "#EDF3EC" + stroke: "#5C805F" + border-radius: 8 + } + } + repository: { + style: { + fill: "#FBF3DB" + stroke: "#A9873E" + border-radius: 8 + } + } + embedded: { + style: { + fill: "#E1F3FE" + stroke: "#4A84A8" + border-radius: 8 + } + } + warning: { + style: { + fill: "#FDEBEC" + stroke: "#A64B48" + border-radius: 8 + } + } +} + +mode: "SCAN MODE\nnormal or trusted¹\n¹ --no-config-discovery" {class: warning} + +detectors: "DETECTOR SOURCE\nKEYWATCH_CONFIG_PATH → repository¹\n→ user → executable → embedded" {class: embedded} + +policy: "POLICY SOURCE\nexplicit --config, or repository¹\n.keywatch.toml → keywatch.toml → .kw.toml" {class: repository} + +merge: "VALIDATE + MERGE\ncustom rules · overrides · excludes" {class: trusted} +result: "FINAL RULES\ndetectors + exclusion policy" {class: trusted} + +mode -> detectors -> merge +mode -> policy -> merge +merge -> result diff --git a/docs/architecture/detector-config-trust.svg b/docs/architecture/detector-config-trust.svg new file mode 100644 index 0000000..94f72d4 --- /dev/null +++ b/docs/architecture/detector-config-trust.svg @@ -0,0 +1,95 @@ +SCAN MODEnormal or trusted¹¹ --no-config-discoveryDETECTOR SOURCEKEYWATCH_CONFIG_PATH → repository¹→ user → executable → embeddedPOLICY SOURCEexplicit --config, or repository¹.keywatch.toml → keywatch.toml → .kw.tomlVALIDATE + MERGEcustom rules · overrides · excludesFINAL RULESdetectors + exclusion policy + + + diff --git a/docs/architecture/scan-pipeline.d2 b/docs/architecture/scan-pipeline.d2 new file mode 100644 index 0000000..c96950f --- /dev/null +++ b/docs/architecture/scan-pipeline.d2 @@ -0,0 +1,61 @@ +direction: right + +classes: { + boundary: { + style: { + fill: "#FBFBFA" + stroke: "#C8C6C1" + stroke-width: 2 + border-radius: 10 + } + } + action: { + style: { + fill: "#EDF3EC" + stroke: "#5C805F" + border-radius: 8 + } + } + decision: { + shape: diamond + style: { + fill: "#FBF3DB" + stroke: "#A9873E" + } + } + output: { + style: { + fill: "#E1F3FE" + stroke: "#4A84A8" + border-radius: 8 + } + } + error: { + style: { + fill: "#FDEBEC" + stroke: "#A64B48" + border-radius: 8 + } + } +} + +start: "SCAN\nvalidated input" {class: output} + +prepare: "1 · PREPARE\npolicy + detector rules" {class: action} + +input: "2 · INPUT\npaths in parallel\nstdin / git in chunks" {class: action} + +detect: "3 · DETECT\nkeyword → regex → entropy\n→ suppression → findings" {class: action} + +baseline: "4 · BASELINE\nkeep or filter" {class: decision} + +baseline_done: "BASELINE UPDATED\nsave + return\nskip reporting" {class: output} + +report: "5 · REPORT\nJSON / SARIF\nsummary + optional file" {class: action} + +exit: "6 · EXIT\n0 clean · 1 findings\n2 validation / config / runtime" {class: output} + +start -> prepare -> input -> detect -> baseline +baseline -> baseline_done: --update-baseline +baseline -> report: normal scan +report -> exit diff --git a/docs/architecture/scan-pipeline.svg b/docs/architecture/scan-pipeline.svg new file mode 100644 index 0000000..efc2eda --- /dev/null +++ b/docs/architecture/scan-pipeline.svg @@ -0,0 +1,103 @@ +SCANvalidated input1 · PREPAREpolicy + detector rules2 · INPUTpaths in parallelstdin / git in chunks3 · DETECTkeyword → regex → entropy→ suppression → findings4 · BASELINEkeep or filterBASELINE UPDATEDsave + returnskip reporting5 · REPORTJSON / SARIFsummary + optional file6 · EXIT0 clean · 1 findings2 validation / config / runtime --update-baselinenormal scan + + + + diff --git a/scripts/render-diagrams.sh b/scripts/render-diagrams.sh new file mode 100755 index 0000000..a6d91c6 --- /dev/null +++ b/scripts/render-diagrams.sh @@ -0,0 +1,50 @@ +#!/bin/sh +set -eu + +expected_version="v0.7.1" +d2_bin="${D2_BIN:-d2}" +mode="${1:-check}" + +if [ "$("$d2_bin" version)" != "$expected_version" ]; then + echo "error: D2 $expected_version is required (set D2_BIN to the pinned binary)" >&2 + exit 1 +fi + +case "$mode" in + render | check) ;; + *) + echo "usage: $0 [render|check]" >&2 + exit 1 + ;; +esac + +diagram_directory="docs/architecture" +diagrams="cli-modules scan-pipeline detector-config-trust" + +render_diagram() { + name="$1" + output="$2" + "$d2_bin" validate "$diagram_directory/$name.d2" + "$d2_bin" fmt --check "$diagram_directory/$name.d2" + "$d2_bin" --layout=elk --theme=0 --pad=40 --omit-version --salt="$name" \ + "$diagram_directory/$name.d2" "$output" +} + +if [ "$mode" = "render" ]; then + for name in $diagrams; do + render_diagram "$name" "$diagram_directory/$name.svg" + done + exit 0 +fi + +temporary_directory="$(mktemp -d)" +trap 'rm -rf "$temporary_directory"' EXIT HUP INT TERM + +for name in $diagrams; do + rendered="$temporary_directory/$name.svg" + render_diagram "$name" "$rendered" + if ! cmp -s "$rendered" "$diagram_directory/$name.svg"; then + echo "error: $diagram_directory/$name.svg is stale; run $0 render" >&2 + exit 1 + fi +done diff --git a/src/detector.rs b/src/detector.rs index 0097cc4..c19d677 100644 --- a/src/detector.rs +++ b/src/detector.rs @@ -5,7 +5,9 @@ pub use error::DetectorInitError; use crate::report::{ParseSeverityError, Severity}; use regex::Regex; use serde::Deserialize; -use std::{fmt, fs, str::FromStr}; +use std::{borrow::Cow, fmt, fs, str::FromStr}; + +const EMBEDDED_DETECTORS_CONFIG: &str = include_str!("../detectors.toml"); #[derive(Debug)] pub enum DetectorError { @@ -204,13 +206,15 @@ pub(crate) fn initialize_trusted_detectors() -> Result, DetectorIn fn initialize_detectors_from_config( include_repository_config: bool, ) -> Result, DetectorInitError> { - let config_path = find_detectors_config(include_repository_config) - .ok_or(DetectorInitError::ConfigNotFound)?; - let toml_contents = - fs::read_to_string(&config_path).map_err(|source| DetectorInitError::ReadConfig { - path: config_path.clone(), - source, - })?; + let toml_contents = match find_detectors_config(include_repository_config) { + Some(config_path) => Cow::Owned(fs::read_to_string(&config_path).map_err(|source| { + DetectorInitError::ReadConfig { + path: config_path, + source, + } + })?), + None => Cow::Borrowed(EMBEDDED_DETECTORS_CONFIG), + }; let config: DetectorsConfig = toml::from_str(&toml_contents) .map_err(|source| DetectorInitError::ParseConfig { source })?; diff --git a/tests/exit_tests.rs b/tests/exit_tests.rs index 1b24262..e6469b7 100644 --- a/tests/exit_tests.rs +++ b/tests/exit_tests.rs @@ -73,10 +73,13 @@ fn test_exit_code_on_no_secrets() { fn test_runtime_errors_exit_with_code_two() { let test_dir = setup_scan_dir("exit_runtime_error", false); let temp_file = test_dir.join("secret.txt"); + let invalid_detectors = test_dir.join("invalid-detectors.toml"); fs::write(&temp_file, "AWS_KEY=AKIAIOSFODNN7EXAMPLE").expect("Write test file"); + fs::write(&invalid_detectors, "[[detectors]").expect("Write invalid detector config"); let status = Command::new(env!("CARGO_BIN_EXE_key-watch")) .current_dir(&test_dir) + .env("KEYWATCH_CONFIG_PATH", invalid_detectors) .arg("scan") .arg(&temp_file) .status() @@ -87,6 +90,34 @@ fn test_runtime_errors_exit_with_code_two() { fs::remove_dir_all(test_dir).expect("Cleanup"); } +#[test] +fn test_embedded_detectors_enable_standalone_scan() { + // Given a standalone binary with no detector configuration on disk. + let test_dir = setup_scan_dir("embedded_detectors", false); + let temp_file = test_dir.join("secret.txt"); + fs::write(&temp_file, "AWS_KEY=AKIAIOSFODNN7EXAMPLE").expect("Write test file"); + + // When the binary scans a file containing a built-in detector match. + let output = Command::new(env!("CARGO_BIN_EXE_key-watch")) + .current_dir(&test_dir) + .env_remove("KEYWATCH_CONFIG_PATH") + .env("HOME", &test_dir) + .env("XDG_CONFIG_HOME", &test_dir) + .env("APPDATA", &test_dir) + .env("USERPROFILE", &test_dir) + .arg("scan") + .arg(&temp_file) + .output() + .expect("Run standalone key-watch"); + + // Then embedded detectors identify the secret instead of producing a config error. + assert_eq!(output.status.code(), Some(1)); + assert!(output.stderr.is_empty()); + assert!(String::from_utf8_lossy(&output.stdout).contains("potential secret(s) detected")); + + fs::remove_dir_all(test_dir).expect("Cleanup"); +} + #[test] fn test_exit_mode_always() { let test_dir = setup_scan_dir("exit_always", true);