Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,6 +42,7 @@ All notable changes to this project will be documented in this file.

### Fixed

- Cargo-installed and standalone binaries now fall back to embedded detector rules when no external `detectors.toml` is available
- CRITICAL severity was silently downgraded to LOW at runtime
- All clippy warnings resolved (`Default` impl, redundant closures, identity maps)
- Public API unit tests moved to `tests/` directory (only private API tests remain in `src/`)
Expand All @@ -61,7 +62,7 @@ All notable changes to this project will be documented in this file.

### Documentation

- README architecture documentation now uses two inline GitHub-compatible Mermaid diagrams for the system overview and detection pipeline, alongside the layer-by-layer overview and core data type reference
- README architecture documentation now uses three source-controlled D2 diagrams with generated SVGs for CLI modules and adapters, the scan pipeline, and detector/configuration trust boundaries

## [1.1.0] - 2026-05-05

Expand Down
126 changes: 24 additions & 102 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -196,109 +196,25 @@ password = 'known-test-password' # keywatch:ignore

## Architecture

### System Overview

```mermaid
flowchart TD
CLI["key-watch CLI"]
Scan["scan command"]
Hooks["hook install / uninstall"]
Setup["init / verify-integrity"]

Sources["Scan sources<br/>files, directories, stdin, or git history"]
BuiltIns["detectors.toml<br/>built-in rules"]
UserConfig[".keywatch.toml or --config<br/>custom rules, overrides, excludes"]
Detectors["Merged detector set"]
Pipeline["Detection pipeline"]
Findings["Findings + ScanMetadata"]
BaselineAction{"Baseline action"}
BaselineFilter["Filter known findings<br/>--baseline"]
BaselineUpdate["Write updated baseline<br/>--update-baseline"]
BaselineFile["Baseline JSON + exit"]
Report["JSON or SARIF 2.1.0 report"]
Destination["stdout or --output<br/>summary + exit code"]

HookTargets["Git hook targets<br/>local or global"]
PreCommit["pre-commit<br/>scan staged files"]
PrePush["pre-push<br/>check policy, then scan repository"]

CLI --> Scan
CLI --> Hooks
CLI --> Setup

Hooks --> HookTargets
HookTargets --> PreCommit
HookTargets --> PrePush
PreCommit --> Scan
PrePush --> Scan

Scan --> Sources
Scan --> BuiltIns
Scan --> UserConfig
BuiltIns --> Detectors
UserConfig --> Detectors
Sources --> Pipeline
Detectors --> Pipeline
Pipeline --> Findings
Findings --> BaselineAction
BaselineAction -->|none| Report
BaselineAction -->|filter| BaselineFilter
BaselineAction -->|update| BaselineUpdate
BaselineFilter --> Report
BaselineUpdate --> BaselineFile
Report --> Destination
```
KeyWatch is a single Rust CLI organized as a modular monolith. `main.rs` owns startup and maps validation, configuration, or runtime failures to exit code `2`. `run_cli()` validates and routes commands, while the scan coordinator currently terminates successful scan execution with code `0` or `1`. Focused modules own detector loading, repository policy, scanning, baselines, reports, hooks, and filesystem or process adapters.

### Architecture Overview

KeyWatch is organized into five layers. Data flows top to bottom: input sources and configuration feed the detection pipeline, findings pass through post-processing, and results are serialized to stdout or a file.

1. **Input** — the CLI accepts files, directories, stdin, or git history (`--git-history`). Flags control exclusion (`--exclude`), baselineing (`--baseline`, `--update-baseline`), output format (`--format`), config path (`--config`), and exit behavior (`--exit-mode`).
2. **Configuration** — `detectors.toml` ships with the binary and holds the built-in rules. An optional `.keywatch.toml` adds custom rules, per-detector severity/enable overrides, and exclude patterns. Configuration merges — it never replaces defaults.
3. **Detection pipeline** — six stages run per file: collect files (recursive walk, skipping `.git` and binary files), apply exclude globs, pre-filter by keyword (fast path that avoids regex on irrelevant files), match regexes (single-line and multiline `(?s)`), gate on Shannon entropy, and apply allowlists plus inline `keywatch:ignore` suppression. Files are scanned in parallel with rayon.
4. **Post-processing** — an optional baseline filter suppresses findings already recorded in the baseline file, keyed by a salted SHA-256 fingerprint of the matched content.
5. **Output** — findings serialize as JSON or SARIF 2.1.0 and are written to stdout or an output file, followed by a severity summary and an exit code derived from the exit mode.

### Detection Pipeline

```mermaid
flowchart TD
Start["scan command"]
Config["Load optional configuration"]
Detectors["Initialize built-in and custom detectors"]
Mode{"Input mode"}

Paths["Files or directories"]
Stdin["stdin stream"]
History["git log patch stream"]

Collect["Collect targets<br/>recursive walk, skip symlinks and .git"]
Dedupe["Sort and deduplicate targets"]
Exclude["Apply CLI and config exclude globs"]
Read["Read text files<br/>skip binary and non-UTF-8 content"]
Parallel["Scan files in parallel with rayon"]
Stream["Scan stream in overlapping chunks"]

Keyword["Keyword pre-filter"]
Regex["Line and multiline regex matching"]
Entropy["Entropy threshold"]
Suppress["Detector allowlist + inline suppression"]
Emit["Emit Finding"]
Result["Return findings + metadata"]

Start --> Config --> Detectors --> Mode
Mode -->|paths| Paths
Mode -->|stdin| Stdin
Mode -->|git history| History

Paths --> Collect --> Dedupe --> Exclude --> Read --> Parallel
Stdin --> Stream
History --> Stream

Parallel --> Keyword
Stream --> Keyword
Keyword --> Regex --> Entropy --> Suppress --> Emit --> Result
```
### CLI Modules and Adapters

![KeyWatch CLI module and adapter architecture](docs/architecture/cli-modules.svg)

The green boxes are internal modules, blue boxes mark entry or output boundaries, and yellow boxes are external runtime or distribution adapters. Rust hook management renders and installs scripts; the shell templates are separate runtime adapters that invoke `key-watch scan`.

### Scan Pipeline

![KeyWatch scan pipeline](docs/architecture/scan-pipeline.svg)

Path scans collect and process files in parallel, while stdin and git history use overlapping stream chunks. Baseline updates short-circuit normal report generation. Scan results exit with code `0` or `1`; validation, configuration, and runtime failures are mapped to code `2` at the process boundary.

### Detector and Configuration Trust Boundaries

![KeyWatch detector and configuration trust boundaries](docs/architecture/detector-config-trust.svg)

Detector definitions and repository policy are separate configuration systems. External detector sources retain precedence, with compiled-in rules as the final fallback. Trusted scans skip repository-owned discovery but still honor explicit configuration and non-repository detector sources.

### Core Data Types

Expand All @@ -309,6 +225,8 @@ flowchart TD
- **Baseline** — versioned collection of fingerprint entries; filters out already-known findings.
- **ScanMetadata** — files scanned, total lines, and excluded files, reported alongside findings.

The canonical diagram sources are in `docs/architecture/*.d2`. Run `scripts/render-diagrams.sh render` with D2 v0.7.1 after editing them, or `scripts/render-diagrams.sh check` to detect stale SVGs.

## Development

```sh
Expand All @@ -317,3 +235,7 @@ cargo test
cargo fmt
cargo clippy
```

# LICENSE - GPLv3

[LICENSE](LICENSE)
49 changes: 49 additions & 0 deletions docs/architecture/cli-modules.d2
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
direction: right

classes: {
boundary: {
style: {
fill: "#FBFBFA"
stroke: "#C8C6C1"
stroke-width: 2
border-radius: 10
}
}
entry: {
style: {
fill: "#E1F3FE"
stroke: "#4A84A8"
border-radius: 8
}
}
core: {
style: {
fill: "#EDF3EC"
stroke: "#5C805F"
border-radius: 8
}
}
support: {
style: {
fill: "#F7F6F3"
stroke: "#9A9892"
border-radius: 8
}
}
external: {
style: {
fill: "#FBF3DB"
stroke: "#A9873E"
border-radius: 8
}
}
}

distribution: "DISTRIBUTION\nRelease · Action · Docker" {class: support}
entry: "ENTRY\nTerminal · hooks · automation" {class: external}
facade: "CLI FACADE\nparse · validate · dispatch" {class: entry}
commands: "COMMANDS\nscan · hooks · init · integrity" {class: core}
engine: "SCAN ENGINE\nconfig → detector → scanner\n→ baseline → report" {class: core}
runtime: "RUNTIME\nfiles · stdin · git · output" {class: external}

distribution -> entry -> facade -> commands -> engine -> runtime
95 changes: 95 additions & 0 deletions docs/architecture/cli-modules.svg
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
53 changes: 53 additions & 0 deletions docs/architecture/detector-config-trust.d2
Original file line number Diff line number Diff line change
@@ -0,0 +1,53 @@
direction: right

classes: {
boundary: {
style: {
fill: "#FBFBFA"
stroke: "#C8C6C1"
stroke-width: 2
border-radius: 10
}
}
trusted: {
style: {
fill: "#EDF3EC"
stroke: "#5C805F"
border-radius: 8
}
}
repository: {
style: {
fill: "#FBF3DB"
stroke: "#A9873E"
border-radius: 8
}
}
embedded: {
style: {
fill: "#E1F3FE"
stroke: "#4A84A8"
border-radius: 8
}
}
warning: {
style: {
fill: "#FDEBEC"
stroke: "#A64B48"
border-radius: 8
}
}
}

mode: "SCAN MODE\nnormal or trusted¹\n¹ --no-config-discovery" {class: warning}

detectors: "DETECTOR SOURCE\nKEYWATCH_CONFIG_PATH → repository¹\n→ user → executable → embedded" {class: embedded}

policy: "POLICY SOURCE\nexplicit --config, or repository¹\n.keywatch.toml → keywatch.toml → .kw.toml" {class: repository}

merge: "VALIDATE + MERGE\ncustom rules · overrides · excludes" {class: trusted}
result: "FINAL RULES\ndetectors + exclusion policy" {class: trusted}

mode -> detectors -> merge
mode -> policy -> merge
merge -> result
Loading