Skip to content

feat: add filesystem watcher for event-driven model re-validation - #172

Open
miyunari wants to merge 1 commit into
mainfrom
feat/file-change-watcher
Open

miyunari wants to merge 1 commit into
mainfrom
feat/file-change-watcher

Conversation

@miyunari

@miyunari miyunari commented Aug 7, 2026

Copy link
Copy Markdown
Member

Summary

  • Add fsnotify-based file watcher to the validation agent that triggers model re-validation immediately when files on the PVC change (create/write/remove/rename), instead of waiting for the next polling interval
  • Add --watch flag to the validation-agent binary and watch field to the ContinuousValidation CRD spec
  • Recursive directory watching with 2-second debounce to coalesce rapid multi-file writes
  • The --interval polling ticker remains as a safety net for network filesystems (NFS, CIFS) where inotify cannot observe remote writes

Supported storage backends

  • Local/block-backed: NVMe, SSD, HDD, iSCSI, Ceph RBD, AWS EBS, GCE PD, Azure Disk, local PVs, emptyDir, hostPath

Not supported (use interval fallback)

  • Network filesystems: NFS, CIFS/SMB, GlusterFS, CephFS (FUSE-mounted)

Resolves: SECURESIGN-3615

Test plan

  • Unit tests for watcher: file write, create, remove, subdirectory creation, debounce coalescing, context cancellation (6 tests, all pass)
  • Existing webhook tests pass (19 tests)
  • Full build passes
  • CI checks pass
  • e2e validation with a PVC-backed model on a Kind cluster

🤖 Generated with Claude Code

Use fsnotify/inotify to watch the model path for file changes and trigger
re-validation immediately, instead of waiting for the next polling interval.
The interval-based ticker remains as a fallback for network filesystems
(NFS, CIFS) where inotify cannot observe remote writes.

Adds --watch flag to validation-agent, Watch field to ContinuousValidation
CRD, and watcher package with recursive directory watching and debounce.

Resolves: SECURESIGN-3615

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Signed-off-by: Nina Bongartz <pninak@web.de>
@miyunari
miyunari force-pushed the feat/file-change-watcher branch from 5833601 to 8681af7 Compare August 7, 2026 14:46
@miyunari
miyunari requested review from SequeI and knrc September 14, 2026 08:41
@miyunari

Copy link
Copy Markdown
Member Author

please have a look too. @ompushkara @sampras343 😃

})
}

func (w *Watcher) addRecursive(fsw *fsnotify.Watcher, root string) error {

@ompushkara ompushkara Sep 15, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@miyunari ,Overall looks good . Just few : addRecursive only registers directories. When spec.model.path points to a file—supported by the existing API and tests—filepath.WalkDir visits the file but adds no watch. Run succeeds and logs that it is watching, but no file events are ever delivered.

@ompushkara

Copy link
Copy Markdown
Contributor

@miyunari , Please regenerate and commit config/crd/bases/ml.sigstore.dev_modelvalidations.yaml; otherwise the new watch field is not accepted/persisted by the deployed CRD.

Comment on lines +60 to +61
defer fsw.Close()
defer close(ch)

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Potential bug: Send on closed channel panic during shutdown

Consider stopping the timer before closing the channel.

@sampras343 sampras343 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regenerate the CRD manifests

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants