feat: repo known files tracking [CM-1329]

[mbani01]

This pull request adds support for tracking well-known files (such as README, SECURITY.md, CONTRIBUTING.md, etc.) in GitHub repositories, both in the enrichment pipeline and in the database. The changes introduce a new database table to inventory these files, update the enrichment worker to extract and classify them, and ensure the data is efficiently upserted and soft-deleted as repositories change.

Database changes:

• Added a new table repo_well_known_files to store inventory of well-known files per repository, including file type, directory, path, and blob OID, with support for soft deletion and change tracking.

Enrichment pipeline updates:

• Modified the GraphQL query in fetchLightRepo.ts to fetch the root, .github, and docs directory trees, and removed the previous REST-based check for SECURITY.md. [1] [2]
• Added logic to classify well-known files and determine if a security file is present using the new directory trees, storing results in the enrichment output. [1] [2] [3] [4] [5]
• Updated types to include well-known files in LightRepoResult and added a new RepoWellKnownFilesUpdate type for batch updates. [1] [2]

Batch processing and upsert:

• Extended the enrichment write buffer to collect, flush, and clear well-known file updates in batches, ensuring deduplication and transactional integrity. [1] [2] [3] [4] [5] [6]
• Added a new function bulkUpsertRepoWellKnownFiles to upsert and soft-delete well-known file records efficiently in the database.

Well-known file classification logic:

• Introduced wellKnownFiles.ts with logic to classify files by type and directory, and to derive the presence of a security file from the directory trees.

These changes enable the system to efficiently track, update, and query the presence and state of well-known files across all monitored repositories.

[cursor]

PR Summary

Medium Risk
Changes how security_file_enabled is computed (boolean from tree vs nullable REST failures) and adds high-volume upsert/soft-delete logic in the enrichment transaction; wrong classification or tree gaps could skew repo metadata.

Overview
Adds repo_well_known_files so the GitHub repos enricher can persist an inventory of governance-style files (README, SECURITY, CONTRIBUTING, etc.) per repo, with blob OIDs, soft deletes, and change_detected_at when content or presence changes.

Enrichment fetch now loads root, .github, and docs directory trees in the same GraphQL call and drops the extra REST Contents API probes for SECURITY.md. security_file_enabled is derived from those trees (intended to match the old probe) and each successful enrich also emits a classified wellKnownFiles list.

Write path: the enrichment flush buffer includes well-known-file updates and bulkUpsertRepoWellKnownFiles runs in the existing enrichment transaction—upsert by (repo_id, path), soft-delete paths no longer seen, bump change_detected_at on blob change or reappearance.

^{Reviewed by Cursor Bugbot for commit 85d7e45. Bugbot is set up for automated code reviews on this repo. Configure here.}

[Copilot]

Pull request overview

Adds GitHub repository well-known-file inventory and change tracking.

Changes:

• Fetches and classifies files from root, .github, and docs.
• Adds transactional upsert, soft-delete, and reappearance tracking.
• Introduces the repo_well_known_files table and indexes.

Metadata: Rename the PR to feat: track repository well-known files (CM-1329) to match repository conventions.

Reviewed changes

Copilot reviewed 6 out of 6 changed files in this pull request and generated 3 comments.

Show a summary per file

File	Description
wellKnownFiles.ts	Classifies well-known files.
updateWellKnownFiles.ts	Persists inventory changes.
types.ts	Adds inventory types.
runEnrichmentLoop.ts	Buffers and flushes inventory updates.
fetchLightRepo.ts	Fetches repository directory trees.
V1784905809__repo_well_known_files.sql	Creates the inventory schema and indexes.
updateEnrichedRepos.ts	Context reviewed for enrichment persistence integration.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.