Add incremental (ref-addressed) branch indexing (v3.0.0)

[simianhacker]

What

Adds delta (ref-addressed) branch indexing as an alternative to Sourcerer's existing
commit-addressed snapshot indexing, so a fast-moving branch (e.g. a GitOps/IaC repo that deploys
off main) can stay current with a cheap delta update instead of a full re-index on every HEAD
advance.

Set mode: delta on a branch source in sourcerer.yml to opt in; mode: snapshot (the
default) is unchanged. Delta content is stored per-ref (carries git.ref, no git.commit); a
HEAD advance diffs old tip vs. new tip and only deletes/reindexes the changed paths. Snapshot
content is stored per-commit as before (carries git.commit, no git.ref).

Every Agent Builder content tool gains two new optional filters — git_ref and git_ref_type —
alongside the existing git_commit. All three are wildcard filters resolved against the small
sourcerer-refs index first, so one query shape serves both content shapes without the caller
knowing which mode a source uses.

This is a big enough change to the index schema to ship as v3.0.0: backing indices move from
sourcerer-v2-* to sourcerer-v3-*. See Breaking Changes below.

How to Verify

1. Add a branch source with mode: delta to a sourcerer.yml and run
   sourcerer index --config sourcerer.yml. Confirm the indexed content carries git.ref and
   no git.commit, and that a single refs join doc (keyed by host~org~repo~ref) holds the
   citable git.commit for that ref.
2. Re-run the same command — it should report the ref as already up to date and make no content
   changes (sourcerer-files/sourcerer-lines doc counts unchanged).
3. Push a commit to the tracked branch and re-run — only the changed files should be
   deleted/reindexed (visible via the reported files/lines counts).
4. Run sourcerer setup against a fresh cluster (or one only holding sourcerer-v2-* data), then
   sourcerer index github/<org>/<repo> -t <tag> — confirm it creates and populates the new
   sourcerer-v3-* indices from scratch (a v2->v3 index rename has no automatic migration).
5. Run sourcerer setup to push the updated tool definitions, then call
   sourcerer.code.search/sourcerer.files.cat/etc. filtering by git_commit against a
   snapshot source and by git_ref (e.g. main) against a delta source — both return rows with
   a citable git.commit.

Query Change Example

Content docs come in two disjoint shapes: snapshot rows carry git.commit (no git.ref); delta
rows carry git.ref (no git.commit). Each tool takes wildcard filters git_commit, git_ref,
and git_ref_type, resolves them against the small sourcerer-refs index, then selects matching
rows in the large content index:

FROM sourcerer-lines
| WHERE <scope>
    AND (
      -- snapshot rows: keep those whose commit the refs index matched
      (git.commit IS NOT NULL AND git.commit IN (<matching commits from sourcerer-refs>))
      OR
      -- delta rows: keep those whose ref the refs index matched
      (git.ref IS NOT NULL AND git.commit IS NULL AND git.ref IN (<matching refs from sourcerer-refs>))
    )
| FORK
    ( WHERE git.commit IS NOT NULL )                       -- snapshot: commit already present
    ( WHERE git.ref IS NOT NULL AND git.commit IS NULL     -- delta: attach the commit
        | LOOKUP JOIN sourcerer-refs ON git.host, git.org, git.repo, git.ref )

• Filter by commit (git_commit = "a1b2c3d..."): matches snapshot rows, exactly like before — no
  join needed, the commit is already on the row.
• Filter by ref (git_ref = "main"): matches delta rows, and the LOOKUP JOIN attaches that
  ref's current commit so every result is still citable to an exact commit.

Existing commit-filter usage is unchanged; ref-filtering is the new capability. There is no
synthetic join key — delta rows join back to their refs doc on (host, org, repo, ref) directly.

Breaking Changes

Bumps to v3.0.0:

• Indices: backing indices rename sourcerer-v2-* to sourcerer-v3-*. No automatic
  migration — run sourcerer setup to create the v3 templates, then re-index every source. The
  old sourcerer-v2-* indices can be deleted once you've re-indexed.
• Agent Builder tools (additive): content tools (sourcerer.code.search, sourcerer.code.grep,
  sourcerer.files.*) gain two new optional wildcard filters, git_ref and git_ref_type;
  git_commit is unchanged, so existing calls keep working. Re-run sourcerer setup to push the
  updated tool definitions.
• Config (additive): a branch source may set mode: delta; omitting mode keeps the
  historical snapshot behavior, so existing sourcerer.yml files work unmodified.

[davemoore-]

Thanks @simianhacker, acknowledging receipt of this PR. I've been testing it pretty heavily over the last couple days. Some minor fixes, possibly some semantic changes we can review, but most importantly, I'm almost certain I can adjust it to avoid introducing a ref_key on all the files and lines. I'll keep working on that and scale testing it, and will share the changes here. Hoping to have it ready in the next day or two.

[davemoore-]

@simianhacker I've added my contributions to this PR for review.

Good news: I was able to remove ref_key (benefits are described in more detail below) and prevent double ref markers for snapshot-indexed sources. I still consider this PR to be breaking change that warrants v3 due to the introduction of the ref field in the files and lines indices.

More details on the notable changes:

Semantic changes – worth understanding before reviewing changes:

• Renamed update_mode to index_strategy (and in sourcerer.yml: sources[i].index.strategy) which better captures the overarching behaviors of "snapshot" and "incremental" since snapshots generally aren't updated.
  • I actually prefer index_mode, but that's overloaded with Elasticsearch's index.mode setting. A "strategy" feels like two or more approaches that converge on the same outcome, and that's not really what this setting is doing. So I'm open to exploring this name a little longer. Maybe I'm overthinking the collision with Elasticsearch's index.mode. If it's never likely to cause confusion, then I think index_mode / sources[i].index.mode with the values of "snapshot" and "incremental" is most natural. @simianhacker thoughts?

Significant changes – worth reviewing and understanding in depth:

• Removed ref_key and instead modified the queries to use subqueries, FORKs, and JOINs to resolve git.commit, git.ref, and git.ref_type regardless of whether the files or lines were indexed as a snapshot or incrementally (see example of new query syntax).
  • Benefits:
    • Files and lines no longer carry duplicate information in a ref_key, keeping space and indexing throughput optimal.
    • sourcerer-v3-refs no longer needs nor uses two ref markers per snapshot-indexed ref.
    • Agents no longer have to think about a non-standard ref_key syntax when scoping searches.
    • Wildcards ("*") are still allowed on all git_* scoping params.
    • Coincidentally, the queries are faster now by roughly an order of magnitude, despite having more complex syntax.
  • Trade-offs:
    • Requires Elastic v9.5.0+ to use the IN ( FROM ... ) subqueries, and bets that this subquery syntax eventually will be promoted from tech-preview to GA.

Smaller changes:

• Incrementally-indexed sources now support sources[i].index.level and sources[i].index.suffix.
• Queries filter by status == "complete" to prevent reading from incomplete, partially indexed refs. This makes incrementally-indexed sources briefly unreadable while they're being updated (which should be a couple seconds in most cases), which I judge to be a small price to pay for consistency.
• Set the status field in sourcerer-v3-refs to "complete" instead of "ready" for consistency with snapshot indexing.
• Removed the unused ref_type and update_mode field from sourcerer-v3-files and sourcerer-v3-lines to save space. The same information is kept in sourcerer-v3-refs and used at query time with efficient lookups.
• Updated the CLI output to show the number of files and lines indexed during incremental indexing, rather than the total number of files and lines that exist in the repo.
• Fixed a false fail message in the CLI output when indexing snapshots.

[davemoore-]

One more semantic change:

• Renamed index_strategy and sources[i].index.strategy (originally update_mode in your PR) to simply mode and sources[i].mode, keeping the values "snapshot" and "incremental". As a top-level field of a source, it implies that the overall handling of a source is modal, and can take the form of either lifecycled commit snapshots or a non-lifecycled, incrementally updated HEAD. It's a sibling of since and retain since it dictates whether those get used, and I don't think it loses anything by being outside of the index section which currently just describes where the data lives, not how indexing behaves. And it doesn't collide with Elasticsearch's own index.mode setting. Overall, I think this naming convention works.

[davemoore-]

One last semantic change pushed: I think the public-facing values for mode should be "snapshot" and "head" (changed from "incremental" to "head"), so that it's more obvious to an agent what the ref "is." Since the term only appears in the config file and to an agent as a ref marker, I think it's more important to the agent to see that a ref either represents a historical snapshot or the current head of a branch, rather than how the ref was indexed.

[davemoore-]

Note: I might walk back the choice of "head" before we release this and return to "incremental" (or "delta" for brevity) to future-proof the feature to support fast-moving tags later (e.g. Kibana's deploy@ tags), which wouldn't fit the "head" metaphor.

[davemoore-]

Update: I was able to test and confirm that delta mode works as expected for tag refs.

I tested this on the elastic/kibana repo using a match pattern of "deploy@{major}".

1. Before testing it, I patched the index command to exclude the most recent tag that matched that pattern.
2. Then I ran the index command, which backfilled elastic/kibana for the second-most recent "deploy@{major}" tag.
3. Then I removed my patch and re-ran the index command. As expected, only the files and lines that were added/changed/removed in the most recent tag were indexed, and then git.commit and git.ref were correctly updated to their latest values in the ref marker in sourcerer-v3-refs (as well as everything else that should have been updated either permanently or temporarily: status, indexed_at, indexing_started_at, git.commit_target).
4. I repeated the test by patching the index command to exclude the two most recent tags, indexing the backfill, removing the patch, running index one more time, and confirmed that the deltas for those two commits were added to the backfill. All as expected.

I did observe that it took a couple minutes for the deltas to even begin to be indexed, which is unlike what I've seen for branch refs indexed in delta mode. I suspect that's because there were many more deltas to resolve in my tests of elastic/kibana, which had several thousand files added/removed/changed rather than just the small handful that changed on my branch ref test. I'd rather optimize that later and release the working feature now, given that indexing tag refs in delta mode will be the least common combination of git.ref_type and mode.

[davemoore-]

Quite a bit has changed since this PR was opened, so I want to summarize my understanding of where it is now.

How things worked before this PR

All sources defined in sourcerer.yml had their refs indexed as whole commit snapshots. This made it possible to search the entire, materialized state of a repository at a given commit point. It works very well for searching the state of "shipped" code at specific release tags, and can be easily lifecycled using retention policies that prune old commits as they reach a certain age, count, or semantic version.

It also works for indexing the commit points of branches. But branches tend to have different behaviors than release tags. That leads us to...

The problem before this PR

Indexing whole commit snapshots makes it difficult to keep up with branches that have frequent, small updates. For example, documentation repos are frequently updated on their default branch, even for changes as small as fixing a typo. A change that small would still trigger the indexing of another whole commit snapshot of that repo - just to reflect that one typo change. This is an unreasonable amount of processing and storage for such a small change, especially for a repo where the head of the default branch is typically all that matters to search, rather than specific points in its history. It would be better to keep a single snapshot of those repos and then add/update/remove just the files and lines that changed from the latest indexed commit to the most recent commit (i.e. their deltas). IaC has the same behavior: frequent, small updates to a branch where the current state usually matters more than its history.

How this PR solves the problem

This PR introduces the concept of a source "mode" (defined in sourcerer.yml as sources[i].mode) to allow sources to be indexed either as immutable commit snapshots, or as a mutable snapshot whose files and lines are added/updated/removed based on the deltas between commits.

Here's how the two modes work:

1. snapshot mode (sources[i].mode: "snapshot") (the default mode):

   • When a new ref matches sources[i].match, the index command indexes all the files and lines of the ref at that commit point.
   • The ref's files and lines are immutable and addressed by git.host > git.org > git.repo > git.commit.
   • The ref is lifecycled (they support sources[i].since and sources[i].retain). Old commit snapshots can be pruned by age, count, or semantic version.
   • A source can be indexed in snapshot mode if its git.ref_type is "tag", "branch", or "commit".

2. delta mode (sources[i].mode: "delta"):

   • When a new ref matches sources[i].match, the index command indexes only the differing files and lines between the ref at that commit point and the current state of the indexed ref. This also means that a whole snapshot of the repo is indexed; but unlike snapshot mode, that one snapshot is updated incrementally over time, rather than indexing an entirely new snapshot for each new revision of the ref.
   • The ref's files and lines are mutable and addressed by git.host > git.org > git.repo > git.ref_pattern > git.ref_type (where git.ref_pattern is the value of sources[i].match in sourcerer.yml). They do not contain git.commit or git.ref, because those can change between indexing runs and we want to minimize the number of documents that are updated and tombstoned for each indexing run. Instead, the latest git.commit and git.ref is kept in sourcerer-v3-refs and joined at query time, which has proven to be performant in our tests.
   • The ref is not lifecycled (it doesn't support sources[i].since or sources[i].retain because there's no history to manage; there's only ever one snapshot, and it changes over time).
   • A source can be indexed in delta mode if its git.ref_type is "branch" or "tag" (not "commit" which is immutable by definition and therefore cannot have deltas). We expect that "branch" will be a much more common use case for delta mode, but there were valid-enough reasons to support "tag" as well.

The Agent Builder ES|QL tools have been modified to support both new modes seamlessly, without requiring an agent to know if it's searching content that was indexed in snapshot mode or delta mode. Agents can still search broadly across repositories or selectively by commit hashes or ref names, and use wildcards ("*") on any git_* scoping parameter. Our implementation of subqueries, forks, and joins correctly resolves the files and lines to their respective commits and ref names, regardless of whether the files or lines have the shape of snapshot or delta, and does so in a performant way. The main trade-off of this approach is that it requires Elastic v9.5.0+ to make use of a new subquery syntax in ES|QL (specifically IN ( FROM ... )), which is also in tech preview.

Ultimately, I believe this PR successfully refactors Sourcerer to support two very different modes of source curation, while keeping storage, indexing throughput, and query speeds optimal; and keeping tool interfaces intuitive to agents. There's room to make further optimizations in ways that wouldn't incur breaking changes from the current state of this PR. Therefore, I feel comfortable merging this PR and releasing it as v3.0.0, and postponing any non-breaking optimizations to future minor/patch releases.

[davemoore-]

LGTM, validated with lots of offline testing. Great working with you @simianhacker