Apply the neighbour heuristic incrementally on full back-links (1.4–2.9× faster build) - #28
Merged
Merged
Conversation
Every insert creates up to M0 back-links, and once the graph has warmed
up the target list is almost always full. The old code handled that by
re-running the complete selection heuristic over all M0 + 1 candidates:
~2,000 distance computations per back-link, ~130,000 per insert, each a
3 KiB memory-mapped read at 768 dimensions. Profiling the Cohere 100K
build put 66 % of insert time there.
AddNeighborConnection now does what Lucene's findWorstNonDiverse does:
only the new node is unchecked, so an existing neighbour is compared
against the new node alone and the new node against the neighbours
nearer than itself. O(M0) distances instead of O(M0^2).
Lucene's fallback of evicting the farthest candidate assumes the list
only holds mutually diverse nodes. Ours does not, because
SelectNeighborsHeuristic tops up with pruned candidates, and applying
the fallback blindly left 7 of 200 nodes unreachable in an existing
Euclidean test. When the incremental walk finds nothing redundant the
code falls back to the full heuristic instead; instrumented on clustered
data that happens on 0.1-0.7 % of full back-links.
SelectNeighborsHeuristic drops LINQ for a stable index sort with
identical ordering.
The graph is not byte-identical to the old one, so recall is re-measured
rather than assumed. Same machine, same seeds, old graph rebuilt the same
day for the SIFT-1M comparison:
siftsmall ~1,000-1,500 -> 2,485 inserts/s, recall unchanged
Cohere 100K 1,696 s -> 589 s (59 -> 170 inserts/s), recall@100 -0.3 pp
SIFT-1M 2,279 s -> 1,631 s (439 -> 613 inserts/s),
recall@10 -1.5 pp at efSearch 10, -0.3 at 80, 0 at 320
Query throughput at a given efSearch is unchanged within run-to-run
noise on both datasets.
New tests: NeighborDiversityTests pins the heuristic's required outcome
on hand-checkable 2-D/3-D constructions plus a well-formedness sweep,
and passes against both implementations. RawGraph is shared out of
GraphFanoutTests.
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Replaces the O(M0²) re-run of the neighbour-selection heuristic on every full back-link with Lucene's incremental
findWorstNonDiverse(O(M0)), falling back to the full heuristic only when the new node makes nothing redundant (0.1–0.7 % of cases — needed because our lists are topped up with pruned candidates, unlike Lucene's; without it 7/200 nodes went unreachable in an existing test). Also removes LINQ fromSelectNeighborsHeuristic. Design doc:docs/design-insert-prune.md.Measurements (same machine, same seeds)
QPS at a given efSearch is unchanged within ±5 % run-to-run noise on both datasets (measured back to back with
--reuse-index). The README SIFT-1M table is updated with the new graph's numbers and states the trade-off.Tests
NeighborDiversityTests: three hand-checkable geometric cases (reject redundant new node / evict the neighbour it makes redundant / evict farthest when all diverse) + well-formedness sweep under all three metrics. They pass against both the old and new implementation — they pin the heuristic, not the shortcut.RawGraphshared out ofGraphFanoutTests.