Skip to content

Refactor disk index build structure#1271

Open
wuw92 wants to merge 7 commits into
mainfrom
wuw92/refactor-disk-build-structure-main
Open

Refactor disk index build structure#1271
wuw92 wants to merge 7 commits into
mainfrom
wuw92/refactor-disk-build-structure-main

Conversation

@wuw92

@wuw92 wuw92 commented Jul 22, 2026

Copy link
Copy Markdown
Contributor

Why

This is a follow-up to #1254. Removing checkpoint and continuation support eliminated the workflow state that previously justified much of the disk-build indirection, allowing the remaining build path to be simplified and its module ownership clarified.

Module changes

  1. Rename core.rs to merged_index.rs; the renamed module retains the merged-build implementation and RAM estimation.
  2. Move shared builder fixtures and end-to-end disk-build tests from the renamed module to tests.rs.
  3. Move IndexBuildStrategy and determine_build_strategy from the renamed module to build.rs, where the one-shot versus merged decision is made.
  4. Move the single in-memory graph construction pipeline from build.rs to inmem_builder.rs.
build.rs
├── one-shot ────────────────> inmem_builder.rs
└── merged ──> merged_index.rs ──> inmem_builder.rs

wuw92 added 3 commits July 21, 2026 11:08
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
- Moved disk index builder tests to a new module structure for better organization.
- Updated import paths in `disk_provider.rs` to reflect the new location of `IndexBuildFixture` and `TestParams`.
- Enhanced the test suite for disk index building, including additional test cases and improved parameter handling.
Copilot AI review requested due to automatic review settings July 22, 2026 07:19

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR refactors the diskann-disk disk index build path to clarify module ownership: merged-index building logic is separated from top-level orchestration, and the in-memory single-graph build pipeline is consolidated behind an internal entry point, while keeping behavior unchanged.

Changes:

  • Extract merged-index build implementation (partition/shard build/merge/cleanup) into a dedicated merged_index module.
  • Move the in-memory build pipeline into inmem_builder and call it from the top-level disk builder.
  • Centralize builder test fixtures under build::builder::tests and update test imports accordingly.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 1 comment.

Show a summary per file
File Description
diskann-disk/src/search/provider/disk_provider.rs Updates test fixture import path to the new builder test module location.
diskann-disk/src/build/builder/tests.rs Introduces centralized builder test fixtures and helpers under build::builder::tests.
diskann-disk/src/build/builder/quantizer.rs Updates documentation wording for build quantizer types.
diskann-disk/src/build/builder/mod.rs Adjusts module layout (adds merged_index, exposes tests as pub(crate) under cfg(test)).
diskann-disk/src/build/builder/merged_index.rs New module containing merged-index build implementation and RAM estimation tests.
diskann-disk/src/build/builder/inmem_builder.rs Consolidates the one-shot in-memory build pipeline behind an internal entry point.
diskann-disk/src/build/builder/build.rs Keeps top-level orchestration and policy; wires in merged_index + inmem_builder entry points.
diskann-disk/src/build/builder/core.rs Deletes the previous combined “core” module after refactor.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread diskann-disk/src/build/builder/tests.rs Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
Copilot AI review requested due to automatic review settings July 22, 2026 07:37

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated no new comments.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
Copilot AI review requested due to automatic review settings July 22, 2026 07:46
@wuw92
wuw92 marked this pull request as ready for review July 22, 2026 07:47
@wuw92
wuw92 requested a review from a team July 22, 2026 07:47

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Comment thread diskann-disk/src/build/builder/mod.rs
Comment thread diskann-disk/src/build/builder/tests.rs
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
Copilot AI review requested due to automatic review settings July 22, 2026 09:35

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated 3 comments.

Comment thread diskann-disk/src/build/builder/merged_index.rs Outdated
Comment thread diskann-disk/src/build/builder/merged_index.rs Outdated
Comment thread diskann-disk/src/build/builder/merged_index.rs Outdated
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

Copilot-Session: ecf6179f-7467-44be-8a77-bf7a96e319af
Copilot AI review requested due to automatic review settings July 22, 2026 11:20

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 9 out of 9 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants