feat(core): add pluggable semantic vector indexes#1141
Conversation
Signed-off-by: phernandez <paul@basicmachines.co>
![chatgpt-codex-connector[bot]](https://avatars.githubusercontent.com/in/1144995?s=60&v=4){kind=small}
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 0470b7d9db
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
Resolved the failed semantic diagnostic job in signed commit |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 367a2f782a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d780c16f28
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 8fa2182b0c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 29b738901a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 3a9dda503c
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: fdc75d7f6a
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d3cf2133d1
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 4c5e3a0c0b
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 67f27ddc9e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d9aa17f768
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Signed-off-by: phernandez <paul@basicmachines.co>
|
@codex review |
|
Codex Review: Didn't find any major issues. More of your lovely PRs please. Reviewed commit: ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
If Codex has suggestions, it will comment; otherwise it will react with 👍. Codex can also answer questions or update the PR. Try commenting "@codex address that feedback". |
1 participant
Why
Milvus support should not require moving Basic Memory's full-text search, embedding generation,
or SQL metadata into a vendor-specific repository. The contributor work in #1041 demonstrated the
need for a clean extension boundary, while also highlighting that an in-tree Milvus dependency
would create Windows compatibility and long-term ownership costs.
This PR establishes that boundary in core so Milvus can be delivered and maintained as an external
package. It also makes the vector lifecycle explicit enough to support future multimodal embedding
providers without coupling those providers to vector storage.
What Changed
SemanticVectorIndexprotocol for vector initialization, persistence, deletion,reconciliation, and nearest-neighbour lookup.
pgvectorandsqlite-vecimplementations behind the same contract.basic_memory.semantic_vector_indexes, allowing anexternal package to provide
semantic_vector_index = "milvus".migration and backend-neutral project embedding status.
adapter boundary.
Implementation Details
Core continues to own text chunking, embedding generation, and the authoritative SQL manifest.
Vector adapters receive already-computed vectors identified by stable
(entity_id, chunk_key)keys and own only vector storage and similarity search.
Writes first persist manifest rows as
pending, perform adapter I/O outside the SQL transaction,and then mark the matching index/model rows
ready. Search results are hydrated through current,ready manifest rows for the active project and embedding identity, so stale or cross-project
external matches fail closed.
SQLite always resolves to the built-in
sqlite-vecadapter. Postgres defaults topgvectorandmay select an installed extension by name. Missing, duplicate, incompatible, or incorrectly scoped
extensions fail explicitly instead of silently falling back.
This does not include a Milvus implementation. The intended follow-up is for the contributor and
Zilliz team to carry the backend in a separately versioned package built on this contract. Feedback
from @zc277584121 on whether this supports the package split proposed in #1041 would be especially
helpful.
Testing
Automated
git show --check --oneline --no-renames HEAD: passed.just fast-check: passed; type checking reported nine existing Python 3.14 asyncio deprecationwarnings.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q tests/repository/test_pgvector_index.py tests/repository/test_postgres_search_repository_unit.py tests/repository/test_semantic_vector_index.py tests/repository/test_semantic_vector_sync.py tests/repository/test_sqlite_vector_search_repository.py tests/services/test_project_service_embedding_status.py tests/services/test_semantic_search.py tests/test_semantic_vector_index_migration.py test-int/test_embedding_status_vec0.py: 93 passed.BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 just doctor: passed, including real localembedding generation and sqlite-vec indexing.
BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 uv run pytest -q --no-cov -m semantic test-int/semantic/test_search_diagnostics.py::test_similarity_formula_analysis: passed after updating the diagnostic to assert the adapter's normalized similarity contract instead of reading backend-native distances.BASIC_MEMORY_ENV=test LOGFIRE_IGNORE_NO_CONFIG=1 just fast-test: two selected local testspassed and one test skipped; six selected semantic integration cases errored during fixture
setup because Docker is not running locally, so no local Postgres result is claimed.
Manual
signed off.
Risks / Follow-ups
first external Milvus package validates it.
this PR makes vector storage replaceable but does not add such a provider.