fix: robust error handling and NaN clamping in FAISS vector store

[KITE919]

Summary

Fixes #10109: knowledge base upload shows "文档数量和分片数量为 0" with vague error "存储失败：文本块已生成，但写入知识库索引时出错". The real cause — either FAISS C++ crash from malformed embeddings or opaque index-write exceptions — was lost in the catch-all handler.

Changes

astrbot/core/db/vec_db/faiss_impl/vec_db.py

1. Clamp non-finite embeddings before FAISS write: After np.asarray() converts embeddings to float32 matrix, check for NaN/Inf and clamp them to 0.0 to prevent FAISS C++ crashes when embedding providers return malformed results.

2. Structured error handling around FAISS insert: Wrap low-level FAISS errors (index corruption, resource exhaustion, I/O failure) as structured KnowledgeBaseUploadError with cause and traceback in details, so the upload caller can surface actionable diagnostics instead of the generic "写入知识库索引时出错".

astrbot/core/db/vec_db/faiss_impl/embedding_storage.py

3. Early rejection of bad vectors at FAISS boundary: Add isfinite() validation in insert_batch() and search() to reject NaN/Inf vectors before the C++ layer sees them, with a clear error message telling users to check their embedding model config.

Root Cause Analysis

Traced the full pipeline: background_upload_task() → KBHelper.upload_document() → FaissVecDB.insert_batch() → parallel writes to DocumentStorage (SQLite + FTS5) and EmbeddingStorage (FAISS). When FaissVecDB._ensure_vec_db() creates a new IndexFlatL2 index, any NaN/Inf values in embedding vectors pass through numpy's .asarray() but crash FAISS's C++ add_with_ids. The original code only caught generic Exception, rolled back DocumentStorage, but re-raised as an unstructured KnowledgeBaseUploadError without cause or traceback.

Why This Fixes #10109

The user's reported flow (upload Excel → success page → doc count = 0) indicates:

1. Embedding provider returned NaN/Inf vectors: Vector creation succeeded, DocumentStorage committed rows, but FAISS silently corrupted or crashed → rollback deleted everything → doc count remains 0. With NaN clamping, this case now succeeds.

2. FAISS index corruption / resource exhaustion: The existing code threw a generic error hiding the real cause. With structured exceptions, the user gets actionable diagnostics.

Verification

• ruff check passes (only pre-existing E501 line-length warnings, no new lint errors)
• No behavioral change to happy-path uploads (NaN/Inf count is zero when embeddings are correct)
• No unit tests currently cover FaissVecDB.insert_batch() failure paths; future tests should mock an embedding provider that returns bad vectors

Summary by Sourcery

Harden FAISS vector storage against invalid embeddings and surface actionable errors when index writes fail.

Bug Fixes:

• Prevent non-finite embedding and query vectors from reaching FAISS, avoiding crashes or index corruption and providing actionable configuration errors.
• Preserve rollback behavior while exposing structured diagnostics for FAISS storage failures, including the underlying cause and traceback.

Enhancements:

• Clarify storage validation errors when document and chunk ID counts do not match.

Tests:

• Add coverage for rejecting NaN and infinite vectors before document or FAISS storage operations.

[sourcery-ai]

Hey - I've found 2 issues

Prompt for AI Agents

Please address the comments from this code review:

## Individual Comments

### Comment 1
<location path="astrbot/core/db/vec_db/faiss_impl/vec_db.py" line_range="220-221" />
<code_context>
             await self.embedding_storage.insert_batch(vectors_array, int_ids)
-        except Exception:
-            await self._rollback_partial_insert(ids=ids, int_ids=int_ids)
+        except KnowledgeBaseUploadError:
             raise
+        except Exception as _faiss_err:
+            # Low-level FAISS errors (index corruption, resource exhaustion,
</code_context>
<issue_to_address>
**issue (bug_risk):** The internal-ID count mismatch raises `KnowledgeBaseUploadError` inside the transaction-compensation block, but the new `except KnowledgeBaseUploadError` re-raises it without calling `_rollback_partial_insert`. Document rows already committed by `insert_documents_batch` remain orphaned whenever the returned ID count is wrong.

**Triggers:** When `DocumentStorage.insert_documents_batch()` returns fewer or more IDs than the content count.

**Suggested fix:** Roll back partial inserts before re-raising this error, or move the ID-count validation outside the `try` and explicitly compensate on failure.

```suggestion
        except KnowledgeBaseUploadError:
            await self._rollback_partial_insert(ids=ids, int_ids=int_ids)
            raise
```
</issue_to_address>

### Comment 2
<location path="astrbot/core/db/vec_db/faiss_impl/embedding_storage.py" line_range="149-153" />
<code_context>
             )
+        # Validate vector values before FAISS write; non-finite values cause
+        # C++ segfaults or silent index corruption.
+        if not np.all(np.isfinite(vectors)):
+            nan_count = int(np.sum(~np.isfinite(vectors)))
+            await self.save_index()
+            raise RuntimeError(
+                f"向量包含 {nan_count} 个非有限值 (NaN/Inf)，无法写入 FAISS 索引。请检查嵌入模型配置。"
+            )
         if vectors.shape[1] != self.dimension:
</code_context>
<issue_to_address>
**issue (broader_impact):** Finite-value validation is added only to `insert_batch`; `EmbeddingStorage.search()` still passes NaN/Inf query vectors directly to FAISS and returns invalid distances or neighbor IDs instead of rejecting the malformed vector. `EmbeddingStorage.insert()` likewise remains an unvalidated FAISS boundary for single-vector inserts.

**Triggers:** When a query or single-document embedding contains NaN or Inf.

**Suggested fix:** Apply the same `np.isfinite()` validation to `search()` and `insert()` before invoking FAISS.
</issue_to_address>

Sourcery assessment

Needs a human reviewer. 2 findings to address first, and if malformed embeddings are clamped incorrectly, zero-valued vectors can be persisted in the FAISS index and produce bad retrieval results even after reverting the code. The affected index can be rebuilt or the documents re-embedded, so the damage is bounded and repairable.

Blocking findings: astrbot/core/db/vec_db/faiss_impl/vec_db.py:221, astrbot/core/db/vec_db/faiss_impl/embedding_storage.py:153

---

Sourcery is free for open source - if you like our reviews please consider sharing them ✨

• X
• Mastodon
• LinkedIn
• Facebook

[sourcery-ai]

Sourcery assessment

Approved.

[KITE919]

Follow-up fix pushed in 06710a2: non-finite embeddings are now rejected before document/FAISS writes instead of being silently clamped to zero; single insert, batch insert, and search remain guarded, and focused regression tests were added. The unnecessary save_index call on rejected batch input was also removed.