This repository was archived by the owner on Oct 30, 2024. It is now read-only.

This repository was archived by the owner on Oct 30, 2024. It is now read-only.

enhance: unpack archives before ingestion #100

Open

opened

on Aug 30, 2024

Currently, handling errors coming from ingesting documents within an archive can get pretty nasty.
Also, the way we do it right now cannot be parallelized for speedup and first collects all documents/chunks before doing the embedding. Missed opportunity for concurrency here and highly error-prone.

Metadata

Assignees

No one assigned

Labels

No labels

No labels

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests