feat: add ZipFileBuilder for parallelizing compression - #911
Conversation
Compresses a file's contents independently of any ZipWriter, so that multiple files can be compressed on separate threads and then appended serially with ZipWriter::add_prepared_file. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
Adds a new “prepare then append” workflow to support parallelizing per-file compression work outside of ZipWriter, while still producing correct ZIP metadata when appending the pre-compressed payloads.
Changes:
- Introduces
ZipFileBuilderto compress a single file entry into memory while tracking CRC-32 and sizes, producing aPreparedZipFile. - Adds
ZipWriter::add_prepared_fileto append aPreparedZipFileby copying its already-compressed bytes verbatim (including handling ZIP64 header patching and stream/data-descriptor mode). - Adds tests covering roundtrip reading, duplicate-name rejection (and writer recovery), stream mode behavior, and encryption rejection.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Its-Just-Nans
left a comment
Its-Just-Nans
left a comment
There was a problem hiding this comment.
Hi,
Thanks for the MR, a little question in comment.
Also, what do you think about adding a test with a thread (like the documentation)?
| crc32: u32, | ||
| uncompressed_size: u64, | ||
| compressed_size: u64, | ||
| data: Vec<u8>, |
There was a problem hiding this comment.
Why not using a Box<[u8]> ?
and change line 2526
-let data = cursor.into_inner();
+let data = cursor.into_inner().into_boxed_slice();
|
I originally had a test case with some threads (similar to the doc comment), but I removed it on the theory that the borrow checker meant it had to work :-) Happy to add it back. |
Compresses a file's contents independently of any ZipWriter, so that multiple files can be compressed on separate threads and then appended serially with ZipWriter::add_prepared_file.