tarutil: buffer the tar stream and pool the copy buffer

[Chenyi Wang (chw120)]

Archiving and restoring a durable dir runs inside the VM's pause window, so its cost is actor downtime. Two mechanical inefficiencies dominated it, both independent of how much data the dir actually holds.

A tar is a sequence of 512-byte blocks, and the stream was written straight to the file, so every header, short file, and padding tail became its own write(2): 120904 of them for a 30k-file tree, against 31 once a 1 MiB buffer sits in between. The reader is buffered for the same reason.

The larger cost was allocation. io.Copy honours a WriterTo on src or a ReaderFrom on dst before it looks at any buffer, and *os.File implements both. Neither fast path can complete here -- the other end is a tar stream, not a file, so sendfile/copy_file_range do not apply -- and the generic fallback each one drops into allocates a fresh 32 KiB buffer. That is one allocation per entry on both the archive and the restore side: 965 MiB of garbage and 287 collections for a 30k-file tree, all of it swept while the actor is frozen. copyPooled masks both interfaces so io.CopyBuffer actually uses the pooled buffer it is handed.

Measured on a 30000-file, 30874624-byte tree (linux/arm64, 9 repeats):

             before              after
Create    274 ms  964.6 MiB   165 ms  28.6 MiB   GC 287 -> 12
Extract   583 ms  979.1 MiB   490 ms  44.7 MiB   GC 279 -> 19

Output is unchanged: the archive hashes identically to the one origin/main produces (sha256 6f524c45...), and so does a re-archive of the extracted tree. The existing round-trip tests for xattrs, device nodes, FIFOs, ownership, and special mode bits all pass.

It's a good idea to open an issue first for discussion.

• Tests pass
• Appropriate changes to documentation are included in the PR

[Benjamin Elder (BenTheElder)]

overall looks like a good improvement, thanks.

Worth noting: if there are any sparse files in the durabledir (from something the workload is doing), we'll wind up densifying them by wrapping tr in readerOnly, we probably don't have a use case yet but we might have to sort that out at some point.

The 1MiB bufio per tar may get expensive if we have a lot of concurrent suspend with durdir in the multi-actor future.

[Benjamin Elder (BenTheElder)]

should we be pooling the bufio buffers?

[Chenyi Wang (chw120)]

Done, both come from a pool now.

[Chenyi Wang (chw120)]

overall looks like a good improvement, thanks.

Worth noting: if there are any sparse files in the durabledir (from something the workload is doing), we'll wind up densifying them by wrapping tr in readerOnly, we probably don't have a use case yet but we might have to sort that out at some point.

Thanks Benjamin Elder (@BenTheElder) for reviewing.

From my learning, archive/tar doesn't write GNU sparse extensions, so there's no way to represent a hole in the archive however the bytes arrive; and neither os.File.WriteTo nor ReadFrom punches holes — their fast paths only engage when both ends are files, which never happens on this path. Sparse files were already densified on main. The archive hashing identically before and after is the direct evidence.

Looks it'd be SEEK_HOLE/SEEK_DATA on create and hole punching on extract, which is separate from this PR.

The 1MiB bufio per tar may get expensive if we have a lot of concurrent suspend with durdir in the multi-actor future.

It turned out 1 MiB wasn't buying anything. Updated to 64 KiB now, 16× less per archive in flight.

One thing that fell out of the smaller size: at 64 KiB the stream buffer is below copyBufPool's 128 KiB, and bufio.Writer hands a write straight to the file when it's larger than the whole buffer. So file contents now bypass the stream buffer instead of being copied through it, and the buffer only batches the headers and padding it was added for. On a large-file tree (40 × 8 MiB) that's 8× more write(2) than 1 MiB but 320 MiB less memcpy, and it measures slightly faster. Both constants now carry a comment noting the sizes are coupled.

[Benjamin Elder (BenTheElder)]

thanks, I suppose we may need some kind of IFTT presubmit someday

[Benjamin Elder (BenTheElder)]

nit: document the current state, not the previous state. this is a bad behavior claude has this comment is way longer than useful, and it takes a lot of time to read the code when we do this.

[Benjamin Elder (BenTheElder)]

Suggested change

	// copyPooled is io.Copy with a reused buffer.
	//
	// The interface masking is load-bearing, not decoration. io.CopyBuffer honors
	// a WriterTo on src or a ReaderFrom on dst before it ever looks at the supplied
	// buffer, and *os.File implements both. Neither fast path can complete here —
	// the other end is a tar stream rather than a file, so sendfile/copy_file_range
	// do not apply — and the generic fallback each one drops into allocates a
	// buffer of its own. Hiding the two methods keeps the copy on the path that
	// actually uses the pooled buffer.
	// copyPooled masks the fast-path interfaces so io.CopyBuffer uses the pooled buffer.

the implementation shows which methods are masked

[Benjamin Elder (BenTheElder)]

Suggested change

	// copyBufPool holds the scratch buffers used to stream file contents. io.Copy
	// allocates a fresh 32 KiB buffer per call (see copyPooled), and a durable dir
	// holds tens of thousands of files, so the garbage — 965 MiB for a 30k-file
	// tree — and the collections it forces are paid inside the VM's pause window.
	//
	// The size is not free to change: it must stay above streamBufSize, or content
	// stops bypassing the stream buffer (see the note there).
	var copyBufPool = sync.Pool{New: func() any {
	b := make([]byte, 128<<10)
	return &b
	}}
	const copyBufSize = 128 << 10
	// File contents must bypass the stream buffer.
	const _ = uint(copyBufSize - streamBufSize - 1)
	var copyBufPool = sync.Pool{New: func() any {
	b := make([]byte, copyBufSize)
	return &b
	}}

The compile-time check captures the size constraint; the variable name already explains the pool.

[Benjamin Elder (BenTheElder)]

Suggested change

	// tarWriterPool and tarReaderPool hold the stream buffers above. These are one
	// per archive rather than one per file, so on their own they save far less
	// garbage than copyBufPool does; pooling them keeps a worker that checkpoints
	// several actors at once reusing a handful of buffers instead of allocating a
	// fresh one per suspend. A buffer must be Reset(nil) before it goes back, or
	// the pooled entry pins the closed *os.File it was last bound to.
	var (
	tarWriterPool = sync.Pool{New: func() any { return bufio.NewWriterSize(nil, streamBufSize) }}
	tarReaderPool = sync.Pool{New: func() any { return bufio.NewReaderSize(nil, streamBufSize) }}
	)
	// Stream buffers must be Reset(nil) before pooling to avoid retaining files.
	var (
	tarWriterPool = sync.Pool{New: func() any { return bufio.NewWriterSize(nil, streamBufSize) }}
	tarReaderPool = sync.Pool{New: func() any { return bufio.NewReaderSize(nil, streamBufSize) }}
	)

[Benjamin Elder (BenTheElder)]

Suggested change

	// streamBufSize buffers the tar stream against the underlying file. A tar is a
	// sequence of 512-byte blocks, so an unbuffered stream turns every header,
	// every short file, and every padding tail into its own write(2): archiving
	// 30k small files issued 120904 of them, against one per 64 KiB of archive
	// once buffered.
	//
	// 64 KiB rather than something larger: the win is in getting the syscall count
	// off the critical path, and that is spent well before this size. Sweeping
	// 16 KiB..4 MiB over a 30k-file tree is flat to within run-to-run noise, so a
	// bigger buffer buys nothing measurable and only adds memory — which is held
	// per archive in flight, and a worker may checkpoint several actors at once.
	//
	// Staying under copyBufPool's buffer is deliberate, not incidental. bufio hands
	// a write straight to the file when it is larger than the whole buffer, so file
	// contents — which arrive in copyBufPool-sized chunks — bypass this buffer
	// instead of being copied through it, and only the headers and padding it
	// exists for are batched. Raising this above 128 KiB, or shrinking copyBufPool
	// below it, silently puts every content byte back through a memcpy.
	const streamBufSize = 64 << 10
	// streamBufSize batches tar headers and padding into fewer file operations.
	// It must be smaller than copyBufSize so file contents bypass the buffer.
	const streamBufSize = 64 << 10