Skip to content

feat(go): add BlobDescriptor batch and stream readers - #761

Draft
XiaoHongbo-Hope wants to merge 8 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/go-blob-descriptor-reader
Draft

feat(go): add BlobDescriptor batch and stream readers#761
XiaoHongbo-Hope wants to merge 8 commits into
apache:mainfrom
XiaoHongbo-Hope:codex/go-blob-descriptor-reader

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Aug 28, 2026

Copy link
Copy Markdown
Contributor

Summary

  • expose BlobDescriptor batch and lazy stream reads through Rust, C, and Go
  • reuse descriptor validation, FileIO, range merging, and shared I/O limits
  • preserve batch input order and return Go-owned buffers
  • let table readers reuse DLF FileIO and refresh temporary data tokens

Scope

This reads serialized descriptors directly and does not require scanning a Paimon table. Static storage options are not rotated; table.NewBlobReader() retains the table's refreshing FileIO.

This PR does not add a BLOB format, StarRocks dependency, write path, or Blob View support.

Go API

reader, err := paimon.NewBlobReader(storageOptions)
reader, err := table.NewBlobReader()

values, err := reader.ReadBlobs(descriptors)
stream, err := reader.OpenBlob(descriptor) // io.ReadSeekCloser

ReadBlob delegates to ReadBlobs. Reader and stream Close operations are idempotent. OpenBlob is lazy and supports descriptor-relative seeks for HTTP Range responses; independent streams are not range-merged.

Validation

  • Rust Blob tests: 119 passed
  • C binding tests: 67 passed
  • Go BlobReader/BlobStream tests and go vet passed
  • DLF token rotation tests cover batch and stream reads
  • read-only DLF regression E2E sought 32 KiB into a real table BLOB and returned only the requested 1 KiB
  • rustfmt, clippy, and diff checks passed

The PR remains draft while the core and binding changes are prepared for review.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title feat(go): add standalone BlobDescriptor reader feat(go): add BlobDescriptor batch and stream readers Aug 29, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant