feat(drivers): add huggingface cloud storage driver#2799
Open
WUHINS wants to merge 3 commits into
Open
Conversation
实现了HuggingFace仓库作为存储驱动的完整功能,支持模型/数据集/空间三种仓库类型,支持代理配置、LFS大文件上传、目录创建和删除等操作,同时在drivers/all.go中注册该驱动。
重新导入三个编写过程中因意外丢失的存储驱动
清理仓库中不再需要的.tmp/CloudPaste和.tmp/OpenList-Frontend临时子模块
Author
已改正,感谢提醒 |
Contributor
|
HF仓库已经支持S3访问了,数据集还不支持 |
Author
所以主要是支持模型和数据集,只是对接API的话顺手做全可能更好,毕竟小白不一定知道HF支持S3,S3配置也有点难度。 |
Author
我看官方说法是桶支持,但标准的模型、数据集几乎不支持 |
Contributor
确实,不过我试了一下,桶目前是和数据集和模型共享存储空间的 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary / 摘要
实现了 HuggingFace 仓库作为 OpenList 存储驱动的完整功能。
用户可感知的变化
HuggingFace驱动类型,支持模型(model) / 数据集(dataset) / 空间(space) 三种仓库repo_id(必填)、api_token、ref(分支/标签/commit SHA,默认 main)、repo_type(默认 model)、hf_proxy(代理地址)重要实现变化
新增 5 个文件:
drivers/huggingface/driver.go(核心逻辑)、meta.go(驱动元数据)、types.go(API 类型)、util.go(工具函数),以及在drivers/all.go注册LFS 上传流程:saveStream 写临时文件 + SHA256 → preupload 检测 → LFS batch API 获取 S3 presigned URL → PUT 直传 S3(支持 fallback 到 POST)→ verify(可选)→ NDJSON commit 提交 lfsFile 引用
小型文件流式提交:io.Pipe + base64 编码 + NDJSON 内嵌,避免写临时文件
自定义 HTTP 客户端:
s3Client()单例共享连接池(5 分钟 ResponseHeaderTimeout),hfClient()独立客户端(DisableKeepAlives)用于 HF APIcontext.WithoutCancel保护 S3 上传 body 不被任务取消中断过滤未签名的
x-amz-*头部,避免 S3 403(X-Amz-SignedHeaders=host)saveStream 增加
fi.Size() != stream.GetSize()校验,提前捕获数据不完整This PR has breaking changes.
/ 此 PR 包含破坏性变更。
This PR changes public API, config, storage format, or migration behavior.
/ 此 PR 修改了公开 API、配置、存储格式或迁移行为。
This PR requires corresponding changes in related repositories.
/ 此 PR 需要关联仓库同步修改。
Related repository PRs / 关联仓库 PR:
Related Issues / 关联 Issue
无
Testing / 测试
go build -o OpenList.exe .— 编译通过,生成 ~124MB 可执行文件Checklist / 检查清单
/ 我已阅读 CONTRIBUTING。
/ 我确认此贡献符合仓库许可证、贡献规范和行为准则。
gofmt,go fmt, orprettierwhere applicable./ 我已按适用情况使用
gofmt、go fmt或prettier格式化变更代码。/ 我已在适用情况下请求相关维护者或代码所有者审查。
AI Disclosure / AI 使用声明
/ 此 PR 包含 AI 辅助内容。
Tools used / 使用工具:
Usage scope / 使用范围:
Code generation / 代码生成
Refactoring / 重构
Documentation / 文档
Tests / 测试
Translation / 翻译
Review assistance / 审查辅助
I have reviewed and validated all AI-assisted content included in this PR.
/ 我已审核并验证此 PR 中的所有 AI 辅助内容。
I have ensured that all AI-assisted commits include
Co-Authored-Byattribution./ 我已确保所有 AI 辅助提交都包含
Co-Authored-By归属信息。I can reproduce all AI-assisted content included in this PR without any AI tools.
/ 我可以在没有任何 AI 工具的情况下重现此 PR 中包含的所有 AI 辅助内容。
如果需要我调整措辞或增减内容,直接说。