Skip to content

refactor: real pagination for app/dataset/skill list APIs - #7451

Closed
ctlaltlaltc wants to merge 1 commit into
labring:mainfrom
ctlaltlaltc:refactor/list-api-pagination
Closed

refactor: real pagination for app/dataset/skill list APIs#7451
ctlaltlaltc wants to merge 1 commit into
labring:mainfrom
ctlaltlaltc:refactor/list-api-pagination

Conversation

@ctlaltlaltc

@ctlaltlaltc ctlaltlaltc commented Aug 4, 2026

Copy link
Copy Markdown
Collaborator

背景与问题

/core/app/list/core/dataset/list/core/ai/skill/list 三个核心资源列表接口是伪分页:DB 层全量 find(无 skip/limit),分页(如果有)在内存 slice 完成;且最终可见集合依赖内存权限计算(tmb 直授 + 成员组/组织展开 + 目录 inheritPermission 继承 + hasReadPer 过滤),countDocuments 无法直接对「权限过滤后」的集合计数。

方案简述

  1. 请求内先算可读集合getMyResourcePermission 复用现有权限拉取 + 组/组织展开,新增 readableDirectIds(按资源聚合角色位 tmbRole ?? groupAndOrgRole,复用对应 Permission 类判定 read 位),纯内存可计算量;
  2. 三层权限谓词$or: [{_id ∈ readableDirectIds}, {tmbId}(创建者), {type ∉ folder, inheritPermission, parentId ∈ readableDirectIds}(一层继承)] —— 由位运算分配律证明与内存过滤逐条等价(继承判定中父权限为直接权限、非递归,无需 $graphLookup);
  3. count 与 find 严格同 matchfind(match).sort({updateTime:-1,_id:-1}).skip(offset).limit(pageSize)countDocuments(match) 并行,同一 match 常量 + 同一读偏好,total == 权限过滤后可见集合数量,杜绝计数漂移;
  4. 分页参数统一:pageSize/offset/pageNum 三键、offset 与 pageNum 互斥(superRefine)、numbers-only 严格校验,私有 parseV2Pagination 解析(?? 语义正确处理 offset=0);
  5. 页内后处理:Per/private/hasInteractiveNode/withAppCount 仅对当前页计算;sourceMember 缺成员占位不丢项(status: null);
  6. 大 $in fail-fast:可读集合超阈值返回明确错误码;
  7. 索引补齐:app/dataset/skill 追加 8 条 defineIndex(含 _id 决胜键、store 创建者单键索引)。

向后兼容

  • 旧接口契约完全不动:三个旧路由的入参出参 schema、前端调用(web/core/{app,dataset,skill}/api.ts)逐字节零改动(git diff --exit-code 验收);旧路由文件、被 ChatAgentHelper 复用的 manage/list.ts、公共 parsePaginationRequest 均零改动;
  • 新增 v2 端点 POST /core/{app,dataset,ai/skill}/listV2,返回 {list, total},前端迁移字段形状零适配;
  • 行为差异契约(v2 为权限引擎语义,均为「显示更多权限可见资源」,无越权):① 无 parentId 模式跨目录的无记录创建者/继承资源;② searchKey 模式同上;③ searchKey+parentId 的目录外资源;④ app owner 无 parentId 从「仅根目录」改为全团队;⑤ orphan(缺成员 / tmbId=null 的 system skill)保留 + 占位 sourceMember(旧接口丢项)。

验证与测试工作

自动化测试(53 个全部通过)

  • helper 单测:角色聚合 ?? 优先级(permission=0 压过组 read)、组/组织授权、OwnerRoleVal、缺成员占位;parseV2Pagination(缺省/offset=0/互斥);
  • schema parse:负数/NaN/字符串/Infinity/超上限被拒、offset+pageNum 互斥、ObjectId/null 双形态、system skill tmbId:null、sourceMember.status:null;
  • 旧/v2 对账:预期相等(含 permission=0 + parent read 反例)与五项差异「预期不相等」断言;total > list.length 分页用例;fail-fast 阈值用例;
  • typecheck(projects/app 0 错误,service/global 无新增)、eslint 全绿。

端到端验证(chrome-devtools 驱动真实浏览器)

  • dev server 运行最新代码,页面登录后调用三资源 v2 vs 旧接口:dataset/skill 可见集合完全一致;app 差 2 项(子目录资源,符合差异契约 ④,无漏报);
  • 分页 pageSize:10 两页 total 恒定、零重叠、total > list.length;
  • 边界:负数/101/字符串/offset+pageNum/NaN 全部 400 拒绝;
  • 控制台无页面 JS 错误,截图留存。

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Coverage Report

Status Category Percentage Covered / Total
🔵 Lines 24.65% 2211 / 8967
🔵 Statements 24.65% 2325 / 9431
🔵 Functions 20.92% 456 / 2179
🔵 Branches 20.03% 1024 / 5112
File CoverageNo changed files found.
Generated in workflow #2084 for commit 5c38641 by the Vitest Coverage Report Action

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Admin Preview Image Ready!

ghcr.io/labring/fastgpt-pr:admin_5c38641aed06106548255d549f5d6b3f42e364b8

🕒 Time: 2026-08-07 16:05:30 (UTC+8)

@ctlaltlaltc
ctlaltlaltc requested a review from c121914yu August 4, 2026 15:36
@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Docs Preview Deployed!

🔗 👀 Click here to visit preview

ghcr.io/labring/fastgpt-docs-pr:9dd19509d0c33f809083d5b20009e29c01f9d6c4

🕒 Time: 2026-08-05 17:44:14 (UTC+8)

@github-actions

github-actions Bot commented Aug 4, 2026

Copy link
Copy Markdown

Build Successful - Preview fastgpt Image for this PR:

ghcr.io/labring/fastgpt-pr:fastgpt_5c38641aed06106548255d549f5d6b3f42e364b8

🕒 Time: 2026-08-07 16:08:04 (UTC+8)

@ctlaltlaltc
ctlaltlaltc marked this pull request as draft August 5, 2026 01:22
@ctlaltlaltc
ctlaltlaltc force-pushed the refactor/list-api-pagination branch from 26e2b4e to 871b891 Compare August 5, 2026 01:53
@ctlaltlaltc
ctlaltlaltc marked this pull request as ready for review August 5, 2026 01:53
@ctlaltlaltc
ctlaltlaltc requested a review from FinleyGe August 5, 2026 02:21
@ctlaltlaltc
ctlaltlaltc marked this pull request as draft August 5, 2026 08:37
@FinleyGe FinleyGe self-assigned this Aug 5, 2026
@ctlaltlaltc
ctlaltlaltc force-pushed the refactor/list-api-pagination branch from 7f06959 to 9dd1950 Compare August 5, 2026 09:41
@ctlaltlaltc
ctlaltlaltc marked this pull request as ready for review August 5, 2026 09:42
@ctlaltlaltc
ctlaltlaltc marked this pull request as draft August 6, 2026 01:31
@ctlaltlaltc
ctlaltlaltc force-pushed the refactor/list-api-pagination branch 2 times, most recently from 4e11578 to 366a970 Compare August 7, 2026 07:48
@ctlaltlaltc
ctlaltlaltc force-pushed the refactor/list-api-pagination branch from 366a970 to 5c38641 Compare August 7, 2026 07:56
@ctlaltlaltc
ctlaltlaltc marked this pull request as ready for review August 7, 2026 08:25
@c121914yu c121914yu closed this Aug 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants