Skip to content

feat(backup): support selective backups with bounded streaming - #10151

Open
RC-CHN wants to merge 8 commits into
AstrBotDevs:masterfrom
RC-CHN:feat/selective-backups
Open

RC-CHN wants to merge 8 commits into
AstrBotDevs:masterfrom
RC-CHN:feat/selective-backups

Conversation

@RC-CHN

@RC-CHN RC-CHN commented Sep 20, 2026

Copy link
Copy Markdown
Member

Full backups currently bundle every component, and restoration offers no way to recover only part of an instance. This adds explicit export and restore scopes while bounding JSON processing for large database dumps.

Modifications / 改动点

  • Add component selection to the backup API and WebUI, covering the main database, knowledge bases, configuration, attachments, plugin directories/data, and the other existing backup directories. Omitted API component selections retain the full-backup behavior.

  • Group export and restore selections into Main Data, Plugins & Extensions, and Temporary Files, with expandable descriptions, group checkboxes, partial-selection states, and counts. Export selects the first two groups by default; temporary files are opt-in. Restore selects all available items, explains unavailable entries, and summarizes the data to be replaced. Refine typography, spacing, checkbox alignment, and mobile layouts.

  • Verify selected components before modifying data, including entry and component checksums, missing entries, JSON structure, and model validation. Keep main database replacement and KB metadata replacement transactional; clear KB files only after the metadata transaction succeeds.

  • Stream database dumps with ijson, bound ZIP metadata/expanded size and JSON records, and move export JSON work and compression into worker threads. Validate raw JSON before materializing containers and preserve NUL/UTF-8 text during bounded database reads.

  • Reject overlapping backup/restore tasks. Show component progress, warnings, and restored scope after partial failures; derive KB manifest inventory from entries actually written.

  • Update OpenAPI and generated client types, all four UI locales and English diagnostics. Add ijson>=3.4,<4 to both Python dependency files and update the English/Chinese WebUI guides.

  • This is NOT a breaking change. / 这不是一个破坏性变更。

Compatibility is not unconditional: selective archives require an importer supporting component selection, and the new resource limits can reject oversized legacy backups. Limits include 8 MiB per JSON record, nesting depth 64, 2 GiB for main/KB metadata dumps, 256 MiB per KB document dump, 8 MiB for the manifest, and 32 GiB total expanded archive size. Import verification intentionally runs synchronously and can temporarily delay dashboard responses.

Screenshots or Test Results / 运行截图或测试结果

image image

Automated verification:

  • Backup regression suite (tests/test_backup.py and tests/test_backup_resources.py): 165 tests passed, including targeted reruns after updating diagnostic-message assertions. Three existing warnings remain (audioop deprecation and two chunk-upload cleanup coroutine warnings).
  • ruff format ., ruff check ., and git diff --check: passed.
  • Dashboard vue-tsc --noEmit: passed; all four changed locale JSON files parsed successfully.
  • Dashboard Node regression tests: 55 tests passed.
  • Headless Chromium checks with mocked backup APIs: export/restore payloads, group and partial selection, keyboard expansion, empty-selection and incompatible-version guards, and unavailable entries passed. All four locales were checked at mobile widths, including dark mode; checkbox and text alignment was measured at desktop and mobile widths.
  • MDI font subset generation: passed. Generated CSS/font assets are excluded from this PR and regenerated by the existing build script.
  • cd docs && pnpm run docs:build: passed, with a bundle-size warning.

Regression coverage includes selective/legacy restoration, preflight rejection before writes, database rollback, oversized archive/JSON rejection, bounded streaming with native and Python parser backends, large integers, NUL/Unicode round trips, export cancellation, single-task admission, and restoring database plus attachments in two parsing passes.

The Settings → Maintenance entry point is unchanged. The English and Chinese WebUI guides now describe the three expandable groups, export defaults, unavailable restore items, replacement scope, attachment dependencies, and import responsiveness. These guide sections contain no screenshots to replace.


Checklist / 检查清单

  • 😊 If there are new features added in the PR, I have discussed it with the authors through issues/emails, etc.
  • 👀 My changes have been well-tested, and verification steps and test results have been provided above.
  • 📚 I checked the affected WebUI instructions in docs/zh and docs/en and updated them in this PR. No navigation entry was renamed, moved, or merged, so an old-to-new mapping is not applicable.
  • 🤓 New dependencies have been added to the appropriate locations in requirements.txt and pyproject.toml.
  • 😮 My changes do not introduce malicious code.

Validate selected components before restoration, bound archive and JSON processing, and reject overlapping backup tasks. Stream database dumps and move export JSON processing into worker threads.

Include the API contract, generated client types, English diagnostic messages, and regression coverage. Validation: 165 backup tests passed; Ruff and git diff checks passed.
Allow users to choose backup and restore components, show dependency warnings and stage progress, and retain restored scope and warnings on failure. Update the API wrapper and all four locale files.

Validation: vue-tsc --noEmit, locale JSON parsing, and git diff checks passed.
Regenerate the tracked MDI subset from current sources to include the checkbox-multiple-marked and database-refresh icons used by the backup dialog.
Include ijson in pip requirements as well as pyproject.toml, and document selective backup and restoration in both WebUI guides. Validation: docs build, Ruff, and dependency specification checks passed.

@sourcery-ai sourcery-ai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry @RC-CHN, your pull request is larger than the review limit of 150,000 diff characters

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant