feat: Milestone 2 — Structured Run Manifest System (#64) by DhanashreePetare · Pull Request #67 · dbpedia/databus-python-client

DhanashreePetare · 2026-06-30T19:29:12Z

Pull Request

Description

Introduces the Structured Run Manifest System (Milestone 2) for the Databus Python Client. When --manifest is passed to any of the three existing commands (download, deploy, delete), a JSON-LD manifest file is written recording the complete details of the operation — input parameters, per-file URLs, checksums, byte sizes, timestamps, success/failure status, and a structured execution summary.
The manifest uses the DataID vocabulary (the same vocabulary used by the Databus platform itself), is versioned via dbus:schemaVersion, and never writes sensitive credentials. If the manifest file already exists at the given path, it auto-suffixes (run_1.jsonld, run_2.jsonld, ...) with a warning rather than silently overwriting. Passing a directory path instead of a file path raises a clear error. Manifest writing failure warns and continues — the exit code reflects the actual operation result, not the manifest write.
The manifest is written even when the operation itself fails, capturing whatever partial results were recorded before the failure — enabling the debugging use case described in the proposal (Use Case 5).

Related Issues
#64

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to change)
This change requires a documentation update
Housekeeping

Checklist:

My code follows the ruff code style of this project.
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation (if applicable)
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
- poetry run pytest - all tests passed
- poetry run ruff check - no linting errors

…sons

…-client into gsoc-2026

coderabbitai · 2026-06-30T19:29:19Z

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 62b470be-e962-4d90-b4e9-d76f4c04531b

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

Create PR with unit tests

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

_{Comment @coderabbitai help to get the list of available commands.}

Integer-Ctrl · 2026-07-03T08:08:58Z

+```
+
+## Manifest
+
+All three commands support an optional `--manifest` flag that writes a structured JSON-LD record of the operation to disk:
+
+```bash
+databusclient download https://databus.dbpedia.org/dbpedia/generic/labels/2023.12.01 \
+  --manifest ./manifests/labels-download.jsonld
+
+databusclient deploy --version-id https://databus.dbpedia.org/myaccount/mygroup/mydata/1.0 \
+  --title "My Dataset" --abstract "..." --description "..." \
+  --license https://creativecommons.org/licenses/by-sa/3.0/ \
+  --apikey YOUR_KEY --manifest ./manifests/deploy-run.jsonld \
+  myfile.nt
+
+databusclient delete https://databus.dbpedia.org/myaccount/mygroup/mydata/1.0 \
+  --databus-key YOUR_KEY --manifest ./manifests/delete-run.jsonld
+```
+
+The manifest records input parameters, per-file URLs, checksums, byte sizes, timestamps, and success/failure status for each file. It uses the DataID vocabulary and is versioned via `dbus:schemaVersion`.
+
+- If the target path already exists, the manifest is written to an auto-suffixed path (e.g. `run_1.jsonld`) with a warning.
+- Sensitive fields (API keys, vault tokens) are never written.
+- If manifest writing fails, a warning is printed and the exit code reflects the actual operation result.
+
+See `examples/reproducible-download.md` for a full walkthrough.


The placement in the REAMDE could be better. At the top of the README is Table of Contents. There I would sort it in under:

- [CLI Usage](#cli-usage) - [Download](#cli-download) - [Deploy](#cli-deploy) - [Delete](#cli-delete) - [Manifest](#cli-manifest) <-- new

Accordingly the doc for Manifest should be placed below cli-delete

Integer-Ctrl · 2026-07-03T08:14:59Z

+# Explicitly un-ignore the manifest module folder (MANIFEST above is for Python packaging artifacts)
+!databusclient/manifest/
+!databusclient/manifest/**
+databusclient/manifest/__pycache__/
+*.py[cod]


Looks like code from agent :D
Unless there is a reason to keep it, remove it. Moreover *.py[cod] is already present on line 8, and __pycache__/ on line 7

Actually, these lines were added to fix a Windows-specific gitignore conflict — the MANIFEST pattern on line 33 (for Python packaging artifacts) was case-insensitively matching databusclient/manifest/ on Windows, preventing the manifest module directory from being committed. The negation lines were the workaround at the time but now since the files are now tracked by git, the conflict no longer applies and I've removed them. Thanks for pointing this out truly.

Integer-Ctrl · 2026-07-03T08:33:35Z

+All three commands support an optional `--manifest` flag that writes a structured JSON-LD record of the operation to disk:
+
+```bash
+databusclient download https://databus.dbpedia.org/dbpedia/generic/labels/2023.12.01 \


Please use existing examples: https://databus.dbpedia.org/dbpedia/generic/labels/2023.12.01 does not exist

Integer-Ctrl · 2026-07-03T09:25:52Z

    if queue is not None:
        queue.add_uri(databusURI)
        return


Real delete manifests are empty for successful deletions.

databusclient/api/delete.py:152 queues every non-dry-run delete and returns before recording. The public delete() always uses a queue, but DeleteQueue.execute() calls _delete_list() without passing
manifest_context at databusclient/api/delete.py:73. Result: databusclient delete ... --manifest out.jsonld can delete resources successfully while the manifest reports zero files.

{ "@context": { "dataid": "http://dataid.dbpedia.org/ns#", "dcat": "http://www.w3.org/ns/dcat#", "dcterms": "http://purl.org/dc/terms/", "xsd": "http://www.w3.org/2001/XMLSchema#", "dbus": "http://databus.dbpedia.org/manifest/ns#" }, "@type": "dbus:OperationManifest", "dbus:schemaVersion": "1.0", "dbus:clientVersion": "0.15", "dbus:command": "delete", "dcterms:issued": { "@value": "2026-07-03T09:15:02.095679+00:00", "@type": "xsd:dateTime" }, "dbus:replayParams": { "databusURIs": [ "https://databus.dev.dbpedia.link/fhofer/group1/artifact1/2027-07-03" ], "dry_run": false }, "dataid:distribution": { "@type": "dataid:Distribution", "dataid:file": [] }, "dbus:executionResult": { "@type": "dbus:ExecutionSummary", "dbus:totalFiles": 0, "dbus:succeeded": 0, "dbus:failed": 0, "dbus:totalBytes": 0 } }

Updated, verified by deleting a deployed test dataset.

Integer-Ctrl · 2026-07-03T09:32:30Z

+    if manifest_context is not None:
+        manifest_context.record_file(
+            url=url,
+            status="success",
+            sha256=actual_checksum or expected_checksum,
+            size_bytes=total_size_in_bytes if total_size_in_bytes else None,
+            downloaded_at=datetime.now(timezone.utc).isoformat(),
+        )


Download manifests can record a file as successful before the requested conversion succeeds.

_download_file() writes the success entry at databusclient/api/download.py:526, but compression/format conversion runs afterward through databusclient/api/download.py:537. If convert_file() or recompression fails later, the manifest still contains a successful file entry for an operation whose final output was not produced

DhanashreePetare added 25 commits June 5, 2026 15:28

gsoc26: Layer 2 with tests initial commit

7b4c635

gsoc26: Refactor Layer 2 with handler architecture and improved tests

5d224f6

Improvements to Format layer implementation

f2fe92e

Review comments resolved under issue dbpedia#59

180c255

dbpedia#61: replacing --convert_from & --convert_to with --compression

34039f2

gsoc26: layer2 complete + implementation for dbpedia#61

cd5e990

gsoc2026: mapping layer initial commit

508bf86

fix: errors from testing fixed.

9aba4f6

docs: update README for --format and --compression flags

549ea3b

docs: update README for --format and --compression flags

2c50924

fix: resolve PR review comments.

4ff66d9

Merge branch 'feature/format-conversion' into gsoc-2026

9a541ee

fix: remove duplicate import after merging layer2 fixes into layer3

6c0200e

docs: add Layer 3 mapping conversion flags and examples

d5db670

merge: sync with upstream/gsoc-2026 after PR dbpedia#62 merge

cac241f

Initial commit: Manifest System

5c905f5

fix: move all test data to tests/resources, add round trip IR compari…

6a3f728

…sons

fix: align Quad->Triple->Quad round trip test as required.

092702a

fix: fixed review comments

3185c6b

fix: fixed review comments(2)

9948f58

Merge branch 'gsoc-2026' of https://github.com/dbpedia/databus-python…

1836d50

…-client into gsoc-2026

Merge branch 'gsoc-2026' into dev

6a9c5e1

Complete Implementation of Milestone 2

9bfcc21

edge case handled

ebe0fe4

docs: add Manifest section to README documenting --manifest flag

393aa6c

DhanashreePetare requested a review from Integer-Ctrl June 30, 2026 19:30

feat: capture operation-level errors in manifest via dbus:operationError

485bd41

Integer-Ctrl requested changes Jul 3, 2026

View reviewed changes

fix: resolve PR review comments

9363657

DhanashreePetare requested a review from Integer-Ctrl July 4, 2026 11:37

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

feat: Milestone 2 — Structured Run Manifest System (#64)#67

feat: Milestone 2 — Structured Run Manifest System (#64)#67
DhanashreePetare wants to merge 27 commits into
dbpedia:gsoc-2026from
DhanashreePetare:gsoc-2026

DhanashreePetare commented Jun 30, 2026

Uh oh!

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading

Review skipped

Uh oh!

Integer-Ctrl Jul 3, 2026

Uh oh!

DhanashreePetare Jul 4, 2026

Uh oh!

Integer-Ctrl Jul 3, 2026

Uh oh!

DhanashreePetare Jul 4, 2026

Uh oh!

Integer-Ctrl Jul 3, 2026

Uh oh!

DhanashreePetare Jul 4, 2026

Uh oh!

Integer-Ctrl Jul 3, 2026

Uh oh!

DhanashreePetare Jul 4, 2026

Uh oh!

Integer-Ctrl Jul 3, 2026

Uh oh!

DhanashreePetare Jul 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

DhanashreePetare commented Jun 30, 2026

Pull Request

Description

Type of change

Checklist:

Uh oh!

coderabbitai Bot commented Jun 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Review skipped

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

coderabbitai Bot commented Jun 30, 2026 •

edited

Loading