Problem
held-out-eval is Python-API-only. Someone integrating it into a shell-script-based or non-Python-orchestrated training pipeline (common in real ML workflows: bash + one Python trainer script + no orchestration framework) has no way to register/check held-out sets without writing custom Python glue code. This narrows who can practically adopt the package to people already comfortable embedding Python calls into their pipeline.
Fix
Add a minimal CLI (e.g. held-out-eval register-train <package_id> <ids-file>, held-out-eval register-held-out <package_id> <ids-file>, held-out-eval check <ids-file>, held-out-eval show) that operates against a registry JSON file on disk (using the existing save()/load()), so the entire workflow is scriptable from bash/CI without any Python glue.
Keep it minimal and stdlib-only (argparse), consistent with the package's zero-runtime-dependency design goal — do not pull in click/typer/etc. unless there's a strong reason to.
Acceptance criteria
held-out-eval --help documents available subcommands.
- Registering train/held-out ids and checking for contamination is fully doable from the command line against a JSON file on disk, with no Python code required.
- CLI exit codes are meaningful (0 = no contamination, non-zero = contamination detected or error) so it's usable directly in a CI/bash pipeline's exit-code checks.
- README gets a short CLI usage example alongside the existing Python API example.
- Test coverage for each subcommand (success path + at least one failure/contamination-detected path).
Priority
Backlog / build only if real adoption interest confirms this is needed — do not block on this before publishing (see PyPI-publish issue), since a working Python API is more important to ship first.
Problem
held-out-evalis Python-API-only. Someone integrating it into a shell-script-based or non-Python-orchestrated training pipeline (common in real ML workflows: bash + one Python trainer script + no orchestration framework) has no way to register/check held-out sets without writing custom Python glue code. This narrows who can practically adopt the package to people already comfortable embedding Python calls into their pipeline.Fix
Add a minimal CLI (e.g.
held-out-eval register-train <package_id> <ids-file>,held-out-eval register-held-out <package_id> <ids-file>,held-out-eval check <ids-file>,held-out-eval show) that operates against a registry JSON file on disk (using the existingsave()/load()), so the entire workflow is scriptable from bash/CI without any Python glue.Keep it minimal and stdlib-only (argparse), consistent with the package's zero-runtime-dependency design goal — do not pull in click/typer/etc. unless there's a strong reason to.
Acceptance criteria
held-out-eval --helpdocuments available subcommands.Priority
Backlog / build only if real adoption interest confirms this is needed — do not block on this before publishing (see PyPI-publish issue), since a working Python API is more important to ship first.