Skip to content

[held-out-eval] Add minimal CLI for non-Python pipeline integration #56

Description

@Rook-CodeVolt

Problem

held-out-eval is Python-API-only. Someone integrating it into a shell-script-based or non-Python-orchestrated training pipeline (common in real ML workflows: bash + one Python trainer script + no orchestration framework) has no way to register/check held-out sets without writing custom Python glue code. This narrows who can practically adopt the package to people already comfortable embedding Python calls into their pipeline.

Fix

Add a minimal CLI (e.g. held-out-eval register-train <package_id> <ids-file>, held-out-eval register-held-out <package_id> <ids-file>, held-out-eval check <ids-file>, held-out-eval show) that operates against a registry JSON file on disk (using the existing save()/load()), so the entire workflow is scriptable from bash/CI without any Python glue.

Keep it minimal and stdlib-only (argparse), consistent with the package's zero-runtime-dependency design goal — do not pull in click/typer/etc. unless there's a strong reason to.

Acceptance criteria

  • held-out-eval --help documents available subcommands.
  • Registering train/held-out ids and checking for contamination is fully doable from the command line against a JSON file on disk, with no Python code required.
  • CLI exit codes are meaningful (0 = no contamination, non-zero = contamination detected or error) so it's usable directly in a CI/bash pipeline's exit-code checks.
  • README gets a short CLI usage example alongside the existing Python API example.
  • Test coverage for each subcommand (success path + at least one failure/contamination-detected path).

Priority

Backlog / build only if real adoption interest confirms this is needed — do not block on this before publishing (see PyPI-publish issue), since a working Python API is more important to ship first.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions