Support running experiments fully locally (no CloudLab dependency)

[milindsrivastava1997]

Currently experiment_run_e2e.py and experiment_run_clickhouse.py require a CloudLab cluster — create_provider() (experiment_utils/providers/factory.py) is hardcoded to always return CloudLabProvider, and config validation requires cloudlab.username/cloudlab.hostname_suffix.

Goal: add a provider that runs an experiment entirely on a single dev machine, no CloudLab account/cluster needed at all.

Note on CloudLabLocalProvider: there's already a cloudlab_local.py provider, but it's for a different use case — running locally after SSHing into a CloudLab node, still using CloudLab paths/conventions (CLOUDLAB_HOME_DIR, etc). It's not a dev-machine-independent provider as-is, though some of its no-SSH execute_command logic may be reusable as a starting point.

What's already reusable:

• The InfrastructureProvider base interface (providers/base.py) — execute_command, execute_command_parallel, get_node_ip, get_home_dir, get_query_log_file — is provider-agnostic and most services call it correctly.
• factory.py already has commented-out scaffolding for provider-type detection (detect_provider_type), just needs wiring up.

Suggested approach: introduce a new LocalProvider (distinct from CloudLabLocalProvider) implementing the base interface for single-machine execution — get_node_ip → 127.0.0.1, get_home_dir → some local scratch dir, file "sync" → no-op since everything's already on one filesystem — then fix the sync.py/services rsync bypass to go through the provider instead of hardcoding SSH.

Scope refinement (post-grilling)

Implementation landed with a narrower, more concrete scope than the original checklist:

• Single local node only (local.home_dir=..., equivalent to cloudlab.num_nodes=0 — coordinator only, node_offset hardcoded to 0). No multi-node local cluster, no SSH-to-localhost.
• Streaming engines precompute and arroyo only — Flink dropped from scope. flink.py's CLOUDLAB_HOME_DIR sites are intentionally untouched.
• Both bare-metal and containerized (Docker) deployment paths, for both entrypoints (experiment_run_e2e.py and experiment_run_clickhouse.py).
• New top-level local: config section (added via Hydra's +local.home_dir=/path syntax, not a config-group file) replaces the originally-suggested experiment_type=local approach — checked with hasattr(cfg, "local") before any cfg.cloudlab.* access, since those fields are Hydra ??? mandatory-missing markers that raise on mere attribute access otherwise.
• Provisioning the actual binaries (Prometheus, Arroyo, asap-summary-ingest, query_engine_rust, fake_exporter) into the local home dir is out of scope — separate manual/setup concern.
• Filed follow-up issue Move CloudLab-specific sync logic onto the provider class when a second remote provider is added #435 for moving CloudLab-specific sync logic onto the provider class once a second remote provider is ever added (the is_remote() gate used here is correct for "CloudLab or local," not infinitely extensible).

Checklist

Provider & config wiring

• Add LocalProvider implementing InfrastructureProvider (single-machine: get_node_ip → 127.0.0.1, get_home_dir → configured local.home_dir, is_remote() → False)
• Wire provider-type detection into providers/factory.py (detect_provider_type now checks for local first)
• Decouple experiment_utils/config.py's Args.__init__ from unconditionally reading cfg.cloudlab.num_nodes / node_offset / username / hostname_suffix (new get_node_params()/required_cloudlab_params() helpers, used by both entrypoints)
• Add a local-mode config path (+local.home_dir=..., not experiment_type=local) that skips the cloudlab.* validation requirement

File sync (rsync-over-SSH bypasses the provider)

• experiment_utils/sync.py — all 7 rsync functions now gated on provider.is_remote()
• experiment_utils/services/query_engine.py
• experiment_utils/services/clickhouse_service.py (3 call sites — the dataset-prepare and DDL/script-asset sites fall back to a local copy instead of a no-op, since source and destination genuinely differ there even in local mode)
• experiment_utils/services/docker_victoriametrics.py

Hardcoded constants.CLOUDLAB_HOME_DIR (bypasses provider.get_home_dir())

• experiment_utils/services/flink.py — out of scope for this pass (Flink dropped, see scope refinement above)
• experiment_utils/services/arroyo.py (5 sites)
• experiment_utils/services/fake_exporters.py (2 sites)
• experiment_run_e2e.py / experiment_run_clickhouse.py — experiment_root_output_dir (both entrypoints had this same hardcoding at the top level, not just inside services; found during implementation)

Hardcoded CloudLab IP scheme

• experiment_run_e2e.py — OmegaConf resolver remote_write_ip now routes through provider.get_node_ip, registered after the provider exists (required reordering provider creation ahead of Args(cfg))

Needs audit (not yet scoped) → resolved, not just audited

• experiment_utils/services/prometheus.py reset() — turned out reset_prometheus.py (the CloudLab-only script it shelled out to) just duplicated provider.execute_command(...) with hardcoded SSH; removed the branch entirely rather than adding a provider-type check

Verification

Code-complete; not yet run against a real local experiment (binaries aren't provisioned by this change — see scope note above). Outstanding before closing:

• Run experiment_run_e2e.py locally with local.home_dir=..., streaming_engine=precompute, bare-metal — confirm no ssh/rsync subprocess is ever invoked
• Repeat with streaming_engine=arroyo
• Repeat both with containerized flags (use_container_query_engine, use_container_arroyo, etc.)
• Run experiment_run_clickhouse.py locally
• Confirm PrometheusService.reset() actually deletes local prometheus/data between runs

[milindsrivastava1997]

Follow-up: the CloudLab-coupling is more widespread than the issue body lists. Fuller audit:

SSH-hostname rsync pattern (f"node{node_offset}.{hostname_suffix}" via scp/rsync), beyond sync.py:

• experiment_utils/services/clickhouse_service.py (lines 127, 232, 330 — three separate call sites)
• experiment_utils/services/docker_victoriametrics.py (line 196)
• experiment_utils/services/query_engine.py (already noted)

Direct constants.CLOUDLAB_HOME_DIR use (bypassing provider.get_home_dir()), beyond flink.py:

• experiment_utils/services/arroyo.py (4 sites — paths to asap-summary-ingest and the arroyo release binary)
• experiment_utils/services/fake_exporters.py (2 sites)

Not previously listed at all:

• experiment_utils/config.py (Args.__init__, ~lines 473-476) unconditionally reads cfg.cloudlab.num_nodes, cfg.cloudlab.node_offset, cfg.cloudlab.username, cfg.cloudlab.hostname_suffix. Even after swapping the provider, Args itself throws without a cloudlab: config section — this needs to become provider-agnostic too.
• experiment_utils/services/prometheus.py (~lines 126-131): when the provider has a hostname_suffix attr, shells out to reset_prometheus.py --cloudlab_username ... --hostname_suffix ..., which does its own SSH/rsync internally. reset_prometheus.py itself hasn't been audited yet and likely needs the same treatment.

Net effect: this isn't a one- or two-file fix. The CloudLab home-dir constant and the manual SSH-hostname string are sprinkled across at least 6 service files plus the central Args config class, so anyone picking this up should expect to touch each service file individually rather than fixing it in one or two central places.