RL Systems Bench

Reproducible synthetic end-to-end benchmarks for ART and SkyRL's native asynchronous reinforcement-learning pipelines. Four workloads share pinned models, forced completion lengths, and an explicit throughput measurement contract. Each run uses two H200s for training and two for inference.

Run The Benchmark

The controller needs Linux, Python 3.12, uv, and NVIDIA's nvidia-smi. Local Docker runs additionally need Docker with NVIDIA Container Toolkit. Direct Kubernetes image execution does not require Docker on the controller. Each run uses four H200 GPUs: two for training and two for inference.

Clone the repository, build both backend images, and run the complete comparison:

git clone https://github.com/OpenPipe/rl-systems-bench.git
cd rl-systems-bench
./bench plan --suite comparison
./bench gpus
./bench build --backend art
./bench build --backend skyrl
./bench run --suite comparison --gpus 0,1,2,3

The comparison runs all four workloads on each backend, serially, and stops on failure. To select individual runs or use separate GPU sets on an eight-GPU host:

./bench run --backend art --workload multi_turn --gpus 0,1,2,3
./bench run --backend skyrl --workload multi_turn --gpus 4,5,6,7

The last two commands can run in separate terminals/agents on an eight-GPU machine. GPU indices are host physical indices; exact UUIDs from bench gpus are accepted too. Four distinct devices are required. The launcher reserves UUID-keyed locks on the host before starting work, rejects overlapping leases, and checks NVIDIA compute processes and memory independently. It never kills other users' jobs or silently chooses a different topology.

If a supervisor is killed or cleanup cannot be confirmed, its persistent reservation record blocks reuse even when no CUDA context is currently visible. Inspect the recorded run and owned processes before manually clearing such a record. There is deliberately no automatic stale-lock override.

Reservations coordinate this launcher on one host. They are not a cluster scheduler and cannot prevent arbitrary external programs from starting later. Run the controller on the host, not separately inside isolated containers. Use the same host account for agents. A machine with only four GPUs must run these benchmarks serially. CPU, memory bandwidth and filesystem contention can still affect disjoint-GPU runs; retain their manifests when comparing rates.

Native Environments

For environments provisioned by scripts/setup_art.sh and setup_skyrl.sh, activate the matching wrapper around the host controller. Substitute the actual absolute prefixes and CUDA 12.8 toolkit path:

env RL_BENCH_ENV_ROOT=/absolute/fresh/art_env CUDA_HOME=/usr/local/cuda-12.8 \
  bash containers/art_env.sh ./bench run --runtime native --backend art --workload multi_turn \
  --python /absolute/fresh/art_env/art/.venv/bin/python --gpus 0,1,2,3
env RL_BENCH_ENV_ROOT=/absolute/fresh/sky_env \
  bash containers/skyrl_env.sh ./bench run --runtime native --backend skyrl --workload multi_turn \
  --python /absolute/fresh/sky_env/skyrl/.venv/bin/python \
  --skyrl-source /absolute/fresh/sky_env/skyrl --gpus 0,1,2,3

Install this package into each backend environment first. Native runs require the same compiler/runtime setup as their container recipe; the launcher does not repair or mutate a pre-existing environment. Use separate native commands for each backend rather than a single interpreter for the suite. --hf-cache /path/to/huggingface reuses a downloaded model snapshot in either runtime. It denotes the HF_HOME directory containing hub/, not an arbitrary model-weight directory. The pinned model revision is still enforced. On a qualified older-driver host requiring SkyRL's packaged NVIDIA compatibility libraries, add --skyrl-cuda-compat; for native execution also add SKYRL_USE_CUDA_COMPAT=1 to the env command so the wrapper activates the libraries before starting the controller. --python alone does not activate either backend's runtime paths or ART wheel provenance. See containers/README.md for the exact driver/toolchain contract; the launcher does not change host drivers.

Workloads

CLI Name	Workload
multi_turn	Four trajectories/group, two historical turns and three new model calls; variable prompt/completion lengths and tool observations
long_generation	Four trajectories/group, one new call; completion lengths 512-4096 tokens
tool_history	Eight trajectories/group, six historical turns, one new call; large tool schema/history and short completions
multi_turn_salted	Same multi-turn workload with per-call conditioning/cache salting to remove substantial shared prefixes

The workload source contains the exact piecewise-quantile distributions and seeding rules. Completion length is forced and checked on every call. These are synthetic systems workloads, not task-quality evaluations. The salted case permits ordinary packing but removes substantial shared-prefix reuse.

Measurement Contract

• Qwen3.5-35B-A3B, BF16, per-expert LoRA with rank 8 and alpha 32.
• Both backends explicitly request enable_thinking=False and preserve_thinking=True. ART retains these in native _internal_config, alongside chat_template_tool_schema_format="vllm_openai" and init_args.max_seq_length=131072. Per-request forced-length options retain the model's chat-template defaults through the native OpenAI client merge.
• ART uses native PipelineTrainer, online autotuning, in-flight updates and routing replay; TP1 CP2 EP2, packed target 131072, mean policy age limit 4.
• SkyRL uses its native fully asynchronous pipeline, B4/workers12/staleness4, TP1 CP1 EP2 DP2 training. --skyrl-batch-size 1 selects B1/workers3.
• --rank 1 is supplemental. It does not select the same native grouped adapter path as rank 8 and must be reported separately.
• Five warmup plus fifteen measured policy/publication updates. These are not necessarily twenty optimizer invocations. Headline rates use updates 16-20, measured from completion 15 to completion 20; all 15 are also retained.
• E2E completion includes native publication and train-log completion, packing, cleanup and offload/reload. Actual accepted trajectories and assistant tokens are counted, not offered work.
• Trainer rates use native intervals with cross-rank interval union, not the sum of concurrent durations or extra CUDA synchronizations.
• Native acceptance, objectives, adapter layouts and trainer scopes differ. This is not exact replay, numerical parity or an isolated compute speedup.

Artifacts And Pins

Each invocation creates a fresh directory under results/, prints its path, and retains manifest.json, run.log, per-update evidence and summaries. Failures remain recorded. Run validation checks completed updates, counts, endpoints and trainer ranks; ART also checks autotuner health.

Container preflight compares the image's benchmark Python and baked recipe hashes against the controller; missing or stale recipe identity fails before GPU reservation. requested_recipe_sha256 records controller recipes. For native runs this is not proof of executed setup: retain the prefix's setup_manifests/ alongside run evidence, rather than inferring setup history from live source.

ART is pinned to 2ebfc1c2a38dbeeaeefac998323b476989602942. SkyRL is pinned to 52063b6567a350e02d326699cd7d0f380bbad466. The model revision is 59d61f3ce65a6d9863b86d2e96597125219dc754.

ART retains the canonical model ID for its support registry. A run-local model directory symlink binds native tokenizer/processor lookups to the same immutable snapshot used by training and inference. Its single-host packing process verifies that binding; keep the snapshot and run directory available until the run exits.

The Docker build/run commands do not publish images.

Kubernetes Images

Kubernetes can run the images without Docker inside the pod. Build and push an image from this checkout using your own registry tooling. Record its immutable linux/amd64 manifest digest and generate the expected source identity from the same unchanged checkout:

uv run python -c 'import json; from rl_pipeline_bench.cli import ROOT, recipe_manifest, source_manifest; print(json.dumps(dict(source_sha256=source_manifest(), recipe_sha256=recipe_manifest(ROOT))))' > source.json

The external controller verifies the pod UID and actual imageID before releasing the benchmark and again after completion. Baked Python/recipe hashes must match the externally mounted source record. Only data/cache/provenance are mounted; dependencies and benchmark code are not replaced at runtime.

Supply your Kubernetes context, writable PVC, and HF cache directory relative to that PVC. Use a fresh output directory and unique pod name for each invocation:

uv run python scripts/qualify_kubernetes.py \
  --backend art --name rl-bench-art-test \
  --image REGISTRY/IMAGE@sha256:MANIFEST_DIGEST \
  --source-provenance source.json --output results/kubernetes_art \
  --context YOUR_CONTEXT --pvc YOUR_PVC --cache-subpath caches/huggingface

Supply --pull-secret when the registry requires authentication and your cluster does not already provide it. --namespace, --scheduler, --node-selector and --runtime-class select cluster-specific placement. The namespace and PVC must already exist. The cache directory must contain the HF_HOME hub/ layout. SkyRL uses --backend skyrl; use --skyrl-cuda-compat only when the packaged compatibility libraries are appropriate for the selected host.

Each pod runs all four workloads serially, then exits. Kubernetes allocates four GPUs; launcher locks operate inside the pod, not across arbitrary pods. Artifacts are collected even when the controller and target do not share a filesystem. Completed pod records are retained. --dry-run prints the requested pod without launching it; --resume observes the same saved pod and never reruns workloads. A failure never triggers an automatic benchmark rerun.