Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
116 changes: 116 additions & 0 deletions src/content/docs/platform/self-hosting/external-orchestrators.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,116 @@
---
title: Running agents with an external orchestrator
description: >-
Connect an external job scheduler to self-hosted agents with the Direct backend
in one-shot mode or the Command backend.
---
import { VARS } from '@data/vars';

Use an existing CI system, Kubernetes controller, or internal scheduler to allocate compute while the {VARS.WARP_AUTOMATION_PLATFORM} routes and tracks each run. The worker supports two patterns for this setup: start a Direct worker with `--one-shot` for each job, or use the Command backend to delegate runs to your runtime.

## Choosing an orchestration pattern

Both patterns keep execution on your infrastructure and require outbound connectivity to Warp.

| Pattern | External orchestrator responsibility | Worker behavior | Use when |
| --- | --- | --- | --- |
| **Direct backend** with `--one-shot` | Starts a worker process and creates a run for its unique worker ID | Runs one task on the allocated host, then exits | Your scheduler allocates a VM, pod, or CI runner for each job |
| **Command backend** | Exposes an API or command that can accept a task payload and start the agent | Stays connected and invokes your dispatch command for each task | Your runtime already has its own job API, queue, or compute lifecycle |

One-shot mode works only with the Direct backend. It forces `max_concurrent_tasks` to `1` and waits for one accepted task's `oz` CLI process to exit before the worker exits. If no task arrives, the worker stays connected until your orchestrator stops it.

The Command backend is fire-and-forget. The dispatch command returns after the external runtime durably accepts the task. The remote agent, not the worker process, reports progress and completion to Warp.

## Running a Direct worker in one-shot mode

This example starts a worker and routes one cloud agent run to it from the same job. Use a unique worker ID so another worker cannot claim the run.

By default, the CLI stays open for 45 minutes after a conversation completes so users can send follow-up prompts. The example sets `--idle-on-complete 0s` so the CLI and worker exit without that idle period. A task-level `config.idle_timeout_minutes` value takes precedence, so leave it unset or set it to `0` for these tasks.

### Prerequisites

* **Self-hosting enabled for your Enterprise team** — [Contact sales](https://www.warp.dev/contact-sales) if self-hosting is not enabled.
* **The worker and CLI binaries** — Install `oz-agent-worker` from a [published release](https://github.com/warpdotdev/oz-agent-worker/releases) and install the {VARS.WARP_AGENT_CLI} by following the [CLI installation instructions](/reference/cli/#installing-the-cli).
* **An agent API key** — Create one in the <a href={`${VARS.WEB_APP_URL}/settings`}>{VARS.WEB_APP}</a>. Store it in your orchestrator's secret manager as `WARP_API_KEY`.

### Start and route the run

Add the following script to the job your orchestrator starts. Replace `CI_JOB_ID` with a unique job identifier from your system.

```bash title="run-agent.sh"
#!/usr/bin/env bash
set -euo pipefail

: "${WARP_API_KEY:?Set WARP_API_KEY in the job environment}"
: "${CI_JOB_ID:?Set CI_JOB_ID to a unique job identifier}"

worker_id="external-${CI_JOB_ID}"

oz-agent-worker \
--worker-id "$worker_id" \
--backend direct \
--one-shot \
--idle-on-complete 0s &
worker_pid=$!

trap 'kill "$worker_pid" 2>/dev/null || true' EXIT

oz agent run-cloud \
--host "$worker_id" \
--prompt "Run the test suite, fix failures, and open a pull request."

wait "$worker_pid"
trap - EXIT
```

The run may enter the queue before the worker finishes connecting. Warp assigns it after the matching worker ID is online. After the conversation completes, the `oz` CLI exits, then the one-shot worker and job exit.

Use Direct backend [setup and teardown commands](/platform/self-hosting/managed-direct/#setup-and-teardown-commands) to prepare the workspace on the allocated host.

## Delegating runs with the Command backend

Use the Command backend when the external runtime owns job creation and cleanup. The worker invokes `dispatch_command` once for each assigned task and writes a versioned JSON payload to standard input.

The payload includes:

* `base_args` — The `oz agent run` argument vector for the external runtime.
* `docker_image` and `sidecars` — The task image and required sidecar mounts.
* `env` — Task environment variables and credentials. Keep this payload out of logs.
* `run_id` and `server_root_url` — Values the agent uses to report status to Warp.

The public [`command-backend` example](https://github.com/warpdotdev/oz-agent-worker/tree/main/examples/command-backend) includes dependency-free Python dispatch and cancellation scripts for an HTTP runtime. Copy those scripts to the worker host, then configure the worker:

```yaml title="worker.yaml"
worker_id: "external-runtime"
backend:
command:
dispatch_command: "python3 /opt/warp/dispatch.py"
cancel_command: "python3 /opt/warp/cancel.py"
dispatch_timeout: "60s"
environment:
- name: OZ_DISPATCH_URL
value: "https://runtime.internal.example.com/agent-runs"
- name: OZ_CANCEL_URL
value: "https://runtime.internal.example.com/agent-runs/cancel"
- name: OZ_DISPATCH_AUTH_HEADER
```

Start the worker with the authentication header and Warp API key supplied by your secret manager:

```bash
export OZ_DISPATCH_AUTH_HEADER="Bearer YOUR_RUNTIME_TOKEN"
export WARP_API_KEY="YOUR_AGENT_API_KEY"

oz-agent-worker --config-file worker.yaml
```

Adapt the example script's `transform()` function to your runtime's request schema. Your runtime must launch `base_args` with the supplied task environment, image, and sidecars. After the CLI exits, it must run `oz harness-support --run-id RUN_ID report-shutdown` so Warp receives the terminal state.

An exit code of `0` from `dispatch_command` means the external runtime accepted responsibility for the task. A nonzero exit or a dispatch timeout fails the task. `max_concurrent_tasks` limits simultaneous dispatch calls, not the number of agents running in the external runtime.

## Related pages

* [Managed: Direct backend](/platform/self-hosting/managed-direct/) — Run agent tasks directly on a worker host.
* [Unmanaged architecture](/platform/self-hosting/unmanaged/) — Invoke `oz agent run` directly when Warp does not need to route the run.
* [Self-hosted worker reference](/platform/self-hosting/reference/) — Look up worker flags and backend configuration fields.
* [Routing runs to self-hosted workers](/platform/self-hosting/#routing-runs-to-self-hosted-workers) — Route runs from the CLI, API, schedules, integrations, or the web app.
48 changes: 22 additions & 26 deletions src/content/docs/platform/self-hosting/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -20,18 +20,18 @@ Self-hosting lets your team run cloud agent workloads on your own infrastructure

Self-hosting has two architectures. The core distinction is **who orchestrates agent runs** — not who owns the compute. Both models keep code and execution on your infrastructure.

* **Managed** — The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates agent runs. You run the `oz-agent-worker` daemon on your infrastructure; it connects to the {VARS.WARP_AUTOMATION_PLATFORM} and waits for work. [Slack](/platform/integrations/slack/) mentions, Linear comments, [schedules](/platform/triggers/scheduled-agents/), API calls, and `oz agent run-cloud` commands all route tasks to your worker, which executes them in isolated Docker containers, Kubernetes Jobs, or directly on the host. Similar to a [GitHub self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners).
* **Managed** — The {VARS.WARP_AUTOMATION_PLATFORM} orchestrates agent runs. You run `oz-agent-worker` on your infrastructure; it connects to the {VARS.WARP_AUTOMATION_PLATFORM} and waits for work. [Slack](/platform/integrations/slack/) mentions, Linear comments, [schedules](/platform/triggers/scheduled-agents/), API calls, and `oz agent run-cloud` commands route tasks to the worker. The Docker, Kubernetes, and Direct backends execute tasks on worker infrastructure. The Command backend dispatches tasks to an external runtime. Similar to a [GitHub self-hosted runner](https://docs.github.com/en/actions/hosting-your-own-runners).
* **Unmanaged** — You orchestrate agent runs. You invoke `oz agent run` directly from your existing CI pipeline, Kubernetes pod, VM, or dev box. The {VARS.WARP_AUTOMATION_PLATFORM} provides session tracking and observability for each run, but does not start or stop agents for you.

### At a glance

| Aspect | **Managed** | **Unmanaged** |
| --- | --- | --- |
| **Who triggers runs** | The {VARS.WARP_AUTOMATION_PLATFORM} (Slack, Linear, schedules, API, `run-cloud`) | Your system (CI, cron, scripts) |
| **What runs on your infra** | Long-lived `oz-agent-worker` daemon | One-shot `oz agent run` invocations |
| **What runs on your infra** | `oz-agent-worker`, either long-lived or started for one externally allocated job | One-shot `oz agent run` invocations |
| **OS support** | Linux (macOS/Windows coming) | Linux, macOS, Windows |
| **Execution isolation** | Docker container, Kubernetes Job, or direct host | Whatever your host provides |
| **Automatic environment setup** | Yes (via Warp [environments](/platform/environments/)) | No (you manage it) |
| **Execution isolation** | Docker container, Kubernetes Job, direct host, or your external runtime | Whatever your host provides |
| **Automatic environment setup** | Docker, Kubernetes, and Direct: yes; Command: the external runtime applies the supplied task configuration | No (you manage it) |
| **Session tracking and steering** | Yes | Yes |

The two architectures are not mutually exclusive. Some teams run managed workers for integration-triggered work and unmanaged agents in CI pipelines. The deployment models diagram on [Deployment patterns](/platform/deployment-patterns/) compares what runs where in each model.
Expand Down Expand Up @@ -69,45 +69,40 @@ Use these questions to decide between managed and unmanaged:
1. **Do you need agents to run on Windows or macOS?**
* Yes → Use the [unmanaged](/platform/self-hosting/unmanaged/) architecture. Managed is Linux-only today.
* No, Linux works → Continue to the next question.
2. **Do you want the {VARS.WARP_AUTOMATION_PLATFORM} to handle starting and stopping agents** (from Slack, the web interface, the Warp app, schedules, or the API)?
* Yes → Use the [managed](#managed-architecture) architecture.
* No, you have your own triggering mechanism → Use the [unmanaged](/platform/self-hosting/unmanaged/) architecture.
3. **Can your development environment run in a Docker container or Kubernetes pod?**
* Yes, Docker → [Managed: Docker](/platform/self-hosting/managed-docker/) backend.
* Yes, Kubernetes → [Managed: Kubernetes](/platform/self-hosting/managed-kubernetes/) backend.
* No (multi-service stacks that don't fit a single container, or environments where container runtimes aren't available) → [Unmanaged](/platform/self-hosting/unmanaged/) or [Managed: Direct](/platform/self-hosting/managed-direct/).
4. **Do you have your own orchestrator** (CI/CD, Kubernetes, internal job scheduler) **that starts agents on demand?**
* Yes → [Unmanaged](/platform/self-hosting/unmanaged/), using `oz agent run` as a drop-in.
* No → [Managed](#managed-architecture).
2. **How should runs start?**
* The {VARS.WARP_AUTOMATION_PLATFORM} should route each run to a worker that manages execution → Use the [managed](#managed-architecture) architecture, then choose a backend below.
* The {VARS.WARP_AUTOMATION_PLATFORM}, with your existing scheduler or runtime owning the compute lifecycle → Use a [managed external-orchestrator pattern](/platform/self-hosting/external-orchestrators/).
* Your system, by invoking `oz agent run` directly → Use the [unmanaged](/platform/self-hosting/unmanaged/) architecture.

### Choosing a managed backend

The managed architecture supports three backends for task execution:
The managed architecture supports four backends for task handling. Docker, Kubernetes, and Direct execute tasks on worker infrastructure. Command dispatches tasks to an external runtime.

1. **Are you deploying the worker into a Kubernetes cluster?**
1. **Should a long-lived worker hand each task to an existing job API, queue, or runtime?**
* Yes → Use the [Command backend](/platform/self-hosting/external-orchestrators/#delegating-runs-with-the-command-backend).
* No → Continue.
2. **Does an external scheduler start a dedicated worker process for each job?**
* Yes → Use the [Direct backend in one-shot mode](/platform/self-hosting/external-orchestrators/#running-a-direct-worker-in-one-shot-mode).
* No → Continue.
3. **Are you deploying the worker into a Kubernetes cluster?**
* Yes → Use the [Kubernetes backend](/platform/self-hosting/managed-kubernetes/). Each task runs as a Kubernetes Job in your cluster; install with the included Helm chart.
* No → Continue.
2. **Is Docker available on your worker host?**
4. **Is Docker available on your worker host?**
* Yes → Use the [Docker backend](/platform/self-hosting/managed-docker/) (default). Tasks run in isolated containers.
* No → Use the [Direct backend](/platform/self-hosting/managed-direct/). Tasks run directly on the host.
3. **Do you need container-level isolation between tasks?**
* Yes → [Docker](/platform/self-hosting/managed-docker/) or [Kubernetes](/platform/self-hosting/managed-kubernetes/) backend.
* No → Any backend works.
4. **Do you need Kubernetes-native scheduling, resource management, or policy enforcement?**
* Yes → [Kubernetes backend](/platform/self-hosting/managed-kubernetes/).
* No → [Docker](/platform/self-hosting/managed-docker/) or [Direct](/platform/self-hosting/managed-direct/) is simpler to set up.

---

## Managed architecture

With the managed architecture, you run the `oz-agent-worker` daemon on your infrastructure. The daemon connects to the {VARS.WARP_AUTOMATION_PLATFORM}'s backend, waits for tasks to be assigned to it, and executes those tasks on its host using one of three backends:
With the managed architecture, you run `oz-agent-worker` on your infrastructure. The worker connects to the {VARS.WARP_AUTOMATION_PLATFORM}'s backend, waits for tasks, and handles them with one of four backends:

* **[Docker backend](/platform/self-hosting/managed-docker/)** (default) — Runs each task in an isolated Docker container.
* **[Kubernetes backend](/platform/self-hosting/managed-kubernetes/)** — Runs each task as a Kubernetes Job in your cluster.
* **[Direct backend](/platform/self-hosting/managed-direct/)** — Runs each task directly on the host without a container runtime.
* **[Command backend](/platform/self-hosting/external-orchestrators/#delegating-runs-with-the-command-backend)** — Dispatches each task to an external runtime through a configured command. The worker does not run the agent on its host.

The managed architecture enables full orchestration by the {VARS.WARP_AUTOMATION_PLATFORM} — it can remotely start agents via Slack, Linear, the <a href={VARS.WEB_APP_URL}>{VARS.WEB_APP}</a>, the API/SDK, and the `oz agent run-cloud` command. Agents can access host resources through volume mounts (Docker), Kubernetes-native configuration (Kubernetes), and injected environment variables.
The managed architecture enables full orchestration by the {VARS.WARP_AUTOMATION_PLATFORM} — it can remotely start agents via Slack, Linear, the <a href={VARS.WEB_APP_URL}>{VARS.WEB_APP}</a>, the API/SDK, and the `oz agent run-cloud` command. Agents can access resources through volume mounts (Docker), Kubernetes-native configuration (Kubernetes), the worker host (Direct), or the configuration applied by an external runtime (Command).

## Unmanaged architecture

Expand All @@ -119,7 +114,7 @@ You're responsible for executing `oz agent run` on your infrastructure — simil

## Routing runs to self-hosted workers

This section applies to **all managed backends** (Docker, Kubernetes, and Direct). Once a worker is connected, route cloud agent runs to it by specifying the `--host` flag (or equivalent) with your worker ID. The `--host` value must match the `--worker-id` of a connected worker exactly.
This section applies to **all managed backends**. Once a worker is connected, route cloud agent runs to it by specifying the `--host` flag (or equivalent) with your worker ID. The `--host` value must match the `--worker-id` of a connected worker exactly. Docker, Kubernetes, and Direct workers execute the assigned task on worker infrastructure; Command workers dispatch it to the configured external runtime.

:::note
Unmanaged runs don't need routing — you invoke `oz agent run` directly on the host where you want the agent to execute. Routing is only relevant for managed workers.
Expand Down Expand Up @@ -214,6 +209,7 @@ For infrastructure-level observability, the `oz-agent-worker` daemon can export
* [Managed: Docker](/platform/self-hosting/managed-docker/) — Default managed setup with the Docker backend.
* [Managed: Kubernetes](/platform/self-hosting/managed-kubernetes/) — Managed setup with the Kubernetes backend and Helm chart.
* [Managed: Direct](/platform/self-hosting/managed-direct/) — Managed setup with no container runtime.
* [External orchestrators](/platform/self-hosting/external-orchestrators/) — Run a Direct worker in one-shot mode or dispatch tasks with the Command backend.
* [Self-hosted worker reference](/platform/self-hosting/reference/) — CLI flags and config file schema.
* [Monitoring](/platform/self-hosting/monitoring/) — OpenTelemetry metrics for worker health, task throughput, and capacity.
* [Security and networking](/platform/self-hosting/security-and-networking/) — Data boundaries, network egress, and security considerations.
Expand Down
1 change: 1 addition & 0 deletions src/content/docs/platform/self-hosting/managed-direct.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -126,6 +126,7 @@ backend:
## Related pages

* [Self-hosted worker reference](/platform/self-hosting/reference/#direct-backend-config) — Full config schema for the Direct backend.
* [External orchestrators](/platform/self-hosting/external-orchestrators/) — Start one Direct worker per external job or delegate runs through the Command backend.
* [Self-hosting overview](/platform/self-hosting/) — Managed vs unmanaged and the backend decision guide.
* [Routing runs to self-hosted workers](/platform/self-hosting/#routing-runs-to-self-hosted-workers) — How to send tasks to your connected worker from the CLI, schedules, integrations, the API, and the web UI.
* [Security and networking](/platform/self-hosting/security-and-networking/) — Data boundaries and security considerations for the Direct backend.
Expand Down
Loading
Loading