OpenForgeRL: Train Harness-native Agents in Any Environment

OpenForgeRL: Training Harness-Native Agents End-to-End§

OpenForgeRL is an open-source framework that enables end-to-end reinforcement learning (RL) training of agents that rely on complex inference harnesses (e.g., Claude Code, Codex, OpenClaw). The key challenge is that these harnesses are stateful, multi-process systems that are incompatible with standard RL training pipelines. OpenForgeRL solves this with two core components: a lightweight proxy and a Kubernetes orchestrator.

Methodological Core§

The proxy acts as a man-in-the-middle: it intercepts all model calls made by the harness (typically to an LLM API) and records the input/output pairs. These pairs form the training data. During RL training, the proxy serves the harness's original model calls using the current policy (which is being trained), and simultaneously logs the trajectories for RL optimization. This effectively decouples the harness's inference logic from the model training.

The Kubernetes orchestrator manages distributed rollouts: each rollout (a complete episode of the agent interacting with the environment) is executed in its own remote container. This allows scaling to many parallel environments. The orchestrator collects all recorded trajectories and sends them to the RL trainer (e.g., veRL) for policy updates.

Training Objective§

OpenForgeRL uses Proximal Policy Optimization (PPO) with a standard clipped surrogate objective:

$$\mathcal{L}^{CLIP}(\theta) = \mathbb{E}_{t} \left[ \min\left( r_t(\theta) \hat{A}_t, \text{clip}(r_t(\theta), 1-\epsilon, 1+\epsilon) \hat{A}_t \right) \right]$$

where $r_t(\theta) = \frac{\pi_\theta(a_t|s_t)}{\pi_{\theta_{\text{old}}}(a_t|s_t)}$ is the probability ratio, $\hat{A}_t$ is the generalized advantage estimate, and $\epsilon$ is a hyperparameter (typically 0.2). The trajectory data recorded by the proxy is used to compute $\hat{A}_t$ and the policy gradient.

Implementation Details§

Proxy Server. The proxy is a simple HTTP server that wraps the harness's model endpoint. It implements a custom /v1/chat/completions endpoint that returns completions from the current policy model (loaded into GPU memory) while logging the request/response pairs.

Kubernetes Integration. The orchestrator defines a rollout as a Kubernetes pod that runs the harness and environment inside a container. It uses a job queue to manage parallel pods, each of which communicates with the proxy. After completion, the logged data is stored in a shared volume (e.g., S3) and the trainer reads from it.

Code Snippet (Proxy Server).

import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
from flask import Flask, request, jsonify

app = Flask(__name__)
model = AutoModelForCausalLM.from_pretrained("policy_model_path")
tokenizer = AutoTokenizer.from_pretrained("policy_model_path")
model.half().cuda()

def log_trajectory(conversation, response):
    # Append to shared storage (e.g., Redis queue)
    pass

@app.route("/v1/chat/completions", methods=["POST"])
def chat_completions():
    data = request.get_json()
    messages = data["messages"]
    input_ids = tokenizer.apply_chat_template(messages, return_tensors="pt").cuda()
    with torch.no_grad():
        output_ids = model.generate(input_ids, max_new_tokens=1024, do_sample=True)
    response = tokenizer.decode(output_ids[0][input_ids.shape[-1]:], skip_special_tokens=True)
    log_trajectory(messages, response)
    return jsonify({"choices": [{"message": {"content": response}}]})

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=8000)

Orchestrator Sketch (simplified).

from kubernetes import client, config
import uuid

config.load_kubeconfig()
v1 = client.CoreV1Api()

for task in task_queue:
    pod_name = f"rollout-{uuid.uuid4().hex[:8]}"
    pod_manifest = {
        "apiVersion": "v1",
        "kind": "Pod",
        "metadata": {"name": pod_name},
        "spec": {
            "containers": [{
                "name": "harness",
                "image": "openforgerl/harness:latest",
                "env": [{"name": "PROXY_URL", "value": "http://proxy-server:8000"}],
                "volumeMounts": [{"mountPath": "/data", "name": "shared-storage"}]
            }],
            "volumes": [{"name": "shared-storage", "persistentVolumeClaim": {"claimName": "rl-data"}}],
            "restartPolicy": "Never"
        }
    }
    v1.create_namespaced_pod(body=pod_manifest, namespace="rl")
    # wait for completion and collect logs

Evaluation and Findings§

OpenForgeRL was validated on two agent categories: tool/claw-based agents (using ZeroClaw, OpenClaw) and GUI agents (browser/computer use). With only hundreds to a few thousand tasks, training significantly improved performance: e.g., OpenForgeClaw achieved 31.7% pass^3 and 55.9% pass@3 on ClawEval, and OpenForgeGUI reached 37.7 on OSWorld-Verified. The framework allowed analyzing how harness choice affects learnability and how RL improves reliability (self-verification, tool coverage) but not error recovery.