OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models

Overview§

OSReward introduces a standardized evaluation suite for VLM-based judges that verify computer-using agent (CUA) trajectories. The authors collect trajectories from diverse agent backbones executing human-verified instructions across platforms, then rigorously label them with ground-truth verdicts via multi-stage human annotation. From this, they derive OSReward-Hard (genuinely hard cases) and OSReward-Multi (fine-grained efficiency and alignment scoring).

Key Findings§

Evaluation of state-of-the-art VLM judges reveals a systematic leniency bias: failed trajectories are frequently mislabeled as successes. The few reliable models are too expensive for large-scale use, while affordable open models trail far behind. This motivates the construction of OS-Shepherd-100K, an open corpus of reasoning-annotated trajectory judgments, and the training of OS-Shepherd (9B/35B) reward models that supply low-cost, stable reward signals, matching commercial judges at 30–60% lower cost.

Formulation§

A CUA trajectory is a sequence of interleaved observations (e.g., screenshots) and actions:

$$ \tau = (s_1, a_1, s_2, a_2, \dots, s_T, a_T) $$

The goal of a reward model $R_\theta$ is to predict whether trajectory $\tau$ satisfies instruction $I$. The VLM judge is modeled as:

$$ p_\theta(y=1 \mid \tau, I) $$

where $y=1$ indicates success. In practice, the model is trained by maximizing the log-probability of the correct verdict token(s) in a language modeling objective:

$$ \mathcal{L} = -\log p_\theta(y^ \mid \tau, I) = - \sum_{w \in y^} \log p_\theta(w \mid \tau, I, w_{<t}) $$

For multi-dimensional scoring (efficiency/alignment), the objective is extended to predict multiple structured outputs, e.g., a multi-label classification head or a sequence of verdict tokens per dimension.

Implementation Example§

The following simplified PyTorch snippet illustrates how OS-Shepherd-style reward models are fine-tuned on trajectory-verdict data using a vision-language backbone like LLaVA:

import torch
from transformers import LlavaForConditionalGeneration

model = LlavaForConditionalGeneration.from_pretrained("llava-v1.6-9b")
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-5)

def encode_trajectory(instruction, frames, actions, label_mask):
    # Convert interleaved screenshots and action text into a VLM prompt
    prompt = build_prompt(instruction, frames, actions)
    inputs = tokenizer(prompt, return_tensors="pt").to(device)
    # Zero out loss on prompt tokens; only reason/verdict tokens are supervised
    inputs["labels"] = inputs["input_ids"].masked_fill(~label_mask, -100)
    return inputs

for batch in dataloader:  # instruction, frames, actions, verdict
    inputs = encode_trajectory(batch, label_mask=batch.verdict_mask)
    outputs = model(**inputs)
    loss = outputs.loss
    loss.backward()
    optimizer.step()
    optimizer.zero_grad()

Impact§

OSReward provides a reproducible yardstick for assessing judge reliability, and OS-Shepherd offers a cost-effective, trustworthy reward signal for CUA training and evaluation. The released benchmark, dataset, and model checkpoints enable further research on reliable agentic reward modeling.