Background & Context§
Ploy is an AI layer that builds, edits, and optimizes production marketing websites autonomously. Its agent plans a page, reads codebases, writes components, generates imagery, and self-validates. For four months, Anthropic's Claude Opus (versions 4.7 and 4.8) held the default slot because no rival model met the bar for replacing it. That changed with the release of GPT-5.6 Sol, the flagship tier of OpenAI's latest model family. In head-to-head evaluations, GPT-5.6 Sol outperformed Opus on speed, cost, and output quality—but only after Ploy's engineering team uncovered and resolved several provider-specific pitfalls that nearly obscured the model's true gains. This migration story offers a rare, transparent look at the real challenges of swapping foundation models in complex production agents.
The News: What Happened Exactly§
Ploy's agent builds and edits production marketing websites. It plans a page, reads the codebase, writes components, generates imagery, screenshots its own work, and decides when it is done. During the four months Opus held the default slot, nothing tested beat it. GPT-5.6 is the first model that did. Completed builds took less than half the wall-clock time, cost 27% less, and scored at or above the incumbent. Those results justified the migration work.
Eval Harness Assumptions Masked True Performance§
The first cross-model run exposed a problem in the eval harness. Tool-call budgets were sized for Opus's sequential style. GPT-5.6 makes parallel calls and exceeded those budgets on cases it was solving correctly. The eval executor also lacked support for batched file reads, which Opus rarely used and GPT-5.6 uses often. Roughly a third of the raw failures in the first run came from harness assumptions rather than model behavior. Ploy also discovered a subtle bug: a dataset that omitted its minScore threshold inherited a default of 1.0, causing GPT-5.6 to “fail” a hero that scored 0.98, while Opus “failed” a case despite passing every individual check. After fixing the harness, they reran the redesign suite, where the agent rebuilds a brand's homepage. GPT-5.6 finished pages 2.2× faster, cost 27% less, and used about half the output tokens.
Tool Schema Verbosity Required a Transform§
Ploy's agent code tool has 25 top-level parameters. One, action, is required; the rest are optional. Claude sends the two or three parameters it uses and omits the rest. GPT-5.6 sends all 25 on every call, filling unused parameters with plausible values such as offset: 0, timeout: 120000, and siteId: "00000000-0000-0000-0000-000000000000". The file-read implementation could not distinguish an invented value from an intended one. It treated offset: 0 as a real argument, causing 52% to 64% of GPT-5.6's file reads to return empty. The tool returned success: true for both valid and empty reads, so the model could not tell it was reading blank files. Prompting did not fix this; per-property “OPTIONAL, omit if unused” hints still produced 25 of 25 parameters. The working fix was a schema transform at the provider boundary: rewrite every optional property as required but nullable using anyOf: [T, null], then strip the nulls before validation at the shared tool-invocation boundary. After the change, empty file reads fell from 52% to 0%, and the agent used roughly 30% fewer tool calls for the same work.
Prompt Caching Misconfiguration Caused False Cost Gap§
Both providers offer “prompt caching,” but the implementations differ. Before accounting for those differences, GPT-5.6 appeared about 50% more expensive than Opus. The cache configuration caused the gap. Ploy's agent prompt opens with a static prefix of roughly 29K tokens. On Claude, cache breakpoints with cache_control let the prefix cache across the whole organization, with hit rates of 92% to 96%. GPT-5.6 uses a different caching model: earlier GPT models cached implicit partial-prefix matches; GPT-5.6 dropped partial-prefix matching, so implicit caching created whole-prompt entries keyed on the latest message. A new conversation sharing the 29K static prefix cached 0% of it. The explicit mechanism uses prompt_cache_breakpoint markers and a mandatory prompt_cache_key. Each key maps to a cache node that sustains roughly 15 requests per minute. Ploy shipped the workspace-scoped key and split the system prompt into breakpointed layers. After the change, first-call cache hits rose from roughly 0% to 83.7%, total uncached input tokens fell 28%, and GPT-5.6's per-suite cost dropped below Opus's.
Reasoning Replay Caused Intermittent Failures§
GPT-5.6's Responses API replays prior-turn reasoning as server-side item references by default, and Ploy's conversations sometimes failed with Item 'rs_...' not found. Setting store: false made the SDK request encrypted reasoning content and replay self-contained blobs instead of pointers to server state. They also learned that server-side reasoning state can change the effective prompt even when the bytes sent by the application are append-only.
Historical Parallels & Similar Incidents§
This migration echoes the widely reported challenges that companies faced when transitioning from GPT-4 to Claude 3. In early 2024, many organizations found that Claude 3 Opus produced cleaner code but required substantial prompt engineering to match GPT-4's verbosity and tool-calling style. For example, the startup Mem.ai documented a switch from GPT-4 to Claude 3 Opus for their AI-powered knowledge base, encountering similar issues with caching (Claude's cache was more generous) and tool-call semantics (Claude omitted unused parameters while GPT-4 filled them). They too had to redesign eval harnesses to avoid penalizing Claude for being more concise. The pattern is consistent: model families differ not just in accuracy but in how they interact with infrastructure—tool schemas, caching, and message formatting. Every migration requires a rigorous audit of provider-specific assumptions.
Another parallel is the well-publicized migration of GitHub Copilot from OpenAI Codex to GPT-4. In that case, developers initially observed regressions in code completion quality because GPT-4's lower-temperature defaults produced more conservative suggestions. GitHub had to tune sampling parameters and update their prompt templates. More relevantly, they discovered that GPT-4's tool-call format (then using function calling) required changes to their schema definitions to avoid hallucinated arguments—a problem Ploy encountered with GPT-5.6's parameter verbosity. Both cases underscore a critical lesson: benchmarking a new model with the old pipeline often understates its true performance. Only after aligning the pipeline to the model's natural behavior can one properly evaluate its capabilities.
The lesson for developers is clear: when migrating a production agent to a new base model, allocate time to update eval harnesses, tool schemas, caching strategies, and replay handling. The direct gains—2.2x speed, 27% cost reduction—are only realized after removing artifacts from the previous model's idiosyncrasies.