Background & Context§
The rapid proliferation of large language models has created a paradox of choice for developers: while more options promise better outcomes, selecting the right model for a task has become increasingly difficult. Netlify, a leading web development platform, has addressed this by integrating a range of models into its Agent Runners feature, which now includes OpenRouter's diverse ecosystem. This move reflects a broader industry trend toward model-agnostic development environments, where developers can leverage multiple models without switching tools. By open-sourcing their evaluation tool, AXIS, Netlify is contributing to a more transparent AI landscape, enabling developers to make data-driven decisions about model selection.
The News: What Happened Exactly§
Netlify announced a partnership with OpenRouter, expanding its Agent Runners to support a broader variety of models through the open-source agent OpenCode. Until now, Agent Runners supported only Claude Agent, OpenAI Codex, and Gemini CLI—each optimized for specific providers. With OpenCode, developers can now run coding agents powered by any of 11 models, including Anthropic's Opus and Sonnet, OpenAI's Sol and Terra, Google's Gemini 3.1 Pro and 3.6 Flash, and open-weight models like Kimi K3 and Kimi K2.7 Code.
To help developers navigate this new choice, Netlify conducted a series of subjective tests using a simple prompt: "Build a one-page site for a neighbourhood coffee shop: opening hours, the address, a short menu and a photo. Nothing on it changes unless I edit it myself." This prompt was run through each model with default settings on Netlify, and the results were published, including credit usage and live sites for comparison.
The credit usage varied widely across models. Claude Opus averaged 515 credits per run (with one run consuming a staggering 1,055 credits), while Claude Sonnet averaged 143 credits. OpenAI's Sol (low effort) averaged 141 credits, Terra averaged 39 credits, Gemini 3.1 Pro averaged 53 credits, Gemini 3.6 Flash averaged 103 credits, Kimi K3 averaged 102 credits, and Kimi K2.7 Code averaged just 19 credits. The notable outlier was OpenWeights GLM, whose credit usage was not detailed in the article but was implied to be low.
Beyond credit costs, the quality of the generated sites differed dramatically. Opus produced the most visually rich and detailed sites, particularly one run that included custom vector graphics, a stamp-like logo, and a functional map. However, this came at a 4x credit premium. Sonnet's sites were simpler but still included some design flair, albeit with less content. OpenAI's Sol low-effort mode delivered solid design intuition with rich content, while Terra, despite being a tier below, produced visually distinct and arguably as appealing results, though with occasional glitches like missing images or low-contrast text.
Google's models showed a stark generational gap: Gemini 3.1 Pro generated bare-bones pages that met the prompt's requirements but nothing more, while Gemini 3.6 Flash produced more modern and content-rich sites, albeit with some repetitiveness. The open-weight Kimi models fared poorly on this design-focused task; Kimi K3, marketed for agentic tasks, did not shine in UI generation, while Kimi K2.7 Code, with a token architecture, produced minimal sites.
These observations underscore that model selection significantly impacts both cost and output quality, and that higher cost does not always guarantee better results. For instance, Opus's 1,055-credit run was only marginally better than its 253-credit runs, and OpenAI's Terra at 39 credits outperformed Gemini 3.1 Pro at 53 credits in terms of design and content richness.
Historical Parallels & Similar Incidents§
Netlify's model evaluation exercise echoes similar initiatives in the AI industry, most notably the emergence of model comparison platforms like the LMSYS Chatbot Arena, which crowdsources ELO ratings for LLMs. In 2023, Chatbot Arena became a de facto standard for comparing model performance, but it focused primarily on conversational ability, not real-world task execution. Netlify's AXIS tool, which evaluates models on actual coding outcomes, addresses a gap by providing a functional benchmark. This shift from subjective preference ratings to objective task-specific evaluations is reminiscent of how the coding assistant Copilot was assessed in its early days: early benchmarks like HumanEval measured code generation accuracy, but they did not capture the full user experience of building a complete application.
Another parallel is the beta launch of GPT-5 by OpenAI, which happened around the same time as this article's publication. GPT-5 was integrated into Netlify's Agent Runners, but it was not tested in this scenario. The article notes that models like Kimi K3 were designed for long-horizon agentic tasks, similar to how GPT-5 was marketed. The contrast between models optimized for complex reasoning versus those tuned for design tasks highlights a critical lesson: a single benchmark cannot assess a model's utility across all use cases. An LLM that excels at mathematical reasoning may fail at web design, as demonstrated by Kimi K3's lackluster performance.
Historically, the industry has seen similar lessons with models like Google's BERT, which revolutionized NLP tasks but was not suited for generative tasks. This niche specificity is becoming more pronounced as models are increasingly specialized. Netlify's approach of running multiple models on the same prompt to reveal trade-offs is a practical extension of this trend, offering developers a glimpse into the operational costs and quality variances they can expect.
In summary, Netlify's experiment provides valuable data for developers navigating the crowded LLM marketplace. It underscores that there is no one-size-fits-all model, and that choices must be made based on specific project requirements and budget constraints. While this article focuses on a simple static page, follow-up posts promise to explore more complex scenarios, which will be crucial as AI agents take on increasingly sophisticated tasks.