Background & Context§
Autonomous AI agents capable of executing long-horizon tasks with tool use are a frontier of LLM research. While benchmarks often focus on isolated tool calls or simple multi-step tasks, open-ended creative production — where the model must research, generate, assemble, and edit a final artifact — remains a challenging testbed. The open-source project Music Video Arena, built by Hershal B. and hosted on GitHub (hershalb/music-video-arena), introduces a minimal agentic harness that gives an LLM a song, a hard dollar budget, and a set of tools, then steps aside to let the model produce a full music video from scratch. The results of a recent head-to-head between two frontier models — Anthropic's Claude Fable 5 and OpenAI's GPT-5.6 Sol — at two budget levels ($25 and $100) provide a rare transparent look into how different models approach resource-constrained, multi-modal creativity.
The News: What Happened Exactly§
Each model was given the same source material: the song "Uptown Funk" by Bruno Mars and Mark Ronson, along with a short text description and a time-stamped lyric transcript. The models operated in an autonomous tool-calling loop with six tools: text-to-image, text-to-video, image-to-video, video-to-audio, web search, and a custom ffmpeg wrapper for editing. The harness logged every model message, tool call, charge, and error. The budget capped paid generation costs on FAL (a model inference provider) at $25 or $100, after which only editing was allowed. The LLM token costs were metered separately.
All four runs — Claude Fable 5 at $25, Claude Fable 5 at $100, GPT-5.6 Sol at $25, and GPT-5.6 Sol at $100 — finished without hitting step or time limits and produced a valid full-length video with the original song muxed in. However, the approaches diverged significantly in tool selection, generation volume, and cost efficiency.
Generation Spend and Tool Use§
At the $25 budget, both models nearly exhausted the generation budget. At $100, Sol spent $36.57 of the budget, while Fable spent $48.60, meaning higher budget did translate into more footage (distinct clips generated per run ranged from 46 to 80). Notably, three of the four runs chose a pure text-to-video pipeline — generating clips directly from text prompts. The lone exception was GPT-5.6 Sol at $25, which used an image-to-video pipeline: first generating still images, then animating them. At $100, Sol mixed three different video models (Hailuo 2.3 Standard, Seedance 1.0 Pro, and Metamorph) in a single run, showing greater willingness to diversify generation sources.
Token Cost Asymmetry§
A striking finding was the asymmetry in LLM token costs. Claude Fable 5 consumed between $16.99 and $25.05 in tokens alone, accounting for about 30–40% of each run's total cost. GPT-5.6 Sol, on the other hand, incurred only $3–4 in token costs despite similar token volumes. This cost difference arises directly from the models' pricing: Claude Fable 5 costs $10 per million input tokens and $50 per million output tokens, while GPT-5.6 Sol costs $5 and $30 respectively. For a task that required extensive planning and iterative refinement (as seen in the logs), the cheaper token costs of Sol gave it a significant economic edge.
Failed Calls and Retry Patterns§
Both models encountered generation failures — mostly transient network errors from the provider. These failures were not charged but forced the models to retry, consuming LLM thinking steps. The transcripts reveal that Fable 5 tended to retry with slightly modified prompts, while Sol more often switched to a different generation tool after repeated failures. This difference reflects the models' different "personalities" in handling errors, which can affect overall runtime and cost.
Final Video Quality§
The article notes that "none of the music videos were great," but the process was revealing. Key observed gaps include: lack of camera motion across clips, minimal character consistency, limited scene transitions/effects, over-reliance on generic prompts, and piecemeal footage lacking a coherent narrative arc. These shortcomings underscore how frontier models still struggle with subjective, stylistically coherent long-form creative tasks.
Historical Parallels & Similar Incidents§
This test echoes an earlier landmark experiment in autonomous AI content creation: in early 2023, researchers pitted GPT-4 and earlier versions of Claude against each other in a task to write, produce, and narrate a short animated film using AI tools (e.g., DALL-E 2 for images, Whisper for speech, and ffmpeg for editing). That experiment, known as the "AI Film Festival" by a collective of independent researchers, also found that models struggled with narrative coherence and visual consistency across scenes. However, the models then had far less sophisticated tool-use capabilities and often required human intervention to fix broken pipelines. In the current Music Video Arena, both models successfully executed the entire pipeline autonomously — a significant step forward. The earlier experiment highlighted the difficulty of maintaining a consistent style and plot, while this newer test reveals that even with improved autonomy, stylistic control remains elusive.
Another parallel is the rise of "agentic coding" benchmarks where LLMs generate software projects end-to-end. For instance, in the SWE-bench and similar code generation tasks, GPT-4 and Claude have shown proficiency in writing isolated functions but often fail to integrate them into a working application without iterative human feedback. The Music Video Arena extends this to multi-modal content, where the model must not only generate but also assemble and edit. The same failure mode appears: piecemeal generation without a global plan. In coding, this manifests as mismatched interfaces; in video, it shows as disjointed clips.
A lesson from these parallels is that current LLMs are strong at individual steps but weak at orchestrating a coherent whole, especially under a budget constraint that forces trade-offs. The cost asymmetry between models also recalls the early days of LLM APIs when GPT-3.5 was far cheaper than GPT-4, leading many developers to optimize for cost by using smaller models for planning and larger models for execution. In this case, GPT-5.6 Sol emerges as both cheaper and more flexible in tool choice, suggesting that for resource-constrained autonomous agents, cost per token is a dominant factor beyond raw capability.
Open Source and Reproducibility§
The entire Music Video Arena is open source at hershalb/music-video-arena, allowing anyone to run the same test with their own song and budget, and to swap in different models. This transparency is critical for the community to validate and build upon the findings. The full transcripts of each run are also available, providing a rare detailed look at the models' internal decision chains. As the field moves toward autonomous agents, such benchmarks will be essential for measuring progress and identifying remaining bottlenecks.