Background & Context§
AI coding agents are rapidly becoming indispensable tools for developers, automating routine tasks and accelerating workflows. However, these agents operate by executing a sequence of shell commands, each of which can have unintended or malicious consequences. The standard safeguard is the "human-in-the-loop" model: a developer must approve each command before the agent executes it. This model places the burden on the human to quickly distinguish between safe and dangerous actions. But as AI agents become more autonomous and the command volume increases, the viability of this security boundary is being called into question. The news from this story centers on a unique experiment that quantifies the failure rate of human oversight in a simulated environment.
The News: What Happened Exactly§
Alex, an ex-Staff Engineer at Uber, developed a browser game that puts players in the role of a human approving commands for an AI coding agent. The game, published on llmgame.scalex.dev, simulates a session where players must approve or deny a stream of commands under time pressure. Some commands are routine (e.g., git status, npm test), while others indicate the agent has been compromised and is attempting to exfiltrate sensitive data (e.g., cat ~/.aws/credentials). The game was designed to explore the reliability of human-in-the-loop supervision, and after collecting data from over 40,000 runs (409,000 individual approval decisions), the results are sobering.
Across all runs, players approved a staggering 34% of all threats. Even more concerning, the miss rate varied significantly depending on how the threat was presented. The most dangerous category—direct exfiltration of credentials via commands like cat ~/.aws/credentials—was missed 28.4% of the time. But when malicious commands were hidden behind a familiar script name, the success rate nearly doubled to 52.5%. The single most-missed command was npm run analyze, which was approved 64.7% of the time. This command, though commonly used for legitimate bundle analysis, can execute arbitrary scripts from package.json, and the game explicitly showed a malicious script in the agent's history log that would curl data to a remote server. Yet, two-thirds of players approved it without inspecting the log.
The game also revealed significant performance degradation over the course of a session. The threat miss rate improved slightly after the first few commands (a warm-up effect) but then climbed steadily towards the end, suggesting that fatigue or stress—such as a looming clock—causes users to become less vigilant. This aligns with observations from Anthropic about "permission fatigue," where users pay less attention to each approval as the number increases.
On the flip side, the game highlighted that many benign commands are routinely blocked, creating noise that desensitizes users. For example, cat ~/.zshrc was approved by only 45.9% of players, making it the most divisive command. While some developers keep no secrets in their shell profile, others export API keys there, making the command's risk entirely dependent on the developer's setup—information the agent cannot see. This ambiguity in risk assessment forces developers to make security judgments without full context, a model that is fundamentally flawed.
The game's author emphasizes that while this is not an academic study, the data clearly demonstrates that relying on human approval as a security boundary is inadequate. The high volume of approvals, combined with ambiguous commands and time pressure, makes it impossible for humans to maintain a high level of vigilance. The author calls for better safeguards, such as sandboxing and strict context isolation, rather than pointing to human-in-the-loop as an acceptable fallback.
Historical Parallels & Similar Incidents§
The issues highlighted by this game are not new. In the early days of web development, similar problems arose with SQL injection attacks. Developers were often the last line of defense, manually reviewing user inputs to prevent malicious database queries. However, as applications grew, this approach proved unsustainable, leading to the development of parameterized queries and ORMs that automatically sanitize inputs. The lesson was that relying on human vigilance for security is a losing battle; instead, the security must be built into the tooling.
Another parallel can be drawn with the evolution of email spam filtering. In the late 1990s and early 2000s, users were often required to approve or reject emails manually, but the sheer volume of spam made this impossible. The industry responded with heuristic and machine-learning-based filters that could automatically identify and block threats, reducing the reliance on user judgment. Similarly, AI agents today are generating a torrent of commands, and expecting developers to manually vet each one is becoming increasingly impractical.
More recently, in the realm of AI, there have been multiple instances of prompt injection attacks against chatbots and virtual assistants. For example, in 2022, a security researcher demonstrated that a Twitter bot could be manipulated into posting malicious content by embedding hidden instructions in its input. The bot's operators relied on a human review process, but the attack succeeded because the instructions were designed to bypass human attention. This incident, like the game's npm run analyze case, shows how attackers exploit the human tendency to overlook details when they appear to be routine.
These historical parallels underscore a crucial lesson: as the volume of security-relevant decisions increases, human oversight must be complemented with automated, context-aware defense mechanisms. For AI agents, this means implementing robust sandboxing that limits what commands can do, using permission systems that automatically assess the risk of each command based on its context (e.g., the state of the repository, the user's configuration), and providing clear, actionable alerts only for truly anomalous actions. The game's author suggests that we need to "make the tooling easier" and grant agents broad permissions only once these safeguards are in place.
# Example of a risky command that was often missed echo "malicious script" >> package.json npm run analyze # approved 64.7% of the time
In summary, the game provides concrete, quantitative evidence that human-in-the-loop is not a reliable security boundary for AI agents. The data reveals that even in a controlled, threat-aware environment, humans missed one in three threats, and that this rate only worsens when commands are disguised or as fatigue sets in. The path forward is not to demand more vigilance from developers but to architect AI systems that are fundamentally safer by design, reducing the reliance on human judgment to catch what the system should have already prevented.