Background & Context§
The rapid adoption of AI assistants like ChatGPT, Perplexity, and Claude has created a new class of web traffic: automated crawlers that fetch pages not for human readers, but to train and inform large language models. Publishers have long struggled with bot traffic, but most have relied on blanket blocks or rate-limiting. However, as AI models become a primary distribution channel for information, some publishers are beginning to experiment with serving AI bots a different experience altogether. TIME's new approach—forking its site for AI crawlers and embedding ads directly into the markdown—represents a significant shift in how publishers monetize AI access. This move, first reported by developer Vincent Schmalbach on August 5, 2026, signals a potential future where the web's primary audience is not human but algorithmic.
The News: What Happened Exactly§
On August 5, 2026, Vincent Schmalbach, while working on his AI coding tool Vroni, discovered that TIME is serving two distinct versions of its website based on the User-Agent header of the requester. By repeatedly fetching the same health article, "The Morning Light Habit Sleep Experts Swear By," he observed that as a human browser (Chrome, Safari) or Googlebot, TIME returned the full HTML page, approximately 303,235 bytes, with all design, images, and scripts. However, when the request came from AI assistant crawlers such as ClaudeBot, PerplexityBot, or OAI-SearchBot, TIME responded with a 200 OK status but served a stripped-down markdown file of only 13,409 bytes—exactly identical across these bots. The markdown contained the article's text but no HTML, layout, or images, making it ideal for language model processing.
Notably, some bots were completely blocked: GPTBot and ChatGPT-User (used by OpenAI for training and live fetches) received a 406 Not Acceptable response. In contrast, OAI-SearchBot, which powers ChatGPT's search index, was granted access to the markdown version. This indicates a deliberate, per-bot policy rather than a blanket bot block.
The response headers of the markdown version revealed the involvement of Mobian, an ad-tech vendor. The headers included:
content-type: text/markdown; charset=utf-8 cache-control: no-store x-mobian-registry-version: 2026-07-28.v9 x-mobian-impression: 46dfff3c-fb40-41cc-85e1-8b1fa637083a x-mobian-tokens: 3323 x-mobian-format: md
The markdown page began with <!-- mobian-agent-page publisher="time" -->, and each request generated a fresh UUID for x-mobian-impression. Paired with cache-control: no-store, this means every bot fetch is logged as a unique ad impression. The x-mobian-tokens: 3323 header counts the tokens fed into the model—a unit of measurement distinct from pageviews.
Schmalbach also found sponsored content embedded directly within the markdown of list and section pages. For example, fetching TIME's "Best Inventions of 2025" collection as ClaudeBot returned a full Ally Bank FAQ, formatted as sponsored content, with questions like "Who is Ally Bank?" and "Can you deposit cash at Ally Bank?" Each answer was crafted in Ally's marketing language, alongside a FAQPage JSON-LD block and tracking links tagged campaign="ally-2026-q3". Similarly, the business section contained sponsored content for the Project Management Institute, including a "Reference Facts and FAQ" table and a claim that certified project managers earn 16% more. None of this sponsored content appears in the human-readable HTML versions of those pages; loading them as a person yields zero hits for "Ally Bank" or "Mobian."
This arrangement is not undisclosed advertising in the traditional sense—the sponsored content is labeled as such in the markdown. However, it is hidden from human readers, and the existence of this separate bot-facing layer is unknown to the general public. TIME's decision to serve different content to AI crawlers than to humans raises questions about transparency and the nature of web publishing as AI models become a primary audience.
Historical Parallels & Similar Incidents§
The practice of serving different content to different user agents is not new. For years, websites have used cloaking—a black-hat SEO technique where users are given one version of a page and search engines another—to manipulate rankings. Search engines like Google have penalized such practices, as they violate guidelines that require the same content to be shown to both users and crawlers. However, TIME's approach is distinct: it serves the full page to Googlebot (the search engine crawler) but a markdown version to AI assistant crawlers. This is closer to legitimate content negotiation based on the client's capabilities, similar to how websites serve mobile-friendly versions to smartphones or reduced-feature versions to text-based browsers.
A more recent parallel is the rise of AI-specific bot blocking and paywalls. For instance, in 2024, major publishers like The New York Times and Reuters blocked OpenAI's GPTBot from crawling their sites, while still allowing access to other bots. In contrast, TIME allows certain AI bots access but monetizes that access through embedded ads. This is a business model innovation rather than a technical one.
Historical lessons from cloaking and bot management suggest that transparency and clear differentiation are crucial. If TIME's bot-specific ads are discovered by the public and perceived as deceptive, it could damage trust. However, the labeling of sponsored content in the markdown may mitigate this. Another lesson is that as AI crawlers become a significant source of traffic—TIME has reportedly seen bot traffic outnumber human traffic on most days—publishers must adapt their business models. The token-based ad impression tracking used by Mobian is an early attempt to quantify the value of AI-driven page views, which could become a standard metric in the industry.
Unlike cloaking attempts of the past, TIME's approach is likely sustainable because it aligns with the needs of both parties: AI assistants get clean, easily processable content, while TIME generates revenue from a nearly untapped audience. However, the lack of human oversight on this bot-facing content could lead to unintended errors or controversies, especially if the sponsored content is factually misleading. As the web evolves, publishers will need to carefully balance monetization, transparency, and ethical considerations when serving non-human audiences.