Cost Optimization for LLM Applications: Model Routing, Caching, and Token Compression

What Just Happened§

It’s official: the “just call GPT-4 for everything” era is over. Over the last quarter, LLM providers like Anthropic, DeepSeek, and OpenAI all debuted or expanded explicit cost-control mechanisms—Claude’s prompt caching (now 90% cheaper on input tokens), DeepSeek’s automatic context caching (1-hour TTL at 0.1x cost), and OpenAI’s new cached prompt token pricing. Meanwhile, a wave of open-source routers (RouteLLM, Martian, llm-router) and caching tools (GPTCache, LangChain’s LLMCache) matured from experimental to production-ready. The pattern is clear: cost optimization is no longer an afterthought—it’s a core architectural principle.

Another major shift is the rising cost of context. As apps embed larger knowledge bases and longer conversation histories, the input token bill can dwarf generation costs. Providers are responding with explicit caching tiers, but they’re also pushing the responsibility onto developers. We’re seeing a new best practice: treat every prompt as a cost unit and every API call as an opportunity to reuse, compress, or route. This is not a vendor fad. It’s a fundamental change in how we design LLM pipelines.

Why This Matters for AI Practitioners§

For AI practitioners, the cost of deploying LLMs at scale can be the difference between a profitable product and a money pit. If you’re building a customer support bot that handles 10,000 queries a day, a naive setup using a large model like GPT-4 Turbo for every interaction could cost more than $100 per day—before caching or routing. Multiply that across a fleet of microservices or user-facing features, and the annual run rate becomes a board-level concern. This is why I’ve seen startup teams spend as much time on cost engineering as on model quality. The economics force you to.

Cost optimization also affects latency and user experience. Caching and routing aren’t just about saving money—they reduce round-trip times by hitting a fast cache or a smaller model for trivial tasks. Perplexity, for example, uses a hybrid search-and-generate pipeline where model routing sends simple queries to instant answers and complex ones to heavier models. Users get the illusion of an omniscient assistant at a fraction of the cost. For practitioners, this means you need to understand the quality/latency/cost triangle. If you can shave 30% off your token spend without affecting quality, you can reinvest that into more features, better retrieval, or simply lower prices for your customers.

The hidden cost of inaction is even more dangerous. As your application grows, the exponential increase in context length and conversation depth compounds. Without a cost architecture, you’ll hit a wall where adding users actively loses money. That’s why the techniques detailed below are not optional padding—they’re survival instincts for anyone shipping LLM features in 2025.

Who Is Affected§

This trend affects everyone who touches LLM APIs: indie developers building side projects, SaaS founders scaling to thousands of users, enterprise architects integrating LLMs into internal knowledge portals, and even researchers running batch evaluations. If you’re using a paid LLM API—be it OpenAI, Anthropic, Azure OpenAI, or Google’s Vertex AI—you are directly impacted by these cost dynamics. Even open-source self-hosted models aren’t immune; they have GPU and electricity costs that favor the same routing and caching strategies.

Specific roles that need to pay attention include backend engineers (who own the API integration), DevOps/platform engineers (who manage API keys and monitoring), and product managers (who prioritize features based on unit economics). The rise of agentic workflows—like those in Cursor or the new Claude Code—has multiplied the number of requests per user, making cost optimization even more critical. In fact, Cursor’s rapid iteration on request routing was one of the first visible cases where a tool had to drop from GPT-4 to smaller models for autocomplete to keep subscription costs viable.

If you’re not affected yet, you will be. The moment your application gains traction, the cost math changes. I’ve personally seen projects with sub-$50 monthly spends jump to $5,000 overnight after a viral feature. Without a cost control strategy, that growth becomes a liability.

How to Use This Right Now§

The first step is to implement model routing: send each request to the smallest model that can handle it. A simple, production-tested approach is to use a classifier or a set of rules to decide between a cheap fast model (like gpt-4o-mini) and a premium model (like gpt-4o or claude-opus-4). For example, you can route based on task complexity, expected token length, or keyword analysis. Here’s a minimal but effective Python pattern:

import openai

def route_and_complete(user_query: str, cached_responses: dict) -> str:
    # 1. Check cache first (exact match, normalized query)
    normalized = user_query.strip().lower()
    if normalized in cached_responses:
        print("[CACHE HIT]")
        return cached_responses[normalized]

    # 2. Route based on complexity keywords
    simple_keywords = ["what is", "define", "summarize", "convert", "short"]
    needs_large_model = any(kw in normalized for kw in ["explain in depth", "compare", "write a full report", "legal", "code review"])

    if any(kw in normalized for kw in simple_keywords) and len(normalized) < 40:
        model = "gpt-4o-mini"
        max_tokens = 150
    elif needs_large_model or len(normalized) > 100:
        model = "gpt-4o"
        max_tokens = 500
    else:
        model = "gpt-4o-mini"
        max_tokens = 250

    print(f"[ROUTE] Using {model}, max_tokens={max_tokens}")
    response = openai.chat.completions.create(
        model=model,
        messages=[{"role": "user", "content": user_query}],
        max_tokens=max_tokens
    )
    answer = response.choices[0].message.content
    cached_responses[normalized] = answer  # cache the result
    return answer

Next, you need caching. The simplest form is exact-match caching in a dictionary or Redis, but for real-world apps, semantic caching is more powerful. Google’s GPTCache and LangChain’s CacheBackedLLM are great starting points. The key is to cache not just the final answer but also the prompt embedding so you can detect similar questions. For example, a customer support bot can cache common greetings and troubleshooting steps. The total input token cost for a typical conversation often includes repeated system prompts and context—caching those can cut costs by up to 90%.

Token compression is the third pillar. This doesn’t mean changing the model; it means being aggressive about what you send. Use max_tokens to limit output length, truncate long system prompts, and dynamically summarize older conversation turns before sending them. Some providers like DeepSeek already compress context internally, but you should still do your part. For example, instead of sending 50 prior messages, keep a running summary of key facts. Anthropic’s Claude also supports explicit prompt caching by adding cache_control to system prompts—do this for any static system context that repeats.

Finally, adopt observability tools like Helicone or Portkey to measure token usage per feature. You can’t optimize what you don’t track. Set up dashboards for cost per request, cache hit rate, and model distribution. It’s surprising how quickly a small change—like dropping max_tokens from 300 to 180—can reduce your bill by 15% without users noticing.

Related Tools on LLMDB.APP§

LLMDB.APP curates the exact tools you need to implement these strategies. I highly recommend searching the platform for the following categories:

• Model routing and orchestration: Look up Portkey (gateway with model fallback and routing), LiteLLM (proxy with 100+ providers and cost management), and OpenRouter (unified API with automatic best-model routing). These tools abstract the routing logic so you don’t have to write it from scratch.
• Prompt caching and semantic cache: Search for GPTCache, RedisVL, and LangChain CacheBackedLLM. These are battle-tested libraries for reducing repeated calls.
• Token and cost tracking: Helicone, Langsmith, and Lunary are popular choices for per-token cost observability. They integrate with your existing code in minutes.
• Compression and summarization: Look for LLMLingua (prompt compression) and Recursively Summarize chains in LangChain. Also check Claude’s official extension for built-in caching.

On LLMDB.APP, each tool page includes open-source status, API compatibility, and user reviews—so you can compare trade-offs before committing. For example, if you’re already on OpenAI, Portkey can add caching and routing without changing your underlying calls. If you’re building a home-grown solution, the code patterns above plus a Redis instance are enough to start.

Let’s face it: the era of API-money-burning is over. The tools and techniques described here are not experimental—they’re the core of any production LLM application today. Start with caching, add a routing layer, and measure the impact on your cloud bill. Your CFO (or your own wallet) will thank you.