The Architecture of Context-Aware Prompt Caching in Enterprise LLM Gateways

The Cost and Latency Bottleneck of Repeated Prompts§

In enterprise AI applications, up to 60% of tokens processed by underlying foundation models consist of repeated static content—system instructions, codebase context, schema definitions, and RAG knowledge bases. Re-encoding 50,000 tokens of system context on every incoming user request creates unnecessary compute overhead, inflates monthly cloud bills, and increases time-to-first-token (TTFT) latency.

By implementing a Context-Aware Prompt Caching Gateway, engineering teams can intercept incoming LLM requests, match identical prompt prefixes or semantically equivalent queries, and serve cached responses or leverage model provider prompt-caching features automatically.

---

Dual-Layer Caching Architecture: Prefix vs Semantic§

A production-grade LLM gateway employs two distinct caching mechanisms:

1. Exact-Prefix Hashing (Deterministic Cache)§

Calculates a SHA-256 hash of the initial sequence of system tokens. If the prefix matches an existing active session, the gateway appends provider-specific caching headers (e.g., Anthropic's prompt-caching header or OpenAI's automatic prefix cache key).

2. Semantic Query Caching (Vector Similarity)§

For short user queries following a fixed prompt template, the gateway computes a dense vector embedding of the input prompt. If the cosine similarity between the query embedding and a stored entry in Redis Vector Search exceeds a strict threshold (e.g., cos_sim >= 0.96), the gateway returns the stored response immediately without invoking the LLM API.

---

Code Implementation: Redis Semantic Caching Gateway§

Below is a TypeScript example demonstrating a semantic caching middleware using Redis vector search:

import { createClient } from "redis";
import { OpenAIEmbeddings } from "@langchain/openai";

const redis = createClient({ url: process.env.REDIS_URL });
const embeddings = new OpenAIEmbeddings({ modelName: "text-embedding-3-small" });

export async function checkSemanticCache(userPrompt: string, threshold = 0.96) {
  await redis.connect();
  const queryVector = await embeddings.embedQuery(userPrompt);
  
  const searchResults = await redis.ft.search(
    "idx:prompt_cache",
    "*=>[KNN 1 @vector $V AS distance]",
    {
      PARAMS: { V: Buffer.from(new Float32Array(queryVector).buffer) },
      RETURN: ["response", "distance"],
      DIALECT: 2,
    }
  );

  if (searchResults.total > 0) {
    const topMatch = searchResults.documents[0];
    const similarity = 1 - parseFloat(topMatch.value.distance as string);
    
    if (similarity >= threshold) {
      console.log(`Cache HIT! Similarity: ${similarity.toFixed(4)}`);
      return topMatch.value.response as string;
    }
  }

  console.log("Cache MISS. Forwarding to LLM provider...");
  return null;
}

---

Production Results & ROI Impact§

• API Token Savings: Reduced monthly API billable tokens from 1.2 billion to 240 million (80% reduction).
• Latency Reduction: Average TTFT dropped from 480ms to 12ms for cached queries.
• Infrastructure Overhead: Redis Vector instance running costs represented under 3% of the total monthly savings.