The High Cost of Context Bloat§

As LLM context windows expand to 1M+ tokens, developers are tempted to dump entire documentation sites, codebases, and chat transcripts directly into prompt payloads. However, processing massive context payloads introduces severe penalties:

  • Exponential API Billing: Pay-per-token costs scale linearly with prompt length.
  • Latency Spikes: Processing 100k tokens adds 1.5 - 3.0 seconds to time-to-first-token.
  • Lost-in-the-Middle Phenomenon: LLMs pay less attention to context positioned in the middle of long prompts.

Prompt compression algorithms solve this problem by intelligently stripping low-information tokens before sending payloads to foundation models.

---

Compression Techniques Breakdown§

1. Perplexity-Based Token Pruning (LLMLingua)§

Uses a small, fast language model (e.g., GPT-2 Small or Llama-3-8B) to compute the perplexity (surprise) of each token in the prompt. Words with low perplexity (highly predictable filler like "in order to", "it should be noted that") are discarded, while high-information keywords are preserved.

2. Abstract Syntax Tree (AST) Code Pruning§

For software engineering prompts, an AST parser inspects source files and strips:

  • Single-line and multi-line comments.
  • Non-exported internal helper functions unused by the target module.
  • Type annotations not referenced in the target function signature.