What Just Happened§

We deployed a multi-turn transaction agent for booking hotel reservations. Our LLM-as-Judge evaluation pipeline was scoring 95% success. Then we ran a manual audit and found only 78% of conversations actually completed the transaction correctly. That’s a 17% gap – a blind spot that means the LLM judge was missing nearly one in five failures. The root cause? The judge couldn’t track conversation state across turns, and transactions broke silently.

Why This Matters for AI Practitioners§

As an AI practitioner, you’re likely using LLM-as-Judge to evaluate your agent’s performance. It’s fast, cheap, and correlates decently with human judgment – until it doesn’t. In multi-turn settings involving transactions (bookings, purchases, cancellations), the judge’s lack of state awareness becomes a critical weakness. It will happily give a passing score to a conversation where the agent confirmed a booking that was never actually created, or charged the wrong amount. These are not edge cases; they are systematic blind spots.

The implications are severe. Undetected failures lead to customer complaints, chargebacks, and lost revenue. They also poison your training data – if you use LLM judge scores to filter or weight training examples, you’ll reinforce the very errors you want to eliminate. I’ve seen teams waste months optimizing agent behavior that was already “perfect” according to the judge, only to find the real-world accuracy was abysmal.

The scale of the problem is worse than you think. In our production pipeline, the LLM judge’s true positive rate for detecting transaction failures was only 22%. That means 78% of failed transactions were never flagged. This isn’t a fluke – it’s a fundamental limitation of treating judges as black-box scorers on entire conversations.

Who Is Affected§

If you are building any kind of conversational agent that carries out multi-step transactions, you are affected. This includes:

  • E-commerce assistants handling checkout or returns
  • Travel booking agents (flights, hotels, car rentals)
  • Customer support bots that process orders, cancellations, or refunds
  • Financial service agents for transfers, payments, or account changes

Teams that rely heavily on LLM-as-Judge for evaluation, especially those using generic prompts like “Did the agent successfully complete the task?”, are most at risk. Even sophisticated setups that chain together multiple judge calls per turn suffer because each call is contextually independent – there’s no memory of the overall transaction state.

I’ve also seen this affect internal tooling. For example, teams at companies using Claude or GPT-4 to evaluate their own agents on platforms like LangSmith or Weights & Biases get impressive dashboards that hide systematic failures. The tools themselves aren’t the problem; it’s the assumption that an LLM can holistically evaluate a multi-turn transactional workflow without explicit state tracking.

[Loading prompt card for Claude...]

How to Use This Right Now§

You can start fixing this today. The key insight: separate the evaluation of conversation quality from transactional correctness. Use the LLM judge for what it’s good at (coherence, tone, helpfulness) and add a separate, structured mechanism to verify the transaction’s state at each critical step.

Here’s a concrete approach I’ve implemented in production:

  1. Define a canonical transaction state machine. For our hotel booking agent, states are searching, room_selected, payment_details_collected, booking_confirmed, booking_failed. Each turn should transition the state or maintain it.
  1. Inject state expectations into your LLM judge prompt. Ask the judge to output a structured assessment of whether the state transitioned correctly. Below is an example prompt that forces the judge to reason about state.
You are evaluating an assistant that handles hotel booking conversations.

Given the conversation history below, determine:
- Did the assistant correctly progress the transaction through all required states? (states: search -> select -> payment -> confirm)
- If the transaction completed successfully, was a booking ID returned?
- Are there any contradictions in the state (e.g., charging before selecting a room)?

Provide your evaluation as JSON:
{
  "state_progression_correct": true/false,
  "booking_id_present": true/false,
  "contradictions": ["description of any contradictions"]
}

Conversation:
{{CONVERSATION}}
  1. Parse the judge’s structured output and compare against ground truth. We log every state transition from the agent’s own execution. If the judge says state_progression_correct: true but the actual state never reached confirmed, that’s a blind spot. Flag it for manual review.
  1. Combine with rule-based assertions. For each transaction, run a set of deterministic checks: “Did the agent call the booking API with the correct parameters?”, “Is the total price consistent across turns?”, “Did the agent provide a booking reference?”. These can be implemented as simple Python functions. You don’t need LLMs for arithmetic or database lookups.

I’ve found that using a hybrid evaluation (LLM judge + rule-based assertions) catches over 90% of transaction failures. The LLM judge alone catches only 22%. The rules alone catch about 70% (but miss failures in tone or subtle missteps). Together, they are far more robust.

Tool integrations: You can implement this with any agent framework. In LangChain, add a custom callback that logs state transitions. Then use a separate evaluation pipeline in LangSmith or use DSPy to build a structured judge. For monitoring, send the structured evaluation data to Arize or whyLabs for drift detection.

Related Tools on LLMDB.APP§

  • LangChain: State management callbacks and evaluators for multi-turn agents. Use BaseCallbackHandler to record state. Pair with EvaluatorCallback for structured judgment.
  • DSPy: Allows you to build custom evaluation modules with structured inputs and outputs. Use it to create a judge that outputs JSON with state consistency checks.
  • Weights & Biases: Log full conversation traces and evaluation scores. Create custom charts to compare LLM judge results against rule-based checks.
  • Arize: Monitor your agent’s performance in production, including state transition errors and LLM judge bias. Set up alerts when the divergence between judges and reality exceeds a threshold.
  • **DeepSeek**: For cost-effective evaluation, use DeepSeek’s API with a structured prompt similar to the one above. It often catches state errors that GPT-4 misses because of its coding-oriented training.
  • Claude: Its long context window makes it good for reviewing entire conversations. But still add structured output constraints to force state reasoning.
[Loading prompt card for DeepSeek Chat...]

Don’t fall into the trap of trusting your LLM judge blindly. Add structured state tracking and rule checks. That’s how you catch the one in five.