Optimizing RAG Pipelines with Hybrid Search and Re-ranking: A Technical Blueprint

In this post, I’m going to walk you through the exact process I used to optimize a RAG pipeline that was returning disappointingly irrelevant results. The fix wasn’t a better LLM — it was a better retrieval strategy. I combined hybrid search (dense embeddings + sparse keyword matching) with a cross-encoder re-ranking step. This blueprint is based on a real project I built for a legal-tech client, but the principles apply across domains.

The Problem I Was Trying to Solve§

My client had a knowledge base of 5,000+ policy documents, legal contracts, and compliance guidelines. They wanted a chat interface that could answer questions like “What are the termination clauses for force majeure?” or “Which subcontractor is responsible for data protection?” The existing RAG pipeline used vanilla vector search with OpenAI embeddings (text-embedding-3-large) over a Pinecone index. It worked fine for generic questions, but failed on any query that contained exact terms like "Force Majeure" or "§ 3.2(b)" — the semantic embedding often matched conceptually similar but legally irrelevant clauses.

The core problem was twofold. First, dense embeddings alone miss exact keyword matches. If a user asks about "termination for convenience," the embedding might pull chunks about termination for breach, just because they’re semantically related. Second, vector search returns the top-k chunks based on cosine similarity, which is a weak proxy for actual relevance. The LLM then has to do the heavy lifting of picking out the right context, and when the context is wrong, the answer is wrong.

I needed a retrieval system that could understand both lexical and semantic similarity, and then re-rank the initial results to ensure the most relevant chunks landed in the LLM’s context window.

Tools and Setup§

I’m a Python developer, and I was already using Cursor as my IDE. For this project, I decided to migrate from Pinecone to Qdrant because Qdrant natively supports hybrid search with a sparse vector representation (using BM25 or SPLADE) alongside dense vectors. This eliminated the need to maintain two separate indexes.

Here’s the stack I used:

• Vector database: Qdrant (open-source, running in Docker)
• Dense embeddings: fastembed (the BAAI/bge-small-en-v1.5 model) — faster and cheaper than OpenAI embeddings, and we were already seeing decent results.
• Sparse embeddings: Qdrant’s built-in BM25 implementation (using their Qdrant/BGE-SPLADE model for sparse encoding, or just raw BM25 tokenization).
• Re-ranker: I initially used Cohere Rerank, but for local testing I moved to a cross-encoder model from sentence-transformers (cross-encoder/ms-marco-MiniLM-L-6-v2).
• LLM: I experimented with DeepSeek (via their API) and Claude (via Anthropic SDK). DeepSeek was the final choice for its cost-to-quality ratio on legal text.
• Framework: I strayed from LangChain and used raw Python with the Qdrant client. This gave me more control and fewer magic abstractions.

For research, I used Perplexity to quickly check the latest Qdrant hybrid query syntax and re-ranking best practices. This saved me a lot of time.

Step-by-Step: What I Actually Did§

Step 1: Index both dense and sparse vectors.

I created a Qdrant collection with named vectors: a dense vector from fastembed, and a sparse vector from Qdrant’s BM25 tokenizer. I set up the collection like this:

from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, SparseVectorParams, Distance, 

client = QdrantClient(":memory:")

client.create_collection(
    collection_name="documents",
    vectors_config={
        "dense": VectorParams(size=384, distance=Distance.COSINE)
    },
    sparse_vectors_config={
        "sparse": SparseVectorParams()
    }
)

Then I wrote a function that splits documents into 512-token chunks with 128-token overlap, and for each chunk, computed both the dense embedding and the sparse indices/values using Qdrant’s BM25 encoder.

Step 2: Perform hybrid search.

When a user query came in, I’d search the collection using a HybridFusion query that merges the dense and sparse results. I used RRF (Reciprocal Rank Fusion) with a rank constant of 60. This combines the scores from both searches, giving more weight to items that appear high in both result sets.

Step 3: Re-rank with a cross-encoder.

The hybrid search returns top 50 candidates. But 50 is too many to stuff into an LLM context window, so I needed to narrow it down to 5. Re-ranking was the answer. I used a cross-encoder that takes a (query, chunk) pair and outputs a relevance score. This is more accurate than vector similarity because the model actually reads both texts together.

I batched the pairs and ran inference:

from sentence_transformers import CrossEncoder

cross_encoder = CrossEncoder('cross-encoder/ms-marco-MiniLM-L-6-v2')

pairs = [(query, chunk) for chunk in candidates]
scores = cross_encoder.predict(pairs)
top_indices = np.argsort(scores)[::-1][:5]

Step 4: Build the final prompt.

I then passed the top 5 re-ranked chunks to the LLM. I formatted the prompt with a clear instruction that the context may not all be relevant, but to prioritize the first chunk, which is the highest ranked.

Code Samples / Prompts Used§

Here’s the core retrieval function I ended up with. It’s simplified for readability but captures the full flow.

import numpy as np
from qdrant_client import QdrantClient
from qdrant_client.models import (
    Filter, FieldCondition, MatchValue, 
    HybridFusion, RRF, ShardKeySelector, PointIdsList,
)

client = QdrantClient(host="localhost", port=6333)

def retrieve_and_rerank(query, top_k=50, final_k=5):
    # 1. Build dense and sparse query vectors
    dense_vec = embedder.embed_query(query)  # from fastembed
    sparse_vec = sparse_encoder.encode(query)  # BM25 sparse

    # 2. Hybrid search
    hybrid_results = client.query_points(
        collection_name="documents",
        prefetch=[
            {
                "query": (dense_vec.tolist()),
                "using": "dense",
                "limit": top_k
            },
            {
                "query": (sparse_vec.indices, sparse_vec.values),
                "using": "sparse",
                "limit": top_k
            }
        ],
        query=HybridFusion(
            fusion=RRF(rank_const=60)
        ),
        limit=top_k
    )
    candidates = [hit.payload for hit in hybrid_results.points]

    # 3. Re-rank with cross-encoder
    pairs = [(query, chunk["text"]) for chunk in candidates]
    scores = cross_encoder.predict(pairs)
    top_indices = np.argsort(scores)[::-1][:final_k]

    # 4. Return top chunks
    return [candidates[i] for i in top_indices]

For the LLM prompt, I used a template like this with DeepSeek:

You are a legal assistant. Use the following context segments to answer the user's question. The segments are ordered by relevance. If the answer is not found, say "I don't know."

Context:
1. {chunk1}
2. {chunk2}
...

Question: {query}
Answer:

I also tested with Claude via Cursor’s prompt panel, but the retrieval logic remained the same.

What Worked Well§

The hybrid search + re-ranking combo was a game-changer. Here’s what stood out:

Recall improved significantly. The sparse retrieval caught exact phrase matches that dense vectors missed. Queries with legal citations like “§ 3.2(b)” or company-specific acronyms suddenly worked. The dense retrieval still brought in semantically related concepts, so we got the best of both worlds.

Re-ranking added precision. The cross-encoder didn’t just filter out noise; it also reordered the chunks. In a test case where the correct answer was in chunk #12 according to hybrid ranking, the re-ranker moved it to #1. The LLM’s final answer quality improved dramatically because the context window contained the right information first.

RRF fusion was simple but robust. I didn’t have to tune weights for the dense vs. sparse relative importance. RRF handled the blending automatically, and the rank constant of 60 gave good results across all queries.

Development speed. Using Qdrant’s native hybrid support saved me from writing a lot of glue code. The Qdrant Python client is well-documented, and I had the whole pipeline running in a couple of days.

What Failed and Why§

The first failure was with chunk size. I originally used 1,024 tokens per chunk, thinking longer context would help. But it made retrieval coarse — a chunk might contain a lot of irrelevant text alongside the key sentence. The cross-encoder scores dropped because the model saw too many distractors. Reducing to 512 tokens with 128 overlap solved this.

The second failure was using a naive word-level BM25 sparse encoder. Qdrant’s default tokenizer breaks on punctuation, which is terrible for legal text with “§” and parentheses. It kept returning empty sparse vectors. I had to switch to a SPLADE-based sparse model (Qdrant/bge-sparse-encoder) that handles tokenization better and produces subword embeddings. That stabilized the sparse retrieval.

The third failure was over-relying on the re-ranker. I initially set top_k=100 for the hybrid search, thinking more candidates would be better. But the cross-encoder is O(N^2) pairwise, and with 100 candidates, inference time spiked to over 2 seconds per query. Dropping to top_k=50 reduced latency to ~800ms without measurable quality loss.

Results and Takeaways§

After the changes, I ran a small evaluation set of 100 real user queries from the client’s log. I manually scored the relevance of the top 5 retrieved chunks on a 1–5 scale. The average score went from 2.8 (with dense-only) to 4.4 (with hybrid + re-ranking). The LLM answer accuracy (defined as “exactly matches the client’s legal team review”) jumped from 61% to 83%.

Latency was a trade-off. Dense-only retrieval took about 200ms end-to-end. Hybrid + re-ranking took about 850ms. That’s acceptable for a chat interface, but we would need to consider caching or async processing for high-traffic production.

The single biggest takeaway is: retrieval is the true bottleneck in RAG. You can fine-tune the LLM, but if the right context isn’t in the prompt, the model will hallucinate or give a shallow answer. Hybrid search plus re-ranking is a low-effort, high-impact upgrade.

Try It Yourself§

If you want to apply this to your own project, I recommend a step-by-step approach:

1. Take an existing RAG pipeline that uses dense vector search and has a set of hard queries where it fails.
2. Add sparse retrieval using Qdrant’s hybrid search or use a dedicated BM25 index (Elasticsearch or Tantivy) and merge results with RRF.
3. Integrate a cross-encoder re-ranker — either Cohere Rerank (easy API) or a local model like cross-encoder/ms-marco-MiniLM-L-6-v2 (free, but requires a GPU for scale).
4. Evaluate on your specific domain as I did, not just on generic QA benchmarks.

Don’t trust a single similarity score. Combine signals and re-rank. Your users will notice.

Key Takeaways: here’s the core value of this approach:

• Hybrid search (dense + sparse) catches both semantic and exact-term matches, which is critical for domain-specific RAG.
• Re-ranking with a cross-encoder dramatically improves precision and subsequent LLM answer quality.
• RRF fusion is a simple, effective way to combine dense and sparse results without hyperparameter tuning.
• Always test on real, messy queries — synthetic benchmarks will hide the failures that matter.