RAG Efficiency, Part 1: Boost Retrieval Accuracy 25% With Rerankers

Your RAG isn't slow because of the LLM. It's slow because you're asking the wrong model to rank.

But before the how, the claim everyone repeats: "rerankers boost accuracy 25%." Is that real?

How Real Is the 25%?

It's real, and it's also the most cherry-picked number in RAG. Both things are true.

The honest version, from what I've measured and seen across recent benchmarks:

• Adding a reranker routinely lifts Hit@1 from the low 60s to the low 80s. That's roughly +20 points, meaning 20 of every 100 queries that used to return the wrong document first now return the right one, for under 250ms of added latency.
• Precision gains over embeddings alone land in the 15 to 40% range depending on the corpus.
• Out-of-domain and long-tail query sets see the biggest jumps, often +15 to +35% relative nDCG.

So where does "25%" come from? The big lifts all share a profile: a weak or generic first-stage retriever, and out-of-domain or ambiguous queries. That's exactly where a bi-encoder's compression hurts most, so the reranker has the most room to fix things.

Flip the conditions and the number collapses. On a strong, in-domain retriever, the same lift shrinks to roughly a single nDCG point. Not 25%. Closer to 1%.

The weaker your retrieval and the further your queries drift from your training distribution, the bigger the reranker win. Quote the 25% if you measured it on your data. Don't quote it because a blog did.

Why Cross-Attention Wins

The reason any of this works comes down to architecture. Bi-encoders compress the query and each document into separate vectors at ingest, then compare with cosine similarity. Two independent compressions, one dot product. Nuance dies before the comparison starts.

Cross-encoders don't compress. They take query and document together, run them through a transformer in one pass, and let cross-attention work across every token pair. Three signals this catches that bi-encoders miss:

• Term disambiguation by context. "Bank" near "river" vs. "bank" near "deposit." A bi-encoder embeds the word once and prays.
• Semantic entailment. Query asks "causes of inflation," document says "why prices rise." No keyword overlap. Cross-encoders almost always catch it.
• Negation. "Not recommended for patients with X" vs. "recommended for patients with X." A bi-encoder's cosine score barely moves. A cross-encoder flips hard.

If your domain is legal, medical, or support over policy docs, this is the gap between RAG you ship and RAG you demo.

The Three-Stage Architecture

Most teams stop at one stage. The pattern that actually holds up in production is three, each one filtering cheaper than the next can afford to.

Stage 1: retrieve wide (recall). Cast the widest cheap net you can. Run hybrid retrieval: a sparse signal (BM25 or a learned sparse model) for exact terms and rare words, plus a dense bi-encoder (Qwen3-Embedding, BGE-M3, or your embedding model of choice) for synonyms and paraphrase. Fuse the two with Reciprocal Rank Fusion. Return the top 100 to 200. You over-retrieve on purpose, because the good answers routinely sit at ranks 30 to 80 where the bi-encoder buried them. Embeddings are precomputed at ingest, so query-time cost is one encode plus an ANN lookup, comfortably under 50ms.

Stage 2: cross-encoder filter (precision). Take those 100 to 200 candidates and score every (query, document) pair with a fast encoder cross-encoder. This is where gte-reranker-modernbert (149M) or bge-reranker-v2-m3 earns its place: a single forward pass per pair, cross-attention across both texts, one scalar relevance score out. Narrow to the top 20 to 25. On a GPU with batching this runs in roughly 100 to 150ms, and it throws out the obvious junk so the expensive stage never sees it.

Stage 3: heavy judge (optional, precision-critical). For the queries that actually matter, reorder the surviving 20 to 25 with a heavyweight: a modern LLM used as a listwise judge, or a large decoder reranker like Qwen3-Reranker-8B. Listwise means the model sees all the candidates together and reasons about relative ordering, not just pointwise scores. It's slow, often seconds, which is exactly why it only ever touches a tiny pre-filtered set. Return the top 5 to the generator.

The discipline is in the funnel. Retrieval proposes cheaply, the cross-encoder filters at moderate cost, and the heavy judge verifies only what's left. Skipping the middle stage and feeding 200 candidates straight to an LLM judge is how you set fire to your latency budget and your bill.

Cross-Encoder vs. LLM Reranker

Since "just use an LLM to rerank" is now a common reflex, the real question is whether it beats a purpose-built cross-encoder. From what I've measured:

• A strong modern LLM as a listwise reranker is on par with a good cross-encoder, and sometimes better on out-of-domain queries.
• Smaller or older LLMs fall short and will sometimes degrade the retriever's own ordering.
• The killer is latency and cost. An LLM reranking a few dozen documents runs in seconds; a cross-encoder does the same in tens of milliseconds. At any real QPS, the LLM-as-workhorse math doesn't close.

So LLM rerankers aren't a replacement for cross-encoders. They're the top of the cascade, used sparingly on a tiny candidate set, exactly as Stage 3 above.

The Latest Models (Bigger Is Not Better)

The reranker landscape shifted hard in the last year. What's worth knowing:

• Qwen3-Reranker (0.6B / 4B / 8B, Apache 2.0). A decoder-style cross-encoder that scores relevance via the probability of a "yes" token. 32K context, 100+ languages, and instruction-aware (custom instructions add 1 to 5%). Strong quality, but autoregressive decoding makes it expensive.
• gte-reranker-modernbert-base (149M). A classic encoder cross-encoder, single forward pass. On recent benchmarks it beats rerankers an order of magnitude larger on both accuracy and latency. 149M params out-ranking 4B is not a typo.
• jina-reranker-v3. Listwise, processes up to 64 docs in a 131k-token window. Around 81% Hit@1 at sub-200ms is best-in-tier for latency.
• bge-reranker-v2-m3. The boring, correct open baseline. Lightweight, multilingual, easy to self-host. If a newer model doesn't clearly beat it on your data, don't pay the extra cost.
• Cohere Rerank 3.5 / v4 and Voyage 2.5. Closed APIs, strong English robustness, roughly 100 to 210ms, zero ops.

The lesson isn't "use the newest." The field split into two families: encoder cross-encoders (one forward pass, fast) and decoder/LLM rerankers (more capable, autoregressive, slower). Parameter count stopped predicting quality. A 149M encoder can beat a multi-billion-parameter decoder on your latency budget. Benchmark on your own corpus before you believe any leaderboard.

Fine-Tuning Cross-Encoders

A general reranker is good. A reranker fine-tuned on your domain is the single highest-leverage upgrade most teams skip. Three things that actually matter:

• Hard negatives are everything. Don't train on random negatives, they're too easy and the model learns nothing. Mine negatives from your own first-stage retriever: take its top-k, drop the known positives, and the leftovers are the confusable documents you actually need the reranker to separate.
• Distillation gets you LLM quality at encoder speed. Use a slow, strong teacher (a large reranker or an LLM judge) to score pairs, then train a small student to match those scores with MarginMSE. This is how you ship LLM-grade ranking on a model small enough to serve cheaply.
• Domain adaptation pays most where general models are weakest, the jargon-heavy, negation-sensitive domains. The CrossEncoder trainer in Sentence-Transformers makes the loop accessible.

You usually need far less labeled data than you'd expect, because you're adapting a strong pretrained base, not training from scratch.

Where Latency Dies (and How to Fix It)

Reranking is O(N) forward passes. The naive version destroys p99. Real numbers for a MiniLM-class encoder over 100 candidates:

• CPU, batch=1: 2500ms. The trap.
• CPU, batch=32: 800ms.
• GPU, batch=32: 150ms. Defensible.
• GPU, batch=64: 120ms.

Levers: batch the pairs (single-pair predict calls are the most common mistake), use a GPU, cache scores for repeated (query, doc) pairs (30 to 50% hit rates are achievable), and pick model size deliberately.

Production Deployment Patterns

Two patterns take reranking from notebook to production.

Async re-ranking service. Don't block the user's request on a GPU-bound reranker. Return the first-stage top-k immediately as provisional results, then rerank asynchronously and stream the refined order. Run the reranker as its own microservice behind a queue so it can batch requests across users (dynamic batching is the biggest throughput win you'll find), scale independently from CPU-bound retrieval, and fall back to first-stage order via a circuit breaker if it's slow or down. Retrieval proposes, the reranker verifies, but the verifier never holds the hot path hostage.

Model serving with Triton. NVIDIA Triton Inference Server is the right home for the cross-encoder. The features that matter: dynamic batching (Triton groups concurrent requests within a small delay window, which is where most of your throughput comes from), multiple concurrent model instances per GPU, and an ONNX or TensorRT backend that typically buys 2 to 4x over raw PyTorch. Serve the tokenizer and model as an ensemble so clients send raw text, and use the metrics endpoint for real p50/p99 visibility. This is the difference between a reranker that works in a demo and one that holds SLA under load.

When to Skip It

• Sub-50ms total RAG budget. Improve the first stage instead.
• Top-10 already right 95%+ of the time. You're paying to reorder what the LLM reads anyway.
• Retrieval not yet validated as the bottleneck. Measure first. Most teams instrument the wrong stage.

What I'd Actually Ship

• Stage 1: hybrid retrieval (sparse plus a modern bi-encoder, fused with RRF), top 100 to 200.
• Stage 2: a fine-tuned encoder cross-encoder (start from gte-reranker-modernbert or bge-reranker-v2-m3) on GPU via Triton with dynamic batching, narrow to top 20 to 25.
• Optional Stage 3: a modern LLM judge or Qwen3-Reranker over those 20 to 25 only, for precision-critical queries, top 5 out.
• Cache (query, doc) scores. Bypass switch for high-confidence exact matches.
• Offline eval with NDCG@10 and MRR before and after. If reranking doesn't move the needle 5+ points on your data, your retriever is already good enough. Don't add a stage you can't justify with numbers.

Reranking is one of the highest-leverage moves in RAG and one of the most over-applied. Use it where compression hurts you. Skip it everywhere else. And measure your own 25%.

Sources

• Qwen3 Embedding
• Cross-Encoder Rediscovers a Semantic Variant of BM25
• Hugging Face: Training a reranker