RAG-Accuracy Reward for Retriever Fine-Tuning

Each cell = full-corpus recall R@1·5·10·15 (×100) and acc = downstream RAG answer accuracy (cover-EM, fixed Qwen2.5-7B reader over top-5). Bright text = best R@5 in column. Mean = avg recall over the 5 datasets.
Cell fill = the cell's average recall vs zero-shot on that dataset: greener = better redder = worse (saturates at 6 pts).
Rows. base baselines: zero-shot and Tevatron (standard relevance recipe, InfoNCE+BM25 hard negs, no reader). ours anchored MNRL + precomputed reader guide, opt. WiSE-FT: vLogit = guide by reader logP(answer); vJudge = guide by k/8 cover-EM accuracy. online live RAG-acc reward (PPO, vLogit). Same training queries within a section.
anchor=1.0 marks runs trained with the L2-to-frozen-base anti-forgetting regularizer (all train_mnrl-based rows); anchor=0 = none (Tevatron). Comparisons are within-anchor-policy; zero-shot is the unmodified base model (same across all data mixes).

Setup. Pools are the policy's own retrievals only — the gold passage is never force-seeded and never used as a contrastive positive. The only supervision is the gold answer inside the frozen-reader reward. This is the honest annotation-free regime.
Headline (negative, robust). Across ~25 runs — PPO/GRPO × deterministic/sampled pools × dense/binary-EM reward × LR/N/KL × 20k–149k data × MiniLM→4B — no annotation-free variant beats its zero-shot base, and all trail gold-supervised Rel-SFT. Stochastic pool sampling collapses unconditionally (R@5 < 5, rows omitted); deterministic top-N is stable but capped at ≈base. Weak backbones: MiniLM [no-gold] 32.0 HotpotQA cEM (≈base), bert collapses (near-random base index). Full grid: EXPERIMENTS.md §11b.

Supervision caveat. In every “Ours” row below the gold passage is force-seeded into each candidate pool and used as the InfoNCE positive — the same passage-level supervision Rel-SFT consumes. These rows compare training signals (reader-reward RL vs contrastive) at equal supervision; they are not annotation-free and are kept only as reference.

Setup. Retriever trained by PPO where the reward is a frozen reader LLM's answer success (log p(gold answer | query, retrieved context)); the gold passage is seeded into every pool ([gold]). Baseline Rel-SFT = direct contrastive supervision on gold passages, the best of a 24-config sweep. All trained on MuSiQue; evaluated in-domain + 4 OOD sets.
Metrics. R@k = full-corpus recall (retrieval quality). Acc = downstream RAG cover-EM of a frozen Qwen2.5-7B reader over the top-5 (what the reward actually targets). Bold = best in its backbone×dataset group.
Headline. Indirect RL matches or beats a well-tuned relevance-SFT baseline on multi-hop (2Wiki/HotpotQA) retrieval and downstream accuracy, at every backbone — largest on the weak bert (2Wiki R@5 53.6 vs 36.3) — using no relevance annotation. It is a multi-hop phenomenon: on single-hop NQ/TriviaQA a MuSiQue-trained retriever trails (out-of-domain transfer).