RAG-Accuracy Reward for Retriever Fine-Tuning

Each cell = full-corpus recall R@1·5·10·15 (×100) and acc = downstream RAG answer accuracy (cover-EM, fixed Qwen2.5-7B reader over top-5). Bright text = best R@5 in column. Mean = avg recall over the 5 datasets.
Cell fill = the cell's average recall vs zero-shot on that dataset: greener = better redder = worse (saturates at 6 pts).
Rows. base baselines: zero-shot and Tevatron (standard relevance recipe, InfoNCE+BM25 hard negs, no reader). ours anchored MNRL + precomputed reader guide, opt. WiSE-FT: vLogit = guide by reader logP(answer); vJudge = guide by k/8 cover-EM accuracy. online live RAG-acc reward (PPO, vLogit). Same training queries within a section.
anchor=1.0 marks runs trained with the L2-to-frozen-base anti-forgetting regularizer (all train_mnrl-based rows); anchor=0 = none (Tevatron). Comparisons are within-anchor-policy; zero-shot is the unmodified base model (same across all data mixes).

Setup. Retriever trained by PPO where the reward is a frozen reader LLM's answer success (log p(gold answer | query, retrieved context)) — no passage-level relevance labels. Baseline Rel-SFT = direct contrastive supervision on gold passages, the best of a 24-config sweep. All trained on MuSiQue; evaluated in-domain + 4 OOD sets.
Metrics. R@k = full-corpus recall (retrieval quality). Acc = downstream RAG cover-EM of a frozen Qwen2.5-7B reader over the top-5 (what the reward actually targets). Bold = best in its backbone×dataset group.
Headline. Indirect RL matches or beats a well-tuned relevance-SFT baseline on multi-hop (2Wiki/HotpotQA) retrieval and downstream accuracy, at every backbone — largest on the weak bert (2Wiki R@5 53.6 vs 36.3) — using no relevance annotation. It is a multi-hop phenomenon: on single-hop NQ/TriviaQA a MuSiQue-trained retriever trails (out-of-domain transfer).