8 tiếng nói · chưa gắn nhãn trưởng lab · 2 nền tảng · 3 tương tác đã đo · Nguồn crawl
WJJTY
Wen Wang, Jindong Wang và 6 nguồn khác 1 bài viết · Hugging Face Papers, arXiv cs.AI Biên tập tiếng Việt Chi tiết nguồn
Bài báo nói rằng reinforcement learning từ phần thưởng cuối cùng cho phép search-augmented language models xen kẽ suy luận và truy xuất mà không cần nhãn từng bước, nhưng giám sát theo kết quả gán credit giống nhau cho mọi lần tìm kiếm trong một trajectory, dù có truy xuất hữu ích hay không. Họ giới thiệu Evidence-based Credit Assignment (ECA) để bổ sung tín hiệu phụ (training-only) đo việc các tài liệu đã truy xuất cải thiện xác suất của câu trả lời tham chiếu. ECA dùng evidence-only prompt với chỉ câu hỏi và các tài liệu đã quan sát (đã loại trùng), để tách tiện ích của đoạn truy xuất khỏi biến thiên do câu chữ suy nghĩ/ truy vấn sinh ra.
Reinforcement learning from terminal answer rewards enables search-augmented language models to interleave reasoning and retrieval without step-level annotations, yet outcome supervision assigns identical credit to every search in a trajectory regardless of whether it retrieved useful information. This limitation is especially severe when all sampled rollouts for a question fail, collapsing group-centered outcome advantages to zero even when trajectories gather markedly different evidence. We present Evidence-based Credit Assignment (ECA), which augments terminal outcome rewards with a training-only auxiliary signal measuring how retrieved documents improve the likelihood of a reference answer. Rather than scoring the full dialogue prefix, ECA queries a frozen snapshot of the current policy under an evidence-only prompt containing solely the question and the deduplicated documents observed so far, isolating the utility of retrieved passages from surface variations in generated thoughts or search queries. Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment. Across seven Search QA benchmarks, ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO. Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.
ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO.
ECA-Horizon improves strict macro exact match over an outcome-only GRPO baseline by 1.03 and 4.80 percentage points on Qwen2.5-3B- and 7B-Instruct, respectively, outperforming both RLOO and GiGPO.
Zero-shot evaluation on MultiHop-RAG further demonstrates strong transfer to multi-hop inference queries.
Within this framework, we compare two return formulations, ECA-Step (which accumulates step-clipped potential increments) and ECA-Horizon (which preserves the telescoping horizon gain in potential), paired with cross-fold group normalization, novelty gating, and search-local token assignment.