PReMind: When Personal Memory Meets Visual Hazards
5 tiếng nói · chưa gắn nhãn trưởng lab · 1 nền tảng · 4 tương tác đã đo · Nguồn crawl
Chi tiết nguồn
Bài báo nói rằng trợ lý thị giác-ngôn ngữ được cá nhân hóa sử dụng quan sát hình ảnh và bộ nhớ cá nhân để đưa ra lời khuyên phù hợp. Khi cảnh có nguy hiểm vật lý, bối cảnh cá nhân như dị ứng, bệnh lý hoặc ràng buộc vận động có thể làm thay đổi mức độ trợ giúp phù hợp; do đó cần nhận biết nguy hiểm và áp dụng thông tin đúng của người dùng. Họ giới thiệu PReMind-Bench để đánh giá đồng thời kết hợp bộ nhớ cá nhân và các nguy hiểm, và đề xuất PReMind—khung reinforcement learning hai pha—để huấn luyện đồng thời grounded memory use và hazard response.
Personalized vision-language assistants use visual observations and personal memories to provide advice tailored to the user. When the observed scene contains a physical hazard, personal context, such as allergies, medical conditions, or mobility constraints, can change what help is appropriate. Responding appropriately therefore requires recognizing the visible hazard and applying the relevant person's information. We introduce PReMind-Bench, a diagnostic benchmark that evaluates this combination across diverse hazards and personal-memory conditions. It separately measures memory access, grounding, abstention, and hazard response, and assesses joint success on interdependency cases. Existing models often satisfy one requirement while failing the other, and training only for safety or personalization leaves a gap in joint success. We therefore propose PReMind, a two-phase reinforcement learning framework that combines on-demand and forced-call interactions (memory always retrieved) to jointly train grounded memory use and hazard response. At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test. Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups. These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test.
At 4B, PReMind raises joint success from 12.9% to 28.6% over its base on the interdependency set and improves hazard adequacy and correct abstention on the main test.
Joint-success gains persist under forced-call evaluation and under a second LLM judge across three Phase II seeds; averaged over these seeds, joint success also improves on unseen hazard groups.
These findings highlight the need to evaluate and train personal assistants to connect visual evidence with the relevant person's context.
- PReMind: When Personal Memory Meets Visual Hazards Hugging Face Papers · 18 thg 11, 2026