Aligning Language Models with Observational Data: Opportunities and Risks from a Causal Perspective
利用观测数据对齐语言模型：因果视角下的机会与风险AI 解读 ↓
← 返回论文列表与筛选
阅读说明
本页整理公开书目信息，不生成学术质量评分。AI 解读根据题名及所注明的摘要或原文片段生成，未完成全文核验，可能有误；具体假设、证明与结论请核对原文。摘要保留来源文本，未经模型改写。
DOI 首次登记不等于接收或发表日期。部分更正、评论等条目没有独立摘要；缺少可靠材料时保留待补状态。
本页为预印本版本；arXiv 收录、作者填写的期刊信息以及作者既有发表经历，都不单独证明本篇已接收。作者发表记录按 OpenAlex 身份与完整姓名、本站覆盖期刊 ISSN 及正式卷期核对，只展示已核实例，档案仍可能误合并。
引用量采用 OpenAlex 单条关联记录，不合计多个记录；关联记录可能包含正式发表版本，不等于此版本单独引用。缺失不视为零，引用指标不代表论文质量。
关注作者按书目中的完整姓名匹配，同名和姓名变体需人工辨别。相关论文先按共享细分方向和具体方法排列，再参考通用方法与发表日期。
分类依据与纠错
AI 根据所提供的论文材料归类：摘要以观测数据中的混杂、因果关系恢复和去混杂奖励为方法核心，直接支持因果推断方向。
反馈分类问题
正在核对关联版本…
代码、数据与补充材料
正在核对对应版本的资源…
关注这篇论文的作者
利用观测数据对齐语言模型：因果视角下的机会与风险
- 研究问题
- 利用历史观测结局微调语言模型时，如何避免模型学习混杂造成的虚假关联，并使生成内容更好地服务目标结局？
- 方法与思路
- 作者分析观测监督的混杂风险，提出DeconfoundLM，从奖励信号中显式移除已知混杂变量影响，并进行实证与模拟比较。
- 材料陈述的贡献
- 摘要报告真实数据中的虚假关联证据，并称在存在纠缠混杂的模拟中，所提方法较ODIN等基准的目标分数高出16%以上。
- 阅读关注点
- 应核对已知混杂变量如何进入奖励调整、因果恢复所需条件，以及模拟混杂结构、目标分数定义和基准比较是否公平。
- 阅读边界
- 这是arXiv预印本，同行评审或接收未核实；摘要未说明调整的具体识别条件，已知混杂调整不能据此保证消除未知混杂。
原文摘要
Large language models are being widely used across industries to generate text that contributes directly to key performance metrics, such as medication adherence in patient messaging and conversion rates in content generation. Pretrained models, however, often fall short when it comes to aligning with human preferences or optimizing for business objectives. As a result, fine-tuning with good-quality labeled data is essential to guide models to generate content that achieves better results. Controlled experiments, like A/B tests, can provide such data, but they are often expensive and come with significant engineering, logistical, and ethical challenges. Meanwhile, companies have access to a vast amount of historical (observational) data that remains underutilized. In this work, we study the challenges and opportunities of fine-tuning LLMs using observational data. We show that while observational outcomes can provide valuable supervision, directly fine-tuning models on such data can lead them to learn spurious correlations. We present empirical evidence of this issue using various real-world datasets and propose DeconfoundLM, a method that explicitly removes the effect of known confounders from reward signals. In simulation experiments, DeconfoundLM more accurately recovers causal relationships and mitigates failure modes of methods that assume counterfactual invariance, achieving over 16% higher objective score than ODIN and other baselines, when entangled confounding is present. Please refer to the project page for code and related resources.