Off-Policy Evaluation in Markov Decision Processes under Weak Distributional Overlap
弱分布重叠下马尔可夫决策过程的离策略评估AI 解读 ↓
← 返回论文列表与筛选
阅读说明
本页整理公开书目信息，不生成学术质量评分。AI 解读根据题名及所注明的摘要或原文片段生成，未完成全文核验，可能有误；具体假设、证明与结论请核对原文。摘要保留来源文本，未经模型改写。
DOI 首次登记不等于接收或发表日期。部分更正、评论等条目没有独立摘要；缺少可靠材料时保留待补状态。
本页为预印本版本；arXiv 收录、作者填写的期刊信息以及作者既有发表经历，都不单独证明本篇已接收。作者发表记录按 OpenAlex 身份与完整姓名、本站覆盖期刊 ISSN 及正式卷期核对，只展示已核实例，档案仍可能误合并。
引用量采用 OpenAlex 单条关联记录，不合计多个记录；关联记录可能包含正式发表版本，不等于此版本单独引用。缺失不视为零，引用指标不代表论文质量。
关注作者按书目中的完整姓名匹配，同名和姓名变体需人工辨别。相关论文先按共享细分方向和具体方法排列，再参考通用方法与发表日期。
分类依据与纠错
AI 根据所提供的论文材料归类：摘要在序贯可忽略性与分布重叠条件下评估目标策略，对应序贯干预的因果评估问题。
反馈分类问题
正在核对关联版本…
代码、数据与补充材料
正在核对对应版本的资源…
关注这篇论文的作者
弱分布重叠下马尔可夫决策过程的离策略评估
- 研究问题
- 在序贯可忽略性下，当目标策略与数据采集策略的平稳分布比值不再有界时，如何可靠估计目标策略价值并刻画收敛速度？
- 方法与思路
- 作者提出截断双重稳健估计量，以较弱的分布重叠替代有界比值假设，并分别分析分布比值平方可积与不可积时的行为。
- 材料陈述的贡献
- 摘要称平方可积时可恢复既有大样本性质；不可积时仍一致但慢于根号时间长度速率，并在仅以混合条件定义的模型类上达到极小极大速率。
- 阅读关注点
- 应核对策略价值的目标定义、序贯可忽略性与混合条件、截断水平的选择，以及初步估计误差和弱重叠程度如何共同影响速率。
- 阅读边界
- 依据 arXiv 预印本摘要，不能确认同行评审或期刊接收；摘要未给出完整弱重叠条件及慢速率公式，数值表现仅是作者报告。
原文摘要
Doubly robust methods hold considerable promise for off-policy evaluation in Markov decision processes (MDPs) under sequential ignorability: They have been shown to converge as $1/\sqrt{T}$ with the horizon $T$, to be statistically efficient in large samples, and to allow for modular implementation where preliminary estimation tasks can be executed using standard reinforcement learning techniques. Existing results, however, make heavy use of a strong distributional overlap assumption whereby the stationary distributions of the target policy and the data-collection policy are within a bounded factor of each other -- and this assumption is typically only credible when the state space of the MDP is bounded. In this paper, we re-visit the task of off-policy evaluation in MDPs under a weaker notion of distributional overlap, and introduce a class of truncated doubly robust (TDR) estimators which we find to perform well in this setting. When the distribution ratio of the target and data-collection policies is square-integrable (but not necessarily bounded), our approach recovers the large-sample behavior previously established under strong distributional overlap. When this ratio is not square-integrable, TDR is still consistent but with a slower-than-$1/\sqrt{T}$-rate; furthermore, this rate of convergence is minimax over a class of MDPs defined only using mixing conditions. We validate our approach numerically and find that, in our experiments, appropriate truncation plays a major role in enabling accurate off-policy evaluation when strong distributional overlap does not hold.