Deep approximate policy iteration

原文摘要

In this paper, we consider deep approximate policy iteration (DAPI) with the Bellman residual minimization in reinforcement learning. In each iteration of DAPI, we apply convolutional neural networks (CNNs) with ReLU activation, called ReLU CNNs, to estimate the fixed point of the Bellman equation by minimizing an unbiased minimax loss. To bound the estimation error in each iteration, we control the statistical and approximation errors using the tools of the empirical process theory with dependent data and deep approximation theory, respectively. We establish a novel statistical error bound for ReLU CNNs on dependent data that is C-mixing, and an approximation error bound for ReLU CNNs on Hölder class. Combining with error propagation, we obtain a nonasymptotic error bound between the optimal action-value function Q∗ and the estimated Q function induced by the greedy policy in DAPI. This bound depends on the sample size and ambient dimension of the data, as well as the size, weight bound, and depth of the CNNs, providing prior guidance on how to set these hyperparameters to achieve the desired convergence rate when training DAPI in practice. Moreover, this bound circumvents the curse of dimensionality if the distribution of state-action pairs is supported on a set with a low intrinsic dimension.

摘要来源：OpenAlex 索引摘要（稿件版本未注明）