New JMLR paper accepted: “Non-parametric Policy Search with Limited Information Loss.”

  • H. van Hoof, G. Neumann, and J. Peters, “Non-parametric policy search with limited information loss,” Journal of Machine Learning Research, 2018.
    [BibTeX] [Abstract] [Download PDF]

    Learning complex control policies from non-linear and redundant sensory input is an important challenge for reinforcement learning algorithms. Non-parametric methods that approximate values functions or transition models can address this problem, by adapting to the complexity of the dataset. Yet, many current non-parametric approaches rely on unstable greedy maximization of approximate value functions, which might lead to poor convergence or oscillations in the policy update. A more robust policy update can be obtained by limiting the information loss between successive state-action distributions. In this paper, we develop a policy search algorithm with policy updates that are both robust and non-parametric. Our method can learn non-parametric control policies for infinite horizon continuous Markov decision processes with non-linear and redundant sensory representations. We investigate how we can use approximations of the kernel function to reduce the time requirements of the demanding non-parametric computations. In our experiments, we show the strong performance of the proposed method, and how it can be approximated effi- ciently. Finally, we show that our algorithm can learn a real-robot underpowered swing-up task directly from image data.

    @article{lirolem28020,
    author = {Herke van Hoof and Gerhard Neumann and Jan Peters},
    year = {2018},
    title = {Non-parametric policy search with limited information loss},
    publisher = {Journal of Machine Learning Research},
    journal = {Journal of Machine Learning Research},
    month = {December},
    keywords = {ARRAY(0x56147fc33978)},
    url = {http://eprints.lincoln.ac.uk/28020/},
    abstract = {Learning complex control policies from non-linear and redundant sensory input is an important
    challenge for reinforcement learning algorithms. Non-parametric methods that
    approximate values functions or transition models can address this problem, by adapting
    to the complexity of the dataset. Yet, many current non-parametric approaches rely on
    unstable greedy maximization of approximate value functions, which might lead to poor
    convergence or oscillations in the policy update. A more robust policy update can be obtained
    by limiting the information loss between successive state-action distributions. In this
    paper, we develop a policy search algorithm with policy updates that are both robust and
    non-parametric. Our method can learn non-parametric control policies for infinite horizon
    continuous Markov decision processes with non-linear and redundant sensory representations.
    We investigate how we can use approximations of the kernel function to reduce the
    time requirements of the demanding non-parametric computations. In our experiments, we
    show the strong performance of the proposed method, and how it can be approximated effi-
    ciently. Finally, we show that our algorithm can learn a real-robot underpowered swing-up
    task directly from image data.}
    }