Non-Prehensile Manipulation in Clutter with Human-In-The-Loop

Rafael Papallas and Mehmet R. Dogar

IEEE International Conference on Robotics and Automation (ICRA) 2020, Paris, France


We propose a human-operator guided planning approach to pushing-based manipulation in clutter. Most recent approaches to manipulation in clutter employs randomized planning. The problem, however, remains a challenging one where the planning times are still in the order of tens of seconds or minutes, and the success rates are low for difficult instances of the problem. We build on these control-based randomized planning approaches, but we investigate using them in conjunction with human-operator input. In our framework, the human operator supplies a high-level plan, in the form of an ordered sequence of objects and their approximate goal positions. We present experiments in simulation and on a real robotic setup, where we compare the success rate and planning times of our human-in-the-loop approach with fully autonomous sampling-based planners. We show that with a minimal amount of human input, the low-level planner can solve the problem faster and with higher success rates.

ICRA 2020 Full Presentation Video

Demo Video



Papallas, R. and Dogar, M.R., 2020. Non-Prehensile Manipulation in Clutter with Human-In-The-Loop. IEEE International Conference on Robotics and Automation.


  title={Non-Prehensile Manipulation in Clutter with Human-In-The-Loop},
  author={Papallas, Rafael and Dogar, Mehmet R},
  booktitle={{IEEE} International Conference on Robotics and Automation},


Authors are with the School of Computing, University of Leeds, United Kingdom.

This research has received funding from the European Union’s Horizon 2020 research and innovation programme under the Marie Skłodowska-Curie grants agreement No. 746143, and from the UK Engineering and Physical Sciences Research Council under grant EP/N509681/1, EP/P019560/1 and EP/R031193/1.