Deep Learning for Robust Robot Control (DL-foRCe)

Project type

NWO Natural Artificial Intelligence; 2015-2019

Abstract

While robots can flawlessly execute a set of commands to achieve a task, these commands are mostly encoded by hand. There is a need for effective learning methods that can deal with the uncertainty in the robot's environment, in particular when only broad goals are specified, and the learning algorithm has to learn motor commands to achieve these goals. This typically involves reinforcement learning (RL). However, current RL for robotics tasks relies on ad hoc function approximators and is typically not robust to changes in the task, environment, or robot uncertainty (compliant robot actuators, or wear and tear). The aim of this project is to integrate two emerging notions in order to make reinforcement learning for robot control more robust and efficient: dynamic feedback control policies for robust control combined with deep neural networks to learn low-dimensional parameterizations of such control policies. This approach promises a generic and robust approach to reinforcement learning for robotic control.

Project members

ir. Tim de Bruin, Dr.-Ing. Jens Kober, Prof Dr Sander Bohté, Prof Karl Tuyls, prof.dr. Robert Babuška

Publications

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Fine-tuning Deep RL with Gradient-Free Optimization. In 21th IFAC World Congress, pp. 8049-8056, 2020. [bibtex] [pdf] [doi] [code]

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Integrating State Representation Learning into Deep Reinforcement Learning. IEEE Robotics and Automation Letters, 3(3):1394–1401, 2018. The contents of this paper were also selected by ICRA'18 Program Committee for presentation at the Conference. [bibtex] [pdf] [doi]

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Experience Selection in Deep Reinforcement Learning for Control. Journal of Machine Learning Research, 19(9):1–56, 2018. [bibtex] [pdf] [html] [code] [video]

Lucian Buşoniu, Tim de Bruin, Domagoj Tolić, Jens Kober, and Ivana Palunko. Reinforcement Learning for Control: Performance, Stability, and Deep Approximators. Annual Reviews in Control, 46:8–28, 2018. [bibtex] [pdf] [doi]

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Improved Deep Reinforcement Learning for Robotics Through Distribution-based Experience Retention. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3947–3952, 2016. [bibtex] [pdf] [doi] [video]

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. Off Policy Experience Retention for Deep Actor Critic Learning. In Deep Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NIPS), 2016. [bibtex] [pdf]

Tim de Bruin, Jens Kober, Karl Tuyls, and Robert Babuška. The Importance of Experience Replay Database Composition in Deep Reinforcement Learning. In Deep Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NIPS), 2015. [bibtex] [pdf] [video]

Robot-Assistance in Understanding and Educating Tooth Removal Procedures

NWO OpenMind logo

Project type

NWO OpenMind; 2019

Abstract

Though tooth removal is the most commonly performed surgical treatment worldwide, there has not been any real technological advancement in over 2000 years. There is still a lot of stress involved for patients, dental students, and (young) dentists. Up until now there is no scientific understanding of tooth removal procedures in terms of what motions and forces should be applied and this has its consequence on the quality of education. Students are highly undertrained when it comes tooth removal. The only way to prepare for a procedure is to read very limited instructions on ‘rocking and twisting’ motions and, in contrast to all other fields of dentistry, there is no way to practice these treatments pre-clinically. Practicing tooth removal therefore takes place on actual patients.

The goal of this project is to utilize modern robot technology to gain understanding of tooth extraction procedures and to use the acquired data to improve dental education. Bridging the scientific gap that exists in this field will have direct and significant clinical consequences and will lead to more confident dental students and dentists when practicing/performing these procedures, more efficient referral behavior and last but not least reduce discomfort and complication rates for the patient.

Project members

Willem de Graaf (TUD), Tom van Riet (AMC), Dr.-Ing. Jens Kober (TUD)

Publications

Maaike Geertruida Beuling, Tom Cornelis Theodorus van Riet, Jan van Frankenhuyzen, Reinier van Antwerpen, Bas de Blocq van Scheltinga, Arnout Harm Hendrik Dourleijn, Dzan Ireiz, Sander Streefkerk, Jonathan C. van Zanten, Jan de Lange, Jens Kober, and Dimitra Dodou. Development and Testing of a Prototype of a Dental Extraction Trainer with Real-Time Feedback on Forces, Torques, and Angular Velocity. In 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), 2022. [bibtex] [pdf]

Tom Cornelis Theodorus van Riet, Willem Mathys de Graaf, Reinier van Antwerpen, Jan van Frankenhuyzen, Jan de Lange, and Jens Kober. Robot Technology in Analyzing Tooth Removal - A Proof of Concept. abstract: 25th Congress of the European Association for Cranio Maxillo Facial Surgery, 2021. [bibtex]

Tom C.T. van Riet, Kevin T.H. Chin Jen Sem, Jean-Pierre T.F. Ho, René Spijker, Jens Kober, and Jan de Lange. Robot Technology in Dentistry, Part One of a Systematic Review: Literature Characteristics. Dental Materials, 37(8):1217–1226, 2021. [bibtex] [url] [doi]

Tom C.T. van Riet, Kevin T.H. Chin Jen Sem, Jean-Pierre T.F. Ho, René Spijker, Jens Kober, and Jan de Lange. Robot Technology in Dentistry, Part Two of a Systematic Review: An Overview of Initiatives. Dental Materials, 37(8):1227–1236, 2021. [bibtex] [url] [doi]

Tom Cornelis Theodorus van Riet, Willem Mathys de Graaf, Reinier van Antwerpen, Jan van Frankenhuyzen, Jan de Lange, and Jens Kober. Robot Technology in Analyzing Tooth Removal – a Proof of Concept. In 42nd Annual International Conferences of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4721–4727, 2020. [bibtex] [pdf] [doi]

Tom Cornelis Theodorus van Riet, Jens Kober, and Jan de Lange. Robottechnologie, is er een toekomst voor in de tandheelkunde? Quality Practice Tandheelkunde 12(5):30–35, 2017. [bibtex] [url]

Tom Cornelis Theodorus van Riet, Jens Kober, Maarten Griffioen, Xiang Zhang, Piet-Hein van Twisk, Robert Babuška, and Jan de Lange. Op zoek naar toepassingen van robottechnologie in de MKA-chirurgie. In Annual Scientific Meeting of the Dutch Association of Oral and Maxillofacial Surgery, 2016. [bibtex]

Wind Turbine Brain (WT Brain)

WT Brain logo

Project type

TKI Wind op Zee; 2017-2020

Description

Background

Wind turbines behave well in a lot of different wind conditions, in the lightest breeze and the heaviest storms. In the end, it boils down to producing energy at the lowest possible cost. That is why the system that controls the response of a wind turbine to the wind, must keep finding a balance between producing energy and the loads acting on the wind turbine. This project focusses on finding new ways to finding this balance. These new ways are becoming available due to the rapid scientific progress in the fields of (deep) machine learning and the use of, amongst other things, neural networks. By applying these methods in wind turbine control, we expect to realise a 5-10% design equivalent load reduction.

Goal/ objective

The goal of the project is to improve the wind turbine controller and its design process. This will be achieved by using machine learning (ML) to tune the controller in the design phase. The controller will contain conventional estimators and controller structures. Neural or Bayesian networks (hereafter: artificial networks (AN)) will be used to search for opportunities to further improve the conventional controller design.

Work programme
  1. ML methods and AN architectures: AN types are compared and the architecture of an AN-based controller is selected. Based on that selection, appropriate ML methods are selected.
  2. Conventional controller tuning with ML: The selected target function may be difficult to capture for the conventional controller. ML can tune the parameters based on the target function directly.
  3. AN-controller: Incorporate AN in the controller and tune it.
  4. Tuning and testing: The new controller structures are tuned for manufacturer and theoretical wind turbine models
Outcome

Situations where improvements are sought are low, normal and highly turbulent winds, starts and stops and extreme wind conditions, such as extreme operating gusts or extreme direction changes. The improvement is expected to reduce design equivalent loads on the blades and the wind turbine tower and to reduce extreme loads. These reductions can directly reduce the cost of a wind turbine. Alternatively, the load reduction on the blades can be used to employ longer blades, resulting in a higher production.

TUD project members

ir. Sebastiaan P. Mulders, Dr.-Ing. Jens Kober, prof.dr.ir. Jan-Willem van Wingerden

Project consortium

Energy Research Centre of the Netherlands (ECN), Lagerwey Group B.V., TU Delft, Siemens Gamesa

Publications

Nikolaos Moustakis, Sebastiaan Paul Mulders, Jens Kober, and Jan-Willem van Wingerden. A Practical Bayesian Optimization Approach for the Optimal Estimation of the Rotor Effective Wind Speed. In American Control Conference (ACC), pp. 4179–4185, 2019. [bibtex] [pdf] [doi]