Deep Learning for Robust Robot Control (DL-foRCe)
Project type
NWO Natural Artificial Intelligence; 2015-2019
Abstract
While robots can flawlessly execute a set of commands to achieve a task, these commands are mostly encoded by hand. There is a need for effective learning methods that can deal with the uncertainty in the robot's environment, in particular when only broad goals are specified, and the learning algorithm has to learn motor commands to achieve these goals. This typically involves reinforcement learning (RL). However, current RL for robotics tasks relies on ad hoc function approximators and is typically not robust to changes in the task, environment, or robot uncertainty (compliant robot actuators, or wear and tear). The aim of this project is to integrate two emerging notions in order to make reinforcement learning for robot control more robust and efficient: dynamic feedback control policies for robust control combined with deep neural networks to learn low-dimensional parameterizations of such control policies. This approach promises a generic and robust approach to reinforcement learning for robotic control.
Project members
ir. Tim de Bruin, Dr.-Ing. Jens Kober, Prof Dr Sander Bohté, Prof Karl Tuyls, prof.dr. Robert Babuška
Publications
. Fine-tuning Deep RL with Gradient-Free Optimization. In 21th IFAC World Congress, pp. 8049–8056, 2020.
[bibtex] [pdf] [doi] [code] 
. Integrating State Representation Learning into Deep Reinforcement Learning. IEEE Robotics and Automation Letters, 3(3):1394–1401, 2018. The contents of this paper were also selected by ICRA'18 Program Committee for presentation at the Conference.
[bibtex] [pdf] [doi] 
. Experience Selection in Deep Reinforcement Learning for Control. Journal of Machine Learning Research, 19(9):1–56, 2018.
[bibtex] [pdf] [html] [code] [video]
. Reinforcement Learning for Control: Performance, Stability, and Deep Approximators. Annual Reviews in Control, 46:8–28, 2018.
[bibtex] [pdf] [doi]
. Improved Deep Reinforcement Learning for Robotics Through Distribution-based Experience Retention. In IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 3947–3952, 2016.
[bibtex] [pdf] [doi] [video]
. Off Policy Experience Retention for Deep Actor Critic Learning. In Deep Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NIPS), 2016.
[bibtex] [pdf] [url] 
. The Importance of Experience Replay Database Composition in Deep Reinforcement Learning. In Deep Reinforcement Learning Workshop, Advances in Neural Information Processing Systems (NIPS), 2015.
[bibtex] [pdf] [url] [video]
Robot-Assistance in Understanding and Educating Tooth Removal Procedures
Project type
NWO OpenMind; 2019
Abstract
Though tooth removal is the most commonly performed surgical treatment worldwide, there has not been any real technological advancement in over 2000 years. There is still a lot of stress involved for patients, dental students, and (young) dentists. Up until now there is no scientific understanding of tooth removal procedures in terms of what motions and forces should be applied and this has its consequence on the quality of education. Students are highly undertrained when it comes tooth removal. The only way to prepare for a procedure is to read very limited instructions on ‘rocking and twisting’ motions and, in contrast to all other fields of dentistry, there is no way to practice these treatments pre-clinically. Practicing tooth removal therefore takes place on actual patients.
The goal of this project is to utilize modern robot technology to gain understanding of tooth extraction procedures and to use the acquired data to improve dental education. Bridging the scientific gap that exists in this field will have direct and significant clinical consequences and will lead to more confident dental students and dentists when practicing/performing these procedures, more efficient referral behavior and last but not least reduce discomfort and complication rates for the patient.
Project members
Willem de Graaf (TUD), Tom van Riet (AMC), Dr.-Ing. Jens Kober (TUD)
Publications
. Using Robot Technology to Analyze Forces and Torques in Tooth Removal. Advances in Oral and Maxillofacial Surgery, 9:100400, 2023.
[bibtex] [url] [doi]
. Analysis of Movements in Tooth Removal Procedures Using Robot Technology. PLOS ONE, 18:1–12, 2023.
[bibtex] [file] [doi]
. Forces and Movements During Tooth Extraction: A Scoping Review. Advances in Oral and Maxillofacial Surgery, 9:100391, 2023.
[bibtex] [url] [doi]
. A Multiclass Classification Model for Tooth Removal Procedures. Journal of Dental Research, 101(11):1357–1362, 2022.
[bibtex] [url] [doi]
. Development and Testing of a Prototype of a Dental Extraction Trainer with Real-Time Feedback on Forces, Torques, and Angular Velocity. In 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 3285–3290, 2022.
[bibtex] [pdf] [doi]
. Robot Technology in Analyzing Tooth Removal - A Proof of Concept. Abstract: 25th Congress of the European Association for Cranio Maxillo Facial Surgery, 2021.
. Robot Technology in Dentistry, Part One of a Systematic Review: Literature Characteristics. Dental Materials, 37(8):1217–1226, 2021.
[bibtex] [url] [doi]
. Robot Technology in Dentistry, Part Two of a Systematic Review: An Overview of Initiatives. Dental Materials, 37(8):1227–1236, 2021.
[bibtex] [url] [doi]
. Robot Technology in Analyzing Tooth Removal – a Proof of Concept. In 42nd Annual International Conferences of the IEEE Engineering in Medicine & Biology Society (EMBC), pp. 4721–4727, 2020.
[bibtex] [pdf] [doi]
. Robottechnologie, is er een toekomst voor in de tandheelkunde? Quality Practice Tandheelkunde 12(5):30–35, 2017.
[bibtex] [pdf]
. Op zoek naar toepassingen van robottechnologie in de MKA-chirurgie. In Annual Scientific Meeting of the Dutch Association of Oral and Maxillofacial Surgery, 2016.
Wind Turbine Brain (WT Brain)
Project type
TKI Wind op Zee; 2017-2020
Description
Background
Wind turbines behave well in a lot of different wind conditions, in the lightest breeze and the heaviest storms. In the end, it boils down to producing energy at the lowest possible cost. That is why the system that controls the response of a wind turbine to the wind, must keep finding a balance between producing energy and the loads acting on the wind turbine. This project focusses on finding new ways to finding this balance. These new ways are becoming available due to the rapid scientific progress in the fields of (deep) machine learning and the use of, amongst other things, neural networks. By applying these methods in wind turbine control, we expect to realise a 5-10% design equivalent load reduction.
Goal/ objective
The goal of the project is to improve the wind turbine controller and its design process. This will be achieved by using machine learning (ML) to tune the controller in the design phase. The controller will contain conventional estimators and controller structures. Neural or Bayesian networks (hereafter: artificial networks (AN)) will be used to search for opportunities to further improve the conventional controller design.
Work programme
- ML methods and AN architectures: AN types are compared and the architecture of an AN-based controller is selected. Based on that selection, appropriate ML methods are selected.
- Conventional controller tuning with ML: The selected target function may be difficult to capture for the conventional controller. ML can tune the parameters based on the target function directly.
- AN-controller: Incorporate AN in the controller and tune it.
- Tuning and testing: The new controller structures are tuned for manufacturer and theoretical wind turbine models
Outcome
Situations where improvements are sought are low, normal and highly turbulent winds, starts and stops and extreme wind conditions, such as extreme operating gusts or extreme direction changes. The improvement is expected to reduce design equivalent loads on the blades and the wind turbine tower and to reduce extreme loads. These reductions can directly reduce the cost of a wind turbine. Alternatively, the load reduction on the blades can be used to employ longer blades, resulting in a higher production.
TUD project members
ir. Sebastiaan P. Mulders, Dr.-Ing. Jens Kober, prof.dr.ir. Jan-Willem van Wingerden
Project consortium
Energy Research Centre of the Netherlands (ECN), Lagerwey Group B.V., TU Delft, Siemens Gamesa
Publications
. A Practical Bayesian Optimization Approach for the Optimal Estimation of the Rotor Effective Wind Speed. In American Control Conference (ACC), pp. 4179–4185, 2019.
[bibtex] [pdf] [doi]