How can robots perceive the world and their own motion so that they can accomplish navigation and manipulation tasks? In this course, we will study how this age-old question has been approached specifically if the robot is equipped with visual sensing capabilities. We focus on studying how a robot can make decisions based on raw, high-dimensional sensory data that contains only partial, noisy observations of the hidden state of the environment. Therefore, the course is divided into two main themes (i) Estimation and (ii) Decision-Making and Control where in each theme we will study traditional approaches, learning-based methods and combinations of those.
For estimation, we will study how images and videos acquired by cameras mounted on robots are transformed into representations like features and optical flow. Such 2D representations allow us then to extract 3D information about where the camera is, in which direction the robot moves, or how objects in front of the camera are moved. We will also study linear and nonlinear filtering methods that allow the robot to keep track of itself, other objects and the environment state over time. And we will study how these problems can be formulated as learning problems to mitigate some of the biases introduced by traditional methods.
For decision-making and control, we will study the basic formulations of optimal and adaptive control and relate it to learning-based approaches such as model-based and model-free reinforcement learning. We will apply what we learned to vision-based control methods, i.e. image-based and position-based visual servoing. We will also study how learning from demonstration or imitation learning can mitigate some of the sampling problems in RL.
We bridge these two themes of estimation and decision-making by also studying optimal representations of raw sensory data and how they can be learned such that they are optimal for a task. We will also discuss how perception and action influence each other.
Both for estimation and decision-making, we draw connections to what we know about human and animal intelligence from cognitive science. We will observe similarities and differences between artificial and biological systems and revisit the most relevant experiments and explanations for human perception and decision making.