Course Description

How can robots perceive the world and their own motion so that they can accomplish navigation and manipulation tasks? In this course, we will study how this age-old question has been approached specifically if the robot is equipped with visual sensing capabilities. We focus on studying how a robot can make decisions based on raw, high-dimensional sensory data that contains only partial, noisy observations of the hidden state of the environment. Therefore, the course is divided into two main themes (i) Estimation and (ii) Decision-Making and Control where in each theme we will study traditional approaches, learning-based methods and combinations of those.

For ​estimation​, we will study how images and videos acquired by cameras mounted on robots are transformed into representations like features and optical flow. Such 2D representations allow us then to extract 3D information about where the camera is, in which direction the robot moves, or how objects in front of the camera are moved. We will also study linear and nonlinear filtering methods that allow the robot to keep track of itself, other objects and the environment state over time. And we will study how these problems can be formulated as learning problems to mitigate some of the biases introduced by traditional methods.

For ​decision-making and control,​ we will study the basic formulations of optimal and adaptive control and relate it to learning-based approaches such as model-based and model-free reinforcement learning. We will apply what we learned to vision-based control methods, i.e. image-based and position-based visual servoing. We will also study how learning from demonstration or imitation learning can mitigate some of the sampling problems in RL.

We bridge these two themes of estimation and decision-making by also studying optimal representations of raw sensory data and how they can be learned such that they are optimal for a task. We will also discuss how perception and action influence each other.

Both for estimation and decision-making, we draw connections to what we know about human and animal intelligence from cognitive science. We will observe similarities and differences between artificial and biological systems and revisit the most relevant experiments and explanations for human perception and decision making.

Teaching Assistants

Mengyuan Yan
Michelle Lee

Class Time and Location

Fall quarter (September - December, 2019).
Lecture: Monday, Wednesday 3:00pm-4:20pm
Gates Building, B12

Office Hours

You can find a full list of times and locations on the calendar.

Grading Policy

Assignment #1: 15%
Assignment #2: 15%
Assignment #3: 15%
Midterm: 20%
Course Project: 35%
See the Grading Policy for more details.

Course Discussions

Stanford students: Piazza

Assignment Details

See the Assignment Page for more details on how to hand in your assignments.
Gradescope Code: M7KJ88.

Course Project Details

See the Project Page for more details on the course project.


Course Materials


Can I take this course on credit/no cred basis?
Yes. Credit will be given to those who would have otherwise earned a C- or above.
Can I audit or sit in?
In general we are very open to sitting-in guests if you are a member of the Stanford community (registered student, staff, and/or faculty). Out of courtesy, we would appreciate that you first email us or talk to the instructor after the first class you attend. If the class is too full and we're running out of space, we would ask that you please allow registered students to attend.
Can I work in groups for the Final Project?
Yes, in groups of up to two people.
I have a question about the class. What is the best way to reach the course staff?
Almost all questions should be asked on Piazza. If you have a sensitive issue you can email the instructors directly.
Can I combine the Final Project with another course?
Yes, you may; however before doing so you must receive permission from the instructors of both courses.