EE 267: Virtual Reality

Spring 2017
Lecture: Mon, Wed 3:00pm-4:20pm, Lab: 3 slots on Friday
Location: 380-380c (Mo/Wed)
Instructors: Gordon Wetzstein, Robert Konrad (TA), Hayato Ikoma (TA), Keenan Molner (TA)

Parts and design of the light field stereoscope, an experimental head mounted display composed of parts that we will work with in this class: liquid crystal displays, IMUs, magnifying optics, etc.

More info about the light field stereoscope.
Lab 4: students are enjoying the VR head mounted displays they have just built.


Course Description

OpenGL and WebGL, real-time rendering, 3D display systems, display optics & electronics, IMUs and sensors, tracking, haptics, rendering pipeline, multimodal human perception and depth perception, stereo rendering, presence. Emphasis on VR technology. Hands-on programming assignments. The 3-unit version requires a final programming assignment in which you create your own virtual environment. The 4-unit version requires a final course project and written report in lieu of the final assignment. Prerequisites: strong programming skills and EE 103 or equivalent. Helpful: basic computer graphics / WebGL / OpenGL. Course Catalog Entry

Topics include:

Lectures, Labs, Office Hours, and Contact



Helpful Background

This course requires programming experience (especially JavaScript, C or C++) as well as basic knowledge of linear algebra. We will review most of the required mathematical concepts, but we assume that you have strong programming skills. Homework require JavaScript and GLSL programming. Even though we don't assume prior experience in JavaScript and GLSL, you should be able to understand object-oriented programs and modify them. Previous knowledge of computer graphics or computer vision would be helpful but is not absolutely required.

Courses that may be very helpful, but are not prerequisites:
Related courses at Stanford that you may also find interesting:
A few of the course topics overlap with different parts of related courses.

Requirements and Grading

The course requirements include: (1) 7 assignments and (2) either a major final project with report (4 unit version) or a long 8th homework (3 unit version).

Your final grade will be made up from There are no "late days" for the assignments.

Enrollment

This is an experimental class that is taught for the second time. Enrollment is limited to 80 students, working in teams of 2. All hardware will be provided in the lab but must be returned at the end of the course. Unfortunately, our hardware resources and the lab space are currently limited. If you did not get a chance to enroll, come to the first few classes anyway and sign up for the waiting list.

Class Format

Each week, there are 2 lectures and 1 lab. The lab is a hands-on lecture where we go over the hardware and software environments and start working on the weekly assignment or aspects relevant for that. Students will complete the assignment after the lab.

Course Goals & Equipment

This is a technical class. Students will learn about all hardware (optics, electronics, display, microcontroller, ...) and software (JavaScript, WebGL, GLSL) aspects of Virtual Reality (VR). The goal for this class is to learn all of these aspects in a hands-on manner. Each assignment is a small piece of a bigger project. The goal for each student or small team of students is to build a fully functional head mounted display, including optics, display, IMU, rendering, lens distortion shader, model loader etc., from off-the-shelf parts. The HMD we will build is inspired by the Oculus DK1. We will NOT learn game development, but study the fundamental building blocks of VR systems and implement all of them.

Textbook and Course Notes

Students are required to read additional resources as indicated for each lecture or week. These readings will help get a better intuition and deeper insights into the topics of this course. The readings may also be required to complete tasks for assignments.

Detailed course notes are available for the topics covered in weeks 5 and 6: orientation tracking with inertial measurement units (IMUs) and pose tracking with the VRduino. You can download these notes here: Other topics of in this class are adequately covered by existing textbooks, even though these are optional for the course. The material covered in the lectures should be self-contained, but it's always a good idea to read up on more details. In particular, parts of the following two textbooks will be useful for EE 267: Both of these textbooks are freely available online or through the Stanford library, just click on the titles above. We will only be reading a few chapters of these books, so there's no need to purchase them.

The hands-on parts of this course, i.e. labs and assignments, are based on JavaScript, WebGL, and GLSL. It's really important for students to go through tutorials on their own, such as these

Final Project

The final project is required for the 4 unit version of this course and can be either a hardware or a software project. Hardware projects include, but are not limited to, implementing Kalman filtering of the IMU on the microcontroller, designing different optics for the HMD, or changing the hardware design in some other way. Possible software projects include the development of a complex virtual environment, a game, or some other VR application. We will provide a few professional HMDs in addition to the kits we build in class, so the final project could be implemented on those .

Previous Course Projects



Tentative Syllabus

Class Date Topic Details Slides Comments Labs and Assignments Reading
Week 1
  4/3
Lecture: Introduction to VR and AR overview of class, logistics, history of VR/AR lecture1.pdf
  4/5 Lecture: The Graphics Pipeline and OpenGL I: Overview and Transformations rotation, translation, scaling, modelview matrix, projection matrix lecture2.pdf Marschner, Chap. 6 & 7
Lab: Hello, WebGL! HW1.pdf
HW1.zip
Lab1.pdf
Week 2
  4/10
Lecture: The Graphics Pipeline and OpenGL II: Lighting and Shading lecture3.pdf Marschner, Chap. 10 & 11
  4/12 Lecture: The Graphics Pipeline and OpenGL III: OpenGL Shading Language (GLSL) GLSL vertex and fragment shaders lecture4.pdf
Lab: Lighting and Shading with GLSL HW1 due (Thu 11:59pm) HW2.pdf
HW2.zip
Lab2.pdf
Week 3
  4/17
Lecture: The Human Visual System perception of depth, color, contrast, resolution lecture5.pdf LaValle, Chap. 5 & 6
  4/19 Lecture: The Graphics Pipeline and OpenGL IV: Stereo Rendering lecture6.pdf
Lab: stereo rendering, anaglyph HW2 due (Thu 11:59pm) HW3.pdf
HW3.zip
Lab3.pdf
Week 4
  4/24
Lecture: Head Mounted Display Optics I magnifier designs, stereo rendering for HMDs, lens distortion correction lecture7.pdf
  4/26 Lecture: Head Mounted Display Optics II advanced HMD optics lecture8.pdf HW4 out
Lab: Build Your Own HMD HW3 due (Thu 11:59pm) HW4.pdf
HW4.zip
Lab4.pdf
Week 5
  5/1
Lecture: Inertial Measurement Units I gyros, accelerometers, magnetometers lecture9.pdf Course Notes on IMUs

LaValle, Chap. 9.1 & 9.2
  5/3 Lecture: Inertial Measurement Units II sensor fusion, complementary filter, arduino lecture10.pdf
Lab: Build Your Own IMU, Arduino Programming HW4 due (Thu 11:59pm) HW5.pdf
HW5.zip
Lab5.pdf
Week 6
  5/8
Lecture: Positional Tracking I tracking with the light house lecture11.pdf Course Notes on Tracking
  5/10 Lecture: Positional Tracking II advanced positional tracking lecture12.pdf HW6.pdf
HW6.zip
Lab6.pdf
Lab: Positional Tracking HW5 due (Thu 11:59pm)
Week 7
  5/15
Lecture: Spatial Sound lecture13.pdf
project ideas
LaValle, Chap. 11
  5/17 Guest Lecture by Prof. Allison Okamura (Stanford) on Haptics lecture14.pdf
Lab: Spatial Sound HW6 due (Sat 11:59pm) HW7.pdf
HW7.zip
Lab7.pdf
Week 8
  5/22
Lecture: Panoramic Imaging and Cinematic VR lecture15.pdf
  5/24 Lecture: VR Engines and Other Aspects of VR
(latency, eye tracking, post-rendering warp,...)
lecture16.pdf
Optional Lab: content creation with unity HW7 due (Sat 11:59pm) UnityLab.pdf
EE267Unity.zip
Week 9
  5/29
Memorial Day - NO CLASS
  5/31 Guest Lecture by Prof. Steve Mann (University of Toronto, Meta)
Optional Lab: work with TAs on project
Week 10
  6/5
Guest Lecture by Prof. Jeremy Bailenson (Stanford)
  6/9 Final Project Poster and Demo Session
see program
HW8 due (Thu 11:59pm, only for 3 units). Presented during lab sessions.
Reports (only for 4 units) and code (everyone) due on 6/12/2017, 11:59pm


Prototype DIY Head Mounted Display

The primary goal of this class is to understand the fundamental building blocks of VR by building a head-mounted display from scratch. We designed this HMD to be educational, accessible, reproducible, and low-cost. Here are all the components we will use in this class and in the labs:

Hardware Component Item used in Class Link Comments Possible Alternatives Additional Info
HMD Housing ViewMaster VR Starter Pack Walmart Nice and sturdy housing. Difficult to use with eyeglasses - not enough space. We needed to drill extra holes in the housing to route the cables into it. Google Cardboard, Durovis Dive, ...
Display Topfoison 6'' 1080p LCD Topfoison Comes with HDMI driver board, get without audio jack due to poor mechanical design. This is the most difficult component to get at low cost with reasonably high resolution and driver board. spec sheet
IMU InvenSense MPU-9250 Amazon Low cost, small-scale, high-quality 9DOF IMU with breakout board. 3DOF gyro, 3DOF accelerometer, 3DOF magnetometer Lower-cost models with fewer DOF are available on Amazon and elsewhere on the web. datasheet
register map
Microcontroller Arduino Teensy 3.2 PJRC Ardunio rocks! Small, low-cost, easy to use, great support, tons of libraries... We use the Arduino Teensy 3.2 for its small form factor, affordability, and abundence of discrete interrupt vectors, used for position tracking. Other microncontrollers or FPGAs with more than 8 configurable pin change interrupt vectors.
Cables 2x USB cables (for Arduino and LCD driver board power) and HDMI Amazon Using the ultra-thin versions is critical.

VRduino rev 0.3

The EE267 uses custom hardware for position tracking. This shield for the Teensy provides the proper support circuitry for the photodiodes used with the HTC Vive Lighthouses. For the EE267 class, we provide this hardware to students in the kits. If you would like to build your own VRduino tracker board, the manufacturing files are availble here: vrduino_rev03_06142017.zip. We will updated the manufacturing files as the hardware continues to develop.

Homework Demos

Homework 1 - Getting Started with WebGL and Transformations

Homework 2 - Lighting, Shading and GLSL

Homework 3 - Foveated Rendering, Depth of Field, Stereo Rendering, Anaglyph 3D

Homework 4 - Build your HMD, Implement Stereo Rendering and Lens Distortion Correction

Homework 5 - Orientation tracking with inertial measurement units and sensor fusion



Acknowledgements

This course is generously supported by AMD and Oculus VR. Thanks for making this happen!