EE367/CS448I: Computational Imaging and Display

Winter 2021
Lectures: Mondays and Wednesdays, 2:30-3:50 pm
Problem sessions: Fridays, 2:30-3:20 pm
Instructors: Gordon Wetzstein, Mark Nishimura (TA)


Light field photograph of the 2017 class. Top row, from left: front focus, center focus, rear focus. Click on the images for high-resolution pictures that were refocused from the light field in post-processing. Bottom row, from left: contrast-enhanced depth map computed from the light field and rectified, raw light field. Click on the images to see the original, full-resolution data.


Course Description

Computational imaging systems have a wide range of applications in consumer electronics, scientific imaging, HCI, medical imaging, microscopy, and remote sensing. We discuss light fields, time-of-flight cameras, multispectral imaging, thermal IR, computational microscopy, compressive imaging, computed tomography, computational light transport, compressive displays, phase space, and other topics at the convergence of applied mathematics, optics, and high-performance computing related to imaging. Hands-on assignments. Prerequisites: EE 261 or equivalent (basic signal processing) and EE 263 or equivalent (linear systems/algebra). Course Catalog Entry

Topics include:

Helpful Background

This course requires programming experience (especially Matlab) as well as knowledge of linear algebra, basic calculus, and optimization. The second class will review most of the required mathematical concepts (see tentative schedule below). Previous knowledge of computer graphics and computer vision will be helpful.
Courses that will be very helpful, but which are not absolutely required:
Related courses at Stanford that you may also find interesting:
A few of the course topics overlap with different parts of related courses.

Important Class Information


Requirements and Grading

The course requirements include: (1) six assignments, (2) an in-class a remote midterm (80 minutes long), and (3) a major final project, including a project proposal, final report, source code, and a poster (or video) presentation.

Your final grade will be made up from There are no "late days" for the assignments. If you choose to submit an assignment late, we will accept it for up to 24h after the submission deadline with a 30% penalty (final grade multiplied by 0.7).

The final project grade takes into account your poster presentation (organization of poster, clarify of presentation, ability to answer question), your source code submission (code organization and documentation), and your final project report (approriate format and length, abstract, introduction, related work, description of your method, quantitative and qualitative evaluation of your method, results, discussion & conclusion, bibliography).

Course Projects, Project Proposal, and Final Report

You can work in teams of up to 3 students for the project. Submit only one proposal and final report for each team. The expected amount of work is relative to the number of team members, so if two teams work on a similar project, we'd expect less work from a smaller team. Before you start to work on the proposal or the report, take a look at some of the past project proposals and reports to give you sense for what's expected.

The project proposal is a 1-2 page document that should contain the following elements: clear motivation of your idea, a discussion of related work along at least 3 scientific references (i.e., scientific papers not blog articles or websites), an overview of what exactly your project is about and what the final goals are, milestones for your team with a timeline and intermediate goals. Once you send us your proposal, we may ask you to revise it and we will assign a project mentor to your team.

The final project report should look like a short (~6 pages) conference paper. We expect the following sections, which are standard practice for conference papers: abstract, introduction, related work, theory (i.e., your approach), analysis and evaluation, results, discussion and conclusion, references. To make your life easier, we provide an MS Word and a LaTex template that you can use to get started on your report.

Equipment

Different equipment will may be available for use, depending on COVID regulations, in the projects: Intel RealSense, Lytro Illum, Time of Flight cameras, machine vision and SLR cameras, The Eye Tribe (gaze tracker), Olympus Air / Open Camera Platform, ...

Textbook

No textbook, students are expected to read relevant literature as discussed in class and outlined in the "additional readings" section of the syllabus.

Pinhole Camera Gallery

You can find some notable pinhole camera photos from previous offerings here

Class Projects of Previous Offerings


Zoom Links and Video Recordings of Lectures and Problem Sessions

The zoom link for the first lecture is here. All zoom links for future lectures, office hours, and other events will be posted on canvas. The recorded lecture videos are released soon after each lecture on canvas. The video recordings of the problem sessions will also be released on canvas.

Tentative Syllabus

Class Date Topic Details Slides Additional Readings Assignments
Week 1

1/11

Introduction and fast forward overview of class, logistics, discussion of project ideas lecture1.pdf
zoom link

1/13

The human visual system perception of color, depth, contrast, resolution lecture2.pdf - Hybrid images paper

1/15

Problem session
Week 2

1/18

Martin Luther King Day
No Class!

1/20

Digital photography I optics, aperture, depth of field, exposure, noise, sensors lecture3.pdf - archived course CS 178

1/22

Problem session HW1 due at noon
Week 3

1/25

Digital photography II image processing pipeline lecture4.pdf - Demosaicing paper
- Non-local means paper
- Intro to bilateral filter

1/27

Sampling, Linear Systems review of sampling, regularized linear systems lecture5.pdf

1/29

Problem session HW2 due at noon
Week 4

2/1

Deconvolution inverse filtering, Wiener filtering, total variation, ADMM lecture6.pdf - Lecture notes: deconvolution
- ADMM paper

2/3

Burst photography HDR, tone mapping, super-resolution, burst denoising
Guest lecture by Dr. Orly Liba
lecture7.pdf - HDR paper
- Tone mapping paper

2/5

Problem session HW3 due at noon
Week 5

2/8

Light field photography camera arrays, lytro, coded masks, refocus, fourier slice theorem lecture8.pdf - original light field paper
- other light field paper
- light field thesis

2/10

Coded computational photography extended depth of field, motion invariance, flutter shutter lecture9.pdf - flutter shutter paper

2/12

Problem session HW4 due at noon
Week 6

2/15

President's Day
No Class!

2/17

Noise signal independent noise, signal-dependent noise, image reconstruction with noise lecture10.pdf - Lecture notes: noise and deconvolution with noise
Project proposal due

2/19

Problem session HW5 due at noon
Week 7

2/22

Compressive imaging single pixel camera, compressive sensing, compressive hyperspectral imaging, compressive light field imaging lecture11.pdf - Lecture notes: single pixel camera

2/24

Time-of-flight imaging LiDAR, single-photon detectors, 3D imaging systems, looking around corners
Guest lecture by Dr. David Lindell
lecture12.pdf

2/26

HW6 due at noon
Week 8

3/1

Conventional and Emerging Image and Vision Sensors sensor electronics, event sensors, emerging sensor technology
Guest lecture by Dr. Julien Martel
lecture13.pdf

3/3

In-class Midterm!

3/5

Week 9

3/8

Imaging Black Holes Guest lecture by Prof. Katie Bouman (Caltech) lecture14.pdf

3/10

Display Blocks and
Computational displays
LCDs, SLMs, OLEDs, stereo displays, light field displays, HDR displays, vision-correcting displays, ... lecture15.pdf

3/12

Week 10

3/15

Wearable displays head-mounted displays (HMDs), virtual reality (VR), augmented reality (AR) lecture16.pdf

3/17

Final project poster presentation
must submit video by 3 pm
ProjectPosterTemplate.ppt
ProjectReportTemplate.doc
ProjectReportTemplateTex.zip
Poster printing instructions

3/19

Project reports and code due (until Fri, 3/19, 11:59pm)


Computing Resources

In the past, we have only supported MATLAB for all assignments, but we are (slowly) transitioning to Python. This is the first year that a Python option will be offered, but we will support both MATLAB and Python.

PYTHON. We'll only support Python 3.7 and recommend that you install it using Anaconda or Miniconda (see installation instructions here).

MATLAB (Image Processing Toolbox is required). If you do not have a license, which you can get at a discounted student price on Stanford's Software licensing web store, you can use FarmShare. FarmShare allows you to use MATLAB and other software remotely from your computer. For basic tutorials on MATLAB, please look here.

Poster Printing Instructions (for in-person poster sessions)

Detailed instructions for printing your poster can be found here: https://eelabs.stanford.edu/posters.html

The quick summary:

Video Recordings and Privacy

Video cameras located in the back of the lecture room will capture the instructor presentations in this course. For your convenience, you can access these recordings by logging into the course Canvas site. These recordings might be reused in other Stanford courses, viewed by other Stanford students, faculty, or staff, or used for other education and research purposes. Note that while the cameras are positioned with the intention of recording only the instructor, occasionally a part of your image or voice might be incidentally captured. If you have questions, please contact a member of the teaching team.

Acknowledgements

Some of the materials used in class build on that from other instructors. In particular, we will use some materials from Marc Levoy, Fredo Durand, Ramesh Raskar, Shree Nayar, Paul Debevec, Matthew O'Toole and others, as noted in the slides. The website was adopted from James Hays (thanks!). Feel free to use these slides for academic or research purposes, but please maintain all acknowledgments.