Image formation

CMPSCI 670: Computer Vision

Grant Van Horn

February 15, 2022
Announcements
HW1 due next Tuesday (2-20)

Project Proposals: Friday, March 15

Project Poster Presentation: Friday May 10th, 2-4pm, LGRC A112.

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 2

Overview
The pinhole projection model
• qualitative properties

Cameras with lenses

• Depth of focus
• Field of view
• Lens aberrations

Digital cameras
• Sensors
• Colors
• Artifacts

Computational photography
• Novel sensors and cameras

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 3

Cameras

Albrecht Dürer early 1500s Brunelleschi, early 1400s

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 4

Lets design a camera

Object Film
A

Idea 1: Lets put a film in front of an object

Do we get a reasonable image?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 5

Pinhole camera

Object Barrier Film

Add a barrier to block of most rays

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 6

Pinhole camera

Object Barrier Film

• Captures pencil of rays - all rays through a single point: aperture,

center of projection, focal point, camera center
• The image is formed on the image plane

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 7

Camera obscura
Basic principle known to Mozi (470-390 BCE), Aristotle (384-322 BCE)
Drawing aids for artists: described by Leonardo Da Vinci (1452-1519 AD)

Gemma Frisius, 1558

“Camera obscure” Latin for “darkened room”
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 8
Pinhole cameras are everywhere

Tree shadow during a solar eclipse

photo credit: Nils van der Burg
http://www.physicstogo.org/index.cfm

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 9
Accidental pinhole cameras

A. Torralba and W. Freeman, Accidental Pinhole and Pinspeck Cameras, CVPR 2012

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 10

Home-made pinhole camera

http://www.pauldebevec.com/Pinhole
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 11
Dimensionality reduction: 3D to 2D
3D world 2D image

Point of observation

• What is preserved?
• Straight lines, incidence
• What is not preserved?
• Angles, lengths

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A. Efros 12
Modeling projection
y
f
z

To compute the projection P’ of a scene point P, form a visual ray connection P to the camera
center O and find where it intersects the image plane
• All scene points that lie on this visual ray have the same projection on the image
• Are there points for which this projection is not defined?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 13
Modeling projection
y
f
z

The coordinate system

• The optical center (O) is at the origin
• The image plane is parallel to the xy-plane (perpendicular to the z axis)

Projection equations
• Derive using similar triangles (x, y, z) ! ( f x/z, f y/z)
<latexit sha1_base64="unIWoc8K4384RASlWb83SD265UQ=">AAACC3icbVDLTsJAFJ3iC/FVdelmAjGBhEeLJrokunGJiTwSaMh0mMKE6SMzU6U07N34K25caIxbf8Cdf+MAXSh4kpucnHNv7r3HDhgV0jC+tdTa+sbmVno7s7O7t3+gHx41hR9yTBrYZz5v20gQRj3SkFQy0g44Qa7NSMseXc/81j3hgvrenYwCYrlo4FGHYiSV1NOz+XExKk4KsMvpYCgR5/4DzJeccWVShCUnqkwKPT1nlI054CoxE5IDCeo9/avb93HoEk9ihoTomEYgrRhxSTEj00w3FCRAeIQGpKOoh1wirHj+yxSeKqUPHZ+r8iScq78nYuQKEbm26nSRHIplbyb+53VC6VxaMfWCUBIPLxY5IYPSh7NgYJ9ygiWLFEGYU3UrxEPEEZYqvowKwVx+eZU0q2XzrFy9Pc/VrpI40uAEZEEemOAC1MANqIMGwOARPINX8KY9aS/au/axaE1pycwx+APt8wfV4JkC</latexit>

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 14
3D - 2D Projections

Adapted from B. Hariharan

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 15
3D - 2D Projections

Adapted from B. Hariharan

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 16
3D - 2D Projections

Adapted from B. Hariharan

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 17
3D - 2D Projections

Adapted from B. Hariharan

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 18
3D - 2D Projections

Adapted from B. Hariharan

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 19
Projection of a line
image plane

camera vanishing point

center

line in the scene

• What if we add another line parallel to the first one?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 20
Vanishing points
Each direction in space has its own
vanishing point
• All lines going in the that direction
converge at that point
• Exception: directions that are parallel
to the image plane

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 21
Vanishing points
Each direction in space has its own
vanishing point
• All lines going in the that direction
converge at that point
• Exception: directions that are parallel
to the image plane
• What happens to the ground plane?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 22

The horizon

camera
center

ground plane

Vanishing line of the ground plane

• All points at the same height of the camera project to the horizon
• Points above the camera project above the horizon
• Provides a way of comparing heights of objects

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 23

The horizon

Is the person above or below the viewer?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 24
Perspective cues

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 25

Comparing heights
vanishing point

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 26

Measuring heights
5.4
5
camera height
4
3.7
3
2.5
2

What is the height of the camera?

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 27

Fun with Projective Geometry

Illusion Credit: RN Shepard, Mind Sights: Original Visual Illusions, Ambiguities, and other Anomalies

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 28

Perspective in art
Masaccio,
Trinity, Santa
Maria Novella,
Florence,
1425-28

One of the first

consistent uses
of perspective
in Western art

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 29

Perspective in art
(At least partial) Perspective projections in art well
before the Renaissance

From ottobwiersma.nl

Also some Greek examples,

So apparently pre-renaissance…
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 30
Orthographic projection
Special case of perspective projection
• Distance of the object from the image plane is infinite
• Also called the “parallel projection”

Image World

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 31
Orthographic projection
Special case of perspective projection
• Distance of the object from the image plane is infinite
• Also called the “parallel projection”

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 32

Pinhole camera

Object Barrier Film

image
aperture

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 33

Shrinking the aperture

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 34
Shrinking the aperture

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by Steve Seitz 35
Adding a lens
Object Lens Film

A lens focuses light on to the film

Thin lens model:
• Rays passing through the center are not deviated (pinhole projection model still holds)

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 36
Adding a lens
Object Lens Film

A lens focuses light on to the film

Thin lens model:
• Rays passing through the center are not deviated (pinhole projection model still holds)
• All parallel rays converge to one point on a plane located at the focal length f

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 37
Adding a lens
Object Lens Film

circle of
confusion

A lens focuses light on to the film

• There is a specific distance at which objects are “in focus”
• other points project on to a “circle of confusion” in the image

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 38
Thin lens formula
What is the relation between the focal length (f ), the distance of the object from the optical center
(D ) and the distance at which the object will be in focus (D’ )?

D′ D
f

image lens object

plane
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 39
Thin lens formula
Similar triangles everywhere! y′/y = D′/D

D′ D
f
y
y′

image lens object

plane
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 40
Thin lens formula
Similar triangles everywhere! y′/y = D′/D
y′/y = (D′−f )/f

D′ D
f
y
y′

image lens object

plane
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 41
Thin lens formula
1 +1 =1 Any point satisfying the thin lens
D′ D f equation is in focus

D′ D
f
y
y′

image lens object

plane
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F. Durand 42
Depth of field

http://www.cambridgeincolour.com/tutorials/depth-of-field.htm

DOF is the distance between the nearest and farthest objects in a scene that appear
acceptably sharp in an image
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros 43
Varying the aperture

Large aperture = small DOF Small aperture = large DOF

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros 44
Controlling depth of field

Changing the aperture size affects the depth of field

• A smaller aperture increases the range in which the object is approximately in focus
• But small aperture reduces the amount of light — need to increase the exposure for contrast
• Pinhole camera has an infinite depth of field

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 image credit Wikipedia 45
Field of view

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros 46
Field of view

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros 47
Field of view

Field of view (FOV) depends on the focal length and the size of the camera retina
✓ ◆
1 d
= tan
2f

Larger focal length = smaller FOV

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros 48
Field of view, focal length

Large FOV, small f — Camera close to the car

tan( ) ⇥ 2f = d
⇠ ( ) ⇥ 2f = d Small FOV, large f — Camera far from the car
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by A.Efros, F.Durand 49
Same effect for faces

wide-angle standard telephoto

(short focus) (long focus)

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F.Durand 50
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Source: Hartley & Zisserman 51
The dolly zoom
Continuously adjusting the camera focal length while the camera moves away from (or towards)
the subject

http://en.wikipedia.org/wiki/Dolly_zoom
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 52
The dolly zoom
Continuously adjusting the camera focal length while the camera moves away from (or towards)
the subject
Also called as “Vertigo shot” or the “Hitchcock shot”

Example of dolly zoom from Goodfellas

Example of dolly zoom from La Haine
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 53
Lens flaws: Chromatic aberration
Lens have different refractive indices (Snell’s law) for different wavelengths: causes color fringing

near lens center near lens outer

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by S.Seitz 54
Lens flaws: Spherical aberration
Spherical lenses don’t focus light perfectly (thin lens model)
• Rays farther from the optical axis are focussed closer

objects lack sharpness

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by S.Seitz 55
Lens flaws: Vignetting
Reduction of image brightness in the periphery

Not all rays reach

the sensor

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by S.Seitz 56
Lens flaws: Radial distortion
Caused by asymmetry of lenses
Deviations are most noticeable near the periphery

barrel distortion pincushion distortion mustache distortion

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring

http://clanegesselphotography.blogspot.com/ 24
http://parkingandyou.com 57
Real photographic lens
Many uses: cameras, telescopes, microscopes, etc

fixed focal length adjustable zoom

Example of a prime lens - Carl Zeiss Tessar Nikkor 28-200 mm zoom lens,
extended to 200 mm at left and
collapsed to 28 mm focal
length at right.
http://en.wikipedia.org/wiki/Zoom_lens

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 58

Overview
The pinhole projection model
• qualitative properties

Cameras with lenses

• Depth of focus
• Field of view
• Lens aberrations

Digital cameras
• Sensors
• Colors
• Artifacts

Novel cameras
• Computational photography

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 59

Measuring light
Photographic film — strip of transparent plastic film base coated on one side with a gelatin
emulsion containing light-sensitive materials
Creates a latent image when exposed to light for short duration
Films are then chemically developed to form a photograph
Early films/photographic plates could only capture intensity

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 60

Early color photography
Sergey Prokudin-Gorskii (1863-1944)
Photographs of the Russian empire (1909-1916)

align

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 61

Measuring light: color films
Color photographic film — many layers of dyes and light sensitive materials to capture light of
different frequencies simultaneously
• Kodak pioneered color films for making paper prints

Simultaneous measurement solves the alignment problem

• But needs complex film design and development process

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 62

Digital images
Color images are commonly represented using 3 channels [R, G, B]
• The color of each pixel is given by the (r,g,b) value

red green blue

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 63

Digital camera
A digital camera replaces the film with a sensor array
• Each cell in the array is a light-sensitive diode that converts photons to electrons
• Two common types of sensor arrays
• Charge Coupled Device (CCD)
• Complementary Metal Oxide Semiconductor (CMOS)

http://electronics.howstuffworks.com/digital-camera.htm
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by S.Seitz 64
Color sensing in the camera
Color filter array
Bayer grid
Estimate missing
components from
neighboring values
(demosiacing)

Why more green?

Human luminance sensitivity function

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by S.Seitz 65
“Demosaicing” — estimate missing values

Red Green Blue

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 66
Demosaicing

Problem: estimate the missing values

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 67

Why is this even possible?
Adjacent pixel values are strongly correlated

left right

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 68

Why is this even possible?
Adjacent pixel values are strongly correlated

left right

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 69

Why is this even possible?
Adjacent pixel values are strongly correlated

left right

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 70

Interpolation
gt
gl ? gr
gb
nearest neighbor
copy one of your
neighbors
? ←gl

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 71

Interpolation
gt gt
gl ? gr gl ? gr
gb gb
nearest neighbor linear interpolation
copy one of your average values of
neighbors your neighbors
? ←gl ? ←(gt+gl+gr+gb)/4

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 72

Interpolation
gt gt gt
gl ? gr gl ? gr gl ? gr
gb gb gb
nearest neighbor linear interpolation adaptive gradient
copy one of your average values of average based on
neighbors your neighbors nbhd. structure
? ←gl ? ←(gt+gl+gr+gb)/4 if |gt-gb| > |gl-gr|
? ← (gl+gr)/2
else
? ← (gt+gb)/2

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 73

Problem: color moiré

COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F.Durand 74
The cause of color moiré

detector

Fine black and white detail in the image scene

is misinterpreted as color information
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 Slide by F.Durand 75
Alternatives to Bayer filter

Three new Kodak RGBW filter patterns Fujifilm "X-Trans" filter

White or “panchromatic” cells allow lights across all wavelengths

• Better light efficiency

How would you go about picking the best one?

Source: https://en.wikipedia.org/wiki/Bayer_filter
COMPSCI 670 Grant Van Horn — UMass Amherst, Spring 24 76
77