2

Singular Value Decomposition

The singular value decomposition (SVD) allows us to transform a matrix A Cmn to

diagonal form using unitary matrices, i.e.,
V
.
A=U

(4)

Cmn has orthonormal columns,

Cnn is diagonal, and V Cnn is
Here U
unitary. This is the practical version of the SVD also known as the reduced SVD. We
will discuss the full SVD later. It is of the form
A = U V
with unitary matrices U and V and Cmn .
Before we worry about how to find the matrix factors of A we give a geometric
interpretation. First note that since V is unitary (i.e., V = V 1 ) we have the equivalence
V
AV = U
.

A=U
Considering each column of V separately the latter is the same as
Av j = j uj ,

j = 1, . . . , n.

(5)

Thus, the unit vectors of an orthogonal coordinate system {v 1 , . . . , v n } are mapped

under A onto a new scaled orthogonal coordinate system {1 u1 , . . . , n un }. In other
words, the unit sphere with respect to the matrix 2-norm (which is a perfectly round
sphere in the v-system) is transformed to an ellipsoid with semi-axes j uj (see Figure 2). We will see below that, depending on the rank of A, some of the j may be zero.
Therefore, yet another geometrical interpretation of the SVD is: Any m n matrix A
maps the 2-norm unit sphere in Rn to an ellipsoid in Rr (r min(m, n)).
1

2v2

0.5

1u1

0.5

0.5

v2

0.5

v1

Figure 2: Geometrical interpretation of singular value decomposition.

They
In (5) we refer to the j as singular values of A (the diagonal entries of ).
are usually ordered such that 1 2 . . . n . The orthonormal vectors uj (the
) are called the left singular vectors of A, and the orthonormal vectors v j
columns of U
(the columns of V ) are called the right singular vectors of A).
Remark For most practical purposes it suffices to compute the reduced SVD (4). We
will give examples of its use, and explain how to compute it later.
23

QR Compression ratio 0.2000, 25 columns used

Compression ratio 0.2000, relative error 0.0320, 25 singular values used

20

20

40

40

60

60

80

80

100

100

120

120

140

140

160

160

180

180

200

200
50

100

150

200

250

300

50

100

150

200

250

300

Figure 3: Image compressed using QR factorization (left) and SVD (right).

Besides applications to inconsistent and underdetermined linear systems and least
squares problems, the SVD has important applications in image and data compression
(see our discussion of low-rank approximation below). Figure 3 shows the difference
between using the SVD and the QR factorization (to be introduced later) for compression of the same image. In both cases the same amount (20%) of information was
retained. Clearly, the SVD does a much better job in picking out what information
is important. We will also see below that a number of theoretical facts about the
matrix A can be obtained via the SVD.
2.0.4

Full SVD

to an orthonormal basis of Cmm by adding appropriate

The idea is to extend U
orthogonal (but otherwise arbitrary) columns and call this new matrix U . This will
to be extended to an m n matrix . Since we do not want to alter the
also force
product of the factors, the additional rows (or columns depending on whether m > n
or m < n) of will be all zero. Thus, in the case of m n we have
A = U V
h
i 

U
e
=
V .
U
O

(6)

Since U is now also a unitary matrix we have

U AV = ,
i.e., unitary transformations (reflections or rotations) are applied from the left and
right to A in order to obtain a diagonal matrix .
Remark Note that the diagonal matrix is in many cases rectangular and will
contain extra rows/columns of all zeros.
It is clear that the SVD will simplify the solution of many problems since the
transformed system matrix is diagonal, and thus trivial to work with.

24

2.0.5

Existence and Uniqueness Theorem

Theorem 2.1 Let A be a complex m n matrix. A has a singular value decomposition

of the form
A = U V ,
where is a uniquely determined m n (real) diagonal matrix, U is an m m unitary
matrix, and V is an n n unitary matrix.
Proof We prove only existence. The uniqueness part of the proof follows directly from
the geometric interpretation. A (more rigorous?) algebraic argument can be found,
e.g., in [Trefethen/Bau].
We use induction on the dimensions of A. All of the following arguments assume
m n (the case m < n can be obtained by transposing the arguments).
= kAk2
For n = 1 (and any m) the matrix A is a column vector. We take V = 1,
A

and U = kAk2 . Then, clearly, we have found a reduced SVD, i.e., A = U V . The full
to U by the Gram-Schmidt algorithm and adding the
SVD is obtained by extending U

necessary zeros to .
We now assume an SVD exists for the case (m 1, n 1) and show it also exists
for (m, n). To this end we pick v 1 Cn such that kv 1 k2 = 1 and
kAk2 = sup kAv 1 k2 > 0.
v 1 Cn
kv 1 k2 =1

Now we take
u1 =

Av 1
.
kAv 1 k2

(7)

Next, we use the Gram-Schmidt algorithm to arbitrarily extend u1 and v 1 to unitary

e1 and Ve1 , i.e.,
matrices by adding columns U
h
i
h
i
e1
U1 = u1 U
V1 = v 1 Ve1 .
This results in
U1 AV1 =

u1
e
U
1

"
=


A

v 1 Ve1

u1 Av 1 u1 AVe1
e Av 1 U
e AVe1
U
1
1

i
#
.

We now look at three of these four blocks:

Using (7) and the specific choice of v 1 we have
(Av 1 )
Av 1
kAv 1 k2
kAv 1 k22
=
= kAv 1 k2
kAv 1 k2
= kAk2 .

u1 Av 1 =

For this quantity we introduce the abbreviation 1 = kAk2 .

25

 Again, using (7) we get

e Av 1 = U
e u1 kAv 1 k2 .
U
1
1
e u1 = 0.
This, however, is a zero vector since U1 has orthonormal columns, i.e., U
1


We show that u1 AVe1 = 0 0 by contradiction. If it were nonzero then
we could look at the first row of the block matrix U1 AV1 and see that
h
i
U1 AV1 (1, :) = 1 u1 AVe1
h
i
with k 1 u1 AVe1 k2 > 1 . On the other hand, we know that unitary matrices
leave the 2-norm invariant, i.e.,
kU1 AV1 k2 = kAk2 = 1 .
Since the norm of the first row of the block matrix U1 AV1 cannot exceed that of
the entire matrix we have reached a contradiction.
We now abbreviate the fourth block with
matrix as

1
U1 AV1 =
0
To complete the proof we apply the
e = U2 2 V . Then
A
2

1

U1 AV1 =
0

1
=
0
or


A = U1

1 0T
0 U2

e = U
e AVe1 and can write the block
A
1

0T
e .
A

e i.e., we use the SVD

induction hypothesis to A,

0T
U2 2 V2



0T
1 0T
1 0T
U2
0 2
0 V2
1 0T
0 2





1 0T
0 V2

V1 ,

another SVD (since the product of unitary matrices is unitary).

2.1

SVD as a Change of Basis

We now discuss the use of the SVD to diagonalize systems of linear equations. Consider
the linear system
Ax = b
with A Cmn . Using the SVD we can write
Ax = U V x

b = U b0 .

Thus, we can express b range(A) in terms of range(U ):

U b0 = b

26

b0 = U b

where we have used the columns of U as an orthonormal basis for range(A).

Similarly, any x Cn (the domain of A) can be written in terms of range(V ):
x0 = V x.
Now
Ax = b U Ax = U b
U U V x = U b
IV x = U b
x0 = b0 ,
and we have diagonalized the linear system.
In summary, expressing the range space of A in terms of the columns of U and the
domain space of A in terms of the columns of V converts Ax = b to a diagonal system.
2.1.1

Connection to Eigenvalues

If A Cmm is square with a linearly independent set of eigenvectors (i.e., nondefective), then
AX = X X 1 AX = ,
where X contains the eigenvectors of A as its columns and = diag(1 , . . . , m ) is a
diagonal matrix of the eigenvalues of A.
If we compare this eigen-decomposition of A to the SVD we see that the SVD is a
generalization: A need not be square, and the SVD always exists (whereas even a square
matrix need not have an eigen-decomposition). The price we pay is that we require
two unitary matrices U and V instead of only X (which is in general not unitary).
2.1.2

Theoretical Information via SVD

A number of theoretical facts about the matrix A can be obtained via the SVD. They
are summarized in
Theorem 2.2 Assume A Cmn , p = min(m, n), and r p denotes the number of
positive singular values of A. Then
1. rank(A) = r
2. range(A) = range(U (:, 1 : r))
null(A) = range(V (:, r + 1, : n))
3. kAk2 = p
1
kAkF = 12 + 22 + . . . + r2
4. The eigenvalues of A A are the i2 and the v i are the corresponding (orthonormalized) eigenvectors. The eigenvalues of AA are the i2 and possibly m n
zeros. The corresponding orthonormalized eigenvectors are given by the ui .

27

5. If A = A (Hermitian or real symmetric), then the eigen-decomposition A =

XX and the SVD A = U V are almost identical. We have U = X, i = |i |,
and v i = sign(i )ui .
Q
6. If A Cmm then |det(A)| = m
i=1 i .
Proof We discuss items 13 and 6.
1. Since U and V are unitary matrices of full rank and rank() = r the statement
follows from the SVD A = U V .
2. Both statements follow from the fact that the range of is spanned by e1 , . . . , er
and that U and V are full-rank unitary matrices whose ranges are Cm and Cn ,
respectively.
3. The invariance of the 2-norm and Frobenius norm under unitary transformations
imply kAk2 = kk2 and kAkF = kkF . Since is diagonal we clearly have
kk2 = max xCn kxk2 = max1ir i = 1 . The formula for kkF follows
kxk2 =1

directly from the definition of the Frobenius norm.

6. We know that the determinant of a unitary matrix is either plus or minus 1,
and that of a diagonal matrix is the product of the diagonal entries. Finally, the
determinant of a product of matrices is given by the product of their determinant.
Thus, the SVD yields the stated result.

2.1.3

Low-rank Approximation

Theorem 2.3 The m n matrix A can be decomposed into a sum of r rank-one matrices:
r
X
A=
j uj v j .
(8)
j=1

Moreover, the best 2-norm approximation of rank (0 r) to A is given by

A =

j uj v j .

j=1

In fact,
kA A k2 = +1 .

(9)

Proof The representation (8) of the SVD follows immediately from the full SVD (6)
by splitting into a sum of diagonal matrices j = diag(0, . . . , 0, j , 0, . . . , 0).
Formula (9) for the approximation error follows from the fact that U AV = and
the expansion for A so that U (A A )V = diag(0, . . . , 0, +1 , . . .) and kA A k2 =
+1 by the invariance of the 2-norm under unitary transformations and item 3 of the
previous theorem.
The claim regarding the best approximation property is a little more involved, and
omitted.
28

Remark There are many possible rank- decompositions of A (e.g., by taking partial
sums of the LU or QR factorization). Theorem 2.3, however, says that the -th partial
sum of the SVD captures as much of the energy of A (measured in the 2-norm) as possible. This fact gives rise to many applications in image processing, data compression,
data mining, and other fields. See, e.g., the Matlab scripts svd compression.m and
qr compression.m.
A geometric interpretation of Theorem 2.3 is given by the best approximation of a
hyperellipsoid by lower-dimensional ellipsoids. For example, the best approximation of
a given hyperellipsoid by a line segment is given by the line segment corresponding to
the hyperellipsoids longest axis. Similarly, the best approximation by an ellipse is given
by that ellipse whose axes are the longest and second-longest axis of the hyperellipsoid.
2.1.4

Computing the SVD by Hand

We now list a simplistic algorithm for computing the SVD of a matrix A. It can be
used fairly easily for manual computation of small examples. For a given m n matrix
A the procedure is as follows:
1. Form A A.
2. Find the eigenvalues and orthonormalized eigenvectors of A A, i.e.,
A A = V V .
3. Sort the eigenvalues according to their magnitude, and let j =

j , j = 1, . . . , n.

4. Find the first r columns of U via

uj = j1 Av j ,

j = 1, . . . , r.

Pick the remaining columns such that U is unitary.

Example Find the SVD for

1 2
A = 2 2 .
2 1
1.

A A=

9 8
8 9


.

2. The eigenvalues (in order of decreasing magnitude) are 1 = 17 and 2 = 1, and

the corresponding eigenvectors
 


1
1
e1 =
e2 =
v
, v
,
1
1
so that (after normalization)
"
V =

1
2
1
2

29

1
2
12

#
.

3. 1 =

17 and 2 = 1, so that

=

17 0
0
1 .
0
0

4. The first two columns of U can be computed as

1
Av 1
17

1 2  
1 1
1

2 2
1
17 2 2 1

3
1

4 ,
34 3

u1 =
=

and
1
Av 2
1

u2 =


1 2 
1
1

2 2
1
2 2 1

1
1
0 .
2
1

Thus far we have

U =

3
34
4
34
3
34

u3 (1)
u3 (2)
.
u3 (3)

0
1
2

In order to determine u3 (i), i = 1, 2, 3, we need to satisfy

uj u3 = j3 ,

j = 1, 2, 3.

The following choice satisfies this requirement

2
1
u3 = 3 ,
17
2
so that

A = U V =

3
34
4
34
3
34

0
1
2

2
17
3

17
2
17

30

17 0 "
0
1
0
0

1
2
1
2

1
2
12

#
.

The reduced SVD is given by

3

V
=
A=U

34
4
34
3
34

1
2

17 0
0
1

"

1
2
1
2

1
2
12

#
.

Remark A practical computer implementation of the SVD will require an algorithm

for finding eigenvalues. We will study this later. The two most popular SVD implementations use either a method called Golub-Kahan-Bidiagonalization (GKB) (from
1965), or some Divide-and-Conquer strategy (which have been studied since around
1980).

31