6.

891 Approximation Algorithms

Lecturer: David P. Williamson

February 4, 2000

Lecture 1

Scribe: John Dunagan

1 An Introduction to Approximation Algorithms

How can we solve NP-hard optimization problems eciently (assuming P 6= NP )?

Chvatal told a recent joke on this subject at SODA which we repeat here. In Communist Eastern Block countries in the 60s and 70s, it was possible to be intelligent,
honest, or a member of the communist party, but no more than two of the three at
once. Algorithms for NP-hard optimization problems are like this in the sense that
we would like them to:
(i) nd optimum solutions
(ii) in polynomial time
(iii) for any instance
However, if P 6= NP , no algorithm meets more than two out of the three requirements.
If we drop requirement (iii), then we are lead to the study of special cases. If we
drop requirement (ii), then we are lead to the eld of integer programming, and the
many techniques there, such as branch-and-bound, branch-and-cut, etc. If we drop
requirement (i), then we are lead to heuristics, which includes greedy algorithms,
local search algorithms (including the more sophisticated variants such as simulated
annealing and genetic algorithms), and approximation algorithms. This class will
focus on approximation algorithms.
Denition 1 An algorithm is an -approximation algorithm for an optimization
problem  if
1. The algorithm runs in polynomial time
2. The algorithm always produces a solution which is within a factor of  of the
value of the optimal solution
The factor  is known as the \performance guarantee" of the algorithm.
Throughout the course we will use the following convention: for minimization
problems,  > 1 (this is universal), while for maximization problems,  < 1 (this is
not universal). In the literature, 1= is sometimes used for maximization problems.
Why do we study approximation algorithms?
1-1

1. NP-hard problems exist and need solutions.

2. As ideas for algorithms that are used in practice for item #1.
3. As a mathematically rigorous way of studying heuristics.
4. Because it's fun!
5. Because it tells us how hard problems are.
Let us brie y touch on item 5 above, beginning with another denition:

Denition 2 A polynomial-time approximation scheme (PTAS) for a minimization

problem is a family of algorithms fA :  > 0g such that for each  > 0, A is a

(1 + )-approximation algorithm which runs in time polynomial in the input size for
xed . For a maximization problem, we require that A is a (1 )-approximation
algorithm.
Some problems for which PTAS's exist are knapsack, Euclidean TSP (Arora 1996,
Mitchell 1996), and many scheduling problems. Other problems like MAX SAT, MAX
CUT and Metric TSP are harder. They all belong to the class of MAX SNP-hard
problems, which are characterized by the following theorem:

Theorem 1 (Arora, Lund, Motwani, Sudan, Szegedy 1992) There does not exist a
PTAS for any MAX SNP-hard problem unless P = NP .

There is a similar, but even more discouraging result with respect to MAX CLIQUE:

Theorem 2 (Hastad 1996) There does not exist a O(n  ) approximation algorithm
for any  > 0 for MAX CLIQUE unless NP  RP .
What is MAX CLIQUE ? Given a graph G = (V; E ), nd the clique S  V of
maximum size jS j. And what is a clique?
Denition 3 A clique S is a set of vertices for which each vertex pair has its corresponding edge included (that is, i 2 S , j 2 S implies (i; j ) 2 E ).
1

Note that there is a trivial approximation algorithm for MAX CLIQUE with
performance guarantee n = jV j. Simply take a single vertex; this is trivially a clique.
The size of any clique cannot be more than n, so the algorithm has a performance
guarantee of n=1 = n. Hastad's result tells us that we do not expect to do much
better than this trivial algorithm.
This brief survey is meant to convey that some NP-hard optimization problems
are much harder than others. While some problems are approximable to within
any factor you want, other problems are not approximable beyond a trivial amount.
1-2

The previous results about MAX CLIQUE and MAX SNP come from the theory of
probabilistically checkable proofs.
The theory of approximation algorithms is a set of positive results showing that
some problems are eciently approximable. The theory of probabilistically checkable
proofs (PCP) is a set of negative results showing that many problems cannot be
approximated better than a certain factor. Together the two form a complete theory
of NP-hard optimization problems. In this class we will only mention PCP results;
our focus will be on deriving approximation techniques. This will be done by studying
a central body of techniques, mostly from the 90's, that will enable you to go forth
and design your own approximation algorithms.
We now post two central theses of this course:
 There is a central core of techniques for designing approximation algorithms.
Know these, and you know a lot.
 Linear Programming and Integer Programming are central tools for designing
and analyzing approximation algorithms.
Both of these theses will be illustrated in our discussion of Set Cover, an NP-hard
optimization problem.

2 Set Cover

Weighted Set Cover (SC)

 Input:
{ Ground elements E = fe ; e ;


; eng
{ Subsets S ; S ;


; Sm  E
{ Weights w ; w ;


; wm  0
 Goal: Find a set I  f1; 2;


; mg that minimizes Pi2I wi, such that Si2I Si =
1

E.

For the unweighted SC problem, we take wj = 1 for all j .

Why should we care about the Set Cover Problem? First, the problem shows up
in various applications. A colleague of Dr. Williamson at IBM applied the set cover
problem to try to nd relevant features of boot sector viruses for the IBM product
IBM AntiVirus.
 Elements: Known boot-sector viruses (about 150 of them).
1-3

 Sets: 3 byte sequences in viral boot sectors not found in \good" code (about

21,000 of them).
Once a relevant set cover was found, the 3 byte sequences were used as \features" for
a neural classier that would determine when a boot sector virus was present or not.
Since many boot sector viruses are written by modifying previous ones it was hoped
that this would allow detection of previously unknown boot sector viruses. A small
set cover (much less than 150 sets) was desired in order to avoid overtting the data.
Using the greedy algorithm discussed later in the lecture, a set cover of 50 sets was
found (actually, the cover was such that each element was contained in four distinct
sets of the solution). The neural classier successfully found many unknown boot
sector viruses in practice.
A second reason to care about Set Cover is that it generalizes other problems.
Consider the following problem:

Weighted Vertex Cover (VC)

 Input:
{ An undirected graph G = (V; E )
{ Weights wi  0 8i 2 V
 Goal: Find a set C that minimizes Pi2C wi, such that for every (i; j ) 2 E , we
have either i 2 C or j 2 C .

To see that VC is a special case of SC, consider the following identication:
 Elements in SC: all edges in G (the VC edges).
 Sets in SC: Si = fall edges incident to vertex ig.

2.1 An Integer Programming Formulation

Now we progress to the second thesis. Let's write Set Cover as an integer program.
Here, we create a variable xj for each subset Sj . If j 2 I , then xj = 1, otherwise
xj = 0.
Min

m
X
j =1

wj xj

subject to: X

xj  1
j :ei 2Sj
xj 2 f0; 1g

8ei 2 E
1-4

The inequality constraints just state that for each item in our ground set, we must
choose at least one set that covers that item. This can't be solved in polynomial time,
so we relax the integrality requirement. The resulting linear program (LP) can be
solved in polynomial time.
Min

m
X
j =1

wj xj

subject to: X

xj  1

8ei 2 E

j :ei 2Sj
xj  0

If you are not familiar with any of the myriad ways to solve LP's in polynoimal
time, don't worry. We will just be using polynomial time LP solvers as a black box.
We close our eyes and the LP solver returns a solution x.

Let OPT be the optimal objective value for the integer program, and let ZLP
  OPT since the
be the optimal objective value for the linear program. Then ZLP
solution space for the integer program is a subset of the solution space of the linear
program. We can rewrite this as
m
X
j =1

wj xj  OPT:

We now proceed to our rst approximation algorithm.

2.2 Method I: Rounding

We dene fi = jfj : ei 2 Sj gj and f = maxi fi. Our rst algorithmic idea is just to

round up any variable xj that is suciently large. Let
I = fj jxj  f1 g:
Lemma 3 I is a set cover.

Suppose there is an element ei such that ei 2= Sj2I Sj . Then for each set
Sj of which ei is a member, we have xj < 1=f . So
X 
xj < f1
jfj : ei 2 Sj gj
j ei 2Sj
 1;
since jfj : ei 2 Sj gj  f . But this violates the linear programming constraint for ei.

Proof:

1-5

Theorem 4 (Hochbaum '82) Rounding is an f -approximation algorithm for set cover.

Proof: It is clear that the rounding algorithm is a polytime algorithm. Furthermore,

X
j 2I

wj 

n
X
j =1

= f

wj (f
xj )

X
j

wj xj

 f
OPT:
The rst inequality follows since j 2 I only if xj f  1.

Now we consider a second rounding technique. If you are unfamiliar with duals,
the course handouts should help.

2.3 Method II: Dual Rounding

Another way to use the rounding method is to apply it to the dual solution. The dual
of the linear programming relaxation for set cover is:
X
Max
yi
i

subject to: X

yi  wj

8Sj

i:ei 2Sj
yi  0

8ei 2 E:

If we have a feasible dual solution y, then

X
  OPT
yi  ZLP
i

by weak duality. Taking advantage of the fact that nding a dual solution is just
as easy as nding a primal solution, we present the following algorithm. This time,
we take all sets whose corresponding dual inequality is tight; that is, it is met with
equality. We let our solution I 0 be
X 
I 0 = fj j
yi = wj g:
i;ei 2Sj

Lemma 5 I 0 is a set cover.

Proof: Suppose 9ei 2= Sj2I Sj . Then for each Sj containing ei
0

X 
yi < wj ;

i:ei 2Sj

1-6

so we can increase yi by some positive amount and remain feasible, which contradicts
the optimality of y. Just to make the positive amount explicit, it's given by

 = j min
fw
ei 2Sj j
:

k:ek 2Sj

ykg

Theorem 6 (Hochbaum '82) Dual rounding is an f -approximation algorithm.

Proof: Because we choose set Sj only if its constraint is tight, we have
X

j 2I

wj =

X X 
yi

j 2I i:ei 2Sj
0

X 
=
yi jfj 2 I 0 : ei 2 Sj gj
i
X 

 f yi
i
 f
OPT:

Maria Minko observed in class that complementary slackness (a property of LP's

explained in the course handouts) guarantees that whenever the Rounding algorithm
includes a set Sj (because xj  f ), the corresponding dual constraint is tight, so Dual
rounding also includes the set Sj in its solution. Thus, Dual-Rounding never obtains
a better solution than Rounding (I  I 0).
Now we are going to present an algorithm which \uses LP" in the sense that its
design and analysis are substantially motivated by our understanding of LP's, but
which never needs to call an LP solver. This leads to an approximation algorithm
with the same performance guarantee as before, but with a signicantly improved
running time.
1

2.4 Method III: Primal-Dual

The following algorithm behaves much like Dual rounding above, except that it constructs its own dual solution, rather than nding the optimal dual LP solution. The
motivation for this is that we only needed three facts in our above analysis. We
needed that y was feasible, that every set included in our solution corresponded to
a tight dual constraint, and that our solution was a set cover.
In fact, Lemma 5 gives us exactly the idea we need on how to construct a dual
solution. Let I 0 contain all the indices of the sets whose dual inequality is currently
tight. Then the lemma shows that if I 0 is not a set cover, there is a way to improve
the overall value of the dual solution, and create a new tight constraint for some set
that contains an item not currently covered.
1-7

We formalize the algorithm below. In the following algorithm, let y~ represent the
dual solution that we iteratively construct. During the entire algorithm, we maintain
that y~ is a dual feasible solution.

Primal-Dual
I ;

y~i 0 8i
while 9ek : ek 2= Sj2I Snj
o
l = arg minPj ek 2Sj wj Pi ei2Sj y~i
l wl i ei2Sl y~i
y~k y~k + l
I I [ flg.
Note that the function arg min returns the argument (index, in this case) that
minimizes the expression. It is not hard to give this algorithm in words: at each
step, we look for an element ek that is not covered. For that element, we consider
all the sets that include the element ek , and their corresponding dual constraints.
Whichever dual constraint is closest to being tight, we increase the dual variable yk
until the constraint is tight (satises the inequality as an equality) and include the
same set (corresponding to the dual constraint) in our set cover. Now the analysis is
nearly identical to that for Dual-Rounding.
:

Lemma 7 Primal-Dual returns a set cover.

Proof: This is the termination condition for the while loop.

Lemma 8 Primal-Dual constructs a dual feasible solution.
Proof: We proceed by induction on the loops of the algorithm. The base case is
trivial since initially

i:ei 2Sj

y~i = 0  wj 8j

For the inductive step, assume that upon entering an iteration of the while loop
we have
X
y~i  wj 8j
i:ei 2Sj

The only dual variable value changed by the while loop is y~k , so the inequalities for
Sj where ek 2= Sj are unaected. If ek 2 Sj , then by our choice of l
X
X
X
y~i + l =
y~i + (wl
y~i)
i:ei 2Sj

i:ei 2Sj


y~i + (wj
i e 2S
 wj :
: i

1-8

i:ei 2Sl

i:ei 2Sj

y~i)

Lemma 9 If j 2 I then Pi e 2S y~i = wj .

Proof: In the step where set j was added to I , we increased y~k by exactly enough
to make constraint j tight.

: i

The rest of the proof is essentially the same as Theorem 6 in the case of dual
rounding.

Theorem 10 (Bar-Yehuda, Even '81 ) Primal-Dual is an f -approximation algo1

rithm for the set cover problem.

Proof:

X
j 2I

wj =

X X

y~i
j 2I i:ei 2Sj
n
X
y~ijfj : ei 2 Sj gj
i=1
X
f
y~i
i


 f
OPT:

The rst equality follows from Lemma 9. The next inequality follows since each y~i
can appear in the double sum at most jfj : ei 2 Sj gj times. The next inequality
follows by the denition of f , and the last inequality follows from weak duality. 
It is worth noting that the observation by Maria Minko no longer holds; since we
are not constructing an optimal dual solution, we cannot apply the complementary
slackness analysis.

2.5 Method IV: Greedy Algorithm

So far every technique we have tried has led to the same result: an f -approximation
algorithm for set cover. In the case of set cover, using a natural greedy heuristic
yields an improved approximation algorithm.
The intuition here is straightforward. At each step choose the set that gives
the most \bang for the buck." That is, select the set that minimizes the cost per
additional element covered.
Before examining the algorithm, we dene two more quantities:
Hn  1 + 12 + 13 + : : : + n1  ln n
g  max
jSj j:
j
1

although the date is before '82, this algorithm followed Hochbaum's work

1-9

Greedy
I ;
S~j SSj 8j
while j2I Sj 6= E w
l arg minj Sj 6 ; jSjjj
I I [ fl g
y~i jSlwjHl g 8ei 2 S~l (y)
S~j S~j Sl 8j
:~ =

Note: The y step has been added only to aid the proof and is not actually part of
the algorithm.
The last step is simply meant to explain how we update the universe to account
for our previous action. After we include a new set, we subtract out of the universe
all the elements covered by that set. We now make the following shocking claim.

Claim 11 Greedy is an Hg approximation algorithm.

In contrast to the primal-dual algorithm, we are no longer using the dual variables
to guide our solution. Instead, we are simply using them for purposes of analysis.
The step where we assign y~i denes the dual solution we construct.
First we note that y~i is set only once for each ei. It is set when that element is
included and not touched again.
We now assume for the moment that y~ is a dual feasible solution. Given this, we
can prove the claim.

Theorem 12 Greedy is an Hg approximation algorithm.

Proof: The cost of the solution is
X
j 2I

wj = Hg

n
X
i=1

y~i  Hg
OPT:

The rst equality follows from the observation that when j is added to I ,

wj = Hg

i:ei 2S

~j

y~i

where S~j is the set Sj with all the elements that have been covered in previous steps
removed. This is simply by the denition of the dual variables as they are set in step
(y). The second inequality is just weak duality.

We now need to prove that the solution is dual feasible.

Lemma 13 The solution y~ is dual feasible.

1-10

Proof:

Pick an arbitrary Sj . We will show that

X

i:ei 2Sj

y~i  wj

Let ak be the number of uncovered elements in Sj at beginning of kth iteration of

the algorithm, and let l  total number of iterations. Then a = jSj j and al = 0.
Finally, let Ak be the set of previously uncovered elements of Sj covered in iteration
k. We immediately nd that jAk j = ak ak .
Suppose that set Sr is chosen in the kth iteration. Then, we have that for every
element ei (and its corresponding dual variable y~i),
y~i = HwjSr j  Hwaj
g r
g k
The equality is by denition on y~i, and the inequality follows since Sr minimizes the
ratio wSrr .
We now nish proving the lemma as follows.
1

+1

+1

X
i:ei 2Si

y~i =


=




l X
X

y~i
k=1 i:ei 2Ak
l
X
(ak ak+1)
Hwaj
g k
k=1
wj Xl ak ak+1
Hg k=1 ak
!
wj Xl 1 + 1 +


+ 1
Hg k=1 ak ak 1
ak+1 + 1
a1 1
wj X
Hg i=1 i  wj :

In the rst line, we simply note that the sum of all the y~i variables is the sum over
all iterations of the yi variables set in a particular iteration. In the second line, we
use our identity for the number of elements in Ak , and we also plug in our bound on
y~i. The rest is just algebraic manipulation.

The following complexity results give a sense of how good our approximation
algorithms are in relation to what is possible.

Theorem 14 (Lund, Yannkakis '92, Feige '96, Raz, Safra '97, Arora, Sudan '97)
Let n = jT j = the size of the ground set. Then:
 If there exists a c ln n-approximation algorithm where c < 1 then NP  DTIME (nO
1-11

(logk

n) )

 There exists some 0 < c < 1 such that if there exists a c log n-approximation
algorithm for set cover, then P = NP .

So in one direction, our approximation seems to be best possible. However, f

and Hg are not strictly comparable measures. f measures the maximum number of
times an element appears in dierent sets, while g measures the size of the largest
set. Although f can be much greater than log n, and thus Hg is a much better
approximation ratio in general, we do not have a matching PCP bound for the case
of constant f .
For example, the Vertex Cover problem corresponds to the case of f = 2. In this
case, there is no known algorithm that achieves better than a 2-approximation, but
the strongest inapproximability result is

Theorem 15 (Hastad '97) If there exists an -approximation algorithm for vertex

cover with  < 76 then P = NP .

So even for this extremely simple case we do not know everything there is to know.

1-12