Scribd
Upload
173 views

PDC - Lecture - No. 2

This document discusses key concepts in parallel and distributed computing including: - Amdahl's law, which establishes an upper bound on the maximum speedup that can be achieved from parallelizing a program. It depends on the fraction of the program that must be executed sequentially. - Types of parallelism like data parallelism, functional parallelism, and pipelining. Data parallelism applies the same operation to different data elements, while functional parallelism applies different operations to different data elements. - Differences between multi-processor systems with shared memory versus multi-computer systems with distributed memory, where processors only have direct access to local memory. - Cache coherence and snooping protocols needed to maintain consistency between caches

Uploaded by

nauman tariq
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
173 views

PDC - Lecture - No. 2

This document discusses key concepts in parallel and distributed computing including: - Amdahl's law, which establishes an upper bound on the maximum speedup that can be achieved from parallelizing a program. It depends on the fraction of the program that must be executed sequentially. - Types of parallelism like data parallelism, functional parallelism, and pipelining. Data parallelism applies the same operation to different data elements, while functional parallelism applies different operations to different data elements. - Differences between multi-processor systems with shared memory versus multi-computer systems with distributed memory, where processors only have direct access to local memory. - Cache coherence and snooping protocols needed to maintain consistency between caches

Uploaded by

nauman tariq
Copyright
© © All Rights Reserved
Available Formats
Download as PPTX, PDF, TXT or read online on Scribd
You are on page 1/ 31

Parallel and Distributed

Computing

Lecture 2
Amdahl’s Law
Outline

 Amdahl’s Law of Parallel Speedup

 Karp-Flatt Metric

 Types of Parallelism
 Data-parallelism
 Functional-parallelism
 Pipelining

 Multi-processor vs Multi-computer

 Cluster vs Network of workstations


Amdahl’s Law

 Amdahl’s was formulized in 1967


 It shows an upper-bound on the maximum speedup that can be
achieved
 Suppose you are going to design a parallel algorithm for a
problem
 Further suppose that fraction of total time that the algorithm
must consume in serial executions is ‘F’
 This implies fraction of parallel potion is (1- F)
 Now, Amdahl’s law states that

 Here ‘p’ is total number of available processing nodes.


Amdahl’s Law

Derivation
 Let’s suppose you have a sequential code for a problem that
can be executed in total T(s)time.
 T(p) be the parallel time for the same algorithm over p
processors.
Then speedup can be calculated using:-

 T(p) can be calculated as:


Amdahl’s Law

Derivation
 Again

 What if you have infinite number of processors?


 What you have to do for further speedup?
Amdahl’s Law

 Example 1: Suppose 70% of a sequential algorithm is


parallelizable portion. The remaining part must be calculated
sequentially. Calculate maximum theoretical speedup for
parallel variant of this algorithm using i). 4 processors and ii).
infinite processors.

 F and 1-F use Amdahl’s law to calculate theoretical speedups.


Amdahl’s Law

 Example 2: Suppose 25% of a sequential algorithm is


parallelizable portion. The remaining part must be calculated
sequentially. Calculate maximum theoretical speedup for parallel
variant of this algorithm using 5 processors and infinite processors.
 ???

 Little challenge: Determine, according to Amdahl's law, how


many processors are needed to achieve maximum theoretical
speedup while sequential portion remains the same?
 The answer may be surprising?
 That’s why we say actual achievable speedup is always less-than
or equal to theoretical speedups.
Karp-Flatt Metric
Karp-Flatt Metric

 The metric is used to calculate serial fraction for a given


parallel configuration.
 i.e., if a parallel program is exhibiting a speedup S while using P
processing units then experimentally determined serial fraction e is
given by :-

Example task: Suppose in a parallel program, for 5


processors, you gained a speedup of 1.25x, determine sequential
fraction of your program.
CS3006 - Fall 2021
Types of Parallelism
Types of Parallelism

1. Data-parallelism
 When there are independent tasks applying the same operation
to different elements of a data set
 Example code
for i= to 99 do
a[ i ] = b[ i ] + c [ i ]
Endfor

 Here same operation addition is being performed on first 100


of ‘b’ and ‘c’
 All 100 iterations of the loop could be executed
simultaneously.
Types of Parallelism

2. Functional-parallelism
 When there are independent tasks applying different operations
to different data elements
 Example code
1) a=2
2) b=3
3) m= (a+b)/2
4) s= ( + )/2
5) v= s -
 Here third and fourth statements could be performed
concurrently.
Types of Parallelism

3. Pipelining
 Usually used for the problems where single instance of the
problem can not be parallelized
 The output of one stage is input of the other stage
 Dividing whole computation of each instance into multiple
stages provided that there are multiple instances of the problem
 An effective method of attaining parallelism on the
uniprocessor architectures
 Depends on pipelining abilities of the processor
Types of
Parallelism
3. Pipelining
 Example:
Assembly line
analogy

Sequential Execution
Types of
Parallelism
3. Pipelining
 Example:
Assembly line
analogy

Pipelining
Types of
Parallelism
3. Pipelining
 Example:
Overlap
instructions in a
single instruction
cycle to achieve
parallelism

4-stage Pipelining
Multi-processor
vs
Multi-Computer
Multi-Processor

 Multiple-CPUs with a shared memory

 The same address on two different CPUs refers to the same


memory location.

 Generally two categories:-


1. Centralized Multi-processors
2. Distributed Multi-processor
Multi-Processor

i. Centralized Multi-processor
 Additional CPUs are attached to the
system bus, and all the processors
share the same primary memory
 All the memory is at one place and
has the same access time from every
processor
 Also known to as UMA(Uniform
Memory Access) multi-processor or
SMP (symmetrical Multi-processor )
Multi-Processor

ii. Distributed Multi-processor


 Distributed collection of memories
forms one logical address space
 Again, the same address on different
processors refers to the same memory
location.
 Also known as non-uniform memory
access (NUMA) architecture
 Because, memory access time varies
significantly, depending on the
physical location of the referenced
address
Multi-Computer

 Distributed-memory, multi-CPU computer


 Unlike NUMA architecture, a multicomputer has disjoint
local address spaces
 Each processor has direct access to their local memory only.
 The same address on different processors refers to two
different physical memory locations.
 Processors interact with each other through passing
messages
Multi-Computer
Asymmetric Multi-Computers

 A front-end computer that interacts


with users and I/O devices

 The back-end processors are


dedicatedly used for “number
crunching”

 Front-end computer executes a


full, multiprogrammed OS and
provides all functions needed for
program development

 The backends are reserved for


executing parallel programs
Multi-Computer
Symmetric Multi-
Computers
 Every computer executes same OS
 Users may log into any of the
computers
 This enables multiple users to
concurrently login, edit and
compile their programs.
 All the nodes can participate in
execution of a parallel program
Network of Workstations
vs
Cluster
Cluster Network of workstations

Usually a co-located collection of low-cost A dispersed collection of computers.


computers and switches, dedicated to Individual workstations may have different
running parallel jobs. Operating systems and executable
All computer run the same version of programs
operating system.
Some of the computers may not have User have the power to login and power
interfaces for the users to login off their workstations
Commodity cluster uses high speed Ethernet speed for this network is usually
networks for communication such as fast slower. Typical in range of 10 Mbps
Ethernet@100Mbps, gigabit
Ethernet@1000 Mbps and Myrinet@1920
Mbps.
Reading Assignment

 Cache Coherence and Snooping


 Branch prediction and issues while
pipelining the problem
Assigned reading pointers:

 Cache Coherence:
 When we are in a distributed environment, each CPU’s cache needs to
be consistent (continuously needs to be updated for current values),
which is known as cache coherence.
 Snooping:
 Snoopy protocols achieve data consistency between the cache memory
and the shared memory through a bus-based memory system. Write-
invalidate and write-update policies are used for maintaining cache
consistency.
 Branch Prediction:
 Branch prediction is a technique used in CPU design that attempts to
guess the outcome of a conditional operation and prepare for the most
likely result.
Questions
CS3006 - Fall 2021

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy