Scribd
Upload
81 views

24-25 - Parallel Processing PDF

This document discusses parallel processing and parallel architectures. It covers the following key points in 3 sentences: The document introduces parallel processing and motivations for using parallel computers to solve applications with large numerical computations or data. It then classifies parallel architectures based on instruction and data flows as SISD, SIMD, MISD, and MIMD. The performance of parallel architectures is discussed in terms of speedup, efficiency, and limitations from sequential portions, communication costs, and load balancing.

Uploaded by

fanna786
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
81 views

24-25 - Parallel Processing PDF

This document discusses parallel processing and parallel architectures. It covers the following key points in 3 sentences: The document introduces parallel processing and motivations for using parallel computers to solve applications with large numerical computations or data. It then classifies parallel architectures based on instruction and data flows as SISD, SIMD, MISD, and MIMD. The performance of parallel architectures is discussed in terms of speedup, efficiency, and limitations from sequential portions, communication costs, and load balancing.

Uploaded by

fanna786
Copyright
© © All Rights Reserved
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 36

Rehan Azmat

Lecture24 and 25
Parallel Processing

Introduction and Motivation


Loop Unrolling
The IA-64 Architecture
Features of Itanium

Introduction and Motivation


Architecture classification
Performance of Parallel Architectures
Interconnection Network

Reduction of instruction execution time


Increased clock frequency by fast circuit technology
Simplify instructions (RISC)

Parallelism within processor


Pipelining
Parallel execution of instructions (ILP)
Superscalar
VLIW architectures

Parallel processing

Sequential computers often are not able to meet


performance needs in certain applications:
Fluid flow analysis and aerodynamics;
Simulation of large complex systems;
in physics, economy, biology, etc.

Computer aided design;


Multimedia.

Applications in the above domains are characterized


by a very large amount of numerical computation
and/or a high quantity of input data.
In order to deliver sufficient performance for such
applications, we can have many processors in a single
computer.

Parallel computers refer to architectures in which


many CPUs are running in parallel to implement a
certain application.
Such computers can be organized in very
different ways, depending on several key
features:

number and complexity of individual CPUs;


availability of common (shared) memory;
interconnection topology;
performance of interconnection network;
I/O devices;
etc.

In order to solve a problem using a parallel computer, one must


decompose the problem into small sub-problems, which can be solved
in parallel.
The results of sub-problems may have to be combined to get the final result of the
main problem.

Because of data dependency among the sub-problems, it is not easy to


decompose a complex problem to generate a large degree of
parallelism.

Because of data dependency, the processors may have to communicate


among each other.

The time taken for communication is usually very high when compared
with the processing time.

The communication scheme must therefore be very well planned in


order to get a good parallel algorithm.

Architectures Classification

Flynns classification (1966) is based on the


nature of the instruction flow executed by the
computer and that of the data flow on which
the instructions operate.
The multiplicity of instruction stream and
data stream gives us four different classes:

Single instruction, single data stream - SISD


Single instruction, multiple data stream - SIMD
Multiple instruction, single data stream - MISD
Multiple instruction, multiple data stream- MIMD

Single processor
Single instruction stream
Data stored in single memory

Lockstep systems are fault-tolerant computer systems that run the same set of operations at
the same time in parallel.

Single machine instruction stream

Simultaneous execution on different set of data

Large number of processing elements

Lockstep synchronization among the process elements.

The processing elements can

Have their respective private data memory; or


Share a common memory via an interconnection network.

Array and vector processors are the most common SIMD


machines A processor lockstep is a technique used to achieve high reliability in a microprocessor system. This is done by
adding a second identical processor to a system that monitors and verifies the operation of the system
processor.* The two processors are initialized to the same state during system start-up, and they receive the
same inputs. so during normal operation the state of the two processors is identical from clock to clock. They
are said to be operating in lockstep.

Global memory which can be accessed by all processors


of a parallel computer.
Data in the global memory can be read/write by any of
the processors.

Single sequence of data


Transmitted to set of processors
Each processor executes different instruction
sequences
Never been implemented!

Set of processors
Simultaneously execute different instruction
sequences

Different sets of data

The MIMD class can be further divided:


Shared memory (tightly coupled):

Symmetric multiprocessor (SMP)


Non-uniform memory access (NUMA)

Distributed memory (loosely coupled) = Clusters


Each processor in a parallel computer has its own memory (local memory); no other processor can
access this memory.

IS- Instruction stream


DS= Data stream

LM= Local Memory

Distributed Memory

Shared Memory

Performance of Parallel
Architectures

Important questions:
How fast does a parallel computer run at its
maximal potential?

How fast execution can we expect from a


parallel computer for a given application?
How do we measure the performance of a
parallel computer and the performance
improvement we get by using such a
computer?

Peak rate: the maximal computation rate that can be


theoretically achieved when all modules are fully utilized.
The peak rate is of no practical significance for the user.
It is mostly used by vendor companies for marketing their
computers.

Speedup: measures the gain we get by using a certain parallel


computer to run a given parallel program in order to solve a
specific problem.

Efficiency: this metric relates the speedup to the


number of processors used; by this it provides a
measure of the efficiency with which the processors
are used.

S: speedup;
P: number of processors.
For the ideal situation, in theory:

S = P; which means E = 1.

Practically the ideal efficiency of 1 cannot be


achieved!

Consider f to be the ratio of computations


that, according to the algorithm, have to be
executed sequentially (0 f 1); and P is the
number of processors.

S= speed up

is used to find the maximum expected improvement to an overall


system when only part of the system is improved. It is often used in
parallel computing to predict the theoretical maximum speedup using
multiple processors.

Amdahls law says that even a little ratio of


sequential computation imposes a limit to
speedup.
A higher speedup than 1/f can not be achieved,
regardless of the number of processors, since

To efficiently exploit a high number of


processors, f must be small (the algorithm
has to be highly parallel), since

Beside the intrinsic sequentiallity of parts of an


algorithm, there are also other factors that limit the
achievable speedup:

communication cost
load balancing of processors
costs of creating and scheduling processes
I/O operations (mostly sequential in nature)

There are many algorithms with a high degree of


parallelism.

The value of f is very small and can be ignored;


Suited for massively parallel systems; and
The other limiting factors, like the cost of communications,
become critical.

Consider a highly parallel computation, so that f can be neglected.


We define fc the fractional communication overhead of a processor:
Tcalc: the time that a processor executes computations;
Tcomm: the time that a processor is idle because of communication;

With algorithms having a high degree of parallelism, massively parallel


computers, consisting of large number of processors, can be efficiently
used if fc is small.
The time spent by a processor for communication has to be small compared to its
time for computation.

In order to keep fc reasonably small, the size of processes can not go


below a certain limit.

Interconnection Network

The interconnection network (IN) is a key


component of the architecture. It has a decisive
influence on:
the overall performance; and
total cost of the architecture.

The traffic in the IN consists of data transfer and


transfer of commands and requests (control
information).
The key parameters of the IN are

total bandwidth: transferred bits/second; and


implementation cost.

Single bus networks are simple and cheap.


One single communication is allowed at a
time; the bandwidth is shared by all nodes.
Performance is relatively poor.
In order to keep a certain performance, the
number of nodes is limited (around 16 - 20).
Multiple buses can be used instead, if needed.

Each node is connected to every other one.


Communications can be performed in parallel
between any pair of nodes.
Both performance and cost are high.
Cost increases rapidly with number of nodes.

A dynamic network: the interconnection topology can be modified


by configurating the switches.
It is completely connected: any node can be directly connected to
any other.
Fewer interconnections are needed than for the static completely
connected network; however, a large number of switches is needed.
A large number of communications can be performed in parallel
(even though one node can receive or send only one data at a time).

Cheaper than completely connected networks, while giving


relatively good performance.
In order to transmit data between two nodes, routing through
intermediate nodes is needed (maximum 2(n-1)
intermediates for an nn mesh).
It is possible to provide wrap-around connections:
Torus.

Three dimensional meshes have been also implemented.

2n nodes are arranged in an n-dimensional


cube. Each node is connected to n neighbors.
In order to transmit data between two nodes,
routing through intermediate nodes is
needed (maximum n intermediates).

The growing need for high performance can not always be satisfied by
computers with a single CPU.

With parallel computers, several CPUs are running concurrently in order


to solve a given application.

Parallel programs have to be available in order to make use of the


parallel computers.

Computers can be classified based on the nature of the instruction flow


and that of the data flow on which the instructions operate.

A key component of a parallel architecture is also the interconnection


network.

The performance we can get with a parallel computer depends not only
on the number of available processors but is limited by characteristics of
the executed programs.

End of the Lecture

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy