Parallel Architectures

Ever Faster
There is a continuing drive to create faster computers. t Some large engineering problems require massive computing resources. Increases in clock speed alone will not produce a sufficient increase in computer performance. Parallelism provides an ability to get more done in the same amount of time.

Parallel Architectures
COMP375 Computer Architecture and dO Organization i ti

Intel Microprocessor Performance

8000.0

Intel Performance
7000.0 6000.0

Clock
5000.0 4000.0

Architecture
Architectural Improvements

3000.0

2000.0

1000.0

0.0 1987 1989 1991 1993 1995 1997 1999 2001 2002 2003

From Stallings textbook

Parallel Architectures

Large Computing Challenges

(Memory Requirement)

Parallelism
The key to making a computer fast is to do as much hi in parallel ll l as possible. ibl Processors in modern home PCs do a lot in parallel
Superscalar execution Multiple ALUs Hyperthreading Pipelining

Demands exceed the capabilities of f even the th fastest f t t current t uniprocessor systems

from http://meseec.ce.rit.edu/eecc756-spring2002/756-3-8-2005.ppt

Processes and Threads

All modern operating systems support multiple processes and threads. Processes have their own memory space. Each program is a process. Threads are a flow of control through a process. Each process has at least one thread. thread Single processor systems support multiple processes and threads by sequentially sharing the processor.

Hyperthreading
Hyperthreading allows the apparently parallel ll l execution ti of ft two th threads. d Each thread has its own set of registers, but they share a single CPU. The CPU alternates executing instructions from each thread thread. Pipelining is improved because adjacent instructions come from different threads and therefore do not have data hazards.

Parallel Architectures

Hyperthreading Performance
Also called Multiple Instruction Issue processors Overall throughput is increased Individual threads do not run as fast as they would without hyperthreading
Thread 1 Th d 2 Thread Thread 1 Thread 2 Thread 1 Thread 2 Th d 1 Thread Thread 2

Pipelining

Hyperthreading
1. Runs best in single th d d applications threaded li ti 2. Replaces pipelining 3. Duplicates the user register set 4 Replaces dual 4. dual-core core processors.

How Visible is Parallelism?

Superscalar
Programmer never notices

Multiple Threads
Programmer must create multiple threads in the program.

Multiple processes
Different programs p g
Programmer never notices

Parts of the same program

Programmer must divide the work among different processes.

Parallel Architectures

Flynns Parallel Classification

SISD Single Instruction Single Data
standard uniprocessors

MIMD Computers
Shared Memory
Attached processors SMP
Multiple processor systems Multiple instruction issue processors Multi-Core processors

SIMD Single Instruction Multiple Data

vector and array processors

MISD - Multiple Instruction Single Data

systolic t li processors

NUMA

Separate S t Memory M
Clusters Message passing multiprocessors
Hypercube Mesh

MIMD Multiple Instruction Multiple Data

Single Instruction Multiple Data

Many high performance computing programs perform f calculations l l ti on l large d data t sets. t SIMD computers can be used to perform matrix operations. The Intel Pentium MMX instructions perform SIMD operations on 8 one byte integers.

Vector Processors
A vector is a one dimensional array. Vector processors are SIMD machines that have instructions to perform operations on vectors. A vector process might have vector registers with 64 doubles in each register register. A single instruction could add two vectors.

Parallel Architectures

Vector Example
/* traditional vector addition */ double a[64], b[64], c[64]; for (i = 0; i < 64; i++) c[i] = a[i] + b[i]; /* vector processor in one instruction*/ ci = ai + bi

Non-contiguous Vectors
A simple vector is a one dimensional array with ith each h element l t in i successive i addresses dd In a two dimensional array, the rows can be considered simple vectors. The columns are vectors, but the are not stored in consecutive addresses. The distance between elements of a vector is called the stride.

Vector Stride
Matrix multiplication
b11, b12 a11, a12, a13 a 21, a 22, a 23 * b21, b22 b31, b32

Cray-1 Vector Processor

Sold to Los Alamos National Laboratory in1976 for $8.8 million 80 MHz clock 160 megaflops g y ( (1 M words) ) 8 megabyte Freon cooled

a11 b11

a12 a13 a21 a22 a23 b12 b21 b22 b31 b32

Parallel Architectures

Cray-2
Built in 1985 with ten times the performance of the Cray-1

Intel calls some of their new Pentium instructions SIMD because

1 The 1. They operate on multiple bytes 2. There is a separate processor for them 3 Multiple processing 3. units operate on one piece of data 4. They are executed in the GPU

Data Flow Architecture

MISD Systolic Architectures

Memory Processor Element Processor Element Processor Element

Data flow graph computing N =(A+B) * (B-4)

A computed value is passed from one processor to the next. Each processor may perform a different operation. Minimizes memory accesses.

Parallel Architectures

Systolic Architectures
Systolic arrays meant to be special-purpose processors or coprocessors and were very finegrained Processors implement a limited and very simple computation, usually called cells Communication is very fast, granularity meant to be very fine (a small number of computational operations per communication) Very y fast clock rates due to regular, g synchronous y structure Data moved through pipelined computational units in a regular and rhythmic fashion Warp and iWarp were examples of systolic arrays
from www-csag.ucsd.edu/teaching/cse160s05/lectures/Lecture15.pdf

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Ali Alignments in i time i

Rows of A

b2,0 b1,0 , b0,0

b2,1 b1,1 b0,1 ,

b2,2 b1,2 b0,2

Columns of B

a0,2

a0,1

a0,0

a1 2 a1,2

a1,1 a1 1

a1,0 a1 0

a2,2

a2,1 T=0

a2,0

Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid

Alignments in time

b2,0 b1,0
b0,0

b2,1 b1,1 b0,1

b2,2 b1,2 b0,2

Each processor accumulates one element of the product

Alignments in time

b2 0 b2,0
b1,0

b2,1 b1,1 b1 1
b0,1
a0,0*b0,1

b2,2 b1,2 b0,2 b0 2

a0,2

a0,1

a0,0

a0,0*b0,0

a0,2

a0,1

a0,0*b0,0 + a0,1*b1,0

a0,0

b0,0

a1 2 a1,2

a1,1 a1 1

a1,0 a1 0

a1 2 a1,2

a1,1 a1 1

a1,0

a1,0*b0,0

a2,2 T=1

a2,1

a2,0 T=2

a2,2

a2,1

a2,0

Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Parallel Architectures

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Alignments in time
b2,0 a0,2
a0,0*b0,0 + a0,1*b1,0 + a0,2*b2,0

b2,1
b1,1
a0,0*b0,1 + a0,1*b1,1

b2,2 b1,2
b0,2
a0,0*b0,2

Alignments in time
b2,1
a0,0*b0,0 + a0,1*b1,0 + a0,2*b2,0

b2,2
b1,2
a0,0*b0,2 + a0,1*b1,2

a0,1

a0,0

a0,2

a0,0*b0,1 + a0,1*b1,1 + a0,2*b2,1

a0,1

b1,0

b0,1
a1,0*b0,1

b2,0 a1,2
a1,0*b0,0 + a1,1*b1,0 , , + a1,2*a2,0

b1,1
a1,0*b0,1 +a1,1*b1,1

b0,2
a1,0*b0,2

a1 2 a1,2

a1,1

a1,0*b0,0 + a1,1*b1,0 , ,

a1,0

a1,1

a1,0 ,

b0,0

b1,0

b0,1
a2,0*b0,1

a2,2 T=3

a2,1

a2,0

a2,0*b0,0

a2,2 a2,2 T=4

a2,1

a2,0*b0,0 + a2,1*b1,0

a2,0

Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Alignments in time
b2,2
a0,0*b0,0 + a0,1*b1,0 + a0,2*b2,0 a0,0*b0,1 + a0,1*b1,1 + a0,2*b2,1

Alignments in time

a0,2

a0,0*b0,2 + a0,1*b1,2 + a0,2*b2,2

a0,0*b0,0 + a0,1*b1,0 + a0,2*b2,0

a0,0*b0,1 + a0,1*b1,1 + a0,2*b2,1

a0,0*b0,2 + a0,1*b1,2 + a0,2*b2,2

b2,1
a1,0*b0,0 + a1,1*b1,0 , , + a1,2*a2,0

b1,2
a1,0*b0,2 + a1,1*b1,2 1 1*b1 2 a1,0*b0,0 + a1,1*b1,0 , , + a1,2*a2,0 a1,0*b0,1 +a1,1*b1,1 + a1,2*b2,1

b2,2 a1,2 ,
a1,0*b0,2 + a1,1*b1,2 1 1*b1 2 + a1,2*b2,2

a1,2

a1,0*b0,1 +a1,1*b1,1 + a1,2*b2,1

a1,1 ,

b2,0 a2,2
a2,0*b0,0 + a2,1*b1,0 + a2,2*b2,0

b1,1
a2,0*b0,1 + a2,1*b1,1

b0,2
a2,0*b0,2 a2,0*b0,0 + a2,1*b1,0 + a2,2*b2,0

b2,1 a2,2
a2,0*b0,1 + a2,1*b1,1 + a2,2*b2,1

b1,2
a2,0*b0,2 + a2,1*b1,2

a2,1

a2,0

a2,1

T=5
Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

T=6
Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Parallel Architectures

Systolic Array Example: 3x3 Systolic Array Matrix Multiplication

Processors arranged in a 2-D grid Each processor accumulates one element of the product

Systolic Architectures
Never caught on for General-purpose computation t ti

Alignments in time

a0,0*b0,0 + a0,1*b1,0 + a0,2*b2,0

a0,0*b0,1 + a0,1*b1,1 + a0,2*b2,1

a0,0*b0,2 + a0,1*b1,2 + a0,2*b2,2

a1,0*b0,0 + a1,1*b1,0 , , + a1,2*a2,0

Done
a2,0*b0,0 + a2,1*b1,0 + a2,2*b2,0

a1,0*b0,1 +a1,1*b1,1 + a1,2*b2,1

a1,0*b0,2 + a1,1*b1,2 1 1*b1 2 + a1,2*b2,2

Regular structures are HARD to achieve Real efficiency requires local communication, but flexibility of global communication is often critical

Used mainly in structured settings

High speed multipliers Structured Array Computations
from www-csag.ucsd.edu/teaching/cse160s05/lectures/Lecture15.pdf

b2,2
a2,0*b0,1 + a2,1*b1,1 + a2,2*b2,1

a2,2

a2,0*b0,2 + a2,1*b1,2 + a2,2*b2,2

T=7
Example source: http://www.cs.hmc.edu/courses/2001/spring/cs156/

Multiple Instruction Multiple Data

Multiple independent processors can execute different parts of a program to different programs. programs The processors can be different or identical. Different architectures allow the processors to access:
the same memory separate memory local memory with access to all memory

Specialized Processors
Some architectures have a general purpose processor with additional specialized processors I/O processors
I/O controllers with DMA Graphics processor

Floating point processors

Intel 80387 for Intel 80386

Vector processors

Parallel Architectures

Separate Memory Systems

Each processor has its own memory that none of f the th other th processors can access. Each processor has unhindered access to its memory. No cache coherence problems. Scalable to very large numbers of processors Communication is done by passing messages over special I/O channels

Topologies
The CPUs can be interconnected in different ways. Goals of an interconnection method are:
Minimize the number of connections per CPU Minimize the average distance between nodes Minimize the diameter or maximum distance between any two nodes Maximize the throughput Simple routing of messages

Grid or Mesh
Each node has 2, 3 or 4 connections Routing is simple and many routes are available. The diameter is

Torus
Similar to a mesh with ith connected t d end d nodes. Many similar routes available. The diameter is

2 N 2

10

Parallel Architectures

Hypercube
Each node log2N connections ti Routing can be done by changing one bit of the address at a time. The diameter is log2N

NUMA systems
In NonUniform Memory Access systems each node has local memory but can also access the memory of other nodes nodes. Most memory access are to local memory. The memory of other nodes can be accessed, but the access is slower than local memory memory. Cache coherence algorithms are used when shared memory is accesses. More scalable than SMP.

NUMA system

11