Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

D.

Basak

and

D.

K.

Panda

Technical

Report

Benefits

of

Processor

Clustering

in

Designing

Parallel

Systems

When

and

How

Debashis

Basak

and

Dhabaleswar

K.

Panda

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

Columbus

OH

Email

fbasak,pandag@cis.ohio-state.edu

Contact

Author

Dhabaleswar

K.

Panda

Abstract

Advances

in

multiprocessor

interconnect

technology

are

leading

to

high

performance

networks

However

software

overheads

associated

with

message

passing

are

limiting

the

processors

to

get

maximum

performance

from

these

networks

leading

to

under-utilization

of

network

resources

Several

research

studies

are

ongoing

for

designing

messaging

protocols

and

hardware

to

reduce

such

overheads

However

with

such

protocols

messaging

overheads

can

not

be

eliminated

Even

though

the

overheads

can

be

lowered

these

will

continue

to

be

reasonably

high

compared

to

the

network

speed

In

this

paper

we

take

an

alternative

approach

and

analyze

on

how

to

exploit

emerging

processor-cluster

technology

for

designing

high-performance

and

cost-effective

parallel

systems

given

that

messaging

overheads

can

limit

maximum

performance

Though

processor-clusters

are

being

used

in

some

systems

in

an

ad

hoc

manner

there

is

no

formal

analysis

in

the

literature

to

show

when

and

how

processor

clusters

benefit

in

designing

high

performance

and

scalable

systems

In

this

paper

we

present

a

design-space-graph

framework

for

analyzing

and

solving

this

problem

by

considering

processor-clustering

messaging

overheads

and

network

performance

in

an

integrated

manner

Our

analysis

establishes

the

following

three

design

guidelines

With

messaging

overheads

constraining

performance

in

a

system

processor

clustering

can

be

used

to

build

a

-RRB-

an

equal-sized

system

with

a

smaller

network

or

b

-RRB-

a

larger

system

with

an

equal-sized

network

With

messaging

overheads

not

being

a

constraint

a

combination

of

processor

clustering

and

wider

channels

can

be

used

to

build

a

range

of

larger-sized

systems

These

guidelines

are

validated

through

simulation

and

experimentation

All

these

guidelines

lead

to

designing

cost-effective

and

scalable

parallel

systems

while

delivering

high

performance

This

research

is

supported

in

part

by

NSF

Grant

and

an

Ohio

State

University

Presidential

Fellowship

A

preliminary

version

of

this

paper

-LSB-

-RSB-

has

been

presented

in

Int

l

Parallel

Processing

Symposium

Apr.

Contents

Introduction

Processor-cluster

based

systems

Enhanced

design

space

with

processor-clustering

Characterizing

communication

performance

and

bottlenecks

Messaging

Overheads

Network

bisection

Determining

communication

performance

interplay

of

above

constraints

Experimentation

and

simulation

setup

Validation

of

communication

performance

predictions

Characterizing

parallel

systems

based

on

communication

performance

Designing

a

system

with

a

given

number

of

processors

Deriving

the

balanced

configuration

for

a

given

channel

width

Required

scaling

in

channel

width

with

larger

cluster

sizes

Inter-cluster

interconnection

cost

Scaling

systems

to

support

larger

number

of

processors

Deriving

the

largest

system

size

supportable

on

a

given

inter-cluster

network

Required

increase

in

channel

width

for

further

scaling

Design

under

technological

constraints

Deriving

cost-effective

configuration

for

a

given

total

number

of

processors

Deriving

the

maximal

system

size

while

scaling

with

fixed

topology

and

larger

clusters

Experimental

validation

pattern

and

application

based

approach

Experimental

platform

Traffic

patterns/applications

Designing

a

system

with

a

given

number

of

processors

Deriving

the

balanced

configuration

for

a

given

channel

width

Required

scaling

in

channel

width

with

larger

clusters

Scaling

systems

to

support

larger

number

of

processors

Deriving

the

largest

system

size

supportable

on

a

given

inter-cluster

network

Required

increase

in

channel

width

for

further

scaling

Conclusion

Introduction

The

performance

of

a

multiprocessor

system

is

often

limited

by

the

bisection

bandwidth

of

its

interconnection

network

-LSB-

-RSB-

In

the

recent

past

systems

have

been

built

using

advanced

interconnection

networks

with

and

bit

channels

and

channel

cycle

times

of

ns

The

offered

bandwidth

for

inter-processor

communication

in

such

systems

is

very

high

This

coupled

with

the

recent

trends

in

routing

mechanisms

like

fast

packet

switching

virtual-cut

through

and

wormhole

routing

has

led

to

systems

offering

very

small

point-to-point

network

delay

in

the

range

of

less

than

a

microsecond

The

increasing

availability

of

high

bandwidth

interconnection

networks

raises

the

following

question

Can

the

offered

higher

bandwidth

be

effectively

exploited

to

always

deliver

higher

system

performance

In

point-to-point

communication

a

message

is

associated

with

processing

overheads

from

packetization

checksum

computing

packet-ordering

and

memory

copying

Such

overheads

usually

in

the

range

of

few

to

tens

of

microseconds

can

severely

limit

the

rate

at

which

messages

can

be

sent

or

received

from

a

processor

-LSB-

-RSB-

This

can

lead

to

much

of

the

communication

bandwidth

offered

by

the

expensive

fast

interconnect

to

remain

unutilized

Several

research

studies

are

ongoing

-LSB-

-RSB-

for

designing

messaging

protocols

and

hardware

to

reduce

such

overheads

However

even

with

efficient

protocols

these

overheads

can

not

be

eliminated

Even

though

the

overheads

get

lowered

these

will

continue

to

be

reasonably

high

compared

to

network

speeds

For

example

on

messaging

start-up

overhead

is

around

microseconds

and

network

channel

transmission

time

per-byte

is

only

microseconds

-LSB-

-RSB-

This

leads

to

a

ratio

of

message

overhead

to

per-byte

transmission

delay

as

In

contemporary

systems

like

the

Cray

message

overhead

is

microseconds

and

per-byte

network

transmission

time

is

microseconds

leading

to

a

ratio

of

Thus

messaging

overheads

continue

to

be

a

dominant

factor

for

poor

utilization

of

network

resources

in

current

generation

parallel

systems

Advancements

in

VLSI

and

packaging

technologies

is

now

allowing

the

integration

of

a

small

number

of

processing

elements

interconnected

through

a

network

onto

a

single

multi-chip

or

a

board

-LSB-

-RSB-

module

Such

processor-cluster

modules

are

being

increasingly

used

instead

of

single

processors

to

build

variety

of

computing

vehicles

such

as

servers

workstations

etc.

This

is

leading

to

greater

demand

and

production

for

such

modules

consequently

bringing

down

perunit

cost

Larger

systems

are

being

built

by

interconnecting

multiple

processor-clusters

-LSB-

-RSB-

Such

an

approach

offers

scalability

while

exploiting

the

cost-effectiveness

and

convenience

of

processorcluster

modules

Similar

motivations

are

also

increasingly

guiding

the

development

of

traditional

parallel

systems

using

processor-clusters

-LSB-

-RSB-

Previous

research

on

using

processor-clusters

for

building

large

parallel

systems

have

mainly

focused

on

proposing

and

proving

different

interconnection

topologies

-LSB-

-RSB-

and

studying

the

design

problem

under

very

realistic

packaging

con

straints

-LSB-

-RSB-

Such

processor

clustering

also

demonstrates

potential

to

alleviate

the

network

under-utilization

problem

by

allowing

more

number

of

processors

to

use

a

given

set

of

network

resources

However

no

formal

study

is

available

in

the

literature

which

shows

the

benefits

of

processor

clustering

towards

better

utilization

of

network

resources

In

this

paper

we

take

on

such

a

challenge

and

demonstrate

how

to

exploit

the

emerging

cluster

and

interconnect

technologies

in

designing

balanced

and

cost-effective

systems

We

introduce

a

new

design

space

graph

framework

to

represent

the

widened

spectrum

of

options

to

build

parallel

systems

under

the

flexibility

of

selecting

a

given

inter-cluster

size

cluster

size

and

channel

width

Our

solutions

span

along

two

directions

Systems

with

high

messaging

overheads

can

have

poor

network

utilization

In

this

case

we

show

that

processor

clustering

can

be

used

to

share

network

resources

This

leads

to

effective

exploitation

of

the

communication

bandwidth

of

the

interconnection

network

resulting

in

larger

as

well

as

balanced

and

more

cost-effective

systems

Along

the

second

direction

we

show

that

the

advantages

of

clustering

can

also

be

extended

to

design

large

systems

when

messaging

overheads

are

low

In

this

case

building

larger

systems

is

proposed

by

using

larger

clusters

while

maintaining

the

interconnection

network

topology

The

performance

is

maintained

by

using

proportionately

wider

network

channels

We

demonstrate

that

as

long

as

cluster

size

and

channel

width

can

grow

under

technological

constraints

such

an

approach

offers

solutions

to

design

cost-effective

systems

while

maintaining

performance

The

paper

is

organized

as

follows

In

Secs.

and

we

present

the

processor-cluster

based

approach

and

its

impact

on

designing

large

multiprocessor

systems

In

Sec.

we

discuss

two

potential

bottlenecks

affecting

the

communication

performance

in

a

parallel

system

In

Sec

we

present

a

characterization

of

parallel

systems

based

on

communication

performance

Design

guidelines

to

organize

and

scale

systems

in

a

balanced

and

cost-effective

manner

are

presented

and

verified

in

Secs.

and

Impact

of

technological

constraints

on

design

is

presented

in

Sec.

Application

benchmark

based

evaluation

of

results

is

presented

in

Sec.

Finally

concluding

remarks

and

future

work

are

presented

in

Sec.

Processor-cluster

based

systems

A

parallel

system

consists

of

a

large

number

of

computing

nodes

interconnected

through

a

network

Typical

topologies

for

the

interconnection

are

meshes/tori

hypercubes

multi-stage

interconnection

networks

and

fat-trees

For

inter-processor

communication

each

processor

is

connected

to

a

router

Messages

are

routed

along

these

routers

on

the

interconnection

network

An

example

system

with

nodes

interconnected

by

a

mesh

network

is

shown

in

Fig.

-LRB-

a

-RRB-

It

can

be

observed

that

in

such

an

organization

the

required

number

of

network

routers

denoted

by

R

is

equal

to

the

number

of

processors

in

the

system

i.e.

R

In

the

same

system

the

number

of

channels

in

the

network

can

be

derived

as

The

number

of

parallel

wires

in

a

channel

determines

its

width

denoted

as

W

The

amount

of

data

that

can

be

transferred

across

it

directly

depends

on

W

Let

NW

denote

the

total

number

of

wires

required

to

connect

the

channels

in

a

given

network

For

the

system

shown

in

Fig.

-LRB-

a

-RRB-

NW

single

processor

-LRB-

c

-RRB-

processor

router

-LRB-

a

-RRB-

cluster

-LRB-

c

-RRB-

computing

nodecomputing

node

-LRB-

b

-RRB-

R

NW

R

NW

W

Figure

Examples

of

parallel

systems

using

computing

nodes

based

on

a

-RRB-

single

processors

and

b

-RRB-

processorclusters

of

size

The

system

in

-LRB-

b

-RRB-

interconnects

the

same

number

of

processors

on

a

smaller

network

With

advancements

in

VLSI

and

packaging

technologies

computing

nodes

having

more

than

one

processor

on

a

single

multi-chip

module

or

processor-board

are

becoming

increasingly

viable

Many

current

parallel

systems

like

the

CRAY

-LSB-

-RSB-

Intel

Paragon

Stanford

DASH

-LSB-

-RSB-

and

Sequent

NUMA-Q

-LSB-

-RSB-

are

using

processor-cluster

based

design

to

build

cost-effective

systems

Figure

-LRB-

b

-RRB-

shows

an

example

system

with

processor-clusters

each

having

processors

This

configurations

uses

processor-clusters

connected

by

a

mesh

network

thus

supporting

a

total

of

processors

In

systems

built

using

processor-clusters

let

c

denote

the

number

of

processors

available

in

a

cluster

also

referred

to

as

the

size

of

a

cluster

The

maximum

value

of

c

depends

on

the

level

of

integration

available

inside

clusters

The

network

interconnecting

processor-clusters

is

referred

to

as

inter-cluster

network

For

uniformity

of

representation

a

single

processor

based

computing

node

is

treated

as

a

processor-cluster

of

size

one

For

designing

a

system

with

N

processors

with

processor-clusters

of

size

c

N

c

clusters

are

necessary

Processors

inside

a

cluster

communicate

with

each

other

using

intra-cluster

network

-LRB-

intracluster

communication

-RRB-

provided

as

part

of

the

processor-cluster

module

This

network

connecting

the

processors

inside

a

cluster

can

be

a

bus

crossbar

star

etc

as

discussed

in

-LSB-

-RSB-

Although

the

choice

of

the

cluster

topology

is

an

important

design

issue

in

this

study

we

emphasize

more

on

how

and

when

processor-clusters

can

be

used

in

designing

balanced

and

larger

cost-effective

systems

While

developing

design

guidelines

we

assume

a

processor-cluster

technology

offering

up

to

c

being

available

Inter-processor

communication

across

clusters

is

achieved

along

the

inter-cluster

network

Each

processor-cluster

is

connected

to

a

router

as

shown

in

Fig.

-LRB-

b

-RRB-

A

processor

injects

a

message

into

its

corresponding

router

which

is

routed

along

intermediate

routers

to

the

destination

router

where

it

is

consumed

by

the

destination

processor

The

number

of

inter-cluster

routers

required

is

the

same

as

the

number

of

clusters

-LRB-

R

N

c

-RRB-

For

example

in

the

system

shown

in

Fig.

-LRB-

a

-RRB-

with

N

and

c

we

have

R

Similarly

in

the

system

shown

in

Fig.

-LRB-

b

-RRB-

with

N

and

c

we

require

fewer

routers

-LRB-

R

-RRB-

Thus

such

a

processor-cluster

based

system

with

larger

clusters

requires

lesser

inter-cluster

routers

Similarly

it

can

be

observed

that

a

system

with

c

requires

lesser

NW

compared

to

NW

in

a

system

with

c

This

leads

us

to

the

intuition

that

processor-cluster

based

approach

to

designing

parallel

systems

has

potential

in

offering

lower

interconnection

costs

For

a

fair

cost

comparison

one

should

also

keep

in

mind

the

increased

routing

hardware

inside

clusters

for

intra-cluster

communication

However

with

higher

level

of

integration

inside

clusters

we

expect

the

intra-cluster

routing

capabilities

being

offered

at

comparatively

much

lower

cost

-LSB-

-RSB-

leading

to

an

overall

lowering

in

interconnection

cost

with

clustering

To

match

the

increasing

demand

on

computing

bandwidth

parallel

systems

need

to

be

scaled

to

larger

configurations

supporting

more

processing

nodes

The

default

approach

to

such

scaling

has

been

to

increase

the

network

size

with

more

links

and

routers

to

interconnect

more

processors

Figure

-LRB-

a

-RRB-

illustrates

such

an

approach

to

scale

a

mesh

interconnected

processor

base

system

to

a

larger

mesh

interconnected

processor

system

This

default

approach

maintains

the

cluster

size

at

one

-LRB-

c

-RRB-

The

channel

width

-LRB-

W

-RRB-

is

assumed

to

be

maintained

in

this

example

It

can

be

observed

that

the

scaled

system

requires

R

and

NW

An

alternative

approach

to

scaling

using

larger

processor-clusters

is

illustrated

in

Fig.

-LRB-

b

-RRB-

This

design

maintains

the

interconnection

network

to

R

and

NW

while

using

larger

clusters

of

size

four

to

achieve

similar

scaling

This

once

again

illustrates

the

potential

of

processor-cluster

based

design

in

offering

lower

interconnection

costs

Enhanced

design

space

with

processor-clustering

The

above

illustrations

emphasize

the

potential

of

processor-clustering

in

designing

parallel

systems

Besides

developments

in

processor-clustering

recent

advancements

in

interconnection

technology

are

also

allowing

wider

and

faster

network

channels

Such

channels

are

leading

to

higher

network

bandwidth

being

offered

However

high

bandwidth

networks

are

expensive

and

this

cost

increases

with

offered

bandwidth

Further

as

we

demonstrate

in

the

next

section

having

higher

network

bandwidth

does

not

necessarily

translate

to

higher

performance

Thus

for

a

system

architect

designing

a

parallel

system

it

is

imperative

to

determine

network

configurations

offering

the

best

achievable

performance

at

minimum

cost

In

this

section

we

present

the

spectrum

of

options

to

single

processor

-LRB-

c

-RRB-

processor

router

-LRB-

a

-RRB-

cluster

-LRB-

c

-RRB-

-LRB-

b

-RRB-

computing

node

computing

node

W

W

R

NW

R

NW

R

NW

W

Base

system

-LRB-

N

-RRB-

Scaled

systems

-LRB-

N

-RRB-

Figure

Examples

of

scaling

a

base

parallel

system

using

computing

nodes

based

on

a

-RRB-

single

processors

and

b

-RRB-

processor-clusters

of

size

The

system

in

-LRB-

b

-RRB-

offers

a

more

compact

interconnection

network

build

parallel

systems

under

the

flexibility

of

selecting

a

given

inter-cluster

size

cluster

size

and

channel

width

from

a

wide

range

of

available

choices

In

the

traditional

approach

of

using

single

processors

for

parallel

system

design

larger

configurations

necessarily

lead

to

larger

interconnects

requiring

more

routers

and

interconnection

wires

With

advancements

in

processor-cluster

technology

such

approach

is

undergoing

a

change

Larger

processor-clusters

are

allowing

a

wider

choice

of

design

configurations

Such

extensions

to

design

options

can

be

summarized

as

a

design

space

graph

-LRB-

DSG

-RRB-

as

shown

in

Fig.

The

vertical

axis

of

the

DSG

indicates

the

number

of

inter-cluster

routers

-LRB-

R

-RRB-

in

a

given

system

The

horizontal

axis

in

Fig.

represents

the

upcoming

flexibility

of

using

larger

processor-clusters

-LRB-

c

-RRB-

in

design

The

R

routers

are

interconnected

by

a

given

inter-cluster

network

topology

For

sake

of

illustration

and

to

match

with

the

previous

examples

we

consider

a

mesh

inter-cluster

topology

However

the

approach

is

applicable

to

any

other

topology

On

the

third

axis

we

denote

the

impact

of

advancing

channel

bandwidth

technology

Increase

in

channel

bandwidth

can

be

due

to

increased

channel

speed

and/or

larger

channel

width

technology

To

represent

the

impact

of

both

we

require

two

separate

technological

dimensions

on

the

DSG

However

in

most

analysis

we

are

interested

in

the

overall

impact

on

channel

bandwidth

leading

to

higher

interconnection

performance

Thus

in

this

paper

we

study

increase

in

channel

bandwidth

due

to

increase

in

only

channel

width

-LRB-

W

-RRB-

We

represent

W

in

bytes

to

reflect

a

trend

of

designing

byte-multiple

channel

width

to

maintain

an

integral

relationship

with

processor

and

memory

modules

-LSB-

-RSB-

The

plane

formed

by

the

R

and

c

axes

is

referred

to

as

the

R-c

plane

Each

point

on

the

R-c

plane

denoted

by

tuple

-LRB-

R

c

-RRB-

represents

a

parallel

system

configuration

with

-LRB-

N

Rc

-RRB-

processors

For

any

given

-LRB-

R

c

-RRB-

tuple

we

can

have

configurations

with

different

channel

bandwidth

These

are

depicted

in

the

DSG

as

an

arrow

out

of

the

R-c

plane

parallel

to

the

W

axis

and

represented

as

triple

-LRB-

R

c

W

-RRB-

For

example

in

Fig.

the

configuration

with

R

c

and

W

byte

representing

a

flat

configuration

is

shown

as

tuple

-LRB-

-RRB-

Similar

configurations

with

higher

channel

bandwidth

are

represented

as

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

Let

us

illustrate

how

the

example

systems

considered

earlier

in

Figs

and

fit

into

the

DSG

framework

Let

the

channel

width

-LRB-

W

-RRB-

considered

in

these

examples

be

byte

Thus

these

configurations

lie

on

the

R-c

plane

for

W

The

flat

configuration

in

Fig.

-LRB-

a

-RRB-

with

c

and

R

gets

represented

as

tuple

-LRB-

-RRB-

The

corresponding

inter-cluster

network

-LRB-

-RRB-

is

shown

underlined

The

default

approach

of

scaling

four

times

with

a

larger

interconnection

with

routers

while

maintaining

cluster

size

to

one

-LRB-

c

-RRB-

is

depicted

in

Fig.

as

tuple

-LRB-

-RRB-

A

vertical

arrow

from

-LRB-

-RRB-

to

-LRB-

-RRB-

represents

such

an

approach

Similar

scaling

can

be

alternatively

achieved

by

using

larger

processor-clusters

e.g.

-LRB-

c

-RRB-

This

approach

applied

to

tuple

-LRB-

-RRB-

leads

to

tuple

-LRB-

-RRB-

having

clusters

-LRB-

processors

-RRB-

connected

by

a

inter-cluster

network

In

this

second

approach

the

number

of

routers

is

maintained

at

R

Number

of

routers

-LRB-

R

-RRB-

Processors

connected

to

each

router

-LRB-

c

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Channel

Width

-LRB-

W

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

iso-processorline

for

N

-LRB-

in

bytes

-RRB-

Figure

Depicting

the

enhanced

design

space

for

parallel

system

configurations

using

larger

processorclusters

For

designing

systems

with

more

processors

traditional

approach

only

considered

varying

the

interconnect

size

leading

to

higher

R.

Using

processor-clusters

adds

a

second

dimension

of

variability

the

cluster

size

-LRB-

c

-RRB-

The

third

dimension

W

represents

the

added

flexibility

of

using

higher

bandwidth

network

channels

In

general

case

R

c

and

W

can

be

varied

leading

to

new

configurations

For

example

the

-LRB-

-RRB-

configuration

leads

to

a

-LRB-

-RRB-

configuration

by

reducing

R

four

times

while

compensating

for

this

reduction

by

choosing

c

This

example

demonstrates

the

approach

of

clustering

to

design

systems

having

same

number

of

processors

with

a

smaller

inter-cluster

network

and

fewer

routers

Let

us

introduce

a

set

of

definitions

with

respect

to

the

DSG

to

be

used

in

the

remaining

part

of

the

paper

Definition

The

set

of

configurations

-LRB-

N

c

c

W

-RRB-

having

same

total

number

of

processors

N

for

different

values

of

c

and

W

lie

in

the

R

c

W

space

on

a

plane

defined

as

the

iso-processor

plane

for

N

For

a

given

W

W

the

subset

of

these

configurations

-LRB-

N

c

c

W

-RRB-

lie

on

the

R

c

plane

for

W

W

as

a

line

defined

as

iso-processor

line

for

N.

Let

us

consider

a

given

-LRB-

R

c

W

-RRB-

configuration

reflecting

contemporary

trends

in

designing

parallel

systems

Let

us

assume

the

inter-cluster

topology

as

an

n-dimensional

mesh

Thus

the

R

routers

in

the

-LRB-

R

c

W

-RRB-

configuration

are

interconnected

by

a

kn

mesh

where

Qni

ki

R

In

this

case

a

more

detailed

notation

-LRB-

instead

of

-LRB-

R

c

W

-RRB-

-RRB-

is

useful

to

represent

a

configuration

Definition

A

processor-cluster

based

system

with

cluster

size

c

and

an

n-dimensional

intercluster

network

of

size

R

kn

with

channels

of

width

W

bytes

is

represented

as

-LRB-

kn

c

W

-RRB-

There

are

many

different

ways

to

factorize

R

into

n

such

factors

Factorization

with

nearly

equal

values

for

all

is

preferred

as

it

leads

to

smaller

network

organization

and

benefits

from

packaging

compactness

-LSB-

-RSB-

With

-LRB-

kn

k

-RRB-

the

notation

-LRB-

kn

c

W

-RRB-

can

be

further

shortened

to

-LRB-

kn

c

W

-RRB-

For

example

the

system

configuration

in

Fig.

-LRB-

a

-RRB-

can

be

represented

as

-LRB-

-RRB-

or

-LRB-

-RRB-

The

presented

analyses

in

this

section

can

be

easily

extended

to

other

topologies

as

well

Prior

to

analyzing

the

benefits

of

processor-clustering

we

briefly

discuss

two

main

determinants

of

communication

performance

in

parallel

systems

Characterizing

communication

performance

and

bottlenecks

The

communication

performance

of

a

parallel

system

is

determined

by

the

performance

of

its

subcomponents

Figure

shows

these

subcomponents

as

different

stages

in

the

process

of

a

message

transfer

from

a

source

processor

to

a

destination

processor

Typically

the

message

gets

initiated

at

the

sender

through

a

call

to

a

communication

library

function

The

message

is

then

prepared

and

injected

into

the

network

through

the

network

interface

The

network

interface

provides

to

and

fro

connectivity

between

a

processor

-LRB-

or

cluster

-RRB-

and

its

network

router

The

message

traverses

across

the

network

to

the

destination

processor

where

it

is

consumed

through

the

network

interface

The

destination

processor

acquires

the

message

through

a

call

to

another

communication

function

The

overall

performance

of

the

system

is

clearly

determined

by

the

sub-component

with

lowest

bandwidth

ProcessorSource

ProcessorDestination

intermediate

routers

destinationrouter

communicationmessaging

Interconnection

Network

source

router

Networkinterface

Networkinterface

library

calloverheads

communicationmessaging

library

calloverheads

Figure

The

process

of

a

message

transfer

from

a

source

processor

to

a

destination

processor

showing

the

various

sub-components

The

communication

performance

of

a

parallel

system

is

often

characterized

by

its

average

message

latency

versus

sustained

message

throughput

under

uniform

traffic

Such

latencythroughput

characteristic

is

used

to

derive

the

maximum

sustained

throughput

-LRB-

point

close

to

saturation

-RRB-

of

a

system

We

represent

the

maximum

sustained

throughput

as

messages

sec

Uniform

traffic

is

considered

here

as

it

is

more

representative

of

general

traffic

in

a

parallel

system

-LSB-

-RSB-

In

the

following

subsections

we

use

as

a

metric

to

analyze

the

communication

performance

of

a

parallel

system

We

first

consider

the

overheads

of

messaging

In

-LSB-

-RSB-

it

was

demonstrated

that

under

uniform

traffic

the

network

bandwidth

is

a

stronger

restriction

on

performance

than

network

interface

In

this

paper

we

assume

the

cluster

module

technology

offering

reasonable

bandwidth

at

the

interface

for

it

not

to

be

a

bottleneck

Thus

we

focus

only

on

the

network

bandwidth

and

messaging

overheads

The

derived

observations

are

validated

in

later

sections

using

simulation

and

experimental

studies

with

uniform

traffic

as

well

as

benchmark

applications

Messaging

Overheads

The

overheads

of

messaging

can

be

divided

into

two

parts

The

first

part

involves

a

software

latency

which

includes

the

overhead

of

procedure

calls

and

possibly

memory

copying

from

user

space

to

system

space

We

represent

such

overhead

as

ts

sec/message

Based

on

the

study

presented

by

Dongarra

and

Dunigan

in

-LSB-

-RSB-

representative

values

of

ts

on

current

machines

are

shown

in

Table

On

most

systems

ts

can

be

observed

to

vary

from

a

few

hundreds

of

cycles

up

to

thousands

of

cycles

Even

recent

machines

like

the

IBM

and

Convex

SPP

have

significantly

high

ts

The

value

of

ts

on

a

given

system

can

vary

with

the

communication

library

Libraries

offering

higher

degree

of

functionality

can

have

high

ts

For

example

on

the

Cray

communication

with

PVM

requires

ts

sec/message

Similar

communication

with

Shared

Memory

Library

-LRB-

sm

-RRB-

requires

a

much

lower

ts

sec/message

Although

the

message

startup

overhead

depends

on

the

communication

library

we

observe

that

even

for

faster

libraries

the

overhead

is

significantly

high

in

contemporary

systems

This

trend

is

expected

to

continue

in

the

near

future

Table

Message

Startup

Overhead

-LRB-

ts

-RRB-

on

Some

Recent

Systems

Machine

OS

ts

Clock

Cycles

-LRB-

sec/message

-RRB-

nCUBE

Vertex

Intel

Paragon

OSF

IBM

MPL

IBM

MPI

Cray

-LRB-

sm

-RRB-

MAX

Cray

-LRB-

PVM

-RRB-

Convex

-LRB-

sm

-RRB-

SPP-UX

Convex

-LRB-

PVM

-RRB-

The

second

part

of

messaging

overhead

is

incurred

in

dividing

the

message

into

standard

sized

packets

accessing

memory

forming

packet

headers

checksum

computing

and

possibly

waiting

for

acknowledgments

to

ensure

reliable

delivery

Such

overhead

is

typically

proportional

to

the

length

of

the

message

Let

tp

sec/byte

denote

overhead

incurred

per

byte

of

a

message

The

parameter

tp

determines

the

asymptotic

message

injection

bandwidth

offered

by

a

communication

library

on

a

given

system

Table

provides

representative

values

of

tp

obtained

on

different

machines

and

messaging

platforms

-LSB-

-RSB-

It

can

be

observed

that

on

most

systems

the

asymptotic

messaging

bandwidth

achieved

is

significantly

lower

than

the

theoretical

channel

maximum

indicating

relatively

high

messaging

overheads

on

such

systems

As

with

ts

the

parameter

tp

also

depends

on

the

underlying

communication

mechanism/protocol

Table

Achievable

Asymptotic

Messaging

Bandwidth

Per

Processor

and

Transmission

Overhead

Per

Byte

tp

Machine

Asymptotic

tp

Theoretical

Bandwidth

sec/byte

Link

Bandwidth

-LRB-

MB/sec

-RRB-

-LRB-

MB/sec

-RRB-

nCUBE

Intel

Paragon

IBM

IBM

Cray

-LRB-

sm

-RRB-

Cray

-LRB-

PVM

-RRB-

Convex

-LRB-

sm

-RRB-

Convex

-LRB-

PVM

-RRB-

Based

on

the

above

discussion

the

overheads

for

a

message

of

average

length

L

denoted

by

T

is

the

sum

of

the

overheads

incurred

in

the

two

parts

T

ts

tpL

-LRB-

sec

-RRB-

-LRB-

-RRB-

This

limits

the

maximum

message

handling

rate

from

a

given

processor

denoted

by

sw

messages

sec

Assuming

a

processor

can

not

initiate

the

next

message

before

the

previous

message

is

fully

transmitted

into

the

network

sw

is

obtained

as

T

Let

us

consider

a

more

optimistic

non-blocking

communication

model

in

which

a

processor

continues

program

execution

after

the

first

part

of

messaging

overhead

-LRB-

ts

-RRB-

is

incurred

This

allows

the

initiation

of

another

message

by

the

processor

while

the

previous

one

is

still

in

its

second

part

It

assumes

the

existence

of

hardware

capability

to

inject

a

message

independent

of

processor

support

capability

that

is

commonly

available

in

current

parallel

systems

Thus

the

maximum

rate

of

message

handling

in

this

model

is

only

limited

by

the

larger

overhead

and

is

obtained

as

sw

max

-LRB-

ts

tpL

-RRB-

messages

sec

-LRB-

-RRB-

The

above

expression

leads

us

to

the

following

observation

Observation

The

value

of

the

parameter

sw

is

independent

of

the

system

configuration

Thus

different

system

configurations

with

similar

messaging

overheads

offer

same

sw

Network

bisection

Besides

messaging

overheads

the

other

main

determinant

of

communication

performance

is

the

network

bisection

The

bisection

size

of

a

network

is

the

minimum

number

of

network

channels

to

be

cut

so

as

to

divide

the

network

into

two

equal

halves

-LSB-

-RSB-

Under

uniform

traffic

on

the

average

half

of

the

generated

messages

in

the

system

need

to

cross

the

inter-cluster

network

bisection

Let

the

maximum

rate

of

injection

of

messages

per

processor

imposed

by

the

bisection

size

assuming

no

messaging

overhead

-LRB-

ts

tp

-RRB-

be

denoted

by

nw

messages

sec

Based

on

a

contention

model

presented

in

-LSB-

-RSB-

the

limit

nw

for

a

-LRB-

kn

c

W

-RRB-

system

with

currently

popular

wormhole

routing

in

inter-cluster

network

can

be

analytically

derived

as

nw

cL

-LRB-

k

-RRB-

tc

-LRB-

-LRB-

n

-RRB-

-RRB-

messages

sec

-LRB-

-RRB-

where

tc

-LRB-

in

sec

-RRB-

denotes

the

channel

cycle

time

in

the

network

n

is

the

dimensionality

of

the

inter-cluster

network

and

L

-LRB-

in

bytes

-RRB-

denotes

the

average

message

length

in

the

system

Details

of

the

derivation

of

the

above

expression

and

its

validation

are

presented

in

the

Appendix

Equation

leads

us

to

the

following

observations

Observation

Given

a

-LRB-

kn

c

W

-RRB-

system

the

impact

of

varying

selected

system

parameters

on

the

maximum

throughput

supportable

per

processor

by

network

bisection

-LRB-

nw

-RRB-

can

be

summarized

as

follows

System

Variation

Effect

Parameters

on

nw

k

c

W

An

entry

in

the

table

or

corresponds

to

the

respective

system

parameter

being

increased

or

decreased

respectively

and

its

resultant

impact

on

nw

Similarly

relationships

obtained

by

reversing

the

sense

of

all

arrows

in

the

above

table

also

hold

true

It

can

be

noted

that

the

value

of

nw

may

be

maintained

by

increasing

c

while

decreasing

k

or

vice-versa

Similarly

nw

can

also

be

maintained

by

increasing

c

and/or

k

while

compensating

the

resultant

loss

in

nw

by

increasing

W

Determining

communication

performance

interplay

of

above

constraints

Let

us

derive

the

overall

impact

of

messaging

overhead

-LRB-

sw

-RRB-

and

network

bisection

-LRB-

nw

-RRB-

on

the

communication

performance

-LRB->=-RRB-

of

a

parallel

system

The

maximum

achievable

throughput

per

processor

in

a

system

is

determined

by

the

smaller

of

the

two

parameters

leading

to

min

-LRB-

nw

sw

-RRB-

-LRB-

-RRB-

As

discussed

earlier

in

this

study

we

do

not

propose

new

messaging

protocols

and

hardware

to

reduce

sw

Given

a

messaging

overhead

-LRB-

sw

-RRB-

we

demonstrate

on

how

to

exploit

emerging

cluster

and

interconnect

technologies

for

achieving

the

best

performance

of

sw

with

more

costeffective

designs

We

used

simulation

modeling

to

verify

the

interplay

of

nw

and

sw

on

system

performance

as

predicted

by

Eqn

We

describe

the

experimental

set-up

briefly

before

proceeding

to

discuss

the

results

Design

guidelines

presented

in

later

sections

are

also

similarly

validated

using

experimentation

on

a

variety

of

system

configurations

using

different

traffic

patterns

and

applications

Experimentation

and

simulation

setup

Real

systems

have

fixed

cluster

size

inter-cluster

channel

width

and

other

architectural

parameters

Thus

to

study

the

variation

of

such

parameters

we

conducted

experiments

on

the

Ohio

State

clustered

architecture

simulation

testbed

ClusterSim

ClusterSim

-LSB-

-RSB-

is

an

event-driven

simulation

testbed

developed

using

CSIM

-LSB-

-RSB-

offering

flexibility

to

analyze

the

impact

of

varying

system

parameters

like

cluster

size

topology

channel

width

network

channel

cycle

time

and

messaging

overheads

on

system

performance

ClusterSim

accurately

models

flit-level

transfer

corresponding

to

wormhole/cut-through

routing

as

in

real

systems

like

the

Paragon

and

Cray

The

following

contemporary

timing

parameters

were

assumed

for

the

network

channels

and

routers

network

channel

cycle

time

of

routing

time

at

a

node

of

and

switching

time

across

a

router

crossbar

of

Network

latency

vs.

throughput

result

plots

were

derived

with

uniform

traffic

for

different

system

configurations

In

this

section

we

only

present

simulation

results

to

validate

Eqn

Further

results

from

the

simulation

testbed

and

experimentation

on

the

to

validate

overall

design

guidelines

are

presented

in

Sec.

Validation

of

communication

performance

predictions

Figure

shows

network

latency

versus

throughput

plots

for

a

-LRB-

-RRB-

system

obtained

with

different

values

of

ts

and

sec/message

and

tp

sec/byte

The

message

length

was

fixed

at

L

bits

For

such

value

of

L

it

can

be

shown

using

Eqn

that

sw

ss

ts

Thus

the

above

values

of

ts

correspond

to

sw

and

messages

sec

respectively

The

value

of

nw

in

this

system

derived

as

the

maximum

achievable

network

throughput

corresponding

to

ts

sec

was

messages

sec

From

Fig.

-LRB-

a

-RRB-

it

can

be

observed

that

for

a

low

value

of

ts

corresponding

to

sw

the

maximum

throughput

in

the

system

is

determined

as

nw

messages

sec

However

for

a

higher

value

of

ts

corresponding

to

sw

messages

sec

the

maximum

performance

in

the

system

is

throttled

at

sw

thus

not

achieving

the

full

network

potential

of

nw

messages

sec

Similar

trends

were

observed

with

other

higher

values

of

ts

and

With

a

-LRB-

-RRB-

system

the

maximal

achievable

network

throughput

nw

is

only

messages

sec

corresponding

to

ts

As

shown

in

Fig.

-LRB-

b

-RRB-

in

such

a

system

having

ts

or

lower

corresponding

to

sw

messages

sec

or

higher

allows

the

maximal

possible

system

performance

-LRB-

nw

-RRB-

to

be

achieved

These

observations

validate

Eqn

Characterizing

parallel

systems

based

on

communication

performance

In

systems

having

sw

nw

the

network

bisection

is

used

effectively

-LRB-

nw

-RRB-

while

the

message

processing

capacity

at

processors

remains

under-utilized

-LRB-

sw

-RRB-

Thus

such

system

configurations

are

network

bisection

constrained

Similarly

in

systems

having

high

messaging

overheads

with

sw

nw

the

throughput

is

throttled

by

messaging

overheads

-LRB-

sw

-RRB-

causing

the

network

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

us

us

us

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

us

us

us

us

Figure

Latency

vs.

throughput

plots

with

different

message

overhead

costs

-LRB-

ts

-RRB-

Results

are

shown

for

two

different

system

a

-RRB-

-LRB-

-RRB-

and

b

-RRB-

-LRB-

-RRB-

The

maximum

throughput

corresponding

to

ts

sec

depicts

the

value

of

nw

and

nw

messages

sec

respectively

in

the

two

systems

Interplay

between

nw

and

sw

leads

to

observed

value

of

resources

to

remain

under-utilized

-LRB-

nw

-RRB-

Such

system

configurations

are

therefore

messaging

overhead

constrained

Finally

systems

with

-LRB-

sw

ss

nw

-RRB-

have

balanced

message

processing

capacity

and

network

bisection

In

such

a

system

configuration

both

resources

are

effectively

utilized

This

leads

us

to

the

following

characterization

of

parallel

systems

Definition

A

parallel

system

can

be

classified

as

ffl

network

bisection

constrained

-LRB-

NC

-RRB-

if

-LRB-

nw

sw

-RRB-

ffl

messaging

overhead

constrained

-LRB-

MC

-RRB-

if

-LRB-

sw

nw

-RRB-

ffl

balanced

-LRB-

B

-RRB-

if

-LRB-

sw

ss

nw

-RRB-

In

the

DSG

in

Fig.

we

analyzed

the

design

space

of

configurations

in

building

parallel

systems

Let

the

parameter

nw

for

a

-LRB-

kn

c

W

-RRB-

configuration

be

derived

based

on

Eqn

For

a

given

set

of

messaging

overheads

ts

and

tp

the

value

of

sw

is

similarly

derived

based

on

Eqn

As

derived

in

Observation

the

value

of

sw

remains

same

across

all

configurations

By

comparing

the

values

of

nw

and

sw

a

configuration

can

be

classified

as

NC

MC

or

B

The

following

theorem

analyzes

a

property

on

such

classification

Theorem

Given

a

balanced

-LRB-

B

-RRB-

configuration

-LRB-

R

c

W

-RRB-

another

configuration

having

the

same

network

-LRB-

R

-RRB-

and

channel

width

-LRB-

W

-RRB-

may

be

characterized

as

follows

configuration

with

smaller

cluster

size

c

is

message

overhead

constrained

-LRB-

MC

-RRB-

configuration

with

larger

cluster

size

c

is

network

bisection

constrained

-LRB-

NC

-RRB-

Proof

Given

a

balanced

-LRB-

R

c

W

-RRB-

configuration

it

clearly

satisfies

-LRB-

sw

ss

nw

-RRB-

From

Eqn

we

observe

that

while

maintaining

the

inter-cluster

network

-LRB-

keeping

k

and

W

fixed

-RRB-

a

smaller

cluster

size

-LRB-

c

-RRB-

leads

to

a

higher

value

of

nw

and

vice-versa

The

value

of

sw

remaining

fixed

implies

that

a

configuration

with

smaller

cluster

size

has

sw

nw

making

it

message

overhead

constrained

-LRB-

MC

-RRB-

Similarly

a

configuration

with

larger

cluster

size

-LRB-

c

-RRB-

can

be

shown

to

have

sw

nw

making

it

network

bisection

constrained

-LRB-

NC

-RRB-

The

above

theorem

leads

us

to

divide

a

typical

R-c

plane

into

the

following

three

regions

B-boundary

MC-region

and

NC-region

The

B-boundary

is

derived

by

joining

the

set

of

balanced

configurations

by

a

curve

as

shown

in

Fig.

-LRB-

a

-RRB-

By

Theorem

any

configuration

to

the

left

of

the

B-boundary

is

MC

Thus

the

configurations

enclosed

by

the

B-boundary

and

the

R

and

c

axes

are

MC

This

region

is

denoted

as

the

MC-region

Similarly

any

configuration

to

the

right

of

the

B-boundary

is

NC

Thus

the

open

part

of

the

R-c

plane

is

referred

to

as

the

NC-region

The

following

result

reflects

the

communication

performance

in

the

three

regions

respectively

Result

All

configurations

within

the

MC-region

and

on

the

B-boundary

have

the

same

maximal

throughput

performance

of

sw

Within

the

NC-region

maximal

throughput

performance

is

determined

by

nw

and

based

on

Eqn

this

throughput

falls

with

increasing

cluster

and

intercluster

size

The

impact

of

increasing

inter-cluster

channel

width

-LRB-

W

-RRB-

from

to

a

higher

value

is

shown

in

Fig.

-LRB-

b

-RRB-

Let

us

consider

a

balanced

configuration

-LRB-

R

c

W

-RRB-

on

the

B-boundary

for

W

W

From

Eqn

a

configuration

-LRB-

R

c

W

-RRB-

with

has

higher

nw

Thus

-LRB-

R

c

W

-RRB-

has

sw

nw

leading

us

to

conclude

it

to

be

MC

configuration

From

Theorem

a

balanced

configuration

with

W

has

cluster

size

c.

Consequently

with

increase

in

W

the

B-boundary

shifts

towards

the

right

-LRB-

outwards

-RRB-

resulting

in

the

MC-region

to

also

become

larger

A

system

with

a

given

number

of

processors

can

be

designed

using

different

configurations

Each

configuration

having

different

cost

and

communication

performance

characteristics

Depending

on

the

messaging

and

network

parameters

a

configuration

could

belong

to

the

MC-region

the

B-boundary

or

the

NC-region

Such

variation

in

characteristics

raises

an

interesting

question

for

designing

a

system

with

a

given

number

of

processors

which

configuration

delivers

best

performance

in

a

cost-effective

manner

This

question

is

addressed

in

the

next

section

where

we

analyze

the

impact

on

performance

and

cost

of

a

system

with

a

total

of

N

processors

as

larger

cluster

sizes

are

used

Number

of

routers

c

Processors

connected

to

each

router

-LRB-

processor-cluster

size

-RRB-

MC-region

NC-region

Number

of

routers

c

Processors

connected

to

each

router

-LRB-

processor-cluster

size

-RRB-

BBBBMC-region

NC-region

W

BBBBB

B-boundaryW

BBBB

W

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Figure

a

-RRB-

A

typical

R-c

plane

depicting

three

regions

into

which

parallel

system

configurations

fall

under

B-boundary

-LRB-

balanced

-RRB-

MC-region

-LRB-

messaging

overhead

constrained

-RRB-

and

NC-region

-LRB-

network-bisection

constrained

-RRB-

b

-RRB-

Impact

of

increasing

W

from

to

a

higher

value

W

the

B-boundary

shifts

outwards

resulting

in

a

larger

MC-region

Designing

a

system

with

a

given

number

of

processors

For

designing

a

system

with

N

processors

with

clusters

of

size

c

the

number

of

required

routers

is

given

by

R

N

c.

Higher

integration

is

leading

to

higher

value

of

c

and

as

a

result

smaller

R.

However

there

are

limits

on

what

technology

-LSB-

-RSB-

can

offer

at

a

given

time

For

example

the

maximum

allowable

cluster

size

can

be

limited

The

impact

of

such

constraints

on

the

design

is

studied

in

Sec.

In

this

section

we

present

a

more

general

analysis

without

restraining

the

study

Number

of

routers

Processors

connected

to

each

router

-LRB-

processor-cluster

size

-RRB-

NC-region

B-boundary

-LRB-

W

-RRB-

X

Y

Iso-processor

line

N

MC-region

Z

W

byte

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Configuration

Performance

-LRB-

l

=-RRB-

X

-LRB-

-RRB-

lsw

Y

-LRB-

-RRB-

lsw

Z

-LRB-

-RRB-

lnw

Figure

Three

configurations

on

an

iso-processor

line

for

N

on

an

R

c

plane

with

fixed

channel

width

-LRB-

W

byte

-RRB-

Figure

illustrates

three

different

configurations

X

-LRB-

-RRB-

Y

-LRB-

-RRB-

and

Z

-LRB-

-RRB-

on

an

iso-processor

line

on

the

R

c

plane

for

W

byte

All

three

configurations

can

be

used

to

build

a

system

with

N

processors

In

Fig.

configuration

X

lies

in

the

MC-region

Y

is

balanced

on

the

B-boundary

and

Z

lies

in

the

NC-region

From

Result

the

performances

-LRB->=-RRB-

at

configurations

X

Y

and

Z

are

sw

sw

and

nw

respectively

For

configurations

on

an

iso-processor

line

for

a

given

N

and

W

we

are

interested

in

deriving

the

following

The

balanced

configuration

at

which

network

and

messaging

performance

-LRB-

sw

-RRB-

match

each

other

For

configurations

using

cluster

size

larger

than

at

the

balanced

configuration

the

required

scaling

in

channel

width

to

maintain

performance

at

sw

Estimating

inter-cluster

interconnection

costs

for

configurations

as

larger

cluster

sizes

are

used

We

analyze

each

of

these

in

the

following

subsections

Deriving

the

balanced

configuration

for

a

given

channel

width

Let

cN

W

bal

denote

the

cluster

size

at

the

balanced

configuration

The

routers

-LRB-

R

N

cN

W

bal

-RRB-

in

the

system

are

assumed

to

be

organized

in

a

n-D

mesh

topology

with

radix

k

-LRB-

N

cN

W

bal

-RRB-

n.

For

such

configuration

equating

nw

ss

sw

obtained

from

Eqns

and

leads

us

to

max

-LRB-

ts

tpL

-RRB-

cN

W

bal

L

-LRB-

-LRB-

N

cN

W

bal

-RRB-

n

-RRB-

tc

-LRB-

-LRB-

-LRB-

n

-RRB-

-RRB-

-LRB-

-RRB-

For

n

this

simplifies

into

a

quadratic

equation

in

q

cN

W

bal

as

pN

q

cN

W

bal

cN

W

bal

W

max

-LRB-

ts

tpL

-RRB-

Ltc

-LRB-

-RRB-

Solutions

to

Eqn

which

round

off

to

positive

integers

are

proper

solutions

to

balanced

configurations

An

obtained

solution

should

also

lie

within

the

limits

of

offered

cluster

technology

Depending

on

the

values

of

timing

parameters

average

message

length

in

the

system

channel

width

and

system

size

the

derived

value

of

cN

W

bal

can

vary

over

a

wide

range

Consider

a

typical

example

with

ts

sec

tp

sec/byte

L

bytes

W

byte

tc

sec

and

N

The

two

derived

values

of

cN

W

bal

based

on

Eqn

are

and

Clearly

the

solution

is

too

large

to

be

reasonable

leaving

only

cN

W

bal

We

validated

this

example

experimentally

Three

configurations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

corresponding

to

X

Y

and

Z

in

Fig.

-LRB-

a

-RRB-

respectively

were

considered

Other

parameters

were

chosen

same

as

mentioned

above

Note

that

our

simulator

is

general

enough

to

model

networks

with

unequal

radixes

like

Figure

shows

the

network

latency

versus

throughput

plots

for

these

three

systems

Given

ts

sw

is

messages

sec

We

observe

such

sw

performance

to

be

delivered

by

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

but

not

by

the

-LRB-

-RRB-

configuration

This

indicates

-LRB-

-RRB-

to

be

the

balanced

configuration

with

cluster

size

of

two

leading

us

to

conclude

bal

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Uniform

Traffic

L

bits

us

Figure

Latency

vs.

throughput

plots

for

a

-RRB-

three

configurations

with

N

processors

and

channel

width

W

byte

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

and

b

-RRB-

-LRB-

-RRB-

configuration

with

wider

channel

width

-LRB-

W

bytes

-RRB-

Required

scaling

in

channel

width

with

larger

cluster

sizes

In

configurations

using

c

cN

W

bal

and

channel

width

W

it

can

be

observed

that

network

bisection

performance

becomes

a

bottleneck

leading

to

overall

performance

falling

below

sw

In

the

above

example

the

configuration

-LRB-

-RRB-

with

c

-LRB-

bal

-RRB-

delivered

less

throughput

than

desired

-LRB-

sw

-RRB-

Let

us

derive

the

minimum

increased

channel

width

W

to

maintain

performance

at

sw

in

such

configurations

while

using

clusters

of

size

c

The

network

performance

can

be

derived

using

Eqn

and

k

-LRB-

N

c

-RRB-

n

as

nw

cL

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

tc

-LRB-

-LRB-

n

-RRB-

-RRB-

Equating

this

to

sw

-LRB-

or

the

network

performance

at

the

balanced

configuration

in

Eqn

-RRB-

and

simplifying

we

obtain

W

cN

W

bal

-LRB-

-LRB-

N

cN

W

bal

-RRB-

n

-RRB-

W

c

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

This

leads

to

W

W

c

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

cN

W

bal

-LRB-

-LRB-

N

cN

W

bal

-RRB-

n

-RRB-

-LRB-

-RRB-

where

the

ceiling

function

ensures

byte-multiple

channel

widths

as

discussed

in

Sec.

Using

wider

channels

can

be

visualized

as

transitioning

from

one

Rc

plane

to

another

Rc

plane

with

higher

channel

width

as

depicted

in

gray

on

the

iso-processor

line

in

Fig.

Configuration

Z

with

W

byte

was

depicted

in

Fig.

as

being

in

the

NC-region

offering

nw

sw

It

is

shown

in

Fig.

that

this

configuration

with

W

bytes

moves

within

the

B-boundary

delivering

sw

This

was

verified

by

considering

a

corresponding

-LRB-

-RRB-

configuration

in

our

experiments

The

network

latency

versus

throughput

plot

for

this

system

is

depicted

in

Fig.

-LRB-

b

-RRB-

It

can

be

observed

that

it

delivers

maximal

throughput

matching

sw

messages

sec

Number

of

routers

Processors

connected

to

each

router

-LRB-

processor-cluster

size

-RRB-

NC-region

B-boundary

-LRB-

W

-RRB-

X

Y

Iso-processor

line

N

MC-region

Z

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

B-boundary

-LRB-

W

-RRB-

Configuration

Performance

-LRB-

l

=-RRB-

Z

-LRB-

-RRB-

lsw

transition

to

higher

W

Figure

Increased

channel

width

in

the

Z

configuration

-LRB-

from

W

to

W

-RRB-

leading

to

configuration

Z

becoming

balanced

and

offering

sw

Inter-cluster

interconnection

cost

The

cost

of

the

inter-cluster

interconnection

network

comprises

of

router

and

link

costs

which

depend

on

the

channel

width

of

the

system

We

are

interested

in

comparing

interconnection

costs

across

different

configurations

Such

cost

comparison

is

considered

easier

across

system

configurations

having

same

channel

width

With

same

channel

width

the

per-router

and

link

costs

are

expected

to

be

same

thus

a

system

requiring

fewer

routers

and

interconnection

links

is

expected

to

have

lower

expensive

interconnects

Cost

comparisons

across

configurations

with

different

channel

widths

is

more

difficult

Configurations

with

wider

channel

width

need

routers

and

link

connectors

with

matching

wider

data

paths

inside

Such

routers

and

connectors

are

expected

to

have

higher

costs

Let

us

first

compute

the

total

number

of

links

denoted

earlier

as

NW

in

the

inter-cluster

interconnection

This

is

an

indication

of

connector/connectivity

cost

across

routers

-LSB-

-RSB-

Similarly

we

estimate

the

change

in

router

costs

Overall

the

following

analysis

supports

the

intuition

that

interconnection

costs

reduce

with

larger

cluster

sizes

-LRB-

and

possibly

wider

channels

-RRB-

while

maintaining

per-processor

communication

performance

Let

us

derive

the

expression

for

NW

in

a

-LRB-

R

c

W

-RRB-

design

configuration

with

N

Rc

and

R

kn

where

k

-LRB-

N

c

-RRB-

n

and

the

kn

routers

interconnected

as

a

n-dimensional

mesh

Each

row

along

any

dimension

has

k

routers

with

-LRB-

k

-RRB-

interconnection

channels

For

bidirectional

channels

this

number

is

-LRB-

k

-RRB-

Given

n

dimensions

each

having

kn

rows

the

total

rows

in

the

network

is

nkn

leading

to

a

total

of

-LRB-

k

-RRB-

network

channels

Each

channel

being

W

wires

wide

the

total

number

of

interconnection

wires

in

the

network

can

be

derived

as

NW

-LRB-

k

-RRB-

W

-LRB-

N

c

-RRB-

n

n

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

W

-LRB-

-RRB-

Let

us

consider

the

impact

on

total

inter-cluster

wiring

costs

-LRB-

NW

-RRB-

as

larger

cluster

sizes

-LRB-

and

possibly

wider

channels

as

indicated

in

Eqn

-RRB-

are

used

to

interconnect

a

system

with

N

processors

while

maintaining

performance

Using

Eqns

and

we

have

NW

-LRB-

N

c

-RRB-

n

n

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

W

if

c

cN

W

bal

-LRB-

N

c

-RRB-

n

n

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

W

c

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

cN

W

bal

-LRB-

-LRB-

N

cN

W

bal

-RRB-

n

-RRB-

if

c

cN

W

bal

-LRB-

-RRB-

It

can

be

shown

that

NW

is

a

monotonically

decreasing

function

of

c

proving

the

intuition

that

link

connector/connectivity

costs

reduce

with

larger

clusters

Similarly

a

router

supporting

wider

channels

requires

wider

data

paths

inside

and

higher

pinout

outside

Predicting

the

impact

of

such

increased

pinout

and

wider

paths

on

router

cost

is

not

always

easy

Do

we

model

the

impact

of

increased

channel

width

on

router

cost

as

sublinear

linear

or

super-linear

Market

forces

often

have

a

critical

role

to

play

in

determining

such

costs

Initial

costs

may

be

very

high

-LRB-

super-linear

-RRB-

but

with

increasing

demand

leading

to

cheaper

production

-LSB-

-RSB-

typical

costs

tend

to

become

sub-linear

A

linear

model

is

therefore

reasonable

and

considered

in

further

analysis

Let

Hn

W

denote

the

cost

of

a

router

with

n

ports

-LRB-

supporting

a

n-dimensional

mesh

-RRB-

each

W

bits

wide

For

a

linear

cost

model

we

derive

the

cost

of

a

router

offering

W

bit

wide

ports

as

Hn

W

-LRB-

W

W

-RRB-

Let

Htot

denote

the

total

router

costs

in

a

system

For

a

system

with

R

routers

each

having

n

ports

of

W

bits

wide

leads

to

Htot

RHn

W

As

larger

cluster

sizes

-LRB-

and

possibly

wider

channels

as

derived

in

Eqn

-RRB-

are

used

to

interconnect

a

system

with

N

processors

while

maintaining

performance

the

total

router

cost

becomes

Htot

-LRB-

N

c

-RRB-

Hn

W

if

c

cN

W

bal

-LRB-

N

c

-RRB-

Hn

W

W

c

-LRB-

-LRB-

N

c

-RRB-

n

-RRB-

cN

W

bal

-LRB-

-LRB-

N

cN

W

bal

-RRB-

n

-RRB-

if

c

cN

W

bal

-LRB-

-RRB-

It

can

be

shown

that

the

above

expression

for

Htot

is

a

monotonically

decreasing

function

of

c.

Equations

and

lead

us

to

the

following

result

Result

For

designing

a

system

with

a

given

number

of

processors

larger

processor-clusters

offer

effectiveness

in

terms

of

link

connectivity

and

router

costs

while

offering

equal

performance

Scaling

systems

to

support

larger

number

of

processors

In

the

last

section

we

investigated

the

design

space

for

building

a

system

with

a

given

number

of

processors

In

this

section

we

analyze

scaling

system

size

using

larger

cluster

sizes

The

scaling

of

a

system

can

be

represented

on

the

R

c

W

space

as

moving

from

a

configuration

on

one

iso-processor

plane

to

a

larger

configuration

on

a

higher

iso-processor

plane

Let

us

consider

an

example

of

scaling

a

configuration

Y

with

processors

to

a

larger

configuration

with

processors

as

shown

in

Fig.

-LRB-

a

-RRB-

In

this

illustration

four

possible

scaled

configurations

are

shown

and

The

respective

cluster

sizes

used

increase

in

these

four

configurations

From

discussion

in

the

last

section

maintaining

performance

at

sw

may

require

increasing

channel

width

with

increasing

cluster

size

However

from

Result

link

connectivity

and

router

costs

are

expected

to

decrease

with

larger

clusters

Thus

it

is

reasonable

to

expect

that

given

an

existing

system

Y

using

clusters

of

size

a

scaled

system

with

similar

or

better

available

cluster

technology

need

to

use

c

Number

of

routers

Processors

connected

to

each

router

-LRB-

processor-cluster

size

c

-RRB-

YIso-processor

line

N

-LRB-

-RRB-

Iso-processor

line

N

scaling

approachusing

larger

clusters

Nbase

N

N

R

WbalN

-LRB-

a

-RRB-

-LRB-

b

-RRB-

c

W

B-boundary

MC

NC

Figure

a

-RRB-

Different

options

to

scale

a

configuration

Y

with

processors

to

a

larger

size

of

processors

b

-RRB-

Scaling

with

same

topology

but

larger

clusters

-LRB-

Y

to

option

-RRB-

In

Fig.

the

scaling

from

Y

to

depicts

an

approach

using

smaller

cluster

size

with

larger

network

Based

on

the

above

discussion

this

approach

is

not

attractive

from

link

and

router

cost

considerations

It

can

be

profitable

if

smaller

cluster

technology

offers

a

significant

cost

advantage

over

larger

clusters

so

as

to

offset

the

savings

in

link

and

router

costs

The

scaling

from

Y

to

is

the

default

approach

by

maintaining

cluster

size

while

interconnecting

more

clusters

Such

approach

is

useful

if

clusters

with

larger

number

of

processors

are

not

available

However

given

that

advancements

in

cluster

technology

are

offering

larger

cluster

modules

scaling

approaches

as

depicted

in

Y

to

and

using

larger

cluster

technology

are

becoming

more

attractive

In

this

section

we

study

one

such

approach

-LRB-

from

Y

to

-RRB-

in

more

detail

A

transition

from

to

can

be

achieved

by

the

principles

discussed

in

Sec.

The

approach

is

depicted

as

a

horizontal

dotted

arrow

along

the

c

axis

in

Fig.

-LRB-

a

-RRB-

Scaling

is

achieved

by

using

larger

clusters

while

maintaining

network

topology

Let

Nbase

denote

the

size

of

a

message-constrained

-LRB-

MC

-RRB-

base

configuration

with

R

routers

and

channel

width

W

Figure

-LRB-

b

-RRB-

shows

the

scaling

of

such

a

configuration

with

larger

clusters

With

larger

systems

the

network

is

more

utilized

leading

us

to

balanced

and

then

into

the

NC-region

Let

NR

W

bal

denote

the

size

of

the

balanced

configuration

Thus

scaling

up

to

a

size

of

N

NR

W

bal

can

be

achieved

without

increasing

W

Scaling

beyond

NR

W

bal

to

a

larger

size

N

requires

larger

W

to

maintain

performance

This

leads

us

to

analyze

the

following

subproblems

What

is

the

largest

system

size

achievable

-LRB-

NR

W

bal

-RRB-

on

a

given

network

topology

with

R

routers

and

channel

width

W

using

large

clusters

while

maintaining

per-processor

communication

performance

at

sw

With

further

scaling

using

larger

clusters

while

maintaining

network

topology

fixed

with

R

routers

and

per-processor

communication

performance

at

sw

what

is

the

minimal

required

increase

in

channel

width

Deriving

the

largest

system

size

supportable

on

a

given

inter-cluster

network

For

a

given

R

and

W

let

us

derive

the

value

of

NR

W

bal

Let

cR

W

bal

denote

the

cluster

size

at

such

balanced

configuration

Thus

cR

W

bal

NR

W

bal

R.

Let

us

consider

the

network

performance

as

derived

in

Eqn

Keeping

network

topology

fixed

during

scaling

implies

keeping

k

fixed

leading

to

nw

c.

For

such

scaling

let

us

consider

a

base

configuration

-LRB-

R

c

W

-RRB-

with

nw

sw

Let

f

denote

the

factor

nw

sw

This

indicates

a

base

configuration

with

under-utilized

network

bandwidth

-LRB-

and

therefore

in

MC-region

-RRB-

As

cluster

size

is

increased

from

c

the

per-processor

network

performance

-LRB-

nw

-RRB-

falls

proportional

to

the

increase

in

cluster

size

becoming

closer

to

sw

with

the

overall

network

bandwidth

being

utilized

more

effectively

At

cluster

size

-LRB-

cR

W

bal

cf

-RRB-

the

parameters

nw

and

sw

are

balanced

This

leads

us

to

Result

Given

a

base

system

-LRB-

R

c

W

-RRB-

with

nw

sw

and

f

nw

sw

designing

a

f

-LRB-

f

f

-RRB-

times

larger

system

delivering

similar

performance

requires

using

clusters

of

size

cf

while

keeping

the

inter-cluster

network

size

-LRB-

kn

-RRB-

and

channel

width

-LRB-

W

-RRB-

fixed

For

a

given

base

network

the

value

of

cR

W

bal

can

be

derived

using

Eqns

and

equating

these

expressions

and

solving

for

c.

Thus

cR

W

bal

is

the

solution

of

c

in

max

-LRB-

ts

tpL

-RRB-

cL

-LRB-

k

-RRB-

tc

-LRB-

-LRB-

n

-RRB-

-RRB-

or

cR

W

bal

max

-LRB-

ts

tpL

-RRB-

L

-LRB-

k

-RRB-

tc

-LRB-

-LRB-

n

-RRB-

-RRB-

We

verified

Result

through

simulation

modeling

of

a

fixed

network

interconnecting

processorclusters

of

increasing

size

Figure

shows

the

latency

versus

throughput

characteristics

of

-LRB-

c

-RRB-

configurations

with

c

and

The

plots

are

shown

for

two

different

message

overheads

ts

and

sec/message

-LRB-

corresponding

to

sw

and

messages

sec

-RRB-

Using

simulation

experiments

with

ts

the

value

of

nw

in

a

-LRB-

-RRB-

system

was

derived

as

message

sec

Under

ts

messages

sec

-LRB-

close

to

-RRB-

was

observed

on

a

-LRB-

-RRB-

system

The

maximal

cluster

size

supported

under

ts

without

degradation

in

ss

was

confirmed

to

be

c

nw

sw

ss

With

larger

cluster

sizes

-LRB-

c

-RRB-

the

achieved

value

of

dropped

significantly

Similarly

maximum

cluster

size

under

ts

-LRB-

sw

-RRB-

was

observed

as

-LRB-

from

the

expression

nw

sw

it

should

be

but

in

our

experiments

we

varied

cluster

size

as

powers

of

-RRB-

These

observations

validate

Result

Scaling

using

larger

clusters

while

maintaining

the

same

network

works

until

nw

sw

When

a

scaled

system

configuration

becomes

balanced

-LRB-

f

-RRB-

the

above

approach

is

no

longer

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

c

c

c

c

Figure

Determining

the

largest

system

size

that

can

be

supported

without

degradation

in

performance

on

a

given

base

network

by

using

larger

clusters

Results

are

shown

for

ts

and

sec/message

applicable

Further

scaling

leads

to

degradation

in

system

performance

Required

increase

in

channel

width

for

further

scaling

Let

us

now

consider

the

problem

of

scaling

a

-LRB-

R

c

W

-RRB-

base

system

which

is

not

message-constrained

-LRB-

sw

nw

-RRB-

using

larger

clusters

In

such

a

system

the

network

is

not

under-utilized

and

the

achieved

throughput

per

processor

is

derived

as

nw

In

scaled

configurations

it

is

desired

to

maintain

the

same

performance

In

scaling

a

-LRB-

R

c

W

-RRB-

system

by

a

factor

g

while

keeping

intercluster

topology

fixed

larger

clusters

with

cg

processors

are

used

From

Eqn

the

inter-cluster

channel

width

needs

to

be

proportionally

increased

to

W

g

to

maintain

nw

This

leads

us

to

Result

A

base

system

not

message

constrained

with

-LRB-

R

c

W

-RRB-

configuration

can

be

scaled

g

times

while

maintaining

the

average

throughput

in

the

system

by

using

-LRB-

R

cg

W

g

-RRB-

configuration

We

verified

that

system

performance

remains

unchanged

when

a

base

system

is

scaled

to

larger

sizes

using

the

approach

proposed

in

Result

We

first

considered

scaling

the

balanced

configuration

-LRB-

-RRB-

-LRB-

derived

in

the

above

subsection

with

ts

sec

-RRB-

two

times

to

a

-LRB-

-RRB-

configuration

Figure

compares

the

latency

versus

throughput

results

for

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

It

can

be

observed

that

the

scaled

configuration

offers

very

similar

throughput

thus

validating

Result

To

further

validate

Result

we

also

considered

scaling

a

network-constrained

base

system

with

-LRB-

-RRB-

configuration

up

to

large

cluster

sizes

-LRB-

c

-RRB-

To

ensure

the

base

system

being

indeed

network-constrained

we

chose

a

very

small

message

overhead

of

ts

sec

Figure

shows

the

latency

versus

throughput

results

for

the

-LRB-

-RRB-

configuration

and

the

corresponding

larger

configurations

-LRB-

-RRB-

It

is

observed

that

the

performance

is

maintained

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

-LRB-

-RRB-

-LRB-

-RRB-

Figure

Studying

the

scalability

of

a

balanced

configuration

-LRB-

-RRB-

using

larger

cluster

and

proportionately

larger

channel

width

A

ts

sec

was

assumed

in

larger

systems

resulting

in

ideal

scaling

Similar

results

were

obtained

with

other

configurations

like

-LRB-

-RRB-

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

Uniform

Traffic

L

bits

us

Figure

Studying

the

scalability

of

the

approach

of

using

larger

clusters

and

proportionately

larger

channel

widths

To

ensure

configurations

not

being

MC

-LRB-

i.e.

being

B

or

NC

-RRB-

a

very

small

ts

sec

was

assumed

Design

under

technological

constraints

In

earlier

sections

we

developed

design

guidelines

in

the

absence

of

technological

constraints

In

-LSB-

-RSB-

it

was

demonstrated

that

the

design

of

a

real

system

is

constrained

by

packaging

constraints

limiting

the

maximum

available

channel

width

-LRB-

Wmax

-RRB-

and

cluster

size

-LRB-

cmax

-RRB-

Among

the

various

solution

configurations

to

scale

a

given

system

only

those

having

-LRB-

W

Wmax

-RRB-

and

-LRB-

c

cmax

-RRB-

are

feasible

This

leads

to

the

following

two

sub-problems

Given

a

task

of

designing

a

system

with

N

processors

under

constraints

W

Wmax

and

c

cmax

which

is

the

most

cost-effective

configuration

offering

per-processor

communication

performance

of

ss

sw

For

given

network

topology

with

R

routers

what

is

the

largest

system

size

designable

under

above

constraints

while

offering

ss

sw

Deriving

cost-effective

configuration

for

a

given

total

number

of

processors

In

sec.

we

demonstrated

designing

cost-effective

systems

with

given

number

of

processors

using

balanced

configurations

with

larger

clusters

Let

us

consider

the

same

problem

under

the

constraints

of

W

Wmax

and

c

cmax

From

Result

we

know

that

larger

cluster

sizes

lead

to

more

costeffective

configurations

Thus

to

derive

the

cost-effective

configuration

our

objective

once

again

is

to

derive

the

balanced

configuration

offering

largest

cluster

size

under

the

given

constraints

Number

of

routers

Processors

connected

to

each

router

-LRB-

c

-RRB-

Wider

channels

push

the

Wmax

B-boundary

outwards

iso-processor

line

N

cmaxc

max

Number

of

routers

Processors

connected

to

each

router

-LRB-

c

-RRB-

cmaxc

max

scaling

keeping

same

R

-LRB-

a

-RRB-

-LRB-

b

-RRB-

Wmax

Wmax

cbalN

cbalN

Wmaxc

balN

cbalR

cbalR

Wmaxc

balR

Figure

Impact

of

constraints

cmax

and

Wmax

on

maximum

cluster

size

usable

for

a

-RRB-

designing

a

system

with

N

processors

where

cN

Wi

bal

denotes

balanced

cluster

size

with

channel

width

Wi

and

b

-RRB-

scaling

system

size

to

larger

configurations

keeping

topology

-LRB-

R

-RRB-

same

where

cR

Wi

bal

denotes

balanced

cluster

size

with

channel

width

Wi

Let

us

first

consider

the

impact

of

maximum

channel

width

constraint

Wmax

on

the

largest

cluster

size

From

the

analysis

in

Sec.

we

know

that

for

a

given

number

of

processors

N

increasing

channel

width

-LRB-

W

-RRB-

leads

to

the

corresponding

balanced

configuration

being

also

attained

at

a

larger

cluster

size

-LRB-

cN

bal

cN

bal

-RRB-

Figure

-LRB-

a

-RRB-

shows

such

an

impact

With

wider

channels

the

B-boundary

is

pushed

outwards

leading

to

the

balanced

configuration

along

an

iso-processor

line

shifting

towards

larger

cluster

sizes

The

parameter

Wmax

can

limit

the

largest

balanced

cluster

size

-LRB-

cN

Wmax

bal

-RRB-

attainable

in

a

design

The

value

of

cN

Wmax

bal

is

derived

by

solving

for

cN

W

bal

in

Eqn

with

W

Wmax

This

maximum

cluster

size

supportable

by

channel

width

Wmax

is

compared

to

cmax

the

maximum

cluster

size

allowed

by

cluster-module

technology

Two

cases

are

possible

as

depicted

in

Fig.

-LRB-

a

-RRB-

a

-RRB-

cmax

cN

Wmax

bal

and

b

-RRB-

cmax

cN

Wmax

bal

In

the

first

case

cluster

technology

limits

the

maximum

cluster

size

used

in

the

design

while

in

the

second

case

channel

width

limits

it

Thus

in

the

first

case

a

balanced

configuration

with

c

cmax

offers

the

best

configuration

In

the

other

case

a

balanced

configuration

with

c

cN

Wmax

bal

and

W

Wmax

is

the

best

Deriving

the

maximal

system

size

while

scaling

with

fixed

topology

and

larger

clusters

In

Sec.

we

demonstrated

the

approach

of

scaling

systems

using

larger

clusters

while

keeping

the

same

inter-cluster

network

topology

-LRB-

R

-RRB-

Such

an

approach

is

summarized

in

Fig.

-LRB-

b

-RRB-

With

channel

width

Wi

the

balanced

configuration

-LRB-

or

largest

system

size

offering

sw

-RRB-

is

attained

at

cluster

size

cR

Wi

bal

where

cR

Wi

bal

is

derivable

based

on

the

discussion

following

Result

To

derive

the

largest

system

size

attainable

under

the

constraints

we

first

determine

cR

Wmax

bal

the

largest

balanced

cluster

size

possible

with

W

Wmax

Two

cases

are

possible

as

shown

in

Fig.

-LRB-

b

-RRB-

a

-RRB-

cmax

cR

Wmax

bal

and

b

-RRB-

cmax

cR

Wmax

bal

In

the

first

case

with

cluster

technology

limiting

the

maximum

cluster

size

the

largest

system

size

is

Rcmax

In

the

second

case

channel

width

limiting

the

cluster

size

the

largest

system

size

is

RcR

Wmax

bal

Experimental

validation

pattern

and

application

based

approach

In

the

previous

sections

we

validated

the

theory

using

uniform

traffic

In

this

section

we

present

similar

results

derived

using

a

set

of

specific

traffic

patterns/applications

As

earlier

the

objective

is

to

demonstrate

that

the

architectural

configurations

derived

with

the

proposed

guidelines

using

larger

clusters

maintain

system

performance

while

being

more

cost-effective

Our

methodology

involves

executing

traffic

patterns/applications

on

a

set

of

configurations

and

comparing

the

resultant

execution

times

Experimental

platform

Experiments

were

conducted

on

the

ClusterSim

simulator

as

described

in

Sec.

We

also

conducted

experiments

on

a

node

Cray

for

configurations

supportable

on

it

The

machine

is

a

MIMD

multiprocessor

machine

using

a

direct

communication

network

with

torus

topology

-LSB-

-RSB-

Two

alpha

processors

are

attached

to

each

node

-LRB-

cluster

-RRB-

of

the

torus

and

share

a

common

hardware

interface

to

the

network

Each

processor

has

a

private

memory

For

communication

across

processors

we

used

the

remote

memory

access

routines

shmem

put

-LRB-

-RRB-

and

shmem

get

-LRB-

-RRB-

offering

the

maximum

communication

bandwidth

-LSB-

-RSB-

Traffic

patterns/applications

We

selected

three

traffic

patterns/applications

-LSB-

-RSB-

Bit

Permute

Complement

exchanges

-LRB-

BPC

-RRB-

Fast

Fourier

Transform

-LRB-

FFT

-RRB-

and

LU

matrix

decomposition

These

are

briefly

discussed

below

BPC

-LRB-

Bit

permute

complement

-RRB-

A

communication

round

in

BPC

involves

a

message

being

sent

out

by

each

node

to

a

destination

decided

by

a

bit-permute-complement

operation

on

its

radix

node

number

representation

In

a

given

round

all

nodes

apply

the

same

bitpermute-complement

bit

operation

and

so

that

each

node

receives

exactly

one

message

in

any

round

BPC

is

an

important

benchmark

as

it

exercises

the

communications

of

a

computer

heavily

on

a

realistic

problem

Each

BPC

requires

O

-LRB-

N

-RRB-

number

of

messages

where

N

is

the

number

of

processors

in

the

system

Fast

Fourier

Transform

-LRB-

FFT

-RRB-

The

computation

of

the

Fast

Fourier

transforms

-LRB-

FFTs

-RRB-

is

the

cornerstone

of

many

scientific

applications

It

is

a

rigorous

test

of

long-distance

communication

performance

In

this

paper

we

use

the

Cooley

Tuckey

algorithm

to

compute

the

FFT

with

interleaved

data

allocation

An

FFT

requires

O

-LRB-

N

log

N

-RRB-

number

of

messages

While

executing

FFT

across

systems

with

varying

number

of

processors

we

scale

problem

size

linearly

with

N

Dense

LU

factorization

with

partial

pivoting

This

computation

appears

in

almost

any

numeric

application

Searching

for

a

pivot

is

basically

a

reduction

operation

within

one

column

of

the

processor

mesh

Exchange

of

pivot

rows

is

a

point-to-point

communication

Update

phase

requires

data

to

be

broadcast

along

rows

and

columns

of

the

processor

mesh

The

number

of

messages

required

in

LU

is

O

-LRB-

-RRB-

However

computation

being

O

-LRB-

-RRB-

dominates

communication

The

problem

size

is

scaled

linearly

with

N

when

running

on

a

system

with

more

processors

Designing

a

system

with

a

given

number

of

processors

Deriving

the

balanced

configuration

for

a

given

channel

width

We

evaluated

the

completion

times

of

LU

BPC

and

FFT

on

three

different

configurations

having

N

processors

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

A

messaging

overhead

of

ts

sec/message

closely

matching

that

on

the

Cray

was

used

The

top

row

of

Fig.

shows

three

sets

of

bar

charts

depicting

completion

times

of

LU

BPC

and

FFT

respectively

derived

on

the

ClusterSim

simulator

for

the

three

configurations

For

each

application

the

completion

times

presented

are

normalized

with

respect

to

the

smallest

of

the

timings

obtained

The

completion

time

of

LU

can

be

observed

to

be

similar

on

all

three

systems

BPC

also

yielded

similar

results

However

with

FFT

characterized

by

very

rigorous

long-distance

communication

the

small

network

bisection

bandwidth

in

the

-LRB-

-RRB-

configuration

becomes

a

bottleneck

This

is

reflected

by

a

increase

in

completion

time

over

the

-LRB-

-RRB-

configuration

These

results

lead

us

to

conclude

that

among

these

configurations

performance

can

be

maintained

only

up

to

cluster

size

of

implying

-LRB-

-RRB-

to

be

the

balanced

configuration

Similar

results

were

confirmed

on

a

Cray

system

on

which

only

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

are

supportable

The

-LRB-

-RRB-

configuration

was

attained

by

choosing

a

-LRB-

-RRB-

partition

of

the

system

and

running

the

application

on

exactly

one

processor

in

each

cluster

The

bottom

row

of

Fig.

shows

three

sets

of

bar

charts

depicting

completion

times

of

LU

BPC

and

FFT

respectively

It

can

be

observed

that

for

all

three

applications

the

achieved

performances

of

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

are

similar

This

verifies

the

simulation

results

and

leads

us

to

conclude

cluster

size

of

being

balanced

These

trends

also

validate

the

theory

established

in

Sec.

Normalized

Completion

Time

ClusterSim

LU

Normalized

Completion

Time

ClusterSim

BPC

Normalized

Completion

Time

ClusterSim

FFT

Normalized

Completion

Time

Cray

LU

Normalized

Completion

Time

Cray

BPC

Normalized

Completion

Time

Cray

FFT

Figure

Impact

of

using

larger

clusters

in

designing

systems

deriving

the

balanced

configuration

The

normalized

completion

times

are

depicted

for

LU

BPC

and

FFT

on

three

configurations

on

ClusterSim

and

two

configurations

on

the

Cray

On

ClusterSim

parameters

of

ts

sec/message

and

W

bytes

matching

those

on

the

are

assumed

Number

of

processors

is

kept

fixed

at

N

Required

scaling

in

channel

width

with

larger

clusters

In

Sec.

we

demonstrated

the

required

increase

in

channel

width

to

maintain

performance

beyond

the

balanced

configuration

In

the

previous

example

the

-LRB-

-RRB-

configuration

did

not

match

the

FFT

performance

on

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

A

-LRB-

-RRB-

configuration

with

increased

channel

width

W

-LRB-

derived

based

on

Eqn

-RRB-

can

be

shown

to

offer

matching

performance

Figure

compares

the

completion

times

of

FFT

on

the

-LRB-

-RRB-

configuration

with

that

on

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

derived

on

the

ClusterSim

simulator

with

ts

sec

It

can

be

observed

that

there

is

a

close

match

in

the

completion

times

on

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configuration

thus

confirming

the

theory

in

Sec.

Normalized

Completion

Time

ClusterSim

FFT

Figure

Matching

performance

with

wider

channels

beyond

the

balanced

configuration

The

normalized

completion

times

of

FFT

is

shown

on

three

configurations

Parameters

N

and

ts

sec

are

assumed

Scaling

systems

to

support

larger

number

of

processors

Deriving

the

largest

system

size

supportable

on

a

given

inter-cluster

network

In

Sec.

we

derived

that

with

high

messaging

overheads

a

network

can

support

larger

systems

with

larger

clusters

-LRB-

up

to

a

certain

point

-RRB-

while

maintaining

performance

This

was

verified

with

uniform

traffic

on

a

mesh

inter-cluster

network

with

channel

width

of

bytes

and

ts

sec

The

cluster

size

was

varied

from

to

processors

With

such

parameters

the

largest

system

size

was

derived

as

obtained

with

cluster

size

of

Similar

results

were

also

obtained

with

BPC

and

FFT

traffic

as

discussed

below

To

determine

the

execution

time

of

an

application

on

a

scaled

system

we

considered

problem

size

to

be

similarly

scaled

The

execution

time

of

the

scaled

problem

on

a

scaled

system

is

a

function

of

the

communication

structure/volume

of

the

problem

For

the

two

applications

BPC

and

FFT

we

use

communication

volume

as

a

metric

to

determine

the

expected

increase

in

completion

time

of

these

applications

on

larger

systems

For

example

the

communication

in

BPC

is

O

-LRB-

N

-RRB-

Thus

in

BPC

increasing

the

number

of

processors

and

application

size

by

a

given

factor

leads

to

similar

increase

in

the

volume

of

communication

Such

scaling

is

expected

to

lead

to

minimal

increase

in

the

completion

time

until

the

network

bisection

bandwidth

resources

are

under-utilized

However

beyond

a

certain

system

size

the

network

no

longer

has

under-utilized

bandwidth

and

the

time

required

is

expected

to

sharply

increase

Such

results

were

observed

for

BPC

as

shown

in

Fig.

The

completion

times

for

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

did

not

increase

substantially

over

the

-LRB-

-RRB-

configuration

However

for

the

-LRB-

-RRB-

configuration

the

completion

time

jumped

to

This

indicates

that

for

the

given

network

and

messaging

characteristics

scaling

is

effective

up

to

a

factor

of

This

matches

the

results

presented

in

Fig.

with

uniform

traffic

The

communication

traffic

in

FFT

is

more

complex

It

consists

of

log

-LRB-

N

-RRB-

rounds

with

each

round

requiring

O

-LRB-

N

-RRB-

messages

Thus

increasing

the

system

size

-LRB-

along

with

problem

size

-RRB-

leads

to

completion

time

increasing

by

at

least

the

number

of

extra

rounds

required

As

system

size

is

increased

each

round

becomes

more

communication

intensive

However

initially

when

the

network

is

under-utilized

such

increase

in

communication

translates

to

minimal

increase

in

execution

time

of

a

round

However

beyond

a

certain

system

size

the

network

no

longer

has

under-utilized

bandwidth

and

the

time

required

per

round

is

expected

to

sharply

increase

Such

results

were

observed

with

FFT

as

shown

in

Fig.

The

completion

times

for

the

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

were

derived

to

be

and

respectively

compared

to

the

completion

time

on

a

base

-LRB-

-RRB-

configuration

These

values

are

close

to

the

expected

increase

in

completion

times

of

and

respectively

However

for

the

-LRB-

-RRB-

configuration

the

completion

time

increases

sharply

to

-LRB-

expected

-RRB-

indicating

network

resources

becoming

a

severe

bottleneck

For

the

given

network

and

messaging

characteristics

scaling

is

effective

up

to

a

factor

of

thus

matching

the

results

derived

earlier

Normalized

Completion

Time

ClusterSim

BPC

Normalized

Completion

Time

ClusterSim

FFT

Figure

Impact

of

using

larger

clusters

in

scaling

a

base

system

having

under-utilized

network

bisection

resources

determining

largest

system

-LRB-

cluster

-RRB-

size

supportable

on

same

network

Normalized

completion

times

are

shown

for

BPC

and

FFT

on

a

-LRB-

-RRB-

base

system

and

its

scaled

counterparts

having

larger

clusters

The

parameters

of

ts

sec/message

and

W

bytes

are

assumed

Required

increase

in

channel

width

for

further

scaling

In

Sec

we

studied

the

approach

of

scaling

configurations

which

are

not

message

constrained

i.e.

do

not

have

under-utilized

network

resources

It

was

demonstrated

that

scaling

in

such

systems

could

be

achieved

in

a

cost-effective

manner

by

using

still

larger

clusters

with

wider

network

channels

This

was

achieved

while

maintaining

per-processor

performance

with

uniform

traffic

We

evaluated

the

same

approach

with

FFT

traffic

on

three

system

configurations

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

As

discussed

in

Sec.

a

low

messaging

overhead

of

ts

sec

was

used

in

this

experiment

to

ensure

that

the

base

system

-LRB-

-RRB-

is

not

message-constrained

-LRB-

it

is

networkconstrained

-RRB-

The

normalized

completion

times

are

shown

in

Fig.

As

discussed

earlier

the

communication

in

FFT

has

log

-LRB-

N

-RRB-

rounds

with

each

round

requiring

O

-LRB-

N

-RRB-

messages

The

increased

communication

per

round

is

expected

to

be

matched

by

increased

network

bisection

bandwidth

Thus

the

completion

time

is

expected

to

increase

proportional

to

the

number

of

rounds

required

O

-LRB-

log

N

-RRB-

These

trends

can

be

observed

in

Fig.

validating

the

results

obtained

earlier

in

Sec.

FFT

Figure

Impact

of

using

larger

clusters

in

scaling

a

base

system

having

no

under-utilized

network

bisection

resources

determining

required

increase

in

channel

width

Normalized

completion

times

are

shown

for

FFT

on

a

-LRB-

-RRB-

base

system

and

its

scaled

counterparts

having

larger

clusters

and

proportionately

wider

channels

The

parameter

ts

sec/message

is

assumed

Conclusion

In

this

paper

we

have

demonstrated

that

messaging

overheads

can

limit

interconnection

utilization

Even

with

new

messaging

protocols

such

overheads

have

been

shown

to

remain

high

compared

to

network

speeds

In

order

to

alleviate

the

impact

of

high

messaging

overheads

we

have

analyzed

an

alternative

of

exploiting

the

emerging

processor-cluster

technology

for

designing

high-performance

and

cost-effective

parallel

systems

Using

a

new

design-space-graph

framework

which

considers

processor-clustering

messaging

overheads

and

network

performance

in

an

integrated

manner

we

have

established

the

following

design

guidelines

With

messaging

overheads

constraining

performance

in

a

system

processor

clustering

can

be

used

to

build

a

-RRB-

an

equal-sized

system

with

a

smaller

network

or

b

-RRB-

a

larger

system

with

an

equal-sized

network

With

messaging

overheads

not

being

a

constraint

a

combination

of

processor

clustering

and

wider

channels

can

be

used

to

build

a

range

of

larger-sized

systems

The

design

guidelines

have

been

validated

through

a

large

set

of

simulation

experiments

as

well

as

experimentation

on

Cray

To

the

best

of

our

knowledge

this

is

the

first

formal

analysis

in

literature

that

demonstrates

when

and

how

processor-clusters

benefit

parallel

system

design

As

processor

clustering

is

beginning

to

be

used

in

some

parallel

systems

like

Cray

and

Intel

Paragon

the

results

derived

in

this

paper

provide

a

strong

foundation

for

architects

to

analyze

the

benefits

of

processor

clustering

References

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Oct

-LSB-

-RSB-

R.

H.

Arpaci

and

D.

E.

Culler

et

al.

Empirical

Evaluation

of

the

A

Compiler

Perspective

In

Proc

of

the

Int

Symp

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

ClusterSim

A

Simulator

Testbed

for

Wormhole-Routed

k-ary

ncube

cluster-c

Multiprocessor

Systems

Dept.

of

Computer

Science

The

Ohio

State

University

working

paper

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Clustered

Multiprocessor

Systems

under

Packaging

and

Technological

Advancements

To

appear

in

IEEE

Trans

on

Parallel

and

Distributed

Systems

currently

available

at

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Designing

Large

Hierarchical

Multiprocessor

Systems

under

Processor

Interconnection

and

Packaging

Advancements

In

Proc

of

the

Int

l

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

D.

Basak

D.

K.

Panda

and

M.

Banikazemi

Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

Technical

Report

The

Ohio

State

University

-LSB-

-RSB-

D.

Basak

D.

K.

Panda

and

M.

Banikazemi

Benefits

of

Processor

Clustering

in

Designing

Large

Parallel

Systems

When

and

How

In

Proc

of

the

Int

Parallel

Processing

Symposium

pages

-LCB-

Apr

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

W.

J.

Dally

Performance

Analysis

of

k-ary

n-cube

Interconnection

Networks

IEEE

Trans

on

Computers

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

On

Hierarchical

Hypercube

Multicomputer

Interconnection

Network

Design

Jour

of

Parallel

and

Distributed

Computing

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

J.

J.

Dongarra

and

T.

Dunigan

Message-Passing

Performance

of

Various

Computers

Available

at

Aug

-LSB-

-RSB-

T.

Eicken

D.

E.

Culler

S.

C.

Goldstein

and

K.

E.

Schauser

Active

Messages

a

Mechanism

for

Integrated

Communication

and

Computation

In

The

Annual

International

Symposium

on

Computer

Architecure

pages

-LCB-

May

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

N.

Boden

et

al.

Myrinet

A

Gigabit-per-Second

Local

Area

Network

In

IEEE

Micro

pages

-LCB-

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

In

Proc

of

the

Int

Parallel

Processing

Symposium

pages

-LCB-

Mar

-LSB-

-RSB-

S.

L.

Johnson

Issues

in

High-Performance

Computer

Networks

In

IEEE

Computer

Society

TCCA

Newsletter

pages

-LCB-

Summer-Fall

-LSB-

-RSB-

V.

Karamcheti

and

A.

A.

Chien

Software

Overhead

in

Messaging

Layers

Where

Does

the

Time

Go

In

Proc

of

the

Int

Conference

on

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pages

-LCB-

-LSB-

-RSB-

V.

Karamcheti

and

A.

A.

Chien

A

Comparison

of

Architectural

Support

for

Messaging

on

the

TMC

and

the

Cray

In

Proc

of

the

Int

Symp

on

Computer

Architecture

-LSB-

-RSB-

V.

Kumar

A

Grama

A.

Gupta

and

G.

Karypis

Introduction

to

Parallel

Computing

Design

and

Analysis

of

Algorithms

Benjamin-Cummings

Addison-Wesley

Publishing

Company

-LSB-

-RSB-

D.

K.

Panda

and

D.

Basak

Issues

in

Designing

Scalable

Systems

with

k-ary

n-cube

clusterc

Organization

In

Proc

of

the

First

International

Workshop

on

Parallel

Processing

India

pages

-LCB-

-LSB-

-RSB-

Y.

R.

Potlapalli

and

D.

P.

Agrawal

Enhancing

the

Performance

of

HMINs

using

Express

Links

In

Int

Conf

on

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

M.

T.

Raghunath

and

A.

Ranade

Designing

interconnection

networks

for

multi-level

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

-LSB-

-RSB-

H.D.

Schwetman

Using

CSIM

to

Model

Complex

Systems

In

Proceedings

of

the

Winter

Simulation

Conference

pages

-LCB-

-LSB-

-RSB-

Sequent

Computer

Systems

NUMA-Q

Available

at

-LSB-

-RSB-

A.

Varma

and

C.

S.

Raghavendra

Interconnection

Networks

for

Multiprocessors

and

Multicomputers

IEEE

Computer

Society

Appendix

Deriving

Maximum

Throughput

Per

Processor

from

Network

Bisection

Constraint

In

this

section

we

derive

the

expression

for

maximum

rate

of

message

throughput

per

processor

as

constrained

by

the

inter-cluster

bisection

bandwidth

of

a

given

system

In

-LSB-

-RSB-

Agrawal

presented

an

analytical

model

of

contention

in

a

wormhole-routed

k-ary

n-cube

system

In

-LSB-

-RSB-

we

extended

this

model

to

clustered

systems

with

-LRB-

kn

c

W

-RRB-

topology

The

expression

for

average

message

latency

-LRB-

T

-RRB-

versus

throughput

per

processor

-LRB->=-RRB-

as

derived

in

-LSB-

-RSB-

was

T

tccF

-LRB-

tccF

d

-RRB-

-LRB-

d

-RRB-

d

n

nd

F

-LRB-

-RRB-

where

T

denotes

the

average

message

latency

expressed

in

network

cycles

Network

cycle

is

the

time

to

send

a

flit

across

one

hop

in

the

system

The

parameter

d

represents

the

average

number

of

hops

taken

by

a

message

in

a

dimension

For

bidirectional

channels

this

value

is

-LRB-

k

-RRB-

The

message

size

expressed

in

flits

is

F

L

W

where

L

is

the

message

length

in

bits

and

W

the

inter-cluster

channel

width

The

parameter

is

the

message

injection

rate

by

a

processor

in

terms

of

messages

sec

and

tc

denotes

the

network

cycle

time

in

microseconds

From

the

above

equation

it

can

be

observed

that

nd

F

cycles

is

the

minimum

latency

suffered

by

a

message

This

happens

at

a

very

small

value

of

message

injection

rate

Equation

can

be

rewritten

to

express

the

throughput

per

processor

as

a

function

of

T

and

other

parameters

as

-LRB-

T

F

nd

-RRB-

d

cF

dtc

-LRB-

-LRB-

n

-RRB-

-LRB-

d

-RRB-

F

-LRB-

T

F

nd

-RRB-

d

-RRB-

messages/cycle

-LRB-

-RRB-

We

identified

a

reasonable

maximal

operational

point

on

this

latency

versus

throughput

plot

by

assuming

the

maximum

average

latency

to

be

bounded

by

a

multiple

of

the

minimal

average

latency

-LRB-

T

-RRB-

Let

f

denote

such

a

multiple

Typical

values

of

f

are

chosen

between

and

Without

loss

of

generality

in

the

following

discussion

we

choose

f

to

be

Let

nw

denote

the

throughput

at

the

point

maximum

average

latency

is

attained

This

point

is

identified

using

Eqn

as

nw

-LRB-

F

nd

-RRB-

d

cF

dtc

-LRB-

-LRB-

n

-RRB-

-LRB-

d

-RRB-

F

-LRB-

F

nd

-RRB-

d

-RRB-

messages/cycle

-LRB-

-RRB-

cF

dtc

-LRB-

-LRB-

n

-RRB-

-LRB-

d

-RRB-

F

-LRB-

F

nd

-RRB-

d

-RRB-

messages/cycle

-LRB-

-RRB-

-LRB-

-RRB-

For

reasonably

large

values

of

F

and

d

and

using

-LRB-

F

L

W

-RRB-

and

d

-LRB-

k

-RRB-

the

above

expression

can

be

further

simplified

to

-LRB-

as

stated

in

Eqn

-RRB-

nw

cL

-LRB-

k

-RRB-

tc

-LRB-

-LRB-

n

-RRB-

-RRB-

messages/cycle

-LRB-

-RRB-

Experimental

validation

of

above

analysis

We

conducted

experiments

with

ts

sec

to

determine

nw

supported

by

the

underlying

network

when

messaging

overhead

costs

are

negligible

Figure

-LRB-

a

-RRB-

presents

the

impact

of

increasing

cluster

size

from

c

to

while

the

inter-cluster

network

is

kept

fixed

at

mesh

with

W

bytes

The

latency

versus

throughput

results

derived

for

a

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

are

depicted

in

these

plots

Uniform

traffic

with

message

length

L

bits

was

used

As

cluster

size

is

increased

from

to

the

value

of

nw

falls

from

a

value

of

to

messages

sec

respectively

indicating

a

trend

of

nw

c.

Figure

-LRB-

b

-RRB-

presents

the

impact

of

increasing

inter-cluster

size

from

a

to

network

while

the

cluster

size

is

kept

fixed

at

c

and

inter-cluster

channel

width

at

W

bytes

The

latency

versus

throughput

results

are

depicted

for

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

configurations

Uniform

traffic

with

message

length

L

bits

was

used

As

the

maximum

radix

in

the

inter-cluster

topology

is

increased

from

kmax

to

the

value

of

nw

also

falls

from

a

value

of

to

messages

sec

respectively

indicating

a

trend

of

nw

kmax

Similarly

the

impact

of

increasing

message

length

on

nw

is

presented

in

Fig.

-LRB-

c

-RRB-

The

latency

versus

throughput

plots

depicted

for

a

-LRB-

-RRB-

configuration

with

increasing

message

length

of

and

bits

show

a

fall

in

nw

from

to

messages

sec

respectively

This

indicates

a

trend

of

nw

L.

Figure

-LRB-

d

-RRB-

presents

the

impact

of

increasing

the

intercluster

channel

width

from

to

bytes

for

a

inter-cluster

network

with

cluster

size

of

As

W

is

increased

from

to

the

value

of

nw

increases

from

to

messages

sec

respectively

indicating

a

trend

of

nw

W

Similar

trends

were

observed

with

higher

dimensional

inter-cluster

networks

with

varying

cluster

sizes

and

message

lengths

These

observations

together

validate

Eqn

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

-LRB-

a

-RRB-

-LRB-

-RRB-

L

bits

c

c

c

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

-LRB-

b

-RRB-

c

W

L

bits

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

-LRB-

c

-RRB-

-LRB-

-RRB-

L

bits

L

bits

L

bits

Average

Message

Latency

-LRB-

microsec

-RRB-

Throughput

-LRB-

msgs/microsec

-RRB-

-LRB-

d

-RRB-

-LRB-

W

-RRB-

L

W

bytes

W

bytes

W

bytes

Figure

Latency

vs.

throughput

plots

with

negligible

ts

isolating

messaging

overheads

to

demonstrate

the

impact

of

a

-RRB-

cluster

size

b

-RRB-

inter-cluster

network

radix

c

-RRB-

message

length

and

d

-RRB-

inter-cluster

channel

width

on

the

maximum

network

throughput

per

processor

-LRB-

nw

-RRB-

interconnection networks
messaging overheads
scalability

clustered architectures
hierarchical architectures
parallel architectures