Designing

Large

Hierarchical

Multiprocessor

Systems

under

Processor

Interconnection

and

Packaging

Advancements

Debashis

Basak

and

Dhabaleswar

K.

Panda

Department

of

Computer

and

Information

Science

Ohio

State

University

Columbus

OH

Tel

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

basak,panda@cis.ohio-state.edu

Abstract

In

this

paper

we

present

a

general

framework

for

architectural

design

of

large

hierarchical

multiprocessor

systems

under

rapidly

changing

packaging

processor

and

interconnection

technologies

Processor

boards

with

larger

area

-LRB-

A

-RRB-

and

greater

pinouts

are

becoming

feasible

Board

interconnection

technology

has

advanced

from

only

peripheral

connections

O

-LRB-

pA

-RRB-

to

elastomeric

surface

connections

O

-LRB-

A

-RRB-

As

processor

and

interconnection

technology

is

growing

there

is

a

varying

demand

on

the

interconnection

network

of

the

system

The

proposed

framework

is

capable

of

taking

into

account

all

these

changing

technologies

Each

technology

is

captured

through

one

or

more

parameter

-LRB-

s

-RRB-

which

reflects

its

level

of

advancement

Depending

on

a

given

set

of

values

of

these

parameters

the

framework

guides

us

to

choose

the

most

optimum

topology

The

framework

is

illustrated

by

considering

the

design

problem

of

the

currently

popular

k-ary

n-cube

cluster-c

scalable

architecture

These

architectures

combine

the

scalability

of

k-ary

n-cube

wormhole-routed

networks

with

the

cost-effectiveness

of

processor

cluster

designs

Each

cluster

consists

of

c

processors

interconnected

through

a

bus/MIN/direct

network

Our

results

indicate

that

under

surface

pinout

technology

an

increase

in

cluster

size

-LRB-

c

-RRB-

is

associated

with

a

growth

in

bisection

bandwidth

whereas

in

periphery

pinout

it

leads

to

a

fall

in

bisection

bandwidth

For

systems

supporting

links

cluster

sizes

of

to

processors

with

k-ary

n-cube

interconnections

are

optimal

Similarly

systems

with

links

can

support

larger

cluster

sizes

-LRB-

c

-RRB-

and

interconnections

systems

research

is

supported

by

the

National

Science

Foundation

Grant

Contents

Introduction

Architectural

model

and

alternatives

Deriving

design-feasible

configurations

Processor

board

technology

Two

pinout

technologies

Parameterized

representation

of

inter-cluster

bisection

bandwidth

and

channel

width

Processor

and

communication

link

speeds

Bisection

bandwidth

demand

in

each

hierarchy

Traffic

Patterns

Parameterizing

inter-cluster

bisection

bandwidth

B

Normalizing

demanded

bandwidth

w.r.t.

packaging

technology

Design

choices

Good

configurations

Choice

of

cluster

size

Choosing

the

best

configuration

Conclusions

and

Future

Work

Introduction

Designing

large

parallel

systems

to

provide

high

computing

bandwidth

is

an

issue

of

important

consequence

Rapid

developments

in

the

field

of

processor

memory

network

and

packaging

technologies

make

the

task

of

efficient

design

of

large

multiprocessor

systems

a

difficult

task

Proposed

design

guidelines

need

to

take

into

account

changes

in

these

technologies

to

provide

good

performance

One

basic

guideline

which

holds

good

universally

is

that

large

systems

need

to

have

a

hierarchical

packaging

This

is

naturally

imposed

by

the

packaging

constraints

like

board

size

pinouts

limited

cabinet

size

etc.

In

this

light

it

makes

sense

to

design

large

systems

hierarchically

so

that

there

is

close

relationship

between

the

design

and

the

actual

packaging

A

variety

of

two-level

hierarchical

configurations

have

been

proposed

by

researchers

in

the

past

to

build

scalable

systems

Though

a

larger

number

of

levels

might

be

more

general

it

is

commonly

believed

that

most

systems

in

the

near

future

will

fit

in

two

hierarchies

Moreover

the

techniques

which

work

for

two-level

hierarchies

can

be

easily

extended

to

design

more

levels

Examples

of

previous

work

in

this

area

include

two-level

systems

based

on

hypercube

and

other

network

topologies

-LSB-

-RSB-

MINs

and

n-hop

networks

-LSB-

-RSB-

twolevel

systems

with

k-ary

n-cube

topologies

-LSB-

-RSB-

and

combination

of

bus

and

mesh/hypercube

networks

-LSB-

-RSB-

However

most

of

these

analyses

either

did

not

consider

packaging

and

interconnecting

constraints

or

provided

guidelines

based

only

on

fixed

board

sizes

with

fixed

pinouts

not

considering

changes

in

board

sizes

and

alternate

pinout

technologies

The

changes

in

processor

and

interconnection

technology

and

its

relation

to

packaging

technology

in

designing

systems

was

also

not

studied

Some

of

the

works

suggested

guidelines

based

only

on

uniform

traffic

patterns

However

localized

traffic

patterns

are

more

common

in

distributed

shared

memory

-LRB-

DSM

-RRB-

systems

with

caches

-LSB-

-RSB-

or

cache-only

memories

-LSB-

-RSB-

Localized

traffic

patterns

put

lesser

load

on

the

higher

hierarchies

allowing

system

designs

with

low-bandwidth

networks

to

be

considered

which

may

have

been

rejected

outright

under

uniform

traffic

assumption

In

this

paper

we

present

a

general

framework

to

provide

design

guidelines

for

building

large

hierarchical

multiprocessor

systems

We

consider

a

packaging

technology

with

two

levels

processors

in

the

parallel

computer

are

organized

into

a

number

of

boards

-LRB-

clusters

-RRB-

and

the

boards

are

then

connected

through

inter-board

wires

Though

the

precise

characteristics

of

existing

and

emerging

packaging

technologies

are

hard

to

model

Raghunath

-LSB-

-RSB-

has

shown

that

a

common

characteristic

of

most

packaging

technologies

is

a

progressive

increase

in

costs

and

decrease

in

capacities

as

we

go

up

the

packaging

hierarchy

For

instance

it

is

clear

that

as

we

proceed

up

the

hierarchy

wires

usually

get

longer

with

larger

inter-wire

spacings

Metal

wires

on

VLSI

chips

may

be

about

micron

apart

whereas

wires

connecting

different

boards

may

be

as

long

as

meter

and

spacing

between

wires

on

a

cable

connector

may

be

as

much

as

mm

This

results

in

a

higher

cost

per

wire

at

upper

levels

of

the

hierarchy

and

makes

the

design

of

these

levels

more

critical

We

propose

a

new

framework

as

shown

in

Fig.

for

designing

and

developing

clus

tered

architectures

The

basic

goal

is

to

design

an

efficient

system

with

N

processors

with

Processor

board

Pinout

Applications

Traffic

patterns

uniform

localized

Packaging

Technology

Architecture

characterestics

technology

technology

Size

and

topology

of

each

level

Bisection

bandwidth

and

channel

width

in

each

hierarchy

Maximum

board

size

Pinout

density

Surface

technology

Periphery

technology

Bisection

bandwidth

demanded

by

processors

in

each

hierarchy

match

determines

relationship

Processor

Speeds

Interconnection

Technology

in

each

Maximum

channel

widths

Commmunication

Link

speeds

SUPPLY

DEMAND

System

size

number

of

hierarchies

hierarchy

Good

configurations

Best

Configuration

Design

Guideline

Framework

Simulation

-LRB-

if

multiple

good

choices

-RRB-

Design-Feasible

configurations

-LRB-

Supply

Demand

-RRB-

Figure

Underlying

framework

for

designing

and

developing

hierarchical

architectures

under

technological

constraints

two-hierarchies

Theoretically

a

large

number

of

alternate

configurations

are

possible

to

build

such

a

system

The

configurations

which

conform

to

the

packaging

restrictions

like

maximum

board

size

the

pinout

technology

being

used

peripheral

or

surface

and

the

pinout

densities

are

defined

as

design-feasible

configurations

It

is

clear

that

only

the

design-feasible

configurations

can

translate

into

real

machines

Each

of

the

design-feasible

configurations

offers

some

performance

depending

on

the

size

of

the

bisection

in

each

hierarchy

Referring

to

the

set

of

design-feasible

configurations

and

their

performance

characteristics

as

what

the

packaging

technology

offers

we

call

this

the

supply

side

On

the

other

hand

processors

make

demands

on

the

bisection

bandwidth

in

each

hierarchy

This

demand

depends

on

the

speed

of

processors

The

speed

of

the

communication

links

determine

the

size

of

the

bisection

required

to

handle

the

demanded

traffic

The

subset

of

design-feasible

configurations

which

offer

performance

greater

than

what

is

demanded

are

defined

as

good

configurations

The

final

step

is

to

select

a

best

configuration

from

amongst

the

good

configurations

This

is

done

on

the

basis

of

relative

performance

and

cost

of

these

systems

Simulation

modeling

may

be

employed

to

decide

among

very

close

candidates

The

paper

is

organized

as

follows

In

section

we

discuss

the

two-level

k-ary

ncube

cluster-c

architecture

-LSB-

-RSB-

used

in

this

paper

to

illustrate

the

working

of

the

design

framework

In

section

we

discuss

the

trends

in

growth

of

processor

board

sizes

alternate

pinout

technologies

channel

width

technology

and

derive

the

design-feasible

configurations

and

their

characterestics

Processor

speeds

communication

link

speeds

and

the

performance

demand

are

discussed

in

sec.

In

Section

we

show

how

to

derive

the

good

configurations

and

discuss

important

considerations

in

choosing

the

best

configuration

Finally

concluding

remarks

and

future

work

are

presented

Architectural

model

and

alternatives

In

this

section

we

discuss

the

class

of

k-ary

n-cube

cluster-c

systems

-LSB-

-RSB-

Such

architectures

are

currently

the

trend

in

building

large

scalable

systems

Examples

are

Intel

Paragon

-LSB-

-RSB-

Stanford

DASH

-LSB-

-RSB-

and

Cray

-LSB-

-RSB-

A

k-ary

n-cube

cluster-c

system

consists

of

two

levels

The

lower

level

consists

of

kn

clusters

of

processors

interconnected

by

a

higher

level

direct

k-ary

n-cube

network

-LRB-

also

referred

to

as

inter-cluster

network

or

internet

-RRB-

Each

cluster

consists

of

c

processors

interconnected

through

buses/MINs/direct

networks

The

significance

of

such

systems

is

twofold

processor

clusters

allow

easy

exploitation

of

communication

locality

and

the

k-ary

n-cube

internet

provides

easy

scalability

to

the

system

Figure

shows

the

overall

configuration

while

Fig.

shows

an

example

when

clusters

are

specifically

bus-based

Intercluster

Network

Intra-cluster

Network

Clusters

ClusterInterface

-LRB-

CI

-RRB-

Figure

A

two-level

hierarchical

system

ClusterInterface

PN

PROCESSING

NODE

-LRB-

INCLUDES

A

PROCESSOR

AND

CACHE

-RRB-

k

ary

n-cube

CLUSTER

BUS

CLUSTER

INTER-CLUSTER

NETWORK

CLUSTER

BUS

CLUSTER

Figure

An

example

k-ary

n-cube

bus-c

architecture

made

up

of

bus-based

clusters

A

cluster

bus

connects

c

processing

nodes

and

a

cluster

interface

The

kn

cluster

interfaces

are

interconnected

through

a

k-ary

n-cube

network

Though

in

this

paper

we

work

with

k-ary

n-cube

internets

our

design

framework

is

more

general

and

can

work

for

other

internet

topologies

like

DeBruijn

graphs

and

n-hop

networks

-LSB-

-RSB-

Without

loss

of

generality

in

this

paper

we

emphasize

on

k-ary

n-cube

because

it

is

a

currently

popular

trend

which

we

believe

will

continue

in

the

near

future

also

Thus

the

insights

provided

by

our

analysis

in

this

paper

will

prove

useful

in

designing

and

packaging

future

machines

which

are

based

on

the

k-ary

n-cube

cluster-c

model

Now

let

us

consider

designing

a

system

with

N

processors

There

are

many

different

alternatives

to

form

a

clustered

configuration

There

are

basically

two

degrees

of

freedom

A

-RRB-

Size

of

levels

and

B

-RRB-

Topology

choice

in

each

level

A

-RRB-

Size

of

levels

Consider

N

to

be

One

can

design

this

system

with

clusters

of

processors

each

clusters

of

processors

each

and

so

on

Note

that

for

a

given

system

size

fixing

the

size

of

one

level

automatically

determines

the

size

of

the

other

level

A

processor

cluster

may

be

accommodated

on

a

single

board

-LRB-

referred

to

as

physical

clustering

-RRB-

or

it

may

span

multiple

boards

-LRB-

referred

to

as

logical

clustering

-RRB-

In

the

rest

of

the

paper

we

use

clustering

to

imply

physical

clustering

B

-RRB-

Topology

choices

Having

fixed

the

size

of

each

level

we

still

have

the

freedom

to

vary

the

topology

in

each

level

For

example

in

a

system

with

clusters

with

processors

each

one

can

have

a

internet

with

bus-based

clusters

internet

with

MIN-based

clusters

and

so

on

Each

alternative

leads

to

a

different

clustered

architecture

with

different

properties

in

terms

of

message

latency

message

throughput

synchronization

costs

etc.

Our

objective

is

to

select

the

configuration

that

offers

best

system

performance

Deriving

design-feasible

configurations

For

a

design

alternative

of

a

given

system

size

the

specific

architectural

characteristics

like

bisection

bandwidth

channel

widths

etc.

can

only

be

precisely

stated

with

respect

to

the

packaging

technology

in

which

it

is

molded

We

first

discuss

current

trends

in

board

and

pinout

technologies

Then

we

present

a

general

scheme

in

which

architectural

characteristics

are

parameterized

with

respect

to

the

board

and

pinout

technology

Finally

we

discuss

the

general

trends

and

constraints

on

the

interconnection

channel

widths

which

impose

restrictions

on

the

design

space

Processor

board

technology

Processor

boards

can

not

be

arbitrarily

large

in

size

Their

size

is

restricted

by

electrical

mechanical

and

board

fabrication

constraints

In

our

system

we

assume

each

board

to

contain

c

processors

chips

intra-cluster

interconnection

network

local

memories

and

the

cluster

interface

-LRB-

consisting

of

part

of

the

internet

router

logic

and

cache-coherence

logic

if

it

is

a

DSM

system

-LSB-

-RSB-

-RRB-

The

maximum

size

of

processor

board

restricts

the

number

of

processors

-LRB-

c

-RRB-

that

can

be

put

on

a

board

and

imposes

a

natural

limit

on

the

pinout

It

has

been

shown

in

-LSB-

-RSB-

that

the

area

of

processor

and

memory

chips

dominate

the

space

requirements

Let

us

define

the

area

required

by

a

processor

chip

its

local

memory

and

a

per-processor

fraction

of

the

network

and

other

logic

to

be

a

In

our

analysis

we

assume

this

area

to

be

organized

as

a

square

with

a

perimeter

of

Of

course

a

depends

on

the

amount

of

per-processor

memory

provided

in

the

system

In

the

ensuing

discussion

when

we

talk

about

a

processor

area

a

we

include

in

it

a

reasonable

amount

of

per-processor

memory

We

use

a

as

an

unit

of

board

area

and

measure

board

sizes

in

terms

of

a

In

recent

years

board

sizes

have

grown

in

physical

dimensions

Progressive

improvements

in

the

processor-memory

integration

has

also

resulted

in

a

becoming

smaller

leading

to

more

processors

on

the

same

board

We

now

have

contemporary

prototypes

like

the

MIT

J-machine

-LSB-

-RSB-

which

has

processors

on

a

board

The

technology

at

a

given

time

determines

the

largest

board

size

-LRB-

m

a

-RRB-

which

can

fit

at

most

m

processors

The

maximum

board

size

determines

the

largest

cluster

size

in

our

system

design

Without

loss

of

generality

in

this

paper

we

assume

m

to

be

Given

a

board

size

-LRB-

j

a

-RRB-

with

-LRB-

j

m

-RRB-

we

can

accommodate

upto

j

processors

on

it

This

is

called

the

capacity

of

the

board

One

can

under-populate

a

board

by

placing

smaller

than

the

capacity

number

of

processors

on

it

A

direct

effect

of

under-populating

is

that

it

leads

to

wastage

in

precious

board

space

as

a

consequence

larger

number

of

boards

are

required

leading

to

greater

system

volume

The

advantage

of

under-populating

is

that

the

remaining

area

of

the

board

may

be

used

to

support

larger

number

of

inter-board

links

which

may

improve

system

performance

However

assuming

that

these

links

are

expensive

-LSB-

-RSB-

this

performance

gain

might

not

be

cost-worthy

We

therefore

choose

to

avoid

under-populating

and

develop

design

guidelines

assuming

a

board

area

is

packed

to

its

capacity

Thus

we

assume

that

for

building

a

system

with

N

processors

we

are

given

exactly

-LRB-

N

a

-RRB-

total

board

space

In

other

words

the

sum

total

of

areas

of

all

boards

in

the

system

is

-LRB-

N

a

-RRB-

Two

pinout

technologies

The

pin-count

P

out

of

a

board

has

a

direct

influence

on

the

data

rate

that

can

flow

in/out

of

a

given

processor

cluster

It

has

been

shown

in

-LSB-

-RSB-

that

for

any

given

inter-board

network

topology

-LRB-

mesh

hypercube

-RRB-

P

also

directly

determines

the

bisection

bandwidth

of

the

inter-board

network

and

hence

the

peak

performance

of

the

system

Currently

two

different

types

of

technologies

are

being

employed

by

the

computer

industry

A

-RRB-

Surface

pinouts

In

this

technology

the

surface

of

the

board

is

utilized

for

external

connections

using

elastomeric

connectors

-LSB-

-RSB-

Under

this

technology

the

pin-count

from

a

board

of

area

A

is

O

-LRB-

A

-RRB-

Let

us

define

parameter

to

be

the

pin-count

that

can

be

supported

on

a

given

board

area

a

using

surface

pinout

technology

This

implies

that

a

square

board

area

of

-LRB-

j

a

-RRB-

would

have

a

peripheral

pin-count

of

-LRB-

j

-RRB-

B

-RRB-

Peripheral

pinouts

This

is

the

more

traditional

technology

Only

the

periphery

of

the

board

is

used

for

external

connections

Thus

if

the

area

of

the

board

is

A

the

peripheral

pinout

is

limited

by

O

-LRB-

pA

-RRB-

Let

us

define

parameter

to

be

the

pin-count

that

can

be

supported

from

a

perimeter

of

length

-LRB-

this

corresponds

to

the

perimeter

of

a

square

board

area

a

-RRB-

using

peripheral

pinout

technology

This

implies

that

a

square

board

area

of

-LRB-

j

a

-RRB-

would

have

a

peripheral

pinout

of

-LRB-

-RRB-

It

is

easy

to

see

that

the

pincount

in

the

surface

technology

could

be

potentially

an

order

of

magnitude

more

than

the

O

-LRB-

pA

-RRB-

periphery

technology

This

technology

has

been

used

in

the

J-machine

prototype

and

is

partly

the

reason

why

processors

could

be

packed

on

a

single

board

In

Fig.

we

show

the

growths

of

pincount

with

board

area

under

the

two

different

technologies

We

also

compare

the

relative

growth

of

pincounts

in

the

two

technologies

as

board

sizes

are

increased

under

different

relationship

between

and

The

vertical

clipping

line

in

each

graph

is

to

remind

the

reader

that

the

board

size

can

not

grow

continuously

and

is

limited

by

some

maximum

size

-LRB-

m

a

-RRB-

Board

size

-LRB-

A

-RRB-

Pin-count

O

-LRB-

A

-RRB-

surface

O

-LRB-

A

-RRB-

peripheral

Board

size

A

-LRB-

in

units

of

a

-RRB-

Pincount

in

O

-LRB-

A

-RRB-

surface

tech

periphery

tech

Pincount

in

O

-LRB-

A

-RRB-

Board

size

-LRB-

A

-RRB-

Pin-count

O

-LRB-

A

-RRB-

surface

O

-LRB-

A

-RRB-

peripheral

technology

technology

ma

technology

technology

ma

maxboard

size

maxboard

sizeimprovement

in

board

size

tech

Growth

of

pincount

with

board

area

under

the

two

technologies

are

shown

for

two

different

relationships

between

and

The

parameters

and

reflect

the

pin-count

that

can

be

supported

from

a

unit

board

area

a

under

the

two

respective

technologies

Figure

Comparison

of

two

pinout

technologies

a

-RRB-

Peripheral

and

b

-RRB-

Surface

It

is

to

be

noted

that

as

expected

for

large

board

sizes

the

surface

technology

always

supersedes

the

peripheral

technology

However

if

then

for

some

small

board

sizes

the

peripheral

technology

might

beat

the

surface

technology

We

also

make

the

following

observation

regarding

the

future

trends

in

values

of

and

Based

on

the

development

in

processor-memory

integration

technology

we

believe

that

processor-area

a

will

reduce

with

time

or

remain

around

the

same

if

memory

per

processor

is

also

increased

However

we

do

not

expect

pinout

density

per

unit

area

or

per

unit

length

to

change

in

comparable

leaps

This

implies

that

values

of

and

will

either

fall

or

remain

around

current

values

Without

loss

of

generality

in

this

paper

we

use

values

of

in

the

range

of

and

those

of

in

the

range

of

The

higher

limits

are

derived

from

the

current

design

trends

of

CRAY

-LSB-

-RSB-

and

J-machine

-LSB-

-RSB-

Parameterized

representation

of

inter-cluster

bisection

band

width

and

channel

width

In

this

section

we

develop

a

general

scheme

to

represent

architectural

characteristics

-LRB-

bisection

bandwidth

and

channel

width

-RRB-

of

the

inter-cluster

network

The

representation

is

parameterized

with

respect

to

the

two

pinout

technologies

under

fixed

total

board

area

Ideally

we

should

consider

similar

characterization

of

the

lower-level

cluster-networks

However

as

shown

in

-LSB-

-RSB-

lower

levels

have

more

flexibility

in

packaging

allowing

system

designer

to

choose

channel

widths

and

bisection

bandwidths

with

more

freedom

It

is

the

design

of

the

higher

level

inter-cluster

network

under

more

rigid

packaging

constraints

that

is

critical

in

determining

overall

system

performance

A

-RRB-

Inter-cluster

bisection

bandwidth

B

The

bisection

bandwidth

B

of

an

architecture

is

the

minimum

number

of

wires

that

need

to

be

cut

in

order

to

divide

the

network

into

two

equal

parts

It

limits

the

number

of

messages

that

can

cross

from

one

half

to

another

half

of

the

network

This

puts

a

restriction

on

the

peak

performance

of

the

system

Table

gives

the

bisection

bandwidths

of

the

internets

for

various

topologies

with

N

processors

under

the

two

pinout

technologies

The

same

information

is

also

shown

graphically

at

the

bottom

of

the

table

The

following

observations

can

be

made

Under

O

-LRB-

A

-RRB-

pinout

technology

we

have

-RRB-

the

total

number

of

internet

links

-LRB-

pin-count

-RRB-

remains

constant

at

-RRB-

bisection

bandwidth

increases

with

cluster

-LRB-

board

-RRB-

size

and

-RRB-

larger

cluster

sizes

with

or

hypercube

internets

provide

similar

B.

Similarly

under

O

-LRB-

pA

-RRB-

pinout

technology

we

have

-RRB-

total

internet

links

-LRB-

pin-count

-RRB-

drops

with

larger

cluster

sizes

-RRB-

bisection

bandwidth

falls

with

increasing

cluster

-LRB-

board

-RRB-

size

-LRB-

except

in

the

case

of

internet

where

it

remains

constant

at

-RRB-

and

-RRB-

larger

cluster

sizes

with

or

hypercube

internets

provide

similar

B.

Broadly

speaking

the

O

-LRB-

A

-RRB-

technology

can

potentially

support

larger

cluster

sizes

with

a

corresponding

growth

in

B

and

hence

performance

while

maintaining

same

total

board

area

and

same

total

pinout

The

O

-LRB-

pA

-RRB-

technology

on

the

other

hand

fails

to

keep

up

B

and

hence

performance

with

increasing

cluster

sizes

B

-RRB-

Inter-cluster

channel

width

W

Wires

in

a

channel

can

be

divided

into

two

categories

wires

that

carry

data

and

those

that

carry

channel

control

signals

For

example

in

Cray

-LSB-

-RSB-

a

channel

width

is

-LRB-

data

part

-RRB-

-LRB-

control

part

-RRB-

wires

Actually

the

data

part

in

channels

can

be

observed

as

a

trend

in

most

current

generation

machines

-LSB-

-RSB-

Many

factors

like

path-width

inside

routers

connector

technology

etc.

restrict

channel

widths

from

being

arbitrarily

large

However

it

is

expected

that

technologies

in

near

future

would

allow

channels

to

carry

or

data

-LSB-

-RSB-

corresponding

to

W

ss

and

W

ss

respectively

Table

gives

the

channel

widths

of

the

internets

for

various

topologies

with

N

processors

under

the

two

pinout

technologies

To

illustrate

the

example

we

choose

the

pinout

densities

as

and

for

surface

and

periphery

technologies

respectively

We

also

show

the

absolute

channel

widths

in

parenthesis

corresponding

to

these

values

Bottom

Table

Bisection

Bandwidth

-LRB-

B

-RRB-

of

Different

Inter-cluster

Network

Configurations

with

Increasing

Board

Area

-LRB-

A

c

a

-RRB-

to

design

a

N

processor

system

with

-RRB-

Surface

Pinout

O

-LRB-

A

-RRB-

Technology

and

-RRB-

Periphery

Pinout

O

-LRB-

pA

-RRB-

Technology

Table

Table

Cluster

-LRB-

board

-RRB-

size

c

No.ofclusters

-LRB-

boards

-RRB-

M

N/c

Pincount.

P

in

O

-LRB-

A

-RRB-

technology

Totalsystempin-count

internetB

=P

internetB

M

internet

B

internet

B

HypercueB

MP

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

Bisection

b/w

vs.

cluster

size

for

different

inter-cluster

topologies

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

vs.

cluster

size

for

different

inter-cluster

topologies

O

-LRB-

A

-RRB-

technology

O

-LRB-

A

-RRB-

technology

-LRB-

in

units

of

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

-LRB-

in

units

of

-RRB-

Periphery

Pinout

Technology

Table

Cluster

-LRB-

board

-RRB-

size

c

No.ofclusters

-LRB-

boards

-RRB-

M

N/c

Pincount.

P

in

O

-LRB-

A

-RRB-

technology

Totalsystempin-count

internetB

=P

internetB

M

internet

B

internet

B

HypercueB

MP

Surface

Pinout

Technology

The

above

tables

are

represented

below

through

graphs

Table

Channel

Width

-LRB-

W

-RRB-

of

Different

Inter-cluster

Network

Configurations

with

Increasing

Board

Area

-LRB-

A

c

a

-RRB-

to

design

a

N

processor

system

with

-RRB-

Surface

Pinout

O

-LRB-

A

-RRB-

Technology

and

-RRB-

Periphery

Pinout

O

-LRB-

pA

-RRB-

Technology

Table

O

-LRB-

A

-RRB-

pinout

O

-LRB-

A

-RRB-

pinout

with

c

c

c

c

H

c

c

c

c

c

H

c

Table

O

-LRB-

A

-RRB-

pinout

O

-LRB-

A

-RRB-

pinout

with

c

c

c

c

H

c

c

D

c

c

c

H

c

Table

O

-LRB-

A

-RRB-

pinout

with

O

-LRB-

A

-RRB-

pinout

with

c

c

c

c

H

c

c

c

c

c

H

c

Feasible

system

configurations

-LRB-

inter-cluster

topology

with

associated

cluster

size

c

-RRB-

under

three

channel

width

constraints

The

value

of

c

indicates

the

configuration

is

infeasible

Cluster

-LRB-

board

-RRB-

size

c

No.ofclusters

-LRB-

boards

-RRB-

M

N/c

Pincount.

P

in

O

-LRB-

A

-RRB-

technology

Totalsystempin-count

internetW

=P

internetW

=P

internetW

=P

=P

HypercueW

P

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Table

Cluster

-LRB-

board

-RRB-

size

c

No.ofclusters

-LRB-

boards

-RRB-

M

N/c

Pincount.

P

in

O

-LRB-

A

-RRB-

technology

Totalsystempin-count

internetW

=P

internetW

=P

internetW

=P

internetW

=P

HypercubeW

P

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Surface

Pinout

Technology

-LRB-

is

assumed

to

derive

representative

values

shown

in

parenthesis

-RRB-

Periphery

Pinout

Technology

-LRB-

is

assumed

to

derive

representative

values

shown

in

parenthesis

-RRB-

W

H

hypercube

W

W

of

the

table

shows

the

restrictions

imposed

on

system

design

by

three

different

channel

width

constraints

-RRB-

W

ss

corresponds

to

current

generation

machines

-RRB-

W

ss

corresponds

to

near

future

generation

and

-RRB-

W

ss

corresponds

to

machines

predicted

at

the

turn

of

this

century

-LSB-

-RSB-

It

is

to

be

noted

that

these

constraints

restrict

allowable

cluster

sizes

to

small

ranges

In

O

-LRB-

pA

-RRB-

pinout

technology

we

observe

that

W

ss

restricts

cluster

sizes

to

W

ss

restricts

cluster

sizes

to

and

W

ss

restricts

cluster

sizes

to

A

general

trend

to

be

noted

is

that

larger

channel

widths

allow

larger

cluster

sizes

in

both

pinout

technologies

However

under

O

-LRB-

A

-RRB-

technology

this

is

accompanied

by

a

growth

in

bisection

bandwidth

as

well

whereas

in

O

-LRB-

pA

-RRB-

technology

this

is

accompanied

by

a

fall

in

bisection

bandwidth

Although

future

technologies

will

offer

the

choice

of

using

wider

channels

-LRB-

W

ss

-RRB-

system

designers

will

still

have

the

option

of

using

narrow

channels

-LRB-

W

ss

-RRB-

Figure

shows

an

example

with

the

bisection

bandwidth

curves

obtained

earlier

-LRB-

Table

-RRB-

clipped

so

as

to

show

available

alternatives

under

W

-LRB-

allowing

W

ss

-RRB-

These

form

the

set

of

design-feasible

configurations

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

Bisection

b/w

vs.

cluster

size

for

different

inter-cluster

topologies

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

vs.

cluster

size

for

different

inter-cluster

topologies

O

-LRB-

A

-RRB-

technology

O

-LRB-

A

-RRB-

technology

-LRB-

in

units

of

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

-LRB-

in

units

of

-RRB-

Table

O

-LRB-

A

-RRB-

pinout

O

-LRB-

A

-RRB-

pinout

with

c

c

c

c

H

c

c

D

c

c

c

H

c

Configurations

which

support

channel

width

W

The

darkened

sections

on

the

curves

show

the

feasible

configurations

which

support

channel

width

W

Figure

Impact

of

channel

width

constraint

leading

to

a

set

of

design-feasible

configurations

Processor

and

communication

link

speeds

Increase

in

processing

power

directly

translates

to

greater

demands

on

the

interconnection

network

bandwidth

To

get

an

estimate

of

demand

on

network

bandwidth

we

need

to

know

the

desired

sustainable

computation

to

communication

ratio

in

future

systems

For

this

paper

we

choose

this

ratio

as

as

suggested

in

-LSB-

-RSB-

-LRB-

however

our

analytical

framework

works

for

any

computation

to

communication

ratio

-RRB-

This

implies

that

a

processor

demands

of

network

bandwidth

In

a

hierarchical

system

not

all

the

traffic

reaches

all

networks

We

first

explain

how

to

compute

the

demanded

bisection

bandwidth

in

each

network

in

the

hierarchy

It

is

obvious

that

a

processor

performance

can

only

be

sustained

if

the

demanded

bandwidth

does

not

exceed

the

offered

bandwidth

in

each

network

The

bandwidth

demanded

in

each

network

depends

on

the

underlying

traffic

pattern

We

discuss

two

commonly

used

traffic

patterns

uniform

and

localized

Then

under

each

of

these

traffic

patterns

we

show

how

to

represent

the

demanded

bisection

bandwidth

in

the

inter-cluster

network

parameterized

with

respect

to

both

processor

and

network

speeds

Bisection

bandwidth

demand

in

each

hierarchy

Let

us

consider

a

simple

two-level

hierarchy

with

N

processors

each

low-level

cluster

consists

of

c

processors

and

there

are

M

N

c

clusters

Let

each

processor

demand

total

interconnection

bandwidth

of

D

MB/s

and

let

the

probability

of

a

generated

request

not

going

outside

a

cluster

be

p

We

represent

demanded

bisection

bandwidth

in

high-level

network

as

B

and

demanded

bisection

bandwidth

in

each

low-level

cluster

network

as

It

can

be

shown

-LSB-

-RSB-

that

B

ND

-LRB-

p

-RRB-

and

cD

-LRB-

p

-RRB-

To

support

larger

clusters

the

cluster

bisection

bandwidth

should

scale

linearly

with

cluster

size

c

In

this

paper

we

assume

this

to

be

the

case

since

packaging

constraints

are

less

rigid

in

lower

hierarchies

allowing

thicker

channel/buses

to

be

used

to

scale

up

network

bandwidth

-LSB-

-RSB-

We

focus

on

the

more

critical

upper

level

inter-cluster

network

bandwidth

B

Our

goal

is

to

parameterize

this

B

as

a

function

of

processor

and

network

speeds

and

then

compare

it

with

actual

offered

bisection

bandwidth

by

the

packaging

technology

-LRB-

derived

in

sec.

-RRB-

We

note

that

the

demanded

B

is

heavily

dependent

on

the

underlying

p

determined

by

the

traffic

patterns

We

discuss

this

briefly

in

the

next

section

Traffic

Patterns

Traffic

patterns

depend

on

disparate

factors

like

application

characteristics

underlying

architecture

mapping

efficiency

and

programming

model

etc.

A

convincing

way

to

represent

them

is

to

actually

generate

them

through

application

implementation

-LSB-

-RSB-

However

for

comparing

a

large

number

of

design

alternatives

this

approach

is

not

practical

Researchers

therefore

rely

on

two

synthetic

traffic

patterns

-LSB-

-RSB-

to

give

them

quick

feedback

on

relative

performance

of

systems

A

-RRB-

Uniform

traffic

Under

this

assumption

a

message

or

a

request

generated

by

a

processor

is

equally

probable

to

go

to

any

other

processor

in

the

system

The

justification

for

using

this

pattern

is

-RRB-

the

application

is

dynamic

with

no

communication

locality

or

-RRB-

not

enough

compiler

or

human

effort

is

expended

on

efficient

mapping

on

underlying

interconnection

or

-RRB-

some

high-level

programming

constructs

purposely

hide

details

of

the

architecture

from

the

user

encouraging

simplicity

at

the

cost

of

performance

-LSB-

-RSB-

B

-RRB-

Localized

traffic

Under

this

assumption

a

request

by

a

processor

has

higher

probability

of

being

satisfied

by

closer

neighbors

than

by

distant

ones

This

is

supposed

to

be

more

representative

for

distributed

shared

memory

programming

with

memory

hierarchies

and

caches

-LSB-

-RSB-

A

location

often

referred

by

a

processor

is

attracted

close

to

the

processor

-LSB-

-RSB-

thus

ensuring

locality

However

two

open

problems

associated

with

this

pattern

is

-RRB-

what

kind

of

locality

model

to

choose

and

-RRB-

how

to

fairly

compare

different

alternative

designs

with

different

cluster

sizes

i.e.

a

model

should

reflect

that

with

larger

cluster

size

c

the

probability

of

a

request

being

met

inside

the

cluster

increases

We

suggest

and

use

a

locality

model

described

below

Though

no

claims

can

be

made

on

its

accuracy

we

believe

that

it

represents

the

trend

and

is

reasonably

good

for

comparing

among

different

system

choices

Let

p

be

the

probability

that

a

generated

message

is

for

intra-cluster

network

To

evaluate

the

performance

of

different

system

topologies

we

need

to

account

for

an

increase

in

p

as

cluster

size

c

increases

We

consider

an

application

that

can

be

mapped

efficiently

on

a

grid

architecture

such

that

a

processor

requires

to

communicate

only

with

its

nearest

neighbors

We

can

estimate

the

probability

of

a

message

going

to

each

of

its

neighbors

We

assume

that

the

grid

can

be

mapped

onto

a

clustered

architecture

of

the

same

size

such

that

each

cluster

gets

a

sub-grid

of

size

c

It

is

easy

to

estimate

the

fraction

of

messages

staying

inside

the

sub-grid

This

gives

an

estimate

on

p

of

the

cluster

If

we

assume

a

grid

to

begin

with

then

in

a

pc

x

pc

sub-grid

p

can

be

derived

as

pc

c

Thus

-LRB-

p

-RRB-

ss

pc

for

large

c.

Table

summarizes

p

for

various

cluster

sizes

Using

the

grid

Table

Estimate

of

Increase

in

p

the

Probability

of

a

Message

being

for

Intra-cluster

network

with

Increase

in

Cluster

Size

c.

Cluster

Size

c

p

p

model

model

of

clustering

-LRB-

p

-RRB-

fraction

of

messages

going

outside

a

cluster

would

only

be

destined

to

neighboring

clusters

at

most

away

However

to

avoid

making

our

model

too

restrictive

we

allow

messages

going

out

of

the

cluster

to

be

destined

uniformly

to

any

other

cluster

in

the

system

Parameterizing

inter-cluster

bisection

bandwidth

B

In

this

section

we

show

how

to

parameterize

the

demand

on

B

in

the

inter-cluster

network

as

a

function

of

both

processor

and

network

speeds

The

derivation

is

done

under

both

uniform

and

localized

traffic

We

use

the

following

example

to

motivate

the

discussion

Let

D

be

the

network

bandwidth

demanded

by

a

processor

in

a

N

processor

system

This

translates

to

a

bisection

bandwidth

demand

of

ND

under

uniform

traffic

assumption

Let

the

network

clock

cycle

time

be

Tc

-LRB-

referred

to

from

now

on

as

cycle

-RRB-

the

time

taken

to

transfer

a

bit

across

an

internet

wire

This

implies

that

a

channel

with

W

wires

can

transfer

W

bits

simultaneously

in

time

Tc

Therefore

B

Tc

under

cycle

of

implies

that

B

be

wires

while

under

a

higher

cycle

of

it

implies

a

greater

B

with

wires

Observe

that

an

increase

in

D

will

capture

the

improvement

in

processor

speeds

while

a

decrease

in

Tc

will

capture

the

improvement

in

link

speeds

in

network

To

capture

both

these

technologies

together

we

choose

to

represent

processor

demand

on

network

in

terms

of

a

parameter

bits/cycle

Thus

D

under

Tc

leads

to

bits/cycle

Similarly

under

Tc

it

is

bits/cycle

Bisection

bandwidth

B

under

uniform

traffic

can

be

directly

represented

as

B

N

In

Table

we

show

the

demanded

B

as

a

function

of

bits/cycle

for

various

system

configurations

with

N

under

both

uniform

and

localized

traffic

models

The

values

of

p

used

in

localized

traffic

are

taken

from

Table

Table

Demanded

bisection

bandwidth

B

as

a

function

of

for

both

traffic

models

under

various

system

configurations

with

N

Cluster

size

-LRB-

c

-RRB-

Demanded

inter-cluster

Cluster

-LRB-

board

-RRB-

Demand

under

l

l

l

l

l

l

l

l

l

l

l

l

l

l

size

uniform

traffic

B

Nl

-LRB-

-RRB-

Demand

under

localized

traffic

B

Nl

-LRB-

-RRB-

c

bisection

bandwidth

-LRB-

B

-RRB-

uniform

traffic

localized

traffic

To

get

representative

values

of

we

choose

various

combinations

of

processor

and

network

speeds

and

summarize

these

in

Table

Most

of

the

values

are

taken

from

-LSB-

-RSB-

and

are

predictions

for

the

near

future

It

is

to

be

noted

that

values

will

lie

pretty

much

in

the

range

of

bits/cycle

in

the

near

future

Normalizing

demanded

bandwidth

w.r.t.

packaging

technol

ogy

In

section

we

captured

board

and

pinout

technologies

through

the

parameters

and

-LRB-

corresponding

to

surface

and

peripheral

technologies

respectively

-RRB-

We

parameterized

the

bisection

bandwidth

offered

by

a

system

configuration

in

terms

of

and

To

compare

the

processor

demanded

bandwidth

against

the

offered

bandwidth

we

need

to

normalize

the

Table

Representative

processor

demand

on

network

in

bits/channel

cycle

for

various

combinations

of

processor

and

network

speeds

Processor

Network

channel

Processor

MB/s

ns

demand

on

n/w

D

-LRB-

MB/s

-RRB-

cycle

time

Tc

-LRB-

ns

-RRB-

demand

on

n/w

l

bits/cycle

ns

ns

ns

ns

demanded

bandwidth

also

in

terms

of

and

This

can

be

easily

achieved

by

dividing

the

computed

bandwidth

demands

in

Table

by

either

or

-LRB-

depending

on

surface

or

peripheral

technology

being

considered

-RRB-

Design

choices

A

good

configuration

is

a

design-feasible

configuration

which

supports

the

bandwidth

demanded

from

it

In

this

section

we

first

decide

on

good

configurations

by

comparing

the

bisection

bandwidth

offered

by

design-feasible

configurations

-LRB-

derived

in

sec.

-RRB-

to

the

demanded

bisection

bandwidth

-LRB-

derived

in

sec.

-RRB-

Among

all

the

good

configurations

the

choice

which

provides

best

performance

with

cheapest

cost

is

defined

as

the

best

configuration

-LSB-

-RSB-

We

take

an

example

to

show

how

to

decide

the

set

of

good

configurations

The

search

for

the

best

configuration

however

is

not

a

very

clear-cut

process

Unknown

and

inestimable

factors

like

exact

link

costs

connector

costs

and

other

packaging

costs

make

this

decision

extremely

hard

Through

the

same

example

we

show

some

of

the

factors

that

may

be

considered

in

the

choice

of

the

best

configuration

Simulation

modeling

might

be

useful

in

cases

where

relative

performances

are

not

clear

Good

configurations

Let

us

consider

the

following

example

of

designing

a

system

with

processors

under

the

following

technological

parameters

and

W

-LRB-

ss

-RRB-

Figure

shows

the

good

configurations

for

the

two

pinout

technologies

and

both

uniform

and

localized

traffic

patterns

The

configurations

which

are

above

the

line

of

satisfy

the

bisection

bandwidth

demands

As

we

saw

earlier

in

Fig.

the

darkened

sections

on

the

offered

bandwidth

curves

of

various

internets

show

the

feasible

configurations

which

support

the

channel

width

W

-LRB-

ss

-RRB-

It

is

to

be

noted

that

in

all

cases

internet

is

not

good

at

all

With

O

-LRB-

pA

-RRB-

technology

internet

with

c

is

good

under

both

traffic

patterns

However

with

O

-LRB-

A

-RRB-

technology

internet

is

good

with

c

only

under

localized

traffic

Similarly

internet

is

good

with

c

under

O

-LRB-

pA

-RRB-

and

c

under

O

-LRB-

A

-RRB-

technology

internet

is

good

with

c

under

O

-LRB-

pA

-RRB-

and

c

cluster

size

-LRB-

c

-RRB-

O

-LRB-

A

-RRB-

peripheral

pinout

technology

Bisection

b/w

-LRB-

B

-RRB-

-LRB-

in

units

of

-RRB-

Technology

parameters

l

bits/cycle

and

W

-LRB-

shown

by

darkened

sections

on

offered

bisection

curves

-RRB-

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

O

-LRB-

A

-RRB-

surface

pinout

technology

-LRB-

in

units

of

-RRB-

Good

configurations

with

c

l

l

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

-LRB-

in

units

of

-RRB-

l

uniform

localized

uniform

cluster

size

-LRB-

c

-RRB-

Bisection

b/w

-LRB-

B

-RRB-

-LRB-

in

units

of

-RRB-

l

localized

with

c

Good

configurations

with

c

with

c

Good

configurations

with

c

with

c

with

c

hcube

with

c

hcube

with

c

with

c

hcube

with

c

Good

configurations

with

c

with

c

with

c

hcube

with

c

Figure

Search

for

good

configurations

from

the

set

of

feasible

configurations

under

two

traffic

patterns

a

-RRB-

uniform

and

b

-RRB-

localized

and

two

pinout

technologies

-RRB-

Surface

O

-LRB-

A

-RRB-

technology

and

-RRB-

Peripheral

O

-LRB-

pA

-RRB-

technology

under

O

-LRB-

A

-RRB-

technology

Hypercube

internet

follows

closely

with

good

choices

being

c

under

O

-LRB-

pA

-RRB-

and

c

under

O

-LRB-

A

-RRB-

technology

It

is

also

to

be

observed

how

the

localized

traffic

modeling

adds

more

choices

to

the

design

space

With

surface

technology

the

internet

with

c

is

a

good

configuration

under

localized

traffic

model

but

not

so

under

the

uniform

traffic

model

It

is

to

be

noted

that

in

this

search

for

the

best

system

we

are

still

left

with

around

ten

to

fifteen

good

choices

amongst

which

we

have

to

choose

one

We

discuss

below

some

guidelines

and

considerations

to

reduce

the

size

of

this

set

Choice

of

cluster

size

We

discuss

the

advantages

and

disadvantages

of

large

cluster

sizes

Larger

cluster

sizes

have

the

following

advantages

The

number

of

clusters

-LRB-

boards

-RRB-

in

the

system

decreases

leading

to

lesser

number

of

inter-board

channels

This

results

in

fewer

connectors

being

required

If

connector

costs

are

high

this

will

result

in

savings

in

cost

Channel

widths

in

internet

can

be

higher

leading

to

shorter

messages

This

will

lead

to

low

contention

and

low

latency

In

case

of

DSM

implementation

with

directory-based

cache-coherence

the

cache-coherence

network

interface

logic

and

directory

memory

has

to

be

replicated

less

number

of

times

This

can

yield

cheaper

systems

However

arbitrarily

large

clusters

may

not

be

possible

due

to

The

scalability

restrictions

of

the

cluster

network

-LRB-

if

it

is

bus-based

-RRB-

or

the

cluster

interface

may

limit

cluster

sizes

to

small

values

In

O

-LRB-

pA

-RRB-

pinout

technology

the

offered

bisection

bandwidth

by

any

internet

falls

with

growth

in

cluster

size

Thus

if

demand

on

bandwidth

were

very

high

one

has

to

use

smaller

cluster

sizes

In

the

following

discussion

without

loss

of

generality

we

assume

that

cluster

networks

and

interfaces

can

be

scaled

to

support

larger

clusters

However

this

is

not

an

unreasonable

assumption

considering

that

our

analysis

indicates

that

in

the

near

future

cluster

sizes

greater

than

will

not

be

feasible

from

other

constraints

This

indicates

that

a

system

designer

has

flexibility

in

choosing

the

cluster

sizes

upto

ten

Choosing

the

best

configuration

A

-RRB-

Surface

pinout

O

-LRB-

A

-RRB-

technology

In

this

technology

irrespective

of

internet

configuration

the

total

volume

of

wires

in

the

internet

-LRB-

total

number

of

pins

-RRB-

is

fixed

Larger

clusters

offer

greater

bisection

bandwidth

under

this

technology

Thus

a

reasonable

approach

is

to

go

for

the

larger

cluster

sizes

in

each

internet

among

the

good

choices

In

our

example

under

O

-LRB-

A

-RRB-

technology

and

uniform

traffic

we

pick

among

the

good

configurations

the

internet

with

c

internet

with

c

and

hypercube

with

c

Some

of

these

cluster

values

are

chosen

so

as

to

make

the

system

size

as

close

to

as

possible

while

maintaining

the

desired

topology

In

Table

we

compare

various

characteristics

of

these

three

choices

actual

bisection

bandwidths

average

distances

to

be

traveled

number

of

internet

channels

-LRB-

connectors

-RRB-

and

internet

channel

width

We

see

that

most

of

these

properties

are

quite

close

making

it

tough

to

make

the

choice

of

the

best

configuration

The

hypercube

may

be

rejected

on

the

basis

of

variable

degree

which

makes

scalability

difficult

But

this

still

leaves

us

with

two

choices

Simulation

modeling

may

be

used

to

decide

amongst

these

However

in

this

case

since

cluster

sizes

are

very

close

and

internet

sizes

are

also

similar

the

analysis

for

the

best

internet

is

equivalent

to

the

choice

of

the

best

dimensionality

in

a

flat

network

given

nodes

with

fixed

pinouts

The

equivalence

holds

good

because

clusters

can

be

treated

as

supernodes

connected

through

a

flat

network

-LRB-

internet

-RRB-

Such

an

analysis

has

been

done

by

Agrawal

in

-LSB-

-RSB-

where

it

was

shown

that

for

arbitrary

length

messages

and

reasonable

switch

delays

a

topology

will

slightly

outperform

the

topology

Hence

we

conclude

that

configuration

will

be

the

best

This

configuration

also

uses

less

number

of

internet

channels

-LRB-

connectors

-RRB-

to

build

a

less

costlier

system

Table

Comparing

Various

Performance

and

Cost

Characteristics

of

the

Three

Final

Good

Configurations

to

Design

N

ss

Processor

System

under

Surface

Pinout-O

-LRB-

A

-RRB-

Technology

Table

Internettopology

cluster

sizec

Actualbisection

bandwidth

Averagedistance

traveled

ininternet

Number

ofinternetchannels

-LRB-

connectors

-RRB-

InternetchannelwidthW

Hcube

Demanded

bisection

bandwidth

B

-RRB-

Under

Peripheral

Pinout

O

-LRB-

pA

-RRB-

technology

In

this

technology

the

total

volume

of

wires

in

the

internet

-LRB-

total

number

of

pins

-RRB-

falls

with

increasing

cluster

size

This

decreasing

system

volume

could

translate

to

a

fall

in

packaging

costs

Keeping

in

mind

the

benefit

of

large

clusters

we

go

for

the

larger

cluster

sizes

in

each

internet

among

the

good

choices

This

implies

we

choose

with

c

with

c

and

with

c

In

Table

we

compare

various

characteristics

of

these

three

choices

actual

bisection

bandwidths

average

distances

to

be

traveled

number

of

internet

channels

-LRB-

connectors

-RRB-

total

number

of

wires

in

internet

and

internet

channel

widths

All

three

systems

offer

more

than

of

the

demanded

Table

Comparing

Various

Performance

and

Cost

Characteristics

of

the

Three

Final

Good

Configurations

to

Design

N

ss

Processor

System

under

Peripheral

Pinout

O

-LRB-

pA

-RRB-

Technology

Table

Internettopology

cluster

sizec

Actualbisection

bandwidth

Averagedistance

traveled

ininternet

Number

ofinternetchannels

-LRB-

connectors

-RRB-

Volume

-LRB-

number

-RRB-

of

internetwires

Internetchannelwidth

W

Demanded

bisection

bandwidth

bandwidth

This

implies

that

all

three

systems

would

be

operating

at

around

load

and

therefore

offer

low

latencies

However

comparing

the

average

distances

to

be

traveled

in

the

internet

we

find

that

the

-LRB-

with

c

-RRB-

-LRB-

with

c

-RRB-

configurations

outperform

the

-LRB-

with

c

-RRB-

by

a

factor

of

more

than

This

implies

that

basic

latency

in

the

configuration

would

be

substantially

higher

In

terms

of

total

volume

of

wires

in

the

internet

we

see

that

the

-LRB-

with

c

-RRB-

configuration

has

the

least

number

of

internet

wires

This

configuration

also

has

least

number

of

internet

channels

requiring

minimum

number

of

connectors

These

observations

lead

us

to

conclude

that

the

-LRB-

-RRB-

with

clusters

of

size

would

be

the

best

system

configuration

Conclusions

and

Future

Work

In

this

paper

we

have

presented

a

comprehensive

framework

to

design

large

multiprocessor

systems

The

proposed

framework

takes

into

account

advancements

in

packaging

processor

and

interconnection

technologies

while

deciding

the

most

optimum

system

configuration

The

elegance

of

this

framework

lies

in

its

parameterized

representation

of

different

technologies

and

then

deriving

an

optimal

system

configuration

for

any

set

of

technologies

and

constraints

The

significance

lies

in

its

generality

which

has

never

been

taken

into

consideration

by

earlier

researchers

All

earlier

works

have

centered

around

fixed

topology

technology

or

packaging

constraint

We

have

applied

the

framework

to

obtain

design

guidelines

for

the

currently

popular

k-ary

n-cube

cluster-c

architectures

This

research

suggests

the

following

guidelines

to

system

designers

for

building

large-scale

k-ary

n-cube

cluster-c

parallel

systems

Larger

channel

width

technology

will

allow

larger

cluster

sizes

under

both

surface

and

peripheral

pinout

technologies

Some

of

the

more

specific

obtained

results

are

as

follows

-LRB-

a

-RRB-

The

current

channel

width

technology

of

data

bits

and

periphery

pinout

technology

can

use

cluster

sizes

of

to

processors

with

internets

to

deliver

maximum

system

performance

-LRB-

b

-RRB-

The

near

future

channel

width

technology

of

data

bits

together

with

periphery

pinout

technology

can

go

for

cluster

sizes

of

with

internet

Similarly

under

surface

pinout

technology

cluster

sizes

of

with

internet

becomes

optimum

-LRB-

c

-RRB-

With

channel

width

technology

of

data

bits

cluster

sizes

as

large

as

are

possible

This

implies

that

in

the

future

a

processor

board

technology

which

allows

to

integrate

around

processors

on

a

single

board

would

be

desirable

Under

surface

pinout

technology

an

increase

in

cluster

size

is

associated

with

growth

in

bisection

bandwidth

whereas

it

is

associated

with

a

fall

in

bisection

bandwidth

in

periphery

pinout

technology

As

processor

and

interconnect

technologies

get

advanced

in

future

the

processor

demand

rate

on

network

bandwidth

will

be

expected

to

stay

between

bits/channel

cycle

For

designing

distributed-shared-memory

system

one

must

consider

an

appropriate

localized

traffic

model

to

derive

optimal

system

configuration

from

a

large

set

of

designfeasible

configurations

This

research

has

emphasized

on

the

design

of

two-level

hierarchical

systems

with

more

emphasis

on

the

design

of

the

top-level

network

However

our

framework

is

general

enough

to

consider

more

levels

In

the

next

phase

of

this

research

we

are

investigating

on

the

design

of

lower

levels

It

will

also

be

interesting

to

see

how

to

extend

our

analysis

to

a

combination

of

surface

and

periphery

pinout

technology

as

this

might

be

a

more

common

practice

in

the

future

References

-LSB-

-RSB-

A.

Agrawal

Limits

on

Interconnection

Network

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

Oct

-LSB-

-RSB-

D.

Basak

and

D.

K.

Panda

Scalable

Architectures

with

k-ary

n-cube

cluster-c

organization

In

Proc

of

the

Symposium

of

Parallel

and

Distributed

Processing

-LSB-

-RSB-

C.

Chen

D.

P.

Agrawal

and

J.

R.

Burke

dBcube

A

new

class

of

Hierarchical

Multiprocessor

Interconnection

Network

with

area

efficient

layout

to

appear

in

IEEE

Trans

on

Parallel

and

Distributed

Systems

Dec

-LSB-

-RSB-

Cray

Reasearch

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

S.

Dandamudi

and

D.

Eager

Hierarchical

Interconnection

Networks

for

Multicomputer

Systems

IEEE

Transactions

on

Computers

-LRB-

-RRB-

June

-LSB-

-RSB-

D.

Lenoski

et

al.

The

Stanford

DASH

Multiprocessor

IEEE

Computer

pages

-LCB-

-LSB-

-RSB-

D.

Lenoski

et

al.

The

DASH

Prototype

Implementation

and

Performance

In

Proc

of

the

Int

l

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

D.

Lenoski

et

al.

The

DASH

Prototype

Logic

Overhead

and

Performance

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Jan

-LSB-

-RSB-

Erik

Hagersten

and

Seif

Haridi

A

quantitative

comparison

of

efficiency

for

large

shared

memory

architectures

In

Third

Workshop

on

Scalable

Shared

Memory

Multiprocessors

International

Symposium

on

Computer

Architecure

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Performance

of

Hierarchical

Systems

with

wiring

constraints

In

Proc

of

the

Int

Conf

on

Parallel

Processing

Aug

-LSB-

-RSB-

W.

Hsu

and

P.

C.

Yew

The

Impact

of

Wiring

Constraints

on

Hierarchical

Network

Performance

In

Proc

of

the

Int

Parallel

Processing

Symposium

Mar

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

Kendall

Square

Research

KSR

Technical

Summary

-LSB-

-RSB-

Michael

D.

Noakes

Deborah

A.

Wallach

and

William

J.

Dally

The

j-machine

multicomputer

An

architectural

evaluation

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

K.

Padmanabhan

Effective

Architectures

for

Data

Access

in

a

Shared

Memory

Hierarchy

Jour

of

Parallel

and

Distributed

Computing

-LSB-

-RSB-

David

A.

Patterson

Observations

in

Massive

Parallelism

Trends

and

Predictions

for

to

Technical

Report

DIMACS

Sept

-LSB-

-RSB-

M.

T.

Raghunath

Interconnection

Network

Design

Based

on

Packaging

Considerations

PhD

thesis

U.

C.

Berkeley

Nov

-LSB-

-RSB-

M.

T.

Raghunath

and

Abhiram

Ranade

Designing

interconnection

networks

for

multilevel

packaging

In

Proc

of

the

Supercomputing

pages

-LCB-

-LSB-

-RSB-

E.

Rothberg

J.

P.

Singh

and

Anoop

Gupta

Working

Sets

Cache

Sizes

and

Node

Granularity

Issues

for

Large-Scale

Multiprocessors

In

Proc

of

the

Int

Symposium

on

Computer

Architecture

pages

-LCB-

hierarchical systems
interconnection network
packaging constraints

clustered systems
k-ary n-cube
pinout constraint