Implementing

Multidestination

Worms

in

Switch

Based

Parallel

Systems

Architectural

Alternatives

and

their

Impact

Craig

B.

Stunkel

Rajeev

Sivaram

and

Dhabaleswar

K.

Panda

Technical

Report

Implementing

Multidestination

Worms

in

Switch

Based

Parallel

Systems

Architectural

Alternatives

and

their

Impact

Craig

B.

Stunkely

Rajeev

Sivaram

Dhabaleswar

K.

Panda

yI.B.M

T.

J.

Watson

Research

Center

Dept.

of

Computer

and

Information

Science

P.

O.

Box

The

Ohio

State

University

Yorktown

Heights

NY

Columbus

OH

Phone

-LRB-

-RRB-

Phone

-LRB-

-RRB-

Fax

-LRB-

-RRB-

Email

stunkel@watson.ibm.com

Email

fsivaram,pandag@cis.ohio-state.edu

Contact

Author

Craig

B.

Stunkel

Abstract

Multidestination

message

passing

has

been

proposed

as

an

attractive

mechanism

for

efficiently

implementing

multicast

and

other

collective

operations

on

direct

networks

However

applying

this

mechanism

to

switch

based

parallel

systems

is

non-trivial

In

this

paper

we

propose

alternative

switch

architectures

with

differing

buffer

organizations

to

implement

multidestination

worms

on

switch

based

parallel

systems

First

we

discuss

issues

related

to

such

implementation

-LRB-

deadlockfreedom

replication

mechanisms

header

encoding

and

routing

-RRB-

Next

we

demonstrate

how

an

existing

central

queue

based

switch

architecture

supporting

unicast

message

passing

can

be

enhanced

fairly

easily

to

accommodate

multidestination

message

passing

Similarly

implementing

multidestination

worms

on

an

input

buffer

based

switch

architecture

is

discussed

Both

these

implementations

are

evaluated

against

each

other

as

well

as

against

a

software-based

scheme

using

the

central

queue

organization

Simulation

experiments

under

a

range

of

traffic

-LRB-

multiple

multicast

bimodal

varying

degree

of

multicast

and

message

length

-RRB-

and

system

size

are

used

for

evaluation

The

study

demonstrates

the

superiority

of

the

central

queue

based

switch

architecture

for

efficient

implementation

of

multidestination

worms

Such

an

implementation

can

perform

well

without

saturating

for

applied

loads

of

or

greater

It

also

indicates

that

under

bimodal

traffic

the

central-queue

based

hardware

multicast

implementation

affects

background

unicast

traffic

less

adversely

compared

to

a

software-based

multicast

implementation

Thus

multidestination

message

passing

can

easily

be

applied

to

switch-based

parallel

systems

to

deliver

good

collective

communication

performance

This

research

is

supported

in

part

by

NSF

Grant

NSF

Career

Award

and

an

IBM

Cooperative

Fellowship

Contents

Introduction

Switch

Based

Parallel

Systems

Issues

in

Implementing

Multidestination

Worms

Deadlock-free

Replication

Header

Encoding

Routing

Output

Buffer

-LRB-

Central

Queue

-RRB-

based

Switch

Architecture

Buffered

Wormhole

Routing

The

Shared

Central

Buffer

Replication

Method

Replication

Implementation

Emulating

Virtual

Cut-Through

Input

Buffer

based

Switch

Architecture

Comparing

the

two

architectures

Performance

Evaluation

Simulation

parameters

and

methodology

Multicast

simulation

issues

Impact

of

multicast

degree

Impact

of

message

size

Impact

of

system

size

Interaction

between

multicast

and

unicast

traffic

Measuring

latency

in

an

alternate

way

Summary

Related

Work

Conclusion

Introduction

The

wormhole-routing

switching

technique

is

the

current

trend

in

building

parallel

systems

due

to

inherent

advantages

such

as

low-latency

communication

and

reduced

communication

hardware

overhead

-LSB-

-RSB-

This

switching

technique

is

being

used

with

a

wide

variety

of

topologies

Examples

include

k-ary

n-cube

networks

-LRB-

Cray

-LSB-

-RSB-

Intel

Paragon

-LSB-

-RSB-

Ncube

-LSB-

-RSB-

-RRB-

fat-tree

networks

-LRB-

-LSB-

-RSB-

Meiko

-LSB-

-RSB-

-RRB-

and

multistage

interconnection

networks

-LRB-

IBM

-LSB-

-RSB-

-RRB-

All

of

these

systems

are

being

used

to

support

either

the

distributed-memory

or

distributed-shared

memory

programming

paradigms

For

efficient

support

of

either

paradigm

these

systems

require

fast

collective

communication

-LSB-

-RSB-

support

as

defined

by

the

Message

Passing

Interface

-LRB-

MPI

-RRB-

Standard

from

the

underlying

communication

subsystem

Among

the

set

of

collective

communication

operations

broadcast

and

multicast

are

quite

fundamental

and

they

get

used

in

several

other

operations

like

barrier

synchronization

and

reduction

-LSB-

-RSB-

Thus

it

is

essential

to

reduce

latency

of

broadcast

and

multicast

operations

on

these

systems

to

achieve

high

performance

parallel

computation

Many

software

schemes

have

been

recently

proposed

in

the

literature

to

efficiently

implement

broadcast

and

multicast

in

wormhole-routed

k-ary

n-cube

networks

-LSB-

-RSB-

and

multistage

interconnection

networks

-LSB-

-RSB-

All

these

schemes

use

point-to-point

-LRB-

unicast

-RRB-

message

passing

and

require

multiple

contention-free

phases

to

achieve

fast

broadcast

and

multicast

For

multicasting

to

d

destinations

these

schemes

typically

require

-LRB-

d

-RRB-

e

communication

phases

Since

the

ratio

of

communication

start-up

time

to

propagation

time

is

quite

high

on

current

parallel

systems

-LSB-

-RSB-

such

a

software

based

unicast

message-passing

approach

leads

to

very

high

latency

for

broadcast

and

multicast

operations

In

addition

under

hybrid

unicast

and

multicast

traffic

the

software-based

approach

leads

to

low

throughput

from

the

network

This

is

due

to

increased

network

contention

because

of

multi-phase

implementation

of

a

multicast

Such

performance

degradation

has

forced

researchers

and

designers

of

parallel

systems

to

go

for

hardware

support

for

multicast

-LSB-

-RSB-

A

new

concept

multidestination

wormhole

message

passing

has

been

introduced

recently

-LSB-

-RSB-

for

efficient

implementation

of

collective

communication

operations

using

less

number

of

communication

phases

Unlike

a

unicast

message

which

has

a

single

destination

such

a

mechanism

allows

a

message

to

have

multiple

destinations

A

multidestination

worm

provides

flexibility

to

distribute

data

to

multiple

nodes

or

gather

data

from

multiple

nodes

using

a

single

message

and

a

single

communication

start-up

These

worms

get

routed

on

the

data

network

together

with

the

unicast

messages

Using

such

worms

it

has

been

shown

that

broadcast

and

multicast

operations

on

k-ary

n-cube

networks

with

different

routing

schemes

-LRB-

e-cube

adaptive

Hamiltonian

etc.

-RRB-

can

be

implemented

with

significantly

reduced

latency

-LSB-

-RSB-

compared

to

unicast-based

schemes

This

mechanism

and

the

associated

schemes

were

specifically

developed

for

direct

networks

with

k-ary

n-cube

topologies

in

mind

and

thus

can

not

be

easily

extended

to

switch-based

systems

In

our

recent

work

-LSB-

-RSB-

we

have

extended

the

multidestination

message

passing

concept

to

multistage

interconnection

networks

This

extension

involves

a

new

concept

called

multiport

encoding

and

an

asynchronous

replication

mechanism

at

the

input

buffers

of

a

switch

to

implement

deadlock-free

multicast

Using

this

new

encoding

a

set

of

algorithms

for

multicast/broadcast

have

been

developed

and

evaluated

In

this

work

it

has

been

shown

that

the

latency

of

a

single

multicast/broadcast

operation

can

be

reduced

up

to

a

factor

of

compared

to

the

software-based

scheme

using

unicast

messages

The

results

in

-LSB-

-RSB-

show

the

proof-of-concept

of

multidestination

message

passing

in

MINs

However

it

does

not

explore

other

architectural

alternatives

and

their

impact

For

example

the

IBM

-LSB-

-RSB-

uses

a

central-queue

buffer

organization

in

a

switch

for

unicast

message

passing

and

this

potentially

provides

a

powerful

basis

for

worm

replication

Similarly

the

multiport-based

encoding

in

-LSB-

-RSB-

allows

a

multicast

to

be

implemented

in

multiple

phases

Even

though

this

approach

requires

less

number

of

phases

than

the

unicast-based

software

scheme

a

challenging

problem

is

to

implement

multicast

in

fewer

phases

using

better

encoding

schemes

In

this

paper

we

address

these

challenges

and

consider

architectural

alternatives

and

their

impact

on

the

implementation

of

multidestination

worms

for

multicast/broadcast

on

switch-based

parallel

systems

We

consider

a

bit-string

encoding

scheme

to

implement

multidestination

worms

This

encoding

allows

a

multicast/broadcast

to

be

implemented

using

a

single

communication

phase

Two

alternative

switch

architectures

central-queue

and

input-buffer

are

considered

Detailed

designs

for

implementing

multidestination

worms

on

these

two

switch

architectures

are

derived

It

is

shown

that

multidestination

worms

can

be

implemented

on

both

switch

architectures

with

very

little

additional

hardware

These

designs

also

provide

realistic

timing

parameters

for

routing

and

propagation

of

multidestination

worms

Next

the

implementations

on

both

switch

architectures

are

evaluated

under

different

traffic

scenarios

Two

performance

measures

-LRB-

latency-throughput

and

received

vs.

applied

load

-RRB-

are

studied

for

multiple

multicast

traffic

as

well

as

hybrid

traffic

consisting

of

multiple

multicast

and

unicast

The

hardware-based

implementations

are

also

compared

with

the

software-based

multicast

scheme

using

unicast

messages

The

results

demonstrate

the

benefits

of

the

hardware-based

multicast

schemes

compared

to

the

software-based

scheme

Furthermore

with

equal

amount

of

buffer

space

in

a

switch

the

central-queue-based

implementation

shows

an

improvement

up

to

a

factor

of

-LCB-

compared

to

the

input-buffer-based

implementation

These

results

clearly

demonstrate

the

superiority

of

central-queue-based

scheme

to

support

multidestination

worms

for

fast

multicast/broadcast

on

switch-based

parallel

systems

The

paper

is

organized

as

follows

Section

provides

an

overview

of

switch-based

parallel

systems

Issues

related

to

implementing

multidestination

worms

are

presented

in

Section

Detailed

designs

to

implement

multidestination

worms

on

central-queue-based

and

input-buffer-based

switch

architectures

are

presented

in

Sections

and

respectively

A

comparison

of

these

two

implementations

is

carried

out

in

Section

Section

shows

performance

evaluation

results

Related

work

is

reviewed

in

Section

Finally

concluding

remarks

and

future

works

are

presented

Switch

Based

Parallel

Systems

In

this

section

we

briefly

describe

the

various

categories

of

switch

based

parallel

systems

and

define

the

scope

of

this

work

Switch

based

parallel

systems

fall

into

two

broad

classes

-LRB-

i

-RRB-

systems

based

on

regular

interconnects

and

-LRB-

ii

-RRB-

systems

based

on

irregular

interconnects

Communication

and

synchronization

issues

on

parallel

systems

based

on

regular

interconnects

have

been

studied

quite

extensively

Architectures

using

unidirectional

multistage

interconnection

networks

-LRB-

MINs

-RRB-

and

bidirectional

MINs

or

fat-tree

networks

-LSB-

-RSB-

fall

into

this

category

In

such

networks

the

regular

pattern

of

interconnection

between

switches

allows

for

simple

routing

functions

to

be

defined

for

routing

messages

between

nodes

In

unidirectional

MINs

messages

must

traverse

all

switch

stages

to

reach

the

destination

processors

In

bidirectional

MINs

on

the

other

hand

turnaround

routing

can

be

used

which

optimizes

communication

between

processors

connected

to

the

same

switch

Figures

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

show

examples

of

unidirectional

and

bidirectional

MIN

networks

respectively

Another

class

of

switch

based

parallel

system

that

is

becoming

popular

is

networks

of

workstations

-LRB-

NOWs/COWs

-RRB-

These

system

typically

consist

of

workstations

interconnected

with

switches

Processing

Elements

Switching

Elements

-LRB-

Switches

-RRB-

Interconnection

Pattern

Unidirectional

Links

Omega

Butterfly

Cube

Baseline

-LRB-

a

-RRB-

Unidirectional

MIN

Links

Bidirectional

Switches

-LRB-

Unidirectional

Switches

-RRB-

Turnaround

Routing

-LRB-

b

-RRB-

Bidirectional

MIN

Processing

Element

-LRB-

Workstation

-RRB-

Switching

Elements

Bidirectional

Links

-LRB-

c

-RRB-

Irregular

Interconnect

Figure

Examples

of

switch

based

parallel

systems

-LRB-

such

as

Myrinet

-LSB-

-RSB-

-RRB-

in

irregular

topologies

Deadlock

free

routing

of

messages

in

such

networks

is

a

challenging

problem

-LSB-

-RSB-

Figure

-LRB-

c

-RRB-

shows

an

example

of

an

irregular

network

of

workstations

The

schemes

and

designs

proposed

in

this

paper

are

applicable

to

all

categories

of

switch

based

parallel

systems

However

for

performance

comparisons

we

restrict

ourselves

to

bidirectional

MIN

topologies

In

the

next

section

we

introduce

the

concept

of

multidestination

worms

in

the

context

of

switch

based

parallel

systems

and

discuss

some

of

the

issues

to

be

considered

in

implementing

them

Issues

in

Implementing

Multidestination

Worms

The

path

based

multicast

proposed

for

direct

networks

-LSB-

-RSB-

has

been

shown

to

lead

to

deadlock

in

unidirectional

MINs

-LSB-

-RSB-

Even

without

this

constraint

path

based

multicast

is

highly

inefficient

because

the

network

has

to

be

traversed

multiple

times

and

flits

of

the

message

have

to

be

copied

and

forwarded

by

the

network

interface

associated

with

the

nodes

An

alternative

method

for

implementing

multidestination

worms

is

to

augment

switches

with

replication

capability

A

switch

replicates

flits

of

an

arriving

multidestination

worm

to

one

or

more

of

its

output

ports

In

strict

or

pure

wormhole

routing

switches

buffer

only

a

single

flit

of

an

arriving

worm

and

can

not

accept

the

next

flit

till

the

buffer

is

freed

For

a

multidestination

worm

that

has

to

be

forwarded

to

multiple

output

ports

this

buffer

can

not

be

freed

till

the

switch

can

forward

the

flit

to

all

-LRB-

required

-RRB-

output

ports

Thus

deadlock

can

easily

occur

if

two

multidestination

packets

that

have

to

be

forwarded

to

the

same

set

of

output

ports

arrive

at

a

switch

with

one

of

the

worms

reserving

a

subset

of

the

output

ports

and

the

other

worm

reserving

the

rest

This

deadlock

problem

exists

even

if

the

switch

can

buffer

more

than

one

flit

Figure

presents

an

example

of

such

a

deadlock

scenario

In

the

next

section

we

describe

two

replication

mechanisms

that

have

been

proposed

in

the

literature

and

the

methods

that

they

adopt

to

overcome

this

deadlock

problem

Deadlock

Scenario

ReservedOutputPortsRequestedOutputPorts

Figure

Typical

case

of

deadlock

for

multidestination

worms

that

use

replication

at

the

switches

Deadlock-free

Replication

Two

replication

methods

have

been

proposed

in

the

literature

Synchronous

replication

requires

that

flits

of

a

multidestination

worm

proceed

in

lock-step

Thus

any

branch

of

the

multidestination

worm

that

is

blocked

can

block

all

other

branches

This

replication

mechanism

was

originally

proposed

in

the

context

of

pure

wormhole

routing

and

is

susceptible

to

deadlock

Furthermore

it

requires

some

kind

of

feedback

architecture

to

ensure

that

the

flits

proceed

in

lock-step

-LSB-

-RSB-

Asynchronous

replication

-LSB-

-RSB-

is

an

alternative

mechanism

that

allows

flits

of

a

multidestination

worm

to

be

forwarded

to

the

subset

of

requested

output

ports

that

the

worm

successfully

reserves

If

a

switch

is

equipped

with

a

buffer

of

size

f

flits

under

asynchronous

replication

at

most

f

flits

can

be

forwarded

to

the

output

ports

that

have

been

reserved

by

the

multidestination

worm

before

the

worm

is

blocked

because

of

the

required

output

ports

which

it

could

not

reserve

This

is

because

of

the

f

flits

in

the

buffer

can

be

freed

as

they

have

n't

been

forwarded

to

all

of

the

required

output

ports

and

more

flits

can

not

enter

the

full

buffer

Bubbles

are

therefore

introduced

if

the

remaining

requested

output

ports

can

not

be

obtained

before

the

input

buffer

fills

up

Even

this

method

of

replication

is

susceptible

to

deadlock

To

prevent

deadlock

under

synchronous

replication

deadlock

avoidance

schemes

have

been

proposed

that

arbitrate

between

multidestination

packets

at

a

switch

to

prevent

cyclic

wait

-LSB-

-RSB-

To

prevent

deadlock

under

asynchronous

replication

switches

must

be

equipped

with

buffers

large

enough

to

store

the

largest

packet

in

the

system

Deadlock

is

prevented

if

the

switches

can

guarantee

that

a

packet

accepted

for

transmission

can

be

eventually

completely

buffered

at

the

switch

-LSB-

-RSB-

a

requirement

that

is

weaker

than

virtual

cut-through

-LSB-

-RSB-

Asynchronous

replication

may

be

preferred

for

a

practical

implementation

due

to

the

following

reasons

Firstly

it

does

not

require

the

costly

feedback

architecture

required

under

synchronous

replication

Secondly

it

is

more

efficient

because

blocked

branches

do

n't

block

other

branches

-LRB-

in

fact

there

is

no

dependence

across

branches

-RRB-

Finally

it

has

been

shown

that

virtual

cut-throughlike

mechanisms

can

deliver

better

performance

than

architectures

with

wormhole

routing

-LSB-

-RSB-

Many

current

day

switches

already

provide

for

fairly

large

buffers

at

the

switches

-LSB-

-RSB-

making

the

satisfaction

of

the

deadlock-freedom

requirement

under

asynchronous

replication

easier

In

this

paper

we

present

two

alternative

implementations

of

multidestination

worms

with

asynchronous

replication

The

two

implementations

differ

primarily

in

the

way

they

guarantee

than

an

arriving

multidestination

worm

can

eventually

be

completely

buffered

One

of

the

implementations

uses

a

central

queue-based

switch

architecture

This

architecture

uses

an

output

buffering

technique

similar

to

the

ones

used

in

switches

of

the

IBM

-LSB-

-RSB-

for

unicast

message

passing

The

other

implementation

uses

an

input

buffer-based

switch

architecture

Detailed

designs

to

implement

multidestination

worms

on

these

two

switch

architectures

are

discussed

in

Section

and

respectively

In

the

remaining

part

of

this

section

we

discuss

two

additional

issues

related

to

implementing

multidestination

worms

header

encoding

and

routing

Header

Encoding

To

route

a

multidestination

message

a

switch

must

be

equipped

with

a

method

for

determining

the

output

ports

to

which

a

multicast

message

must

be

forwarded

The

multidestination

worm

header

encodes

information

that

helps

the

switch

decipher

this

information

A

variety

of

header

encoding

mechanisms

have

been

proposed

in

the

literature

These

schemes

differ

in

the

following

characteristics

size

of

the

header

required

for

encoding

knowledge

required

at

the

switches

about

the

system

topology

complexity

and

speed

of

the

decoding

logic

required

at

the

switches

and

the

multicast

sets

that

can

be

covered

using

these

schemes

-LSB-

-RSB-

It

must

be

emphasized

that

the

encoding

and

decoding

mechanism

used

for

routing

a

multidestination

worm

is

independent

of

the

scheme

adopted

for

replicating

a

multidestination

worm

at

a

switch

the

two

issues

are

completely

orthogonal

In

-LSB-

-RSB-

we

have

proposed

a

multiport

encoding

scheme

for

multidestination

worms

This

encoding

mechanism

requires

extremely

simple

decoding

logic

at

the

switches

In

addition

switches

are

not

required

to

know

the

topology

of

the

MIN

network

Depending

on

the

network

topology

the

source

node

encodes

the

header

of

a

multidestination

worm

based

on

the

path

that

needs

to

be

followed

to

reach

a

set

of

destinations

However

all

arbitrary

multicast

destination

sets

can

not

be

encoded

in

a

single

worm

using

multiport

encoding

Instead

given

an

arbitrary

multicast

destination

set

a

set

of

multiport

encoded

worms

are

required

to

cover

the

destinations

A

multi

phase

multinomial

tree

based

approach

is

adopted

to

perform

multicast

using

this

set

of

multiport

encoded

worms

Even

though

this

scheme

performs

better

than

software-based

multicast

using

unicast

message

passing

a

multi-phase

implementation

is

not

desirable

for

system

level

multicasts

such

as

cache

invalidation

traffic

in

distributed

shared

memory

systems

This

is

because

with

cache

invalidation

only

the

source

node

has

information

about

the

destination

set

-LSB-

-RSB-

Thus

a

single

phase

implementation

may

be

preferable

in

such

systems

perhaps

at

a

slight

additional

decoding

logic

cost

One

form

of

header

encoding

that

accomplishes

multicast

to

arbitrary

multicast

destination

sets

in

a

single

communication

phase

is

known

as

bit-string

encoding

-LSB-

-RSB-

A

multidestination

worm

with

a

bit-string

encoded

header

carries

an

encoding

of

the

destinations

which

must

receive

the

message

The

encoding

consists

of

N

bits

where

N

is

the

system

size

with

a

bit

in

the

ith

bit

position

indicating

that

processor

i

is

a

multicast

destination

To

decode

a

bit-string

encoded

header

a

switch

must

possess

knowledge

about

the

processors

that

can

be

reached

through

each

of

its

output

ports

This

reachability

information

-LSB-

-RSB-

can

be

encoded

using

a

similar

N

bit

string

for

each

output

port

with

bits

denoting

processors

that

are

reachable

via

the

output

port

Such

an

N-bit

string

is

associated

with

each

output

port

For

some

unidirectional

MINs

the

space

required

to

store

this

information

can

be

reduced

by

taking

into

account

the

fact

that

a

processor

may

be

reached

from

only

one

of

the

switch

's

output

ports

because

of

the

unique

path

property

possessed

by

some

of

these

networks

The

reachability

information

can

be

set

up

at

the

switches

at

the

time

of

system

startup

or

reconfiguration

Figure

shows

a

bidirectional

MIN

network

along

with

the

reachability

information

associated

with

some

of

the

output

ports

The

figure

also

shows

a

bit

string

encoded

header

and

the

path

followed

by

the

corresponding

multidestination

worms

1111 0000 0000 0000

0000 1111 0000 0000

0000 0000 1111 0000

0000 0000 0000 1111

1000 0000 0000 0000

0100 0000 0000 0000

0010 0000 0000 0000

0001 0000 0000 0000

0000 0110 0000 1101

0000 1111 1111 1111

Figure

An

example

of

a

bidirectional

MIN

network

with

processors

bit-string

encoded

header

with

the

path

of

the

corresponding

multidestination

worm

and

the

reachability

strings

associated

with

some

of

the

output

ports

When

a

multidestination

worm

with

a

bit-string

encoded

header

arrives

at

a

switch

the

switch

compares

the

bit-string

in

the

header

with

the

reachability

information

associated

with

every

one

of

its

output

ports

The

output

ports

required

by

the

worm

can

thus

be

determined

The

logic

required

to

compare

these

bit-strings

is

quite

simple

and

thus

we

do

not

present

it

in

detail

Routing

Although

the

bit-string

encoding

presented

in

the

previous

section

allows

a

multidestination

worm

to

get

replicated

along

a

fixed

path

from

a

source

to

its

destinations

the

presence

of

alternative

paths

between

nodes

in

some

switch-based

networks

may

not

be

adequately

exploited

using

the

bitstring

encoding

alone

In

networks

like

the

bidi-MINs

there

exists

considerable

choice

in

the

way

a

multidestination

message

can

get

routed

A

multidestination

message

could

replicate

downwards

on

its

forward

path

while

going

to

the

least

common

ancestor

-LRB-

LCA

-RRB-

-LSB-

-RSB-

stage

of

the

source

and

destinations

and

then

cover

the

remaining

destinations

by

replicating

on

the

way

back

from

the

LCA

stage

Alternatively

it

could

just

travel

to

the

LCA

stage

and

then

cover

all

the

destinations

by

replicating

on

the

way

back

-LSB-

-RSB-

Furthermore

since

the

same

set

of

destinations

can

be

reached

from

any

switch

in

the

LCA

stage

choice

also

exists

in

determining

the

path

taken

upto

the

LCA

stage

one

can

decide

to

deterministically

route

messages

to

the

LCA

stage

or

to

make

the

choice

adaptively

In

this

paper

we

assume

that

multidestination

worms

travel

adaptively

upto

the

LCA

stage

and

that

the

destinations

are

covered

by

replicating

on

the

downward

path

from

the

LCA

stage

In

addition

to

the

bit-string

encoded

header

the

worm

may

carry

a

count

field

which

is

initialized

to

the

LCA

stage

The

count

is

decremented

at

each

switch

on

the

worm

's

forward

path

and

the

worm

begins

its

backward

journey

when

the

count

value

becomes

zero

-LRB-

this

field

may

then

be

discarded

-RRB-

It

is

to

be

noted

that

such

choices

of

path

do

not

exist

for

unidirectional

MINs

with

the

unique

path

property

For

irregular

topologies

routing

can

be

performed

much

like

the

routing

described

for

bidiMINs

by

assuming

a

tree

structure

superimposed

on

the

irregular

network

This

tree

structure

is

already

being

used

in

irregular

networks

to

provide

deadlock-free

routing

of

unicast

messages

In

the

up

down

-LSB-

-RSB-

routing

algorithm

for

example

a

tree

structure

is

imposed

on

the

networks

and

links

are

uniquely

identified

as

up

links

or

down

links

depending

on

whether

they

take

you

closer

or

farther

away

from

the

root

Such

a

tree

can

be

used

to

determine

reachability

information

for

the

down

links

of

the

switches

Multidestination

routing

can

then

be

performed

by

adaptively

routing

a

message

to

the

root

by

following

up

links

and

then

replicating

the

message

on

the

down

path

at

the

intermediate

switches

using

the

reachability

information

associated

with

each

of

the

switches

down

links

In

this

section

we

have

described

some

of

the

issues

with

respect

to

implementing

multidestination

worms

in

switch

based

architectures

In

the

next

two

sections

we

discuss

how

asynchronous

replication

of

multidestination

worms

can

be

implemented

on

two

alternative

switch

architectures

output

buffer

-LRB-

central-queue

-RRB-

based

and

input-buffer

based

Output

Buffer

-LRB-

Central

Queue

-RRB-

based

Switch

Architecture

We

now

describe

a

switch

architecture

with

output

buffering

modified

for

implementing

multidestination

worms

We

first

describe

an

output

buffer

based

switch

architecture

very

similar

to

the

high

performance

switch

in

the

IBM

-LSB-

-RSB-

We

then

present

modifications

that

could

be

made

to

this

architecture

to

implement

multidestination

worms

Buffered

Wormhole

Routing

The

buffered

wormhole

routing

used

in

the

IBM

is

a

variation

of

wormhole

routing

wherein

every

switch

in

the

network

is

equipped

with

a

central

buffer

and

a

crossbar

as

illustrated

in

Figure

When

a

packet

arrives

at

an

input

port

and

encounters

no

contention

for

the

chosen

output

Figure

A

switch

equipped

with

a

central

buffer

port

the

packet

uses

the

crossbar

path

for

minimal

delay

However

when

packets

are

blocked

at

an

input

port

due

to

a

busy

output

port

the

switch

attempts

to

store

the

packet

in

this

central

buffer

thus

freeing

the

links

held

by

the

trailing

packet

flits

There

may

be

enough

space

in

the

central

buffer

to

store

the

entire

packet

However

there

is

no

guarantee

that

a

packet

arriving

at

a

switch

will

find

enough

space

in

the

central

buffer

to

be

completely

stored

If

the

central

buffer

does

not

have

adequate

space

to

store

the

entire

blocked

packet

as

many

as

possible

of

the

packet

flits

are

stored

in

the

central

buffer

and

the

remainder

of

the

packet

is

blocked

in

place

Note

that

in

the

absence

of

contention

packets

may

propagate

through

the

network

just

as

in

a

purely

wormhole

routed

network

and

the

central

buffers

will

remain

empty

An

central

buffer

is

an

extremely

attractive

resource

for

packet

replication

because

multiple

output

ports

can

retrieve

the

identical

packet

flits

from

the

same

shared

buffer

However

because

there

is

no

assurance

that

this

central

buffer

can

store

an

entire

multidestination

packet

the

central

buffer

as

described

can

not

guarantee

to

prevent

multicast

deadlock

To

address

this

problem

we

will

describe

minor

modifications

to

the

basic

central

buffer

free-space

logic

that

are

similar

to

virtual

cut-through

operation

Specifically

these

changes

guarantee

that

any

packet

admitted

to

the

central

buffer

can

-LRB-

eventually

-RRB-

be

entirely

stored

This

guarantee

effectively

decouples

the

interdependence

of

the

replicated

output

packets

at

a

switch

eliminating

the

cause

of

multicast

wormhole

routing

deadlock

In

the

buffered

wormhole

implementation

the

central

buffer

effectively

forms

a

separate

FIFO

queue

of

packets

for

each

output

port

Each

input

port

can

write

flits

into

the

buffer

and

each

output

port

can

read

flits

Central

buffer

space

is

dynamically

allocated

to

requesting

input

ports

in

a

fair

manner

For

a

k-port

switch

k

flits

are

buffered

-LRB-

deserialized

-RRB-

by

an

input

port

into

a

k-flit

chunk

before

being

written

into

the

central

buffer

These

k-flit

chunks

are

read

from

the

central

buffer

and

are

disassembled

-LRB-

serialized

-RRB-

into

k

flits

again

by

the

intended

output

port

This

reduces

the

number

of

central

buffer

RAM

read

and

write

ports

required

As

an

example

in

the

switches

up

to

flit

is

received

-LRB-

transmitted

-RRB-

at

each

input

-LRB-

output

-RRB-

port

every

cycle

An

chunk

is

therefore

flits

and

the

central

buffer

only

requires

RAM

write

port

and

RAM

read

port

to

match

the

input

and

output

bandwidth

of

the

switch

In

general

for

a

switch

with

k

input

and

k

output

ports

a

chunk

size

of

k

flits

is

required

for

a

RAM

to

match

the

input

and

output

bandwidth

of

the

switch

The

central

buffer

logic

maintains

a

list

of

free

chunk

locations

A

central

buffer

write

allocates

a

free

chunk

and

a

read

returns

a

free

chunk

There

must

be

a

mechanism

we

will

call

it

the

next-packet

list

to

order

the

packets

within

each

packet

queue

Each

packet

is

divided

into

chunks

and

thus

there

is

also

a

mechanism

the

next-chunk

list

to

order

the

chunks

within

a

packet

Here

we

describe

the

next-packet

lists

The

next-chunk

lists

are

similar

although

slightly

more

complex

To

record

these

two

types

of

linking

information

two

pointer

fields

are

associated

with

each

chunk

of

data

the

next-packet

-LRB-

NP

-LSB-

-RSB-

-RRB-

field

and

the

next-chunk

-LRB-

NC

-LSB-

-RSB-

-RRB-

field

-LRB-

see

Figure

-RRB-

In

ChunkChunkChunkChunkDataNPNC

Figure

Organization

of

a

central

buffer

addition

each

output

port

o

maintains

first-packet

-LRB-

firstP

-LSB-

o

-RSB-

-RRB-

and

last-packet

-LRB-

lastP

-LSB-

o

-RSB-

-RRB-

pointers

into

its

packet

queue

and

a

first-chunk

-LRB-

firstC

-LSB-

o

-RSB-

-RRB-

field

that

points

to

the

next

chunk

to

be

read

if

output

port

o

has

not

read

the

last

chunk

of

the

current

packet

Each

input

port

i

maintains

a

last-chunk

-LRB-

lastC

-LSB-

i

-RSB-

-RRB-

field

that

points

to

the

last

chunk

written

by

input

port

i.

All

pointers

are

assumed

to

be

when

invalid

In

the

following

discussion

we

shall

assume

input

port

i

is

writing

chunks

to

output

port

o

The

next-packet

list

is

updated

each

time

the

first

chunk

-LRB-

the

header

chunk

-RRB-

of

a

packet

is

written

If

no

packets

are

currently

on

the

destination

output

port

's

packet

queue

-LRB-

firstP

-LSB-

o

-RSB-

-RRB-

then

firstP

-LSB-

o

-RSB-

writeloc

where

writeloc

is

the

address

where

the

header

is

written

Otherwise

NP

-LSB-

lastP

-LSB-

o

-RSB-

-RSB-

writeloc

The

last-packet

pointer

is

updated

-LRB-

lastP

-LSB-

o

-RSB-

writeloc

-RRB-

and

the

packet-list

is

terminated

-LRB-

NP

-LSB-

writeloc

-RSB-

-RRB-

The

logical

structure

of

a

typical

output

port

queue

within

the

central

buffer

is

shown

in

Figure

There

are

two

packets

shown

each

with

its

associated

chunks

displayed

in

a

column

The

lightly-shaded

fields

indicate

fields

that

are

not

currently

valid

-LRB-

e.g.

next-packet

fields

are

not

used

except

for

header

chunks

-RRB-

When

a

header

chunk

is

read

from

the

central

buffer

the

next-packet

list

must

also

be

updated

-LRB-

firstP

-LSB-

o

-RSB-

NP

-LSB-

readloc

-RSB-

-RRB-

It

should

be

evident

that

the

order

of

packets

on

a

queue

is

entirely

determined

by

the

order

of

header

chunk

writes

The

next-chunk

fields

provide

a

similar

linking

function

between

packet

chunks

On

a

write

DataNPNCDataNPNCDataNPNCDataNPNCDataNPNCDataNPNCDataNPNCfirstPfirstCla

stP

Figure

Structure

of

an

output

port

queue

within

the

central

buffer

when

a

valid

last-chunk

pointer

exists

the

central

buffer

next-chunk

location

pointed

to

by

lastchunk

is

updated

with

the

location

of

the

currently

written

chunk

-LRB-

if

lastC

-LSB-

i

-RSB-

then

NC

-LSB-

lastC

-LSB-

i

-RSB-

-RSB-

writeloc

-RRB-

When

an

input

port

writes

a

chunk

it

also

updates

its

last-chunk

pointer

with

the

write

location

-LRB-

lastC

-LSB-

i

-RSB-

writeloc

-RRB-

The

one

exception

is

a

write

of

the

last

chunk

of

a

packet

in

this

case

the

last-chunk

pointer

becomes

invalid

-LRB-

lastC

-LSB-

i

-RSB-

-RRB-

On

the

output

port

side

except

for

when

a

header

chunk

is

being

read

the

output

port

firstchunk

field

is

used

to

determine

the

location

of

the

next

central

buffer

chunk

read

-LRB-

readloc

firstC

-LSB-

o

-RSB-

-RRB-

For

header

chunk

reads

the

first-packet

pointer

is

used

-LRB-

readloc

firstP

-LSB-

o

-RSB-

-RRB-

On

every

chunk

read

the

output

port

's

first-chunk

pointer

is

updated

with

the

associated

central

buffer

next-chunk

pointer

-LRB-

firstC

-LSB-

o

-RSB-

NC

-LSB-

readloc

-RSB-

-RRB-

Figure

shows

the

structure

of

the

queue

from

Figure

after

the

first

two

chunks

of

the

first

packet

in

the

queue

have

been

read

by

the

output

port

Note

that

firstP

and

firstC

have

been

updated

and

firstC

is

now

a

valid

pointer

field

required

for

retrieving

the

next

chunk

from

the

queue

The

Shared

Central

Buffer

Replication

Method

We

desire

a

replication

method

that

requires

only

one

copy

of

the

packet

-LRB-

l

flits

-RRB-

to

be

transmitted

from

the

input

port

and

a

method

which

acquires

the

deadlock-avoidance

advantages

of

virtual

cut-through

operation

This

goal

can

be

achieved

via

the

central

buffer

with

replication

occurring

during

the

read

of

the

chunk

out

of

the

central

buffer

by

the

output

port

In

this

single-copy

replication

method

the

input

port

writes

each

chunk

into

the

central

buffer

once

but

also

initializes

an

associated

counter

c

to

ks

where

ks

is

the

degree

of

replication

required

at

that

switch

When

an

output

port

reads

the

chunk

from

the

central

buffer

it

checks

c

If

c

then

the

chunk

is

thrown

away

Otherwise

c

is

decremented

The

relatively

large

size

of

a

chunk

minimizes

the

impact

of

adding

a

counter

c

to

the

storage

space

for

each

central

buffer

chunk

The

multiple-flit

size

of

a

chunk

provides

another

striking

advantage

for

this

method

an

input

port

can

write

a

multidestination

packet

into

the

central

buffer

at

full

bandwidth

while

simultane

DataNPNCDataNPNCDataNPNCDataNPNCDataNPNCfirstPfirstClastP

Figure

Structure

of

the

same

output

port

queue

after

two

chunk

reads

ously

all

ks

output

ports

are

reading

the

packet

at

full

bandwidth

Thus

latency

and

required

buffer

space

are

minimized

We

have

established

the

motivation

for

the

single-copy

replication

method

in

the

next

section

we

examine

the

implementation

issues

Replication

Implementation

In

this

section

we

describe

modifications

to

the

basic

buffer

wormhole

strategy

that

provide

efficient

single-copy

replication

The

basic

change

is

to

provide

a

counter

c

with

each

chunk

that

indicates

the

number

of

output

ports

that

have

not

yet

read

the

chunk

from

the

central

buffer

as

introduced

in

the

preceding

section

However

there

remains

a

problem

with

the

next-packet

lists

If

a

single

header

chunk

is

written

to

the

central

buffer

then

there

exists

only

a

single

nextpacket

pointer

associated

with

this

header

chunk

Therefore

even

if

the

last-packet

and

appropriate

next-packet

pointers

are

updated

for

every

destination

output

port

the

next-packet

lists

will

be

converged

an

unacceptable

situation

One

solution

is

to

write

ks

copies

of

the

header

chunks

one

for

each

destination

output

port

This

immediately

provides

a

distinct

next-packet

pointer

for

each

queue

maintaining

their

disjoint

nature

All

other

chunks

of

the

multidestination

packet

are

written

once

This

method

requires

ks

more

chunks

of

central

buffer

space

Figure

illustrates

this

solution

Output

ports

x

and

y

share

the

same

packet

at

the

start

of

their

queues

yet

their

next-packet

lists

remain

separate

Emulating

Virtual

Cut-Through

As

explained

before

the

equivalent

of

virtual

cut-through

-LRB-

VCT

-RRB-

operation

is

required

to

avoid

the

dependence

between

output

ports

which

can

lead

to

multicast

deadlock

VCT

designs

perform

flow-control

on

a

packet

basis

allowing

transmission

of

a

packet

from

a

switch

only

when

the

entire

packet

can

be

buffered

at

the

downstream

switch

or

node

Wormhole

flow-control

designs

can

implementation

need

not

strictly

follow

this

convention

For

instance

c

could

initially

be

set

to

ks

in

which

case

if

c

then

the

last

output

port

is

reading

the

chunk

This

choice

might

allow

c

to

be

implemented

with

less

bit

firstPfirstPfirstCfirstClastPlastPDataNPNCc

aNPNCc

port

xOutput

port

y

Figure

Replicated

header

chunks

pointing

to

the

same

second

chunk

be

augmented

to

provide

the

aspects

of

VCT

that

are

essential

for

multicasting

given

sufficient

buffering

capability

within

each

switch

To

use

a

central

buffer

for

emulating

VCT

the

total

central

buffer

size

must

as

large

or

larger

than

the

largest

packet

to

be

buffered

When

a

multicast

packet

is

replicated

at

a

switch

each

chunk

of

this

packet

must

be

stored

in

the

central

buffer

before

being

read

by

all

destination

output

ports

The

packet

header

chunk

is

not

allowed

to

enter

the

central

buffer

until

there

is

a

guarantee

that

the

entire

packet

will

eventually

obtain

space

within

the

central

buffer

This

does

not

necessarily

require

that

space

for

the

entire

packet

exists

prior

to

writing

the

header

chunk

When

the

header

chunk

is

written

-LRB-

at

least

before

any

other

chunk

from

any

input

port

is

subsequently

written

-RRB-

the

required

number

of

chunks

are

reserved

for

use

by

that

multicast

packet

only

In

designs

that

maintain

a

free

chunk

counter

this

is

most

easily

accomplished

by

decrementing

the

free

space

count

by

the

total

number

of

multicast

packet

chunks

to

be

written

To

summarize

although

the

normal

wormhole

switch-to-switch

flow-control

is

used

between

switches

the

multicast

packet

header

chunk

is

prevented

from

entering

the

central

buffer

until

the

entire

multicast

is

guaranteed

to

fit

within

this

buffer

This

is

a

weaker

requirement

than

VCT

and

is

only

applied

to

multicast

packets

which

are

to

be

replicated

within

that

switch

Thus

multidestination

worms

can

be

implemented

in

central

queue

based

switch

architectures

with

very

little

additional

hardware

support

Input

Buffer

based

Switch

Architecture

In

this

section

we

describe

an

input

buffer

based

switch

architecture

for

implementing

multidestination

worms

-LSB-

-RSB-

The

crucial

step

is

to

increase

the

size

of

the

input

buffers

in

the

switch

so

that

they

are

larger

than

the

largest

multicast

packet

allowed

in

the

system

Any

multicast

packet

that

the

designer

can

identify

cases

in

which

chunks

currently

within

the

central

buffer

are

guaranteed

to

be

read

and

freed

then

these

chunks

may

be

able

to

be

counted

toward

the

available

buffer

space

These

cases

will

vary

from

design

to

design

and

may

also

be

topology-dependent

arrives

at

an

input

port

is

thus

guaranteed

to

be

completely

buffered

eventually

-LRB-

the

packet

at

the

head

of

the

buffer

will

definitely

be

eventually

freed

implying

that

packets

behind

it

will

definitely

progress

one

by

one

to

the

head

of

the

buffer

-RRB-

Just

as

with

the

previous

architecture

flits

of

an

arriving

multidestination

worm

are

assembled

into

chunks

and

stored

in

a

chunk-wide

RAM

that

constitutes

the

input

buffer

When

a

multidestination

worm

arrives

at

the

switch

a

count

value

is

associated

with

every

one

of

its

chunks

The

switch

decodes

the

multidestination

worm

header

to

determine

the

output

ports

to

which

the

worm

should

be

forwarded

A

request

is

then

enqueued

for

each

of

these

output

ports

and

the

idle

output

ports

begin

reading

chunks

of

the

message

from

the

appropriate

input

ports

As

a

chunk

is

read

the

count

value

associated

with

it

is

decremented

and

the

chunk

is

discarded

once

all

output

ports

have

read

it

The

chunk

that

is

read

is

stored

in

an

output

buffer

from

where

it

is

transmitted

onto

the

next

link

on

a

flit

by

flit

basis

Note

that

the

header

chunk

is

also

read

by

each

of

the

output

ports

as

part

of

the

message

every

branch

of

the

multidestination

worm

carries

the

same

bit

string

header

implying

that

no

header

manipulation

needs

to

be

done

at

the

intermediate

switches

As

discussed

in

the

previous

section

chunking

allows

output

ports

to

maintain

a

flit

per

cycle

rate

while

requiring

only

a

port

-LRB-

read

write

-RRB-

RAM

Figure

shows

a

snapshot

of

a

multidestination

worm

with

asynchronous

replication

making

progress

in

an

input

buffer

based

switch

architecture

In

the

figure

the

multidestination

worm

enqueues

a

request

for

a

busy

port

while

the

ports

that

are

available

read

chunks

of

the

worm

from

the

input

buffer

simultaneously

decreasing

the

associated

count

value

The

ports

take

turns

reading

from

the

input

buffer

Request

Enqueued

Available

Ports

Proceed

Multidestination

Worm

Unicast

Worm

Output

Ports

read

chunks

Chunks

Switch

Crossbar

Count

values

Figure

Snapshot

of

a

multidestination

worm

making

progress

through

a

switch

with

an

input

buffer

based

architecture

An

implication

from

the

above

description

is

that

a

chunk

is

transmitted

in

a

single

cycle

across

the

switch

crossbar

This

would

require

a

chunk-wide

crossbar

in

a

naive

implementation

which

would

increase

the

wiring

complexity

within

the

switch

chip

To

alleviate

this

problem

we

have

proposed

alternative

implementations

which

perform

as

well

as

an

implementation

with

a

chunkwide

crossbar

-LSB-

-RSB-

These

implementations

pipeline

the

transfer

of

flits

in

a

chunk

from

the

input

to

output

by

either

-LRB-

i

-RRB-

using

k

separate

flit-wide

RAMs

instead

of

a

single

chunk-wide

RAM

-LRB-

where

k

is

the

number

of

input

and

output

ports

in

the

switch

which

defines

the

chunk

size

as

described

before

-RRB-

or

-LRB-

ii

-RRB-

using

a

series

of

registers

to

pipeline

the

transfer

of

the

flits

from

the

chunk-wide

RAM

Interested

readers

are

requested

to

refer

to

-LSB-

-RSB-

for

further

details

Another

important

design

decision

in

this

architecture

is

whether

packets

at

different

points

in

the

input

buffer

should

be

allowed

to

make

progress

or

whether

progress

should

be

restricted

to

the

packet

at

the

head

of

the

input

buffer

The

former

option

implies

that

a

fairly

complex

next

packet

list

must

be

associated

with

every

output

port

much

like

the

next

packet

list

described

in

the

context

of

the

central

queue

in

the

previous

section

While

this

list

is

associated

with

the

single

central

queue

in

the

previous

section

the

presence

of

multiple

input

buffers

complicates

the

structure

of

this

list

This

complexity

is

eliminated

if

only

packets

at

the

head

of

the

input

buffer

are

allowed

to

make

progress

an

output

port

need

only

know

which

input

port

it

must

serve

next

the

location

to

read

from

to

begin

transfer

is

implicit

However

there

may

be

a

performance

penalty

due

to

head-of-the-line

blocking

Since

our

preliminary

simulation

results

showed

little

difference

between

the

two

schemes

especially

in

terms

of

saturation

loads

we

decided

to

adopt

the

approach

with

packets

being

processed

in

FIFO

order

for

our

study

Such

an

approach

also

simplifies

the

design

Comparing

the

two

architectures

In

this

section

we

qualitatively

examine

the

two

proposed

architectures

before

comparing

them

to

software

multicasting

in

the

next

section

Both

schemes

will

benefit

from

requiring

less

aggregate

links

for

a

multicast

compared

to

the

-LRB-

m

-RRB-

e

messages

required

for

the

software

multicast

The

central

buffer

approach

is

significantly

more

complex

thus

it

is

not

worth

considering

unless

it

is

expected

to

provide

better

performance

It

is

well-known

that

shared

central

buffers

provide

superior

performance

for

most

unicast

traffic

-LSB-

-RSB-

The

central

buffer

based

architecture

provides

a

dynamically-shared

resource

for

input

and

output

ports

In

contrast

the

input

buffer

implementation

statically

and

evenly

divides

the

available

storage

among

the

input

ports

If

the

load

among

the

input

ports

is

unbalanced

the

central

queue

based

scheme

is

likely

to

benefit

because

the

ports

have

shared

access

to

a

larger

buffer

and

can

potentially

use

nearly

the

entire

space

available

even

when

some

or

all

of

the

intended

output

ports

are

busy

with

other

traffic

In

a

bidi-MIN

network

multidestination

worms

adaptively

travel

from

the

source

to

the

least

common

ancestor

and

then

turn

back

They

then

replicate

on

their

downward

path

tending

to

cause

more

blocking

on

the

right

input

ports

than

on

the

left

input

ports

and

a

centrally-buffered

scheme

can

adjust

to

this

imbalance

Another

expected

drawback

of

the

input

buffer

based

implementation

is

head-of-the-line

blocking

a

blocked

packet

at

the

head

of

the

FIFO

blocks

all

packets

behind

it

Multicast

is

likely

to

exacerbate

this

problem

as

the

packet

at

the

head

of

the

input

FIFO

will

not

be

dislodged

until

all

destination

output

ports

have

read

the

entire

packet

In

the

next

section

our

simulation

experiments

attempt

to

provide

quantify

these

comparisons

Performance

Evaluation

To

assess

the

performance

impact

of

input

FIFO

replication

and

central

buffer

replication

schemes

we

conducted

simulations

on

a

C++

CSIM-based

model

incorporating

these

options

To

judge

hardware

multicasting

versus

software

multicasting

methods

we

also

implemented

the

U-Min

algorithm

-LSB-

-RSB-

that

minimizes

contention

between

different

stages

of

a

software

multicast

Simulation

parameters

and

methodology

The

default

performance

parameters

and

the

central

buffer

model

were

based

roughly

on

a

contemporary

switch

the

SP

Switch

of

the

IBM

SP

systems

Each

switch

in

these

systems

has

input

ports

and

output

ports

where

input

and

output

ports

are

typically

paired

and

connected

to

another

switch

's

input/output

port

pair

over

a

bidirectional

link

Each

output

-LRB-

input

-RRB-

port

can

send

-LRB-

receive

-RRB-

a

flit

every

nsec

cycle

for

a

maximum

of

MB/s

of

bidirectional

traffic

over

the

link

For

our

simulations

we

count

latency

in

cycles

and

we

assume

flits

Minimum

latency

through

each

switch

is

assumed

to

be

cycles

with

cycles

required

for

reading

data

out

of

the

input

FIFO

determining

the

route

requesting

output

port

-LRB-

s

-RRB-

and

granting

the

request

For

input

port

FIFO

replication

or

for

traversing

the

switch

via

the

central

buffer

flits

are

deserialized

into

a

chunk

requiring

extra

cycles

for

the

chunks

As

in

the

SP

Switch

we

assume

the

central

buffer

accommodates

chunks

each

of

which

contains

eight

flits

for

a

total

of

Kbytes

of

storage

In

the

central

buffer

model

each

of

the

input

FIFOs

contains

flits

for

a

total

on-chip

packet

buffer

space

of

Kbytes

To

equalize

storage

between

the

central

buffer

and

input

queuing

models

we

assume

bytes

-LRB-

flits

-RRB-

of

storage

for

each

input

FIFO

in

the

input

queuing

model

We

assume

that

packet

flits

are

immediately

pulled

from

the

network

upon

arrival

at

a

node

We

simulated

random

traffic

for

and

messages

and

we

assume

that

both

of

these

message

sizes

fit

within

a

single

packet

For

an

m-way

multicast

simulation

on

an

n-node

system

for

each

message

we

randomly

choose

m

destinations

among

the

n

non-source

nodes

Message

transmission

times

for

each

node

are

exponentially

distributed

Latency

results

include

queuing

time

at

the

source

node

and

latency

curves

are

not

plotted

above

the

saturation

bandwidth

-LRB-

latency

is

infinite

in

a

stable

but

saturated

system

-RRB-

To

create

a

stable

environment

for

statistics

collection

we

typically

ran

the

simulations

with

-LCB-

cycles

-LRB-

or

more

-RRB-

of

cold

start

time

before

commencing

data

collection

Multicast

simulation

issues

Multicast

traffic

evaluation

raises

new

issues

beyond

unicast

traffic

The

definition

of

multicast

latency

is

problematic

latency

can

be

defined

as

-LRB-

a

-RRB-

the

latency

of

the

last

received

message

of

the

multicast

or

-LRB-

b

-RRB-

as

average

of

the

latencies

of

each

received

message

of

the

multicast

Nupairoj

and

Ni

-LSB-

-RSB-

argue

that

-LRB-

a

-RRB-

is

the

more

important

term

in

assessing

message-passing

collective

communication

performance

In

another

example

shared-memory

systems

multicasts

might

be

used

for

cache

line

invalidations

to

multiple

destinations

and

the

source

must

wait

for

the

last

received

-LRB-

and

acknowledged

-RRB-

invalidation

before

modifying

the

cache

line

Except

for

graphs

in

which

we

directly

compare

definition

-LRB-

a

-RRB-

with

-LRB-

b

-RRB-

we

choose

definition

-LRB-

a

-RRB-

for

recording

latency

A

second

issue

when

the

network

is

operating

below

the

saturation

bandwidth

for

m-way

hardware

multicasts

the

packet

injection

bandwidth

Bi

does

not

equal

the

packet

receive

bandwidth

Bo

Instead

Bo

mBi

Thus

mBi

is

effectively

the

load

being

injected

into

the

network

One

is

tempted

to

simply

plot

latency

versus

Bo

but

we

prefer

to

plot

performance

measures

against

a

stimulus

not

a

response

Most

of

our

graphs

display

effective

input

load

-LRB-

mBi

-RRB-

on

the

x-axis

Let

a

load

of

for

Bo

represent

full

bandwidth

received

at

each

node

Although

it

is

possible

that

mBi

a

stable

system

will

already

be

in

saturation

for

mBi

anyway

Thus

we

do

not

show

results

for

mBi

A

final

issue

is

message

startup

overhead

In

network

simulations

unicast

messages

are

typically

assumed

to

have

negligible

startup

overhead

With

this

assumption

the

network

can

easily

be

stressed

beyond

saturation

during

simulation

yielding

more

insights

into

its

behavior

In

implementing

software

multicast

most

real

systems

have

a

non-negligible

overhead

between

receiving

a

multicast

packet

and

forwarding

the

packet

along

the

next

stage

of

the

binomial

tree

In

our

simulations

we

zero

this

overhead

minimizing

the

latency

of

software

multicasting

to

give

it

the

best

chance

of

competing

against

hardware

multicasting

We

conducted

simulations

for

both

and

fat-trees

constructed

from

switching

elements

with

parents

and

children

-LRB-

ports

-RRB-

Our

default

message

size

is

bytes

-LRB-

flits

-RRB-

which

could

correspond

roughly

to

message

sizes

of

request

and

reply

packets

for

sharedmemory

systems

To

further

stress

the

network

we

also

plot

performance

for

messages

In

the

rest

of

this

section

we

discuss

the

effects

of

varying

a

number

of

input

loading

and

network

characteristics

Impact

of

multicast

degree

To

understand

the

effect

multicast

on

network

performance

we

begin

by

varying

the

degree

of

the

multicast

m

for

a

system

for

switches

with

and

without

central

buffers

The

left

graphs

in

Figure

compare

software

versus

hardware

multicasting

latency

using

centrally-buffered

switches

in

both

cases

As

m

increases

the

software

multicast

latency

increases

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Software

-LRB-

CQ

-RRB-

Multicasting

msgs

SW-CQ

SW-CQ

SW-CQ

SW-CQ

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Hardware

-LRB-

FIFO

-RRB-

Multicasting

msgs

HW-FIFO

HW-FIFO

HW-FIFO

HW-FIFO

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Software

-LRB-

CQ

-RRB-

Multicasting

msgs

SW-CQ

SW-CQ

SW-CQ

HW-CQ

HW-CQ

HW-CQ

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Hardware

-LRB-

FIFO

-RRB-

Multicasting

msgs

HW-FIFO

HW-FIFO

HW-FIFO

HW-CQ

HW-CQ

HW-CQ

Figure

Latency

vs.

Effective

applied

load

for

software

multicast

hardware

multicast

with

central

queue

and

hardware

multicast

with

input

buffering

and

saturates

more

quickly

The

hardware

multicast

latency

is

much

lower

and

increases

much

less

dramatically

The

right

plots

of

Figure

compare

our

two

proposed

hardware

replication

methods

For

small

m

input

FIFO

replication

actually

performs

worse

than

software

multicasting

with

a

central

buffer

This

follows

from

the

superiority

of

central

buffer

queuing

over

input

queuing

for

unicast

packets

However

in

contrast

to

software

multicasting

input

FIFO

replication

latencies

decrease

with

increasing

m

and

saturation

bandwidth

increases

As

m

increases

a

larger

percentage

of

the

replication

occurs

late

on

the

paths

to

the

destination

nodes

For

example

for

m

in

a

system

most

replication

occurs

at

the

switches

whereas

for

m

most

replication

occurs

at

the

switches

just

before

reaching

the

nodes

And

as

noted

before

for

Bo

mBi

increasing

m

decreases

Bi

Figure

allows

a

better

inspection

of

saturation

bandwidth

by

plotting

output

bandwidth

Bo

versus

effective

input

bandwidth

mBi

The

right

graphs

confirm

that

for

input

FIFO

replication

Received

traffic

-LRB-

fraction

of

max

BW

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Software

-LRB-

CQ

-RRB-

multicasting

msgs

SW-CQ

SW-CQ

SW-CQ

SW-CQ

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Received

traffic

-LRB-

fraction

of

max

BW

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Hardware

-LRB-

FIFO

-RRB-

multicasting

msgs

HW-FIFO

HW-FIFO

HW-FIFO

HW-FIFO

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Received

traffic

-LRB-

fraction

of

max

BW

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Software

-LRB-

CQ

-RRB-

Multicasting

msgs

SW-CQ

SW-CQ

SW-CQ

HW-CQ

HW-CQ

HW-CQ

Received

traffic

-LRB-

fraction

of

max

BW

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Hardware

-LRB-

FIFO

-RRB-

Multicasting

msgs

HW-FIFO

HW-FIFO

HW-FIFO

HW-CQ

HW-CQ

HW-CQ

Figure

Received

vs.

Effective

applied

load

for

software

multicast

hardware

multicast

with

central

queue

and

hardware

multicast

with

input

buffering

saturation

bandwidth

increases

with

m.

For

the

software

multicasts

shown

in

the

left

graphs

the

opposite

is

true

Unlike

the

characteristic

flattened

Bo

lines

for

the

right

graphs

the

software

multicast

Bo

does

not

flatten

out

after

Bo

mBi

This

is

because

the

software

forwarding

of

messages

is

happening

less

frequently

as

latency

increases

and

this

acts

as

a

drag

on

the

input

bandwidth

Thus

it

may

not

be

appropriate

to

characterize

the

software

multicast

as

in

saturation

the

centrally-buffered

switches

continue

to

increase

delivered

bandwidth

for

this

message

size

In

fact

for

the

central

buffer

replication

scheme

saturation

bandwidth

is

near

enough

to

that

it

is

hard

to

distinguish

the

curves

although

the

highest

bandwidth

is

achieved

for

large

m.

To

better

see

the

impact

of

m

on

saturation

for

central

buffer

replication

we

increase

the

message

size

in

the

next

section

Impact

of

message

size

The

graphs

we

have

shown

thus

far

use

messages

which

the

large

central

buffers

typically

handle

quite

easily

to

over

of

the

maximum

load

In

this

section

to

stress

the

network

further

we

examine

the

impact

of

a

larger

message

size

bytes

Figure

is

a

latency

graph

for

a

system

for

small

hardware

and

software

multicasts

The

and

hardware

multicasts

now

saturate

before

effective

load

although

the

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs.

Software

-LRB-

CQ

-RRB-

Multicasting

bytes

msgs

SW-CQ

SW-CQ

SW-CQ

SW-CQ

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Hardware

-LRB-

CQ

-RRB-

vs

Hardware

-LRB-

FIFO

-RRB-

Multicasting

byte

msgs

HW-FIFO

HW-FIFO

HW-FIFO

HW-FIFO

HW-CQ

HW-CQ

HW-CQ

HW-CQ

Figure

Latency

vs.

Effective

applied

load

for

a

system

messages

and

multicasts

do

not

Again

this

is

because

for

equivalent

Bo

mBi

larger

m

implies

smaller

Bi

resulting

in

less

packets

traversing

the

first

stages

of

the

network

although

in

the

last

stage

of

the

network

bandwidth

is

equivalent

The

software

multicasts

also

saturate

more

quickly

than

for

the

case

Just

as

for

the

messages

saturation

bandwidth

decreases

as

m

increases

because

for

software

multicasts

Bo

Bi

The

results

for

and

simulations

are

qualitatively

similar

and

throughout

the

rest

of

the

paper

we

shall

choose

message

size

arbitrarily

Impact

of

system

size

To

this

point

we

have

relied

on

simulations

to

test

various

parameters

To

assess

whether

lessons

learned

from

simulations

are

applicable

to

larger

systems

we

now

compare

and

hardware

multicasting

experiments

for

the

central

buffer

model

We

do

not

show

software

multicast

results

for

systems

because

they

grow

even

worse

as

m

increases

Figure

shows

latency

and

bandwidth

for

low

medium

and

high

values

of

m

for

each

system

size

The

system

exhibits

higher

latency

and

slightly

lower

saturation

bandwidth

than

the

system

for

comparable

multicasts

This

fits

well

with

experience

from

unicast

simulations

on

fat-trees

although

fat-trees

theoretically

scale

bandwidth

linearly

in

a

larger

network

random

traffic

requires

more

hops

on

average

and

therefore

encounters

more

contention

lowering

achievable

bandwidth

Just

as

for

the

system

the

low

m

multicast

has

low

latency

but

saturates

earliest

Interaction

between

multicast

and

unicast

traffic

The

experiments

so

far

have

stressed

the

network

applying

a

single

mode

in

traffic

input

random

m-way

multicast

traffic

In

a

typical

parallel

application

however

message

traffic

will

not

be

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Multicast

in

and

node

systems

byte

msgs

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

Received

traffic

-LRB-

fraction

of

max

BW

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Multicast

in

and

node

systems

byte

msgs

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

nodes

HW-CQ

Figure

Performance

of

and

systems

with

central

buffers

messages

entirely

multicast

or

entirely

unicast

at

any

moment

To

better

understand

the

interaction

of

multicast

and

unicast

traffic

we

simulate

random

bimodal

traffic

in

which

of

the

received

load

is

due

to

multicasts

Thus

Bo

consists

of

two

components

Bou

and

Bom

and

Bom

Bo

For

example

if

Bo

then

Bom

and

Bou

Figure

shows

the

result

of

combining

unicasts

with

multicasts

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Bimodal

Traffic

byte

msgs

All

traffic

SW-CQ

Unicast

traffic

SW-CQ

Mcast

traffic

SW-CQ

All

traffic

HW-CQ

Unicast

traffic

HW-CQ

Mcast

traffic

HW-CQ

Figure

Bimodal

latency

for

a

system

with

central

buffers

messages

For

unicast

traffic

it

is

natural

that

the

average

latency

should

closely

follow

the

unicast

message

latency

unless

multicast

latency

grows

disproportionately

large

Hardware

multicasts

perturb

unicast

traffic

much

less

than

do

software

multicasts

resulting

in

a

lower

combined

latency

This

is

because

hardware

multicasts

occupy

network

links

for

less

time

than

software

multicasts

due

to

both

lower

latency

and

a

lower

number

of

links

traversed

during

the

multicast

Measuring

latency

in

an

alternate

way

We

mentioned

in

Section

that

there

are

two

alternatives

for

calculating

the

average

latency

of

multicasts

To

this

point

we

have

chosen

definition

-LRB-

a

-RRB-

the

latency

of

the

last

received

message

of

the

multicast

In

this

section

we

compare

-LRB-

a

-RRB-

to

-LRB-

b

-RRB-

the

average

of

the

latencies

of

each

received

message

of

the

multicast

Figure

displays

this

comparison

for

m

on

the

left

and

m

on

the

right

For

the

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Latency

definitions

for

multicasting

msgs

SW-CQ

def

-LRB-

a

-RRB-

SW-CQ

def

-LRB-

b

-RRB-

HW-FIFO

def

-LRB-

a

-RRB-

HW-FIFO

def

-LRB-

b

-RRB-

HW-CQ

def

-LRB-

a

-RRB-

HW-CQ

def

-LRB-

b

-RRB-

Message

latency

-LRB-

cycles

-RRB-

Effective

applied

load

-LRB-

fraction

of

max

bandwidth

-RRB-

Latency

definitions

for

multicasting

msgs

SW-CQ

def

-LRB-

a

-RRB-

SW-CQ

def

-LRB-

b

-RRB-

HW-FIFO

def

-LRB-

a

-RRB-

HW-FIFO

def

-LRB-

b

-RRB-

HW-CQ

def

-LRB-

a

-RRB-

HW-CQ

def

-LRB-

b

-RRB-

Figure

Comparing

latency

definitions

for

a

system

messages

m

case

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

are

nearly

equal

for

the

hardware

central

buffer

multicast

at

low

loads

but

for

higher

loads

-LRB-

a

-RRB-

exceeds

-LRB-

b

-RRB-

by

almost

For

hardware

input

FIFO

replication

the

difference

between

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

grows

more

rapidly

as

does

the

magnitude

of

the

latencies

For

the

software

multicast

there

is

always

a

significant

difference

As

noted

in

Section

for

small

m

software

multicasting

with

central

buffers

actually

saturates

later

than

hardware

multicasting

with

input

FIFO

replication

hence

the

crossovers

in

the

graph

For

the

m

hardware

multicast

cases

the

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

definitions

match

closely

throughout

a

large

load

range

Software

multicast

has

no

such

property

due

to

the

-LRB-

m

-RRB-

e

steps

required

to

complete

it

To

summarize

when

-LRB-

a

-RRB-

and

-LRB-

b

-RRB-

definitions

are

nearly

equal

it

indicates

that

different

branches

of

the

multicast

messages

are

arriving

nearly

synchronously

This

is

an

attractive

property

for

loosely

synchronous

algorithms

and

collective

operations

such

as

barrier

synchronization

that

can

be

aided

by

multicasting

Hardware

multicasting

displays

this

nearly

synchronous

behavior

for

low

loads

or

high

degree

of

multicast

-LRB-

when

all

other

traffic

is

multicast

traffic

of

similar

degree

and

the

network

is

not

saturated

-RRB-

Summary

Here

we

summarize

our

simulated

performance

results

The

central

queue-based

hardware

replication

scheme

exhibited

a

number

of

desirable

characteristics

in

comparison

to

software

multicasting

Latency

was

much

lower

and

saturation

bandwidth

was

higher

for

the

entire

range

of

multicast

degree

m

system

size

n

and

message

size

When

random

multicasts

are

part

of

bimodal

traffic

along

with

random

unicast

messages

the

hardware

multicasts

perturb

the

unicasts

less

than

do

the

software

multicasts

Hardware

multicast

latency

varies

less

over

the

range

of

destinations

of

a

particular

multicast

especially

for

light

loads

or

large

m

The

input

FIFO-based

hardware

replication

scheme

did

not

fare

as

well

especially

at

high

loads

but

for

large

m

this

scheme

outperformed

the

software

multicasting

with

a

central

buffer

This

is

significant

considering

that

central

buffers

perform

significantly

better

than

input

FIFOs

for

unicast

traffic

Obviously

a

software

multicast

scheme

using

switches

with

no

central

buffers

would

perform

worse

than

any

option

simulated

Related

Work

Software

multicasting

on

parallel

systems

has

been

extensively

studied

for

k-ary

n-cube

networks

The

focus

of

a

bulk

of

this

work

has

been

on

achieving

multi-phase

contention

free

multicast

implementations

-LSB-

-RSB-

Most

of

these

implementations

require

-LRB-

d

-RRB-

e

communication

phases

to

cover

a

set

of

d

destinations

Software

multicast

has

also

been

studied

in

the

context

of

unidirectional

MINs

by

Chiang

and

Ni

-LSB-

-RSB-

and

in

the

context

of

bidirectional

MINs

by

Xu

et

al

-LSB-

-RSB-

Some

of

the

recent

research

has

focussed

on

extending

such

software

multicast

schemes

to

irregular

networks

-LSB-

-RSB-

Recent

research

has

also

focussed

on

hardware

multicasting

schemes

As

mentioned

before

a

path

based

model

was

proposed

for

multicasting

on

two

dimensional

mesh

networks

by

Lin

and

Ni

-LSB-

-RSB-

This

work

was

extended

to

a

Base

Routing

Conformed

Path

-LRB-

BRCP

-RRB-

model

by

Panda

et

al

-LSB-

-RSB-

applicable

to

any

k-ary

n-cube

network

Kesavan

and

Panda

-LSB-

-RSB-

have

studied

the

problem

of

multiple

multicast

for

meshes

using

both

software

and

hardware

path

based

multicasts

In

the

context

of

switch

based

parallel

systems

Chiang

and

Ni

have

proposed

a

method

for

synchronous

replication

of

multidestination

worms

-LSB-

-RSB-

adopting

a

deadlock

avoidance

scheme

at

the

switches

A

somewhat

related

multicasting

technique

proposed

in

the

context

of

DSM

systems

based

on

direct

networks

is

the

tree

based

multicasting

scheme

proposed

in

-LSB-

-RSB-

Instead

of

guaranteeing

freedom

from

deadlock

a

deadlock

recovery

scheme

is

proposed

that

prunes

blocked

branches

This

method

is

effective

only

for

short

messages

as

prevalent

in

a

distributed

shared

memory

system

The

architectures

proposed

in

the

current

paper

allow

multicast

for

packets

as

large

as

the

buffer

size

at

the

switches

and

the

technique

works

well

for

both

long

and

short

messages

Andrews

et

al

-LSB-

-RSB-

have

proposed

a

method

for

tree

based

multicast

using

bit-string

encoding

in

the

context

of

dance-hall

architectures

However

this

work

only

focuses

on

store

and

forward

networks

and

short

message

lengths

Some

parallel

systems

like

the

-LSB-

-RSB-

Meiko

-LSB-

-RSB-

etc.

provide

facilities

for

multicasting

However

the

uses

a

separate

network

for

multicast

operations

and

only

one

multicast

is

allowed

in

this

network

at

any

given

time

Data-less

multicasts

or

eurekas

are

supported

in

the

Cray

-LSB-

-RSB-

in

the

same

network

used

for

normal

data

communication

This

avoids

the

deadlock

scenarios

outlined

in

this

paper

and

the

solution

does

not

extend

to

multicast

of

data

packets

The

approach

presented

in

this

paper

allows

arbitrary

numbers

of

multicasts

each

of

size

upto

the

maximum

packet

size

allowed

in

the

system

to

proceed

concurrently

while

using

the

same

network

as

all

other

data

to

achieve

multicast

communication

Furthermore

unlike

the

approach

proposed

for

the

Meiko

multicast

to

arbitrary

destination

sets

is

allowed

Conclusion

In

this

paper

we

have

presented

two

switch

architectures

with

differing

buffer

organizations

for

implementing

multidestination

worms

in

switch

based

parallel

systems

We

have

described

in

detail

how

a

central

queue

based

switch

architecture

that

supports

only

unicast

message

passing

can

be

modified

to

support

multidestination

message

passing

at

very

little

additional

cost

We

then

performed

extensive

simulations

to

evaluate

the

relative

performance

of

our

proposed

switch

architectures

and

to

compare

the

achieved

hardware

multicast

performance

with

the

best

software

multicast

algorithm

To

make

the

comparisons

more

interesting

we

factored

out

the

high

start-up

overhead

associated

with

the

software

schemes

in

our

performance

comparisons

The

performance

achieved

by

such

a

software

multicast

algorithm

is

the

best

that

can

be

achieved

using

point

to

point

communication

primitives

alone

Our

performance

studies

have

shown

that

central

queue

based

switch

architectures

prove

extremely

beneficial

in

improving

hardware

multicast

performance

Such

architectures

can

deliver

good

performance

across

an

entire

range

of

applied

loads

message

lengths

multicast

degrees

and

system

sizes

Although

the

alternative

implementation

using

an

input

FIFO

delivers

multicast

performance

similar

to

the

central

queue

based

implementation

for

single

multicasts

and

multiple

multicasts

at

low

loads

the

degradation

in

performance

is

rapid

and

occurs

at

relatively

light

loads

The

central

queue

based

software

multicast

outperforms

hardware

multicast

using

the

input

FIFO

based

implementation

-LRB-

if

the

associated

start-up

overhead

is

neglected

-RRB-

for

low

multicast

degrees

This

implies

that

software

multicast

using

efficient

messaging

layers

and

support

at

the

network

interfaces

to

reduce

start-up

overhead

has

the

potential

to

offer

good

performance

when

used

in

conjunction

with

central-queue-based

switches

We

are

currently

studying

the

impact

of

enhancements

to

the

switch

architecture

to

support

reliable

barrier

synchronization

The

relative

performance

of

such

a

scheme

compared

to

an

efficient

software

based

scheme

in

terms

of

absolute

barrier

latency

and

in

terms

of

their

impact

on

other

network

traffic

is

being

evaluated

We

are

also

working

on

measuring

the

impact

of

these

enhancements

to

the

communication

subsystem

on

benchmark

application

performance

in

the

DM

and

DSM

domains

References

-LSB-

-RSB-

J.

B.

Andrews

C.

J.

Beckmann

and

D.

K.Poulsen

Notification

and

multicast

networks

for

synchronization

and

coherence

Journal

of

Parallel

and

Distributed

Computing

-LCB-

Aug.

-LSB-

-RSB-

V.

Bala

J.

Bruck

R.

Cypher

P.

Elustondo

A.

Ho

C.-T

Ho

S.

Kipnis

and

M.

Snir

CCL

A

Portable

and

Tunable

Collective

Communication

Library

for

Scalable

Parallel

Computers

In

Proceedings

of

the

International

Parallel

Processing

Symposium

-LSB-

-RSB-

M.

Barnett

D.

G.

Payne

and

R.

Van

de

Geijn

Optimal

Broadcasting

in

Mesh-Connected

Architectures

Technical

Report

Dept.

of

Computer

Science

Universityof

Texas

at

Austin

Dec

-LSB-

-RSB-

J.

Beecroft

M.

Homewood

and

M.

McLaren

Meiko

Interconnect

Elan-Elite

Design

Parallel

Computing

-LCB-

Nov

-LSB-

-RSB-

N.

J.

Boden

D.

Cohen

and

et

al.

Myrinet

A

Gigabit-per-Second

Local

Area

Network

IEEE

Micro

pages

-LCB-

Feb

-LSB-

-RSB-

C.-M

Chiang

and

L.

M.

Ni

Multi-Address

Encoding

for

Multicast

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

May

-LSB-

-RSB-

C.-M

Chiang

and

L.

M.

Ni

Deadlock-Free

Multi-Head

Wormhole

Routing

In

Proceedings

of

the

First

High

Performance

Computing-Asia

-LSB-

-RSB-

C.-M

Chiang

and

L.

M.

Ni

Efficient

Software

Multicast

in

Wormhole-routed

Unidirectional

Multistage

Networks

In

Symposium

on

Parallel

and

Distributed

Processing

-LSB-

-RSB-

Chi-Ming

Chiang

Multicasting

in

Multistage

Interconnection

Networks

PhD

thesis

Michigan

State

University

-LSB-

-RSB-

Cray

Research

Inc.

Cray

System

Architecture

Overview

-LSB-

-RSB-

D.

Dai

and

D.

K.

Panda

Reducing

Cache

Invalidation

Overheads

in

Wormhole

DSMs

Using

Multidestination

Message

Passing

In

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

W.

J.

Dally

and

C.

L.

Seitz

Deadlock-Free

Message

Routing

in

Multiprocessor

Interconnection

Networks

IEEE

Transactions

on

Computers

pages

-LCB-

May

-LSB-

-RSB-

B.

Duzett

and

R.

Buck

An

Overview

of

the

Supercomputer

In

Proceedings

of

the

Frontiers

of

Massively

Parallel

Computation

pages

-LCB-

-LSB-

-RSB-

M.

D.

Schroeder

et

al.

Autonet

A

High-speed

Self-configuring

Local

Area

Network

Using

Point-to-point

Links

Technical

Report

SRC

research

report

DEC

Apr

-LSB-

-RSB-

Intel

Corporation

Paragon

XP/S

Product

Overview

-LSB-

-RSB-

P.

Kermani

and

L.

Kleinrock

Virtual

Cut-Through

A

New

Computer

Communications

Switching

Technique

Computer

Networks

-LRB-

-RRB-

-LCB-

Sept.

-LSB-

-RSB-

R.

Kesavan

K.

Bondalapati

and

D.

K.

Panda

Multicast

on

Irregular

Switch-based

Networks

with

Wormhole

Routing

In

Proceedings

of

the

Int

l

Symposium

on

High

Performace

Computer

Architecture

-LRB-

-RRB-

accepted

to

be

presented

-LSB-

-RSB-

R.

Kesavan

and

D.

K.

Panda

Minimizing

Node

Contention

in

Multiple

Multicast

on

Wormhole

k-ary

n-cube

Networks

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

C.

E.

Leiserson

et

al.

The

Network

Architecture

of

the

Connection

Machine

In

Proceedings

of

the

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

pages

-LCB-

-LSB-

-RSB-

X.

Lin

P.

K.McKinley

and

L.

M.

Ni

Deadlock

free

multicast

wormhole

routing

in

mesh

multicomputers

IEEE

Trans

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Aug.

-LSB-

-RSB-

X.

Lin

P.

K.

McKinley

and

L.

M.

Ni

Performance

Evaluation

of

Multicast

Wormhole

Routing

in

Multicomputers

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

-LSB-

-RSB-

X.

Lin

and

L.

M.

Ni

Deadlock-free

Multicast

Wormhole

Routing

in

Multicomputer

Networks

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

M.

P.

Malumbres

J.

Duato

and

J.

Torellas

An

Efficient

Implementation

of

Tree-Based

Multicast

Routing

for

Distributed

Shared-Memory

Multiprocessors

In

Proceedings

of

the

Eighth

IEEE

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

New

Orleans

LA

October

-LSB-

-RSB-

P.

K.

McKinley

and

D.

F.

Robinson

Collective

Communication

in

Wormhole-Routed

Massively

Parallel

Computers

IEEE

Computer

pages

-LCB-

Dec

-LSB-

-RSB-

P.

K.

McKinley

H.

Xu

A.-H

Esfahanian

and

L.

M.

Ni

Unicast-based

Multicast

Communication

in

Wormhole-routed

Networks

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LCB-

Dec

-LSB-

-RSB-

L.

Ni

and

P.

K.

McKinley

A

Survey

of

Wormhole

Routing

Techniques

in

Direct

Networks

IEEE

Computer

pages

-LCB-

Feb.

-LSB-

-RSB-

Lionel

Ni

Should

Scalable

Parallel

Computers

Support

Efficient

Hardware

Multicasting

In

ICPP

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

A.

G.

Nowatzyk

M.

C.

Browne

E.

J.

Kelly

and

M.

Parkin

S-Connect

from

Networks

of

Workstations

to

Supercomputer

Performance

In

Proceedings

of

the

International

Symposium

on

Computer

Architecture

pages

-LCB-

-LSB-

-RSB-

N.

Nupairoj

and

L.M.

Ni

Benchmarking

of

Multicast

Communication

Services

Technical

Report

Michigan

State

University

Sep

-LSB-

-RSB-

D.

K.

Panda

Issues

in

Designing

Efficient

and

Practical

Algorithms

for

Collective

Communication

in

Wormhole-Routed

Systems

In

Workshop

on

Challenges

for

Parallel

Processing

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

S.

Singal

and

P.

Prabhakaran

Multidestination

Message

Passing

Mechanism

Conforming

to

Base

Wormhole

Routing

Scheme

In

Proceedings

of

the

Parallel

Computer

Routing

and

Communication

Workshop

pages

-LCB-

-LSB-

-RSB-

D.

K.

Panda

and

R.

Sivaram

Fast

Broadcast

and

Multicast

in

Wormhole

Multistage

Networks

with

Multidestination

Worms

Technical

Report

Dept.

of

Computer

and

Information

Science

The

Ohio

State

University

April

-LSB-

-RSB-

W.

Qiao

and

L.

M.

Ni

Adaptive

Routing

in

Irregular

Networks

Using

Cut-Through

Switches

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

I

-LCB-

Chicago

IL

Aug

-LSB-

-RSB-

I.

D.

Scherson

and

C.-H

Chien

Least

common

ancestor

networks

In

Proc

Int

Parallel

Processing

Symp.

pages

-LCB-

-LSB-

-RSB-

S.

L.

Scott

Synchronization

and

communication

in

the

multiprocessor

In

ASPLOS-VII

Sept.

-LSB-

-RSB-

R.

Sivaram

D.

K.

Panda

and

C.

B.

Stunkel

Efficient

Broadcast

and

Multicast

on

Multistage

Interconnection

Networks

using

Multiport

Encoding

In

Proceedings

of

the

Eighth

IEEE

Symposium

on

Parallel

and

Distributed

Processing

pages

-LCB-

Oct

-LSB-

-RSB-

Rajeev

Sivaram

Dhabaleswar

K.

Panda

and

Craig

B.

Stunkel

Efficient

Broadcast

and

Multicast

on

Multistage

Interconnection

Networks

using

Multiport

Encoding

Technical

Report

The

Ohio

State

University

March

-LSB-

-RSB-

C.

B.

Stunkel

D.

Shea

D.

G.

Grice

P.

H.

Hochschild

and

M.

Tsao

The

High

Performance

Switch

In

Scalable

High

Performance

Computing

Conference

pages

-LCB-

-LSB-

-RSB-

C.

B.

Stunkel

D.

G.

Shea

B.

Abali

et

al.

The

High-Performance

Switch

IBM

System

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Y.

Tamir

and

G.

L.

Frazier

High-performance

multi-queue

buffers

for

VLSI

communication

switches

In

Proc

Ann

Int

Symp

on

Computer

Architecture

pages

-LCB-

May

-LSB-

-RSB-

K.

Verstoep

K.

Langendoen

and

H.

Bal

Efficient

Reliable

Multicast

on

Myrinet

In

Proceedings

of

the

International

Conference

on

Parallel

Processing

pages

III

-LCB-

Aug

-LSB-

-RSB-

H.

Xu

Y.-D

Gui

and

L.

M.

Ni

Optimal

Software

Multicast

in

Wormhole-Routed

Multistage

Networks

In

Proceedings

of

the

Supercomputing

Conference

pages

-LCB-

broadcast
collective communication
multicast
multistage interconnection networks
performance evaluation
wormhole routing

cut-through routing
parallel computer architecture