Randomized

Priority

Queues

for

Fast

Parallel

Access

Peter

Sanders

Max-Planck-Institute

for

Computer

Science

Saarbr?ucken

Germany

E-mail

sanders@mpi-sb.mpg.de

January

Abstract

We

present

simple

randomized

algorithms

for

parallel

priority

queues

on

distributed

memory

machines

Inserting

O

-LRB-

n

-RRB-

elements

or

deleting

the

O

-LRB-

n

-RRB-

out

of

m

smallest

elements

using

n

pocessors

requires

O?Tcoll

log

mn

amortized

time

with

high

probability

where

Tcoll

bounds

the

time

for

performing

pre?x

sums

and

randomized

routing

The

memory

requirement

is

bounded

by

mn

-LRB-

o

-LRB-

-RRB-

-RRB-

O

-LRB-

logn

-RRB-

whp

These

bounds

are

an

improvement

over

the

best

previously

known

algorithms

for

many

interconnection

networks

and

even

matches

the

speed

of

the

best

known

PRAM

algorithms

Generalizations

for

accessing

the

k

AE

n

smallest

elements

are

even

more

ef?cient

A

portable

implementation

using

MPI

demonstrates

that

our

approach

is

already

useful

for

medium

scale

parallelism

Two

parallel

selection

algorithms

for

randomly

placed

data

are

a

spin-off

One

runs

in

time

O

-LRB-

Tcoll

-RRB-

with

high

probability

beating

a

lower

bound

for

the

worst

case

The

other

requires

only

a

single

reduction

operation

Besides

their

classical

role

as

a

data

structure

parallel

priority

queues

can

also

be

viewed

as

load

balancing

algorithms

able

to

ef?ciently

schedule

prioritized

jobs

This

is

exempli?ed

by

an

application

to

parallel

branch-and-bound

Introduction

Ef?cient

priority

queues

i.e.

data

structures

supporting

the

operations

insert

and

deleteMin

are

a

key

component

of

many

algorithms

such

as

branch-and-bound

discrete

event

simulation

-LSB-

-RSB-

or

some

graph

algorithms

Parallelizing

these

applications

often

calls

for

parallel

access

to

a

globally

visible

priority

queue

which

can

become

a

performance

bottleneck

if

not

implemented

properly

In

addition

a

parallel

priority

queue

can

be

viewed

as

a

scheduler

for

prioritized

jobs

yielding

a

exible

load

balancing

algorithm

Even

if

the

sequential

algorithm

does

not

explicitly

use

prioritization

it

often

Most

of

this

work

was

performed

while

the

author

was

at

the

Department

of

Computer

Science

of

the

University

of

Karlsruhe

makes

sense

to

use

the

sequential

evaluation

order

itself

as

the

priority

for

parallel

execution

In

-LSB-

-RSB-

this

turns

out

to

reduce

-LRB-

adverse

-RRB-

speedup

anomalies

in

particular

in

presence

of

strong

heuristics

Consequently

parallel

priority

queues

have

attracted

considerable

research

interest

The

remainder

of

this

paper

is

structured

as

follows

Models

for

the

parallel

machine

and

the

queue

data

structure

are

introduced

in

Section

Section

surveys

previous

results

A

core

part

of

the

paper

is

Section

on

a

randomized

synchronous

parallel

priority

queue

which

can

be

described

quite

concisely

and

which

turns

out

to

be

asymptotically

very

ef?cient

These

formal

results

are

complemented

by

Section

where

we

present

enhancements

which

make

the

algorithm

more

effective

on

contemporary

machines

with

asynchronous

behavior

preference

for

coarse-grained

communication

and

a

high

penalty

for

collective

operations

Section

describes

an

implementation

which

demonstrates

the

practicability

of

bottleneck-free

priority

queues

even

on

coarse-grained

moderately

parallel

machines

like

the

IBM-SP

The

conclusions

in

Section

summarize

and

discuss

the

results

The

Model

We

rst

describe

our

model

of

a

parallel

machine

in

Section

and

then

de?ne

measures

for

randomized

algorithms

and

some

tools

for

their

analysis

in

Section

Section

introduces

the

branch-and-bound

application

and

Section

the

notion

of

parallel

priority

queue

needed

in

this

paper

The

Parallel

Machine

We

consider

an

MIMD

computer

with

n

processing

elements

-LRB-

PEs

-RRB-

which

asynchronously

interact

by

exchanging

messages

through

a

network

We

start

with

algorithms

written

in

a

synchronized

data

parallel

style

but

synchronizations

will

not

be

a

bottleneck

We

do

not

assume

a

speci?c

network

topology.Instead

the

algorithms

are

based

on

the

following

set

of

building

blocks

-LRB-

if

not

otherwise

stated

we

assume

constant

message

lengths

-RRB-

Routing

Every

PE

either

sends

up

to

k

messages

to

randomly

determined

receivers

or

receives

up

to

k

messages

from

randomly

determined

senders

-LRB-

k

constant

-RRB-

Broadcast

One

PE

sends

a

message

to

all

other

PEs

Reduction

Given

value

vi

on

PE

i

determine

vn

for

an

associative

commutative

operator

Pre?x

For

vi

and

as

above

determine

vi

on

PE

i

The

analysis

can

focus

on

nding

the

number

of

necessary

basic

operations

The

execution

time

for

a

particular

network

is

then

easy

to

determine

by

appropriately

de?ning

TRouting

-LRB-

n

-RRB-

TBroadcast

-LRB-

n

-RRB-

TReduction

-LRB-

n

-RRB-

and

TPre?x

-LRB-

n

-RRB-

The

results

are

also

easy

to

translate

into

abstract

models

like

BSP

-LSB-

-RSB-

although

this

may

be

less

accurate

for

some

machines

with

tuned

implementations

for

the

above

collective

operations

To

further

simplify

the

discussion

we

de?ne

a

common

upper

bound

Tcoll

such

that

fTRouting

-LRB-

n

-RRB-

TBroadcast

-LRB-

n

-RRB-

TReduction

-LRB-

n

-RRB-

TPre?x

-LRB-

n

-RRB-

-LSB-

O

-LRB-

logn

-RRB-

O

-LRB-

Tcoll

-LRB-

n

-RRB-

-RRB-

g

with

high

probability

Note

that

Tcoll

-LRB-

n

-RRB-

is

proportional

to

the

network

diameter

d

for

many

network

types

e.g.

r-dimensional

meshes

hypercubes

and

related

constant

degree

networks

like

butter?ies

or

a

combination

of

a

multistage

network

for

routing

and

a

tree

network

All

the

necessary

results

can

be

found

in

-LSB-

-RSB-

Analysis

of

Randomized

Algorithms

The

analysis

of

the

randomized

algorithms

described

here

is

based

on

the

notion

of

behavior

with

high

probability

Among

the

various

variants

of

this

notion

we

have

adopted

the

one

from

-LSB-

-RSB-

De?nition

A

positive

real

valued

random

variable

X

is

in

O

-LRB-

f

-LRB-

n

-RRB-

-RRB-

with

high

probability

or

X

O

-LRB-

f

-LRB-

n

-RRB-

-RRB-

for

short

iff

P

-LSB-

X

c

f

-LRB-

n

-RRB-

-RSB-

n?b

i.e.

the

probability

that

X

exceeds

the

bound

f

by

more

than

a

constant

factor

is

polynomially

small

In

this

paper

the

variable

used

to

express

high

probability

is

always

n

the

number

of

PEs

One

advantage

of

the

high

probability

approach

is

that

there

are

quite

simple

rules

combining

results

about

simpler

problems

into

more

complex

results

In

this

paper

we

need

the

following

rules

which

we

present

without

proof

because

they

are

based

on

quite

straightforward

elementary

probability

theory

-LRB-

Proofs

can

be

found

in

-LSB-

-RSB-

-RRB-

Lemma

Let

O

-LRB-

-RRB-

Xk

O

-LRB-

fk

-RRB-

be

random

variables

-LRB-

k

constant

-RRB-

k

O

i

Xi

O

k

O

i

fi

for

O

max

-LRB-

-RRB-

Lemma

Let

Xmg

O

-LRB-

f

-RRB-

be

a

set

of

identically

distributed

random

variables

-LRB-

m

at

most

polynomial

in

n

-RRB-

m

max

i

Xi

O

-LRB-

f

-RRB-

-LRB-

-RRB-

m

i

Xi

O

-LRB-

m

f

-RRB-

-LRB-

-RRB-

A

keystone

of

many

probabilistic

proofs

are

the

following

Chernoff

bounds

which

give

quite

tight

bounds

on

the

probability

that

the

sum

of

variables

i.e.

a

binomial

distribution

deviates

from

the

expected

value

by

some

factor

Lemma

-LRB-

Chernoff

bounds

-RRB-

Let

the

random

variable

X

represent

the

number

of

heads

after

n

independent

ips

of

a

loaded

coin

where

the

probability

for

a

head

is

p

Then

-LSB-

-RSB-

P

-LSB-

X

-LRB-

e

-RRB-

np

-RSB-

for

e

-LRB-

-RRB-

P

-LSB-

X

-LRB-

e

-RRB-

np

-RSB-

for

e

-LRB-

-RRB-

P

-LSB-

X

anp

-RSB-

e

-LRB-

loga

-RRB-

anp

for

a

-LRB-

-RRB-

-LRB-

Throughout

this

paper

log

denotes

the

natural

logarithm

-RRB-

Branch-and-Bound

We

adopt

the

model

of

Karp

and

Zhang

-LSB-

-RSB-

Let

H

denote

the

search

tree

with

a

set

of

nodes

V

Node

degrees

are

bounded

by

a

constant

All

node

costs

c

-LRB-

v

-RRB-

are

assumed

to

be

different

and

c

-LRB-

v

-RRB-

is

monotonously

increasing

on

any

path

from

the

root

to

a

leaf

-LRB-

When

we

compare

nodes

we

actually

compare

their

costs

-RRB-

We

are

looking

for

the

leaf

v

with

minimal

cost

Let

V

fv

v

v?g

and

let

H

be

the

subtree

of

H

containing

the

nodes

V

Let

m

j

V

j

and

h

the

length

of

the

longest

root-leaf

path

in

H.

Clearly

V

is

a

set

of

nodes

which

have

to

be

expanded

by

any

algorithm

which

wants

to

nd

v

together

with

a

proof

that

there

can

not

be

better

solutions

We

do

not

want

to

look

at

very

small

problems

and

therefore

assume

that

m

W

-LRB-

nlogn

-RRB-

var

Q

froot

nodeg

Heap

-LRB-

frontier

set

-RRB-

var

c

-LRB-

best

solution

so

far

-RRB-

while

Q

do

select

some

v

Q

and

remove

it

if

c

-LRB-

v

-RRB-

c

then

if

v

is

a

leaf

node

then

process

new

solution

c

c

-LRB-

v

-RRB-

else

insert

successors

of

v

into

Q

Figure

Sequential

branch-and-bound

Most

branch-and-bound

algorithms

can

be

viewed

as

a

variant

of

the

abstract

sequential

algorithm

depicted

in

Figure

The

processor

repeatedly

selects

a

node

v

from

the

frontier

set

Q

and

expands

it

A

node

with

a

higher

value

than

the

best

one

investigated

so

far

can

not

lead

to

a

solution

and

is

pruned

The

remaining

inner

nodes

are

expanded

and

their

successors

are

inserted

into

the

frontier

set

Leaf

nodes

are

solution

candidates

When

Q

becomes

empty

we

know

that

the

best

leaf

node

found

so

far

is

the

solution

Expanding

a

node

-LRB-

i.e.

generating

its

successors

-RRB-

takes

time

Tx

all

other

operations

on

nodes

node

values

and

solutions

take

unit

time

If

the

best

rst

selection

strategy

is

used

i.e.

Q

is

organized

as

a

priority

queue

the

algorithm

expands

the

-LRB-

optimal

-RRB-

number

of

m

nodes

Selection

and

insertion

can

be

done

in

time

O

-LRB-

logm

-RRB-

e.g.

by

using

a

binary

implementation

of

Q.

So

for

sequential

best

rst

branch-and-bound

we

get

the

execution

time

Tseq

m

-LRB-

Tx

O

-LRB-

logm

-RRB-

-RRB-

Parallel

Priority

Queues

The

semantics

of

sequential

priority

queues

is

quite

simple

A

set

of

elements

with

totally

ordered

keys

is

managed

The

operation

insert

inserts

an

element

The

operation

deleteMin

deletes

the

element

with

the

smallest

key

and

returns

it

For

a

parallel

priority

queue

this

is

in

general

more

dif?cult

We

start

with

a

simple

data

parallel

variant

insert

Each

PE

inserts

up

to

d

elements

-LRB-

d

constant

-RRB-

are

more

sophisticated

data

structures

for

priority

queues

which

only

require

constant

time

for

some

operations

However

for

applications

with

about

the

same

number

of

insert

and

deleteMin

operations

these

variants

imply

no

asymptotic

improvement

deleteMin

The

n

smallest

elements

are

retrieved

and

each

PE

gets

exactly

one

of

the

elements

For

simplicity

assume

that

all

key

values

in

the

queue

are

different

and

that

empty

queues

returns

dummy

elements

with

key

If

necessary

unique

elements

can

be

enforced

transparently

by

appending

an

identi?er

to

the

least

signi?cant

bits

of

a

key

Generalizations

are

possible

and

mostly

straightforward

Roughly

access

to

fewer

elements

will

not

be

much

faster

but

may

still

be

better

than

a

centralized

queue

Batched

access

to

many

elements

is

discussed

in

Section

Refer

to

Section

for

a

more

exible

asynchronous

semantics

For

example

using

these

operations

we

get

the

synchronous

parallel

global

best

rst

branch-and-bound

algorithm

described

in

-LSB-

-RSB-

if

we

replace

Q

in

Algorithm

by

a

parallel

priority

queue

The

parallel

algorithm

expands

n

nodes

in

each

iteration

and

the

number

of

iterations

required

lies

between

max

-LRB-

mn

h

-RRB-

and

mn

h

i.e.

in

Q

mn

h

The

lower

bound

is

easy

to

understand

since

every

branch-and-bound

algorithm

must

expand

at

least

m

nodes

and

has

a

sequential

component

of

h

node

expansions

For

the

upper

bound

if

jQ

V

j

n

all

PEs

get

nodes

from

H

In

all

other

iterations

the

maximum

path

length

to

a

leaf

in

H

is

reduced

Approaches

to

Parallelization

Perhaps

the

simplest

implementation

of

a

parallel

priority

queue

is

to

use

a

centralized

server

PE

which

manages

a

sequential

priority

queue

In

this

case

the

operations

insert

and

deleteMin

require

time

Q

-LRB-

nlogm

-RRB-

which

is

quite

slow

for

large

n.

Also

the

memory

of

the

server

PE

limits

the

size

of

the

queue

This

can

be

improved

by

algorithms

parallelizing

an

individual

queue

access

Refer

to

-LSB-

-RSB-

for

a

recent

survey

The

best

algorithms

perform

all

operations

in

constant

time

using

O

-LRB-

logm

-RRB-

PEs

More

scalable

algorithms

exploit

the

fact

that

our

de?nition

of

parallel

priority

queue

calls

for

a

method

for

quickly

removing

a

rather

large

number

of

elements

at

once

One

intensively

studied

approach

is

to

use

a

generalized

k-bandwidth

heap

which

contains

multiple

elements

in

each

heap

node

and

substitutes

the

compare

and

exchange

operations

of

the

usual

heap

algorithm

by

sorting

and

merging

operations

For

a

recent

literature

survey

refer

to

-LSB-

-RSB-

n

insertions

and

deletions

can

be

performed

in

time

O

-LRB-

logm

-RRB-

on

EREW

PRAMs

-LSB-

-RSB-

and

on

pipelined

hypercubes

-LSB-

-RSB-

This

algorithm

requires

newly

inserted

elements

to

be

sorted

so

we

must

add

another

O

-LRB-

lognloglogn

-RRB-

or

O

-LRB-

logn

-RRB-

term

for

the

sorting

operation

The

parallel

sorting

and

merging

routines

required

by

these

algorithms

are

slower

on

weaker

models

of

parallel

computation

On

single-ported

hypercubes

and

meshes

access

times

are

O

-LRB-

logm

logn

-RRB-

and

O?pnlogm

respectively

-LSB-

-RSB-

A

radical

approach

is

to

relax

the

priority

queue

semantics

and

to

distribute

the

elements

over

more

or

less

independent

local

queues

which

exchange

elements

in

order

to

approximate

the

behavior

of

a

global

priority

queue

For

example

in

the

algorithm

by

Karp

and

Zhang

-LSB-

-RSB-

inserted

elements

are

sent

to

random

PEs

while

deleteMin

requests

simply

access

the

locally

present

queue

For

branch-and-bound

this

only

increases

the

number

of

expanded

nodes

by

a

constant

factor

-LRB-

with

high

probability

-RRB-

The

starting

point

for

this

paper

is

to

use

Karp

and

Zhang

's

approach

of

local

queues

with

random

placement

of

elements

for

the

insert

operation

but

to

extract

the

globally

best

elements

for

a

deleteMin

operation

It

turns

out

that

instead

of

parallel

sorting

and

merging

the

only

nonlocal

operations

are

routing

and

parallel

selection

now

The

basic

approach

has

been

independently

developed

in

-LSB-

-RSB-

and

-LSB-

-RSB-

These

simple

versions

are

already

asymptotically

optimal

on

mesh

connected

machines

In

-LSB-

-RSB-

improvements

for

logarithmic

diameter

networks

like

butter?ies

are

introduced

by

avoiding

work

imbalance

due

to

local

queue

access

and

by

modifying

an

ef?cient

selection

algorithm

to

exploit

the

random

placement

of

the

elements

Here

we

explain

and

re?ne

this

approach

The

selection

algorithm

from

-LSB-

-RSB-

is

also

used

in

-LSB-

-RSB-

for

accessing

k

AE

n

priority

queue

elements

in

parallel

These

algorithms

can

be

considered

to

be

a

combination

of

k-bandwidth

heaps

and

random

placement

Actual

implementations

of

parallel

priority

queues

are

quite

rare

One

reason

is

that

start-up

times

for

parallel

primitives

are

so

large

that

they

dwarf

the

time

for

sequentially

accessing

a

single

element

Consequently

parallelizing

a

single

queue

access

turned

out

to

be

impractical

in

-LSB-

-RSB-

Note

that

this

was

on

a

machine

with

hardware

support

for

shared

memory

and

that

the

speed

gap

between

local

and

global

operation

has

widened

since

then

Signi?cant

speedups

have

so

far

only

been

achieved

for

batched

operation

involving

thousands

of

accesses

per

PE

-LSB-

-RSB-

But

note

that

the

traditional

notion

of

speedup

neglects

the

fact

that

requests

and

answers

have

to

involve

communication

in

a

parallel

setting

Therefore

a

parallel

priority

queue

with

speedup

S

can

still

be

useful

if

it

avoids

the

communication

bottleneck

implied

by

a

centralized

queue

An

Ef?cient

Algorithm

and

its

Analysis

We

now

describe

and

analyze

an

algorithm

for

synchronous

parallel

access

to

priority

queues

The

algorithm

is

designed

to

combine

simplicity

and

analyzability

with

asymptotic

ef?ciency

For

a

practical

implementation

some

of

the

modi?cations

described

in

Section

should

be

used

The

description

starts

with

an

overview

of

the

basic

algorithm

in

Section

We

then

elaborate

probabilistic

bounds

on

the

sizes

of

some

intermediate

data

structures

in

Section

which

play

a

key

role

in

the

analysis

of

the

algorithm

After

investigating

a

nontrivial

subroutine

in

Section

we

can

nally

integrate

the

pieces

into

an

analysis

of

the

full

algorithm

in

Section

The

Basic

Algorithm

Figure

outlines

the

algorithms

for

insert

and

deleteMin

Every

PE

holds

a

local

queue

stored

in

two

parts

a

sorted

array

which

acts

as

a

buffer

for

fast

access

to

the

smallest

elements

and

a

heap

for

the

remaining

elements

Let

the

ff

accent

denote

the

global

union

operation

e.g.

denotes

the

union

of

all

buffers

Newly

inserted

elements

are

routed

to

random

PEs

and

then

inserted

into

Every

logn

calls

of

insert

-LRB-

we

call

this

a

cycle

-RRB-

is

emptied

into

Lemma

states

that

this

measure

is

suf?cient

to

keep

small

-LRB-

O

-LRB-

logn

-RRB-

-RRB-

most

of

the

time

deleteMin

proceeds

in

two

phases

First

elements

are

moved

from

to

until

the

smallest

n

elements

must

be

in

the

buffers

This

can

be

checked

by

counting

the

buffer

elements

smaller

than

the

minimum

element

in

i.e.

a

global

minimum

computation

followed

by

a

global

sum

are

the

communication

operations

required

in

each

iteration

Lemma

will

establish

that

the

amortized

number

of

required

iterations

through

the

while-loop

is

constant

-LRB-

per

call

of

deleteMin

-RRB-

Then

a

distributed

algorithm

var

i

Integer

-LRB-

call

counter

-RRB-

var

Sorted

Array

-LRB-

buffer

-RRB-

var

Heap

-LRB-

main

queue

-RRB-

Procedure

insert

-LRB-

e

Array

of

Element

-RRB-

send

each

ei

to

a

randomly

selected

PE

insert

received

elements

into

i

i

if

i

mod

logn

then

-LSB-

Function

deleteMin

-LRB-

-RRB-

Element

-LRB-

One

per

PE

-RRB-

while

jfe

e

min

n

do

-LSB-

fdeleteMin

-LRB-

-RRB-

g

extract

the

n

smallest

elements

in

enumerate

the

smallest

elements

en

send

ei

to

PE

i

and

return

it

Figure

Parallel

priority

queue

access

described

later

nds

the

n

smallest

elements

in

These

can

be

enumerated

using

a

pre?x

operation

Finally

element

number

j

is

returned

on

PE

j.

Figure

sketches

an

example

for

the

ow

of

data

during

the

execution

of

deleteMin

Q

Q

PE

Filter

n

best

Assign

to

PEs

Figure

Example

for

extracting

the

n

best

elements

Queue

Sizes

Lemma

The

maximum

number

of

new

elements

to

be

inserted

into

on

any

PE

after

a

call

of

insert

is

in

O

logn

loglogn

Proof

Let

Xi

denote

the

number

of

new

elements

to

be

inserted

on

PE

i

Since

the

number

of

elements

inserted

by

each

PE

is

bounded

by

a

constant

there

is

a

constant

k

such

that

kn

is

a

bound

for

the

overall

number

of

new

elements

The

placements

of

elements

can

be

viewed

as

independent

Bernoulli

experiments

with

success

probability

-LRB-

We

count

the

placement

of

an

element

at

PE

i

as

a

success

-RRB-

Therefore

we

can

apply

the

Chernoff

bound

-LRB-

-RRB-

Let

c

be

some

constant

we

are

free

to

choose

P

Xi

clogn

loglogn

exp

log

clogn

kloglogn

clogn

loglogn

n?c

logloglogn

log

-LRB-

k

c

-RRB-

loglogn

n?b

for

suf?ciently

large

n

and

an

appropriate

choice

of

c.

Using

the

maximum

rule

-LRB-

-RRB-

we

conclude

that

maxn

i

Xi

is

also

in

O

logn

loglogn

Using

similar

techniques

we

can

also

derive

the

following

bounds

Lemma

The

maximum

number

of

new

elements

to

be

inserted

into

on

any

PE

during

a

cycle

is

in

O

-LRB-

logn

-RRB-

Lemma

The

maximum

number

of

elements

moved

from

to

on

any

PE

during

logn

calls

to

deleteMin

is

in

O

-LRB-

logn

-RRB-

Lemma

At

any

moment

O

-LRB-

logn

-RRB-

Furthermore

the

elements

are

very

evenly

distributed

over

the

local

memories

Lemma

At

any

moment

mn

O

pmn

logn

logn

Proofs

of

Lemmata

Similar

to

the

proof

of

Lemma

For

Lemma

we

change

the

number

of

Bernoulli

trials

to

knlogn

For

Lemma

observe

that

during

logn

calls

the

nlogn

globally

smallest

elements

are

removed

In

the

worst

case

they

all

have

to

be

extracted

from

But

since

they

have

been

placed

randomly

no

PE

will

have

more

than

O

-LRB-

logn

-RRB-

of

them

Lemma

is

a

consequence

of

the

Lemmata

and

and

of

the

summation

rule

-LRB-

-RRB-

For

m

O

-LRB-

nlogn

-RRB-

Lemma

can

also

be

proved

in

an

analogous

way

Otherwise

let

Xi

denote

the

number

of

elements

in

at

PE

i

We

can

use

the

Chernoff

bound

-LRB-

-RRB-

to

see

that

P

h

Xi

m

n

p

logn

i

P

h

Xi

nm

logn

m

n

i

e

mn

n?b

for

suf?ciently

large

n

The

remainder

of

the

argument

is

again

analogous

to

the

proof

of

Lemma

Extracting

Elements

Now

assume

we

want

to

remove

the

smallest

elements

from

-LRB-

In

our

case

n.

-RRB-

Figure

outlines

an

ef?cient

probabilistic

algorithm

for

doing

this

We

maintain

a

candidate

set

-LRB-

initially

-RRB-

and

a

set

of

elements

Qout

known

to

belong

to

the

smallest

ones

is

the

size

of

The

algorithm

is

related

to

the

well

known

sequential

quicksortlike

median

selection

algorithm

-LSB-

Section

-RSB-

However

instead

of

choosing

a

single

pivot

for

partitioning

the

remaining

candidates

we

try

to

partition

into

three

parts

elements

which

are

certain

to

belong

to

the

smallest

elements

those

which

are

certainly

not

among

the

smallest

ones

and

a

-LRB-

hopefully

small

-RRB-

set

of

remaining

candidates

for

the

next

iteration

First

a

random

sample

of

size

n

is

selected

-LRB-

It

simpli?es

the

analysis

to

assume

that

this

is

done

with

replacement

i.e.

elements

may

be

selected

for

multiple

samples

-RRB-

We

then

rank

the

sample

elements

and

choose

two

pivots

u

and

l

spaced

by

n

e

from

an

estimate

for

an

element

with

rank

close

to

e

is

a

small

constant

which

we

are

free

to

choose

We

argue

that

with

high

probability

at

least

elements

will

be

smaller

than

u

such

that

elements

larger

than

u

can

be

excluded

from

consideration

and

similarly

that

there

are

no

more

than

elements

smaller

than

l

such

that

those

that

are

smaller

can

safely

be

selected

This

randomized

partitioning

process

is

repeated

until

a

trivial

case

occurs

Lemma

There

is

an

e

such

that

the

number

of

partitioning

iterations

is

in

O

-LRB-

-RRB-

The

proof

is

based

on

Chernoff

bounds

and

on

the

fact

that

for

an

appropriate

choice

of

e

the

number

of

elements

between

l

and

u

is

small

with

high

probability

Very

similar

arguments

can

also

be

found

in

-LSB-

-RSB-

Proof

In

the

worst

case

we

have

logn

O

-LRB-

nlogn

-RRB-

-LRB-

At

most

O

-LRB-

nlogn

-RRB-

new

elements

during

a

cycle

and

at

most

nlogn

elements

from

each

-RRB-

If

we

choose

e

then

for

suf?ciently

large

n

iterations

decrease

below

n

with

high

probability

-LRB-

We

use

the

relations

and

and

the

fact

that

the

conjunction

of

a

constant

number

of

propositions

which

are

true

with

high

probability

is

also

true

with

high

probability

-RRB-

By

carefully

implementing

all

the

required

operations

we

get

the

following

time

bound

for

the

selection

algorithm

forall

PEs

dopar

synchronously

-LRB-

solution

candidates

-RRB-

j

-LRB-

number

of

remaining

candidates

-RRB-

Qout

-LRB-

elements

already

found

-RRB-

while

n

do

choose

n

random

samples

from

with

replacement

let

si

denote

the

i-th

smallest

sample

-LRB-

sort

-RRB-

-LRB-

si

for

i

si

for

i

n

-RRB-

u

s

n

n

e

-LRB-

upper

pivot

-RRB-

l

s

n

n

e

-LRB-

lower

pivot

-RRB-

if

jfv

v

ugj

then

nfv

v

ug

if

jfv

v

lgj

then

jfv

v

lgj

Qout

Qout

-LSB-

fv

v

lg

nfv

v

lg

j

if

then

Qout

Qout

-LSB-

else

if

then

-LRB-

n

sorting

is

easy

now

-RRB-

sort

insert

the

smallest

elements

into

their

local

Qout

Figure

Algorithm

for

nding

the

smallest

elements

from

Lemma

Extracting

the

n

smallest

elements

from

takes

time

O

-LRB-

Tcoll

-RRB-

Proof

An

iteration

involves

the

following

communication

operations

ffl

A

pre?x

sum

for

enumerating

the

elements

in

ffl

Routing

randomly

selected

samples

to

different

PEs

Since

the

elements

have

been

placed

randomly

the

samples

will

be

evenly

distributed

over

the

PEs

such

that

routing

them

is

possible

in

time

O

-LRB-

Tcoll

-RRB-

ffl

Ranking

the

n

samples

An

algorithm

performing

this

in

time

O

-LRB-

Tcoll

-RRB-

can

for

example

be

found

in

-LSB-

Section

-RSB-

ffl

A

constant

number

of

reductions

and

broadcasts

for

counting

elements

and

disseminating

pivots

All

other

operations

are

performed

locally

By

representing

and

Qout

as

two

indices

into

the

sorted

array

the

local

operations

can

be

performed

in

time

O

-LRB-

-RRB-

O

-LRB-

-RRB-

O

-LRB-

logn

-RRB-

using

binary

search

All

operations

inside

the

loop

can

be

performed

in

time

O

-LRB-

Tcoll

logn

-RRB-

O

-LRB-

Tcoll

-RRB-

The

same

is

true

for

the

operations

outside

the

loop

Since

the

number

of

iterations

is

in

O

-LRB-

-RRB-

we

can

conclude

that

the

total

time

for

selection

is

in

O

-LRB-

Tcoll

logn

-RRB-

O

-LRB-

Tcoll

-RRB-

-LRB-

using

the

product

rule

-LRB-

-RRB-

-RRB-

Putting

the

Pieces

Together

We

now

have

all

the

required

results

to

come

back

to

the

analysis

of

Algorithm

Theorem

A

call

to

insert

or

deleteMin

requires

amortized

time

in

O

Tcoll

log

mn

-LRB-

amortized

over

logn

calls

-RRB-

Proof

We

look

at

the

total

execution

time

for

logn

calls

of

both

functions

Since

O

-LRB-

logn

-RRB-

-LRB-

Lemma

-RRB-

emptying

into

takes

time

O

log

mn

logn

According

to

Lemma

there

are

only

O

-LRB-

logn

-RRB-

iterations

of

the

while

loop

-LRB-

summed

over

logn

calls

-RRB-

for

moving

elements

from

to

each

of

which

involves

two

reductions

and

one

priority

queue

access

-LRB-

time

O

Tcoll

log

mn

-RRB-

The

remaining

operations

have

to

be

counted

for

each

call

-LRB-

i.e.

logn

times

-RRB-

Extracting

the

n

smallest

elements

takes

time

O

-LRB-

Tcoll

-RRB-

-LRB-

Lemma

-RRB-

and

assigning

these

elements

to

the

PEs

can

be

done

using

a

pre?x

sum

and

a

routing

operation

-LRB-

time

O

-LRB-

Tcoll

-RRB-

-RRB-

O

logn

loglogn

new

elements

-LRB-

for

each

call

of

insert

Lemma

-RRB-

can

be

inserted

into

in

time

O

-LRB-

logn

-RRB-

by

rst

sorting

them

-LRB-

e.g.

by

heap-sort

-RRB-

and

then

merging

them

with

Summing

all

this

together

-LRB-

Relations

-LRB-

-RRB-

-RRB-

yields

time

O

Tcoll

log

mn

logn

This

result

can

for

example

be

used

to

devise

a

parallel

branch-and-bound

algorithm

which

is

provably

ef?cient

if

Tx

W?Tcoll

log

mn

Theorem

The

parallel

execution

time

of

synchronous

best

rst

branch-and-bound

using

algorithm

is

Tpar

m

n

h

Tx

O

Tcoll

log

m

n

Proof

mn

h

iterations

are

suf?cient

to

complete

the

search

For

m

polynomial

in

n

the

theorem

is

an

immediate

consequence

of

Theorem

and

the

summation

rule

-LRB-

-RRB-

Else

we

must

additionally

exploit

that

the

execution

times

of

different

cycles

are

independent

of

each

other

because

a

cycle

deterministically

removes

the

nlogn

smallest

elements

without

moving

the

other

elements

Therefore

with

high

probability

only

a

polynomially

small

fraction

of

the

cycles

will

require

a

time

exceeding

the

limit

from

Theorem

These

slow

cycles

can

not

signi?cantly

change

the

total

execution

time

because

the

worst

case

time

for

a

cycle

involves

only

a

polynomial

number

of

operations

Re?nements

Algorithm

is

mainly

designed

for

simplicity

and

analyzability

We

now

describe

improvements

which

might

be

useful

for

a

real

implementation

After

introducing

some

simple

enhancements

in

Section

we

show

how

the

number

of

communication

operations

can

be

reduced

at

the

cost

of

increasing

message

sizes

in

Section

and

we

explain

how

an

asynchronous

machine

can

be

used

more

ef?ciently

in

Section

In

Section

we

outline

how

the

algorithm

can

be

implemented

ef?ciently

on

shared

memory

machines

Simple

Enhancements

As

a

rst

improvement

note

that

the

loop

in

Algorithm

while

jfv

v

min

n

do

-LSB-

fdeleteMin

-LRB-

-RRB-

g

yields

two

kinds

of

information

that

can

be

used

to

speed

up

the

subsequent

extraction

process

While

the

condition

is

true

all

elements

smaller

than

min

can

safely

be

selected

When

it

is

false

elements

larger

than

min

can

be

excluded

from

consideration

Queue

maintenance

costs

can

be

reduced

by

using

the

leftist

tree

variant

for

representing

and

-LSB-

-RSB-

Emptying

into

can

then

be

performed

in

time

O

-LRB-

logm

-RRB-

by

merging

the

two

trees

In

addition

elements

immediately

fetched

back

into

when

deleteMin

is

called

after

the

end

of

a

cycle

can

be

retrieved

in

constant

time

Furthermore

if

sending

elements

involves

long

messages

we

will

only

put

the

key

and

a

reference

to

the

element

in

Q

This

reduces

the

number

of

times

an

element

is

moved

from

two

to

one

In

many

branch-and-bound

applications

most

inserted

elements

are

never

actually

expanded

and

do

not

need

to

be

moved

at

all

Using

Coarser

Grained

Communication

Algorithm

employs

a

relatively

large

number

of

ne-grained

communication

operations

On

many

machines

it

is

better

to

use

proportionately

more

coarse-grained

but

fewer

operations

in

total

Starting

with

some

simple

measures

we

go

on

to

a

new

algorithm

for

parallel

selection

and

nally

consider

accessing

many

more

than

n

elements

at

once

Coalescing

Loop

Iterations

We

can

coalesce

several

iterations

of

the

while

loop

for

moving

elements

from

to

into

two

vector

operations

for

minimum

determination

and

counting

At

the

cost

of

increasing

the

candidate

set

for

selection

we

can

even

completely

eliminate

the

above

while

loop

Since

the

n

best

elements

are

randomly

placed

we

know

that

considering

the

rst

O

logn

loglogn

elements

of

is

suf?cient

to

catch

all

the

n

best

ele

ments

with

high

probability

The

rare

cases

when

this

is

insuf?cient

can

be

caught

later

on

by

checking

on

each

PE

whether

the

result

of

the

selection

routine

is

smaller

than

the

smallest

remaining

local

element

In

this

case

an

exception

can

be

raised

triggering

a

fallback

implementation

which

can

be

arbitrarily

slow

if

it

is

needed

suf?ciently

rarely

For

example

in

our

implementation

we

looked

at

the

min

-LRB-

n

logn

loglogn

-RRB-

locally

smallest

elements

and

using

the

Chernoff

bound

-LRB-

-RRB-

it

is

easy

to

show

that

the

probability

of

a

buffer

over?ow

on

any

PE

is

below

for

all

n.

Similarly

the

number

of

iterations

of

the

selection

procedure

in

Figure

can

be

reduced

by

sorting

a

larger

sample

or

using

more

than

two

pivots

at

a

time

A

particularly

interesting

scheme

for

choosing

pivots

is

to

use

ui

s

n

i

li

s

n

-LRB-

i

logn

-RRB-

The

number

of

pivots

is

quite

moderate

and

the

pivots

lie

very

densely

around

the

estimate

for

the

smallest

element

Also

there

are

no

tuning

parameters

-LRB-

like

e

-RRB-

which

we

would

have

to

adapt

for

maximal

performance

Selection

with

a

Single

Reduction

For

moderate

n

or

machines

with

very

coarse-grained

communication

characteristics

we

could

simply

perform

a

reduction

over

using

merge

as

the

associative

operator

The

root

PE

will

then

receive

in

sorted

order

and

simply

needs

to

broadcast

the

n-th

smallest

element

This

crude

approach

can

be

improved

by

throwing

away

very

large

and

very

small

elements

after

each

merging

operation

Since

the

elements

are

allocated

randomly

it

is

rather

improbable

that

the

n-th

smallest

element

is

among

the

omitted

elements

So

it

suf?ces

to

count

them

Figure

explicates

this

idea

It

is

described

for

the

general

case

of

selecting

the

k-th

smallest

out

of

m

randomly

allocated

and

locally

sorted

elements

based

on

sequences

of

size

The

function

prunedMerge

processes

two

-LRB-

s

j

x

j

D

j

y

j

-RRB-

-LRB-

j

-RRB-

containing

from

left

to

right

the

sequences

to

be

merged

the

largest

element

omitted

as

too

small

the

number

of

elements

omitted

as

too

small

the

smallest

element

omitted

as

too

large

and

the

number

of

processors

from

which

the

data

has

been

collected

The

two

sequences

are

merged

and

the

information

regarding

omitted

elements

is

rst

computed

based

on

the

assumption

that

no

pruning

takes

place

The

resulting

is

then

pruned

First

the

position

i

of

the

k-th

smallest

element

is

estimated

based

on

the

available

information

Then

the

elements

outside

the

index-range

-LSB-

i

i

-RSB-

are

thrown

away

and

the

information

on

omitted

elements

is

updated

accordingly

At

the

root

of

the

reduction

tree

s

-LSB-

k

D

-RSB-

will

be

the

correct

answer

if

k

D

is

a

legal

index

and

x

s

-LSB-

k

D

-RSB-

y.

Otherwise

we

can

use

a

slow

fallback

implementation

The

following

lemma

answers

the

question

of

how

large

the

sequence

size

needs

to

be

in

order

to

make

a

successful

selection

highly

probable

Lemma

It

suf?ces

to

choose

O

max

-LRB-

pklogn

logn

-RRB-

in

order

to

make

selection

with

pruned

merge

succeed

with

high

probability

Proof

Let

em

denote

the

considered

element

Let

X

denote

the

number

of

elements

ei

ek

allocated

to

the

processors

under

consideration

and

consider

some

b

Function

select

-LRB-

k

m

s

-RRB-

-LRB-

s

x

D

y

-RRB-

globalReduce

-LRB-

prunedMerge

-LRB-

k

m

-RRB-

prune

-LRB-

k

m

-LRB-

s

-RRB-

-RRB-

-RRB-

if

k

D

x

s

-LSB-

k

D

-RSB-

y

then

return

s

-LSB-

k

D

-RSB-

else

return

slowSelect

-LRB-

k

m

s

-RRB-

Function

prunedMerge

-LRB-

k

m

-LRB-

-RRB-

-LRB-

-RRB-

-RRB-

return

prune

-LRB-

k

m

-LRB-

merge

-LRB-

-RRB-

max

-LRB-

-RRB-

min

-LRB-

-RRB-

-RRB-

-RRB-

Function

prune

-LRB-

k

m

-LRB-

s

x

D

y

-RRB-

-RRB-

i

round

k

n

D

-LRB-

omitted

cases

where

-RRB-

-LRB-

not

both

sides

of

the

sequence

need

pruning

-RRB-

return

-LRB-

s

-LSB-

i

i

-RSB-

max

-LRB-

x

s

-LSB-

i

-RSB-

-RRB-

D

i

min

-LRB-

y

s

-LSB-

i

-RSB-

-RRB-

-RRB-

Figure

Selection

by

pruned

merging

If

the

selection

fails

there

must

be

at

least

one

call

to

prune

where

s

still

contains

ek

but

at

a

position

where

it

is

thrown

away

We

now

prove

that

the

probability

for

this

event

is

polynomially

small

for

any

of

the

calls

to

prune

regardless

of

the

order

in

which

the

reduction

is

performed

Hence

the

total

failure

probability

is

also

polynomially

small

Case

ek

is

thrown

away

as

too

small

This

implies

that

X

k

n

-LRB-

-RRB-

k

n

The

probability

of

this

event

can

be

bounded

using

Chernoff

bound

-LRB-

-RRB-

P

X

n

e

n

logn

n?b

if

logn

b

This

is

equivalent

to

q

n

logn

p

Case

ek

is

thrown

away

as

too

large

Case

If

k

n

we

can

use

Chernoff

bound

-LRB-

-RRB-

in

a

way

analogous

to

Case

Case

If

k

n

but

for

some

e

we

can

also

use

bound

-LRB-

-RRB-

by

estimating

P

h

X

k

n

P

h

X

-LRB-

e

-RRB-

k

n

i

Case

Otherwise

a

rather

crude

estimation

using

bound

-LRB-

-RRB-

suf?ces

P

X

n

P?X

e

log

n?b

for

log

blogn

For

it

suf?ces

to

have

log

blogn

and

this

is

true

if

logn

O

-LRB-

logn

-RRB-

Lemma

can

be

applied

to

the

selection

problem

in

deleteMin

by

setting

k

n.

Note

that

for

the

case

that

we

apply

selection

to

the

O

logn

loglogn

best

elements

on

each

PE

these

are

not

placed

independently

But

the

entire

set

of

m

elements

was

placed

independently

and

the

n

best

elements

are

among

the

considered

elements

with

high

probability

so

we

will

get

the

same

result

Corollary

Selecting

the

best

n

elements

in

deleteMin

can

be

done

using

a

single

reduction

with

input

data

of

length

O

pnlogn

This

is

asymptotically

not

as

ef?cient

as

the

more

complicated

algorithms

but

for

many

practical

machines

it

will

be

faster

It

is

also

a

considerable

improvement

over

the

unpruned

merge

Batched

Operation

The

number

of

communications

per

element

can

be

further

decreased

by

inserting

and

deleting

larger

batches

of

elements

at

once

This

is

quite

straightforward

for

deleteMin

In

-LSB-

-RSB-

it

is

proved

that

for

inserting

k

W?n

-LRB-

logn

-RRB-

elements

it

is

not

necessary

to

place

them

completely

independently

Rather

it

suf?ces

to

randomly

put

them

into

O

-LRB-

logn

-RRB-

containers

and

to

place

the

containers

randomly

In

addition

by

using

a

generalized

kn

bandwidth

heap

which

stores

kn

elements

in

every

node

the

local

queue

access

can

be

made

more

ef?cient

-LSB-

-RSB-

Using

large

batches

is

often

equivalent

to

emulating

k

logical

PEs

on

each

physical

PE

This

only

works

if

there

is

suf?cient

parallelism

in

the

problem

instance

Furthermore

the

required

work

can

increase

For

example

for

branch-and-bound

the

number

of

expanded

nodes

can

grow

from

Q

mn

h

to

Q

mn

kh

A

special

form

of

batched

operation

is

initialization

This

is

even

faster

than

batched

insertion

because

we

can

use

the

linear

time

sequential

initialization

algorithm

locally

we

do

not

need

specialized

algorithms

as

described

in

-LSB-

-RSB-

for

k-bandwidth

heaps

Note

that

if

a

batch

of

w

-LRB-

m

logm

-RRB-

is

to

be

inserted

it

is

faster

to

reinitialize

all

the

local

queues

rather

than

inserting

all

new

elements

inidividually

Asynchronous

Operation

For

some

applications

synchronized

parallel

calls

of

deleteMin

are

not

adequate

The

time

required

to

process

an

element

will

vary

Therefore

it

would

be

wasteful

to

wait

until

all

n

PEs

have

completed

their

work

before

a

new

queue

access

is

started

One

way

to

solve

this

problem

is

to

separate

priority

queue

server

threads

from

client

threads

doing

the

real

work

The

server

threads

could

start

a

collective

queue

access

whenever

en

client

threads

have

sent

a

request

Depending

on

the

machine

architecture

the

server

threads

could

run

on

the

same

PE

as

the

clients

on

communication

coprocessors

or

on

a

smaller

set

of

dedicated

server

PEs

However

the

above

solution

implies

a

delicate

tradeoff

between

long

waiting

times

for

large

e

and

high

communication

overhead

for

small

e

It

is

more

elegant

to

always

have

some

of

the

smallest

queue

elements

in

reserve

which

can

be

retrieved

asynchronously

using

a

distributed

FIFO

Such

a

data

structure

can

be

implemented

with

little

contention

using

parallel

counting

algorithms

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

When

-LRB-

or

even

before

-RRB-

this

reserve

is

exhausted

a

new

batch

of

-LRB-

say

n

-RRB-

elements

is

retrieved

A

price

we

pay

for

this

approach

is

that

the

elements

in

the

reserve

might

already

be

outdated

when

they

are

retrieved

since

smaller

elements

have

been

inserted

in

the

meantime

But

a

closer

consideration

shows

that

this

is

not

really

a

new

problem

No

priority

queue

which

allows

asynchronous

parallel

insertion

can

guarantee

that

every

element

passed

to

insert

will

instantaneously

become

visible

everywhere

In

fact

if

we

use

a

centralized

queue

it

may

take

time

in

W

-LRB-

nlogm

-RRB-

before

an

element

is

inserted

This

is

much

more

than

the

delay

incurred

by

holding

a

reserve

The

faster

our

routines

the

better

can

we

approximate

the

behavior

of

a

sequential

algorithm

The

maximum

time

it

takes

until

new

elements

become

visible

can

be

bounded

by

periodically

invalidating

and

recomputing

the

reserve

Invalidation

need

not

be

triggered

as

long

as

no

elements

smaller

than

the

largest

reserve

element

are

inserted

Invalidation

can

be

triggered

ealier

then

prescribed

by

the

timeout

when

more

than

a

certain

number

of

elements

smaller

than

reserve

elements

have

been

inserted

Since

such

a

trigger

point

need

not

be

accurate

we

can

also

use

approximated

triggers

as

described

in

-LSB-

-RSB-

which

incur

very

low

overhead

Asynchronous

access

implies

a

number

of

additional

bene?ts

ffl

The

total

work

required

for

inserting

or

deleting

O

-LRB-

n

-RRB-

elements

is

in

O?nlog

mn

so

for

m

not

too

large

most

server

threads

spend

most

of

their

time

waiting

for

the

completion

of

collective

communication

primitives

This

time

can

be

used

by

client

threads

ffl

We

do

not

need

to

determine

exactly

the

n-th

smallest

element

en

during

selection

since

the

size

of

the

reserve

is

exible

So

we

could

break

the

loop

in

Algorithm

after

a

single

iteration

or

invent

a

variant

of

Algorithm

which

does

not

keep

track

of

all

elements

around

the

estimated

position

of

en

but

only

of

a

sample

which

gets

the

sparser

the

farther

away

it

is

from

the

estimated

position

Exploiting

Shared

Memory

In

the

architecture

of

parallel

computers

there

is

a

trend

to

support

shared

memory

-LRB-

with

nonuniform

memory

access

costs

-LRB-

NUMA

-RRB-

-RRB-

even

for

quite

large

n

Our

algorithm

does

not

require

shared

memory

but

it

can

pro?t

from

the

fact

that

remote

memory

accesses

are

often

much

faster

than

sending

and

receiving

small

messages

In

particular

delivery

of

result

elements

in

the

deleteMin

operation

can

be

replaced

by

a

single

remote

write

The

collective

communications

can

also

pro?t

if

there

is

an

ef?cient

way

to

wait

for

the

arrival

of

a

remote

write

-LRB-

Busy

waiting

is

one

possibility

but

this

is

expensive

for

the

asynchronous

algorithms

of

Section

-RRB-

The

insert

operation

is

more

complicated

because

multiple

elements

can

queue

up

at

a

PE

but

some

machines

support

an

ef?cient

fetch-and-increment

operation

which

can

be

used

on

the

element

counter

of

the

target

PE

Hardware

support

for

this

operation

is

also

very

useful

for

maintaining

the

FIFO

queue

required

by

the

asynchronous

algorithm

because

there

are

few

commercial

machines

with

suf?cient

hardware

support

for

message

passing

-LRB-

With

the

notable

exception

of

the

shared

memory

machine

Cray

-LSB-

-RSB-

-RRB-

Implementation

In

order

to

get

an

idea

of

how

practical

the

bottleneck-free

priority

queues

are

on

contemporary

machines

we

have

implemented

the

algorithm

in

a

portable

way

using

the

library

MPI

-LRB-

Message

Passing

Interface

-LSB-

-RSB-

-RRB-

Since

a

quite

large

message

startup

overhead

is

common

it

was

clear

from

the

start

that

the

cost

of

local

queue

access

was

not

so

important

So

we

experimented

only

with

a

single

-LRB-

large

-RRB-

queue

size

For

the

same

reason

algorithms

parallelizing

a

single

queue

access

were

bound

to

be

useless

in

this

setting

Therefore

the

parallel

algorithm

was

compared

with

a

centralized

priority

queue

located

on

a

dedicated

PE

serving

queue

accesses

by

the

other

PEs

Requests

are

collected

using

MPI

Waitsome

for

ef?ciency

and

to

enforce

fairness

-LRB-

The

source

code

is

available

at

-RRB-

Also

the

size

of

the

queue

elements

was

bound

to

have

little

in?uence

-LRB-

besides

limiting

the

maximal

size

of

the

queue

-RRB-

So

we

used

integer

keys

and

integer

data

elds

for

the

measurements

On

the

other

hand

the

number

of

communication

operations

had

to

be

minimized

Therefore

a

simpli?ed

algorithm

was

implemented

which

keeps

the

size

of

below

min

-LRB-

n

logn

loglogn

-RRB-

at

all

times

Furthermore

the

reduction

based

selection

method

from

Section

was

used

So

a

deleteMin-operation

boils

down

to

one

call

of

MPI

Reduce

and

one

call

of

MPI

Bcast

for

the

selection

a

call

to

MPI

Scan

for

enumerating

the

best

elements

O

logn

loglogn

sends

for

delivering

the

re

sults

and

a

single

MPI

Recv

The

complete

source

code

is

available

electronically

at

T

-LSB-

ms

-RSB-

n

parallel

centralized

n

Figure

Median

execution

times

for

inserting

plus

extracting

n

elements

on

an

IBM-SP

with

n

nodes

-LRB-

based

on

repetitions

-RRB-

Figure

shows

the

execution

times

for

one

collective

call

to

deleteMin

plus

one

call

to

insert

inserting

n

elements

into

a

queue

of

size

distributed

over

n

we

can

not

give

the

average

execution

time

since

at

the

time

of

the

measurements

the

machine

was

operated

in

such

a

way

that

some

huge

delays

due

to

external

reasons

occured

-LRB-

This

also

processors

of

the

IBM-SP

at

the

University

of

Karlsruhe

A

centralized

implementation

needs

about

s

per

queue

access

for

large

n.

So

for

more

than

about

PEs

the

new

algorithm

is

faster

By

omitting

parts

of

the

code

and

timing

inner

loops

the

contributions

of

different

kinds

of

operations

were

measured

More

than

half

of

the

execution

time

is

consumed

by

the

reduction

and

scan

operations

Most

of

the

rest

is

due

to

the

other

communication

operations

A

local

queue

access

takes

about

s

for

m

so

it

does

not

in?uence

the

execution

time

very

much

measurments

T

-LSB-

ms

-RSB-

Figure

Execution

time

distribution

for

an

insert

deleteMin

call

on

a

Cray

with

n

m

Both

MPI-programs

were

also

executed

on

a

Cray

For

n

and

m

the

parallel

priority

queue

needs

an

average

of

ms

for

inserting

plus

deleting

n

elements

This

is

about

times

faster

than

the

centralized

code

-LRB-

with

m

-RRB-

which

needs

about

s

for

inserting

plus

deleting

one

element

-LRB-

i.e.

ms

for

elements

-RRB-

We

did

not

have

the

opportunity

to

run

a

complete

measurement

series

on

the

but

even

if

the

parallel

program

would

not

be

faster

for

smaller

n

the

break-even

point

where

the

parallel

algorithm

is

faster

than

a

centralized

queue

would

lie

below

on

the

The

histogram

in

Figure

for

the

run

times

of

calls

to

insert

plus

deleteMin

demonstrates

that

substantial

variations

are

very

rare

happens

with

other

applications

-RRB-

But

the

results

on

the

show

that

the

median

is

within

one

percent

of

the

average

value

under

normal

conditions

would

like

to

thank

the

university

computing

centre

in

Karlsruhe

and

its

staff

for

making

available

Wide

Nodes

-LRB-

with

MHz

clock

-RRB-

connected

by

a

High

Performance

Switch

This

required

a

special

setup

wich

could

not

be

repeated

for

the

measurements

of

the

centralized

algorithm

But

for

n

not

too

small

the

measured

data

is

modeled

quite

reliably

by

a

linear

function

in

n.

Conclusions

Parallel

priority

queues

based

on

random

placement

of

elements

and

parallel

selection

are

very

ef?cient

for

parallel

access

by

all

processors

On

powerful

models

of

computation

like

EREW-PRAMs

the

algorithm

is

as

ef?cient

as

previously

known

but

more

complex

algorithms

More

importantly

the

algorithm

also

performs

well

on

networks

of

processors

Both

insert

and

deleteMin

of

O

-LRB-

n

-RRB-

out

of

m

elements

can

be

performed

in

amortized

time

O

Tcoll

log

mn

on

n

PEs

In

particular

on

-LRB-

single

ported

-RRB-

hypercubes

on

constant

degree

hypercubic

networks

like

butter?ies

CCC

perfect

shuf?e

or

DeBruijn

and

on

combinations

of

multistage

networks

with

tree

networks

the

execution

time

is

in

O

logn

log

mn

which

is

an

improvement

on

all

previously

known

algorithms

The

algorithm

is

also

of

practical

interest

since

it

outperforms

the

centralized

approach

for

n

on

mainstream

parallel

computers

Other

implementations

could

only

clearly

demonstrate

this

for

batched

access

to

k

AE

n

elements

-LRB-

using

a

similar

approach

-RRB-

In

part

this

success

is

due

to

the

new

selection

algorithm

based

on

a

single

reduction

Nevertheless

the

asymptotically

more

ef?cient

selection

algorithm

from

Figure

is

of

independent

interest

since

with

its

execution

time

in

O

-LRB-

Tcoll

-RRB-

for

randomly

placed

data

it

beats

the

worst

case

lower

bound

for

deterministic

algorithms

given

in

-LSB-

-RSB-

Future

Work

We

are

currently

working

on

a

re?nement

of

the

algorithm

which

also

supports

delete

and

decreaseKey

ef?ciently

This

can

be

done

by

representing

and

as

relaxed

heaps

-LSB-

-RSB-

and

introducing

an

additional

relaxed

heap

Qd

holding

decreased

keys

Elements

are

placed

in

a

reproducible

way

using

a

hash

function

The

probabilistic

analysis

still

applies

as

long

as

the

operations

performed

by

the

application

do

not

depend

on

the

hash

function

An

ef?cient

and

portable

implementation

of

asynchronous

parallel

priority

queues

raises

some

interesting

questions

On

the

implementation

side

there

is

some

hope

that

in

the

near

future

ef?cient

implementations

of

MPI

combined

with

POSIX

Threads

will

be

available

This

might

be

the

right

platform

for

developing

algorithms

along

the

lines

of

Section

Acknowledgements

I

would

like

to

thank

Gerth

Brodal

Sajal

Das

Wolfgang

Dittrich

Christina

Pinotti

Thomas

Worsch

and

an

anonymous

referee

for

many

helpful

comments

References

-LSB-

-RSB-

A.

B?aumker

W.

Dittrich

F.

Meyer

auf

der

Heide

and

I.

Rieping

Realistic

parallel

algorithms

Priority

queue

operations

and

selection

for

the

BSP

model

In

Europar

number

in

LNCS

pages

-LSB-

-RSB-

G.

S.

Brodal

J.

L.

Tr?aff

and

C.

D.

Zaroliagis

A

parallel

priority

data

structure

with

applications

Journal

of

Parallel

and

Distributed

Computing

This

issue

-LSB-

-RSB-

T.

H.

Cormen

C.

E.

Leiserson

and

R.

L.

Rivest

Introduction

to

Algorithms

McGraw-Hill

-LSB-

-RSB-

S.

K.

Das

M.

C.

Pinotti

and

F.

Sarkar

Optimal

and

load

balanced

mapping

of

parallel

priority

queues

in

hypercubes

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

-LSB-

-RSB-

N.

Deo

and

S.

Prasad

Parallel

heap

In

International

Conference

on

Parallel

Processing

volume

III

pages

-LSB-

-RSB-

J.

R.

Driscoll

H.

N.

Gabow

R.

Shrairman

and

R.

E.

Tarjan

Relaxed

heaps

An

alternative

to

bonacci

heaps

with

applications

to

parallel

computation

Communications

of

the

ACM

-LRB-

-RRB-

-LSB-

-RSB-

R.

Fujimoto

Parallel

discrete

event

simulation

Communications

of

the

ACM

-LRB-

-RRB-

-LSB-

-RSB-

A.

V.

Gerbessiotis

and

C.

J.

Siniolakis

Selection

on

the

bulk-synchronous

parallel

model

with

applications

to

priority

queues

In

International

Conference

on

Parallel

and

Distributed

Processing

Techniques

and

Applications

-LSB-

-RSB-

A.

K.

Gupta

and

A.

G.

Phoutiou

Load

balanced

priority

queue

implementations

on

distributed

memory

parallel

machines

In

Sixth

International

Conference

on

Parallel

Architectures

and

Languages

Europe

number

in

LNCS

pages

Athens

July

Springer

-LSB-

-RSB-

D.

Ierardi

in

average

time

O

-LRB-

pN

lg

N

-RRB-

In

ACM

Symposium

on

Parallel

Architectures

and

Algorithms

pages

-LSB-

-RSB-

L.

V.

Kale

B.

Ramkumar

V.

Saletore

and

A.

B.

Sinha

Prioritization

in

parallel

symbolic

computing

In

R.

H.

Halstead

editor

Parallel

Symbolic

Computing

number

in

LNCS

pages

Springer

-LSB-

-RSB-

L.

V.

Kale

B.

H.

Richards

and

T.

D.

Allen

Ef?cient

graph

coloring

with

prioritization

In

Parallel

Symbolic

Languages

and

Systems

number

in

LNCS

pages

Springer

-LSB-

-RSB-

R.

M.

Karp

and

Y.

Zhang

Parallel

algorithms

for

backtrack

search

and

branch-andbound

Journal

of

the

ACM

-LRB-

-RRB-

-LSB-

-RSB-

D.

E.

Knuth

The

Art

of

Computer

Programming

Sorting

and

Searching

volume

Addison

Wesley

-LSB-

-RSB-

T.

Leighton

Introduction

to

Parallel

Algorithms

and

Architectures

Morgan

Kaufmann

-LSB-

-RSB-

F.

Mattern

and

H.

Mehl

Diskrete

Simulation

Prinzipien

und

Probleme

der

Ef?zienzsteigerung

durch

Parallelisierung

Informatik

Spektrum

-LSB-

-RSB-

W.

F.

McColl

Scalable

computing

In

Computer

Science

Today

number

in

LNCS

pages

Springer

-LSB-

-RSB-

S.

Olariu

and

Z.

Wen

Optimal

parallel

initialization

algorithms

for

a

class

of

priority

queues

IEEE

Transactions

on

Parallel

and

Distributed

Systems

-LRB-

-RRB-

October

-LSB-

-RSB-

C.

G.

Plaxton

On

the

network

complexity

of

selection

In

Foundations

of

Computer

Science

pages

IEEE

-LSB-

-RSB-

S.

Rajasekaran

Randomized

parallel

selection

In

Tenth

Conference

on

Foundations

of

Software

Technology

and

Theoretical

Computer

Science

number

in

LNCS

pages

Bangalore

Springer

-LSB-

-RSB-

A.

Ranade

A

simpler

analysis

of

the

Karp-Zhang

parallel

branch-and-bound

method

Technical

report

University

of

California

Berkeley

-LSB-

-RSB-

A.

Ranade

S.

Cheng

E.

Deprit

J.

Jones

and

S.

Shih

Parallelism

and

locality

in

priority

queues

In

Sixth

IEEE

Sypmposium

on

Parallel

and

Distributed

Processing

pages

October

-LSB-

-RSB-

V.

N.

Rao

and

V.

Kumar

Concurrent

access

of

priority

queues

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LSB-

-RSB-

R.

Reischuk

Probabilistic

parallel

algorithms

for

sorting

and

selection

SIAM

Journal

on

Computing

-LRB-

-RRB-

-LSB-

-RSB-

P.

Sanders

Flaschenhalsfreie

parallele

Priority

queues

In

PARS

Workshop

Potsdam

number

in

PARS

Mitteilungen

pages

September

-LSB-

-RSB-

P.

Sanders

Fast

priority

queues

for

parallel

branch-and-bound

In

Workshop

on

Algorithms

for

Irregularly

Structured

Problems

number

in

LNCS

pages

Lyon

Springer

-LSB-

-RSB-

P.

Sanders

Lastverteilungsalgorithmen

f?ur

parallele

Tiefensuche

Number

in

Fortschrittsberichte

Reihe

VDI

Verlag

-LSB-

-RSB-

S.

L.

Scott

Synchronization

and

communication

in

the

multiprocessor

In

ASPLOS-VII

Architectural

Support

for

Programming

Languages

and

Operating

Systems

pages

ACM

-LSB-

-RSB-

N.

Shavit

and

A.

Zemach

Diffracting

trees

In

ACM

Symposium

on

Parallel

Architectures

and

Algorithms

pages

Cape

May

New

Jersey

-LSB-

-RSB-

M.

Snir

S.

W.

Otto

S.

Huss-Lederman

D.

W.

Walker

and

J.

Dongarra

MPI

the

Complete

Reference

MIT

Press

analysis of randomized algorithms
distributed memory
load balancing
median selection
parallel priority queue