CS

-LCB-

-LCB-

Algorithms

for

Parallel

Memory

I

Two-Level

Memories

Jeffrey

Scott

Vitter

Elizabeth

A.

M.

Shriver

Department

of

Computer

Science

Duke

University

Durham

North

Carolina

January

Algorithms

for

Parallel

Memory

I

Two-Level

Memories

Jeffrey

Scott

Vitter

y

Dept.

of

Computer

Science

Duke

University

Box

Durham

N.

C.

-LCB-

Elizabeth

A.

M.

Shriver

z

Courant

Institute

New

York

University

Mercer

Street

New

York

N.

Y.

Abstract

We

provide

the

first

optimal

algorithms

in

terms

of

the

number

of

input/outputs

-LRB-

I/Os

-RRB-

required

between

internal

memory

and

multiple

secondary

storage

devices

for

the

problems

of

sorting

FFT

matrix

transposition

standard

matrix

multiplication

and

related

problems

Our

two-level

memory

model

is

new

and

gives

a

realistic

treatment

of

parallel

block

transfer

in

which

during

a

single

I/O

each

of

the

P

secondary

storage

devices

can

simultaneously

transfer

a

contiguous

block

of

B

records

The

model

pertains

to

a

large-scale

uniprocessor

system

or

parallel

multiprocessor

system

with

P

disks

In

addition

the

sorting

FFT

permutation

network

and

standard

matrix

multiplication

algorithms

are

typically

optimal

in

terms

of

the

amount

of

internal

processing

time

The

difficulty

in

developing

optimal

algorithms

is

to

cope

with

the

partitioning

of

memory

into

P

separate

physical

devices

Our

algorithms

performance

can

be

significantly

better

than

those

obtained

by

the

well-known

but

nonoptimal

technique

of

disk

striping

Our

optimal

sorting

algorithm

is

randomized

but

practical

the

probability

of

using

more

than

times

the

optimal

number

of

I/Os

is

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

where

M

is

the

internal

memory

size

This

paper

will

appear

in

a

special

issue

of

Algorithmica

on

the

subject

of

Large-Scale

Memories

A

summarized

version

of

this

research

was

presented

at

the

Annual

ACM

Symposium

on

Theory

of

Computing

Baltimore

MD

May

yThis

work

was

done

while

the

author

was

at

Brown

University

Support

was

provided

in

part

by

a

National

Science

Foundation

Presidential

Young

Investigator

Award

with

matching

funds

from

IBM

by

NSF

research

grants

DCR

-LCB-

and

CCR

-LCB-

by

Army

Research

Office

grant

-LCB-

-LCB-

G

-LCB-

and

by

the

Office

of

Naval

Research

and

the

Defense

Advanced

Research

Projects

Agency

under

contract

-LCB-

-LCB-

J

-LCB-

ARPA

order

zThis

work

was

done

in

part

while

the

author

was

at

Brown

University

supported

by

a

Bellcore

graduate

fellowship

and

at

Bellcore

Contents

Introduction

The

Two-Level

Memory

Model

Problem

Definitions

Main

Results

Shuffle-Merge

and

its

Applications

External

Sorting

and

Permuting

Phase

Phase

Completing

the

Sort

Finding

the

Partitioning

Elements

Permuting

for

Very

Small

P

and

B

Standard

Matrix

Multiplication

Conclusions

References

i

Introduction

Sorting

is

the

canonical

information-processing

application

It

accounts

for

roughly

-LCB-

percent

of

the

computing

resources

on

large-scale

computers

-LSB-

Knu

LiV

-RSB-

In

applications

where

the

file

of

records

can

not

fit

into

internal

memory

the

records

must

be

stored

on

-LRB-

external

-RRB-

secondary

storage

usually

in

the

form

of

disks

Sorting

in

this

framework

is

called

external

sorting

The

bottleneck

in

external

sorting

and

many

other

applications

is

the

time

for

the

input/output

-LRB-

I/O

-RRB-

between

internal

memory

and

the

disks

-LSB-

Knu

LiV

-RSB-

This

bottleneck

is

accentuated

as

processors

get

faster

and

parallel

computers

are

used

The

remedy

we

explore

in

this

paper

is

to

use

secondary

storage

systems

with

parallel

capabilities

-LSB-

HGK

Jil

Mag

PGK

Uni

-RSB-

We

restrict

our

attention

in

this

paper

to

two-level

storage

systems

with

random

access

secondary

storage

Magnetic

disks

for

example

provide

the

functionality

needed

in

our

model

of

secondary

storage

so

for

simplicity

we

shall

refer

to

secondary

storage

as

disk

storage

consisting

of

one

or

more

disk

drives

Efficient

algorithms

for

multilevel

hierarchical

memory

are

considered

in

the

companion

paper

-LSB-

ViS

-RSB-

In

previous

work

Aggarwal

and

Vitter

-LSB-

AgV

-RSB-

presented

optimal

upper

and

lower

bounds

on

the

I/O

needed

for

sorting-related

problems

of

size

N

using

a

two-level

memory

model

where

internal

memory

can

store

M

records

and

the

secondary

memory

size

is

limitless

In

their

model

an

I/O

can

simultaneously

transfer

P

physical

blocks

each

consisting

of

B

contiguous

records

Their

results

generalized

the

groundbreaking

work

done

by

Floyd

-LSB-

Flo

-RSB-

who

gave

optimal

bounds

for

sorting

realized

by

standard

two-way

merge

sort

for

the

special

case

P

and

M

pN

The

model

in

-LSB-

AgV

-RSB-

is

somewhat

unrealistic

however

because

secondary

storage

is

usually

partitioned

into

separate

physical

devices

each

capable

of

transferring

only

one

block

per

I/O

We

are

interested

in

optimal

algorithms

for

realistic

two-level

storage

systems

that

allow

P

simultaneous

data

transfers

By

realistic

we

mean

that

each

block

transfer

must

be

associated

with

a

separate

secondary

storage

device

In

the

next

section

we

define

a

realistic

two-level

memory

model

with

parallel

block

transfer

which

consists

of

an

internal

memory

-LRB-

capable

of

storing

M

records

-RRB-

and

P

disks

each

disk

capable

of

simultaneously

transferring

one

block

of

B

records

as

shown

in

Figure

Another

version

of

our

model

that

turns

out

to

be

sufficient

for

our

purposes

is

to

have

the

P

disks

controlled

by

P

processors

each

with

internal

memory

capable

of

storing

M

=P

records

If

P

P

each

of

the

P

processors

can

drive

about

P

=P

disks

if

P

P

M

each

disk

is

associated

with

about

P

=P

processors

The

P

processors

are

connected

by

a

network

such

as

a

hypercube

or

cube-connected

cycles

that

allows

some

basic

operations

like

sorting

of

the

M

records

in

the

internal

memories

to

be

performed

quickly

in

parallel

in

O

-LRB-

-LRB-

M

=P

-RRB-

log

M

-RRB-

time

The

special

case

in

which

P

P

is

pictured

in

Figure

Our

main

measure

of

performance

is

the

number

of

parallel

I/Os

required

but

we

also

consider

the

amount

of

internal

computation

The

bottleneck

in

the

problems

we

consider

is

generally

the

I/O

at

least

in

the

multiprocessor

versions

of

the

two-level

memory

model

described

above

In

this

paper

we

consider

large-scale

instances

of

the

following

important

problems

which

are

defined

in

Section

sorting

permuting

matrix

transposition

FFT

permutation

networks

and

standard

matrix

multiplication

In

Section

we

state

our

main

results

namely

tight

upper

and

lower

bounds

on

the

number

of

I/Os

and

amount

of

internal

processing

THE

TWO-LEVEL

MEMORY

MODEL

needed

to

solve

these

problems

for

each

of

the

uniprocessor

and

multiprocessor

models

mentioned

above

The

standard

matrix

multiplication

algorithm

is

simultaneously

optimal

in

terms

of

internal

processing

time

Our

sorting

FFT

and

permutation

network

algorithms

are

also

simultaneously

optimal

in

terms

of

the

internal

processing

time

when

P

O

-LRB-

-RRB-

or

log

M

O

-LRB-

log

-LRB-

M

B

-RRB-

-RRB-

For

large-scale

computer

configurations

in

which

P

and

PB

are

large

our

algorithms

for

sorting

permuting

FFT

and

permutation

networks

can

be

significantly

faster

than

the

algorithms

obtained

by

applying

the

well-known

technique

of

disk

striping

to

good

single-disk

algorithms

Sections

-LCB-

are

devoted

to

the

algorithms

and

their

analyses

In

Section

we

develop

optimal

algorithms

for

matrix

transposition

FFT

and

permutation

networks

by

making

use

of

the

shuffle-merge

primitive

Even

though

these

problems

are

sorting-related

it

is

much

easier

to

develop

optimal

algorithms

for

them

than

it

is

for

sorting

since

their

I/O

schedules

are

fixed

and

nonadaptive

The

main

contribution

of

this

paper

is

the

optimal

randomized

algorithm

for

sorting

-LRB-

and

permuting

-RRB-

and

its

probabilistic

analysis

in

Section

The

probability

that

it

uses

more

than

times

the

optimal

number

of

I/Os

is

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

The

sorting

algorithm

is

a

variant

of

a

distribution

sort

a

combination

of

two

randomized

techniques

is

used

to

do

the

partitioning

so

as

to

take

full

advantage

of

parallel

block

transfer

The

algorithm

requires

that

the

disks

operate

independently

in

read

mode

but

in

each

disk

write

the

track

written

to

is

the

same

among

all

the

disks

which

facilitates

writing

error

correction

information

in

fact

some

of

the

initial

disk

arrays

being

developed

require

that

parallel

writes

have

this

same-track

property

In

Section

we

cover

standard

matrix

multiplication

Conclusions

and

open

problems

are

discussed

in

Section

The

Two-Level

Memory

Model

First

we

define

the

parameters

for

our

two-level

memory

model

with

parallel

block

transfer

as

pictured

in

Figures

and

Definition

The

parameters

are

defined

by

N

records

in

the

file

M

records

that

can

fit

in

internal

memory

B

records

per

block

P

disk

drives

P

internal

processors

where

B

M

M

N

P

bM

Bc

and

P

M

The

parameters

N

M

B

P

and

P

are

referred

to

as

the

file

size

memory

size

block

size

number

of

disks

and

number

of

processors

respectively

We

denote

the

P

disks

by

DP

Each

disk

is

partitioned

into

consecutive

tracks

each

capable

of

storing

one

block

of

B

records

as

pictured

in

Figure

-LRB-

For

simplicity

in

our

model

we

assume

that

a

block

our

unit

of

transfer

is

the

same

size

as

a

track

-RRB-

If

no

disk

is

specified

when

we

refer

to

the

kth

track

we

mean

the

kth

track

of

all

P

disks

collectively

simplicity

of

notation

we

use

log

x

where

x

to

denote

the

quantity

xg

CPU

Internal

memory

P

Figure

The

uniprocessor

two-level

storage

model

with

parallel

block

transfer

The

measure

of

performance

is

the

number

of

I/Os

During

an

I/O

each

of

the

P

disks

can

simultaneously

transfer

a

block

of

B

contiguous

records

to

or

from

the

internal

memory

of

size

M

For

purposes

of

making

the

problem

definitions

we

give

in

the

next

section

more

concrete

the

locations

on

disk

are

numbered

track-by-track

in

the

following

cyclical

fashion

Track

contains

the

first

PB

memory

locations

track

of

contains

memory

locations

B

track

of

contains

memory

locations

B

B

track

of

DP

contains

memory

locations

-LRB-

P

-RRB-

B

-LRB-

P

-RRB-

B

PB

Track

contains

the

next

PB

memory

locations

track

of

contains

memory

locations

PB

PB

-LRB-

P

-RRB-

B

track

of

contains

memory

locations

-LRB-

P

-RRB-

B

-LRB-

P

-RRB-

B

-LRB-

P

-RRB-

B

track

of

DP

contains

memory

locations

-LRB-

-RRB-

B

-LRB-

-RRB-

B

The

numbering

continues

in

this

fashion

for

tracks

Parallelism

appears

in

our

model

in

two

basic

ways

First

records

are

transferred

concurrently

in

blocks

of

B

contiguous

records

It

takes

roughly

the

same

amount

of

time

to

access

and

transfer

one

block

as

it

does

one

record

This

reflects

the

fact

that

the

seek

time

THE

TWO-LEVEL

MEMORY

MODEL

The

Net

CPU

Internal

memory

CPU

Internal

memory

CPU

Internal

memory

P

Figure

An

alternate

multiprocessor

version

of

the

model

for

P

P

Each

of

the

P

disks

is

controlled

by

a

separate

processor

with

its

own

internal

memory

of

size

M

=P

The

interprocessor

communication

is

assumed

to

be

sufficiently

fast

so

that

internal

sorting

can

be

done

rapidly

in

parallel

for

a

record

greatly

dominates

the

time

to

transmit

a

record

The

second

type

of

parallelism

arises

because

P

blocks

can

be

transferred

in

a

single

I/O

We

make

the

realistic

restriction

that

the

P

blocks

must

be

associated

with

tracks

from

P

different

disks

That

is

only

one

track

per

disk

can

be

accessed

but

there

is

no

constraint

on

which

track

is

accessed

on

each

disk

The

restriction

that

only

one

block

can

be

accessed

per

disk

during

an

I/O

is

what

distinguishes

our

model

from

the

less

realistic

model

of

-LSB-

AgV

-RSB-

This

distinction

is

akin

to

the

difference

in

parallel

computation

between

the

MPC

-LRB-

module

parallel

computer

-RRB-

model

and

the

less

realistic

PRAM

model

However

general

PRAM

simulation

techniques

use

logarithmic

time

per

step

if

they

were

applied

to

the

algorithms

in

-LSB-

AgV

-RSB-

the

resulting

DP

B

records

Disks

Tracks

Figure

The

disks

are

represented

by

horizontal

lines

and

the

tracks

by

vertical

lines

algorithms

would

not

be

optimal

in

terms

of

I/O

The

algorithms

we

develop

on

our

model

use

the

same

number

of

I/Os

as

those

in

-LSB-

AgV

-RSB-

for

the

less

realistic

model

Problem

Definitions

The

problems

we

consider

in

this

paper

have

been

well

described

in

the

literature

Most

of

the

following

definitions

are

those

from

-LSB-

AgV

-RSB-

with

suitable

modifications

Sorting

Problem

Instance

The

internal

memory

is

empty

and

the

N

records

are

stored

in

the

first

N

locations

of

secondary

storage

Goal

The

internal

memory

is

empty

and

the

N

records

are

stored

in

sorted

non-decreasing

order

in

the

first

N

locations

of

secondary

storage

Permuting

The

Problem

Instance

and

Goal

are

the

same

as

for

the

Sorting

problem

except

that

the

key

values

of

the

N

records

are

required

to

form

a

permutation

of

Ng

Matrix

Transposition

Problem

Instance

The

internal

memory

is

empty

and

a

p

q

matrix

A

-LRB-

Ai

j

-RRB-

of

N

pq

records

is

stored

row

by

row

in

the

first

N

locations

of

secondary

storage

Goal

The

internal

memory

is

empty

and

the

transposed

matrix

AT

is

stored

row

by

row

in

the

first

N

locations

of

secondary

storage

-LRB-

The

q

p

matrix

AT

is

called

the

transpose

of

A

if

ATi

j

Aj

i

for

all

i

q

and

j

p.

-RRB-

The

matrix

transposition

problem

above

is

a

special

case

of

permuting

in

which

the

permutation

to

be

realized

corresponds

to

converting

a

matrix

from

row-major

order

to

column-major

order

Fast

Fourier

Transform

-LRB-

FFT

-RRB-

Problem

Instance

Let

N

be

a

power

of

The

internal

memory

is

empty

and

the

N

records

are

stored

in

the

first

N

locations

of

secondary

storage

PROBLEM

DEFINITIONS

Goal

The

internal

memory

is

empty

the

N

output

nodes

of

the

FFT

digraph

are

pebbled

-LRB-

as

explained

below

-RRB-

and

the

N

records

are

stored

in

the

first

N

locations

of

secondary

storage

The

FFT

digraph

consists

of

log

N

levels

each

containing

N

nodes

level

contains

the

N

input

nodes

and

level

log

N

contains

the

N

output

nodes

Each

non-input

node

has

indegree

and

each

non-output

node

has

outdegree

We

shall

denote

the

ith

node

-LRB-

i

N

-RRB-

in

level

j

-LRB-

j

log

N

-RRB-

in

the

FFT

digraph

by

ni

j

For

j

the

two

predecessors

to

node

ni

j

are

nodes

ni

j

and

ni

j

where

denotes

the

exclusive-or

operation

on

the

binary

representations

-LRB-

Note

that

nodes

ni

j

and

ni

j

each

have

the

same

two

predecessors

-RRB-

The

ith

node

in

each

level

corresponds

to

record

Ri

We

are

allowed

to

pebble

node

ni

j

if

its

two

predecessors

ni

j

and

ni

j

have

already

been

pebbled

and

if

the

records

Ri

and

Ri

corresponding

to

the

two

predecessors

both

reside

in

internal

memory

Intuitively

the

FFT

problem

can

be

phrased

as

the

problem

of

pumping

the

records

into

and

out

of

internal

memory

in

a

way

that

permits

the

computation

implied

by

the

FFT

digraph

Permutation

Networks

The

problem

instance

and

goal

are

the

same

as

for

the

FFT

problem

except

that

the

permutation

network

digraph

-LRB-

see

below

-RRB-

is

pebbled

instead

of

the

FFT

digraph

A

permutation

network

is

a

sorting

network

-LSB-

Knu

-RSB-

consisting

of

comparator

modules

or

switches

that

can

be

set

by

external

controls

so

that

any

desired

permutation

of

the

inputs

can

be

realized

at

the

output

level

of

the

network

It

consists

of

J

levels

for

some

J

log

N

each

containing

N

nodes

Level

contains

the

N

input

nodes

and

level

J

contains

the

N

output

nodes

All

edges

are

directed

between

adjacent

levels

in

the

direction

of

increasing

index

For

i

N

and

j

J

we

denote

the

ith

node

in

level

j

as

ni

j.

For

each

j

there

is

an

edge

from

ni

j

to

ni

j

In

addition

ni

j

can

have

one

other

predecessor

call

it

j

but

in

that

case

there

is

also

an

edge

from

ni

j

to

j

that

is

nodes

ni

j

and

j

have

the

same

two

predecessors

We

can

think

of

there

being

a

switch

between

nodes

ni

j

and

j

that

can

be

set

either

to

allow

the

data

from

the

previous

level

to

pass

through

unaltered

-LRB-

that

is

the

data

in

node

ni

j

goes

to

ni

j

and

the

data

in

j

goes

to

j

-RRB-

or

else

to

swap

the

data

-LRB-

so

that

the

data

in

ni

j

goes

to

j

and

the

data

in

j

goes

to

ni

j

-RRB-

A

digraph

like

this

is

called

a

permutation

network

if

for

each

of

the

N

permutations

pN

i

we

can

set

the

switches

in

such

a

way

to

realize

the

permutation

that

is

data

at

each

input

node

ni

is

routed

to

output

node

npi

J

The

ith

node

in

each

level

corresponds

to

the

current

contents

of

record

Ri

and

we

can

pebble

node

ni

j

if

its

predecessors

have

already

been

pebbled

and

if

the

records

corresponding

to

those

predecessors

reside

in

internal

memory

There

is

an

important

difference

between

permutation

networks

and

general

permuting

In

the

latter

case

the

I/Os

may

depend

upon

the

desired

permutation

whereas

with

permutation

networks

all

N

permutations

can

be

generated

by

the

same

sequence

of

I/Os

or

memory

accesses

Standard

Matrix

Multiplication

Problem

Instance

The

internal

memory

is

empty

The

elements

of

two

k

k

matrices

A

and

B

where

N

are

each

stored

in

the

first

N

locations

of

secondary

storage

Goal

The

internal

memory

is

empty

and

the

product

C

A?B

formed

by

the

standard

matrix

multiplication

algorithm

that

uses

O

-LRB-

-RRB-

arithmetic

operations

is

stored

in

the

first

N

locations

of

secondary

storage

Main

Results

In

this

section

we

state

our

main

results

Theorems

-LCB-

which

report

optimal

algorithms

in

terms

of

the

number

of

I/Os

and

internal

processing

time

in

our

two-level

model

with

parallel

block

transfer

to

solve

the

problems

defined

in

the

previous

section

Theorem

The

number

of

I/Os

required

for

sorting

N

records

is

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

the

internal

processing

time

is

O

-LRB-

N

log

-LRB-

N

B

-RRB-

-LRB-

log

M

-RRB-

=P

log

-LRB-

M

B

-RRB-

-RRB-

which

is

the

optimal

time

O

-LRB-

-LRB-

N

log

N

-RRB-

=P

-RRB-

for

example

when

log

M

O

-LRB-

log

-LRB-

M

B

-RRB-

-RRB-

The

upper

bounds

are

given

by

a

randomized

algorithm

the

probability

of

using

more

than

the

average

number

of

I/Os

or

internal

processing

time

falls

off

exponentially

The

lower

bounds

apply

to

both

the

average

case

and

the

worst

case

The

lower

bound

on

internal

processing

time

is

the

well-known

O

-LRB-

-LRB-

N

log

N

-RRB-

=P

-RRB-

bound

from

the

comparison

model

of

computation

The

I/O

lower

bound

does

not

require

the

use

of

the

comparison

model

of

computation

except

for

the

case

when

M

and

B

are

extremely

small

with

respect

to

N

namely

when

B

log

-LRB-

M

B

-RRB-

o

-LRB-

log

-LRB-

N

B

-RRB-

-RRB-

Theorem

The

number

of

I/Os

required

for

computing

the

N-input

FFT

digraph

is

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

the

internal

processing

time

is

O

-LRB-

N

log

-LRB-

N

B

-RRB-

-LRB-

log

M

-RRB-

=P

log

-LRB-

M

B

-RRB-

-RRB-

which

is

the

optimal

time

O

-LRB-

-LRB-

N

log

N

-RRB-

=P

-RRB-

for

example

when

log

M

O

-LRB-

log

-LRB-

M

B

-RRB-

-RRB-

The

lower

bound

on

internal

processing

time

is

O

-LRB-

-LRB-

N

log

N

-RRB-

=P

-RRB-

The

average-case

and

worst-case

number

of

I/Os

required

for

computing

any

N-input

permutation

network

is

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

furthermore

there

are

permutation

networks

such

that

the

number

of

I/Os

needed

to

compute

them

is

O

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

and

the

internal

processing

time

is

O

-LRB-

N

log

-LRB-

N

B

-RRB-

-LRB-

log

M

-RRB-

=P

log

-LRB-

M

B

-RRB-

-RRB-

which

is

optimal

when

log

M

O

-LRB-

log

-LRB-

M

B

-RRB-

-RRB-

All

lower

bounds

apply

to

both

the

average

case

and

the

worst

case

MAIN

RESULTS

Theorem

The

number

of

I/Os

required

to

permute

N

records

is

min

-LRB-

N

P

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

-RRB-

and

the

internal

processing

time

is

O

-LRB-

minfN

-LRB-

log

P

-RRB-

=P

N

log

-LRB-

N

B

-RRB-

-LRB-

log

M

-RRB-

=P

log

-LRB-

M

B

-RRB-

g

-RRB-

The

I/O

lower

bound

applies

to

both

the

average

case

and

the

worst

case

The

second

term

in

the

I/O

upper

bound

corresponds

to

the

randomized

algorithm

of

Theorem

and

the

first

term

in

the

I/O

upper

bound

makes

use

of

a

modification

of

Phase

of

the

randomized

algorithm

Theorem

The

number

of

I/Os

required

to

transpose

a

p

q

matrix

of

N

pq

elements

is

N

PB

log

minfM

p

q

N

Bg

log

-LRB-

M

B

-RRB-

and

the

internal

processing

time

is

greater

by

a

multiplicative

factor

of

O

-LRB-

-LRB-

PB

log

P

-RRB-

=P

-RRB-

Theorem

The

number

of

I/Os

required

to

multiply

two

k

k

matrices

using

the

standard

matrix

multiplication

algorithm

is

minfk

pM

gPB

and

the

internal

processing

time

required

is

-LRB-

=P

-RRB-

The

I/O

bounds

in

the

theorems

can

be

regarded

is

in

terms

of

the

number

of

passes

through

the

file

One

pass

corresponds

to

the

number

of

I/Os

needed

to

read

and

write

the

file

once

which

is

PB

A

linear-time

algorithm

-LRB-

defined

to

be

one

that

requires

a

constant

number

of

passes

through

the

file

-RRB-

would

use

O

-LRB-

N

PB

-RRB-

I/Os

The

logarithmic

factors

that

multiply

the

N

PB

term

in

the

above

expressions

indicate

the

degree

of

nonlinearity

The

I/O

lower

bounds

for

Theorems

-LCB-

follow

from

the

lower

bounds

proved

in

-LSB-

AgV

-RSB-

for

the

less

realistic

model

in

which

P

tracks

can

be

accessed

on

the

same

disk

in

a

single

I/O

Since

any

algorithm

in

our

model

automatically

applies

to

the

model

in

-LSB-

AgV

-RSB-

the

same

lower

bounds

apply

The

I/O

lower

bounds

proved

in

-LSB-

AgV

-RSB-

are

based

only

on

routing

concerns

and

thus

hold

for

an

arbitrarily

powerful

adversary

except

in

the

case

of

sorting

for

the

extreme

case

mentioned

in

Theorem

when

M

and

B

are

extremely

small

in

which

case

the

comparison

model

is

used

Thus

the

hard

part

of

sorting

in

the

non-extreme

case

is

the

routing

of

the

records

not

the

determination

of

the

records

order

The

well-known

technique

of

key

sorting

-LSB-

Knu

-RSB-

which

attempts

to

reduce

sorting

to

permutation

routing

by

using

a

special-purpose

method

of

determining

the

order

of

the

records

is

therefore

not

going

to

use

significantly

fewer

I/Os

than

will

general

sorting

algorithms

The

lower

bound

in

Theorem

for

standard

matrix

multiplication

follows

by

taking

the

bound

for

the

case

P

in

-LSB-

SaV

-RSB-

and

dividing

by

P

The

algorithms

that

meet

the

lower

bounds

of

Theorems

-LCB-

will

be

described

and

analyzed

in

the

following

sections

The

previously

best

way

to

sort

with

multiple

disks

when

P

was

the

following

combination

of

the

well-known

techniques

of

disk

striping

and

two-way

merge

sort

The

read/write

heads

of

the

P

disks

are

synchronized

so

that

during

each

I/O

all

the

disk

drives

access

the

same

track

number

on

their

respective

disks

This

striping

of

the

data

on

the

disks

effectively

reduces

the

multiple

disks

to

only

one

logical

disk

but

with

a

larger

block

size

PB

The

number

of

I/Os

needed

for

two-way

merge

sort

with

the

one

logical

disk

is

-LRB-

-LRB-

N

-RRB-

log

-LRB-

N

-RRB-

log

-LRB-

M

-RRB-

-RRB-

-LRB-

-LRB-

N

PB

-RRB-

log

-LRB-

N

PB

-RRB-

log

-LRB-

M

PB

-RRB-

-RRB-

This

bound

can

be

larger

than

the

one

in

Theorem

by

a

multiplicative

log

-LRB-

M

B

-RRB-

factor

which

can

be

significant

when

P

and

PB

are

relatively

large

For

small

values

of

P

striping

is

efficient

in

terms

of

I/O

within

a

constant

factor

of

optimal

Shuffle-Merge

and

its

Applications

In

this

section

we

exploit

a

simple

but

useful

merging

operation

called

shuffle-merge

that

can

be

used

to

achieve

the

optimal

I/O

bounds

mentioned

in

Theorems

and

for

the

problems

of

FFT

permutation

networks

and

matrix

transposition

The

algorithms

which

consist

of

a

series

of

shuffle-merges

are

the

ones

described

in

-LSB-

AgV

-RSB-

except

that

the

disk

placement

of

the

blocks

of

the

merged

runs

must

be

done

in

a

staggered

way

so

that

the

merging

in

the

next

pass

can

be

done

using

full

parallelism

Without

loss

of

generality

we

assume

for

simplicity

of

exposition

that

N

M

P

and

B

are

powers

of

The

operation

of

shuffle-merge

consists

of

performing

a

perfect

shuffle

-LSB-

Sto

-RSB-

on

the

elements

of

M

B

runs

of

r

records

each

and

the

result

is

a

single

shuffled

run

of

rM

B

elements

Pictorially

suppose

the

sorted

runs

initially

look

like

this

Run

Run

Run

MB

aM

B

aM

B

aM

B

r

After

the

perfect

shuffle

a

single

shuffled

run

remains

aM

B

aM

B

aM

B

r

It

is

easy

to

do

shuffle-merges

and

take

full

advantage

of

parallel

block

transfer

if

the

input

runs

are

blocked

and

the

blocks

are

staggered

with

respect

to

one

another

on

the

disk

so

that

in

a

single

I/O

we

can

read

the

next

track

from

each

of

the

next

P

runs

For

example

it

suffices

if

the

kth

block

of

records

from

the

ith

run

-LRB-

consisting

of

records

ai

-LRB-

k

-RRB-

B

aikB

-RRB-

is

stored

on

track

-LRB-

i

-RRB-

dr

=P

Be

dk

=P

e

of

disk

-LRB-

-LRB-

k

i

-RRB-

modP

-RRB-

-LRB-

If

r

PB

then

this

placement

can

be

modified

so

that

more

than

one

run

is

packed

per

track

-RRB-

The

algorithm

consists

of

a

series

of

parallel

block

transfers

On

reads

-LRB-

k

-RRB-

M

PB

kM

PB

we

bring

into

internal

memory

the

kth

block

from

each

of

the

M

B

runs

The

records

are

shuffled

appropriately

in

internal

memory

and

then

written

to

the

disks

The

total

number

of

I/Os

for

the

entire

shuffle-merge

is

O

-LRB-

rM

PB

-RRB-

which

is

best

possible

since

each

record

is

read

once

from

disk

and

written

once

to

disk

making

full

use

of

parallelism

and

blocking

Permutation

Networks

and

FFT

Every

permutation

of

N

elements

can

be

realized

by

three

passes

through

an

FFT

network

by

an

appropriate

setting

of

the

switches

in

the

FFT

network

that

depends

on

the

permu

SHUFFLE-MERGE

AND

ITS

APPLICATIONS

Figure

Decomposition

used

for

optimal

pebbling

of

the

FFT

digraph

for

N

M

The

M

pebbles

can

be

slid

forward

log

M

levels

before

the

pebbles

have

to

be

regrouped

tation

-LSB-

WuF

-RSB-

So

we

can

get

optimal

I/O

strategies

for

an

FFT-based

permutation

network

by

getting

optimal

I/O

strategies

for

FFT

digraphs

The

FFT

digraph

is

defined

in

Section

For

simplicity

we

assume

that

log

M

divides

log

N

evenly

We

divide

the

N

records

into

N

M

groups

of

M

contiguous

records

Each

group

corresponds

to

a

set

of

rows

of

the

FFT

digraph

whose

nodes

have

links

to

only

each

other

in

the

next

log

M

levels

of

the

FFT

digraph

For

each

of

N

M

I/Os

we

input

the

M

records

in

a

group

pebble

forward

in

the

FFT

digraph

log

M

levels

and

then

write

the

group

back

to

the

disks

in

a

staggered

way

Afterwards

a

series

of

shuffle-merges

are

done

to

realign

the

records

into

new

groups

of

size

M

so

that

pebbling

of

each

group

can

proceed

for

the

next

log

M

levels

as

shown

in

Figure

This

continues

until

the

entire

FFT

digraph

is

pebbled

The

above

algorithm

stops

and

performs

a

series

of

shuffle-merges

log

N

log

M

times

Each

series

consists

of

logM

B

minfM

N

Mgg

shuffle-merges

each

requiring

O

-LRB-

N

PB

-RRB-

I/Os

Thus

the

total

number

of

I/Os

used

is

N

PB

log

N

log

M

logM

B

min

ae

M

N

M

oe

I/Os

which

can

be

shown

by

some

algebraic

manipulation

to

equal

the

bound

given

in

Theorem

The

bound

on

internal

processing

time

in

Theorem

follows

from

the

fact

that

the

processing

is

done

one

memoryload

at

a

time

Each

memoryload

accounts

for

-LRB-

M

PB

-RRB-

I/Os

and

O

-LRB-

-LRB-

M

log

M

-RRB-

=P

-RRB-

time

is

used

to

do

the

internal

processing

of

each

memoryload

Matrix

Transposition

Let

us

denote

the

B

records

that

end

up

in

the

same

block

in

the

transposed

matrix

as

a

target

group

Initially

in

the

original

untransposed

matrix

several

members

of

a

given

target

group

may

be

in

the

same

block

We

call

these

members

a

target

subgroup

Each

target

subgroup

initially

has

size

x

if

B

minfp

qg

B

minfp

qg

if

minfp

qg

B

maxfp

qg

N

if

maxfp

qg

B.

-LRB-

-RRB-

The

transposition

algorithm

consists

of

a

series

of

shuffle-merges

Records

in

the

same

target

subgroup

remain

together

throughout

the

course

of

the

algorithm

In

each

pass

we

merge

together

sets

of

M

B

target

subgroups

thus

increasing

the

size

of

the

resulting

target

subgroups

by

a

factor

of

M

B

The

number

of

passes

is

logM

B

B

x

ss

-LRB-

-RRB-

each

requiring

O

-LRB-

N

PB

-RRB-

I/Os

The

upper

bound

for

I/O

in

Theorem

follows

by

substituting

the

three

cases

of

-LRB-

-RRB-

into

-LRB-

-RRB-

Each

memoryload

accounts

for

-LRB-

M

PB

-RRB-

I/Os

and

O

-LRB-

-LRB-

M

log

P

-RRB-

=P

-RRB-

time

is

used

to

do

the

internal

processing

of

each

memoryload

thus

yielding

the

bound

on

internal

processing

time

in

Theorem

External

Sorting

and

Permuting

In

this

section

we

present

and

analyze

the

optimal

algorithms

for

sorting

and

permuting

on

the

two-level

memory

model

which

achieve

the

bound

listed

in

Theorems

and

We

assume

for

simplicity

of

exposition

that

N

M

P

and

B

are

powers

of

we

show

below

that

this

assumption

does

not

result

in

any

loss

of

generality

For

sorting

we

also

make

the

simplifying

assumption

that

all

key

values

are

distinct

This

assumption

is

satisfied

for

example

if

we

append

to

the

key

field

of

each

record

the

original

memory

location

of

the

record

Permuting

records

is

a

special

case

of

sorting

The

bounds

for

sorting

and

permuting

given

in

Theorems

and

are

the

same

except

when

the

internal

memory

size

M

and

block

size

B

are

extremely

small

with

respect

to

the

file

size

N

In

the

latter

case

permuting

can

be

done

using

O

-LRB-

N

=P

-RRB-

I/Os

by

a

modification

of

the

sorting

technique

we

discuss

below

so

we

shall

restrict

our

attention

to

the

sorting

problem

The

application

of

the

sorting

method

to

the

special

case

of

permuting

will

be

discussed

in

Section

The

FFT

and

matrix

transposition

algorithms

described

in

the

previous

section

were

easy

to

implement

using

an

optimal

number

of

I/Os

because

the

merging

pattern

in

each

pass

was

predetermined

it

consisted

of

a

series

of

shuffle-merges

This

made

it

easy

to

distribute

the

records

onto

the

disks

so

that

the

merges

in

the

next

pass

accessed

the

records

in

a

balanced

fashion

among

all

the

disks

However

this

does

not

seem

applicable

to

sorting

When

merge

sort

is

used

for

external

sorting

the

merges

in

each

pass

are

not

in

general

perfect

shuffles

And

when

the

merges

are

not

perfect

shuffles

it

is

difficult

to

know

how

to

distribute

the

records

onto

the

disks

so

as

to

guarantee

balanced

access

to

the

disks

in

the

EXTERNAL

SORTING

AND

PERMUTING

next

merge

pass

An

interesting

question

of

whether

merge

sort

can

be

implemented

using

the

optimal

number

of

I/Os

is

discussed

in

Section

In

order

to

get

an

optimal

sorting

algorithm

we

use

the

following

practical

randomized

approach

which

is

a

recursive

distribution

sort

If

N

M

we

sort

the

file

internally

Otherwise

we

do

the

following

steps

-LSB-

Find

partitioning

elements

-RSB-

We

deterministically

find

S

partitioning

elements

bS

that

break

the

file

into

S

roughly

equal-sized

buckets

The

parameter

S

will

be

defined

shortly

it

is

always

small

enough

so

that

the

partitioning

elements

can

be

stored

easily

in

internal

memory

For

convenience

we

define

the

dummy

partitioning

elements

and

bS

The

jth

bucket

consists

of

all

the

records

R

in

the

file

whose

key

values

are

in

the

range

bj

key

-LRB-

R

-RRB-

bj

-LSB-

Partition

into

buckets

-RSB-

We

partition

the

file

into

buckets

based

upon

the

partitioning

elements

and

distribute

the

records

in

each

bucket

evenly

among

the

P

disks

-LSB-

Recurse

-RSB-

We

sort

each

bucket

by

applying

the

sorting

algorithm

recursively

-LRB-

With

high

probability

the

records

in

each

bucket

are

distributed

evenly

among

the

P

disks

and

thus

they

can

be

read

into

internal

memory

with

O

-LRB-

N

SPB

-RRB-

I/Os

-RRB-

The

output

of

the

sorting

algorithm

is

the

concatenation

of

the

sorted

buckets

The

partitioning

in

Step

is

done

in

one

of

two

ways

which

we

call

Phase

and

Phase

Phase

which

is

described

in

detail

in

Section

is

used

for

the

partitioning

when

N

pMBP

ln

-LRB-

M

B

-RRB-

It

can

be

thought

of

intuitively

as

a

hashing

approach

to

distribute

the

blocks

of

each

bucket

among

the

disks

It

works

effectively

when

the

hash

function

distributes

the

records

evenly

and

by

analogy

to

the

maximum

bucket

occupancy

problem

in

hashing

-LSB-

ViF

-RSB-

the

distribution

is

even

when

the

expected

number

of

blocks

per

disk

for

each

bucket

is

at

least

a

logarithmic

amount

However

if

N

is

not

much

larger

than

M

the

distribution

using

the

hashing

approach

can

be

quite

uneven

resulting

in

nonoptimal

performance

In

the

latter

case

when

M

N

pMBP

ln

-LRB-

M

B

-RRB-

the

partitioning

is

done

by

Phase

which

is

described

in

Section

Phase

uses

a

partitioning

technique

motivated

by

a

different

instance

of

the

hashing

problem

and

works

with

overwhelming

probability

After

the

Phase

partitioning

each

bucket

will

have

at

most

M

records

and

can

be

sorted

internally

as

described

in

Section

An

alternative

to

the

Phase

technique

for

small

N

is

the

deterministic

approach

based

on

Leighton

's

Columnsort

algorithm

-LSB-

Lei

-RSB-

as

mentioned

in

-LSB-

AgV

-RSB-

and

described

in

detail

in

-LSB-

NoVc

Theorem

-RSB-

However

we

use

Phase

here

because

it

complements

Phase

nicely

in

approach

and

has

fairly

small

constant

factors

As

noted

above

we

assume

for

simplicity

that

N

M

P

and

B

are

powers

of

We

also

choose

S

to

be

a

power

of

The

assumption

on

M

P

and

B

whose

values

do

not

change

during

the

course

of

the

algorithm

clearly

does

not

affect

the

generality

of

the

algorithm

We

can

make

N

a

power

of

at

each

level

of

recursion

by

appending

to

the

file

hypothetical

dummy

records

with

key

value

These

dummy

records

do

not

need

to

be

written

to

the

buckets

so

the

cumulative

sizes

of

the

buckets

is

not

affected

The

running

time

increases

as

a

result

by

at

most

a

small

constant

factor

Definition

We

denote

the

S

buckets

by

SS

The

number

of

records

in

the

file

that

belong

to

bucket

Sj

is

denoted

Nj

In

our

two-level

model

with

parallel

block

transfer

we

can

look

at

only

M

records

at

a

time

so

partitioning

will

be

done

one

memoryload

at

a

time

We

denote

the

ith

memoryload

by

Mi

A

write

cycle

is

defined

as

the

collection

of

P

blocks

that

we

write

to

the

disks

concurrently

in

a

single

I/O

We

denote

write

cycles

by

Read

cycles

are

defined

analogously

For

the

time

being

let

us

assume

that

we

can

deterministically

compute

in

Step

the

approximate

partitioning

elements

bS

using

O

-LRB-

N

PB

-RRB-

I/Os

-LRB-

The

algorithm

and

the

analysis

for

computing

the

partitioning

elements

are

given

in

Section

-RRB-

For

Phase

we

set

S

ss

pM

B

-LRB-

M

B

-RRB-

in

particular

we

set

S

to

be

maximum

of

and

the

largest

power

of

that

is

pM

B

-LRB-

M

B

-RRB-

We

show

later

in

Lemma

that

N

-LRB-

S

-RRB-

Nj

-LRB-

S

-RRB-

-LRB-

-RRB-

For

Phase

we

set

S

M

we

show

later

in

Lemma

that

Nj

-LRB-

-RRB-

The

upper

bound

for

sorting

in

Theorem

follows

from

the

following

bound

which

is

the

main

result

of

this

paper

Theorem

The

number

of

I/Os

used

by

the

above

distribution

sort

algorithm

to

sort

N

records

is

O

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

and

the

internal

processing

time

is

O

-LRB-

N

log

-LRB-

N

B

-RRB-

-LRB-

log

M

-RRB-

=P

log

-LRB-

M

B

-RRB-

-RRB-

with

overwhelming

probability

In

particular

the

probability

that

the

number

of

I/Os

or

the

internal

processing

time

is

more

than

times

the

average

is

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

Proof

We

define

T

-LRB-

N

-RRB-

to

be

the

number

of

I/Os

used

to

sort

N

records

and

-LRB-

N

-RRB-

to

be

the

number

of

I/Os

used

for

all

the

calls

to

Phase

We

shall

see

from

Theorem

that

with

high

probability

Phase

uses

O

-LRB-

N

PB

-RRB-

I/Os

to

partition

N

records

and

to

store

each

bucket

evenly

across

the

disks

so

that

the

buckets

can

be

retrieved

one-by-one

in

the

next

level

of

recursion

with

a

total

of

O

-LRB-

N

PB

-RRB-

I/Os

The

above

construction

gives

us

-LRB-

N

-RRB-

X

j

S

-LRB-

Nj

-RRB-

O

N

PB

-LRB-

-RRB-

with

high

probability

In

particular

from

Theorem

the

probability

that

the

quantity

represented

by

the

big-oh

term

in

-LRB-

-RRB-

is

more

than

N

PB

is

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

In

Phase

we

set

S

ss

pM

B

-LRB-

M

B

-RRB-

by

-LRB-

-RRB-

we

have

Nj

-LRB-

S

-RRB-

Substituting

this

bound

into

-LRB-

-RRB-

and

iterating

the

recurrence

until

N

M

by

which

time

Phase

certainly

ends

we

get

-LRB-

N

-RRB-

O

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

-LRB-

-RRB-

EXTERNAL

SORTING

AND

PERMUTING

with

high

probability

In

all

the

instances

of

partitioning

during

the

recursive

levels

of

Phase

imbalance

by

more

than

a

factor

of

occurs

independently

with

exponentially

small

probability

By

convexity

arguments

we

can

combine

these

probability

bounds

and

bound

the

probability

that

-LRB-

N

-RRB-

is

more

than

times

the

expression

in

the

big-oh

term

in

-LRB-

-RRB-

by

a

quantity

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

In

Theorem

we

show

that

with

high

probability

Phase

uses

O

-LRB-

N

PB

-RRB-

I/Os

in

order

to

perform

the

last

level

of

partitioning

The

remaining

buckets

each

contain

at

most

M

records

and

can

be

sorted

internally

This

gives

us

T

-LRB-

N

-RRB-

O

N

PB

log

-LRB-

N

B

-RRB-

log

-LRB-

M

B

-RRB-

-LRB-

-RRB-

with

high

probability

Since

the

partitioning

in

Phase

is

independent

of

those

in

Phase

we

can

bound

the

probability

that

T

-LRB-

N

-RRB-

is

more

than

times

the

expression

in

the

big-oh

term

in

-LRB-

-RRB-

by

a

quantity

exponentially

small

in

-LRB-

log

-RRB-

log

-LRB-

M

B

-RRB-

The

bound

on

internal

processing

time

follows

because

the

internal

processing

is

done

one

memoryload

at

a

time

Each

memoryload

accounts

for

-LRB-

M

PB

-RRB-

I/Os

and

O

-LRB-

-LRB-

M

log

M

-RRB-

=P

-RRB-

time

is

used

for

the

internal

sorting

partitioning

and

overhead

of

each

memoryload

Phase

We

use

Phase

to

partition

the

records

of

a

file

of

N

records

when

N

pMBP

ln

-LRB-

M

B

-RRB-

The

number

S

of

partitions

is

approximately

pM

B

-LRB-

M

B

-RRB-

in

particular

we

set

S

to

be

maximum

of

and

the

largest

power

of

that

is

pM

B

-LRB-

M

B

-RRB-

In

order

to

read

a

file

into

internal

memory

using

full

parallelism

the

records

of

the

file

must

be

evenly

distributed

over

the

disks

as

a

result

of

the

previous

pass

of

Phase

This

is

crux

of

the

problem

We

shall

show

in

Theorem

that

Phase

does

the

partitioning

using

O

-LRB-

N

PB

-RRB-

I/Os

We

read

the

records

of

the

file

into

internal

memory

one

memoryload

at

a

time

We

assign

the

records

to

buckets

based

upon

the

partitioning

elements

and

organize

the

records

so

that

records

in

the

each

bucket

are

contiguous

in

internal

memory

We

then

write

the

records

in

each

bucket

of

the

memoryload

to

the

disks

using

full

parallelism

We

use

a

randomized

approach

to

distribute

the

records

The

main

result

of

this

subsection

is

showing

that

the

requirement

N

pMBP

ln

-LRB-

M

B

-RRB-

assures

with

high

probability

that

the

records

of

a

bucket

-LRB-

among

all

the

memoryloads

-RRB-

will

be

spread

out

evenly

among

the

disks

In

order

to

link

together

the

records

in

each

bucket

-LRB-

to

allow

fast

retrieval

of

the

bucket

in

the

next

level

of

recursion

-RRB-

we

need

to

remember

the

last

track

on

each

disk

where

a

block

belonging

to

that

bucket

was

written

we

store

these

pointers

in

internal

memory

In

order

to

reduce

the

number

of

pointers

so

that

they

can

be

kept

in

internal

memory

we

cluster

the

disks

into

C

logical

clusters

as

pictured

in

Figure

We

set

C

to

be

minfP

Sg

Definition

A

cluster

is

a

logical

grouping

of

consecutive

disks

The

C

clusters

are

denoted

CC

The

P

C

disks

in

the

kth

cluster

Ck

are

denoted

Dk

Dk

Dk

;P

C

The

ith

track

of

a

cluster

refers

collectively

to

the

ith

tracks

of

all

the

disks

that

comprise

the

cluster

Records

are

written

to

the

disks

in

cluster-size

units

of

P

C

blocks

which

we

call

a

group

Phase

;P

C

;P

C

Figure

Decomposition

of

the

P

disks

into

C

clusters

Algorithm

Phase

Let

last

disk

j

k

and

last

track

j

k

represent

the

last

disk

and

the

last

track

respectively

written

to

in

cluster

Ck

by

bucket

Sj

Let

next

trackk

represent

the

first

track

on

Ck

that

has

not

been

assigned

to

a

bucket

We

initialize

last

disk

j

k

last

track

j

k

and

next

trackk

The

file

is

processed

memoryload

by

memoryload

For

each

i

N

M

the

ith

memoryload

is

brought

into

internal

memory

The

records

are

partitioned

into

buckets

based

upon

the

partitioning

elements

The

records

in

each

bucket

are

formed

into

blocks

and

the

blocks

within

a

bucket

are

formed

into

groups

of

size

P

C

except

possibly

the

last

group

which

might

be

only

partially

filled

We

choose

C

groups

to

be

written

during

this

write

cycle

and

we

assign

these

groups

to

clusters

by

choosing

a

random

permutation

of

Cg

This

is

repeated

C

groups

at

a

time

until

the

memoryload

is

written

-LRB-

This

is

the

only

place

where

randomness

is

used

in

Phase

-RRB-

What

remains

is

to

assign

the

blocks

in

a

group

to

the

disks

in

a

cluster

In

each

group

we

have

a

maximum

of

P

C

blocks

We

do

not

want

to

have

empty

tracks

on

the

disks

so

we

cycle

through

the

disks

in

the

cluster

Let

us

assume

that

a

group

belonging

to

bucket

Sj

is

assigned

to

cluster

Ck

We

assign

the

first

P

C

last

disk

j

k

blocks

to

disks

last

disk

j

k

P

C

on

track

last

trackk

we

assign

the

remaining

blocks

if

any

to

disks

on

track

next

trackk

We

then

update

the

value

of

last

disk

j

k

and

when

the

current

track

gets

filled

we

set

last

track

j

k

to

next

trackk

and

increment

next

track

k.

For

each

memoryload

we

retain

partially-filled

blocks

in

internal

memory

until

they

are

completely

filled

but

groups

are

written

to

the

disks

even

if

they

contain

fewer

than

P

C

blocks

Each

time

a

group

from

bucket

Sj

is

written

to

cluster

Ck

we

fill

up

the

last

track

written

on

that

cluster

for

Sj

before

we

start

a

new

track

that

is

once

a

bucket

writes

to

a

particular

track

of

a

given

cluster

it

will

not

write

to

another

track

of

that

cluster

until

the

current

track

is

completely

filled

This

has

the

effect

of

making

each

track

of

each

cluster

completely

filled

except

possibly

for

the

last

track

of

the

cluster

for

each

bucket

In

order

for

the

recursion

to

work

we

must

link

together

the

records

of

each

bucket

This

EXTERNAL

SORTING

AND

PERMUTING

will

be

done

with

pointers

being

made

part

of

the

blocks

when

they

are

written

to

disk

Since

records

from

a

bucket

are

written

as

a

group

we

only

have

to

save

pointers

for

the

groups

Also

since

an

entire

track

in

a

cluster

is

written

to

by

only

one

bucket

the

linking

can

be

done

by

pointers

in

the

block

on

the

first

disk

of

each

track

in

the

cluster

To

do

this

we

have

one

previous

group

pointer

for

each

track

of

each

cluster

which

we

call

pg

Each

pg

pointer

links

together

the

groups

of

a

given

bucket

that

are

on

a

given

cluster

in

reverse

order

If

a

block

in

a

group

from

Sj

is

written

to

the

first

disk

of

a

cluster

Ck

the

pg

pointer

of

that

block

is

set

to

last

track

j

k

Once

the

assigning

is

done

we

can

write

the

chosen

C

groups

to

their

assigned

disk

locations

When

we

finish

processing

the

file

we

save

on

the

disks

the

pointers

last

disk

j

k

and

last

track

j

k

so

that

we

can

locate

the

records

for

each

bucket

during

the

next

level

of

recursion

last

disk

j

k

for

all

j

k

next

track

k

for

all

k

last

track

j

k

for

all

j

k

for

each

memoryload

of

records

Mi

-LRB-

i

N

M

-RRB-

do

begin

read

Mi

into

internal

memory

partition

the

records

into

buckets

based

upon

the

partitioning

elements

for

each

bucket

Sj

-LRB-

j

S

-RRB-

do

begin

form

the

records

into

blocks

of

size

B

form

the

blocks

to

groups

of

blocks

of

size

P

C

end

for

each

write

cycle

Wt

do

begin

choose

C

groups

of

blocks

to

be

written

in

Wt

assign

the

groups

to

clusters

via

a

random

permutation

of

Cg

f

assign

the

blocks

in

each

group

to

the

disks

in

a

cluster

g

for

each

cluster

Ck

-LRB-

k

C

-RRB-

do

begin

let

Sj

be

the

bucket

whose

group

is

assigned

to

Ck

for

each

disk

Dk

;d

such

that

P

C

last

disk

j

k

d

P

C

do

schedule

the

next

block

to

be

assigned

to

last

track

j

k

on

Dk

;d

if

still

more

blocks

to

be

assigned

then

begin

temp

pg

last

track

j

k

last

track

j

k

next

track

k

next

track

k

next

track

k

for

each

disk

Dk

;d

such

that

d

P

C

last

disk

j

k

do

begin

schedule

the

next

block

to

be

assigned

to

last

track

j

k

on

Dk

;d

if

d

then

set

the

pg

pointer

of

block

to

temp

pg

end

end

Phase

update

last

disk

j

k

end

write

the

blocks

in

Wt

to

the

desired

disks

end

end

write

pointers

last

disk

j

k

and

last

track

j

k

for

all

j

k

Analysis

of

Phase

Theorem

With

overwhelming

probability

each

pass

of

Phase

uses

O

-LRB-

N

PB

-RRB-

I/Os

to

partition

a

file

of

N

records

In

particular

the

probability

that

the

number

of

I/Os

used

is

more

than

times

the

average

is

exponentially

small

in

-LRB-

log

-RRB-

maxflog

-LRB-

M

B

-RRB-

N

PBSg

Proof

The

file

is

read

into

internal

memory

one

memoryload

at

a

time

The

actual

number

of

records

read

in

each

time

might

be

less

than

a

memoryload

since

the

pointers

-LRB-

last

disk

last

track

and

next

track

-RRB-

and

partially-filled

blocks

are

retained

in

memory

during

the

partitioning

process

There

are

C

-LRB-

-RRB-

pointers

needed

assuming

each

pointer

does

not

exceed

a

record

the

pointers

take

up

C

-LRB-

-RRB-

records

Since

each

of

the

S

buckets

might

have

a

partially-filled

block

of

B

records

the

partially-filled

blocks

can

take

up

at

most

S

-LRB-

B

-RRB-

records

And

we

need

space

for

the

S

partitioning

elements

Therefore

at

least

M

C

-LRB-

-RRB-

SB

records

can

be

read

in

For

convenience

we

redefine

M

to

be

M

C

-LRB-

-RRB-

SB

so

that

a

full

memoryload

can

be

read

into

or

written

from

internal

memory

This

changes

the

value

of

M

by

at

most

a

small

constant

factor

Let

Z

be

the

number

of

I/Os

required

during

the

next

pass

of

Phase

or

Phase

to

read

in

all

the

subfiles

corresponding

to

the

buckets

formed

from

the

current

file

by

Phase

We

want

to

show

that

Z

O

N

PB

with

high

probability

We

do

that

by

showing

that

Pr

ae

Z

N

PB

oe

is

exponentially

small

in

-LRB-

log

-RRB-

maxflog

-LRB-

M

B

-RRB-

N

PBSg

The

number

of

inputs

needed

in

the

next

pass

of

Phase

or

Phase

in

order

to

read

into

internal

memory

the

subfile

corresponding

to

some

bucket

formed

by

the

current

pass

of

Phase

is

the

maximum

number

of

tracks

devoted

to

that

bucket

among

all

the

clusters

Let

Xj

k

represent

the

number

of

tracks

of

cluster

Ck

that

have

been

assigned

to

bucket

Sj

We

have

Z

X

j

S

max

k

C

fXj

kg

This

gives

us

Pr

ae

Z

N

PB

oe

Pr

X

j

S

max

k

C

fXj

kg

N

PB

-LRB-

-RRB-

The

max

term

in

-LRB-

-RRB-

is

the

difficult

expression

to

analyze

We

shall

use

the

fact

that

N

pMBP

ln

-LRB-

M

B

-RRB-

in

Phase

to

show

that

the

Xj

k

are

very

evenly

distributed

with

EXTERNAL

SORTING

AND

PERMUTING

respect

to

k

We

have

Pr

X

j

S

max

k

C

fXj

kg

N

PB

Pr

ae

max

k

C

fXj

kg

Nj

PB

oe

S

Pr

ae

max

k

C

kg

PB

oe

SC

Pr

ae

PB

oe

-LRB-

-RRB-

where

Nj

is

the

number

of

records

in

bucket

Sj

and

is

any

value

of

j

that

a

priori

-LRB-

before

the

partitioning

elements

are

chosen

-RRB-

maximizes

k

CfXj

kg

Nj

PBg

To

bound

-LRB-

-RRB-

we

use

Chernoff

's

bound

-LSB-

Kle

-RSB-

Lemma

If

X

is

a

nonnegative

random

variable

and

r

we

have

PrfX

ug

E

-LRB-

erX

-RRB-

eru

Before

we

apply

Chernoff

's

bound

we

must

construct

the

appropriate

scenario

We

let

gt

denote

the

number

of

clusters

written

to

from

bucket

during

write

Wt

for

t

R

where

R

is

the

total

number

of

write

cycles

used

in

Phase

We

have

X

t

R

gt

PB

C

C

PB

C

-LRB-

-RRB-

The

extra

C

term

in

-LRB-

-RRB-

appears

because

the

last

track

on

each

of

the

C

clusters

might

be

only

partially

filled

We

define

Gt

to

be

the

number

of

groups

belonging

to

bucket

in

write

Wt

that

are

assigned

to

cluster

Because

only

one

group

can

be

written

to

any

one

cluster

in

a

write

cycle

Gt

is

restricted

to

the

values

and

We

have

PrfGt

gt

C

and

PrfGt

g

gt

C.

Let

GGt

-LRB-

z

-RRB-

be

the

probability

generating

function

for

Gt

GGt

-LRB-

z

-RRB-

PrfGt

PrfGt

gt

C

gt

C

z

gt

C

-LRB-

z

-RRB-

-LRB-

-RRB-

Let

-LRB-

z

-RRB-

be

the

probability

generating

function

for

We

can

bound

by

the

sum

of

independent

random

variables

GR

For

purposes

of

bounding

-LRB-

-RRB-

let

us

consider

that

GR

Using

-LRB-

-RRB-

we

have

-LRB-

z

-RRB-

GR

-LRB-

z

-RRB-

-LRB-

z

-RRB-

-LRB-

z

-RRB-

GGR

-LRB-

z

-RRB-

Y

t

R

gt

C

-LRB-

z

-RRB-

-LRB-

-RRB-

By

Chernoff

's

bound

in

Lemma

we

have

Pr

ae

PB

oe

E

-LRB-

exp

-LRB-

-RRB-

-RRB-

exp

r

PB

-LRB-

-RRB-

Phase

for

each

r

We

can

express

the

numerator

in

-LRB-

-RRB-

using

-LRB-

-RRB-

and

the

definitions

of

expected

value

and

probability

generating

function

as

E

-LRB-

exp

-LRB-

-RRB-

-RRB-

X

t

tgert

-LRB-

er

-RRB-

Y

t

R

gt

C

-LRB-

er

-RRB-

-LRB-

-RRB-

To

bound

-LRB-

-RRB-

we

use

the

following

lemma

which

follows

easily

from

convexity

arguments

Lemma

If

i

R

ai

Q

and

ai

for

i

R

then

i

R

ai

is

maximized

when

aR

Q

R.

By

-LRB-

-RRB-

and

Lemma

we

can

maximize

-LRB-

-RRB-

by

setting

gt

RPB

CR

for

each

t.

Thus

E

-LRB-

exp

-LRB-

-RRB-

-RRB-

Y

t

R

-LRB-

PB

-RRB-

-LRB-

er

-RRB-

RPB

-LRB-

PB

-RRB-

-LRB-

er

-RRB-

RPB

R

Substituting

this

bound

into

-LRB-

-RRB-

we

get

Pr

ae

PB

oe

-LRB-

PB

-RRB-

-LRB-

er

-RRB-

RPB

R

exp

r

PB

Hence

by

-LRB-

-RRB-

and

-LRB-

-RRB-

we

have

Pr

ae

Z

N

PB

oe

SC

-LRB-

PB

-RRB-

-LRB-

er

-RRB-

RPB

R

exp

r

PB

-LRB-

-RRB-

By

Lemma

in

Section

and

the

Phase

bounds

N

pMBP

ln

-LRB-

M

B

-RRB-

and

S

pM

B

-LRB-

M

B

-RRB-

we

have

PB

PB

-LRB-

S

-RRB-

N

-LRB-

-RRB-

where

-LRB-

S

-RRB-

N

ln

-LRB-

M

B

-RRB-

Substituting

this

bound

into

-LRB-

-RRB-

we

have

Pr

ae

Z

N

PB

oe

SC

-LRB-

-RRB-

-LRB-

er

-RRB-

RPB

R

exp

r

PB

-LRB-

-RRB-

From

the

bound

-LRB-

a

-RRB-

b

eab

for

a

we

can

approximate

the

numerator

in

-LRB-

-RRB-

and

get

Pr

ae

Z

N

PB

oe

SC

exp

-LRB-

-RRB-

-LRB-

er

-RRB-

r

PB

SC

exp

-LRB-

-LRB-

-RRB-

-LRB-

er

-RRB-

r

-RRB-

PB

-LRB-

-RRB-

EXTERNAL

SORTING

AND

PERMUTING

Picking

r

ln

-LRB-

-LRB-

-RRB-

-RRB-

for

we

have

Pr

ae

Z

N

PB

oe

SC

exp

ln

PB

-LRB-

-RRB-

By

plugging

into

-LRB-

-RRB-

the

inequality

ln

-LRB-

x

-RRB-

x

for

x

we

get

Pr

ae

Z

N

PB

oe

SC

exp

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

PB

For

example

when

we

have

the

exponentially

small

bound

Pr

ae

Z

N

PB

oe

SC

exp

-LRB-

-RRB-

-LRB-

-RRB-

PB

-LRB-

-RRB-

As

gets

larger

we

get

directly

from

-LRB-

-RRB-

the

exponentially

small

bound

Pr

ae

Z

N

PB

oe

O

SC

exp

-LRB-

log

-RRB-

PB

-LRB-

-RRB-

Note

that

SC

M

B

and

by

Lemma

we

have

PB

N

PBS

ln

-LRB-

M

B

-RRB-

thus

the

exponential

term

in

-LRB-

-RRB-

gets

smaller

exponentially

more

quickly

than

the

SC

term

grows

even

for

relatively

small

And

in

the

early

passes

of

Phase

PB

can

be

substantially

larger

than

ln

-LRB-

M

B

-RRB-

Phase

In

Phase

we

want

to

sort

N

pMBP

ln

-LRB-

M

B

-RRB-

records

in

one

pass

of

distribution

sort

using

O

-LRB-

N

PB

-RRB-

I/Os

which

is

optimal

We

use

S

M

partitions

This

range

of

values

of

N

is

the

case

that

can

not

be

handled

by

Phase

since

when

N

is

relatively

small

the

records

in

each

bucket

will

generally

not

be

distributed

evenly

among

the

clusters

We

want

to

guarantee

that

each

bucket

will

consist

of

at

most

M

records

so

that

no

further

partitioning

is

needed

and

we

want

to

distribute

the

blocks

of

each

of

the

S

buckets

evenly

over

the

disks

Before

we

give

the

algorithm

we

first

define

the

notions

of

logical

track

and

diagonal

Definition

A

logical

track

is

defined

as

I

M

PB

consecutive

tracks

that

are

accessed

in

I

consecutive

I/Os

When

accessing

the

ith

logical

track

we

actually

access

tracks

I

-LRB-

i

-RRB-

Ii

This

is

pictured

in

Figure

Definition

The

ith

diagonal

for

i

N

M

is

defined

as

the

memoryload

of

M

B

blocks

in

which

the

first

set

of

BN

blocks

consists

of

the

ith

logical

track

of

disks

DMP

N

the

second

set

of

BN

blocks

consists

of

the

-LRB-

i

-RRB-

st

logical

track

of

disks

DMP

N

N

and

so

on

wrapping

back

to

i

when

i

exceeds

N

M

Diagonal

is

illustrated

in

Figure

It

follows

from

the

condition

N

pMBP

ln

-LRB-

M

B

-RRB-

of

Phase

that

MP

N

P

and

hence

each

diagonal

is

well-defined

Every

block

in

the

file

is

a

part

of

a

unique

diagonal

and

every

diagonal

contains

the

same

number

of

blocks

from

each

logical

track

Our

algorithm

consists

of

two

passes

Phase

I

Figure

The

shaded

tracks

collectively

represent

a

logical

track

DMP

N

DMP

N

N

DP?MP

N

DP

N

records

oe

I

Figure

The

shaded

areas

collectively

represent

diagonal

Pass

We

scramble

the

records

memoryload

by

memoryload

and

write

them

back

to

the

disks

-LRB-

This

is

where

we

use

randomness

in

Phase

-RRB-

This

step

can

be

done

concurrently

with

the

choosing

of

the

partitioning

elements

Pass

We

read

in

the

file

one

diagonal

-LRB-

memoryload

-RRB-

at

a

time

For

each

memoryload

we

partition

the

records

into

buckets

based

upon

the

partitioning

elements

-LRB-

The

number

of

records

in

each

bucket

of

a

memoryload

will

be

evenly

distributed

with

high

probability

-RRB-

We

write

one

block

to

disk

from

each

bucket

cycling

through

the

tracks

EXTERNAL

SORTING

AND

PERMUTING

on

disk

we

repeat

this

process

until

all

the

records

of

the

memoryload

are

written

At

the

end

of

Pass

each

bucket

will

contain

at

most

M

records

the

sorting

can

then

conclude

with

a

final

series

of

internal

sorts

It

is

convenient

to

think

of

Phase

as

an

instance

of

the

maximum

occupancy

problem

in

hashing

but

different

from

the

instance

we

used

for

motivation

of

Phase

in

Section

in

which

we

scrambled

buckets

randomly

among

the

disks

Here

we

consider

for

each

separate

diagonal

scrambling

the

M

records

of

the

diagonal

randomly

among

the

S

buckets

The

upper

bound

N

pMBP

ln

-LRB-

M

B

-RRB-

guarantees

that

each

diagonal

read

during

Pass

contains

some

number

of

blocks

from

each

of

the

N

=P

logical

tracks

the

random

scrambling

of

the

logical

tracks

in

Pass

means

that

any

given

diagonal

can

contain

any

given

record

in

the

file

Since

the

number

M

of

records

per

diagonal

is

substantially

more

than

the

number

S

of

buckets

the

analogy

to

hashing

suggests

for

each

diagonal

that

the

distribution

of

the

records

in

the

buckets

will

very

likely

be

even

This

general

intuition

is

verified

formally

for

the

actual

problem

at

hand

in

Theorem

Algorithm

Pass

of

Phase

For

simplicity

of

exposition

we

assume

that

the

file

resides

in

packed

format

across

the

disks

track

by

track

In

reality

the

file

formed

by

Phase

is

not

packed

but

it

can

be

read

into

internal

memory

using

full

parallelism

so

our

assumption

is

valid

We

read

in

all

the

records

processing

them

one

logical

track

-LRB-

memoryload

-RRB-

at

a

time

For

each

memoryload

we

randomly

permute

the

records

in

internal

memory

Next

we

form

blocks

based

upon

the

permuted

ordering

and

we

write

the

blocks

back

to

the

logical

track

from

which

they

were

read

for

each

memoryload

of

records

Mi

-LRB-

i

N

M

-RRB-

do

begin

read

Mi

from

the

ith

logical

track

into

internal

memory

randomly

permute

the

M

records

write

Mi

to

the

ith

logical

track

end

Algorithm

Pass

of

Phase

We

read

in

all

of

the

records

one

diagonal

-LRB-

memoryload

-RRB-

at

a

time

The

records

for

each

memoryload

are

partitioned

into

buckets

based

upon

the

partitioning

elements

and

then

written

to

the

disks

as

follows

The

records

within

each

bucket

are

formed

into

blocks

We

then

write

the

blocks

to

the

disks

including

partially-filled

blocks

in

the

following

order

Phase

block

of

block

of

block

of

SS

block

of

block

of

block

of

SS

and

so

on

If

one

of

the

buckets

runs

out

of

blocks

before

the

others

dummy

blocks

for

that

bucket

are

written

The

disks

are

written

to

track

by

track

in

the

cyclical

order

P

k

for

each

diagonal

Mi

-LRB-

i

N

M

-RRB-

do

begin

read

Mi

from

the

ith

diagonal

into

internal

memory

partition

the

M

records

into

buckets

based

upon

the

partitioning

elements

form

the

records

into

blocks

of

size

B

while

a

nonempty

bucket

remains

do

for

j

to

S

do

begin

schedule

next

block

from

Sj

to

be

written

on

next

available

track

of

Dk

assigning

a

dummy

block

if

Sj

is

empty

k

-LRB-

k

mod

P

-RRB-

end

write

the

memoryload

to

the

desired

disks

end

By

our

assumptions

N

M

and

P

are

powers

of

and

thus

S

M

is

relatively

prime

to

P

If

however

the

least

common

multiple

d

of

S

and

P

were

to

satisfy

d

SP

then

the

above

code

would

have

to

be

modified

so

that

after

each

d

S

write

cycles

the

order

that

the

disks

are

written

to

would

be

cyclically

shifted

by

one

The

shifting

would

prevent

each

bucket

from

being

written

to

only

a

small

subset

of

the

disks

Analysis

of

Phase

Theorem

With

overwhelming

probability

Phase

of

the

distribution

sort

algorithm

uses

O

-LRB-

N

PB

-RRB-

I/Os

to

complete

the

sort

of

N

records

The

probability

that

the

number

of

I/Os

used

is

more

than

times

the

average

is

exponentially

small

in

-LRB-

log

-RRB-

-LRB-

M

B

-RRB-

Proof

The

actual

number

of

records

read

in

each

memoryload

might

be

less

than

M

records

since

in

Pass

the

S

M

partitioning

elements

are

retained

in

memory

The

maximum

size

of

each

bucket

formed

must

be

less

than

M

in

order

for

the

sorting

to

be

completed

by

a

series

of

internal

sorts

in

the

next

pass

as

described

in

Section

since

the

final

sorting

pass

requires

that

a

track/disk

pointer

and

partially-filled

block

be

retained

in

memory

At

most

M

S

records

can

be

read

per

memoryload

during

Pass

and

it

is

EXTERNAL

SORTING

AND

PERMUTING

possible

that

only

M

-LRB-

B

-RRB-

M

B

records

can

be

read

per

memoryload

during

the

final

internal

sorting

step

assuming

that

the

disk/track

pointer

does

not

exceed

one

record

For

convenience

we

redefine

M

to

be

M

B?S

so

that

a

full

memoryload

can

be

read

into

or

written

from

internal

memory

in

Pass

This

changes

the

value

of

M

by

at

most

a

small

constant

factor

The

reading

of

the

records

in

Pass

the

reading

and

writing

of

the

records

in

Pass

and

the

writing

of

the

records

in

the

final

pass

described

in

Section

use

O

-LRB-

N

PB

-RRB-

I/Os

We

can

restrict

our

attention

to

the

remaining

set

of

I/Os

we

have

to

consider

namely

the

write

operations

in

Pass

and

the

read

operations

in

the

final

pass

These

two

quantities

are

equal

so

we

restrict

our

attention

to

the

number

of

write

operations

in

Pass

Let

Z

be

the

number

of

I/Os

needed

to

write

all

the

records

one

bucket

at

a

time

in

Pass

We

want

to

show

that

Z

O

N

PB

-LRB-

-RRB-

with

high

probability

In

particular

we

shall

show

that

Pr

ae

Z

N

PB

oe

-LRB-

-RRB-

is

exponentially

small

in

-LRB-

log

-RRB-

-LRB-

M

B

-RRB-

Let

be

the

number

of

times

a

set

of

S

blocks

is

written

in

Pass

and

let

Yi

j

represent

the

number

of

records

found

in

memoryload

Mi

belonging

to

bucket

Sj

We

have

X

i

N

M

max

j

S

ae?Yi

j

B

ssoe

-LRB-

-RRB-

Since

the

last

write

for

each

bucket

may

be

partial

we

get

the

bound

Z

S

P

ss

-LRB-

-RRB-

Note

that

Z

is

expressed

in

-LRB-

-RRB-

in

terms

of

which

by

-LRB-

-RRB-

is

the

sum

over

i

of

j

SfdYi

j

Beg

The

hard

part

of

the

analysis

is

showing

that

j

S

fdYi

j

Beg

is

with

very

high

probability

comparable

to

the

average

value

of

each

dYi

j

Be

We

have

Pr

ae

Z

N

PB

oe

Pr

ae

S

P

ss

N

PB

oe

Pr

ae

P

N

PBS

P

P

oe

Pr

ae

N

BS

-LRB-

P

-RRB-

oe

Pr

X

i

N

M

max

j

S

ae?Yi

j

B

ssoe

N

BS

-LRB-

P

-RRB-

-LRB-

-RRB-

Let

us

define

to

be

any

value

of

j

that

a

priori

-LRB-

before

the

partitioning

elements

are

chosen

-RRB-

maximizes

the

expression

j

Be

M

BS

-LRB-

P

-RRB-

M

Ng

We

can

bound

-LRB-

-RRB-

as

Phase

follows

Pr

X

i

N

M

max

j

S

ae?Yi

j

B

ssoe

N

BS

-LRB-

P

-RRB-

N

M

Pr

-LRB-

max

j

S

ae

j

B

ssoe

M

BS

-LRB-

P

-RRB-

M

N

-RRB-

SN

M

Pr

-LRB-

B

ss

M

BS

-LRB-

P

-RRB-

M

N

-RRB-

SN

M

Pr

-LRB-

M

S

-LRB-

P

-RRB-

MB

N

-LRB-

B

-RRB-

-RRB-

-LRB-

-RRB-

Let

us

define

so

that

M

S

M

S

-LRB-

P

-RRB-

MB

N

-LRB-

B

-RRB-

-LRB-

-RRB-

Substituting

the

Phase

bound

S

M

into

-LRB-

-RRB-

we

find

after

some

algebraic

manipulation

that

M

Since

N

pMBP

ln

-LRB-

M

B

-RRB-

in

Phase

it

follows

that

is

at

most

a

small

constant

amount

less

than

in

particular

we

have

ln

-LRB-

M

B

-RRB-

Substituting

this

value

of

into

-LRB-

-RRB-

we

get

Pr

ae

Z

N

PB

oe

SN

M

Pr

ae

M

S

oe

-LRB-

-RRB-

Let

Yi

j

k

represent

the

number

of

records

found

in

memoryload

Mi

belonging

to

bucket

Sj

read

from

kth

logical

track

In

particular

we

have

k

N

M

k

Let

k

represent

the

expected

value

of

k.

By

Lemma

each

bucket

contains

at

most

M

records

of

the

file

and

there

are

N

M

memoryloads

Thus

we

have

X

k

N

M

k

E

-LRB-

-RRB-

M

N

M

N

Let

Tj

k

be

the

number

of

records

belonging

to

bucket

Sj

on

the

kth

logical

track

We

have

k

kM

N

We

want

to

bound

Sg

We

do

that

by

considering

two

cases

-RRB-

small

k

and

-RRB-

large

k

The

two

cases

are

determined

by

comparing

k

with

the

average

size

that

k

would

be

if

all

the

k

terms

were

equal

to

N

Let

ffi

We

have

Pr

ae

M

S

oe

X

k

N

M

k

Pr

-LRB-

k

k

ffi

-RRB-

X

k

N

M

k

Pr

k

k

ffi?kg

-LRB-

-RRB-

The

above

bound

-LRB-

-RRB-

holds

since

X

k

N

M

k

k

ffi

X

k

N

M

k

-LRB-

k

ffi?k

-RRB-

ffi

N

EXTERNAL

SORTING

AND

PERMUTING

ffi

M

S

ffi

M

S

M

S

The

probability

term

in

-LRB-

-RRB-

is

expressed

in

-LRB-

-RRB-

as

a

sum

of

tails

of

distributions

of

k

where

the

starting

point

of

each

tail

is

sufficiently

far

from

the

mean

k

so

that

the

result

is

exponentially

small

as

we

shall

see

Intuitively

in

order

to

get

a

small

bound

on

the

probability

term

when

k

is

small

the

tail

should

start

at

some

absolute

distance

from

k

and

when

k

is

larger

the

tail

should

start

at

some

multiple

of

k

We

want

to

get

tight

upper

bounds

for

the

tails

of

the

probability

distribution

of

k

listed

in

the

summations

in

-LRB-

-RRB-

Both

summands

have

the

form

PrfX

vg

where

X

k

k

is

the

mean

of

X

and

v

is

a

positive

value

Let

L

N

be

the

number

of

records

read

from

ith

logical

track

by

any

memoryload

The

random

variable

X

k

has

the

hypergeometric

probability

distribution

PrfX

tg

kt

k

L?t

ML

-LRB-

-RRB-

with

mean

kL

M

kM

N

-LRB-

-RRB-

One

approach

to

bounding

PrfX

vg

is

to

use

Chernoff-type

bounds

as

we

did

in

Section

except

in

this

case

we

would

have

to

use

a

two-variable

version

involving

the

supergenerating

function

of

-LRB-

-RRB-

which

has

a

simple

closed

form

A

simpler

approach

is

to

consider

the

ratio

R

-LRB-

t

-RRB-

defined

by

PrfX

t

PrfX

tg

We

have

PrfX

vg

PrfX

vg

PrfX

v

PrfX

gPrfX

vg

PrfX

g

PrfX

v

PrfX

-LRB-

-RRB-

Using

-LRB-

-RRB-

it

is

easy

to

see

that

R

-LRB-

v

-RRB-

is

monotone

decreasing

in

v

and

hence

we

have

PrfX

vg

PrfX

g

PrfX

v

PrfX

Substituting

this

bound

into

-LRB-

-RRB-

we

get

PrfX

vg

PrfX

vg

PrfX

g

-LRB-

PrfX

g

PrfX

-RRB-

PrfX

vg

PrfX

g

-LRB-

-RRB-

Phase

Note

that

we

can

write

PrfX

vg

in

the

following

form

PrfX

vg

PrfX

gPrfX

PrfX

g

PrfX

PrfX

PrfX

vg

PrfX

v

PrfX

g

Y

t

v

R

-LRB-

t

-RRB-

-LRB-

-RRB-

Using

-LRB-

-RRB-

and

-LRB-

-RRB-

we

get

PrfX

vg

Y

t

v

R

-LRB-

t

-RRB-

-LRB-

-RRB-

From

-LRB-

-RRB-

-LRB-

-RRB-

the

definition

of

R

-LRB-

t

-RRB-

and

the

bounds

t

k

M

we

find

after

some

algebraic

manipulation

that

R

-LRB-

t

-RRB-

-LRB-

k

-LRB-

t

-RRB-

-RRB-

-LRB-

L

-LRB-

t

-RRB-

-RRB-

-LRB-

t

-RRB-

-LRB-

M

k

L

t

-RRB-

t

-LRB-

-RRB-

Thus

by

-LRB-

-RRB-

PrfX

vg

Y

t

v

t

A

Taking

the

logarithm

of

both

sides

and

bounding

the

sum

by

an

integral

we

get

ln

PrfX

vg

X

t

v

ln

t

A

Z

v

ln

y

dy

-LRB-

v

-RRB-

ln

v

v

-LRB-

v

-RRB-

ln

v

v

-LRB-

v

-RRB-

ln

-LRB-

-RRB-

Taking

the

exponential

of

both

sides

we

get

Pr

fX

vg

v

-LRB-

v

-RRB-

exp

v

-LRB-

v

-RRB-

ln

-LRB-

-RRB-

Let

us

start

with

the

summand

of

the

second

sum

in

-LRB-

-RRB-

By

applying

-LRB-

-RRB-

and

the

bound

ln

-LRB-

=-LRB-

-RRB-

-RRB-

we

get

k

k

ffi?kg

-LRB-

ffi

-RRB-

-LRB-

ffi

-RRB-

k

exp

-LRB-

ffi?k

ffi

-RRB-

-LRB-

-RRB-

Since

k

we

get

by

some

analysis

that

-LRB-

-RRB-

is

maximized

when

k

and

we

get

k

k

ffi?kg

-LRB-

ffi

-RRB-

-LRB-

ffi

-RRB-

exp

ffi

ffi

-LRB-

-RRB-

EXTERNAL

SORTING

AND

PERMUTING

Note

that

in

Phase

we

always

have

-LRB-

M

B

-RRB-

The

bound

in

-LRB-

-RRB-

is

exponentially

small

in

-LRB-

M

B

-RRB-

when

ffi

is

small

and

exponentially

small

in

ffi

-LRB-

log

ffi

-RRB-

-LRB-

M

B

-RRB-

when

ffi

is

large

For

example

if

ffi

we

can

use

the

bound

ln

-LRB-

x

-RRB-

x

for

x

to

get

k

k

ffi?kg

exp

-LRB-

ffi

-RRB-

M

B

ffi

exp

ffi

M

B

ffi

exp

ffi

-LRB-

ffi

-RRB-

M

B

exp

ffi

M

B

which

is

exponentially

small

when

ffi

-LRB-

M

B

-RRB-

Similarly

by

-LRB-

-RRB-

and

the

bound

ln

-LRB-

=-LRB-

-RRB-

-RRB-

the

summand

of

the

first

sum

in

-LRB-

-RRB-

for

k

is

Pr

-LRB-

k

k

ffi

-RRB-

ffi

k

-LRB-

k

-RRB-

exp

k

-LRB-

-RRB-

which

can

be

bounded

by

an

expression

similar

to

the

right-hand

side

of

-LRB-

-RRB-

For

smaller

k

we

have

from

-LRB-

-RRB-

Pr

-LRB-

k

k

ffi

-RRB-

ffi

k

-LRB-

k

-RRB-

exp

k

ln

k

-LRB-

-RRB-

which

is

exponentially

decreasing

in

a

way

similar

to

-LRB-

-RRB-

In

conclusion

by

combining

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

and

-LRB-

-RRB-

we

can

bound

the

probability

-LRB-

-RRB-

by

times

a

term

exponentially

small

in

-LRB-

log

-RRB-

-LRB-

M

B

-RRB-

The

term

is

very

quickly

masked

by

the

exponentially

small

term

since

in

Phase

we

have

N

pMBP

ln

-LRB-

M

B

-RRB-

which

implies

that

-LRB-

M

B

-RRB-

Completing

the

Sort

After

Phase

is

completed

we

can

read

the

blocks

belonging

to

each

bucket

Sj

using

an

optimal

number

O

-LRB-

N

PBS

-RRB-

of

I/Os

the

disk

and

track

location

of

every

block

-LRB-

including

the

dummy

blocks

-RRB-

belonging

to

each

partition

can

be

easily

computed

because

the

placement

of

the

blocks

was

deterministic

Bucket

Sj

contains

at

most

M

records

so

it

can

be

sorted

internally

We

sort

the

records

of

the

bucket

form

blocks

and

write

the

blocks

to

the

next

available

track/disk

cycling

through

the

disks

We

retain

in

internal

memory

the

last

block

if

partially

full

The

records

in

the

final

partially-filled

block

from

Sj

can

be

treated

as

members

of

Sj

when

Sj

is

processed

Finding

the

Partitioning

Elements

k

for

each

bucket

Sj

-LRB-

j

S

-RRB-

from

Phase

do

begin

read

Sj

into

internal

memory

sort

the

records

in

internal

memory

by

key

values

form

blocks

of

size

B

for

each

full

block

do

begin

schedule

the

block

to

be

written

to

the

next

available

track

on

Dk

k

-LRB-

k

mod

P

-RRB-

end

write

the

full

blocks

of

Sj

end

Finding

the

Partitioning

Elements

All

that

remains

is

to

show

how

to

compute

with

O

-LRB-

N

PB

-RRB-

I/Os

the

S

partitioning

elements

bS

that

break

up

the

file

into

S

roughly

equal-sized

buckets

The

jth

bucket

Sj

consists

of

those

records

R

such

that

bj

key

-LRB-

R

-RRB-

bj

where

and

bS

We

need

to

show

that

conditions

-LRB-

-RRB-

and

-LRB-

-RRB-

of

Section

are

satisfied

Without

loss

of

generality

we

assume

for

simplicity

of

exposition

that

N

M

and

S

are

powers

of

Our

procedure

for

computing

the

approximate

partitioning

elements

must

work

for

the

recursive

step

of

the

algorithm

so

we

assume

that

the

N

records

are

stored

in

O

-LRB-

N

B

-RRB-

blocks

of

contiguous

records

each

of

size

at

most

B.

First

we

describe

a

subroutine

that

uses

O

-LRB-

n

PB

-RRB-

I/Os

to

find

the

record

with

the

kth

smallest

key

-LRB-

or

simply

the

kth

smallest

record

-RRB-

in

a

set

containing

n

records

in

which

the

records

are

stored

on

disk

in

at

most

O

-LRB-

n

B

-RRB-

blocks

We

load

the

n

records

into

memory

one

memoryload

at

a

time

and

sort

each

of

the

dn

Me

memoryloads

internally

We

pick

the

median

record

from

each

of

these

sorted

sets

and

find

the

median

of

the

medians

using

the

linear-time

sequential

algorithm

developed

in

-LSB-

BFP

-RSB-

The

number

of

I/Os

required

for

these

operations

is

O

-LRB-

n

PB

n

M

-RRB-

O

-LRB-

n

PB

-RRB-

We

use

the

key

value

of

this

median

record

to

partition

the

n

records

into

two

sets

It

is

easy

to

verify

that

each

set

can

be

partitioned

into

blocks

of

size

B

-LRB-

except

possibly

for

the

last

block

-RRB-

in

which

each

group

is

stored

contiguously

on

disk

It

is

also

easy

to

see

that

each

of

the

two

sets

has

size

bounded

by

The

algorithm

is

recursively

applied

to

the

appropriate

half

to

find

the

kth

smallest

record

the

total

number

of

I/Os

is

O

-LRB-

n

PB

-RRB-

We

now

describe

how

to

apply

this

subroutine

to

find

the

S

approximate

partitioning

elements

in

a

set

containing

N

records

Let

p

and

q

denote

powers

of

to

be

specificed

later

As

above

we

start

out

by

sorting

N

M

memoryloads

of

records

which

can

be

done

with

O

-LRB-

N

PB

-RRB-

I/Os

Let

us

denote

the

ith

sorted

set

by

Mi

We

construct

a

new

set

of

size

at

most

N

=p

consisting

of

the

kpth

record

-LRB-

in

sorted

order

-RRB-

of

Mi

for

k

M

=p

and

i

N

M

The

records

in

Mi

can

be

output

one

block

at

a

time

The

total

number

of

contiguous

blocks

of

records

comprising

is

O

-LRB-

B

-RRB-

so

we

can

apply

the

subroutine

EXTERNAL

SORTING

AND

PERMUTING

above

to

find

the

record

of

rank

jq

in

with

only

O

-LRB-

PB

-RRB-

O

-LRB-

N

pPB

-RRB-

I/Os

we

call

its

key

value

bj

Thus

if

p

-LRB-

S

-RRB-

the

S

bj

's

can

be

found

with

a

total

of

O

-LRB-

SN

pPB

-RRB-

O

-LRB-

N

PB

-RRB-

I/Os

The

above

description

can

be

expressed

in

the

following

pseudo-code

for

each

memoryload

of

records

Mi

-LRB-

i

N

M

-RRB-

do

begin

read

Mi

into

internal

memory

sort

the

records

in

internal

memory

by

key

values

construct

so

that

it

consists

of

every

pth

record

in

memory

write

end

M

for

j

to

S

do

bj

record

of

rank

qj

in

The

two

lemmas

below

show

that

the

partitioning

is

done

evenly

in

Phases

and

respectively

Lemma

In

the

above

partitioning

algorithm

the

number

of

partitioning

elements

S

satisfies

S

N

pq

If

we

choose

p

-LRB-

S

-RRB-

and

q

-LRB-

S

-RRB-

in

Phase

where

-LRB-

S

-RRB-

then

condition

-LRB-

-RRB-

of

Section

is

true

that

is

N

-LRB-

S

-RRB-

Nj

-LRB-

S

-RRB-

Lemma

If

p

and

q

M

in

the

above

partitioning

algorithm

for

Phase

then

S

M

and

condition

-LRB-

-RRB-

of

Section

is

true

that

is

Nj

The

choices

of

p

in

Lemmas

and

satisfy

p

-LRB-

S

-RRB-

as

can

be

verified

by

using

the

condition

N

pMBP

ln

-LRB-

M

B

-RRB-

for

Phase

and

thus

the

partitioning

can

be

done

with

O

-LRB-

N

PB

-RRB-

I/Os

as

mentioned

above

Lemmas

and

are

special

cases

of

the

following

general

partitioning

lemma

Lemma

The

size

Nj

of

the

jth

bucket

Sj

produced

by

the

above

partitioning

algorithm

satisfies

pq

p

N

M

Nj

pq

p

N

M

Proof

Each

element

in

corresponds

to

a

collection

of

p

elements

in

the

original

file

Since

the

chosen

partitioning

elements

are

q

apart

in

this

gives

us

pq

elements

that

could

be

in

Sj

Let

ei

and

ei

represent

the

pith

record

and

p

-LRB-

i

-RRB-

th

record

from

the

file

in

some

memoryload

If

ei

bj

ei

then

the

p

elements

between

ei

and

ei

may

also

be

in

Sj

Thus

there

may

be

p

additional

records

from

each

of

the

memoryloads

except

from

the

memoryload

that

contributed

bj

This

gives

us

the

upper

bound

Nj

pq

N

M

-LRB-

p

-RRB-

pq

p

N

M

Permuting

for

Very

Small

P

and

B

By

similar

reasoning

we

get

the

lower

bound

Nj

pq

N

M

-LRB-

p

-RRB-

pq

p

N

M

Lemmas

and

follow

directly

from

Lemma

The

condition

-LRB-

S

-RRB-

in

Lemma

is

satisfied

by

the

setting

S

pM

B

-LRB-

M

B

-RRB-

for

Phase

and

we

have

pN

M

-LRB-

S

-RRB-

N

N

-LRB-

S

-RRB-

Permuting

for

Very

Small

P

and

B

Aggarwal

and

Vitter

-LSB-

AgV

-RSB-

show

in

their

one-disk

model

with

P

block

transfers

per

I/O

that

the

optimal

way

to

permute

when

P

and

B

are

very

small

is

the

ve

method

of

repeatedly

moving

P

records

in

each

I/O

from

their

inputed

positions

to

the

desired

final

positions

This

makes

no

use

of

blocking

each

block

transfer

is

used

to

transfer

a

single

record

The

resulting

number

of

I/Os

is

O

-LRB-

N

=P

-RRB-

This

algorithm

does

not

translate

directly

to

our

more

realistic

two-level

model

with

parallel

block

transfer

because

there

is

no

way

to

guarantee

that

the

P

block

transfers

involve

separate

disks

Instead

we

achieve

the

desired

O

-LRB-

N

=P

-RRB-

I/O

bound

by

using

the

following

technique

inspired

by

Phase

In

the

first

pass

the

records

are

read

one

memoryload

at

a

time

Each

memoryload

is

permuted

randomly

in

internal

memory

and

written

back

to

disk

in

the

new

permuted

order

As

a

result

the

records

that

need

to

ultimately

end

up

on

a

particular

disk

i

-LRB-

call

them

Ri

-RRB-

are

spread

with

high

probability

uniformly

among

all

the

disks

This

is

true

for

each

i

P

-LRB-

It

might

take

two

passes

to

do

this

using

pP

bins

each

time

instead

of

P

so

that

there

's

enough

internal

data

structure

space

to

manage

the

placement

of

all

the

bins

on

the

disks

-RRB-

In

the

deterministic

second

pass

for

each

i

P

in

parallel

one

block

of

Ri

is

read

into

internal

memory

Then

B

writes

are

executed

during

each

write

one

record

from

Ri

is

written

to

disk

i

in

parallel

for

each

i

P

The

total

number

of

I/Os

is

O

-LRB-

N

=P

-RRB-

as

desired

assuming

that

the

first

pass

spreads

each

Ri

uniformly

among

the

P

disks

This

uniformity

condition

can

be

proven

using

a

modification

of

the

analysis

of

Section

Standard

Matrix

Multiplication

The

following

is

a

basic

divide

and

conquer

approach

for

scheduling

the

multiplication

of

two

k

k

matrices

using

the

standard

algorithm

If

k

pM

we

multiply

the

matrices

internally

Otherwise

we

do

the

following

steps

We

subdivide

A

and

B

into

eight

k

k

submatrices

-LCB-

and

-LCB-

A

B

We

reposition

the

records

of

the

eight

submatrices

so

that

each

submatrix

is

stored

in

row-major

order

CONCLUSIONS

We

use

the

algorithm

recursively

to

compute

We

reposition

-LCB-

so

that

C

is

stored

in

row-major

order

We

partition

secondary

storage

into

four

contiguous

parts

one

part

for

each

set

of

submatrices

We

define

T

-LRB-

k

-RRB-

to

be

the

number

of

I/Os

used

to

add

two

k

k

matrices

Step

takes

O

-LRB-

PB

-RRB-

I/Os

since

in

the

worst

case

we

can

have

at

most

four

blocks

that

are

assigned

to

be

written

to

the

same

disk

The

number

of

I/Os

needed

to

do

eight

multiplications

of

submatrices

recursively

in

Step

is

T

-LRB-

k

-RRB-

and

the

additions

take

a

linear

number

O

-LRB-

PB

-RRB-

of

I/Os

since

all

of

the

submatrices

are

packed

in

blocks

Step

takes

O

-LRB-

PB

-RRB-

I/Os

it

is

similar

to

Step

When

k

pM

we

get

the

following

recurrence

T

-LRB-

k

-RRB-

T

k

O

PB

where

T

-LRB-

pM

-RRB-

M

PB

This

gives

us

the

desired

I/O

bound

from

Theorem

The

recurrence

for

I

-LRB-

k

-RRB-

the

amount

of

work

done

internally

satisfies

I

-LRB-

k

-RRB-

I

k

O

log

P

where

I

-LRB-

pM

-RRB-

M

PB

which

yields

the

desired

upper

bound

I

-LRB-

k

-RRB-

O

-LRB-

-RRB-

The

internal

processing

time

is

O

-LRB-

I

-LRB-

k

-RRB-

=P

-RRB-

Conclusions

In

this

paper

we

have

introduced

a

new

and

realistic

model

of

two-level

storage

with

parallel

block

transfer

between

the

internal

memory

and

secondary

storage

We

have

developed

practical

algorithms

for

sorting

permuting

matrix

transposition

FFT

permutation

networks

and

standard

matrix

multiplication

that

use

an

optimal

number

of

I/O

steps

The

algorithms

for

sorting

and

permuting

are

based

upon

a

randomized

version

of

distribution

sort

The

partitioning

is

done

by

a

combination

of

two

interesting

probabilistic

techniques

in

order

to

guarantee

that

the

accesses

are

spread

uniformly

over

the

disks

Applications

of

these

techniques

to

obtain

optimal

algorithms

for

the

P-HMM

and

P-BT

hierarchical

memory

models

are

developed

in

the

companion

paper

-LSB-

ViS

-RSB-

Preliminary

work

suggests

that

the

amount

of

randomness

in

our

distribution

sort

algorithm

can

be

greatly

reduced

by

applying

universal

hashing

-LSB-

CaW

-RSB-

in

an

interesting

way

But

the

problem

of

removing

randomness

completely

from

this

technique

is

more

difficult

The

study

of

I/O

efficiency

has

many

applications

besides

the

ones

we

studied

in

this

paper

For

example

graphics

applications

multidimensional

search

problems

and

iterated

lattice

computations

often

involve

I/O-bound

tasks

We

expect

that

the

algorithms

and

insights

we

develop

in

this

paper

will

have

many

applications

in

those

domains

Addendum

At

the

beginning

of

Section

we

gave

some

intuitions

as

to

why

merge

sort

seemed

especially

hard

to

implement

with

an

optimal

number

of

I/Os

in

our

two-level

disk

model

Oddly

enough

a

practical

and

optimal

deterministic

sorting

algorithm

was

recently

developed

by

Nodine

and

Vitter

-LSB-

NoVc

-RSB-

using

a

greedy

merge

sort

Unfortunately

this

merge

sort

algorithm

does

not

seem

to

lead

to

optimal

deterministic

sorting

algorithms

in

most

cases

of

the

P-HMM

P-BT

and

other

parallel

hierarchical

memory

models

Nodine

and

Vitter

have

subsequently

developed

an

optimal

distribution

sort

algorithm

that

is

deterministic

and

that

does

generalize

to

give

optimal

deterministic

parallel

hierarchy

algorithms

-LSB-

NoVa

NoVb

-RSB-

Acknowledgments

We

thank

Bowen

Alpern

Larry

Carter

Tom

Cormen

Mike

Goodrich

and

Mark

Nodine

for

several

helpful

comments

References

-LSB-

AgV

-RSB-

A.

Aggarwal

and

J.

S.

Vitter

The

Input/Output

Complexity

of

Sorting

and

Related

Problems

Communications

of

the

ACM

-LRB-

September

-RRB-

-LCB-

-LSB-

BFP

-RSB-

M.

Blum

R.

W.

Floyd

V.

Pratt

R.

Rivest

and

R.

E.

Tarjan

Time

Bounds

for

Selection

in

Complexity

of

Computer

Calculations

Miller

and

Thatcher

eds.

Plenum

NY

-LCB-

-LSB-

CaW

-RSB-

J.

L.

Carter

and

M.

N.

Wegman

Universal

Classes

of

Hash

Functions

Journal

of

Computer

and

System

Sciences

-LRB-

April

-RRB-

-LCB-

-LSB-

Flo

-RSB-

R.

W.

Floyd

Permuting

Information

in

Idealized

Two-Level

Storage

in

Complexity

of

Computer

Calculations

R.

Miller

and

J.

Thatcher

ed.

Plenum

-LCB-

-LSB-

HGK

-RSB-

L.

Hellerstein

G.

Gibson

R.

M.

Karp

R.

H.

Katz

and

D.

A.

Patterson

Coding

Techniques

for

Handling

Failures

in

Large

Disk

Arrays

Algorithmica

this

issue

-LSB-

Jil

-RSB-

W.

Jilke

Disk

Array

Mass

Storage

Systems

The

New

Opportunity

Amperif

Corporation

September

-LSB-

Kle

-RSB-

L.

Kleinrock

Queueing

Systems

Volume

I

Theory

Wiley

and

Sons

New

York

-LSB-

Knu

-RSB-

D.

Knuth

The

Art

of

Computer

Programming

Volume

Sorting

and

Searching

Addison-Wesley

Reading

MA

-LSB-

Lei

-RSB-

F.

T.

Leighton

Tight

Bounds

on

the

Complexity

of

Parallel

Sorting

IEEE

Transactions

on

Computers

-LRB-

April

-RRB-

-LCB-

-LSB-

LiV

-RSB-

E.

E.

Lindstrom

and

J.

S.

Vitter

The

Design

and

Analysis

of

BucketSort

for

Bubble

Memory

Secondary

Storage

IEEE

Transactions

on

Computers

-LRB-

March

-RRB-

-LCB-

-LSB-

Mag

-RSB-

N.

B.

Maginnis

Store

More

Spend

Less

Mid-Range

Options

Around

Computerworld

-LRB-

November

-RRB-

-LSB-

NoVa

-RSB-

M.

H.

Nodine

and

J.

S.

Vitter

Optimal

Deterministic

Sorting

on

Parallel

Memory

Hierarchies

Department

of

Computer

Science

Duke

University

Technical

Report

January

-LSB-

NoVb

-RSB-

M.

H.

Nodine

and

J.

S.

Vitter

Optimal

Deterministic

Sorting

on

Parallel

Disks

Department

of

Computer

Science

Duke

University

Technical

Report

January

REFERENCES

-LSB-

NoVc

-RSB-

M.

H.

Nodine

and

J.

S.

Vitter

Large-Scale

Sorting

in

Parallel

Memories

Proceedings

of

the

Annual

ACM

Symposium

on

Parallel

Algorithms

and

Architectures

-LRB-

July

-RRB-

-LSB-

PGK

-RSB-

D.

A.

Patterson

G.

Gibson

and

R.

H.

Katz

A

Case

for

Redundant

Arrays

of

Inexpensive

Disks

-LRB-

RAID

-RRB-

Proceedings

of

the

ACM

SIGMOD

International

Conference

on

Management

of

Data

-LRB-

June

-RRB-

-LCB-

-LSB-

SaV

-RSB-

J.

Savage

and

J.

S.

Vitter

Parallelism

in

Space-Time

Tradeoffs

in

Advances

in

Computing

Research

Volume

F.

P.

Preparata

ed.

JAI

Press

-LCB-

-LSB-

Sto

-RSB-

H.

S.

Stone

Parallel

Processing

with

the

Perfect

Shuffle

IEEE

Transactions

on

Computers

-LRB-

February

-RRB-

-LCB-

-LSB-

Uni

-RSB-

University

of

California

at

Berkeley

Massive

Information

Storage

Management

and

Use

-LRB-

NSF

Institutional

Infrastructure

Proposal

-RRB-

Technical

Report

No

UCB/CSD

January

-LSB-

ViF

-RSB-

J.

S.

Vitter

and

Ph.

Flajolet

Average-Case

Analysis

of

Algorithms

and

Data

Structures

in

Handbook

of

Theoretical

Computer

Science

Jan

van

Leeuwen

ed.

NorthHolland

-LSB-

ViS

-RSB-

J.

S.

Vitter

and

E.

A.

Shriver

Algorithms

for

Parallel

Memory

II

Hierarchical

Multilevel

Memories

Algorithmica

this

issue

-LSB-

WuF

-RSB-

C.

Wu

and

T.

Feng

The

Universality

of

the

Shuffle-Exchange

Network

IEEE

Transactions

on

Computers

-LRB-

May

-RRB-

-LCB-

disk
distribution sort
fft
matrix multiplication
permutation
secondary memory
sorting
transposition