Declustering

spatial

databases

on

a

multi-computer

architecture

Nikos

and

Christos

and

Ibrahim

Computer

Systems

Research

Institute

University

of

Toronto

AT&T

Bell

Laboratories

Murray

Hill

NJ

Matsushita

Information

Technology

Laboratory

Abstract

We

present

a

technique

to

decluster

a

spatial

access

method

on

a

shared-nothing

multi-computer

architecture

-LSB-

DGS

-RSB-

We

propose

a

software

architecture

with

the

R-tree

as

the

underlying

spatial

access

method

with

its

non-leaf

levels

on

the

master-server

and

its

leaf

nodes

distributed

across

the

servers

The

major

contribution

of

our

work

is

the

study

of

the

optimal

capacity

of

leaf

nodes

or

chunk

size

-LRB-

or

striping

unit

-RRB-

we

express

the

response

time

on

range

queries

as

a

function

of

the

chunk

size

and

we

show

how

to

optimize

it

We

implemented

our

method

on

a

network

of

workstations

using

a

real

dataset

and

we

compared

the

experimental

and

the

theoretical

results

The

conclusion

is

that

our

formula

for

the

response

time

is

very

accurate

-LRB-

the

maximum

relative

error

was

the

typical

error

was

in

the

vicinity

of

-RRB-

We

illustrate

one

of

the

possible

ways

to

exploit

such

an

accurate

formula

by

examining

several

what-if

scenarios

One

major

practical

conclusion

is

that

a

chunk

size

of

page

gives

either

optimal

or

close

to

optimal

results

for

a

wide

range

of

the

parameters

Introduction

One

of

the

requirements

for

the

database

management

systems

-LRB-

DBMSs

-RRB-

of

the

future

is

the

ability

to

handle

spatial

data

Spatial

data

arise

in

many

applications

including

Cartography

-LSB-

-RSB-

Computer-Aided

Design

-LRB-

CAD

-RRB-

-LSB-

OHM

-RSB-

-LSB-

-RSB-

computer

vision

and

robotics

-LSB-

-RSB-

traditional

databases

where

a

record

with

k

attributes

corresponds

to

a

point

in

a

k-d

space

rule

indexing

in

expert

database

systems

-LSB-

-RSB-

temporal

databases

-LSB-

-RSB-

where

time

can

be

treated

as

one

more

dimension

-LSB-

-RSB-

scientific

databases

with

spatial-temporal

data

etc.

In

several

of

these

applications

the

volume

of

data

is

huge

necessitating

the

use

of

multiple

units

For

example

NASA

expects

Terabyte

-LRB-

-RRB-

of

data

per

day

this

corresponds

to

bytes

per

year

of

satellite

data

Geographic

databases

can

be

large

for

example

the

TIGER

database

of

the

U.S.

Bureau

of

On

leave

from

the

University

of

Maryland

College

Park

His

research

was

partially

funded

by

the

Institute

for

Systems

Research

-LRB-

ISR

-RRB-

and

by

the

National

Science

Foundation

under

Grants

and

-LRB-

PYI

-RRB-

with

matching

funds

from

EMPRESS

Software

Inc.

and

Thinking

Machines

Inc.

Census

is

Gigabytes

Historic

and

temporal

databases

tend

to

archive

all

the

changes

and

grow

quickly

in

size

In

the

above

applications

one

of

the

most

typical

queries

is

the

range

query

Given

a

rectangle

retrieve

all

the

elements

that

intersect

it

A

special

case

of

the

range

query

is

the

point

query

or

stabbing

query

where

the

query

rectangle

degenerates

to

a

point

We

study

the

use

of

parallelism

in

order

to

improve

the

response

time

of

spatial

queries

We

plan

to

use

R-trees

-LSB-

-RSB-

as

our

underlying

data

structure

because

they

guarantee

good

space

utilization

they

treat

geometric

objects

as

a

whole

and

they

give

good

response

time

We

envision

a

shared-nothing

architecture

with

several

workstations

connected

to

a

LAN

The

challenge

is

to

organize

the

data

on

the

available

units

in

order

to

minimize

the

response

time

for

range

queries

The

paper

is

organized

as

follows

Section

briefly

describes

the

R-tree

and

its

variants

Also

it

surveys

previous

efforts

to

parallelize

other

file

structures

Section

proposes

our

architecture

and

describes

its

parameters

and

components

Section

presents

the

analysis

for

computing

the

optimal

chunk

size

Section

presents

experimental

results

and

validates

the

formulas

derived

from

our

analysis

Section

lists

the

conclusions

and

highlights

directions

for

future

work

Survey

Several

spatial

access

methods

have

been

proposed

A

recent

survey

and

classification

can

be

found

in

-LSB-

-RSB-

This

classification

includes

-LRB-

a

-RRB-

methods

that

transform

rectangles

into

points

in

a

higher

dimensionality

space

-LSB-

-RSB-

subsequently

using

a

point

access

method

like

a

grid

file

-LSB-

-RSB-

-LRB-

b

-RRB-

methods

that

use

linear

quadtrees

-LSB-

-RSB-

or

equivalently

the

z-ordering

-LSB-

-RSB-

or

other

space

filling

curves

-LSB-

-RSB-

-LSB-

-RSB-

and

finally

-LRB-

c

-RRB-

methods

based

on

trees

-LRB-

kd-trees

-LSB-

-RSB-

k-d-B-trees

-LSB-

-RSB-

cell-trees

-LSB-

-RSB-

the

BANG

file

-LSB-

-RSB-

hB-trees

-LSB-

-RSB-

n-d

generalizations

of

B-trees

-LSB-

-RSB-

e.t.c.

-RRB-

Fig.

-LRB-

a

-RRB-

Data

-LRB-

dark

rectangles

-RRB-

organized

in

an

R-tree

with

fanout

-LRB-

b

-RRB-

the

resulting

R-tree

on

disk

Among

the

above

three

approaches

we

focus

on

the

R-tree

family

for

the

following

reasons

-LCB-

the

R-trees

do

not

cut

data

rectangles

into

pieces

-LRB-

unlike

the

linear-quadtree

approach

-RRB-

Cutting

data

into

pieces

results

in

an

artificially

increased

database

size

-LRB-

linear

on

the

number

of

pieces

-RRB-

moreover

it

requires

a

duplicateelimination

step

because

a

query

may

retrieve

the

same

object-id

several

times

-LRB-

once

for

each

piece

of

the

qualifying

object

-RRB-

-LCB-

the

R-trees

operate

on

the

native

address

space

which

is

of

lower

dimensionality

in

contrast

transforming

rectangles

into

points

in

a

higher-dimensionality

space

invites

the

dimensionality

curse

problems

early

on

-LCB-

R-trees

seem

more

robust

for

higher

dimensionalities

-LSB-

FBF

-RSB-

Scientific

medical

and

statistical

databases

may

involve

several

dimensions

eg.

-LRB-

x

y

z

time

pressure

wind

velocity

temperature

-RRB-

or

-LRB-

gender

age

cholesterollevel

blood-pressure

etc.

-RRB-

Thus

we

mainly

focus

on

the

R-tree

-LSB-

-RSB-

and

its

variants

The

R-tree

is

an

extension

of

the

B-tree

for

multidimensional

objects

A

geometric

object

is

represented

by

its

minimum

bounding

rectangle

-LRB-

MBR

-RRB-

Non-leaf

nodes

contain

entries

of

the

form

-LRB-

ptr

R

-RRB-

where

ptr

is

a

pointer

to

a

child

node

in

the

R

tree

R

is

the

MBR

that

covers

all

rectangles

in

the

child

node

Leaf

nodes

contain

entries

of

the

form

-LRB-

obj

id

R

-RRB-

where

obj

id

is

a

pointer

to

the

object

description

and

R

is

the

MBR

of

the

object

The

main

innovation

in

the

R-tree

is

that

parent

nodes

are

allowed

to

overlap

This

way

the

R-tree

can

guarantee

good

space

utilization

and

remain

balanced

Figure

illustrates

data

rectangles

-LRB-

in

black

-RRB-

organized

in

an

R-tree

with

fanout

-LRB-

a

-RRB-

while

-LRB-

b

-RRB-

shows

the

file

structure

for

the

same

R-tree

where

nodes

correspond

to

disk

pages

The

R-tree

inspired

much

subsequent

work

whose

main

focus

was

to

improve

the

search

time

A

packing

technique

is

proposed

in

-LSB-

-RSB-

to

minimize

the

overlap

between

different

nodes

in

the

R-tree

for

static

data

An

improved

packing

technique

based

on

the

Hilbert

Curve

is

proposed

in

-LSB-

-RSB-

it

is

extended

for

dynamic

environments

in

-LSB-

-RSB-

The

R

tree

-LSB-

-RSB-

avoids

the

overlap

between

non-leaf

nodes

of

the

tree

by

clipping

data

rectangles

that

cross

node

boundaries

Beckmann

et

al.

proposed

the

R

tree

-LSB-

-RSB-

which

seems

to

have

very

good

performance

The

main

idea

is

the

concept

of

forced

re-insert

which

tries

to

defer

the

splits

to

achieve

better

utilization

When

a

node

overflows

some

of

its

children

are

carefully

chosen

and

they

are

deleted

and

re-inserted

usually

resulting

in

a

better

structured

R-tree

There

is

also

much

work

on

how

to

organize

spatial

access

methods

on

multidisk

or

multi-processor

machines

The

majority

of

them

examine

the

parallelization

of

grid-based

structures

Typical

representatives

include

the

disk

modulo

allocation

method

and

its

variants

-LSB-

-RSB-

-LSB-

-RSB-

methods

using

minimum

spanning

trees

-LSB-

-RSB-

the

field-wise

exclusive-OR

-LRB-

FX

-RRB-

method

-LSB-

-RSB-

methods

using

error

correcting

codes

-LSB-

-RSB-

methods

based

on

the

Hilbert

curve

-LSB-

-RSB-

and

methods

using

lattices

-LSB-

-RSB-

The

objective

in

all

these

methods

is

to

maximize

the

parallelism

for

partial

match

or

range

queries

An

adaptive

algorithm

to

achieve

dynamic

re-declustering

to

cool-off

hot

spots

is

presented

in

-LSB-

-RSB-

However

the

above

methods

try

to

do

the

best

possible

declustering

without

taking

into

account

the

communication

cost

One

of

the

exceptions

is

the

work

by

Ghandeharizadeh

et

al.

-LSB-

-RSB-

which

considers

a

grid

file

with

a

certain

profile

of

range

queries

against

it

a

major

contribution

of

this

work

is

a

formula

to

estimate

the

optimal

number

of

activated

processors

for

a

given

query

Here

we

focus

on

the

parallelization

of

R-trees

The

difference

between

the

R-tree

and

the

grid

file

is

that

the

latter

will

suffer

if

the

attributes

are

correlated

Moreover

the

grid

file

is

mainly

designed

for

point

data

if

the

data

are

rectangles

the

R-tree

is

better

equipped

to

handle

them

Little

work

has

been

done

on

the

parallelization

of

R-trees

In

-LSB-

-RSB-

we

studied

the

multi-disk

architecture

with

no

communication

cost

and

we

proposed

the

so-called

proximity

index

to

measure

the

dis-similarity

of

two

rectangles

in

order

to

decide

which

unit

to

assign

each

rectangle

to

DeWitt

et

al.

-LSB-

DKL

-RSB-

discuss

a

client-server

architecture

for

Geographical

Information

Systems

with

a

rich

data

model

and

a

storage

manager

that

uses

a

variation

of

R-trees

In

this

paper

we

present

a

software

design

to

achieve

efficient

parallelization

of

R-trees

on

a

multi-computer

-LRB-

shared-nothing

-RRB-

architecture

using

commodity

hardware

and

interconnect

as

well

as

fast

processors

Unlike

previous

work

our

major

focus

is

to

optimize

the

unit

of

declustering

itself

which

will

implicitly

optimize

the

number

of

activated

processors

We

derive

closed-form

formulas

and

we

validate

our

results

experimentally

on

a

network

of

SUN

workstations

operating

on

real

data

Proposed

Method

and

System

Architecture

An

overview

of

the

hardware

architecture

is

in

figure

It

consists

of

a

number

of

workstations

connected

together

with

a

LAN

-LRB-

e.g.

Ethernet

-RRB-

that

does

not

support

multi-casting

-LRB-

ability

to

send

a

message

at

all

or

a

subset

of

nodes

at

the

cost

of

one

message

-RRB-

The

problem

is

defined

as

follows

Given

a

set

of

n-dimensional

rectangles

Organize

them

on

the

available

machines

to

minimize

the

response

time

for

range

queries

Target

applications

are

for

example

GIS

applications

-LRB-

retrieve

the

elevation

data

for

a

given

region

-RRB-

statistical

queries

-LRB-

eg.

find

the

average

salary

in

the

state

of

Michigan

for

a

census

database

-RRB-

data

mining

applications

-LSB-

-RSB-

and

decision

support

systems

-LRB-

eg.

find

correlations/rules

among

demographic

data

and

symptoms

for

a

collection

of

patient

records

-RRB-

Given

a

spatial

database

our

first

reaction

would

be

to

do

declustering

over

the

available

machines

Thus

similar

-LRB-

nearby

-RRB-

data

rectangles

should

not

be

stored

on

the

same

machine

This

approach

is

only

partially

correct

because

it

neglects

the

communication

cost

In

this

case

it

is

not

optimal

to

decluster

at

the

data-rectangle

level

The

reason

is

the

following

a

small

query

who

retrieves

say

data

rectangles

will

have

to

engage

machines

-LRB-

sending

at

least

messages

on

the

network

-RRB-

For

such

a

small

query

it

would

be

best

if

those

data

rectangles

were

stored

on

the

same

machine

which

hopefully

could

be

identified

easily

and

could

get

activated

with

only

message

Thus

we

still

want

to

do

declustering

but

not

on

the

rectangle

level

Therefore

we

propose

-LRB-

a

-RRB-

to

group

similar

data

rectangles

in

chunks

-LRB-

also

termed

striping

units

eg.

in

the

RAID

literature

-RRB-

and

-LRB-

b

-RRB-

to

decluster

the

chunks

We

have

the

following

three

design

decisions

what

is

the

optimal

chunk

size

i.e.

the

unit

of

the

amount

of

data

that

we

should

place

in

every

server

such

that

the

search

time

is

minimized

data

placement

Once

the

chunk

size

has

been

determined

what

is

the

best

algorithm

to

distribute

the

chunks

among

several

servers

to

optimize

the

response

time

and

the

parallelism

How

to

do

the

book-keeping

to

record

the

server-id

for

each

chunk

We

discuss

briefly

the

last

two

issues

in

the

next

two

subsections

The

first

issue

the

optimal

chunk

size

selection

is

the

main

focus

of

this

work

and

is

discussed

in

detail

in

section

Book-keeping

As

mentioned

before

we

have

to

do

some

book-keeping

to

record

which

chunks

reside

on

which

machine

This

is

necessary

because

we

need

to

know

which

machine

should

be

activated

as

well

as

where

to

put

new

data

rectangles

if

insertions

-LRB-

and

eventually

splits

-RRB-

are

allowed

We

propose

to

dedicate

one

of

the

machines

to

the

book-keeping

This

machine

will

be

referred

to

as

the

master

server

-LRB-

host

in

the

terminology

of

database

machines

-RRB-

The

rest

of

the

machines

are

plain

servers

they

receive

a

-LRB-

range

-RRB-

query

and

some

chunk

requests

from

the

master

server

they

retrieve

the

appropriate

chunks

from

their

local

disks

and

ship

the

-LRB-

qualifying

-RRB-

data

rectangles

to

the

master

server

A

natural

way

for

the

master

server

to

do

the

book-keeping

is

through

a

spatial

access

method

For

the

reasons

explained

in

section

we

propose

to

use

an

R-tree

-LRB-

any

R-tree

variant

would

work

-RRB-

Figure

gives

an

example

of

an

R

tree

distributed

across

the

master

server

and

N

servers

Chunks

are

stored

on

the

servers

and

they

are

represented

by

their

MBRs

Chunks

correspond

to

the

leaves

of

the

R-tree

We

have

decided

to

store

all

the

non-leaf

nodes

in

the

master

server

for

the

following

reasons

-LCB-

the

non-leaf

portion

of

the

R-tree

in

the

master

server

will

be

small

-LRB-

for

an

R-tree

with

a

fanout

of

f

the

non-leaf

nodes

will

take

roughly

of

the

total

space

of

the

SAM

-RRB-

Thus

the

non-leaf

portion

of

the

R-tree

might

even

fit

in

the

main

memory

of

the

master

server

and

definitely

on

its

disk

-LCB-

It

does

not

pay

off

to

decluster

the

non-leaf

levels

of

the

R-tree

across

the

servers

because

this

will

introduce

inter-server

pointers

higher

communication

cost

and

longer

delays

as

each

node

access

may

have

to

go

through

a

potentially

slow/congested

network

The

proposed

method

has

only

the

following

structural

differences

from

a

traditional

centralized

R-tree

-LCB-

the

leaf

nodes

-LRB-

chunks

-RRB-

span

C

pages

where

C

is

not

necessarily

page

-LCB-

the

pointers

from

the

master

server

to

the

chunks

-LRB-

denoted

by

dashed

lines

in

Figure

-RRB-

consist

of

a

chunk-identifier

and

a

machine-identifier

The

query-resolution

algorithm

for

the

proposed

system

is

as

follows

-LCB-

A

query

starts

at

the

master

server

who

examines

the

-LRB-

locally

stored

-RRB-

nonleaf

nodes

of

the

R-tree

Note

that

this

search

is

done

in

the

normal

way

that

R-trees

are

searched

by

comparing

MBR

's

This

search

will

result

in

some

chunk-ids

and

the

corresponding

machine-ids

-LCB-

The

master

server

activates

the

servers

identified

in

the

previous

step

by

sending

a

message

The

message

contains

the

MBR

of

the

query

and

the

chunk

id

's

to

be

retrieved

from

the

specific

server

Notice

that

the

activation

can

be

done

either

by

broadcasting

or

by

selective

activation

Both

our

implementation

and

our

analysis

can

handle

either

case

with

small

modifications

however

we

focus

on

the

selective

activation

because

it

scales-up

better

This

is

the

typical

approach

that

commercial

parallel

DBMSs

follow

-LRB-

eg.

the

parallel

version

of

IBM

's

-LSB-

BFG

-RSB-

-RRB-

Ethernet

o

o

o

Fig.

The

proposed

architecture

-LCB-

Each

activated

server

fetches

the

relevant

chunks

from

its

local

disk

processes

them

-LRB-

to

discard

data

rectangles

that

do

not

intersect

the

query

rectangle

-RRB-

and

ships

the

qualifying

data

back

to

the

master

server

R-tree

nodes

-LRB-

page

each

-RRB-

Master

serverServer

Fig.

Parallel

Decomposition

for

servers

Data

placement

The

goal

is

that

similar

chunks

should

be

declustered

that

is

if

two

chunks

have

MBR

's

that

are

nearby

in

space

and

therefore

likely

to

qualify

under

the

same

range

query

these

chunks

should

not

be

stored

on

the

same

server

We

distinguish

two

cases

-LRB-

a

-RRB-

a

static

database

-LRB-

no

insertions/deletions/updates

-RRB-

and

-LRB-

b

-RRB-

a

dynamic

one

Fig.

Hilbert

Curves

of

order

and

For

a

static

database

following

-LSB-

-RSB-

we

propose

to

use

Hilbert

Curves

to

achieve

good

declustering

A

Hilbert

curve

is

a

type

of

space

filling

curve

that

visits

all

points

in

a

k-dimensional

grid

exactly

once

and

never

crosses

itself

Thus

it

can

be

used

to

linearize

the

points

of

a

grid

In

-LSB-

-RSB-

it

was

shown

experimentally

that

the

Hilbert

curve

achieves

better

clustering

than

several

other

space-filling

curves

We

can

achieve

good

declustering

by

using

a

variation

of

the

Hilbert-based

declustering

method

-LSB-

-RSB-

as

applied

in

the

so-called

Hilbertpacked

R-trees

-LSB-

-RSB-

-RRB-

For

a

static

collection

of

data

rectangles

the

method

works

briefly

as

follows

we

sort

the

data

rectangles

on

the

Hilbert

values

of

their

centers

and

pack

them

into

R-tree

leaf-nodes

-LRB-

chunks

each

of

size

C

pages

-RRB-

we

scan

the

resulting

list

of

chunks

assigning

chunks

to

servers

in

a

round-robin

fashion

Thanks

to

the

good

clustering

properties

of

the

Hilbert

curve

successive

chunks

will

be

similar

thanks

to

the

round-robin

assignment

they

will

be

assigned

to

different

servers

Thus

similar

chunks

will

be

de-clustered

For

a

dynamic

database

the

quality

of

declustering

depends

on

the

insertion

and

deletion

routines

Insertions

and

deletions

in

our

proposed

structure

can

be

handled

by

modifying

the

corresponding

R-tree

routines

The

only

thing

that

needs

to

be

added

is

that

during

a

split

we

have

to

decide

where

to

put

the

newly

created

chunk

There

are

several

criteria

ranging

from

a

random

assignment

to

elaborate

methods

such

as

the

so-called

proximity

index

-LSB-

-RSB-

This

method

tries

to

put

a

new

chunk

on

a

server

that

has

the

most

un-similar

chunks

To

keep

the

discussion

simple

and

the

emphasis

on

the

optimal

chunk-size

selection

we

restrict

ourselves

to

the

static

case

ignoring

insertions

and

splits

Optimal

chunk-size

selection

In

this

section

we

describe

our

approach

to

optimize

the

chunk

size

Figure

illustrates

the

problems

of

a

poor

choice

for

the

chunk

size

a

tiny

chunk

size

will

result

in

activating

several

servers

even

for

small

queries

-LRB-

see

Q

small

in

Figure

-LRB-

b

-RRB-

-RRB-

resulting

in

high

message

traffic

a

huge

chunk

size

will

limit

parallelism

unnecessarily

even

for

large

queries

-LRB-

see

Q

large

in

Figure

-LRB-

c

-RRB-

-RRB-

The

goal

of

this

whole

section

is

to

find

the

optimal

chunk-size

Copt

so

that

to

minimize

the

response

time

for

a

range

query

Thus

the

major

part

of

the

effort

is

to

express

the

response

time

as

a

function

of

C

and

apply

some

optimization

technique

-LRB-

something

like

setting

the

first

derivative

to

zero

-RRB-

Initial

Data

FileSmall

Chunk

SizeBig

Chunk

Size

-LRB-

a

-RRB-

-LRB-

b

-RRB-

-LRB-

c

-RRB-

Server

Fig.

Illustration

of

problems

with

too

small

and

too

large

chunk

sizes

A

small

chunk

size

will

force

small

queries

-LRB-

Q

small

-RRB-

to

execute

on

many

servers

resulting

in

high

message

traffic

A

large

chunk

size

will

force

large

queries

-LRB-

Q

large

-RRB-

to

execute

in

one

only

server

resulting

in

little

parallelism

Symbols

Explanation

Values

N

Number

of

servers

in

the

architecture

PS

Average

packet

size

-LRB-

in

pages

-RRB-

page

CCidle

Time

to

send

a

message

on

an

idle

Network

CC

Time

to

send

a

message

and

wake

up

cost

under

load

qx

qy

query

sides

D

Total

#of

records

in

the

data

base

Prec

Records

per

page

rec

Tlocal?per?page

Disk

plus

CPU

time

to

fetch

and

process

page

ms

Q

Average

#of

qualifying

pages

for

query

qx

qx

C

chunk

size

in

Pages

Copt

optimal

chunk

size

in

Pages

Table

Symbols

definitions

and

typical

values

As

a

first

step

we

shall

express

the

response

time

as

a

function

of

the

number

K

of

servers

that

get

activated

-LRB-

also

called

promising

servers

from

now

on

-RRB-

Then

we

express

K

as

a

function

of

C

-LRB-

and

the

volume

of

qualifying

data

Q

-RRB-

and

finally

we

optimize

the

resulting

-LRB-

piece-wise

continuous

-RRB-

function

Each

subsection

is

devoted

to

each

of

the

three

above

steps

Response

time

as

a

function

of

number

of

activated

servers

K

Recall

that

as

soon

as

a

range

query

arrives

to

the

master

server

it

consults

the

R-tree

index

and

determines

which

K

servers

to

activate

it

sends

a

message

to

each

of

them

and

it

waits

for

results

from

all

of

the

K

servers

The

method

can

be

applied

to

any

n-dimensional

address

space

Without

loss

of

generality

we

use

examples

from

space

to

simplify

the

presentation

and

the

drawing

of

the

examples

For

further

clarity

we

assume

that

the

address

space

has

been

normalized

to

the

unit

square

Let

Q

be

the

expected

number

of

data

pages

that

a

query

of

size

qx

qy

will

retrieve

There

are

various

ways

to

estimate

Q

with

varying

degrees

of

accuracy

Using

the

uniformity

assumption

-LSB-

-RSB-

we

have

Q

qx?qy?D

Prec

where

D

is

the

total

number

of

records

in

the

database

and

Prec

is

the

number

of

records

per

page

More

accurate

estimates

require

formulas

that

use

the

fractal

dimension

of

the

specific

dataset

-LSB-

-RSB-

However

as

the

experiments

section

shows

even

the

uniformity

assumption

gives

satisfactory

results

because

the

fractal

dimension

of

the

specific

data

set

was

-LSB-

-RSB-

rather

close

to

the

value

of

that

corresponds

to

the

uniformity

assumption

Let

Tlocal?per?page

be

the

time

that

each

server

takes

to

process

one

disk

page

locally

including

the

disk

access

time

and

the

CPU

processing

time

The

response

time

or

round

trip

time

RT

-LRB-

qx

qy

-RRB-

for

a

query

of

size

qx?qy

can

be

expressed

as

a

sum

of

-LRB-

at

most

-RRB-

three

terms

The

first

is

the

local

processing

time

in

each

server

This

time

includes

CPU

time

and

disk

access

time

and

is

referred

as

TLOCAL

The

second

is

the

time

required

to

send

the

query

to

each

server

that

is

involved

in

the

execution

TCOMMUNICATION

The

last

is

the

time

required

to

ship

all

the

qualifying

data

to

the

master

server

TRESULT

Notice

that

the

time

to

traverse

the

R-tree

on

the

master

server

is

ignored

because

it

is

constant

regardless

of

the

chunk

size

and

thus

does

not

participate

in

the

optimization

Moreover

the

R-tree

of

the

master

server

will

most

probably

be

small

typically

fitting

in

core

and

thus

requiring

only

some

small

CPU

time

to

be

traversed

Thus

the

response

time

can

be

expressed

as

RT

-LRB-

qx

qy

-RRB-

TCOMMUNICATION

TLOCAL

TRESULT

-LRB-

-RRB-

To

keep

the

analysis

simple

-LRB-

but

still

accurate

as

the

experiments

show

-RRB-

we

make

the

optimistic

assumption

of

perfect

load

balance

Thus

we

assume

that

all

the

servers

will

have

the

same

local

processing

cost

and

will

ship

back

the

same

amount

of

qualifying

data

In

figures

-LRB-

a

b

-RRB-

we

show

how

each

server

spends

its

time

Gray

black

and

striped

boxes

indicate

the

time

receive

a

message

to

do

local

processing

and

to

ship

back

the

results

respectively

Lack

of

a

box

indicates

an

idle

period

for

this

server

Notice

that

messages

on

the

network

may

not

overlap

The

dashed

lines

in

Figure

help

illustrate

exactly

this

fact

Also

for

the

moment

we

assume

that

there

are

no

other

users

on

the

network

we

shall

see

shortly

how

to

take

them

into

account

Figure

illustrates

the

two

possible

scenarios

depending

on

which

is

the

dominating

delay

-LRB-

communication

time

vs.

local

processing

-RRB-

Domination

of

communication

time

Figure

-LRB-

a

-RRB-

illustrates

this

situation

The

first

server

receives

the

message

collects

its

results

and

has

to

wait

because

the

network

is

still

busy

with

the

messages

from

the

master

server

to

the

rest

of

the

K

servers

In

this

case

TLOCAL

will

not

appear

in

the

expression

for

RT

-LRB-

qx

qy

-RRB-

because

this

term

is

completely

overlapped

by

the

communication

time

Therefore

we

have

TCOMMUNICATION

K

CC

-LRB-

-RRB-

-LRB-

a

-RRB-

Local

processing

timeTimeTime

to

send

back

the

resultsTime

to

send

chunk

requestsTime

to

send

chunk

requests

Server

to

get

back

the

results

-LRB-

b

-RRB-

TimeTime

tosendfirstmessage

Local

processing

timeServer

to

get

back

the

results

Fig.

Timings

when

the

dominating

factor

is

-LRB-

a

-RRB-

the

communication

and

-LRB-

b

-RRB-

the

local

processing

where

K

is

the

number

of

activated

-LRB-

ie.

promising

-RRB-

servers

and

CC

is

the

time

to

send

a

message

-LRB-

of

size

PS

-RRB-

from

the

master

server

to

one

of

the

servers

TRESULT

CC

Q

PS

-LRB-

-RRB-

which

is

the

total

time

to

ship

the

results

back

to

the

master

site

-LRB-

in

packets

of

size

PS

-RRB-

Thus

RT

-LRB-

qx

qy

-RRB-

becomes

RT

-LRB-

qx

qy

-RRB-

K

CC

CC

QP

S

when

-LRB-

K

-RRB-

CC

Q

Tlocal?per?page

K

-LRB-

-RRB-

Domination

of

local

processing

time

Figure

-LRB-

b

-RRB-

illustrates

this

setting

Since

the

local

processing

time

is

long

the

first

server

is

still

retrieving

data

even

after

all

K

servers

have

been

activated

Thus

the

response

time

RT

-LRB-

qx

qy

-RRB-

can

be

estimated

as

the

time

to

send

the

first

message

the

time

for

the

first

server

to

finish

its

local

processing

and

the

time

to

ship

all

the

results

back

Thus

TCOMMUNICATION

CC

-LRB-

-RRB-

-LRB-

for

the

first

message

only

-RRB-

TLOCAL

Q

Tlocal?per?page

K

-LRB-

-RRB-

which

is

the

local

processing

time

for

the

first

-LRB-

as

well

as

every

other

-RRB-

server

activated

and

TRESULT

CC

Q

PS

-LRB-

-RRB-

as

before

Thus

we

finally

have

RT

-LRB-

qx

qy

-RRB-

CC

Q?Tlocal?per?page

K

Q?CC

PS

when

-LRB-

K

-RRB-

CC

Q

Tlocal?per?page

K

-LRB-

-RRB-

where

Q?Tlocal?per?page

K

represents

the

local

processing

time

in

each

node

Recall

that

it

includes

the

disk

access

time

-LRB-

to

fetch

the

page

-RRB-

plus

the

CPU

time

to

check

whether

the

data

rectangles

indeed

intersect

the

query

rectangle

It

is

important

to

highlight

how

overhead

from

other

users

using

the

network

can

be

represented

in

the

formulas

above

As

analyzed

in

-LSB-

-RSB-

the

time

to

send

a

packet

grows

linearly

with

the

number

of

active

nodes

transmitting

that

is

CC

CCidle

-LRB-

-RRB-

where

is

the

number

of

active

-LRB-

transmitting

-RRB-

nodes

simultaneously

with

the

node

in

question

and

CCidle

is

the

time

to

transmit

a

message

on

an

idle

network

Thus

expresses

the

overhead

due

to

other

use

of

the

network

Ideally

the

time

to

send

a

small

message

on

an

Ethernet

under

very

light

load

is

on

the

average

CCidle

K

as

a

function

of

C

The

next

step

is

to

find

the

connection

between

the

number

of

activated

servers

-LRB-

K

-RRB-

and

the

chunk

size

-LRB-

C

-RRB-

For

a

given

chunk

size

C

the

average

number

K

of

servers

that

will

be

activated

can

be

estimated

by

the

formula

K

-LRB-

-LRB-

N

-RRB-

QC

-RRB-

N

-LRB-

-RRB-

This

formula

assumes

that

the

qualifying

chunks

are

randomly

and

independently

distributed

among

the

servers

-LRB-

eg.

through

good

hashing

function

-RRB-

The

justification

of

the

formula

is

as

follows

-LCB-

Q

C

estimates

the

number

of

chunks

that

the

query

will

retrieve

-LCB-

N

is

the

probability

that

a

given

server

will

contain

a

specific

chunk

of

interest

-LCB-

N

is

the

probability

that

a

given

server

will

not

contain

a

specific

chunk

-LCB-

-LRB-

N

-RRB-

Q

C

is

the

probability

that

a

given

server

will

not

contain

any

of

the

Q

C

qualifying

chunks

-LCB-

-LRB-

N

-RRB-

Q

C

is

the

probability

that

a

given

server

will

contain

at

least

one

of

the

requested

chunks

Final

optimization

Combining

the

results

of

the

previous

steps

we

can

express

the

response

time

as

a

function

of

C.

From

equations

-LRB-

-RRB-

-LRB-

repeated

here

for

convenience

-RRB-

we

have

RT

-LRB-

qx

qy

-RRB-

K

CC

CC

QP

S

when

-LRB-

K

-RRB-

CC

Q

Tlocal?per?page

K

-LRB-

-RRB-

RT

-LRB-

qx

qy

-RRB-

CC

Q?Tlocal?per?page

K

Q?CC

PS

when

-LRB-

K

-RRB-

CC

Q

Tlocal?per?page

K

-LRB-

-RRB-

Our

goal

is

to

find

the

optimal

value

Kopt

of

K

-LRB-

and

eventually

Copt

-RRB-

that

minimizes

RT

-LRB-

qx

qy

-RRB-

Notice

that

Eq

increases

with

K

-LRB-

being

linear

on

K

-RRB-

while

Eq

decreases

with

K

-LRB-

since

K

is

in

the

denominator

-RRB-

Moreover

the

combination

of

Eq

-LRB-

-RRB-

forms

a

piece-wise

continuous

function

whose

minimum

is

achieved

at

the

point

of

discontinuity

that

is

when

-LRB-

K

-RRB-

CC

Q

Tlocal?per?page

K

-LRB-

-RRB-

Solving

for

K

we

have

K

Q

Tlocal?per?page

CC

-LRB-

-RRB-

and

after

discarding

the

negative

root

-LRB-

which

has

no

physical

meaning

-RRB-

Kopt

q

Q?Tlocal?per?page

CC

-LRB-

-RRB-

The

above

can

be

solved

for

Copt

using

-LRB-

-RRB-

giving

Copt

Q

N?log

-LRB-

Kopt

N

-RRB-

when

Kopt

N

-LRB-

-RRB-

In

the

formula

above

the

computation

for

the

chunk

size

holds

as

long

as

Kopt

N

When

the

number

of

servers

available

-LRB-

N

-RRB-

is

less

than

Kopt

then

the

optimal

value

of

chunk

size

tends

to

become

from

equation

Intuitively

this

is

expected

since

in

this

case

the

best

we

can

hope

is

that

all

units

participate

in

the

query

execution

and

this

is

guaranteed

by

making

the

chunk

size

as

small

as

possible

However

choosing

the

chunk

size

to

be

less

than

page

will

increase

the

I/O

cost

several

pages

will

be

retrieved

in

order

to

access

only

a

tiny

portion

of

their

contents

This

observation

is

confirmed

in

our

experiments

-LRB-

see

figure

-RRB-

Thus

Copt

page

when

Kopt

N

-LRB-

-RRB-

Summarizing

the

goal

of

this

section

was

to

provide

a

formula

for

the

optimal

value

of

the

chunk

size

Copt

this

is

achieved

with

Eqs

-LRB-

-RRB-

where

Kopt

is

given

by

Eq

Notice

that

Eqs

-LRB-

-RRB-

hold

for

any

dimensionality

of

the

address

space

Experimental

Results

In

this

section

we

present

experimental

results

with

the

proposed

architecture

All

the

experiments

were

conducted

on

a

set

of

rectangles

representing

the

MBRs

-LRB-

minimum

bounding

rectangles

-RRB-

of

road-segments

from

the

Montgomery

County

MD.

The

data

set

consisted

of

line

segments

the

address

space

was

normalized

to

the

unit

square

In

all

cases

we

built

a

Hilbert-packed

R-tree

and

declustered

it

as

explained

in

subsection

The

above

R-tree

package

stored

the

tree

in

main-memory

thus

we

had

to

simulate

each

disk

access

with

a

delay

A

disk

page

was

taken

to

be

The

CPU

time

to

process

a

page

was

timed

and

found

two

orders

of

magnitude

smaller

than

the

disk

access

time

thus

we

ignored

it

in

our

formulas

-LRB-

Tlocal?per?page

disk-access-time

-RRB-

The

implementation

of

the

communication

software

was

done

in

the

C

language

under

UNIX

using

TCP

sockets

For

the

experiments

we

used

several

Sun

SPARC

-LRB-

IPX

-RRB-

workstations

with

of

main

memory

and

sparc

processors

at

connected

via

Ethernet

at

To

avoid

variations

in

the

message

traffic

due

to

other

users

we

ran

the

experiments

at

night

where

the

load

was

light

and

the

communication

cost

CC

per

message

was

fairly

stable

The

queries

were

squares

-LRB-

qx

qy

-RRB-

of

varying

side

q

they

were

randomly

distributed

in

the

address

space

For

each

query

side

q

we

report

the

averages

over

queries

We

performed

the

following

sets

of

experiments

-LCB-

In

the

first

set

we

illustrate

the

accuracy

of

the

formula

for

the

response

time

-LCB-

In

the

second

set

we

experimented

with

various

chunk

sizes

to

show

that

indeed

our

choice

of

chunk

size

minimizes

the

response

time

-LCB-

The

third

set

of

graphs

uses

the

formulas

to

plot

analytical

results

in

what-if

scenarios

Each

set

of

experiments

is

discussed

in

the

corresponding

subsection

next

Accuracy

of

the

formula

Our

very

first

step

was

to

estimate

the

communication

cost

CC

-LRB-

ie.

startup

costs

and

transmission

of

a

message

-RRB-

in

our

environment

Since

our

network

was

not

isolated

random

overhead

was

imposed

to

our

system

from

other

users

of

the

network

We

performed

several

measurements

to

obtain

estimates

at

different

times

during

the

night

when

the

network

traffic

was

low

All

our

experiments

were

also

done

at

night

To

obtain

one

measurement

we

set

up

the

communication

interface

between

two

processes

and

exchanged

messages

of

size

PS

page

at

random

times

The

averages

times

are

presented

in

table

The

estimate

of

CC

was

very

consistent

and

is

the

one

we

use

from

now

on

Measurement

CC

-LRB-

msec

-RRB-

Measurement

Measurement

Measurement

Measurement

Measurement

Measurement

Average

Table

Experiments

to

measure

the

communication

cost

CC

for

a

single

message

CC

Tlocal?per?page

N

Response

Time

-LRB-

msec

-RRB-

Query

Side

Experimental

Theoretical

error

Table

Theoretical

and

Experimental

values

for

response

time

-LRB-

in

ms

-RRB-

for

N

servers

Given

an

accurate

estimate

for

CC

we

can

use

it

in

Eq

's

-LRB-

-RRB-

and

-LRB-

-RRB-

to

make

predictions

about

the

response

time

Tables

and

compare

theoretical

CC

Tlocal?per?page

N

Response

Time

-LRB-

msec

-RRB-

Query

Side

Experimental

Theoretical

error

Table

Theoretical

and

experimental

values

for

response

time

for

N

servers

CC

Tlocal?per?page

N

Response

Time

-LRB-

msec

-RRB-

Query

Side

Experimental

Theoretical

error

Table

Theoretical

and

Experimental

values

for

response

time

-LRB-

in

ms

-RRB-

for

N

servers

and

experimental

values

for

the

response

time

-LRB-

in

milli-seconds

-RRB-

for

N

servers

respectively

and

for

chunk

size

C

page

We

also

present

the

percentage

of

error

of

each

experiment

performed

rounded

up

to

the

nearest

integer

The

main

observation

is

that

our

formula

for

the

response

time

is

accurate

within

or

better

for

all

the

experiments

we

performed

Optimal

chunk-size

selection

In

this

set

of

experiments

we

vary

the

chunk

size

and

we

plot

the

response

time

for

a

given

setting

-LRB-

number

of

servers

N

query

side

q

-RRB-

The

goal

is

to

find

the

chunk

size

that

minimizes

the

response

time

Figures

-LRB-

a

b

-RRB-

present

our

results

for

N

and

servers

respectively

The

various

lines

correspond

to

different

query

sides

q

In

all

these

plots

the

response

time

is

minimized

when

the

chunk

size

is

one

page

-LRB-

Copt

-RRB-

This

result

agrees

with

the

outcome

of

our

formula

for

the

estimation

of

the

optimal

chunk

size

-LRB-

Eq

-RRB-

Substituting

the

parameter

values

of

the

experiment

of

Figure

into

-LRB-

Eq

-RRB-

for

query

sides

qx

qy

we

obtain

Copt

using

the

parameter

values

of

figure

we

obtain

Copt

For

the

other

values

of

query

sides

the

optimal

value

of

processors

Kopt

returned

by

-LRB-

Eq

-RRB-

is

greater

than

the

number

of

servers

-LRB-

Kopt

N

-RRB-

thus

-LRB-

Eq

-RRB-

gives

Copt

page

in

agreement

with

the

experiments

Also

notice

that

C

gives

poor

results

justifying

-LRB-

Eq

-RRB-

The

reasons

have

been

explained

in

section

With

a

smaller-than-page

chunk

size

each

server

will

have

to

retrieve

several

disk

pages

only

to

discard

most

of

their

contents

-LRB-

since

it

will

only

need

a

small

chunk

from

each

page

-RRB-

thus

the

I/O

subsystems

will

be

unnecessarily

overloaded

doing

useless

work

Chunk

Size

vs

Responce

time

for

machines

Query

Side

Query

Side

Query

Side

Query

Side

Query

Side

Responce

time

-LRB-

ms

-RRB-

x

Chunk

Size

Chunk

Size

vs

Responce

time

for

machines

Query

Side

Query

Side

Query

Side

Query

Side

Query

Side

Responce

time

-LRB-

ms

-RRB-

x

Chunk

Size

-LRB-

a

-RRB-

N

servers

-LRB-

b

-RRB-

N

servers

Fig.

Response

time

-LRB-

in

ms

-RRB-

vs

chunk

size

-LRB-

in

pages

-RRB-

for

-LRB-

a

-RRB-

N

servers

-LRB-

b

-RRB-

N

servers

Notice

the

spike

for

C

page

Extrapolation

for

other

scenarios

Having

obtained

confidence

in

our

analysis

in

this

subsection

we

study

its

behavior

through

arithmetic

examples

We

choose

some

realistic

settings

and

do

extrapolations

and

what-if

scenarios

Our

goals

are

to

study

large

databases

networks

of

different

speeds

in

anticipation

of

fast

networks

-LRB-

eg.

fiber

optics

-RRB-

as

well

as

slower

networks

-LRB-

eg.

heavily

loaded

ones

-RRB-

For

the

database

size

we

use

two

sizes

-LCB-

a

medium

db

with

D

-LCB-

a

large

db

with

D

From

our

analysis

-LRB-

eq

-RRB-

it

follows

that

CC

Tlocal?per?page

affects

the

choice

of

chunk

size

Intuitively

this

is

expected

When

CC

Tlocal?per?page

o

the

local

processing

time

will

be

the

bottleneck

thus

increasing

the

use

of

the

network

and

decreasing

the

local

processing

per

server

is

the

proper

action

As

a

consequence

more

servers

should

participate

in

query

execution

and

therefore

a

small

chunk

size

is

preferable

In

the

reverse

situation

when

CC

Tlocal?per?page

AE

the

communication

is

the

bottleneck

The

network

should

be

off-loaded

which

can

be

achieved

by

engaging

fewer

servers

thus

a

larger

chunk

size

is

favorable

now

We

experiment

with

three

settings

-LCB-

current

network

where

CC

Tlocal?per?page

-LCB-

fast

network

where

CC

Tlocal?per?page

-LCB-

slow

network

with

CC

Tlocal?per?page

We

present

the

results

in

Figures

-LRB-

-RRB-

For

each

combination

of

network

speed

and

database

size

we

plot

the

response

time

as

a

function

of

the

chunk

size

for

the

following

query

sides

q

qx

qy

and

Response

Time

-LRB-

ms

-RRB-

Chunk

Size

-LRB-

pages

-RRB-

q

q

q

q

q

q

Response

Time

-LRB-

ms

-RRB-

Chunk

Size

-LRB-

pages

-RRB-

q

q

q

q

q

q

-LRB-

a

-RRB-

current

network

medium

db

-LRB-

b

-RRB-

slow

network

medium

db

Fig.

Analytical

results

for

a

D

-LRB-

medium

-RRB-

database

Response

time

-LRB-

in

ms

-RRB-

vs

chunk

size

-LRB-

in

pages

-RRB-

for

N

servers

and

query

sides

q

-LRB-

bottom

to

top

-RRB-

-LRB-

a

-RRB-

current

network

-LRB-

CC

Tlocal?per?page

-RRB-

-LRB-

b

-RRB-

slow

network

-LRB-

CC

Tlocal?per?page

-RRB-

Response

Time

-LRB-

ms

-RRB-

log

scale

Chunk

Size

-LRB-

pages

-RRB-

q

q

q

q

q

q

Response

Time

-LRB-

ms

-RRB-

Chunk

Size

-LRB-

pages

-RRB-

log

scale

q

q

q

q

q

q

-LRB-

a

-RRB-

current

network

large

db

-LRB-

b

-RRB-

slow

network

large

db

Fig.

Analytical

results

for

a

large

database

-LRB-

D

-RRB-

Logarithm

of

response

time

-LRB-

in

ms

-RRB-

vs

chunk

size

-LRB-

in

pages

-RRB-

N

servers

query

sides

q

-LRB-

bottom

to

top

-RRB-

-LRB-

a

-RRB-

current

network

-LRB-

CC

Tlocal?per?page

-RRB-

-LRB-

b

-RRB-

slow

network

-LRB-

CC

Tlocal?per?page

-RRB-

Figure

shows

the

results

for

a

database

of

size

for

the

current

network

-LRB-

CC

Tlocal?per?page

-RRB-

and

for

a

slow

network

-LRB-

CC

Tlocal?per?page

-RRB-

For

the

first

case

the

response

time

is

minimized

for

Copt

page

all

the

curves

increase

with

C

because

messages

are

relatively

cheap

and

therefore

it

pays

off

to

distribute

the

work

among

many

servers

-LRB-

ie.

small

chunk

size

-RRB-

For

the

slow

network

-LRB-

Figure

-LRB-

b

-RRB-

-RRB-

messages

are

more

expensive

and

therefore

larger

chunk

sizes

are

more

favorable

However

even

then

C

gives

a

response

time

that

is

close

to

the

minimum

The

largest

deviation

from

the

minimum

response

time

is

which

occurs

for

q

for

the

query

sizes

in

the

Figure

Notice

that

this

performance

penalty

diminishes

for

larger

queries

-LRB-

top

two

curves

q

-RRB-

The

reason

is

that

for

larger

queries

the

response

time

becomes

insensitive

to

the

chunk

size

The

explanation

is

the

following

Unless

the

chunk

size

has

a

huge

value

a

large

query

will

activate

all

the

N

servers

anyway

each

of

which

will

do

roughly

of

the

total

local

processing

which

will

be

the

bottleneck

Notice

also

that

for

small

queries

the

optimal

chunk

size

is

C

The

reason

is

that

small

queries

retrieve

a

small

amount

of

data

a

large

chunk

size

will

be

an

over-kill

because

all

the

qualifying

data

will

fit

in

a

chunk

with

room

to

spare

-LRB-

ie.

wasted

I/O

time

to

fetch

extraneous

data

from

the

disk

-RRB-

Figures

b

show

the

same

plots

for

a

database

for

the

current

network

-LRB-

CC

Tlocal?per?page

-RRB-

and

for

the

slow

network

-LRB-

CC

Tlocal?per?page

-RRB-

respectively

Notice

that

the

vertical

axis

is

in

logarithmic

scale

because

a

linear

scale

would

visually

collapse

the

curves

of

the

small

queries

The

response

time

is

rather

insensitive

to

the

chunk

size

the

plots

are

almost

straight

lines

indicating

that

Copt

is

a

good

choice

-LRB-

as

good

as

anything

else

-RRB-

Only

for

very

small

queries

-LRB-

eg.

q

-RRB-

the

curves

bend

a

little

even

then

the

choice

of

C

page

gives

response

time

that

is

within

of

the

minimum

Response

Time

-LRB-

ms

-RRB-

Chunk

Size

-LRB-

pages

-RRB-

q

q

q

q

q

q

Response

Time

-LRB-

ms

-RRB-

log

scale

Chunk

Size

-LRB-

pages

-RRB-

q

q

q

q

q

q

-LRB-

a

-RRB-

fast

network

medium

db

-LRB-

b

-RRB-

fast

network

large

db

Fig.

Analytical

results

for

a

fast

network

-LRB-

CC

Tlocal?per?page

-RRB-

Response

time

-LRB-

in

ms

-RRB-

vs

chunk

size

-LRB-

in

pages

-RRB-

N

servers

query

sides

q

-LRB-

bottom

to

top

-RRB-

-LRB-

a

-RRB-

medium

db

-LRB-

D

-RRB-

-LRB-

b

-RRB-

large

db

-LRB-

D

-RRB-

notice

the

logarithmic

y-axis

Figure

gives

the

plots

for

the

fast

network

-LRB-

CC

Tlocal?per?page

-RRB-

plotting

the

response

time

RT

as

a

function

of

the

chunk

size

C

for

several

values

of

the

query

sides

q

If

the

network

is

fast

we

want

to

maximize

parallelism

This

is

achieved

with

the

smallest

allowable

chunk

size

-LRB-

Copt

-RRB-

This

is

very

pronounced

for

the

medium

database

-LRB-

Figure

-LRB-

a

-RRB-

-RRB-

where

the

response

time

increases

with

the

chunk

size

for

all

the

query

sizes

For

the

large

database

-LRB-

notice

the

logarithmic

y-axis

in

Figure

-LRB-

b

-RRB-

-RRB-

we

see

again

that

C

gives

the

minimum

for

small

queries

for

large

queries

the

response

time

is

insensitive

to

the

chunk

size

as

discussed

before

Thus

the

overall

conclusion

is

that

C

is

a

safe

choice

for

a

wide

range

of

parameters

it

will

either

give

the

optimal

response

time

or

very

close

to

it

Conclusions

In

this

paper

we

have

studied

a

method

to

decluster

a

spatial

access

method

-LRB-

and

specifically

an

R-tree

-RRB-

on

a

shared-nothing

multi-computer

architecture

The

nodes

are

connected

through

an

off-the-shelf

LAN

The

major

contributions

of

this

work

are

The

derivation

of

formulas

for

the

optimal

chunk

size

Copt

-LRB-

striping

unit

-RRB-

for

a

given

query

size

-LRB-

Eqs

-RRB-

The

observation

that

Copt

page

is

a

safe

choice

for

a

wide

range

of

the

problem

parameters

-LRB-

query

size

network/disk

speed

database

size

etc

-RRB-

This

choice

either

gives

the

minimum

response

time

or

close

to

the

-LRB-

rather

flat

-RRB-

minimum

of

the

response

time

Additional

smaller

contributions

include

the

software

architecture

with

the

R-tree

at

the

master-server

and

the

leaf

nodes

-LRB-

chunks

-RRB-

at

the

rest

of

the

servers

We

made

and

justified

several

design

decisions

-LRB-

eg.

selective

activation

vs.

broadcasting

no

pointers

of

the

R-tree

across

servers

etc

-RRB-

the

derivation

of

simple

but

accurate

formulas

that

estimate

the

response

time

for

a

given

query

size

-LRB-

Eq

-RRB-

We

implemented

the

proposed

method

and

we

ran

several

experiments

on

a

network

of

SUN

workstations

operating

on

real

data

-LRB-

road

segments

from

the

Montgomery

county

of

Maryland

U.S.A

-RRB-

The

experiments

showed

that

-LCB-

the

formulas

-LRB-

Eq

-RRB-

for

the

response

time

are

accurate

within

or

better

-LCB-

the

formulas

-LRB-

Eq

-RRB-

for

the

optimal

chunk

size

agree

very

well

with

the

experimental

results

Having

an

accurate

formula

for

the

response

time

is

a

strong

tool

One

of

its

uses

is

for

extrapolation

and

what-if

scenarios

We

can

obtain

estimates

about

the

performance

of

our

system

when

the

network

is

faster

or

slower

-LRB-

loaded

-RRB-

when

the

disks

-LRB-

or

in

general

the

I/O

units

-RRB-

are

slower

-LRB-

e.g.

juke-boxes

of

optical

disks

-RRB-

or

when

the

disks

are

faster

-LRB-

e.g.

thanks

to

large

buffer

pools

at

each

server

and

high

buffer-hit

ratios

-RRB-

etc.

All

we

have

to

do

to

simulate

these

cases

is

to

adjust

the

appropriate

values

for

the

parameters

-LRB-

CC

for

the

communication

cost

Tlocal?per?page

for

the

I/O

and

CPU

cost

per

page

etc.

-RRB-

A

second

use

of

the

formula

could

be

the

analytical

study

of

the

throughput

in

case

of

multiple

concurrent

queries

We

believe

that

the

effects

of

the

additional

queries

can

be

modeled

by

appropriately

inflating

the

service

time

-LRB-

CC

-RRB-

of

the

network

as

well

as

of

the

disks

and

CPUs

of

the

servers

-LRB-

Tlocal?per?page

-RRB-

Using

our

formulas

we

studied

several

what-if

scenarios

A

useful

practical

conclusion

from

this

exercise

is

that

using

chunks

typically

leads

to

optimal

performance

or

very

close

to

it

This

is

especially

true

for

fast

networks

as

well

as

for

large

queries

This

conclusion

is

important

from

a

practical

point

of

view

because

it

provides

a

simple

intuitive

rule

for

the

optimal

choice

of

the

chunk

size

Future

work

includes

the

design

of

parallel

R-tree

algorithms

for

other

types

of

queries

such

as

spatial

joins

-LSB-

-RSB-

and

experimentation

with

other

types

of

interconnects

such

as

ATM

switches

References

-LSB-

-RSB-

Rakesh

Agrawal

and

Ramakrishnan

Srikant

Fast

algorithms

for

mining

association

rules

in

large

databases

Proc

of

VLDB

Conf.

pages

-LCB-

September

-LSB-

-RSB-

D.

Ballard

and

C.

Brown

Computer

Vision

Prentice

Hall

-LSB-

-RSB-

J.L.

Bentley

Multidimensional

binary

search

trees

used

for

associative

searching

CACM

-LRB-

-RRB-

-LCB-

September

-LSB-

BFG

-RSB-

C.

K.

Baru

G.

Fecteau

A.

Goyal

H.

Hsiao

A.

Jhingran

S.

Padmanabhan

G.

P.

Copeland

and

W.

G.

Wilson

Parallel

Edition

IBM

Systems

Journal

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Thomas

Brinkhoff

Hans-Peter

Kriegel

and

Bernhard

Seeger

Efficient

processing

of

spatial

joins

using

r-trees

Proc

of

ACM

SIGMOD

pages

-LCB-

May

-LSB-

-RSB-

N.

Beckmann

H.-P

Kriegel

R.

Schneider

and

B.

Seeger

The

r

tree

an

efficient

and

robust

access

method

for

points

and

rectangles

ACM

SIG

MOD

pages

-LCB-

May

-LSB-

-RSB-

David

Boggs

Jeffrey

C.

Mogul

and

Christopher

A.

Kent

Measured

capacity

of

an

ethernet

Myths

and

reality

WRL

Research

Report

-LSB-

-RSB-

Ling

Tony

Chen

and

Doron

Rotem

Declustering

objects

for

visualization

Proc

VLDB

Conf.

August

to

appear

-LSB-

DGS

-RSB-

D.

DeWitt

S.

Ghandeharizadeh

D.

A.

Schneider

A.

Bricker

H.

Hsiao

and

R.

Rasmussen

The

gamma

database

machine

project

IEEE

Transactions

on

Knowledge

and

Data

Engineering

-LRB-

-RRB-

March

-LSB-

DKL

-RSB-

David

J

DeWitt

Navin

Kabra

Jun

Luo

Jignesh

Patel

and

Jie-Bing

Yu

The

client/server

paradise

Proceedings

of

the

VLDB

Santiago

Chile

September

-LSB-

-RSB-

H.C.

Du

and

J.S.

Sobolewski

Disk

allocation

for

cartesian

product

files

on

multiple

disk

systems

ACM

Trans

Database

Systems

-LRB-

TODS

-RRB-

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

Christos

Faloutsos

and

Pravin

Bhagwat

Declustering

using

fractals

In

Int

Conference

on

Parallel

and

Distributed

Information

Systems

-LRB-

PDIS

-RRB-

pages

-LCB-

San

Diego

CA

January

-LSB-

FBF

-RSB-

Christos

Faloutsos

Ron

Barber

Myron

Flickner

J.

Hafner

Wayne

Niblack

Dragutin

Petkovic

and

William

Equitz

Efficient

and

effective

querying

by

image

content

J.

of

Intelligent

Information

Systems

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

Christos

Faloutsos

and

Ibrahim

Kamel

Beyond

uniformity

and

independence

Analysis

of

r-trees

using

the

concept

of

fractal

dimension

Proc

ACM

SIGACT-SIGMOD-SIGART

PODS

pages

-LCB-

May

Also

available

as

-LSB-

-RSB-

M.F.

Fang

R.C.T.

Lee

and

C.C.

Chang

The

idea

of

de-clustering

and

its

applications

In

Proc

International

Conference

on

VLDB

pages

-LCB-

Kyoto

Japan

August

-LSB-

-RSB-

C.

Faloutsos

and

D.

Metaxas

Declustering

using

error

correcting

codes

Eighth

ACM

SIGACT-SIGMOD-SIGART

Symposium

on

Principles

of

Database

Systems

-LRB-

PODS

-RRB-

pages

-LCB-

March

Also

available

as

and

-LSB-

-RSB-

C.

Faloutsos

and

S.

Roseman

Fractals

for

secondary

key

retrieval

Eighth

ACM

SIGACT-SIGMOD-SIGART

Symposium

on

Principles

of

Database

Systems

-LRB-

PODS

-RRB-

pages

-LCB-

March

also

available

as

UMIACS

and

-LSB-

-RSB-

Michael

Freeston

The

bang

file

a

new

kind

of

grid

file

Proc

of

ACM

SIGMOD

pages

-LCB-

May

-LSB-

-RSB-

Michael

Freeston

A

general

solution

of

the

n-dimensional

b-tree

problem

Proc

of

ACM-SIGMOD

pages

-LCB-

May

-LSB-

-RSB-

I.

Gargantini

An

effective

way

to

represent

quadtrees

Comm

of

ACM

-LRB-

CACM

-RRB-

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

Shahram

Ghandeharizadeh

David

J.

DeWitt

and

W.

Qureshi

A

performance

analysis

of

alternative

multi-attribute

declustering

strategies

SIG

MOD

Conf.

June

-LSB-

-RSB-

O.

Gunther

The

cell

tree

an

index

for

geometric

data

Memorandum

No

UCB/ERL

Univ.

of

California

Berkeley

December

-LSB-

-RSB-

A.

Guttman

New

Features

for

Relational

Database

Systems

to

Support

CAD

Applications

PhD

thesis

University

of

California

Berkeley

June

-LSB-

-RSB-

A.

Guttman

R-trees

a

dynamic

index

structure

for

spatial

searching

Proc

ACM

SIGMOD

pages

-LCB-

June

-LSB-

-RSB-

K.

Hinrichs

and

J.

Nievergelt

The

grid

file

a

data

structure

to

support

proximity

queries

on

spatial

objects

Proc

of

the

WG

-LRB-

Intern

Workshop

on

Graph

Theoretic

Concepts

in

Computer

Science

-RRB-

pages

-LCB-

-LSB-

-RSB-

H.V.

Jagadish

Linear

clustering

of

objects

with

multiple

attributes

ACM

SIGMOD

Conf.

pages

-LCB-

May

-LSB-

-RSB-

Ibrahim

Kamel

and

Christos

Faloutsos

Parallel

r-trees

Proc

of

ACM

SIGMOD

Conf.

pages

-LCB-

June

Also

available

as

Tech

Report

UMIACS

TR

-LSB-

-RSB-

Ibrahim

Kamel

and

Christos

Faloutsos

On

packing

r-trees

Second

Int

Conf

on

Information

and

Knowledge

Management

-LRB-

CIKM

-RRB-

November

-LSB-

-RSB-

Ibrahim

Kamel

and

Christos

Faloutsos

Hilbert

r-tree

an

improved

r-tree

using

fractals

In

Proc

of

VLDB

Conference

pages

-LCB-

Santiago

Chile

September

-LSB-

-RSB-

M.H.

Kim

and

S.

Pramanik

Optimal

file

distribution

for

partial

match

retrieval

Proc

ACM

SIGMOD

Conf.

pages

-LCB-

June

-LSB-

-RSB-

Curtis

P.

Kolovson

and

Michael

Stonebraker

Segment

indexes

Dynamic

indexing

techniques

for

multi-dimensional

interval

data

Proc

ACM

SIG

MOD

pages

-LCB-

May

-LSB-

-RSB-

David

B.

Lomet

and

Betty

Salzberg

The

hb-tree

a

multiattribute

indexing

method

with

good

guaranteed

performance

ACM

TODS

-LRB-

-RRB-

-LCB-

December

-LSB-

-RSB-

J.

Nievergelt

H.

Hinterberger

and

K.C.

Sevcik

The

grid

file

an

adaptable

symmetric

multikey

file

structure

ACM

TODS

-LRB-

-RRB-

-LCB-

March

-LSB-

OHM

-RSB-

J.

K.

Ousterhout

G.

T.

Hamachi

R.

N.

Mayo

W.

S.

Scott

and

G.

S.

Taylor

Magic

a

vlsi

layout

system

In

Design

Automation

Conference

pages

-LCB-

Alburquerque

NM

June

-LSB-

-RSB-

J.

Orenstein

Spatial

query

processing

in

an

object-oriented

database

system

Proc

ACM

SIGMOD

pages

-LCB-

May

-LSB-

-RSB-

N.

Roussopoulos

and

D.

Leifker

Direct

spatial

search

on

pictorial

databases

using

packed

r-trees

Proc

ACM

SIGMOD

May

-LSB-

-RSB-

J.T.

Robinson

The

k-d-b-tree

a

search

structure

for

large

multidimensional

dynamic

indexes

Proc

ACM

SIGMOD

pages

-LCB-

-LSB-

-RSB-

H.

Samet

The

Design

and

Analysis

of

Spatial

Data

Structures

AddisonWesley

-LSB-

-RSB-

T.

Sellis

N.

Roussopoulos

and

C.

Faloutsos

The

r

tree

a

dynamic

index

for

multi-dimensional

objects

In

Proc

International

Conference

on

VLDB

pages

-LCB-

England

September

also

available

as

SRC

-LSB-

-RSB-

R.

Stam

and

Richard

Snodgrass

A

bibliography

on

temporal

databases

IEEE

Bulletin

on

Data

Engineering

-LRB-

-RRB-

December

-LSB-

-RSB-

M.

Stonebraker

T.

Sellis

and

E.

Hanson

Rule

indexing

implementations

in

database

systems

In

Proceedings

of

the

First

International

Conference

on

Expert

Database

Systems

Charleston

SC

April

-LSB-

-RSB-

M.

White

N-Trees

Large

Ordered

Indexes

for

Multi-Dimensional

Space

Application

Mathematics

Research

Staff

Statistical

Research

Division

U.S.

Bureau

of

the

Census

December

-LSB-

-RSB-

J.-H

Wang

T.-S

Yuen

and

D.H.-C

Du

On

multiple

random

accesses

and

physical

data

placement

in

dynamic

files

IEEE

Trans

on

Software

Engineering

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

Gerhard

Weikum

Peter

Zabback

and

Peter

Scheuermann

Dynamic

file

allocation

in

disk

arrays

Proc

ACM

SIGMOD

pages

-LCB-

May

Table

of

Contents

Introduction

Survey

Proposed

Method

and

System

Architecture

Book-keeping

Data

placement

Optimal

chunk-size

selection

Response

time

as

a

function

of

number

of

activated

servers

K

K

as

a

function

of

C

Final

optimization

Experimental

Results

Accuracy

of

the

formula

Optimal

chunk-size

selection

Extrapolation

for

other

scenarios

Conclusions

This

article

was

processed

using

the

LaTEX

macro

package

with

LLNCS

style

parallel data bases
shared nothing architecture