Primary

-LCB-

Backup

Protocols

Lower

Bounds

and

Optimal

Implementations

Navin

Budhiraja

Keith

Marzullo

Fred

B.

Schneidery

Sam

Touegz

Department

of

Computer

Science

Cornell

University

Ithaca

NY

USA

Abstract

We

present

a

precise

specification

of

the

primary

-LCB-

backup

approach

Then

for

a

variety

of

different

failure

models

we

prove

lower

bounds

on

the

degree

of

replication

failover

time

and

worst-case

blocking

time

for

client

requests

Finally

we

outline

primary

-LCB-

backup

protocols

and

indicate

which

of

our

lower

bounds

are

tight

Introduction

One

way

to

implement

a

fault-tolerant

service

is

by

using

multiple

servers

that

fail

independently

The

state

of

the

service

is

replicated

and

distributed

among

these

servers

and

updates

are

coordinated

so

that

even

when

a

subset

of

servers

fail

the

service

remains

available

Such

fault-tolerant

services

are

generally

structured

in

one

of

two

ways

One

approach

is

to

replicate

the

service

state

at

all

servers

and

to

present

all

client

requests

in

the

same

order

to

all

non-faulty

servers

This

service

architecture

is

commonly

called

active

replication

or

the

Supported

by

Defense

Advanced

Research

Projects

Agency

-LRB-

DoD

-RRB-

under

NASA

Ames

grant

number

NAG

-LCB-

and

by

grants

from

IBM

Siemens

and

Xerox

Budhiraja

is

also

supported

by

an

IBM

Graduate

Fellowship

The

views

opinions

and

findings

contained

in

this

report

are

those

of

the

authors

and

should

not

be

construed

as

an

official

Department

of

Defense

position

policy

or

decision

ySupported

in

part

by

the

Office

of

Naval

Research

under

contract

the

National

Science

Foundation

under

Grant

No

DARPA/NSF

Grant

No

and

by

a

grant

from

IBM

Endicott

Programming

Laboratory

zSupported

in

part

by

NSF

grants

and

and

by

a

grant

from

IBM

Endicott

Programming

Laboratory

state

machine

approach

-LSB-

-RSB-

and

has

been

widely

studied

from

both

theoretical

and

practical

viewpoints

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

The

other

approach

to

building

replicated

services

is

to

designate

one

server

as

the

primary

and

all

the

others

as

backups

Clients

make

requests

by

sending

messages

only

to

the

primary

If

the

primary

fails

then

a

failover

occurs

and

one

of

the

backups

takes

over

This

service

architecture

is

commonly

called

the

primary-backup

or

the

primary-copy

approach

-LSB-

-RSB-

and

has

been

widely

used

in

commercial

fault-tolerant

systems

However

the

approach

has

not

been

analyzed

nearly

as

extensively

as

the

state

machine

approach

Little

is

known

of

its

costs

and

tradeoffs

the

degree

of

replication

required

or

the

worst-case

response

time

for

various

failure

models

In

this

paper

we

derive

some

of

these

tradeoffs

For

example

in

some

primary

-LCB-

backup

protocols

-LSB-

-RSB-

the

number

of

servers

used

is

more

than

twice

the

number

of

failures

to

be

tolerated

We

are

now

able

to

explain

this

phenomenon

by

showing

that

the

number

of

servers

needed

depends

on

the

failure

model

With

both

active

replication

and

the

primary-backup

approach

the

goal

is

to

provide

a

client

with

the

illusion

of

a

service

that

is

implemented

by

a

single

server

despite

failures

The

key

difference

between

active

replication

and

the

primary-backup

is

how

each

handles

failures

With

active

replication

the

effects

of

failures

are

completely

masked

by

voting

and

the

service

implemented

is

indistinguishable

from

a

single

non-faulty

server

With

the

primary-backup

approach

a

request

to

the

service

can

be

lost

if

it

is

sent

to

a

faulty

primary

Thus

clients

can

observe

the

effects

of

failures

However

the

periods

during

which

requests

can

be

lost

are

bounded

by

the

length

of

time

that

can

elapse

between

failure

of

the

primary

and

takeover

by

a

backup

Such

behavior

is

an

instance

of

what

we

call

bofo

-LRB-

bounded

outage

finitely

often

-RRB-

To

formulate

the

notion

of

a

bofo

server

define

a

server

outage

to

occur

at

time

t

if

some

client

makes

a

server

request

at

that

time

but

never

receives

a

response

to

that

request

In

a

-LRB-

k

-RRB-

-LCB-

bofo

server

all

server

outages

can

be

grouped

into

at

most

k

intervals

of

time

with

each

interval

having

length

at

most

Accordingly

even

though

some

requests

made

to

a

bofo

service

-LRB-

that

is

a

service

that

implements

the

abstraction

of

a

bofo

server

-RRB-

will

be

lost

this

number

is

limited

Note

that

if

clients

of

a

service

are

restricted

to

send

requests

only

to

one

server

then

it

is

not

possible

to

implement

a

specification

that

is

stronger

than

bofo

This

is

because

if

the

client

sends

a

request

to

a

-LRB-

single

-RRB-

server

and

that

server

subsequently

crashes

then

the

request

can

be

lost

and

will

not

be

processed

This

paper

gives

lower

bounds

for

various

costs

associated

with

implementing

a

bofo

service

by

using

the

primary-backup

approach

These

lower

bounds

depend

on

message

delivery

delay

and

the

class

of

failures

to

be

tolerated

These

bounds

characterize

the

degree

of

replication

the

time

during

which

the

service

can

be

without

a

primary

and

the

amount

of

time

it

can

take

to

respond

to

client

requests

-LRB-

blocking

time

-RRB-

In

some

cases

our

results

are

surprising

For

example

more

than

f

servers

are

necessary

to

tolerate

f

failures

of

certain

types

-LRB-

crash

and

link

failures

receive-omission

failures

or

general-omission

failures

-RRB-

Also

we

have

proved

that

if

a

majority

of

the

servers

can

be

faulty

then

any

primary

-LCB-

backup

protocol

for

receive

-LCB-

omission

failures

will

have

a

run

in

which

a

non-faulty

primary

is

is

forced

to

let

a

faulty

server

become

the

primary

in

its

place

Finally

we

outline

some

course

the

client

can

subsequently

resend

a

copy

of

that

request

to

the

new

primary

simplicity

we

assume

in

this

paper

that

every

request

elicits

a

response

primary

-LCB-

backup

protocols

This

allows

us

to

determine

which

of

our

lower

bounds

are

tight

The

paper

is

organized

as

follows

Section

gives

a

precise

specification

of

the

primarybackup

approach

Section

describes

the

system

model

we

consider

Section

discusses

lower

bounds

and

in

Section

we

outline

our

protocols

and

state

which

of

our

bounds

are

tight

We

conclude

in

Section

Specification

of

the

Primary

-LCB-

Backup

Approach

Since

we

wish

to

derive

lower

bounds

we

must

first

give

a

precise

specification

of

primarybackup

that

is

general

enough

to

satisfy

any

protocol

one

would

characterize

as

being

primary-backup

The

following

four

properties

do

this

The

first

property

states

that

no

more

than

one

server

can

be

the

primary

at

any

time

There

exists

a

local

predicate

Prmys

on

the

state

of

each

server

s

At

any

time

there

is

at

most

one

server

s

whose

state

satisfies

Prmys

For

brevity

whenever

we

say

that

s

is

the

primary

-LRB-

at

time

t

-RRB-

we

mean

that

the

state

of

s

satisfies

Prmys

-LRB-

at

time

t

-RRB-

We

define

the

failover

time

of

a

service

to

be

the

longest

period

of

time

during

which

Prmys

is

not

true

for

any

s.

Property

distinguishes

the

primary-backup

approach

from

active

replication

where

each

client

broadcasts

its

request

to

all

the

servers

Each

client

i

maintains

a

server

identity

Desti

such

that

to

make

a

request

client

i

sends

a

message

-LRB-

only

-RRB-

to

Desti

We

assume

that

requests

sent

to

a

server

s

are

enqueued

in

a

message

queue

at

s.

If

a

client

request

arrives

at

a

server

that

is

not

the

current

primary

then

that

request

is

not

enqueued

-LRB-

and

therefore

is

not

processed

-RRB-

Properties

-LCB-

specify

a

protocol

for

client

interactions

with

a

service

but

not

the

obligations

of

the

service

For

example

the

properties

do

not

rule

out

a

primary

that

ignores

all

requests

A

fourth

property

eliminates

such

trivial

implementations

by

stipulating

that

the

service

implements

a

single

bofo

server

for

some

values

of

k

and

There

exist

fixed

values

k

and

such

that

the

service

behaves

like

a

single

-LRB-

k

-RRB-

-LCB-

bofo

server

We

believe

that

the

above

four

properties

characterize

a

primary-backup

approach

and

have

checked

that

many

primary-backup

protocols

in

the

literature

-LRB-

e.g.

-LSB-

-RSB-

-RRB-

do

satisfy

this

characterization

Note

that

is

not

implementable

if

the

number

of

failures

-LRB-

that

is

the

number

of

servers

and

communication

components

that

fail

-RRB-

can

not

be

bounded

a

priori

This

is

because

an

unbounded

number

of

servers

would

be

required

to

implement

the

service

In

a

practical

system

one

can

implement

service

outages

of

bounded

lengths

by

bounding

the

rate

of

failures

and

allowing

reintegration

of

recovered

servers

and

communication

links

We

do

not

address

failure

rates

or

reintegration

in

this

paper

protocol

of

-LSB-

-RSB-

allows

concurrent

primaries

but

only

for

bounded

periods

If

one

replaces

by

this

weaker

property

then

except

for

the

bounds

on

failover

times

the

bounds

shown

in

Section

continue

to

hold

c

Figure

A

Simple

Primary

-LCB-

Backup

Protocol

A

Simple

Primary

-LCB-

Backup

Protocol

As

an

example

of

a

service

based

on

the

primary

-LCB-

backup

approach

consider

the

following

protocol

which

tolerates

crash

failures

of

a

single

server

Assume

that

all

communication

is

over

point-to-point

non-faulty

links

and

that

each

link

has

an

upper

bound

ffi

on

message

delivery

time

Refer

to

Figure

There

is

a

primary

server

and

a

backup

server

connected

by

a

communications

link

A

client

C

initially

sends

all

requests

to

-LRB-

indicated

by

the

arrow

labeled

in

the

figure

-RRB-

Whenever

receives

a

request

it

ffl

processes

the

request

and

updates

its

state

accordingly

ffl

sends

information

about

the

update

to

-LRB-

message

in

the

figure

-RRB-

ffl

without

waiting

for

an

acknowledgement

from

sends

a

response

to

the

client

-LRB-

message

in

the

figure

-RRB-

The

order

in

which

these

messages

are

sent

is

important

because

it

guarantees

that

given

our

assumption

about

failures

if

the

client

receives

a

response

then

either

will

eventually

receive

message

or

will

crash

Server

updates

its

state

upon

receiving

messages

from

In

addition

sends

dummy

messages

to

every

o

seconds

If

does

not

receive

such

a

message

for

o

ffi

seconds

then

becomes

the

primary

Once

has

become

the

primary

it

informs

the

clients

-LRB-

who

update

their

copies

of

Dest

-RRB-

and

begins

processing

subsequent

requests

from

the

clients

We

now

show

how

this

protocol

satisfies

our

characterization

of

a

primary

-LCB-

backup

protocol

Property

requires

that

there

never

be

two

primaries

This

is

satisfied

by

the

following

definitions

of

Prmy

def

-LRB-

has

not

crashed

-RRB-

def

-LRB-

has

not

received

a

message

from

for

o

ffi

-RRB-

Predicate

is

always

false

in

a

system

executing

our

protocol

and

hence

is

satisfied

The

failover

time

for

this

protocol

is

the

longest

interval

during

which

can

hold

and

it

is

o

seconds

Property

follows

trivially

from

simplify

exposition

we

assume

that

the

maximum

message

delay

between

the

clients

and

the

servers

is

the

same

as

the

delay

between

the

servers

However

our

results

can

be

easily

extended

to

the

case

when

the

delays

are

different

the

description

of

the

protocol

Property

is

true

because

requests

are

not

sent

to

until

after

has

failed

Finally

requires

that

the

protocol

implements

a

single

bofo

server

for

some

values

of

k

and

Since

sends

message

before

message

it

will

never

be

the

case

that

sends

a

response

to

the

client

and

does

not

get

information

about

that

response

from

In

this

protocol

there

is

at

most

one

switch

of

the

primary

So

there

is

at

most

one

outage

period

i.e.

k

To

compute

it

suffices

to

compute

the

longest

interval

during

which

a

client

request

may

not

elicit

a

response

Assume

that

crashes

at

time

tc

Thus

any

client

request

sent

to

at

tc

ffi

or

later

may

be

lost

since

crashes

at

tc

Furthermore

may

not

learn

about

's

crash

until

tc

o

and

clients

may

not

learn

that

is

the

primary

for

another

ffi

So

the

total

period

during

which

a

request

may

not

elicit

a

response

is

tc

ffi

through

tc

o

the

protocol

implements

a

single

-LRB-

o

-RRB-

-LCB-

bofo

server

The

Model

We

consider

a

system

consisting

of

n

servers

and

a

set

of

clients

We

assume

that

server

clocks

are

perfectly

synchronized

with

real

time

Clients

and

servers

communicate

by

exchanging

messages

through

a

completely

connected

point-to-point

network

Each

message

sent

is

enqueued

in

a

queue

maintained

by

the

receiving

process

and

a

process

accesses

its

message

queue

by

executing

a

receive

statement

We

assume

that

links

between

processes

are

FIFO

-LRB-

i.e.

if

pi

sends

message

m

followed

by

to

process

pj

then

pj

will

never

receive

m

after

-RRB-

and

there

is

a

known

constant

ffi

such

that

if

processes

pi

and

pj

are

connected

by

a

-LRB-

non-faulty

-RRB-

link

then

a

message

sent

from

pi

to

pj

at

time

t

will

be

enqueued

in

pj

's

queue

at

of

before

t

ffi

We

are

interested

in

identifying

the

costs

inherent

in

primary

-LCB-

backup

protocols

and

so

we

assume

that

it

takes

no

time

for

a

server

to

compute

a

response

Our

theorems

characterize

lower

bounds

they

are

not

invalidated

by

servers

that

require

a

substantial

amount

of

time

to

compute

a

response

We

model

an

execution

of

a

system

by

a

run

which

is

a

sequence

of

timestamped

events

involving

clients

servers

and

message

queues

These

events

include

sending

messages

enqueuing

messages

receiving

messages

and

internal

events

that

model

computation

at

processes

Two

runs

and

of

the

system

are

defined

to

be

indistinguishable

to

a

process

p

if

the

same

sequence

of

events

-LRB-

with

the

same

timestamps

-RRB-

occur

at

p

in

both

and

We

assume

that

if

two

runs

and

are

indistinguishable

to

p

and

p

has

the

same

initial

state

in

both

runs

then

at

any

time

t

the

state

of

p

at

t

in

is

the

same

as

the

state

of

p

at

t

in

It

is

not

hard

to

extend

the

definition

of

indistinguishability

to

handle

nondeterministic

servers

We

assume

that

the

clients

can

send

any

request

at

any

time

If

we

impose

restrictions

on

the

behavior

of

the

clients

then

we

can

derive

protocols

that

violate

the

lower

bounds

in

this

paper

protocols

can

be

extended

to

the

case

where

clocks

are

only

approximately

synchronized

-LSB-

-RSB-

approach

would

be

assume

that

servers

are

interconnected

with

redundant

broadcast

busses

-LSB-

-RSB-

We

have

not

pursued

this

approach

Define

OE

to

be

the

potential

causality

relation

-LSB-

-RSB-

on

server

events

and

Thus

OE

is

the

transitive

closure

of

the

following

relation

iff

both

and

occur

at

the

same

server

s

and

occurs

before

or

is

a

send

event

and

is

the

corresponding

receive

event

Informally

we

say

a

request

m

is

an

update

request

if

it

changes

the

state

of

the

service

in

such

a

way

that

the

responses

to

subsequent

requests

depend

on

m

More

formally

let

m

be

a

request

with

associated

response

r

-LRB-

and

e

-LRB-

m

-RRB-

and

e

-LRB-

r

-RRB-

be

the

events

in

the

run

associated

with

the

receipt

of

m

and

the

sending

of

r

respectively

-RRB-

Then

m

is

an

update

request

if

all

request/response

pairs

where

was

sent

after

r

have

e

-LRB-

m

-RRB-

OE

e

-LRB-

-RRB-

We

assume

that

update

requests

exist

since

otherwise

the

actions

performed

by

the

primary

do

not

have

to

be

communicated

to

the

backups

We

assume

that

failures

occur

independently

from

each

other

We

consider

the

following

hierarchy

of

failure

models

Crash

failures

A

server

may

fail

by

halting

prematurely

Until

it

halts

it

behaves

correctly

-LSB-

-RSB-

Crash

Link

failures

A

server

may

crash

or

a

link

may

lose

messages

-LRB-

but

links

do

not

delay

duplicate

or

corrupt

messages

-RRB-

Receive-Omission

failures

A

server

may

fail

not

only

by

crashing

but

also

by

omitting

to

receive

some

of

the

messages

directed

to

it

over

a

non-faulty

link

Send-Omission

failures

A

server

may

fail

not

only

by

crashing

but

also

by

omitting

to

send

some

of

the

messages

over

a

non-faulty

link

-LSB-

-RSB-

General-Omission

failures

A

server

may

exhibit

send-omission

and

receive-omission

failures

-LSB-

-RSB-

-RRB-

Note

that

crash

link

failures

and

the

various

types

of

omission

failures

are

quite

different

Although

all

of

these

failure

models

concern

loss

of

messages

each

class

of

failures

is

dealt

with

by

a

different

masking

technique

In

particular

crash

link

failures

can

be

masked

by

adding

redundant

communication

paths

while

omission

failures

can

only

be

masked

by

adding

redundant

servers

so

that

faulty

processes

can

detect

their

own

failure

and

halt

We

return

to

these

masking

techniques

in

Section

Failures

are

counted

by

the

number

of

failing

components

-LRB-

either

servers

or

links

-RRB-

We

say

that

a

protocol

tolerates

f

failures

if

it

works

correctly

despite

the

failure

of

up

to

f

faulty

components

-LRB-

note

that

each

faulty

component

may

fail

many

times

during

an

execution

-RRB-

Lower

Bounds

For

each

failure

model

we

now

give

lower

bounds

for

implementing

a

single

-LRB-

k

-RRB-

-LCB-

bofo

server

using

the

primary

-LCB-

backup

approach

lower

bounds

we

derive

for

crash

failures

also

hold

for

fail-stop

failures

-LSB-

-RSB-

except

for

the

bound

on

failover

time

The

lower

bound

on

failover

time

depends

on

the

maximum

duration

between

when

a

server

pi

fails

and

when

failedi

becomes

true

Bounds

on

Replication

The

first

bound

is

obvious

However

to

introduce

our

notation

and

the

proof

technique

that

will

be

used

later

in

the

section

we

give

a

formal

proof

of

the

theorem

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

crash

failures

requires

n

f

Proof

We

prove

the

result

by

contradiction

Suppose

there

is

a

protocol

P

for

n

f

Thus

P

satisfies

Consider

a

run

in

which

all

n

servers

are

crashed

initially

and

clients

submit

R

kd

de

requests

where

d

is

the

minimum

time

between

the

sending

of

any

two

requests

-LRB-

d

-RRB-

By

at

least

one

of

these

requests

must

elicit

a

response

This

is

because

the

number

of

requests

that

can

not

have

responses

must

fall

into

at

most

k

intervals

of

length

at

most

and

each

interval

of

can

contain

at

most

d

de

requests

However

such

a

response

is

impossible

since

by

assumption

all

servers

have

crashed

The

following

lemma

is

used

in

the

rest

of

the

theorems

in

this

section

Lemma

Consider

any

protocol

that

satisfies

Suppose

two

disjoint

and

nonempty

sets

of

servers

A

and

B

can

be

found

that

meet

the

following

three

properties

There

exists

a

run

oea

containing

R

de

requests

where

d

is

the

minimum

time

between

the

sending

of

any

two

client

requests

-LRB-

d

-RRB-

Furthermore

in

this

run

the

servers

in

A

do

not

crash

and

all

other

servers

crash

at

time

There

exists

a

run

oeb

containing

R

requests

Furthermore

in

this

run

the

servers

in

B

do

not

crash

and

all

other

servers

crash

at

time

There

exists

a

run

oeab

containing

R

requests

Furthermore

the

servers

in

A

and

B

do

not

crash

oeab

is

indistinguishable

from

oea

to

all

servers

in

A

and

oeab

is

indistinguishable

from

oeb

to

all

servers

in

B

At

least

one

of

the

above

runs

violates

Proof

Suppose

for

contradiction

that

the

lemma

is

false

and

runs

oea

oeb

and

oeab

all

satisfy

For

oea

by

at

least

R?kd

de

of

the

requests

must

have

been

received

by

servers

in

A.

Similarly

for

oeb

at

least

R

kd

de

of

the

requests

must

have

been

received

by

servers

in

B.

Finally

since

oeab

is

indistinguishable

from

oea

to

servers

in

A

they

must

execute

the

same

number

of

receive

events

in

both

runs

The

same

holds

for

the

servers

in

B.

By

each

request

is

sent

to

at

most

one

server

and

so

at

least

-LRB-

R

kd

de

-RRB-

requests

must

have

been

sent

in

oeab

Since

only

R

requests

were

sent

we

must

have

R

-LRB-

R

kd

de

-RRB-

or

R

de

which

contradicts

the

assumption

that

R

de

Theorems

and

depend

on

two

parameters

of

primary

-LCB-

backup

protocols

Let

be

the

maximum

time

that

can

elapse

between

any

two

successive

client

requests

-LRB-

possibly

from

different

clients

-RRB-

and

let

D

be

a

duration

such

that

if

some

server

s

becomes

the

primary

at

time

and

remains

the

primary

through

time

t

D

when

a

client

ci

sends

a

request

then

Desti

s

at

time

t.

Hence

D

is

the

minimum

delay

until

all

clients

know

the

identity

of

a

new

primary

For

simplicity

of

notation

we

write

D

to

mean

that

D

is

bounded

and

is

either

unbounded

or

bounded

and

greater

than

D.

Note

that

when

D

the

service

must

be

able

to

detect

the

failure

of

a

primary

and

disseminate

the

new

primary

's

identity

to

the

clients

without

using

any

messages

from

clients

With

both

send-omission

failures

and

crash

link

failures

messages

may

fail

to

reach

their

destinations

The

following

theorem

shows

that

crash

link

failures

are

more

expensive

to

tolerate

as

they

require

more

replication

Theorem

Suppose

there

is

at

most

one

link

between

any

two

servers

Then

any

primary

-LCB-

backup

protocol

tolerating

f

crash

link

failures

and

having

D

requires

n

f

Proof

For

contradiction

assume

the

existence

of

a

protocol

P

with

n

f

We

will

show

that

P

has

three

runs

oea

oeb

and

oeab

that

satisfy

the

conditions

of

Lemma

From

the

lemma

at

least

one

of

these

runs

violates

which

implies

that

P

can

not

be

a

primary

-LCB-

backup

protocol

Let

A

be

a

set

containing

the

one

server

sa

and

let

B

be

the

set

of

remaining

servers

Since

jAj

and

jBj

n

f

A

and

B

can

become

disconnected

by

link

failures

We

first

construct

the

run

oeab

in

which

no

server

crashes

postulating

that

the

links

between

the

servers

in

A

and

B

are

faulty

and

do

not

deliver

any

messages

As

required

by

Lemma

clients

send

a

total

of

R

de

requests

Let

d

D

be

the

minimum

interval

between

any

two

such

requests

We

postulate

that

a

request

will

be

sent

at

time

t

iff

no

request

has

been

sent

during

the

interval

-LSB-

t

d

t

-RRB-

and

one

of

the

following

rules

hold

A

server

s

is

the

primary

during

the

interval

-LSB-

t?D

t

-RSB-

This

request

arrives

immediately

and

is

enqueued

-LRB-

at

s

by

and

the

definition

of

D

-RRB-

There

is

no

primary

at

time

t

This

request

arrives

immediately

and

by

will

never

be

enqueued

at

any

server

A

server

s

is

the

primary

at

time

t

but

another

server

is

the

primary

immediately

after

time

t

If

this

request

is

sent

to

s

then

it

arrives

after

t

and

if

it

is

sent

to

any

other

server

then

it

arrives

immediately

In

both

cases

it

arrives

at

a

server

that

is

not

the

primary

and

so

will

not

be

enqueued

-LRB-

again

by

-RRB-

Note

that

by

construction

the

maximum

interval

between

any

two

client

requests

is

D

d

This

interval

occurs

when

a

server

s

becomes

the

primary

just

before

d

after

a

client

message

is

sent

and

s

remains

the

primary

for

at

least

D.

Hence

the

client

will

be

able

to

send

R

requests

within

time

R

-LRB-

D

d

-RRB-

This

completes

the

construction

of

oeab

We

now

construct

oea

and

oeb

recalling

that

in

oea

all

of

the

servers

except

sa

crash

at

time

and

in

oeb

server

sa

crashes

at

time

The

clients

send

the

same

requests

and

at

the

same

times

in

oea

and

in

oeb

as

in

oeab

Furthermore

by

construction

these

requests

will

arrive

at

the

servers

according

to

the

same

rules

used

in

constructing

oeab

Of

course

a

client

request

may

not

be

delivered

to

the

same

servers

in

oea

or

oeb

as

in

oeab

since

different

servers

are

operational

in

these

runs

Since

sa

does

not

receive

any

messages

from

servers

in

B

in

either

oeab

or

oea

these

two

runs

are

indistinguishable

to

sa

as

long

as

it

receives

the

same

client

requests

at

the

same

times

in

both

runs

We

show

that

this

is

the

case

by

contradiction

let

t

be

the

earliest

time

that

sa

can

distinguish

between

these

two

runs

Thus

at

time

t

either

sa

received

a

request

m

in

oeab

but

not

in

oea

or

it

received

a

request

m

in

oea

but

not

in

oeab

We

will

assume

the

former

the

proof

for

the

latter

is

similar

The

request

m

must

have

been

enqueued

at

some

time

t

at

sa

in

oeab

Since

m

was

received

by

sa

m

must

have

been

sent

by

rule

By

rule

sa

must

have

been

the

primary

through

-LSB-

D

-RSB-

in

oeab

and

therefore

by

indistinguishability

in

oea

as

well

By

the

definition

of

D

m

would

have

been

enqueued

at

sa

at

time

in

oea

as

well

Since

sa

can

not

distinguish

between

the

runs

before

t

sa

can

not

receive

m

before

t

in

oea

and

sa

must

execute

a

receive

in

both

oea

and

oeab

at

time

t.

So

it

must

be

the

case

that

sa

receives

another

request

m

at

time

t

in

oea

Assume

that

was

enqueued

at

time

By

an

indistinguishability

argument

similar

to

above

must

be

enqueued

at

time

at

sa

in

oeab

as

well

Therefore

if

s

received

in

oea

at

time

t

it

must

receive

in

oeab

as

well

a

contradiction

A

similar

argument

can

be

used

to

show

that

the

servers

in

B

receive

the

same

requests

in

oeb

and

oeab

and

so

these

two

runs

are

indistinguishable

to

the

servers

in

B.

Thus

by

Lemma

P

can

not

be

a

primary

-LCB-

backup

protocol

The

assumption

in

this

theorem

that

D

is

significant

As

we

discuss

in

Section

when

D

protocols

that

tolerate

f

crash

link

failures

can

be

constructed

that

use

only

f

servers

The

next

theorem

states

that

additional

replication

is

required

in

order

to

tolerate

receiveomission

failures

The

proof

is

similar

to

that

of

Theorem

and

so

it

is

omitted

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

receive-omission

failures

and

having

D

requires

n

c

The

next

lower

bound

holds

independent

of

the

relation

between

D

and

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

general-omission

failures

requires

n

Proof

Assume

for

contradiction

that

there

is

a

protocol

for

n

Partition

the

servers

into

two

disjoint

sets

A

and

B

of

size

at

most

f

each

We

will

construct

two

runs

and

In

each

run

one

set

of

servers

will

be

faulty

and

the

other

set

will

be

non-faulty

The

servers

in

A

are

faulty

and

fail

to

communicate

with

all

servers

in

B

but

behave

correctly

otherwise

Clients

send

update

requests

until

the

first

response

is

sent

-LRB-

this

must

happen

by

-RRB-

Assume

that

the

first

response

r

to

an

update

request

m

is

sent

at

time

t.

Say

that

this

response

is

sent

by

server

s.

The

same

as

up

to

time

t

but

if

s

is

in

B

then

in

it

is

the

servers

in

B

that

are

faulty

and

fail

to

communicate

with

all

servers

in

A

rather

than

the

servers

in

A

that

are

faulty

In

either

case

r

is

sent

by

a

faulty

server

Furthermore

no

server

can

distinguish

from

through

time

t

and

therefore

the

first

response

r

is

sent

at

time

t

in

as

well

Let

all

of

the

faulty

servers

in

crash

immediately

after

r

is

sent

and

have

clients

continue

to

send

requests

until

another

response

is

sent

This

response

must

have

been

sent

by

a

non-faulty

server

which

implies

that

-LRB-

e

-LRB-

m

-RRB-

OE

e

-LRB-

-RRB-

-RRB-

However

this

violates

the

fact

that

m

is

an

update

request

Bounds

on

Blocking

Informally

a

blocking

primary-backup

protocol

is

one

in

which

the

primary

must

after

receiving

a

request

m

either

receive

a

message

from

another

server

or

simply

wait

an

interval

before

it

can

respond

to

m.

Consider

a

failure-free

run

of

a

primary-backup

protocol

that

is

handling

a

request

Let

the

time

that

the

request

is

received

be

tm

and

the

time

that

the

response

is

sent

be

tr

We

say

that

this

protocol

is

C

-LCB-

blocking

if

it

is

guaranteed

that

tr

tm

C

holds

For

example

any

primary-backup

protocol

in

which

the

primary

sends

information

about

a

request

to

the

backups

and

waits

for

acknowledgement

before

sending

the

response

to

the

client

will

be

at

least

-LCB-

blocking

As

shown

in

Section

-LCB-

blocking

primary-backup

protocols

can

be

built

for

crash

and

crash

link

failure

models

For

servers

that

take

no

time

to

compute

the

response

to

a

request

the

simple

protocol

tolerating

crash

failures

presented

in

Section

is

-LCB-

blocking

We

call

such

protocols

nonblocking

because

the

primary

can

send

a

reply

to

the

client

as

soon

as

the

reply

has

been

computed

Nonblocking

protocols

tolerating

receive-omission

failures

also

exist

as

long

as

n

but

there

is

can

be

no

nonblocking

primary

-LCB-

backup

protocol

tolerating

send-omission

or

general

-LCB-

omission

failures

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

receive-omission

failures

with

f

n

and

D

is

C

-LCB-

blocking

for

some

C

Proof

For

contradiction

suppose

there

is

a

primary

-LCB-

backup

protocol

for

n

and

f

that

is

C

-LCB-

blocking

where

C

Partition

the

servers

into

two

sets

A

and

B

where

jAj

f

and

jBj

n

f

f

We

construct

three

runs

In

all

three

runs

assume

that

all

server

messages

take

ffi

to

arrive

There

are

no

failures

and

all

client

requests

take

ffi

to

arrive

Moreover

clients

send

update

requests

until

some

update

request

m

evokes

a

response

r.

Let

m

be

received

at

time

tm

by

server

p

A

and

r

be

sent

at

time

tr

by

a

server

q

A

-LRB-

q

could

be

the

same

as

p

-RRB-

Notice

that

since

the

protocol

is

C

-LCB-

blocking

where

C

tr

tm

Also

since

by

construction

all

requests

take

ffi

to

arrive

all

client

requests

sent

after

time

tm

ffi

will

be

received

after

time

tr

Identical

to

until

p

receives

m

at

time

tm

At

this

point

in

all

servers

in

A

are

assumed

to

crash

and

clients

are

assumed

to

send

no

request

during

the

interval

-LSB-

tm

ffi

tr

-RSB-

Finally

after

time

tr

clients

are

assumed

to

repeatedly

send

requests

at

intervals

of

at

least

d

where

d

D

as

follows

A

request

is

sent

at

time

t

if

no

request

has

been

sent

in

-LSB-

t

d

t

-RRB-

and

one

of

the

following

rules

hold

A

server

s

B

is

the

primary

during

the

interval

-LSB-

t

D

t

-RSB-

This

request

arrives

immediately

and

is

enqueued

-LRB-

at

s

by

and

the

definition

of

D

-RRB-

There

is

no

primary

in

B

at

time

t

This

request

arrives

immediately

by

will

never

be

enqueued

at

any

server

A

server

s

B

is

the

primary

at

time

t

but

another

server

B

is

the

primary

immediately

after

time

t

If

the

request

is

sent

to

s

then

it

arrives

after

t

and

if

it

is

sent

to

any

other

server

it

arrives

immediately

In

both

cases

it

arrives

at

a

server

that

is

no

the

primary

and

so

will

not

be

enqueued

-LRB-

again

by

-RRB-

Notice

that

eventually

there

will

be

a

response

-LRB-

say

-RRB-

in

because

the

protocol

satisfies

and

by

construction

it

must

be

from

a

request

sent

by

rule

The

same

as

except

that

the

servers

in

A

do

not

crash

at

time

tm

Instead

the

servers

in

B

commit

receive

failures

on

all

messages

sent

after

tm

by

servers

in

A.

Clients

send

requests

at

the

same

times

as

in

which

arrive

using

the

same

rules

as

Now

consider

these

three

runs

By

construction

the

runs

are

identical

up

to

time

tm

Since

all

server

messages

take

ffi

to

arrive

clients

can

not

distinguish

and

through

tm

ffi

and

so

clients

send

the

same

requests

to

the

same

servers

in

both

and

Similarly

since

all

server

messages

take

ffi

to

arrive

the

servers

in

B

can

not

distinguish

between

and

through

tm

ffi

Therefore

since

tr

tm

p

-LRB-

the

server

that

received

request

m

at

time

tm

in

-RRB-

and

q

-LRB-

the

server

that

sent

response

r

at

time

tr

in

-RRB-

can

not

distinguish

between

and

through

time

tr

and

so

q

sends

response

r

in

as

well

Then

using

an

argument

similar

to

the

one

in

Theorem

servers

in

B

can

not

distinguish

and

and

so

response

also

occurs

in

However

-LRB-

e

-LRB-

m

-RRB-

OE

e

-LRB-

-RRB-

-RRB-

which

violates

the

assumption

that

m

is

an

update

request

In

run

of

the

above

proof

a

correct

primary

-LRB-

p

in

set

A

-RRB-

becomes

the

backup

while

a

faulty

server

from

set

B

becomes

the

primary

in

p

's

place

It

is

always

possible

to

construct

such

a

run

This

is

a

disconcerting

property

there

does

not

exist

a

primary

-LCB-

backup

protocol

that

tolerates

receive-omission

failures

with

n

in

which

a

primary

cedes

only

when

it

fails

Moreover

this

lower

bound

is

tight

in

-LSB-

-RSB-

we

give

a

receive-omission

primary

-LCB-

backup

protocol

with

n

in

which

a

primary

cedes

only

when

it

fails

And

if

f

then

the

following

theorem

holds

its

proof

is

similar

to

the

proof

of

Theorem

-LRB-

and

is

therefore

omitted

-RRB-

except

that

p

q.

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

receive-omission

failures

with

f

and

n

and

having

D

is

C

-LCB-

blocking

for

some

C

ffi

Primary

-LCB-

backup

protocols

tolerating

send-omission

or

general

-LCB-

omission

failures

exhibit

the

same

blocking

properties

as

those

tolerating

receive-omission

failures

except

that

the

restriction

D

is

no

longer

necessary

Here

we

prove

just

the

results

for

send

-LCB-

omission

failures

The

results

for

general

-LCB-

omission

failures

then

follow

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

send-omission

failures

with

f

is

C

-LCB-

blocking

for

some

C

Proof

For

contradiction

suppose

there

is

a

primary

-LCB-

backup

protocol

that

is

C

-LCB-

blocking

where

C

We

consider

the

following

two

runs

in

which

all

server

messages

take

ffi

to

arrive

There

are

no

failures

and

all

client

requests

take

ffi

to

arrive

Moreover

clients

send

update

requests

until

some

update

request

m

evokes

a

response

r.

Let

m

be

received

at

time

tm

by

server

p

and

r

be

sent

at

time

tr

by

a

server

q

-LRB-

again

q

could

be

p

-RRB-

Notice

that

since

the

protocol

is

C

-LCB-

blocking

where

C

tr

tm

Also

since

by

construction

all

requests

take

ffi

to

arrive

all

client

requests

sent

after

time

tm

ffi

will

be

received

after

time

tr

Identical

to

through

tm

After

tm

p

and

q

fail

and

omit

to

send

all

messages

to

all

servers

except

each

other

Since

by

construction

all

messages

take

ffi

to

arrive

servers

and

clients

can

not

distinguish

between

and

through

tm

ffi

and

as

a

result

p

and

q

can

not

distinguish

the

two

runs

through

tm

ffi

Therefore

since

tr

tm

q

sends

the

response

r

at

time

tr

in

as

well

Now

let

p

and

q

crash

at

time

tr

and

the

clients

send

requests

after

time

tr

By

there

eventually

must

be

some

request

that

results

in

a

response

However

-LRB-

e

-LRB-

m

-RRB-

OE

e

-LRB-

-RRB-

-RRB-

which

violates

the

assumption

that

m

us

an

update

request

Again

if

f

then

the

following

theorem

can

be

proved

using

a

proof

similar

to

Theorem

except

that

p

q.

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

send-omission

failures

with

f

is

C

-LCB-

blocking

for

some

C

ffi

Bounds

on

Failover

Times

Recall

that

the

failover

time

is

the

longest

interval

during

which

Prmys

is

not

true

for

any

server

s

In

this

section

we

give

lower

bounds

for

failover

times

In

order

to

discuss

these

bounds

we

postulate

a

fifth

property

of

primary

-LCB-

backup

protocols

A

correct

server

that

is

the

primary

remains

so

until

there

is

a

failure

of

some

server

or

link

This

is

a

reasonable

expectation

and

it

is

valid

for

all

protocols

that

we

have

found

in

the

literature

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

crash

failures

must

have

a

failover

time

of

at

least

f

ffi

Proof

The

proof

is

by

induction

on

f

Base

case

f

trivially

true

since

a

failover

time

can

not

be

smaller

than

zero

Induction

case

f

Suppose

the

theorem

holds

for

at

most

f

failures

but

-LRB-

for

a

proof

by

contradiction

-RRB-

there

is

a

protocol

P

for

which

the

theorem

is

false

when

there

are

f

failures

From

the

induction

hypothesis

there

is

a

run

oe

with

at

most

f

failures

and

an

interval

-LSB-

-RSB-

at

least

-LRB-

f

-RRB-

ffi

during

which

there

is

no

primary

Let

be

the

server

that

becomes

the

primary

at

Consider

the

two

runs

and

that

extend

oe

as

follows

Assume

crashes

at

time

By

assumption

there

exists

a

new

primary

-LRB-

say

-RRB-

at

time

ffi

Since

crashes

at

time

does

not

receive

any

messages

from

that

were

sent

after

time

Assume

that

is

correct

there

are

no

other

crashes

at

or

after

and

all

messages

sent

by

after

time

take

ffi

to

arrive

Since

can

not

distinguish

from

through

time

becomes

the

primary

at

time

in

By

however

remains

the

primary

at

time

in

This

violates

and

so

P

is

not

a

primary

-LCB-

backup

protocol

Failover

times

for

all

other

failure

models

have

a

larger

lower

bound

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

crash

link

failures

has

a

failover

time

of

at

least

ffi

Proof

The

proof

is

by

induction

on

f

Base

case

f

trivially

true

Induction

case

f

Suppose

the

theorem

holds

for

at

most

f

failures

but

-LRB-

for

a

proof

by

contradiction

-RRB-

there

is

a

protocol

P

for

which

the

theorem

is

false

when

there

are

f

failures

From

the

induction

hypothesis

there

is

a

run

oe

with

at

most

f

failures

and

an

interval

-LSB-

-RSB-

at

least

-LRB-

f

-RRB-

ffi

during

which

there

is

no

primary

Let

be

the

server

that

becomes

the

primary

at

Consider

the

three

runs

and

that

extend

oe

as

follows

Assume

that

crashes

at

time

and

all

messages

sent

after

take

ffi

to

arrive

Furthermore

the

crash

of

is

the

only

failure

at

or

after

By

assumption

there

exists

a

new

primary

-LRB-

say

-RRB-

at

time

Since

crashes

at

time

does

not

receive

any

messages

from

that

were

sent

after

time

Furthermore

since

all

messages

take

ffi

to

arrive

any

message

that

was

sent

after

ffi

can

be

received

by

only

after

time

Assume

that

is

correct

there

are

no

other

failures

at

or

after

and

all

messages

sent

after

time

take

ffi

to

arrive

Since

there

are

no

failures

at

or

after

time

by

continues

to

be

the

primary

through

time

The

same

as

except

that

the

link

between

and

is

faulty

and

does

not

deliver

any

message

sent

by

to

after

time

By

construction

can

not

distinguish

from

through

time

and

so

becomes

the

primary

at

time

in

Similarly

can

not

distinguish

from

through

time

and

so

remains

the

primary

until

time

in

This

violates

and

so

P

is

not

a

primary

-LCB-

backup

protocol

We

omit

the

proofs

of

the

following

two

theorems

because

they

are

similar

to

Theorem

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

receive-omission

failures

has

a

failover

time

of

at

least

ffi

Theorem

Any

primary

-LCB-

backup

protocol

tolerating

f

send-omission

failures

has

a

failover

time

of

at

least

ffi

Outline

of

the

Protocols

In

order

to

establish

that

the

bounds

given

above

are

tight

we

have

developed

primary

-LCB-

backup

protocols

for

the

different

failure

models

-LSB-

-RSB-

In

this

section

we

outline

these

protocols

and

discuss

which

of

our

lower

bounds

are

tight

Our

protocol

for

crash

failures

is

similar

to

the

protocol

given

in

Section

Whenever

the

primary

receives

a

request

from

the

client

it

processes

that

request

and

sends

information

about

state

updates

to

the

backups

before

sending

a

response

to

the

client

All

servers

periodically

send

messages

to

each

other

in

order

to

detect

server

failures

This

protocol

uses

-LRB-

f

-RRB-

servers

and

so

the

lower

bound

in

Theorem

is

tight

Furthermore

it

is

nonblocking

and

so

incurs

no

additional

delay

It

has

the

failover

time

f

ffi

o

for

arbitrarily

small

and

positive

o

and

so

the

lower

bound

in

Theorem

is

tight

In

order

for

the

protocol

to

tolerate

crash

link

failures

we

add

an

additional

server

By

Theorem

this

server

is

necessary

The

additional

server

ensures

that

there

is

always

at

least

one

non-faulty

path

between

any

two

correct

servers

where

a

path

contains

zero

or

more

intermediate

servers

The

protocol

for

crash

failures

outlined

above

is

now

modified

so

that

a

primary

ensures

any

message

sent

to

a

backup

is

sent

across

at

least

one

non-faulty

path

This

protocol

uses

-LRB-

f

-RRB-

servers

and

so

Theorem

is

tight

Furthermore

it

is

nonblocking

and

so

incurs

no

additional

delay

It

has

the

failover

time

ffi

o

for

arbitrarily

small

and

positive

o

and

so

Theorem

is

tight

Most

of

our

protocols

for

the

various

kinds

of

omission

failures

apply

translation

techniques

-LSB-

-RSB-

to

the

protocol

for

crash

failures

outlined

above

These

techniques

ensure

that

a

faulty

server

detects

its

own

failure

and

halts

thereby

translating

a

more

severe

failure

to

a

crash

failure

The

translations

of

-LSB-

-RSB-

assume

a

round-based

protocol

Since

our

crash

failure

protocol

is

not

round-based

we

must

modify

the

translations

so

that

a

server

can

send

and

receive

messages

at

any

time

rather

than

just

at

the

beginning

or

the

end

of

a

round

All

of

these

resulting

omission

-LCB-

failure

protocols

have

failover

time

ffi

o

and

thus

Theorems

and

are

tight

The

protocol

for

send-omission

failures

uses

f

servers

and

is

o

-LCB-

blocking

Furthermore

we

also

have

a

send-omission

protocol

for

f

that

is

ffi

-LCB-

blocking

Thus

Theorems

and

are

tight

The

protocol

for

general-omission

failures

also

uses

servers

and

is

o

-LCB-

blocking

and

so

Theorem

is

tight

and

Theorems

and

are

tight

for

general-omission

failures

as

well

We

have

not

been

able

to

determine

whether

Theorems

and

are

tight

Our

protocol

for

receive-omission

failures

uses

servers

whereas

the

lower

bound

in

Theorem

only

requires

n

c

We

have

constructed

protocols

for

n

f

and

n

f

but

are

unable

to

generalize

these

protocols

We

can

also

show

that

any

protocol

for

n

has

the

following

odd

property

there

is

at

least

one

run

of

the

protocol

in

which

a

non-faulty

primary

is

forced

to

relinquish

control

to

a

backup

that

is

faulty

However

the

protocol

for

n

f

is

ffi

-LCB-

blocking

and

so

Theorem

is

tight

Table

summarizes

all

of

our

results

failure

degree

of

amount

of

failover

model

replication

blocking

time

crash

n

f

f

ffi

crash

link

n

f

y

ffi

receive

omission

n

c

y

ffi

if

n

and

f

y

if

n

and

f

y

if

n

ffi

send

omission

n

f

ffi

if

f

if

f

ffi

general

omission

n

ffi

if

f

iff

ffi

Bound

not

known

to

be

tight

y

D

Table

Lower

Bounds

Discussion

We

give

a

precise

characterization

for

primary

-LCB-

backup

protocols

in

a

system

with

synchronized

clocks

and

bounded

message

delays

We

then

present

lower

bounds

on

the

degree

of

replication

the

blocking

time

and

the

failover

time

under

various

kinds

of

server

and

link

failures

We

finally

outline

a

set

of

primary

-LCB-

backup

protocols

that

show

which

of

our

lower

bounds

are

tight

We

now

briefly

compare

our

results

to

existing

primary

-LCB-

backup

protocols

The

protocol

presented

in

-LSB-

-RSB-

tolerates

one

crash

link

failure

by

using

only

two

servers

This

appears

to

contradict

Theorem

which

states

that

at

least

three

servers

are

required

to

tolerate

one

failure

However

the

protocol

in

-LSB-

-RSB-

assumes

that

there

are

two

links

between

the

two

servers

effectively

masking

a

single

link

failure

Hence

only

crash

failures

need

to

be

tolerated

and

this

can

be

accomplished

using

only

two

servers

-LRB-

Theorem

-RRB-

A

more

ambitious

primary

-LCB-

backup

protocol

is

presented

in

-LSB-

-RSB-

This

protocol

works

for

the

following

failure

model

-LRB-

quoted

from

-LSB-

-RSB-

-RRB-

The

network

may

lose

or

duplicate

messages

or

deliver

them

late

or

out

of

order

in

addition

it

may

partition

so

that

some

nodes

are

temporarily

unable

to

send

messages

to

some

other

nodes

As

is

usual

in

distributed

systems

we

assume

the

nodes

are

fail-stop

processors

and

the

network

delivers

only

uncorrupted

messages

This

failure

model

is

incomparable

with

those

in

the

hierarchy

we

presented

However

the

protocol

does

tolerate

general-omission

failures

and

has

optimal

degree

of

replication

for

general-omission

failures

as

it

uses

servers

In

Theorem

we

assumed

that

D

This

assumption

is

crucial

we

are

able

to

construct

a

two-server

primary

-LCB-

backup

protocol

tolerating

one

crash

link

failure

for

which

D

Recall

that

link

failures

are

masked

by

adding

redundant

paths

between

the

servers

Our

two-server

crash

link

protocol

essentially

uses

the

path

from

the

primary

to

the

backup

through

the

client

as

the

redundant

path

Thus

there

appears

to

be

a

tradeoff

between

the

degree

of

replication

and

the

time

it

takes

for

a

client

to

learn

that

there

is

a

new

primary

The

lower

bounds

on

failover

times

given

in

Section

assume

This

is

necessary

as

we

have

constructed

protocols

that

have

failover

times

smaller

than

the

lower

bounds

given

in

Section

and

these

protocols

do

not

satisfy

This

smaller

failover

time

is

achieved

at

a

cost

of

an

increased

variance

in

service

response

time

Finally

in

this

paper

we

have

attempted

to

give

a

characterization

of

primary

-LCB-

backup

that

is

broad

enough

to

include

most

synchronous

protocols

that

are

considered

to

be

instances

of

the

approach

There

are

protocols

however

that

are

incomparable

to

the

class

of

protocols

we

analyze

-LSB-

-RSB-

since

they

were

developed

for

an

asynchronous

setting

Such

protocols

can

not

be

cast

in

terms

of

implementing

a

-LRB-

k

-RRB-

-LCB-

bofo

server

for

finite

values

of

k

and

We

are

currently

studying

possible

characterizations

for

a

primary

-LCB-

backup

protocol

in

an

asynchronous

system

and

hope

to

extend

our

results

to

this

setting

Acknowledgements

We

would

like

to

thank

Lorenzo

Alvisi

Mike

Reiter

and

the

anonymous

conference

referees

for

their

helpful

comments

on

earlier

drafts

of

this

paper

References

-LSB-

-RSB-

P.A.

Alsberg

and

J.D.

Day

A

principle

for

resilient

sharing

of

distributed

resources

In

Proceedings

of

the

Second

International

Conference

on

Software

Engineering

pages

-LCB-

October

-LSB-

-RSB-

Ozalp

Babao

glu

and

Rog?erio

Drummond

Streets

of

Byzantium

Network

architectures

for

fast

reliable

broadcasts

IEEE

Transactions

on

Software

Engineering

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

J.F.

Barlett

A

nonstop

kernel

In

Proceedings

of

the

Eighth

ACM

Symposium

on

Operating

System

Principles

SIGOPS

Operating

System

Review

volume

pages

-LCB-

December

-LSB-

-RSB-

Anupam

Bhide

E.N.

Elnozahy

and

Stephen

P.

Morgan

A

highly

available

network

file

server

In

USENIX

pages

-LCB-

-LSB-

-RSB-

Kenneth

P.

Birman

and

Thomas

A.

Joseph

Exploiting

virtual

synchrony

in

distributed

systems

In

Eleventh

ACM

Symposium

on

Operating

System

Principles

pages

-LCB-

November

-LSB-

-RSB-

Navin

Budhiraja

Keith

Marzullo

Fred

Schneider

and

Sam

Toueg

Optimal

primary

-LCB-

backup

protocols

In

Proceedings

of

the

Sixth

International

Workshop

on

Distributed

Algorithms

Haifa

Israel

November

To

Appear

-LSB-

-RSB-

IBM

International

Technical

Support

Centers

IBM/VS

extended

recovery

facility

-LRB-

XRF

-RRB-

technical

reference

Technical

Report

IBM

-LSB-

-RSB-

Flaviu

Cristian

Synchronous

atomic

broadcast

for

redundant

broadcast

channels

Journal

of

Real-Time

Systems

-LCB-

September

-LSB-

-RSB-

Flaviu

Cristian

Houtan

Aghili

H.

Ray

Strong

and

Danny

Dolev

Atomic

broadcast

From

simple

message

diffusion

to

Byzantine

agreement

In

Proceedings

of

the

Fifteenth

International

Symposium

on

Fault-Tolerant

Computing

pages

-LCB-

Ann

Arbor

Michigan

June

A

revised

version

appears

as

IBM

Technical

Report

-LSB-

-RSB-

Vassos

Hadzilacos

Issues

of

Fault

Tolerance

in

Concurrent

Computations

PhD

thesis

Harvard

University

June

Department

of

Computer

Science

Technical

Report

-LSB-

-RSB-

Thomas

Joseph

and

Kenneth

Birman

Reliable

Broadcast

Protocols

pages

-LCB-

ACM

Press

New

York

-LSB-

-RSB-

Leslie

Lamport

Time

Clocks

and

the

Ordering

of

Events

in

a

Distributed

System

Communications

of

the

ACM

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

Leslie

Lamport

and

Michael

Fischer

Byzantine

generals

and

transaction

commit

protocols

Op.

SRI

International

April

-LSB-

-RSB-

Leslie

Lamport

and

P.

M.

Melliar-Smith

Synchronizing

clocks

in

the

presence

of

faults

Journal

of

the

ACM

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

Barbara

Liskov

Sanjay

Ghemawat

Robert

Gruber

Paul

Johnson

and

Michael

Williams

Replication

in

the

Harp

file

system

In

Proceedings

of

the

Symposium

on

Operating

System

Principles

pages

-LCB-

-LSB-

-RSB-

Timothy

Mann

Andy

Hisgen

and

Garret

Swart

An

algorithm

for

data

replication

Technical

Report

Digital

Systems

Research

Center

-LSB-

-RSB-

Gil

Neiger

and

Sam

Toueg

Automatically

increasing

the

fault-tolerance

of

distributed

systems

In

Proceedings

of

the

Seventh

ACM

Symposium

on

Principles

of

Distributed

Computing

pages

-LCB-

Toronto

Ontario

August

ACM

SIGOPS-SIGACT

-LSB-

-RSB-

B.

Oki

and

Barbara

Liskov

Viewstamped

replication

A

new

primary

copy

method

to

support

highly

available

distributed

systems

In

Seventh

ACM

Symposium

on

Principles

of

Distributed

Computing

pages

-LCB-

Toronto

Ontario

August

ACM

SIGOPS

SIGACT

-LSB-

-RSB-

M.

Pease

R.

Shostak

and

Leslie

Lamport

Reaching

agreement

in

the

presence

of

faults

Journal

of

the

ACM

-LRB-

-RRB-

-LCB-

April

-LSB-

-RSB-

Kenneth

J.

Perry

and

Sam

Toueg

Distributed

agreement

in

the

presence

of

processor

and

communication

faults

IEEE

Transactions

on

Software

Engineering

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

Richard

D.

Schlichting

and

Fred

B.

Schneider

Fail-stop

processors

an

approach

to

designing

fault-tolerant

computing

systems

ACM

Transactions

on

Computer

Systems

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

Fred

B.

Schneider

Implementing

fault

tolerant

services

using

the

state

machine

approach

A

tutorial

Computing

Surveys

-LRB-

-RRB-

-LCB-

December

availability
fault-tolerance
optimal protocols
primary{backup
reliability