DRAFT

Do

Not

Distribute

Reconfiguration

of

Internet

Servers

Ashish

Singhai

University

of

Illinois

Swee-Boon

Lim

and

Sanjay

R.

Radia

Sun

Microsystems

Inc.

y

Abstract

We

describe

the

design

implementation

and

performance

of

SUN-RAIS

-LRB-

Reliable

Array

of

Internet

Servers

-RRB-

in

the

context

of

a

distributed

service

monitoring

algorithm

SUN-RAIS

provides

highly

available

and

high

performance

internet

services

and

features

automatic

response

to

a

wide

variety

of

error

conditions

e.g.

machine

crash

message

loss

network

interface

failure

buffer

exhaustion

and

machine

and

network

overload

Moreover

high

availability

is

provided

in

an

unobstrusive

manner

without

modifying

either

the

applications

or

the

kernel

SUN-RAIS

uses

a

distributed

self

stabilizing

leader

election

algorithm

The

algorithm

does

not

require

a

priori

knowledge

of

the

number

of

participating

hosts

or

their

identities

It

does

not

require

a

group

membership

protocol

and

dynamically

learns

the

identities

of

the

participating

hosts

The

implementation

is

parameterized

to

allow

adaptation

to

network

and

host

characteristics

Upon

detection

of

a

failure

automatic

reconfiguration

takes

place

and

the

services

are

migrated

to

a

functional

host

The

system

can

be

transparently

scaled

vertically

by

upgrading

existing

machines

as

well

as

horizontally

by

adding

more

machines

We

present

simulation

results

to

demonstrate

the

robustness

of

the

system

Even

under

conditions

as

server

as

packet

loss

and

machine

failures

every

hour

single

address

unavailability

is

restricted

to

of

the

total

time

The

system

has

been

in

active

use

at

our

site

by

of

users

for

over

months

and

forms

the

basis

of

a

future

product

Introduction

Network

computing

systems

using

a

collection

of

off-the-shelf

machines

connected

by

a

high-bandwidth

network

offer

large

computational

power

at

better

price/performance

than

commercial

MPPs

This

is

true

because

of

the

economy

of

scale

-LSB-

-RSB-

Such

systems

can

use

existing

technology

-LRB-

as

opposed

to

developing

Contact

Author

This

work

was

performed

while

the

author

was

at

Sun

Microsystems

Author

's

Address

W.

Springfield

Avenue

Urbana

IL

E-mail:Ashish.Singhai@UIUC.EDU

yAddress

Garcia

Ave.

Mountain

View

CA

E-mail

fSwee.Lim,Sanjay.Radiag@Sun.COM

DRAFT

Do

Not

Distribute

new

technology

for

MPPs

-RRB-

and

have

potential

to

be

more

scalable

because

increasing

their

capacity

is

not

bounded

by

the

physical

law

Network

computing

systems

are

are

increasingly

being

used

in

the

domains

requiring

continuous

service

Some

examples

are

data

access

and

update

-LRB-

files

e-mail

database

records

-RRB-

information

retrieval

-LRB-

web

search

database

search

-RRB-

network

connectivity

-LRB-

provision

of

internet

service

-RRB-

and

disseminating

continuously

updated

data

-LRB-

news

stock

market

data

sports

results

-RRB-

Two

key

requirements

of

such

domains

are

reliability

for

providing

continuous

service

and

scalability

to

handle

varying

demand

However

when

such

systems

are

scaled

up

by

adding

more

machines

the

probability

of

a

component

failure

-LRB-

hardware

or

software

-RRB-

in

the

system

also

increases

For

increased

availability

such

systems

need

to

tolerate

individual

component

failures

In

this

paper

we

discuss

the

design

implementation

and

performance

of

which

is

an

adaptation

of

a

generic

distributed

monitoring

framework

It

provides

continuous

internet

based

services

We

also

present

experimental

results

and

show

that

our

approach

provides

very

high

availability

at

very

low

cost

The

outline

of

this

paper

is

as

follows

In

the

next

section

we

discuss

our

system

Sections

and

discuss

our

approach

to

high

availability

and

briefly

describe

the

design

and

implementation

of

the

service

monitor

framework

Implementation

of

the

SUN-RAIS

is

described

in

section

We

analyze

the

performance

of

the

system

in

the

presence

of

artificially

injected

faults

in

section

In

section

we

discuss

the

related

work

in

the

area

of

highly

available

services

Finally

section

concludes

this

paper

with

a

discussion

of

future

work

and

applicability

of

our

approach

to

other

problems

System

Model

The

techniques

discussed

in

this

paper

are

applicable

to

any

network

computing

system

For

concreteness

in

the

following

description

we

assume

a

LAN

environment

consisting

of

hosts

communicating

using

the

IP

protocol

suite

-LSB-

-RSB-

We

consider

a

set

of

hosts

connected

by

a

network

Our

current

setup

connects

hosts

by

fast

ethernet

These

hosts

provide

various

services

e.g.

web

news

e-mail

ftp

The

hosts

or

the

services

can

completely

or

partially

fail

at

any

time

In

addition

the

communication

may

be

unreliable

i.e.

network

may

drop

packets

We

do

not

require

bounded

communication

delay

or

synchronized

host

clocks

Finally

we

assume

all

failures

are

fail-stop

failures

-LSB-

-RSB-

i.e.

the

failed

components

do

not

behave

maliciously

The

hosts

belong

to

service

groups

A

service

group

provides

a

specific

service

e.g.

netnews

All

the

hosts

in

a

service

group

are

capable

of

providing

the

service

associated

with

that

group

The

hosts

need

not

be

dedicated

to

a

specific

service

and

can

provide

multiple

services

concurrently

The

hosts

are

identified

by

their

network

addresses

called

control

addresses

Services

are

also

identified

by

their

network

addresses

called

service

addresses

Service

groups

are

identified

by

their

logical

names

e.g.

The

clients

access

the

services

using

the

service

addresses

Clients

need

not

and

probably

stands

for

reliable

array

of

internet

servers

DRAFT

Do

Not

Distribute

Client

Network

Router

host-n

Figure

A

possible

system

configuration

will

not

have

access

to

the

control

addresses

Service

addresses

and

control

addresses

can

be

on

the

same

subnet

or

on

different

subnets

We

assume

that

each

host

is

capable

of

supporting

multiple

IP

addresses

This

can

be

achieved

either

by

having

multiple

network

interfaces

per

host

or

by

having

the

same

interface

support

multiple

addresses

Such

hosts

are

referred

to

as

multi-homed

hosts

-LRB-

which

are

not

routers

-RRB-

in

the

IP

host

requirements

document

-LSB-

-RSB-

A

possible

system

configuration

is

illustrated

in

the

figure

The

clients

and

the

services

are

conventional

off-the-shelf

products

using

standard

protocols

We

provide

high-reliability

in

an

unobtrusive

manner

and

do

not

require

application

modification

kernel

modification

or

proprietary

protocols

The

clients

and

the

services

need

not

use

any

special

hardware

or

libraries

and

need

not

be

vendor

specific

The

Service

Monitor

Framework

Design

Principles

We

design

a

framework

to

solve

the

generic

problem

of

monitoring

a

distributed

system

and

establishing

or

maintaining

the

desired

system

propertieswhen

the

system

state

changes

because

of

faults

or

repairs

We

design

our

framework

with

the

following

goals

in

mind

-LCB-

handling

component

failures

gracefully

flexibility

and

ease

of

maintenance

robustness

genericity

cost

effectiveness

and

use

of

standard

components

We

strive

to

provide

high

availability

in

the

presence

of

component

failures

As

long

as

at

least

one

functional

system

in

the

service

group

is

available

the

service

associated

with

that

service

group

is

available

to

its

clients

The

degradation

in

the

quality

of

service

-LRB-

e.g.

response

time

-RRB-

is

proportional

to

the

lost

capacity

We

would

also

like

to

note

that

while

we

use

reconfiguration

to

handle

the

problems

that

arise

due

to

transient

lack

of

capacity

it

is

not

possible

to

solve

the

problem

of

fundamentally

inadequate

system

capacity

within

a

software

framework

We

reduce

the

impact

of

this

problem

by

providing

easy

installation

of

new

hosts

without

disrupting

the

service

We

build

a

system

that

is

easy

to

maintain

we

do

not

require

that

the

number

of

the

participants

in

the

system

be

fixed

or

even

bounded

The

number

of

participating

hosts

can

change

dynamically

making

it

DRAFT

Do

Not

Distribute

easy

to

add

more

capacity

de-install

hosts

or

selectively

bring

hosts

down

for

upgrades

or

maintenance

Our

design

leads

to

a

flexible

system

in

which

the

hosts

need

not

be

identical

and

hosts

can

be

added

-LRB-

removed

-RRB-

simply

by

plugging

them

in

-LRB-

out

-RRB-

Similarly

to

avoid

interfering

with

the

network

architecture

we

do

not

restrict

the

participating

hosts

to

be

on

the

same

subnet

and

do

not

impose

any

addressing

convention

Our

system

is

robust

in

that

it

can

handle

a

variety

of

faults

including

interface

failures

link

failures

process

failures

and

machine

crashes

We

also

increase

the

robustness

by

making

minimal

assumptions

about

the

network

reliability

and

transmission

delay

host

speeds

and

host

clocks

The

self

stabilizing

algorithm

that

we

use

ensures

that

the

system

converges

to

a

desired

state

irrespective

of

the

number

type

and

frequency

of

perturbations

Our

design

is

symmetric

-LRB-

each

host

runs

the

same

algorithm

-RRB-

and

it

can

tolerate

up

to

-LRB-

n

-RRB-

simultaneous

host

crashes

for

an

n

host

system

As

a

result

if

the

availability

of

an

individual

host

is

A

we

achieve

the

overall

system

availability

of

-LRB-

A

-RRB-

n

In

contrast

the

popular

approach

to

providing

high-reliability

using

pairs

of

machines

-LSB-

-RSB-

has

availability

-LRB-

A

-RRB-

tolerates

only

one

server

crash

requires

that

both

the

hosts

in

the

pair

be

the

same

type

and

allows

only

paired

upgrades

These

restrictions

make

the

system

fragile

and

also

increase

the

cost

We

want

to

have

a

generic

solution

that

can

be

customized

to

address

different

problems

We

build

an

object

oriented

framework

consisting

of

independent

reusable

components

and

use

configuration

specification

and

dynamic

loading

to

enable

customization

The

framework

customization

consists

of

specifying

the

monitor

object

to

detect

faults

and

the

action

object

prescribing

the

corrective

action

In

the

following

sections

we

discuss

one

customization

example

-LRB-

IP

failover

-RRB-

in

detail

and

allude

to

the

others

briefly

We

minimize

cost

by

using

a

self-stabilizing

algorithm

which

is

essentially

non-masking

That

is

the

effects

of

faults

are

visible

to

the

outside

world

Making

faults

completely

transparent

-LRB-

referred

to

as

masking

fault

tolerance

-LSB-

-RSB-

-RRB-

requires

very

expensive

hardware

and

software

We

provide

high

availability

by

making

the

errors

extremely

short

lived

Another

measure

contributing

to

the

goal

of

cost

effectiveness

is

an

all

software

implementation

without

requiring

any

special

hardware

or

kernel

modifications

Finally

we

implement

our

system

using

standard

protocols

-LRB-

IP

multicast

SNMP

-RRB-

well

understood

and

proven

concepts

-LRB-

self-stabilization

-RRB-

and

standard

operating

system

features

This

offers

the

advantages

of

rapid

development

portability

and

inter-operability

with

other

systems

The

Service

Monitor

Framework

Implementation

The

service

monitor

monitors

the

status

of

each

host

in

the

service

group

and

takes

actions

towards

establishing

and/or

maintaining

some

consistency

predicate

For

example

in

a

load

monitoring

system

the

per

host

status

would

consist

of

the

load

on

the

host

and

the

consistency

predicate

would

be

The

load

is

evenly

distributed

across

all

hosts

We

use

the

I-am-alive

protocol

for

status

monitoring

The

concept

behind

this

protocol

is

very

simple

Every

host

transmits

periodic

messages

announcing

that

it

is

alive

If

no

such

message

is

received

from

a

host

DRAFT

Do

Not

Distribute

transmitThread

-LRB-

-RRB-

receiveThread

-LRB-

-RRB-

controlThread

-LRB-

-RRB-

loop

periodically

-LCB-

find

leader

from

DB

-RCB-

-RCB-

if

-LRB-

i

am

not

leader

-RRB-

continue

restore

predicate

-LRB-

DB

-RRB-

update

databse

-LRB-

DB

-RRB-

ServiceMonitor

P

consistency

predicate

DB

host

info

database

while

-LRB-

p

-LRB-

DB

-RRB-

is

false

-RRB-

-LCB-

loop

infinite

-LCB-

receive

hostInfo

merge

-LRB-

hostInfo

DB

-RRB-

-RCB-

loop

periodic

-LCB-

-RCB-

multicast

-LRB-

myInfo

-RRB-

get

myInfo

from

DB

Figure

Modified-OMT

diagram

for

the

ServiceMonitor

for

h

units

of

time

-LRB-

called

host

timeout

-RRB-

then

that

host

is

assumed

to

have

failed

The

general

properties

and

stabilization

proofs

for

this

algorithm

are

discussed

by

Arora

and

Paduska

-LSB-

-RSB-

If

the

identities

of

the

hosts

are

unique

and

can

be

totally

ordered

this

protocol

can

also

be

used

as

a

self-stabilizing

leader

election

protocol

by

declaring

say

the

host

with

smallest

identity

to

be

the

leader

Our

implementation

of

this

protocol

uses

the

host

IP

address

-LRB-

control

address

-RRB-

as

the

identity

guaranteeing

uniqueness

and

total

ordering

To

gain

execution

efficiency

we

also

use

IP

multicast

-LSB-

-RSB-

In

the

original

discussion

of

the

I-am-alive

protocol

the

messages

contain

only

an

integer

We

replace

it

by

host

status

information

The

nature

of

included

information

is

dependent

upon

the

service

being

monitored

As

a

result

of

these

modifications

we

have

periodic

transmissions

of

each

host

's

status

Each

host

merges

the

incoming

information

into

a

host

status

database

The

leader

analyzes

its

database

to

determine

the

inconsistencies

in

the

system

and

takes

corrective

actions

towards

establishing

the

consistency

predicate

Our

implementation

is

an

object

system

The

behavior

of

the

objects

is

customizable

through

the

use

of

a

configuration

description

Of

particular

interest

is

the

ServiceMonitor

class

which

abstracts

the

transmission

reception

and

analysis

of

status

information

Its

instances

have

three

active

objects

performing

transmission

reception

and

control

The

transmit

object

transmits

the

periodic

I-am-alive

messages

The

receiver

object

receives

the

information

and

merges

it

with

the

host

status

database

The

control

module

periodically

analyzes

the

information

and

takes

necessary

actions

to

restore

the

consistency

predicate

if

necessary

The

period

of

the

transmit

module

is

the

unit

time

and

the

control

module

runs

every

c

time

units

called

the

control

interval

The

framework

is

quite

general

because

it

neither

restricts

the

kind

of

information

being

collected

nor

prescribes

the

nature

of

corrective

actions

We

have

implemented

an

IP

failover

mechanism

that

monitors

various

internet

services

-LRB-

web

news

e-mail

-RRB-

and

instructs

functional

hosts

to

impersonate

the

IP

addresses

of

failed

hosts

in

order

to

continue

providing

these

services

We

also

use

the

framework

for

load

balancing

DRAFT

Do

Not

Distribute

In

that

case

the

collected

information

pertains

to

the

perceived

load

and

corrective

actions

move

the

load

to

lightly

loaded

machines

Other

applications

include

distributed

instrumentation

deadlock

detection

garbage

collection

network

routing

and

the

like

In

general

it

can

be

used

for

detecting

conditions

that

are

either

stable

-LRB-

stable

predicate

detection

-RRB-

or

are

slowly

varying

IP

Failover

Mechanism

In

this

section

we

describe

the

IP

failover

mechanism

in

the

context

of

the

service

monitor

framework

In

abstract

this

mechanism

provides

highly

available

IP

addresses

and

services

that

are

associated

with

them

e.g.

netnews

We

motivate

the

need

for

highly

available

IP

addresses

using

a

simple

example

Consider

a

web

browser

which

obtains

HTML

pages

from

a

server

If

the

browser

or

the

operating

system

caches

the

server

name

to

IP

address

mapping

then

a

server

crash

will

cause

the

browser

to

declare

the

server

to

be

unavailable

If

there

are

multiple

servers

with

the

same

content

the

user

will

have

to

restart

the

browser

or

reboot

the

host

in

order

to

access

the

On

the

other

hand

if

one

of

the

functionally

equivalent

server

could

impersonate

the

IP

address

of

the

failed

server

a

simple

reconnection

-LRB-

or

retry

-RRB-

would

be

able

to

facilitate

data

access

with

minimum

disruption

The

IP

failover

mechanism

provides

this

functionality

We

consider

the

hosts

to

be

the

servers

of

IP

addresses

and

use

serving

an

IP

address

to

mean

receiving

network

packets

coming

to

that

address

and

providing

specific

services

associated

with

that

address

Upon

detecting

a

failure

-LRB-

due

to

a

machine

crash

interface

failure

buffer

exhaustion

or

other

reasons

-RRB-

the

failover

mechanism

arranges

to

have

another

host

impersonate

the

IP

address

and

to

start

the

services

associated

with

it

In

the

context

of

the

service

monitor

framework

the

consistency

predicate

for

this

mechanism

is

each

service

address

must

be

served

by

exactly

one

host

The

status

information

of

a

host

consists

of

the

IP

address

-LRB-

es

-RRB-

that

it

is

serving

The

error

conditions

are

obtained

by

negating

the

consistency

predicate

i.e.

two

or

more

hosts

are

serving

the

same

IP

address

or

no

host

is

serving

one

of

the

IP

addresses

The

corrective

actions

ensure

that

the

consistency

predicate

is

satisfied

In

the

following

subsections

we

discuss

issues

related

to

detecting

and

responding

to

the

error

conditions

Speeding

up

Convergence

To

expedite

the

rate

of

convergence

of

the

algorithm

we

transmit

the

status

information

from

the

host

as

well

as

the

information

it

has

received

from

others

This

is

similar

to

vector

time-stamps

-LSB-

-RSB-

In

addition

to

expediting

the

convergence

in

the

presence

of

variable

message

delay

and

message

loss

it

can

also

handle

networks

that

are

not

broadcast

based

These

benefits

are

achieved

at

the

cost

of

increasing

the

message

size

are

assuming

that

some

means

to

get

to

the

new

server

exists

DRAFT

Do

Not

Distribute

from

O

-LRB-

n

-RRB-

to

O

-LRB-

-RRB-

for

an

n

host

system

Given

that

the

absolute

message

size

is

quite

small

the

increased

size

does

not

incur

much

overhead

We

call

these

messages

vector

messages

In

the

section

we

show

the

difference

in

performance

of

two

versions

of

the

system

one

has

each

host

transmitting

only

its

own

state

while

the

other

has

each

host

transmitting

its

own

as

well

as

others

state

The

resource

requirement

of

vector

messages

is

seen

to

be

minimal

At

the

same

time

we

witness

a

significant

improvement

in

the

time

taken

to

detect

and

eliminate

duplicate

addresses

Detecting

Failures

The

criterion

for

declaring

a

service

failed

is

that

it

is

not

accessible

to

its

clients

Since

we

are

interested

in

application

availability

the

fact

that

the

ServiceMonitor

is

running

and

is

able

to

communicate

with

its

peers

does

not

necessarily

imply

that

the

services

that

are

being

monitored

are

alive

or

that

they

are

accessible

to

the

clients

Hence

before

asserting

liveness

ServiceMonitor

determines

liveness

by

performing

application

dependent

tests

In

the

case

of

a

web

server

a

simple

test

may

be

opening

a

connection

with

the

server

and

fetching

a

dummy

document

More

sophisticated

tests

could

be

designed

to

test

various

features

of

monitored

services

that

are

considered

essential

Our

framework

allows

specification

of

a

list

of

tests

for

each

service

The

transmit

module

dynamically

loads

these

test

objects

and

executes

them

before

reporting

liveness

The

frequencies

of

these

tests

are

configurable

on

a

per

test

basis

Duplicate

Addresses

Since

we

are

dynamically

assigning

IP

addresses

to

hosts

there

is

a

possibility

that

two

different

hosts

believe

that

they

are

serving

the

same

address

This

happens

because

of

the

asynchronous

nature

of

the

network

For

example

the

communication

between

a

host

and

the

leader

may

be

interrupted

for

a

short

time

The

leader

notices

the

loss

of

communication

from

that

host

and

re-assigns

its

service

addresses

to

some

other

host

As

a

result

there

are

two

hosts

serving

the

same

address

We

note

that

it

is

impossible

to

completely

eliminate

such

problems

in

asynchronous

networks

-LSB-

-RSB-

With

judicious

choice

of

timeouts

and

by

using

vector

messages

we

reduce

the

occurrences

of

such

events

Each

host

sends

periodic

ARP

resolve

messages

to

flush

stale

IP

to

ethernet

address

mappings

from

other

hosts

ARP

caches

We

also

benefit

from

the

fact

that

the

router

is

an

arbitrator

in

our

case

remembering

only

the

last

association

of

IP

address

with

a

physical

address

that

it

has

seen

-LRB-

Our

system

architecture

resembles

VERIFY

figure

in

section

-RRB-

So

even

if

two

or

more

hosts

believe

that

they

are

serving

the

same

address

messages

destined

for

that

address

are

consistently

delivered

to

the

same

host

mitigating

the

effects

of

this

error

In

section

we

show

that

the

average

time

for

which

duplicate

addresses

persist

in

the

system

is

quite

small

even

when

the

network

is

dropping

half

of

the

messages

Our

experience

with

actually

using

the

system

suggests

that

transient

duplicate

addresses

do

not

hinder

the

service

availability

DRAFT

Do

Not

Distribute

Interface

Failures

While

detecting

machine

or

process

failures

is

relatively

easier

detecting

network

interface

failures

presents

numerous

challenges

If

an

interface

completely

fails

the

consequences

are

not

catastrophic

because

the

corresponding

host

is

isolated

from

the

network

However

an

interface

can

fail

in

such

as

way

that

it

is

either

unable

to

receive

or

to

transmit

but

can

perform

the

other

function

well

If

the

interface

can

only

receive

messages

it

is

reduced

to

the

function

of

an

observer

On

the

other

hand

if

it

can

not

receive

but

can

transmit

messages

we

get

a

situation

when

the

host

thinks

that

all

other

machines

in

the

network

are

dead

-LRB-

because

it

is

not

hearing

from

anyone

-RRB-

and

tries

to

acquire

all

service

addresses

Because

it

can

transmit

it

succeeds

in

communicating

its

intentions

to

the

router

and

diverting

all

the

traffic

to

itself

This

is

a

potentially

disastrous

situation

To

handle

this

case

we

incorporate

extra

provisions

in

the

algorithm

If

a

host

believes

itself

to

be

the

only

up

machine

it

uses

the

ICMP

echo

command

to

ping

the

router

If

it

does

not

get

a

response

it

assumes

that

it

is

unable

to

receive

messages

and

shuts

itself

down

after

sending

out

a

notification

This

discussion

points

out

that

the

need

for

testing

the

interfaces

before

reporting

that

the

host

is

alive

Other

tests

such

as

a

dummy

connection

to

the

server

may

pass

with

a

failed

interface

if

the

messages

to

the

local

host

are

looped

back

by

the

network

layer

Another

implication

is

that

it

is

preferable

to

use

the

same

interface

for

control

as

well

as

for

service

messages

If

different

interfaces

are

used

the

case

of

service

interface

failure

may

go

undetected

if

the

control

interface

is

functional

Automatic

Installation

and

De-installation

Most

of

the

failover

mechanisms

that

we

found

have

a

static

configuration

and

need

to

know

the

identities

of

the

participating

hosts

in

advance

This

complicates

the

maintenance

and

upgrade

procedures

Adding

another

machine

potentially

requires

bringing

down

the

entire

system

-LRB-

to

update

the

configuration

-RRB-

This

is

obviously

contrary

to

the

goal

of

high

availability

Our

framework

does

not

require

a

priori

knowledge

of

the

number

of

participants

in

the

service

group

and

dynamically

infers

the

participants

identities

We

implement

this

feature

by

using

IP

multicast

-LSB-

-RSB-

Any

host

sending

messages

on

a

pre-determined

multicast

address

becomes

a

participant

allowing

new

machines

to

be

added

by

simply

plugging

them

in

However

the

de-installation

scenario

is

more

complicated

because

absence

of

a

host

is

interpreted

by

the

rest

of

the

members

of

the

service

group

as

a

failure

As

a

result

the

remaining

machines

try

to

keep

serving

the

missing

IP

address

-LRB-

es

-RRB-

Since

the

intention

is

to

take

the

de-installed

machine

out

of

service

it

is

undesirable

to

keep

serving

the

missing

IP

address

-LRB-

es

-RRB-

We

perform

de-installation

by

associating

a

configurable

time

out

value

with

the

service

addresses

If

the

machine

that

was

originally

intended

to

serve

a

given

service

address

does

not

reclaim

it

within

the

time

out

period

it

is

deconfigured

DRAFT

Do

Not

Distribute

Load

Balancing

We

achieve

load

balancing

in

two

ways

The

first

is

statistical

multiplexing

offered

by

DNS

by

rotation

of

DNS

entires

after

each

resolve

The

other

is

by

zone

modification

The

host

status

contains

two

metrics

load

REFERENCE

and

capacity

These

metrics

are

specific

to

the

service

For

example

for

a

CPU

intensive

service

they

may

represent

CPU

load

and

capacity

in

MIPS

For

a

network

intensive

service

they

may

represent

bandwidth

comsumption

and

maximum

bandwidth

When

the

utilization

increases

above

a

configurable

high-water

mark

the

host

withdraws

itself

from

the

DNS

zone

i.e.

stops

advertizing

itself

as

a

service

provider

This

initiates

a

shift

of

incoming

requests

to

other

machines

Finally

when

the

load

reduces

below

a

low-water

mark

the

host

adds

itself

back

in

the

DNS

zone

Preferred

Addresses

and

Hot

Spares

For

ease

of

administration

each

host

has

a

preferred

service

address

If

the

host

is

functional

it

is

assigned

its

preferred

address

To

handle

the

loss

of

capacity

because

of

failing

hosts

we

have

provisions

for

including

hot

spares

They

participate

in

the

algorithm

but

do

not

have

a

preferred

address

and

under

normal

circumstances

do

not

provide

the

service

When

a

new

host

is

selected

to

take

over

an

address

hot

spares

are

the

natural

candidates

because

they

are

the

least

loaded

hosts

Performance

Analysis

In

this

section

we

present

the

performance

of

our

system

and

analyze

the

factors

affecting

it

We

conducted

many

different

experiments

Each

experiment

consisted

of

running

ServiceMonitor

objects

in

a

service

group

with

artificially

injected

faults

The

experimental

setup

consisted

of

Sun

Sparcstation

and

Sun

workstations

connected

by

Mb/s

fast

ethernet

running

the

SunOS

operating

system

The

transmit

interval

-LRB-

unit

time

-RRB-

was

sec

and

the

host

status

was

logged

every

second

Except

for

the

one

described

in

subsection

all

the

experiments

ran

for

hours

leading

to

samples

The

independent

variables

in

the

experiments

were

Control

interval

-LRB-

c

-RRB-

Host

timeout

-LRB-

h

-RRB-

Packet

drop

function

Host

failure

rate

The

dependents

variables

were

lifetime

of

duplicate

addresses

td

time

for

which

a

service

address

is

not

available

tb

-LRB-

in

the

event

of

a

failure

-RRB-

DRAFT

Do

Not

Distribute

h

h

h

h

c

c

c

c

c

average

duplicate

lifetime

host

timeout

-LRB-

sec

-RRB-

ctrl

ctrl

ctrl

ctrl

ctrl

Figure

Average

Duplicate

Lifetime

td

-LRB-

as

a

fraction

of

the

total

running

time

-RRB-

for

packet

drop

rate

no

failures

We

follow

the

experimental

methodology

of

varying

one

independent

variable

while

keeping

the

others

constant

and

observing

its

effect

on

the

dependent

variables

Finally

we

also

conducted

an

experiment

with

a

specific

combination

of

all

the

variables

to

witness

the

overall

system

performance

Druschel

and

Banga

-LSB-

-RSB-

have

discussed

the

packet

loss

characteristics

for

internet

servers

They

have

found

that

the

packet

loss

as

seen

at

the

application

level

is

bursty

and

can

reach

very

high

proportions

In

the

extreme

case

of

server-livelock

the

application

loses

all

the

packets

Accordingly

we

have

designed

our

experiments

to

measure

the

performance

under

such

conditions

The

leader

election

algorithm

stabilizes

in

-LRB-

h

-RRB-

c

ffi

where

ffi

is

the

communication

latency

-LSB-

-RSB-

-LRB-

the

diameter

of

the

network

is

in

our

case

-RRB-

A

large

stabilization

interval

implies

large

failure

detection

latency

i.e.

service

address

non-availability

-LRB-

tb

-RRB-

Reducing

the

stabilization

interval

requires

reducing

both

h

and

c

However

a

small

h

gives

rise

to

false

failure

detections

This

happens

because

the

leader

may

lose

some

incoming

messages

and

-LRB-

wrongly

-RRB-

conclude

that

the

sending

host

has

failed

reassigning

the

service

to

another

host

This

leads

to

two

hosts

providing

the

same

service

violating

the

consistency

criterion

stated

in

section

Reducing

c

increases

the

load

on

the

host

because

of

more

frequent

execution

of

the

control

module

Hence

we

are

interested

in

an

optimal

combination

of

parameters

that

minimizes

both

the

service

address

non-availability

and

duplicated

services

for

given

system

-LRB-

host

and

network

-RRB-

characteristics

with

minimal

overhead

In

the

following

subsections

we

present

our

observations

Lifetime

of

Duplicate

Addresses

As

discussed

above

existence

of

more

than

one

host

providing

the

same

service

is

an

error

condition

that

is

triggered

by

lost

messages

To

determine

the

effect

of

c

and

h

on

td

-LRB-

the

average

time

for

which

duplicate

DRAFT

Do

Not

Distribute

h

h

h

h

c

c

c

c

c

average

duplicate

lifetime

host

timeout

-LRB-

sec

-RRB-

ctrl

ctrl

ctrl

ctrl

ctrl

Figure

Average

Duplicate

Lifetime

td

-LRB-

as

a

fraction

of

the

total

running

time

-RRB-

for

packet

drop

rate

no

failures

addresses

exist

-RRB-

we

varied

c

from

sec

through

sec

in

steps

of

sec

and

h

from

sec

through

sec

in

steps

of

sec

We

considered

the

receive

packet

drop

functions

of

the

form

runm?n

which

drop

packets

in

bursts

of

m

every

n

packets

Starting

point

of

the

burst

is

uniformly

distributed

in

-LSB-

m

n

-RRB-

The

experiments

used

and

Figures

and

list

the

average

time

as

a

fraction

of

the

total

running

time

-LRB-

td

-RRB-

for

which

the

aforementioned

error

condition

persists

We

observe

that

td

shows

a

decreasing

trend

along

the

rows

-LRB-

in

the

direction

of

increasing

h

-RRB-

for

This

confirms

our

hypothesis

that

a

large

h

implies

more

reliable

failure

detection

td

shows

a

slow

decrease

along

the

columns

-LRB-

in

the

direction

of

increasing

c

-RRB-

for

A

larger

c

implies

increased

latency

of

duplicate

detection

-LRB-

larger

td

-RRB-

as

well

as

a

less

frequent

failure

detection

-LRB-

smaller

td

-RRB-

We

observe

that

the

latter

primarily

dominates

in

this

case

and

a

less

frequent

examination

of

state

leads

to

lesser

reassignments

-LRB-

i.e.

less

duplicates

are

created

-RRB-

For

the

case

of

drop

function

the

trends

are

not

so

clear

because

very

high

packet

drop

rate

-LRB-

-RRB-

continuously

perturbs

the

system

and

does

not

give

it

a

chance

to

fully

stabilize

In

spite

of

such

severe

conditions

td

is

quite

small

-LRB-

less

than

of

the

total

running

time

in

most

cases

-RRB-

From

the

above

observations

we

picked

c

and

h

which

is

in

the

stable

region

of

the

first

set

of

observations

and

experimented

with

a

different

packet

drop

function

randx

which

drops

any

received

packet

with

probability

x

We

varied

x

from

through

in

steps

of

There

were

no

duplicates

in

this

case

We

also

conducted

experiments

in

which

we

drop

transmitted

packets

using

the

function

randx

for

x

ranging

from

through

in

steps

of

In

this

case

also

there

were

no

duplicates

These

observations

demonstrate

the

robustness

of

our

approach

in

the

presence

of

lost

packets

DRAFT

Do

Not

Distribute

h

h

h

h

c

c

c

c

c

average

serivce

address

non-availability

-LRB-

sec

-RRB-

host

timeout

-LRB-

sec

-RRB-

ctrl

ctrl

ctrl

ctrl

ctrl

Figure

Average

service

address

not-availabe

time

per

failure

tb

-LRB-

in

seconds

-RRB-

no

packet

drops

Single

Service

Address

Non-availability

The

primary

reason

for

a

service

address

not

being

available

is

machine

failures

A

service

address

is

unavailable

from

the

time

the

host

serving

it

fails

till

the

time

the

address

is

reassigned

to

another

host

An

unfortunate

sequence

of

message

loss

may

also

lead

to

service

address

non-availability

For

example

if

a

reassignment

message

is

lost

the

service

address

remains

unavailable

until

the

leader

reassigns

it

again

We

would

like

to

emphasize

that

non-availability

of

a

single

service

address

is

not

the

same

as

the

non-availability

of

the

service

itself

because

only

a

fraction

of

the

user

population

is

affected

To

determine

the

effect

of

host

failures

we

simulated

machine

failures

by

running

the

program

from

within

a

wrapper

that

kills

and

restarts

the

program

periodically

with

a

specific

frequency

Figure

shows

the

average

service

downtime

in

seconds

per

failure

-LRB-

tb

-RRB-

when

all

machines

fail

every

-LSB-

-RRB-

minutes

and

restart

sec

after

failing

There

is

no

message

loss

in

this

case

The

length

of

this

experiment

was

min

The

table

confirms

our

qualitative

analysis

that

increasing

h

leads

to

increased

failure

detection

latency

and

increasing

c

leads

to

slower

response

to

failures

The

overall

effect

is

to

increase

tb

for

larger

c

and

h.

Overall

Performance

CPU

Usage

and

Effect

of

Vector

Messages

Finally

we

consider

the

compound

effect

of

combining

various

kinds

of

failures

at

the

same

time

Table

shows

our

observations

both

with

and

without

vector

messages

The

simulation

was

run

for

hours

on

machines

with

Sparc

-LRB-

MHz

CPU

-RRB-

and

MBytes

of

RAM

We

observe

that

under

the

conditions

of

burst

mode

message

loss

amounting

to

for

receive

and

random

transmit

packet

loss

amounting

to

the

unavailability

of

an

address

is

limited

to

be

less

than

as

a

fraction

of

total

time

even

when

each

machine

fails

approximately

once

every

hour

For

the

same

set

of

conditions

no

address

is

duplicated

for

more

than

of

the

total

time

Measurements

used

the

times

-LRB-

-RRB-

system

call

with

a

resolution

of

ms.

We

would

like

to

emphasize

that

our

system

is

extremely

lightweight

This

performance

is

achieved

at

DRAFT

Do

Not

Distribute

variable

with

without

Average

duplicate

lifetime

-LRB-

td

-RRB-

Maximum

duplicate

lifetime

-LRB-

tdmax

-RRB-

sec.

Average

unavailability

per

address

per

failure

sec.

Unavailability

per

address

as

a

fraction

of

total

time

Maximum

address

non-availability

-LRB-

tbmax

-RRB-

sec.

CPU

usage

-LRB-

system

mode

-RRB-

CPU

usage

-LRB-

user

mode

-RRB-

Table

Overall

Performance

with

and

without

vector

messages

The

parameters

are

c

h

recv

drop

transmit

drop

and

failures

every

-LSB-

-RRB-

minutes

the

cost

of

of

the

CPU

time

on

a

relatively

slow

CPU

-LRB-

MHz

-RRB-

for

an

unoptimized

debug

version

of

the

software

that

included

the

logging

overhead

In

the

same

table

we

also

observe

that

using

vector

messages

over

simple

messages

reduces

the

average

lifetime

of

duplicate

addresses

from

to

of

the

total

time

representing

a

reduction

of

about

while

consuming

only

more

CPU

time

The

overhead

is

minimal

because

for

a

small

number

of

participants

a

vector

message

fits

completely

in

an

ethernet

packet

As

a

result

the

number

of

network

packets

remains

constant

leading

to

no

extra

system-mode

overhead

and

neglegible

user-mode

overhead

Performance

Summary

The

results

presented

in

this

section

demonstrate

the

robustness

of

the

system

Under

extreme

circumstances

where

of

the

packets

were

being

dropped

by

the

network

the

duplicate

addresses

persisted

for

less

than

of

the

total

time

and

no

service

address

was

unavailable

for

more

than

of

the

total

time

The

CPU

overhead

is

about

on

a

rather

slow

-LRB-

MHz

-RRB-

CPU

for

a

sample

set

of

parameters

and

will

likely

decrease

for

faster

CPUs

e.g.

Sun

Ultrasparc

We

chose

to

do

the

CPU

time

measurements

on

a

slower

CPU

to

have

higher

accuracy

with

a

low

resolution

-LRB-

ms.

-RRB-

timer

We

also

note

that

our

experiments

simulated

rather

extreme

scenarios

In

the

field

the

conditions

are

expected

to

be

significantly

better

We

have

been

using

the

SUN-RAIS

system

for

our

web-proxy

and

news

servers

and

have

obtained

excellent

results

Related

Work

The

issue

of

self

reconfiguration

in

networks

has

been

previously

addressed

by

Autonet

-LSB-

-RSB-

by

Schroeder

et

al.

Autonet

is

a

local

area

network

consisting

of

switches

connected

by

point

to

point

links

It

deals

with

the

failures

at

the

level

of

the

network

layer

Similar

to

our

work

it

also

uses

a

distributed

algorithm

to

DRAFT

Do

Not

Distribute

gather

information

and

to

take

decisions

However

it

does

not

address

failures

in

higher

layers

e.g.

protocol

failures

or

host

failures

Our

focus

is

on

providing

continuous

service

to

the

clients

at

the

application

level

As

contrasted

with

Autonet

we

do

not

require

specialized

hardware

or

new

drivers

or

protocols

Druschel

and

Banga

-LSB-

-RSB-

discuss

the

load

on

network

servers

and

discuss

the

design

of

their

network

subsystem

architecture

to

address

issues

such

as

traffic

separation

accounting

and

early

packet

discard

They

provide

experimental

measurements

about

packet

loss

as

seen

by

the

applications

There

are

a

number

of

commercial

products

-LSB-

-RSB-

that

offer

highly

available

data

services

using

either

shared

or

mirrored

disk

configuration

Such

products

are

available

from

almost

all

major

hardware

and

software

vendors

e.g.

Sun

Silicon

Graphics

Hewlett-Packard

Tandem

Digital

IBM

Oracle

Informix

Sybase

Veritas

Almost

all

of

these

systems

have

two

hosts

monitoring

each

other

and

taking

over

the

responsibility

of

the

other

one

in

case

it

fails

Failure

detection

is

performed

using

a

heart-beat

mechanism

in

which

a

host

which

is

not

transmitting

periodic

pulses

is

assumed

to

have

failed

A

major

drawback

of

these

systems

is

the

inefficiency

of

the

pairwise

configuration

and

limited

reliability

These

systems

can

tolerate

only

one

failure

The

service

becomes

unavailable

if

both

hosts

in

a

pair

fail

even

if

other

hosts

are

available

If

the

reliability

of

a

host

is

A

then

the

service

availability

in

a

paired

system

is

-LRB-

A

-RRB-

Another

drawback

is

that

the

system

can

only

be

upgraded

by

adding

pairs

which

is

not

cost

efficient

Such

systems

are

suitable

for

data

update

services

using

a

dual

ported

disk

Another

shortcoming

of

these

products

is

that

they

focus

upon

machine

and

disk

crashes

but

do

not

consider

other

types

of

failures

Most

notable

omissions

are

network

interface

failures

and

protocol

errors

due

to

buffer

exhaustion

In

fact

the

use

of

redundant

networks

increases

the

chances

that

an

interface

failure

remains

unnoticed

Tandem

offers

a

commercial

product

-LSB-

-RSB-

that

provides

transparent

failover

for

applications

using

the

socket

library

Unfortunately

the

benefits

are

limited

to

the

applications

that

use

their

library

and

off-theshelf

clients

can

not

take

advantage

of

these

features

We

draw

upon

the

large

body

of

work

in

the

area

of

fault

tolerance

in

distributed

systems

Arora

and

Gouda

-LSB-

-RSB-

provide

a

uniform

framework

for

dealing

with

fault

tolerance

According

to

their

classification

our

work

is

characterized

as

non-masking

fault

tolerant

as

well

as

self-stabilizing

The

I-am-alive

protocol

its

proof

and

properties

have

been

discussed

by

Arora

and

Paduska

-LSB-

-RSB-

There

have

also

been

a

number

of

results

in

literature

that

deal

with

leader

election

-LSB-

-RSB-

We

differentiate

our

work

from

the

above

in

that

we

design

a

distributed

service

monitor

framework

that

can

be

used

to

implement

a

variety

of

highly

available

services

Our

object-oriented

design

allows

dynamic

customization

and

easy

extension

of

functionality

In

addition

we

also

provide

extensive

experimental

results

and

show

that

the

performance

of

our

system

is

very

good

under

very

severe

conditions

The

experimental

results

also

provide

guidelines

for

determining

the

correct

values

of

system

parameters

DRAFT

Do

Not

Distribute

Conclusion

and

Future

Work

We

have

discussed

the

design

algorithms

implementation

and

performance

of

a

distributed

service

monitor

framework

specialized

to

perform

IP

failover

Our

system

is

based

upon

well

established

concepts

of

self-stabilization

leader-election

network

address

impersonation

and

multicast

Our

system

is

quite

flexible

because

it

does

not

assume

the

existence

of

any

globally

shared

information

In

particular

we

do

not

require

that

the

number

or

addresses

of

participating

hosts

be

fixed

and

well

known

or

that

their

clocks

be

synchronized

The

system

is

fully

dynamic

and

adapts

to

the

existing

environmental

conditions

It

provides

for

easy

installation

and

de-installation

making

it

easy

to

deploy

it

in

the

field

with

minimum

operator

training

We

have

shown

through

extensive

performance

measurements

that

the

system

performs

very

well

under

very

severe

conditions

In

addition

we

experimentally

demonstrate

realization

of

various

trade-offs

and

determination

of

appropriate

parameter

values

Through

rigorous

experimentation

we

demonstrate

that

for

as

many

as

dropped

packets

the

single

service

address

unavailability

stays

within

Various

implementation

of

the

system

have

been

operational

at

our

site

for

over

eight

months

and

provide

highly

available

web

proxy

and

news

services

for

our

building

The

users

have

not

noticed

any

service

disruption

when

we

temporarily

turn

off

one

or

more

machines

add

more

machines

or

take

the

existing

machines

out

of

service

We

have

also

customized

it

to

provide

NFS

service

We

are

presently

incorporating

additional

services

in

our

system

We

plan

to

use

it

for

distributed

measurements

load

balancing

and

databases

In

addition

our

present

administrative

interface

is

a

custom

one

consisting

of

command

line

tools

to

control

the

system

We

are

in

the

process

of

integrating

it

with

an

SNMP

-LSB-

-RSB-

module

for

generic

system-wide

controllability

References

-LSB-

-RSB-

A.

Arora

and

M.

Gouda

Closure

and

convergence

A

foundation

of

fault-tolerant

computing

IEEE

Trans

on

Software

Engineering

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

A.

Arora

and

D.

Paduska

A

timing-based

schema

for

stabilizing

information

exchange

in

networks

In

Proc

Intl.

Conf

on

Computer

Networks

-LSB-

-RSB-

WilliamE

Baker

Robert

W.

Horst

David

P.

Sonnier

and

William

J.

Watson

A

flexible

servernet-based

fault-tolerant

architecture

In

Proc

Intl.

Symp

on

Fault-Tolerant

Computing

June

-LSB-

-RSB-

Michael

Barborak

Miroslaw

Malek

and

Anton

Dahbura

The

consensus

problem

in

fault-tolerant

computing

ACM

Computing

Surveys

-LRB-

-RRB-

-LCB-

June

-LSB-

-RSB-

R.

Braden

Requirements

for

Internet

Hosts

-LCB-

Communication

Layers

Network

Working

Group

Internet

Engineering

Task

Force

October

RFC

DRAFT

Do

Not

Distribute

-LSB-

-RSB-

J.

Case

M.

Fedor

M.

Schoffstall

and

J.

Davin

A

Simple

Network

Management

Protocol

-LRB-

SNMP

-RRB-

Network

Working

Group

Internet

Engineering

Task

Force

May

RFC

-LSB-

-RSB-

Digital

Equipment

Corporation

inFORM

-LRB-

-RRB-

Magazine

on

the

world

wide

web

May/June

URL

-LSB-

-RSB-

S.

Deering

Host

Extensions

for

IP

Multicasting

Network

Working

Group

Internet

Engineering

Task

Force

August

RFC

-LSB-

-RSB-

Peter

Druschel

and

Gaurav

Banga

Lazy

receiver

processing

-LRB-

LRP

-RRB-

A

network

subsystem

architecture

for

server

systems

In

Proc

USENIX

Symp

on

Operating

System

Design

and

Implementation

-LRB-

OSDI

-RRB-

volume

October

-LSB-

-RSB-

T.

Anderson

et

al.

A

case

for

networks

of

workstations

NOW

IEEE

Micro

February

-LSB-

-RSB-

M.J.

Fischer

N.A.

Lynch

and

M.

Merritt

Easy

impossibility

proofs

for

distributed

consensus

problems

Distributed

Computing

-LRB-

-RRB-

-LSB-

-RSB-

H.

Garcia-Molina

Elections

in

a

distributed

system

IEEE

Trans

Computers

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Silicon

Graphics

Inc.

Computing

around

the

clock

Keeping

the

enterprise

alive

Product

description

on

the

world

wide

web

URL

-LSB-

-RSB-

Veritas

Software

Inc.

High

availability

failover

management

software

for

client/server

environments

White

paper

on

the

world

wide

web

URL

wp.html

-LSB-

-RSB-

Information

Sciences

Institute

University

of

Southern

California

Internet

Protocol

September

RFC

-LSB-

-RSB-

Friedemann

Mattern

Virtual

time

and

global

states

of

distributed

systems

In

M.

Cosnard

et

al.

editor

Proc

Workshop

on

Parallel

and

Distributed

Algorithms

pages

-LCB-

North-Holland/Elsevier

-LSB-

-RSB-

D.

Peleg

Time-optimal

leader

election

in

general

networks

Journal

of

Parallel

and

Distributed

Computing

-LRB-

-RRB-

-LCB-

-LSB-

-RSB-

Richard

D.

Schlichting

and

Fred

B.

Schneider

Fail-stop

processors

An

approach

to

designing

faulttolerant

computing

systems

ACM

Trans

on

Computer

Systems

-LRB-

-RRB-

-LCB-

May

-LSB-

-RSB-

Michael

D.

Schroeder

Andrew

D.

Birrell

Michael

Burrows

Hal

Murray

Roger

M.

Needham

Thomas

Rodeheffer

Ed

Satterthwaite

and

Chuck

Thacker

Autonet

a

high-speed

self-configuring

local

area

network

using

point-to-point

links

IEEE

Journal

on

Selected

Areas

in

Communications

-LRB-

-RRB-

October

O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

customization
high availability
leader election
reconfiguration
scalability