FORSCHUNGSZENTRUM

J

ULICH

GmbH

Zentralinstitut

f

ur

Angewandte

Mathematik

J

ulich

Tel.

-LRB-

-RRB-

Interner

Bericht

Parallelizing

Iterative

Solvers

for

Sparse

Systems

of

Equations

and

Eigenproblems

on

Distributed

Memory

Machines

Achim

Basermann

Mai

-LRB-

Stand

-RRB-

Diese

Arbeit

ist

zur

Publikation

eingereicht

Parallelizing

Iterative

Solvers

for

Sparse

Systems

of

Equations

and

Eigenproblems

on

Distributed

Memory

Machines

A.

Basermann

Central

Institute

for

Applied

Mathematics

Research

Centre

J?ulich

GmbH

J?ulich

Germany

email

a.basermann@kfa-juelich.de

Abstract

For

the

analysis

and

solution

of

discretized

ordinary

or

partial

differential

equations

it

is

necessary

to

solve

systems

of

equations

or

eigenproblems

with

coefficient

matrices

of

different

sparsity

patterns

depending

on

the

discretization

method

In

many

cases

the

use

of

the

finite

element

method

-LRB-

FE

-RRB-

results

in

largely

unstructured

systems

of

equations

The

main

computational

cost

in

iterative

methods

for

solving

these

problems

consists

of

matrix-vector

products

and

vector-vector

operations

usually

the

main

work

in

each

iteration

is

the

computation

of

matrix-vector

products

When

iterative

solvers

are

parallelized

on

a

multiprocessor

system

with

distributed

memory

the

data

distribution

and

the

communication

scheme

-LCB-

depending

on

the

data

structures

used

for

sparse

matrices

-LCB-

are

of

the

greatest

importance

for

an

efficient

execution

Here

data

distribution

and

communication

schemes

are

presented

that

are

based

on

the

analysis

of

the

column

indices

of

the

non-zero

matrix

elements

Performance

tests

using

the

conjugate

gradient

method

-LRB-

CG

-RRB-

and

the

Lanczos

algorithm

for

the

symmetric

eigenproblem

were

carried

out

on

the

distributed

memory

systems

INTEL

and

PARAGON

XP/S

of

the

Research

Centre

J?ulich

with

sparse

matrices

from

FE

models

The

parallel

variants

of

the

algorithms

showed

good

scaling

behavior

for

matrices

with

very

different

sparsity

patterns

Introduction

For

the

analysis

and

solution

of

discretized

ordinary

or

partial

differential

equations

it

is

necessary

to

solve

systems

of

equations

or

eigenproblems

with

coefficient

matrices

of

different

sparsity

patterns

depending

on

the

discretization

method

In

many

cases

the

use

of

the

finite

element

method

-LRB-

FE

-RRB-

results

in

largely

unstructured

systems

of

equations

Sparse

eigenproblems

play

an

important

role

in

the

analysis

of

elastic

solids

and

structures

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

In

the

corresponding

FE

models

the

natural

frequencies

and

mode

shapes

of

free

vibration

are

determined

as

are

buckling

loads

and

modes

Another

class

of

problem

is

related

to

stability

analysis

e.g.

of

electrical

networks

Moreover

approximations

of

extreme

eigenvalues

are

useful

to

solve

sets

of

linear

equations

e.g.

for

the

determination

of

condition

numbers

of

symmetric

positive

definite

matrices

or

for

conjugate

gradient

methods

with

polynomial

preconditioning

-LSB-

-RSB-

The

main

computational

cost

in

iterative

methods

for

solving

linear

systems

and

eigenproblems

consists

of

matrix-vector

products

and

vector-vector

operations

usually

the

main

The

author

was

supported

by

the

EEC

ESPRIT

Basic

Research

Project

-LRB-

APPARC

-RRB-

and

by

the

Graduiertenkolleg

Informatik

und

Technik

RWTH

Aachen

Germany

work

in

each

iteration

is

the

computation

of

matrix-vector

products

The

access

to

the

vector

is

determined

by

the

sparsity

pattern

and

the

storage

scheme

of

the

matrix

When

iterative

solvers

are

parallelized

on

a

multiprocessor

system

with

distributed

memory

the

data

distribution

and

the

communication

scheme

-LCB-

depending

on

the

data

structures

used

for

sparse

matrices

-LCB-

are

of

the

greatest

importance

for

an

efficient

execution

In

this

context

we

investigate

different

reordering

strategies

of

the

sparse

matrix

to

reduce

waiting

times

by

the

overlapped

execution

of

computation

and

communication

Additionally

the

reverse

Cuthill-McKee

scheme

-LSB-

-RSB-

is

applied

to

diminish

the

bandwidth

of

the

matrix

Depending

on

the

sparsity

pattern

of

the

matrix

bandwidth

reduction

can

result

in

a

considerable

decrease

of

communication

The

data

distribution

and

the

communication

scheme

are

determined

before

the

execution

of

the

solver

by

preprocessing

the

symbolic

structure

of

the

sparse

matrix

and

they

both

are

exploited

in

each

iteration

The

schemes

can

be

reused

as

long

as

the

sparsity

pattern

of

the

matrix

-LRB-

which

is

determined

by

the

discretization

mesh

and

the

element

types

-RRB-

does

not

change

For

example

they

can

be

used

in

each

time

step

of

a

time

dependent

problem

or

in

each

iterative

step

of

a

nonlinear

problem

that

is

solved

by

linearization

In

this

report

we

present

data

distribution

and

communication

schemes

that

are

based

on

the

analysis

of

the

column

indices

of

the

non-zero

matrix

elements

Performance

tests

using

the

conjugate

gradient

algorithm

-LRB-

CG

-RRB-

with

preconditioning

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

to

solve

systems

of

equations

and

the

Lanczos

method

for

the

symmetric

eigenproblem

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

were

carried

out

on

the

distributed

memory

systems

INTEL

and

PARAGON

XP/S

of

the

Research

Centre

J?ulich

with

sparse

matrices

from

two

FE

models

The

first

FE

model

comes

from

environmental

science

it

simulates

the

behavior

of

pollutants

in

geological

systems

-LSB-

-RSB-

-LSB-

-RSB-

In

the

second

FE

model

coming

from

structural

mechanics

stresses

in

materials

induced

by

thermal

expansion

are

calculated

by

applying

the

FE

program

SMART

-LSB-

-RSB-

The

remainder

of

this

report

is

organized

as

follows

Section

briefly

describes

two

iterative

solvers

the

CG

method

and

the

Lanczos

method

for

the

symmetric

eigenproblem

In

section

a

conventional

storage

scheme

for

large

sparse

matrices

is

presented

For

the

parallelization

of

iterative

methods

on

a

distributed

memory

system

section

gives

a

detailed

description

of

data

distribution

and

communication

schemes

based

on

the

storage

technique

above

In

section

we

discuss

numerical

and

performance

results

of

parallel

CG

and

Lanczos

methods

on

the

and

PARAGON

XP/S

and

finally

section

is

devoted

to

concluding

remarks

Iterative

Solvers

In

many

applications

Krylov

subspace

methods

are

applied

to

solve

systems

of

linear

equations

or

eigenproblems

In

the

following

we

consider

some

variants

of

two

frequently

used

algorithms

the

method

of

conjugate

gradients

and

the

Lanczos

method

for

the

symmetric

eigenproblem

The

Method

of

Conjugate

Gradients

The

method

of

conjugate

gradients

-LSB-

-RSB-

is

an

algorithm

for

solving

systems

of

linear

equations

Ax

b

particularly

when

A

is

a

sparse

coefficient

matrix

The

method

applies

to

symmetric

positive

definite

matrices

A

IRn?n

In

Hestenes

and

Stiefel

developed

the

original

CG

method

that

is

described

in

Algorithm

In

each

iteration

the

vectors

xi

gi

and

di

are

computed

xi

approximates

the

solution

vector

gi

is

the

residual

and

di

determines

the

direction

in

which

the

next

approximation

of

the

solution

vector

is

sought

For

some

sparse

matrices

the

main

work

in

each

iteration

consists

solely

of

the

computation

of

the

matrix-vector

product

Adi

for

other

sparse

matrices

this

work

is

comparable

with

the

work

involved

in

the

computation

of

inner

products

and

saxpys

Iteration

is

continued

until

the

Euclidean

norm

of

the

residual

is

less

than

or

equal

to

fflr

Algorithm

The

original

CG

method

Choose

an

arbitrary

IRn

b

i

i

gTi

gi

dTi

Adi

xi

xi

idi

gi

gi

iAdi

ffii

gTi

gi

gTi

gi

di

gi

ffiidi

until

kgi

fflr

Another

stopping

criterion

that

uses

the

maximum

scaled

absolute

difference

of

the

components

of

the

solution

vector

's

latest

two

approximations

is

determined

as

follows

max

j

n

jxji

xji

j

jxji

j

jxji

j

ffls

-LRB-

-RRB-

In

Aykanat

et

al.

-LSB-

-RSB-

suggested

a

modified

CG

algorithm

which

has

better

parallelization

properties

than

the

original

method

Algorithm

The

modified

CG

method

Choose

an

arbitrary

IRn

b

i

i

gTi

gi

dTi

Adi

ffii

i

-LRB-

Adi

-RRB-

TAdi

dTi

Adi

gTi

gi

ffiigTi

gi

xi

xi

idi

gi

gi

iAdi

di

gi

ffiidi

until

kgi

fflr

The

main

difference

between

the

original

and

the

modified

algorithm

is

that

in

the

modified

version

all

dot

products

are

computed

without

any

operations

in

between

Therefore

if

each

iteration

is

performed

in

parallel

on

a

distributed

memory

system

the

local

values

of

the

dot

products

can

be

included

in

one

message

to

determine

the

global

values

Algorithm

was

shown

to

be

less

robust

than

Algorithm

for

some

test

matrices

This

disadvantage

can

be

avoided

by

computing

the

dot

product

gTi

gi

in

each

iteration

and

not

using

the

value

of

ffii

from

the

previous

iteration

This

results

in

an

additional

dot

product

at

the

beginning

of

each

iteration

but

does

not

affect

the

more

advantageous

parallelization

properties

of

Algorithm

compared

with

Algorithm

With

that

modification

Algorithm

shows

the

same

robustness

as

Algorithm

for

all

matrices

tested

In

the

investigations

Algorithms

and

were

performed

with

and

without

diagonal

scaling

-LSB-

-RSB-

a

simple

preconditioner

which

hardly

contributes

to

the

total

execution

time

but

usually

accelerates

the

convergence

considerably

The

Lanczos

algorithm

Let

A

be

an

n

n

real

symmetric

matrix

with

eigensolutions

-LRB-

i

zi

-RRB-

i

n

i.e.

Azi

izi

i

n

For

solving

real

symmetric

eigenproblems

Lanczos

methods

are

most

commonly

used

to

approximate

a

small

number

of

extreme

eigenvalues

and

eigenvectors

of

large

sparse

matrices

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

Starting

with

the

original

sparse

matrix

a

sequence

of

symmetric

tridiagonal

matrices

is

generated

the

eigenvalues

of

the

tridiagonal

matrices

approximate

the

eigenvalues

of

the

original

matrix

Tridiagonalization

The

following

shows

the

variant

of

the

Lanczos

tridiagonalization

suggested

in

-LSB-

-RSB-

Algorithm

The

Lanczos

tridiagonalization

Choose

an

arbitrary

with

and

set

i

i

qTi

Aqi

ri

Aqi

iqi

iqi

i

qi

ri

i

In

this

method

a

vector

sequence

fqigi

with

qi

IRn

and

a

sequence

of

i

i

symmetric

tridiagonal

matrices

Ti

i

are

generated

by

an

iterative

process

starting

with

an

n

n

real

symmetric

matrix

A

and

an

initial

vector

The

orthonormal

vectors

qi

i

are

called

the

Lanczos

vectors

and

the

symmetric

tridiagonal

matrices

Ti

the

Lanczos

matrices

The

matrices

Ti

have

the

following

form

with

m

m

i

and

m

m

i

as

the

diagonal

and

bidiagonal

elements

respectively

Ti

BBBBBB

i

i

i

i

i

CCCCCCA

In

Algorithm

-LRB-

rTi

ri

-RRB-

denotes

the

Euclidean

norm

of

the

residual

vector

ri

IRn

The

main

work

in

each

iteration

is

the

computation

of

the

matrix-vector

product

Aqi

and

in

some

situations

also

the

vector-vector

operations

Because

of

rounding

errors

the

Lanczos

vectors

qi

IRn

i

loose

their

mutual

orthogonality

as

the

number

of

steps

increases

One

way

to

obtain

good

numerical

accuracy

is

to

augment

the

algorithm

with

an

expensive

reorthogonalization

step

-LSB-

-RSB-

-LSB-

-RSB-

However

the

original

Lanczos

method

if

permitted

to

continue

long

enough

is

usually

able

to

produce

very

accurate

approximations

to

all

of

the

distinct

eigenvalues

of

A

-LSB-

-RSB-

-LSB-

-RSB-

That

means

finite-precision

arithmetic

delays

but

it

does

not

prevent

the

determination

of

the

eigenvalues

Therefore

we

do

not

consider

reorthogonalization

in

this

report

The

principle

of

the

modified

Lanczos

tridiagonalization

from

-LSB-

-RSB-

is

described

in

the

following

algorithm

This

variant

shows

more

advantageous

parallelization

properties

than

Algorithm

Algorithm

The

modified

Lanczos

tridiagonalization

Choose

an

arbitrary

with

and

set

i

i

kri

i

rTi

rTi

qi

ri

i

ri

Ari

i

iqi

iqi

The

main

difference

between

Algorithm

and

Algorithm

is

again

that

in

the

modified

version

all

dot

products

are

computed

without

any

operations

in

between

For

the

matrices

tested

no

difference

with

respect

to

robustness

was

found

between

the

two

variants

Tridiagonal

solver

After

a

certain

number

of

steps

of

the

Lanczos

tridiagonalization

the

eigenvalues

of

the

current

tridiagonal

matrix

are

computed

to

decide

which

of

those

are

good

approximations

of

eigenvalues

of

the

original

matrix

A

This

is

repeated

until

a

prespecified

number

of

eigenvalues

of

A

are

approximated

to

a

given

accuracy

The

eigenvalues

of

the

tridiagonal

matrices

are

determined

by

a

bisection

method

based

on

the

parallel

algorithm

ALLEV

-LRB-

ALL

EigenValues

-RRB-

-LSB-

-RSB-

that

uses

the

Sturm

sequence

First

the

eigenvalues

are

isolated

in

intervals

and

then

extracted

to

a

predefined

accuracy

by

a

superlinearly

convergent

zero

finder

the

Pegasus

method

ALLEV

applies

a

parallelization

strategy

over

intervals

For

use

in

the

Lanczos

method

we

modified

the

algorithm

to

find

all

eigenvalues

in

a

given

interval

Moreover

we

integrated

criteria

to

decide

which

eigenvalues

of

the

current

tridiagonal

matrix

are

good

approximations

of

eigenvalues

of

A

The

following

error

estimate

holds

for

the

eigenvalue

j

of

Tk

and

the

eigenvalue

i

of

A

-LSB-

-RSB-

-LSB-

-RSB-

j?j

ij

k

jykj

j

ykj

denotes

the

kth

component

of

the

eigenvector

yj

of

the

eigenpair

-LRB-

j

yj

-RRB-

The

developed

parallel

Lanczos

method

LANSP

-LRB-

LANczos

SParse

-RRB-

accepts

an

eigenvalue

j

of

Tk

as

being

sufficiently

accurate

if

k

jykj

j

ffltj?j

j

-LRB-

-RRB-

for

a

given

tolerance

fflt

However

the

validity

of

-LRB-

-RRB-

only

needs

to

be

checked

for

a

certain

subset

of

the

eigenvalues

of

Tk

-LSB-

-RSB-

Therefore

the

eigenvalues

of

Tk

are

classified

the

classes

are

marked

by

a

corresponding

MP

value

as

follows

-LSB-

-RSB-

-LSB-

-RSB-

Spurious

eigenvalues

-LRB-

MP

-RRB-

Let

Sk

denote

the

tridiagonal

matrix

obtained

by

deleting

the

first

row

and

column

of

Tk

An

eigenvalue

of

Tk

is

defined

to

be

spurious

if

it

is

a

numerically

simple

eigenvalue

of

Tk

and

also

an

eigenvalue

of

Sk

A

spurious

eigenvalue

is

often

a

poor

approximation

to

every

eigenvalue

of

A.

Multiple

eigenvalues

-LRB-

MP

-RRB-

Clusters

of

eigenvalues

i.e.

eigenvalues

that

are

extremely

close

are

assumed

a

priori

to

be

accurate

approximations

to

eigenvalues

of

A.

Non-isolated

simple

eigenvalues

-LRB-

MP

-RRB-

If

any

of

the

remaining

eigenvalues

is

close

to

a

spurious

eigenvalue

-LRB-

close

here

is

with

respect

to

a

larger

tolerance

than

that

used

before

-RRB-

then

that

eigenvalue

is

marked

by

and

assumed

to

be

accurate

Simple

eigenvalues

-LRB-

MP

-RRB-

The

remaining

eigenvalues

and

only

these

are

tested

for

accuracy

according

to

criterion

-LRB-

-RRB-

For

each

such

eigenvalue

the

corresponding

eigenvector

of

Tk

is

computed

using

the

LAPACK

routine

DSTEIN

-LSB-

-RSB-

The

parallel

algorithm

LANSP

provides

the

following

options

Determining

all

eigenvalues

Determining

all

eigenvalues

in

a

given

interval

Determining

a

specific

number

of

eigenvalues

Determining

a

specific

number

of

eigenvalues

in

a

given

interval

The

current

version

of

LANSP

does

not

compute

the

corresponding

eigenvectors

However

this

is

possible

with

only

slight

modifications

because

the

computation

of

the

eigenvectors

of

the

tridiagonal

matrices

is

integrated

For

any

eigenvector

yj

of

Tk

uj

kX

i

yij

qi

is

an

approximation

of

an

eigenvector

of

the

original

matrix

A

i.e.

the

eigenpair

-LRB-

j

uj

-RRB-

approximates

the

eigenpair

-LRB-

i

zi

-RRB-

of

A

The

corresponding

residual

vector

is

rj

Auj

juj

For

the

determination

of

approximations

of

the

eigenvectors

of

A

the

Lanczos

vectors

qi

must

be

either

accumulated

in

secondary

storage

or

recomputed

In

the

latter

case

the

values

of

the

i

and

i

stored

as

elements

of

the

tridiagonal

matrices

can

be

reused

Storage

Schemes

Storage

schemes

for

large

sparse

matrices

depend

on

the

sparsity

pattern

of

the

matrix

the

considered

algorithm

and

the

architecture

of

the

computer

system

used

In

the

literature

many

variants

of

storage

schemes

can

be

found

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

-LSB-

-RSB-

The

storage

scheme

we

consider

here

is

often

used

in

FE

programs

and

is

suitable

for

regular

as

well

as

for

irregular

discretization

meshes

It

can

be

found

in

a

similar

form

in

-LSB-

-RSB-

for

example

The

scheme

is

illustrated

in

-LRB-

-RRB-

for

matrix

-LRB-

-RRB-

The

non-zeros

of

matrix

-LRB-

-RRB-

are

stored

row-wise

in

three

one-dimensional

arrays

aw

contains

the

values

of

the

non-zeros

as

the

corresponding

column

indices

In

az

the

position

of

the

beginning

of

each

row

in

aw

and

as

is

stored

The

subdivisions

in

aw

and

as

were

added

to

mark

the

beginning

of

a

new

row

The

order

of

the

matrix

elements

per

row

in

aw

and

as

is

different

from

that

in

matrix

-LRB-

-RRB-

since

this

is

usually

the

case

in

FE

programs

due

to

the

assembly

of

the

coefficient

matrix

from

single

elements

A

BBBBBBBBBBBB

CCCCCCCCCCCCA

-LRB-

-RRB-

aw

-LRB-

j

j

j

j

j

j

j

-RRB-

as

-LRB-

j

j

j

j

j

j

j

-RRB-

-LRB-

-RRB-

az

-LRB-

-RRB-

Parallelization

For

the

parallelization

of

iterative

solvers

on

a

distributed

memory

system

it

is

essential

to

distribute

the

matrix

and

vector

arrays

suitably

to

each

processor

and

to

determine

an

efficient

communication

scheme

for

matrix-vector

products

depending

on

the

sparsity

pattern

of

the

matrix

Data

Distribution

In

the

data

distribution

schemes

we

consider

below

the

matrix

arrays

aw

and

as

are

distributed

row-wise

the

rows

of

each

processor

succeed

one

another

The

distribution

of

the

vector

arrays

corresponds

component-wise

to

the

row

distribution

of

the

matrix

arrays

Criteria

for

the

data

distribution

can

be

Each

processor

gets

the

same

number

of

rows

or

enough

rows

to

ensure

that

each

processor

has

nearly

the

same

number

of

non-zeros

The

number

of

operations

for

the

computation

of

the

matrix-vector

product

is

proportional

to

the

number

of

non-zeros

the

remaining

vector

operations

of

one

iteration

are

proportional

to

the

number

of

rows

The

criterion

we

consider

here

is

that

each

processor

has

to

compute

nearly

the

same

number

of

operations

If

the

discretization

mesh

is

regular

i.e.

the

sparsity

pattern

of

the

coefficient

matrix

is

regular

then

all

three

criteria

result

in

nearly

the

same

data

distribution

If

the

mesh

is

very

irregular

the

three

distributions

differ

considerably

Our

algorithm

for

distributing

the

rows

onto

the

processors

can

be

described

as

follows

The

row

distribution

is

determined

by

analyzing

the

array

az

Let

nk

denote

the

number

of

rows

assigned

to

processor

k

and

let

p

denote

the

number

of

processors

Also

let

ek

denote

the

number

of

non-zeros

assigned

to

processor

k

and

let

e

denote

the

total

number

of

non-zeros

in

the

matrix

Then

processor

k

is

assigned

rows

until

the

following

requirement

is

satisfied

for

the

first

time

ek

nk

e

n

p

for

ek

nk

AE

-LRB-

-RRB-

First

the

parameter

depends

on

the

number

of

vector

operations

that

are

additional

to

the

operations

of

the

matrix-vector

product

in

each

iteration

Secondly

it

considers

the

execution

times

of

arithmetical

logical

and

memory

operations

on

the

processor

used

it

is

therefore

dependent

on

the

processor

architecture

The

numerator

in

-LRB-

-RRB-

is

proportional

to

the

number

of

operations

of

one

partial

iteration

on

processor

k

the

denominator

is

proportional

to

the

total

number

of

operations

of

one

iteration

It

should

be

noted

that

for

each

processor

gets

nearly

the

same

number

of

non-zeros

and

for

nearly

the

same

number

of

rows

The

first

case

means

that

the

execution

time

of

all

vector-vector

operations

is

negligible

compared

with

the

execution

time

of

the

matrix-vector

product

In

the

second

case

the

execution

time

of

the

matrix-vector

product

hardly

contributes

to

the

total

execution

time

With

these

considerations

the

contribution

of

the

matrix-vector

product

to

one

iteration

can

be

approximated

by

aMVP

ss

e

e

n

mz

for

e

n

AE

-LRB-

-RRB-

Here

mz

e

n

is

the

mean

number

of

non-zeros

per

row

Additionally

-LRB-

-RRB-

provides

a

means

for

measuring

If

aMVP

is

determined

by

timings

then

an

approximation

of

can

be

computed

by

ss

mz

aMVP

On

the

INTEL

XR

or

XP

the

timings

result

in

an

approximative

value

of

about

for

both

variants

of

the

CG

method

and

of

about

for

both

variants

of

the

Lanczos

tridiagonalization

The

data

distribution

according

to

criterion

-LRB-

-RRB-

is

shown

in

-LRB-

-RRB-

by

distributing

matrix

-LRB-

-RRB-

to

four

processors

The

other

arrays

are

distributed

analogously

In

this

small

example

the

data

distribution

is

the

same

for

both

the

CG

method

and

the

Lanczos

tridiagonalization

For

large

sparse

matrices

from

FE

applications

the

data

distribution

usually

varies

for

these

algorithms

due

to

the

different

values

of

Processor

-LRB-

j

j

-RRB-

Processor

-LRB-

j

-RRB-

-LRB-

-RRB-

Processor

-LRB-

j

-RRB-

Processor

-LRB-

-RRB-

Communication

Schemes

On

a

distributed

memory

system

the

computation

of

the

matrix-vector

product

requires

communication

because

each

processor

owns

only

a

partial

vector

For

the

efficient

computation

of

the

matrix-vector

product

it

is

necessary

to

develop

a

suitable

communication

scheme

by

preprocessing

the

distributed

column

index

arrays

Here

we

describe

different

schemes

based

on

different

reorderings

of

the

matrix

First

the

arrays

ask

are

analyzed

on

each

processor

k

to

determine

which

data

results

in

access

to

components

of

the

vector

of

other

processors

Then

ask

and

awk

are

reordered

in

such

a

way

that

the

data

that

results

in

access

to

processor

h

are

collected

in

block

h

The

data

of

block

h

succeeds

one

another

row-wise

with

increasing

column

index

per

row

Block

k

is

the

first

block

in

ask

and

awk

and

contains

the

data

that

results

in

local

access

The

goal

of

this

reordering

is

to

perform

computation

and

communication

overlapped

The

first

reordering

scheme

is

shown

in

-LRB-

-RRB-

for

the

data

distribution

from

-LRB-

-RRB-

and

the

matrix-vector

product

Adi

from

Algorithms

or

Here

only

array

is

analyzed

and

reordered

Processor

-LRB-

j

j

-RRB-

di

-LRB-

-RRB-

Processor

-LRB-

j

-RRB-

di

-LRB-

-RRB-

Processor

-LRB-

j

-RRB-

di

-LRB-

-RRB-

-LRB-

-RRB-

Processor

-LRB-

-RRB-

di

-LRB-

-RRB-

Reordering

-LRB-

j

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

k

j

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

-RRB-

Computing

the

operation

row-times-vector

of

the

matrix-vector

product

of

processor

the

index

results

in

an

access

to

component

of

processor

the

index

to

of

processor

and

the

indices

and

in

access

to

and

of

processor

The

data

blocks

in

-LRB-

-RRB-

are

separated

by

double

dashes

for

elucidation

the

blocks

were

numbered

below

the

brackets

After

reordering

the

data

of

block

results

in

local

access

the

data

of

block

in

access

to

processor

the

data

of

block

in

access

to

processor

and

the

data

of

block

in

access

to

processor

After

having

analyzed

the

column

index

array

ask

each

processor

k

knows

which

components

of

di

are

required

by

which

processors

This

information

is

broadcasted

to

all

processors

Then

each

processor

can

distinguish

which

data

must

be

sent

to

which

processors

This

communication

scheme

is

determined

once

before

the

start

of

the

parallel

CG

algorithm

or

Lanczos

tridiagonalization

and

applies

then

unchanged

to

each

iteration

Fig.

displays

the

communication

scheme

for

the

example

above

Processor

receives

the

third

component

of

di

from

processor

the

sixth

and

seventh

component

from

processor

and

the

eighth

component

from

processor

On

the

other

hand

the

fourth

component

of

processor

is

sent

to

processor

the

fourth

and

fifth

to

processor

and

the

fourth

to

processor

In

Fig.

the

parallel

computation

of

the

matrix-vector

product

is

described

for

the

considered

algorithms

First

on

each

processor

the

data

that

is

necessary

for

other

processors

is

sent

asynchronously

After

having

executed

asynchronous

receive-routines

for

receiving

non-local

data

each

processor

performs

all

local

computations

in

particular

the

local

part

of

the

computation

of

the

matrix-vector

product

Then

each

processor

waits

until

the

data

from

an

arbitrary

processor

arrives

and

continues

the

computation

of

the

matrixvector

product

Thereafter

each

processor

awaits

the

data

of

other

processors

until

the

computation

of

the

matrix-vector

product

is

complete

Computation

and

communication

are

performed

overlapped

While

the

required

data

is

on

the

network

operations

with

local

or

with

data

that

has

already

arrived

from

other

processors

are

executed

In

the

second

reordering

scheme

the

data

blocks

built

as

described

above

are

sent

to

the

processors

that

own

the

corresponding

components

of

the

vector

of

the

matrix-vector

product

The

goal

is

to

increase

the

number

of

local

computations

while

the

required

data

is

on

the

network

In

this

case

the

processors

compute

partial

results

of

the

result

vector

of

the

matrix-vector

product

Then

yk

l

denotes

the

partial

result

of

yk

Akdi

of

processor

k

that

is

computed

on

processor

l

After

the

computation

the

partial

results

except

the

local

Processor

k

Index

j

of

a

vector

component

k

j

Figure

Communication

scheme

reordering

Sending

the

data

that

is

necessary

for

other

processors

asynchronously

Receiving

non-local

data

for

the

matrix-vector

product

asynchronously

Local

vector-vector

operations

Computing

the

matrix-vector

product

with

local

data

k

Matrix-vector

product

with

the

data

of

the

processor

Data

of

a

processor

available

no

no

no

yes

yes

yes

Computation

complete

no

no

yes

yes

yes

no

Figure

The

parallel

matrix-vector

product

reordering

one

are

sent

to

the

corresponding

processors

and

are

then

added

to

the

local

result

of

the

receiving

processor

The

new

distribution

of

the

matrix

data

is

presented

in

-LRB-

-RRB-

Processor

-LRB-

j

j

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

-RRB-

di

-LRB-

-RRB-

Processor

-LRB-

j

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

k

j

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

-RRB-

di

-LRB-

-RRB-

-LRB-

-RRB-

Processor

-LRB-

j

-LCB-

z

-RCB-

k

j

-LCB-

z

-RCB-

-RRB-

di

-LRB-

-RRB-

Processor

-LRB-

-LCB-

z

-RCB-

k

-LCB-

z

-RCB-

-RRB-

di

-LRB-

-RRB-

The

first

number

of

the

blocks

in

-LRB-

-RRB-

denotes

the

processor

to

which

the

partial

result

is

sent

the

second

number

indicates

the

processor

on

which

the

computation

is

performed

Processor

for

example

computes

the

local

result

with

the

first

block

the

partial

result

of

processor

with

the

second

block

the

partial

result

of

processor

with

the

third

block

and

the

partial

result

of

processor

with

the

fourth

block

Fig.

shows

the

communication

scheme

for

the

block

distribution

from

-LRB-

-RRB-

Processor

sends

a

value

to

processor

and

this

value

is

added

to

the

third

component

of

Simultaneously

processor

receives

a

value

from

processor

that

must

be

added

to

the

first

component

of

In

Fig.

the

parallel

computation

of

the

matrix-vector

product

is

presented

for

the

second

reordering

scheme

First

asynchronous

receive-routines

for

receiving

all

necessary

partial

results

of

other

processors

are

executed

on

each

processor

After

that

each

processor

computes

the

partial

results

that

are

sent

to

other

processors

The

computation

is

performed

per

data

block

the

results

are

asynchronously

sent

to

the

corresponding

processors

after

each

computation

Then

all

local

computations

are

performed

in

particular

the

local

part

of

the

computation

of

the

matrix-vector

product

Thereafter

each

processor

waits

until

the

data

of

an

arbitrary

processor

arrives

and

then

adds

the

values

to

the

corresponding

components

of

the

local

result

This

is

repeated

until

the

computation

of

the

matrix-vector

product

is

complete

Computation

and

communication

are

performed

overlapped

Since

partial

results

of

the

matrix-vector

product

are

exchanged

most

computations

are

local

After

having

received

non-local

data

each

processor

merely

performs

a

summation

of

vector

components

However

the

partial

results

must

be

computed

first

then

they

can

be

sent

That

means

that

the

overlap

of

computation

and

communication

is

less

than

for

the

first

scheme

Another

disadvantage

of

this

method

is

that

load

balancing

is

not

guaranteed

any

more

after

the

new

distribution

of

the

blocks

some

processors

can

own

more

or

larger

data

blocks

than

other

ones

However

this

scheme

allows

arbitrary

data

distribution

each

processor

can

get

arbitrary

parts

of

arbitrary

rows

which

need

not

succeed

one

another

For

a

specific

FE

application

a

suitable

data

distribution

for

this

scheme

can

be

found

when

the

data

distribution

considers

the

discretization

mesh

In

the

third

communication

scheme

the

matrix-vector

product

is

performed

columnwise

In

this

case

the

data

distribution

and

the

reordering

scheme

is

the

same

as

shown

in

-LRB-

-RRB-

but

partial

results

of

the

matrix-vector

product

are

sent

and

received

Compared

with

the

first

communication

scheme

the

data

transfer

scheme

is

vice

versa

The

indices

of

the

components

that

are

sent

in

the

first

scheme

are

received

in

the

third

scheme

and

the

indices

of

the

components

that

are

received

in

the

first

scheme

are

sent

Fig.

shows

the

communication

scheme

for

the

example

above

Processor

k

Index

j

of

a

vector

component

k

j

Figure

Communication

scheme

reordering

Local

vector-vector

operations

Computing

the

local

partial

result

k

Adding

the

values

to

the

local

partial

result

Data

of

a

processor

available

no

no

no

yes

yes

yes

Computation

complete

no

no

yes

yes

yes

no

Computing

a

partial

result

of

another

processor

Sending

the

partial

result

asynchronously

Receiving

all

neccessary

non-local

partial

results

asynchronously

Are

there

data

blocks

of

other

processors

Processing

of

all

data

blocks

of

other

proc

yes

yes

yes

yes

yes

yes

no

no

no

nono

no

complete

k

Figure

The

parallel

matrix-vector

product

reordering

Processor

k

Index

j

of

a

vector

component

k

j

Figure

Communication

scheme

Processor

sends

a

partial

result

of

the

third

component

of

the

global

result

vector

of

the

matrix-vector

product

to

processor

and

this

value

is

added

to

the

third

component

of

the

local

result

vector

of

processor

Simultaneously

processor

receives

a

partial

result

of

the

fourth

component

of

the

global

result

vector

of

the

matrix-vector

product

from

processor

that

must

be

added

to

the

first

component

of

the

local

result

vector

of

processor

The

strategy

for

the

overlapped

execution

of

computation

and

communication

is

the

same

as

shown

in

Fig.

Partial

results

must

be

computed

first

then

they

can

be

sent

With

respect

to

the

overlapped

execution

that

means

a

disadvantage

compared

with

the

first

scheme

However

the

number

of

local

computations

are

increased

since

after

having

received

non-local

data

each

processor

merely

performs

a

summation

of

vector

components

Compared

with

the

second

scheme

it

is

advantageous

that

load

balancing

is

not

affected

because

the

data

blocks

are

not

redistributed

The

data

distribution

and

the

communication

schemes

we

present

here

do

not

require

any

knowledge

about

a

specific

discretization

mesh

the

schemes

are

determined

automatically

by

the

analysis

of

the

column

indices

of

the

non-zero

matrix

elements

Moreover

these

schemes

can

be

applied

to

other

algorithms

such

as

CG

methods

with

polynomial

preconditioning

-LSB-

-RSB-

and

the

QMR

algorithm

for

solving

non-Hermitian

systems

of

linear

equations

-LSB-

-RSB-

From

the

operational

point

of

view

polynomial

preconditioning

results

essentially

in

additional

matrix-vector

products

per

CG

iteration

the

number

of

the

matrixvector

products

depends

on

the

degree

of

the

polynomial

In

each

QMR

iteration

two

matrix-vector

products

are

performed

With

respect

to

the

data

distribution

the

parameter

must

be

adapted

to

these

algorithms

For

CG

methods

with

polynomial

preconditioning

the

value

of

the

pure

CG

method

divided

by

the

number

of

matrix-vector

products

per

iteration

is

usually

a

sufficient

approximation

of

With

respect

to

the

parallel

matrix-vector

products

all

three

communication

schemes

can

be

applied

In

the

case

of

CG

methods

with

polynomial

preconditioning

all

matrix-vector

products

per

iteration

are

coupled

whereas

in

the

QMR

iteration

the

two

matrix-vector

products

of

the

form

Asi

and

AT

ti

are

decoupled

i.e.

AT

ti

is

independent

of

the

result

of

Asi

and

vice

versa

For

the

matrix-vector

product

AT

ti

the

third

communication

scheme

is

most

advantageous

since

if

A

is

stored

row-wise

then

AT

is

given

column-wise

The

same

data

structure

can

be

accessed

for

both

Asi

and

AT

ti

if

the

first

scheme

is

used

for

Asi

In

addition

both

schemes

can

be

coupled

i.e.

non-local

data

for

Asi

and

AT

ti

can

be

included

in

the

same

message

The

advantage

to

be

achieved

is

a

decreased

number

of

messages

and

an

increased

overlap

of

computation

and

communication

which

makes

QMR

even

more

attractive

for

parallel

computing

Results

The

numerical

and

performance

tests

of

the

developed

parallel

CG

algorithms

and

the

parallel

Lanczos

methods

were

performed

on

the

distributed

memory

system

and

PARAGON

XP/S

of

the

Research

Centre

J?ulich

The

has

processors

each

with

a

Megabyte

private

memory

interconnected

by

a

hypercube-network

whereas

the

PARAGON

XP/S

has

processors

each

with

a

Megabyte

private

memory

interconnected

by

a

two-dimensional

mesh

The

maximum

transfer

rates

are

and

Megabyte/second

per

channel

in

both

directions

respectively

Numerical

Results

The

tests

we

present

here

were

carried

out

with

one

matrix

from

each

of

the

FE

models

from

environmental

science

and

structural

mechanics

and

in

addition

with

a

sample

matrix

from

-LSB-

-RSB-

The

start

vector

for

Algorithms

and

is

either

given

by

the

FE

models

or

computed

by

bj

ajj

j

n

where

ajj

j

n

denote

the

diagonal

elements

of

A.

For

Algorithm

the

components

of

the

start

vector

are

chosen

as

j

n

j

n

The

corresponding

start

vector

for

Algorithm

is

computed

by

q

i.e.

pn

-LRB-

n

-RRB-

-LRB-

-RRB-

j

n

These

start

vectors

are

advantageous

for

the

parallel

execution

of

the

algorithms

since

they

can

be

generated

fully

in

parallel

and

are

always

the

same

independent

of

the

number

of

processors

used

they

were

applied

in

all

tests

As

a

sample

matrix

of

order

n

we

choose

matrix

-LRB-

-RRB-

from

-LSB-

-RSB-

The

eigenvalues

of

the

matrix

are

given

by

the

formula

-LRB-

i

j

-RRB-

cos

ssi

cos

ssj

with

i

and

j

For

the

tests

with

Algorithms

and

using

matrix

-LRB-

-RRB-

the

right

hand

side

is

chosen

as

bj

nX

i

sji

j

n

so

that

the

exact

solution

vector

is

x

-LRB-

-RRB-

T

Table

shows

numerical

data

of

the

coefficient

matrices

and

for

the

convergence

of

the

CG

method

S

BBBBBBBBBBBBBBBBBBB

CCCCCCCCCCCCCCCCCCCA

-LRB-

-RRB-

Environmental

science

Structural

mechanics

Sample

matrix

Rows

Non-zeros

Density

Non-zeros

per

row

max

mz

aMVP

CG

method

aMVP

Lanczos

tridiagonalization

Condition

number

ss

ss

ss

CG

method

max

scal

abs

diff

Iterations

without

scaling

Iterations

with

scaling

kgi

Table

Numerical

data

of

the

considered

large

sparse

matrices

The

matrix

from

environmental

science

has

rows

that

from

structural

mechanics

In

the

first

case

the

mean

number

of

non-zeros

per

row

is

near

the

maximum

number

This

is

caused

by

a

regular

discretization

mesh

In

the

second

case

the

mean

and

the

maximum

number

are

markedly

different

the

discretization

mesh

is

much

more

irregular

The

operational

contribution

of

the

matrix-vector

product

to

one

iteration

is

for

the

matrix

from

environmental

science

for

the

matrix

from

structural

mechanics

and

for

the

sample

matrix

in

the

case

of

the

CG

method

in

the

case

of

the

Lanczos

tridiagonalization

the

values

are

and

respectively

For

the

first

two

matrices

the

condition

number

with

respect

to

the

spectral

norm

that

is

the

ratio

of

the

largest

and

the

smallest

eigenvalue

in

the

case

of

symmetric

positive

definite

matrices

was

approximated

by

the

developed

parallel

Lanczos

method

applying

Algorithm

In

Table

the

number

of

CG

iterations

with

and

without

diagonal

scaling

is

given

The

iteration

is

stopped

when

the

maximum

scaled

absolute

difference

from

-LRB-

-RRB-

is

less

than

this

corresponds

to

a

precision

of

the

solution

vector

of

about

five

decimals

With

diagonal

scaling

the

number

of

iterations

is

considerably

smaller

in

the

first

two

cases

In

the

third

case

diagonal

scaling

is

not

effective

since

the

diagonal

elements

are

equal

to

one

The

contribution

of

this

preconditioner

to

the

total

execution

time

is

in

all

cases

below

For

the

preconditioned

method

the

Euclidean

norm

of

the

residual

after

and

iterations

respectively

is

given

in

addition

Figure

Top

sparsity

patterns

of

the

matrices

from

environmental

science

-LRB-

left

-RRB-

and

structural

mechanics

-LRB-

right

-RRB-

Bottom

the

same

matrices

with

bandwidth

reduction

Fig.

displays

the

sparsity

patterns

of

the

FE

matrices

The

matrix

from

environmental

science

has

essentially

a

band

structure

with

a

maximum

bandwidth

of

The

matrix

from

structural

mechanics

has

a

much

more

irregular

structure

the

maximum

bandwidth

is

When

the

bandwidth

of

the

matrices

is

diminished

the

communication

overhead

in

each

iteration

of

both

algorithms

is

possibly

reduced

Since

communication

is

necessary

for

the

operation

row-times-vector

of

the

matrix-vector

product

a

smaller

bandwidth

results

in

smaller

message

length

or

even

in

communication

with

fewer

processors

Here

the

matrix

is

reordered

by

the

reverse

Cuthill-McKee

-LRB-

RCM

-RRB-

scheme

-LSB-

-RSB-

which

is

a

similarity

transformation

In

FE

models

this

scheme

is

frequently

used

for

the

assembly

of

the

coefficient

matrix

it

is

performed

only

once

if

the

mesh

does

not

change

whereas

in

many

cases

equation

systems

or

eigenproblems

are

frequently

solved

e.g.

in

each

time

step

of

a

time

dependent

problem

Fig.

also

shows

the

sparsity

patterns

of

both

matrices

with

bandwidth

reduction

For

the

matrix

from

environmental

science

the

bandwidth

is

reduced

by

the

maximum

bandwidth

is

When

the

reverse

Cuthill-McKee

scheme

is

applied

the

maximum

bandwidth

of

the

matrix

from

structural

mechanics

decreases

to

this

is

a

reduction

by

merely

In

all

following

tests

with

respect

to

eigenvalue

approximation

the

tridiagonal

solver

determines

the

eigenvalues

of

the

tridiagonal

matrices

with

a

precision

of

decimals

the

stopping

criterion

is

described

in

-LSB-

-RSB-

Simple

non-spurious

eigenvalues

that

equal

spurious

eigenvalues

with

a

precision

of

decimals

are

marked

as

non-isolated

simple

eigenvalues

The

tolerance

fflt

from

-LRB-

-RRB-

is

set

to

Performance

Results

On

the

the

tests

were

performed

using

the

FORTRAN

compiler

of

the

PortlandGroup

Inc.

release

and

the

operating

system

-LSB-

-RSB-

-LSB-

-RSB-

On

the

PARAGON

we

applied

the

version

of

the

PARAGON

FORTRAN

compiler

and

the

PARAGON

operating

system

release

-LSB-

-RSB-

-LSB-

-RSB-

The

programs

were

compiled

with

the

optimization

switches

Knoieee

on

both

systems

In

the

first

five

investigations

we

do

not

apply

bandwidth

reduction

to

the

matrices

In

Fig.

execution

times

per

iteration

of

the

two

variants

of

the

CG

method

and

the

Lanczos

tridiagonalization

are

compared

With

increasing

number

of

processors

the

execution

times

of

Algorithms

and

are

considerably

less

than

those

of

Algorithms

and

because

the

former

variants

require

only

one

global

communication

On

processors

the

times

are

reduced

by

for

the

matrix

from

environmental

science

and

by

for

the

matrix

from

structural

mechanics

in

the

case

of

the

CG

algorithm

For

the

Lanczos

tridiagonalization

the

corresponding

times

decrease

by

and

respectively

Therefore

we

only

consider

Algorithms

and

in

all

following

investigations

On

the

left

Fig.

displays

execution

times

per

iteration

for

three

different

data

distributions

and

for

the

CG

method

or

for

the

Lanczos

tridiagonalization

For

the

matrix

from

environmental

science

the

execution

times

are

nearly

the

same

since

the

matrix

has

a

regular

structure

In

the

case

of

the

matrix

from

structural

mechanics

the

execution

times

using

the

criteria

same

number

of

non-zeros

and

same

number

of

operations

are

reduced

by

ca.

compared

with

the

time

using

the

criterion

same

number

of

rows

for

both

algorithms

Because

of

the

very

different

number

of

non-zeros

per

row

the

operations

for

the

computation

of

the

matrix-vector

product

are

not

uniformly

distributed

to

each

processor

applying

the

latter

criterion

The

times

for

the

criteria

same

number

of

non-zeros

and

same

number

of

operations

are

nearly

the

same

since

the

operational

contribution

for

the

computation

of

the

matrix-vector

product

Figure

Execution

times

per

iteration

PARAGON

Left

two

variants

of

the

CG

algorithm

right

two

variants

of

the

Lanczos

tridiagonalization

Figure

Execution

times

per

iteration

Left

different

data

distributions

PARAGON

processors

right

different

communication

schemes

CG

method

PARAGON

processors

Figure

CG

method

Left

the

influence

of

overlapping

execution

times

per

iteration

processors

right

speedups

PARAGON

to

one

iteration

is

in

the

case

of

the

CG

method

and

in

the

case

of

the

Lanczos

tridiagonalization

In

all

following

investigations

we

apply

the

criterion

same

number

of

operations

Fig.

also

shows

execution

times

per

iteration

for

the

three

considered

communication

schemes

on

processors

in

the

case

of

the

CG

method

For

the

matrix

from

environmental

science

schemes

to

result

in

almost

the

same

execution

times

This

is

caused

by

a

regular

discretization

mesh

In

the

case

of

the

matrix

from

structural

mechanics

the

time

for

the

second

scheme

increases

markedly

compared

with

the

time

for

the

first

scheme

On

the

one

hand

this

is

caused

by

a

decreased

overlap

of

computation

and

communication

in

the

case

of

the

second

scheme

on

the

other

hand

the

new

distribution

of

the

data

blocks

destroys

the

load

balancing

because

of

the

irregular

structure

of

that

matrix

For

the

third

scheme

the

increase

of

the

execution

time

is

smaller

since

the

data

blocks

are

not

redistributed

Since

schemes

and

do

not

result

in

an

improvement

compared

with

scheme

for

the

matrices

considered

we

apply

the

latter

scheme

in

all

following

investigations

The

times

for

the

first

scheme

on

PARAGON

processors

are

added

to

demonstrate

the

effect

of

the

smaller

cycle

time

of

the

PARAGON

processors

XP

-LRB-

nanoseconds

-RRB-

compared

with

the

iPSC

processors

XR

-LRB-

nanoseconds

-RRB-

The

times

on

the

PARAGON

are

reduced

by

ca.

compared

with

the

corresponding

times

on

the

iPSC

In

Fig.

times

per

iteration

on

processors

with

and

without

the

overlapped

execution

of

computation

and

communication

are

presented

for

the

parallel

CG

algorithm

On

the

iPSC

the

overlapped

execution

reduces

the

execution

times

by

ca.

in

both

cases

whereas

the

overlap

results

on

PARAGON

merely

in

an

decrease

of

the

execution

times

by

ca.

in

the

first

case

and

by

ca.

in

the

second

case

The

main

reason

for

the

difference

is

the

much

higher

transfer

rate

on

the

PARAGON

compared

with

that

on

the

iPSC

On

the

right

Fig.

shows

speedups

on

to

processors

for

the

CG

method

with

and

without

bandwidth

reduction

of

the

matrices

The

equation

system

from

environmental

science

together

with

the

program

code

and

the

remaining

data

requires

the

memory

of

more

than

two

processors

that

from

structural

mechanics

the

memory

of

more

than

four

processors

and

the

sample

system

the

memory

of

more

than

eight

processors

For

up

to

four

in

the

first

case

up

to

eight

in

the

second

case

and

up

to

processors

in

the

third

case

linear

speedup

was

assumed

Bandwidth

reduction

results

in

slightly

higher

speedups

for

up

to

processors

On

and

processors

the

speedups

for

the

matrices

with

bandwidth

reduction

are

even

slightly

less

than

for

the

matrices

without

bandwidth

reduction

On

processors

and

without

bandwidth

reduction

the

speedup

is

in

the

first

case

and

in

the

second

case

This

corresponds

to

efficiencies

of

and

With

bandwidth

reduction

the

speedups

are

and

respectively

the

efficiencies

are

and

For

the

sample

matrix

a

speedup

of

is

achieved

on

processors

the

corresponding

efficiency

is

The

solution

of

the

sample

system

with

a

precision

of

five

decimals

on

processors

requires

an

execution

time

of

seconds

Fig.

displays

the

corresponding

speedups

of

the

total

Lanczos

method

for

the

matrix

from

structural

mechanics

with

bandwidth

reduction

and

for

the

sample

matrix

The

iteration

was

stopped

when

at

least

eigenvalues

of

the

matrix

had

been

determined

with

the

predefined

precision

After

every

steps

of

the

Lanczos

tridiagonalization

the

eigenvalues

of

the

generated

tridiagonal

matrices

were

computed

In

total

steps

were

necessary

to

find

eigenvalues

in

the

first

case

and

eigenvalues

in

the

second

On

processors

speedups

of

and

are

achieved

this

corresponds

to

efficiences

of

and

respectively

The

total

execution

times

on

processors

are

seconds

for

the

matrix

from

structural

mechanics

and

seconds

for

the

sample

matrix

The

contribution

of

the

time

for

the

tridiagonal

solver

to

the

total

time

is

in

the

first

Figure

Total

Lanczos

method

PARAGON

Left

speedups

right

execution

times

matrix

from

environmental

science

with

bandwidth

reduction

case

and

in

the

second

Fig.

also

shows

execution

times

of

the

total

Lanczos

method

for

the

matrix

from

environmental

science

with

bandwidth

reduction

The

same

stopping

criterion

as

before

was

applied

In

this

case

after

every

steps

of

the

Lanczos

tridiagonalization

the

eigenvalues

of

the

generated

tridiagonal

matrices

were

determined

In

total

steps

were

necessary

to

find

eigenvalues

Here

the

time

for

the

tridiagonal

solver

is

dominant

On

processors

the

contribution

to

the

total

time

is

Assuming

linear

speedup

for

up

to

processors

a

total

speedup

of

was

achieved

on

processors

the

efficiency

is

Fig.

demonstrates

that

both

the

developed

parallel

Lanczos

tridiagonalization

and

the

tridiagonal

solver

scale

well

on

distributed

memory

machines

Conclusions

We

presented

a

parallelization

strategy

for

the

iterative

solution

of

both

sparse

systems

of

equations

and

eigenproblems

on

distributed

memory

systems

and

demonstrated

by

case

studies

that

the

developed

data

distribution

and

communication

schemes

which

enable

the

overlapped

execution

of

local

computations

and

communication

do

result

in

efficient

iterative

algorithms

These

algorithms

perform

well

for

large

sparse

matrices

with

very

different

sparsity

patterns

coming

from

real

finite

element

applications

In

recent

investigations

the

parallelization

strategy

above

has

been

applied

successfully

to

CG

methods

with

polynomial

preconditioning

-LSB-

-RSB-

and

the

QMR

algorithm

for

solving

non-Hermitian

systems

of

linear

equations

-LSB-

-RSB-

References

-LSB-

-RSB-

Intel

Supercomputing

Systems

Division

Beaverton

Oregon

Fortran

Compiler

User

's

Guide

January

Order

Number

-LSB-

-RSB-

Intel

Supercomputing

Systems

Division

Beaverton

Oregon

System

User

's

Guide

March

Order

Number

-LSB-

-RSB-

Intel

Supercomputing

Systems

Division

Beaverton

Oregon

Paragon

Fortran

Compiler

User

's

Guide

April

Order

Number

-LSB-

-RSB-

Intel

Supercomputing

Systems

Division

Beaverton

Oregon

Paragon

User

's

Guide

October

Order

Number

-LSB-

-RSB-

a

three-dimensional

finite

element

model

of

water

flow

through

saturated-unsaturated

media

Oak

Ridge

National

Laboratory

-LSB-

-RSB-

SMART

Benutzerhandb?ucher

Institut

f?ur

Statik

und

Dynamik

der

Luft

und

Raumfahrtkonstruktionen

der

Universit?at

Stuttgart

ISD-Berichte

-LSB-

-RSB-

E.

Anderson

et

al.

LAPACK

user

's

guide

SIAM

Philadelphia

-LSB-

-RSB-

S.F.

Ashby

Minimax

polynomial

preconditioning

for

hermitian

linear

systems

SIAM

J.

Matrix

Anal

Appl.

-LCB-

-LSB-

-RSB-

C.

Aykanat

F.

Ozg?uner

D.S.

Scott

Vectorization

and

parallelization

of

the

conjugate

gradient

algorithm

on

hypercube-connected

vector

processors

Microprocessing

and

Microprogramming

-LCB-

-LSB-

-RSB-

V.A.

Barker

C.

Yingqun

LANSYM

A

Fortran

Subroutine

for

Computing

Eigensolutions

of

Symmetric

Sparse

Matrices

on

the

Connection

Machine

Institute

for

Numerical

Analysis

Technical

University

of

Denmark

Lyngby

Report

December

-LSB-

-RSB-

A.

Basermann

Conjugate

gradients

parallelized

on

the

hypercube

International

Journal

of

Modern

Physics

C

Vol

No.

-LCB-

-LSB-

-RSB-

A.

Basermann

P.

Weidner

A

parallel

algorithm

for

determining

all

eigenvalues

of

large

real

symmetric

tridiagonal

matrices

Parallel

Computing

-LCB-

-LSB-

-RSB-

L.

Collatz

Eigenwertaufgaben

mit

technischen

Anwendungen

Akademische

Verlagsgesellschaft

Geest

Portig

K.-G.

Leipzig

-LSB-

-RSB-

J.K.

Cullum

R.A.

Willoughby

Lanczos

Algorithms

for

Large

Symmetric

Eigenvalue

Computations

Volume

I

Theory

Birkh?auser

Boston

Basel

Stuttgart

-LSB-

-RSB-

J.K.

Cullum

R.A.

Willoughby

Lanczos

Algorithms

for

Large

Symmetric

Eigenvalue

Computations

Volume

II

Programs

Birkh?auser

Boston

Basel

Stuttgart

-LSB-

-RSB-

N.

Feistl

Das

Verfahren

der

konjugierten

Gradienten

auf

Vektorrechern

Diplomarbeit

Ludwig-Maximilians-Universit

at

M?unchen

Juli

-LSB-

-RSB-

P.

Fernandes

P.

Girdinio

A

new

storage

scheme

for

an

efficient

implementation

of

the

sparse

matrix-vector

product

Parallel

Computing

-LCB-

-LSB-

-RSB-

R.W.

Freund

N.M.

Nachtigal

QMR

a

quasi-minimal

residual

method

for

nonHermitian

linear

systems

Numerische

Mathematik

-LCB-

-LSB-

-RSB-

M.R.

Hestenes

E.

Stiefel

Methods

of

conjugate

gradients

for

solving

linear

systems

Journal

of

Research

of

the

National

Bureau

of

Standards

-LCB-

-LSB-

-RSB-

T.J.R.

Hughes

The

Finite

Element

Method

Prentice-Hall

Englewood

Cliffs

-LSB-

-RSB-

S.K.

Kim

A.T.

Chronopoulos

A

class

of

Lanczos-like

algorithms

implemented

on

parallel

computers

Parallel

Computing

-LCB-

-LSB-

-RSB-

C.P.

Kruskal

L.

Rudolph

M.

Snir

Techniques

for

parallel

manipulation

of

sparse

matrices

Theoretical

Computer

Science

-LCB-

-LSB-

-RSB-

O.A.

McBryan

E.F.

Van

de

Velde

Matrix

and

vector

operations

on

hypercube

parallel

processors

Parallel

Computing

-LCB-

-LSB-

-RSB-

J.M.

Ortega

Introduction

to

Parallel

and

Vector

Solution

of

Linear

Systems

Plenum

Press

New

York

London

-LSB-

-RSB-

B.N.

Parlett

The

Symmetric

Eigenvalue

Problem

Prentice-Hall

Englewood

Cliffs

-LSB-

-RSB-

S.

Pissanetsky

Sparse

Matrix

Technology

Academic

Press

London

Orlando

-LSB-

-RSB-

Y.

Saad

Numerical

Methods

for

Large

Eigenvalue

Problems

Manchester

University

Press

Manchester

-LSB-

-RSB-

U.

Schendel

Sparse-Matrizen

R.

Oldenbourg

Verlag

M?unchen

Wien

Auflage

-LSB-

-RSB-

H.R.

Schwarz

FORTRAN-Programme

zur

Methode

der

finiten

Elemente

B.

G.

Teubner

Stuttgart

-LSB-

-RSB-

H.

Vereecken

G.

Lindenmayr

A.Kuhr

D.H.

Welte

A.

Basermann

Numerical

modeling

of

field

scale

transport

in

heterogeneous

variably

saturated

porous

media

KFA/ICG

Internal

Report

No.

January

-LSB-

-RSB-

J.H.

Wilkinson

The

Algebraic

Eigenvalue

Problem

Clarendon

Press

Oxford

-LSB-

-RSB-

J.H.

Wilkinson

C.

Reinsch

Handbook

for

Automatic

Computation

Volume

II

Linear

Algebra

Springer-Verlag

Berlin

Heidelberg

New

York

communication scheme
conjugate gradient method
data distribution
lanczos algorithm
parallelization
sparse matrices


Distance

learning

with

the

World-Wide

Web

B.

Ibrahim

Distance

learning

with

the

World-Wide

Web

Bertrand

Ibrahim

University

of

Geneva

rue

du

G?n?ral

Dufour

Geneva

Switzerland

email

<bertrand@cui.unige.ch>

www

Summary

The

World-Wide

Web

-LRB-

WWW

-RRB-

is

often

seen

as

just

a

means

to

navigate

through

hyperdocuments

across

the

Internet

More

sophisticated

uses

are

however

possible

The

statistics

on

the

use

of

the

Internet

backbone

show

that

the

use

of

WWW

is

growing

exponentially

and

is

becoming

one

of

the

most

important

media

for

global

information

sharing

In

this

paper

we

describe

the

main

features

of

WWW

then

discuss

some

reasons

for

its

success

and

finally

we

analyze

those

features

that

should

prove

most

interesting

for

distance

learning

Conference

theme

Dissemination

of

Information

Learning

Sector

Higher

Education

What

is

the

World-Wide

Web

Even

though

WWW

is

often

referred

to

by

the

name

of

its

most

widespread

viewer

that

is

Mosaic

it

can

be

best

characterized

as

the

combination

of

a

client-server

protocol

-LRB-

the

HyperText

Transfer

Protocol

HTTP

-RRB-

and

a

markup

language

-LRB-

the

HyperText

Markup

Language

HTML

-RRB-

The

protocol

specifies

how

programs

at

the

two

ends

of

a

communication

line

-LRB-

a

client

and

a

server

-RRB-

should

communicate

with

each

other

and

the

markup

language

specifies

the

set

of

display

primitives

that

a

-LRB-

client

-RRB-

viewer

should

be

able

to

handle

-LRB-

see

fig

-RRB-

The

most

common

configuration

is

for

an

information

provider

to

run

one

of

the

public

domain

HTTP

servers

available

for

all

sorts

of

platforms

and

for

an

information

consumer

to

run

one

of

the

public

domain

HTML

viewers

among

which

Mosaic

is

probably

the

most

well

known

because

of

its

multimedia

features

The

most

important

features

of

WWW

are

The

integration

of

most

widespread

Internet

protocols

allowing

the

user

to

have

access

to

almost

all

existing

resources

available

on

the

Internet

in

an

integrated

fashion

A

uniform

naming

scheme

for

Internet

resources

the

URLs

-LRB-

Uniform

Resource

Locators

-RRB-

consisting

of

various

fields

path

where

path

is

either

the

full

path

of

a

document

in

the

server

storage

space

or

a

query

to

a

database

server

or

some

information

allowing

the

server

to

locate

unambiguously

a

resource

it

has

access

to

Easy

interfacing

with

external

viewers

The

documents

are

indeed

typed

and

different

viewing

programs

can

be

associated

to

the

various

types

-LRB-

the

most

common

scheme

is

to

infer

the

type

of

the

document

from

the

suffix

of

the

file

name

-RRB-

New

types

can

even

be

defined

with

appropriate

viewers

if

necessary

When

the

user

follows

a

link

that

points

to

a

document

which

Distance

learning

with

the

World-Wide

Web

B.

Ibrahim

requires

an

external

viewer

the

WWW

viewer

-LRB-

for

instance

Mosaic

-RRB-

retrieves

that

document

and

launches

the

corresponding

viewer

to

handle

it

The

advantage

of

this

approach

is

that

it

is

very

easily

extendible

but

its

drawback

is

that

those

documents

that

require

an

external

viewer

are

outside

the

hyperstructure

that

is

they

can

not

hold

links

to

other

documents

Handling

of

hypermedia

documents

since

the

HTML

language

allows

for

the

integration

in

a

single

document

of

text

pictures

and

icons

which

can

all

bear

links

to

other

documents

via

embedded

URL

references

There

are

some

limitations

though

as

video

animations

and

sounds

are

handled

by

external

viewers

There

is

thus

no

synchronization

possible

between

an

HTML

document

and

video

animations

or

sounds

High

portability

since

HTML

documents

are

in

a

simple

and

machine-independent

format

Each

HTML

viewer

can

display

the

documents

according

to

its

capabilities

and

those

of

the

machine

it

is

running

on

However

there

is

presently

very

little

portability

when

it

comes

to

video

animation

or

sound

as

very

different

formats

are

used

in

the

various

computers

available

on

the

market

Platform

independent

standards

are

starting

to

emerge

like

the

MPEG

format

for

video

animations

for

instance

but

they

often

involve

some

limitation

like

requiring

a

color

display

HTTP

Server

sends

actual

or

virtual

documents

sends

a

request

based

displays

an

HTML

document

Request

with

Requested

Fig.

FTP

server

sends

files

from

Request

with

path

and

file

name

Requested

file

HTTP

protocol

WAIS

Gopher

some

protocol

conversion

to

HTML

some

query

reply

to

query

document

on

a

URL

then

-LRB-

usually

-RRB-

document

path

upon

request

workstation

FTP

HTTP

HTML

HyperText

Transfer

HyperText

Markup

Language

Protocol

File

Transfer

Protocol

other

FTP

protocol

URL

Universal

Resource

Locator

disk

disk

disk

Client-Server

configuration

of

the

World-Wide

Web

Depending

on

the

protocol

used

the

WWW

client

might

need

to

convert

the

information

to

HTML

before

displaying

it

provides

information

HyTelnet

server

WWW

client

its

repository

Distance

learning

with

the

World-Wide

Web

B.

Ibrahim

Why

is

WWW

so

successful

It

is

always

very

delicate

and

error-prone

to

try

to

explain

social

phenomena

The

growth

of

the

Internet

usage

via

WWW

has

increased

exponentially

since

its

availability

less

than

two

years

ago

This

extraordinary

growth

can

probably

be

explained

by

a

combination

of

multiple

factors

-LSB-

-RSB-

The

Internet

growth

itself

More

than

two

million

hosts

are

connected

to

it

representing

an

estimate

of

more

than

twenty

million

users

It

has

probably

reached

a

critical

mass

in

which

almost

anybody

can

find

something

of

interest

The

transparent

access

to

all

commonly

available

Internet

resources

giving

access

to

a

gigantic

universe

of

electronic

information

with

the

added

appeal

of

hypermedia

making

it

trivial

for

anybody

to

do

net-surfing

The

ease

of

use

is

certainly

a

very

important

factor

for

widespread

acceptance

The

availability

in

the

public

domain

of

both

server

and

viewer

software

on

multiple

platforms

The

ease

of

installation

of

the

server

software

is

probably

a

key

factor

to

the

increasing

amount

of

documents

available

all

over

the

world

The

relative

simplicity

of

the

HTML

syntax

Thanks

to

this

simplicity

information

providers

do

not

need

sophisticated

tools

to

prepare

their

documents

Distance

learning

with

WWW

Tools

such

as

WWW

have

mostly

been

developed

to

make

it

easier

to

share

information

among

scientists

Even

though

they

are

still

in

their

infancy

these

tools

are

already

being

used

for

distance

learning

using

the

Internet

infrastructure

to

transport

almost

transparently

information

from

the

tutors

to

the

learners

These

pedagogical

uses

have

on

one

hand

limitations

in

what

they

can

attempt

to

do

compared

to

educational

software

that

runs

on

stand-alone

machines

because

of

the

current

limitations

of

the

HTTP

protocol

and

of

the

HTML

syntax

For

instance

current

educational

material

available

through

WWW

makes

only

use

of

the

hypertextual

capabilities

of

the

medium

allowing

the

learner

to

navigate

through

documents

without

much

feedback

from

the

system

-LSB-

-RSB-

-LSB-

-RSB-

But

on

the

other

hand

pedagogical

uses

of

WWW

give

the

unique

opportunity

to

use

the

ocean

of

information

and

knowledge

that

is

accessible

on

the

Internet

and

that

is

now

available

at

a

fingertip

More

sophisticated

uses

of

the

WWW

are

possible

without

much

additional

effort

to

provide

much

more

intelligent

tutoring

We

have

demonstrated

with

an

actual

example

-LSB-

-RSB-

how

educational

programs

can

be

executed

remotely

through

WWW

thus

making

use

of

all

the

interesting

features

of

this

tool

i.e.

hypermedia

and

remote

access

without

giving

up

on

those

features

that

make

tutorials

intelligent

i.e.

maintain

a

profile

of

the

user

and

adapt

the

behavior

of

the

program

to

the

abilities

of

the

learner

The

basic

principle

for

this

sophisticated

use

is

the

following

-LRB-

fig

-RRB-

some

links

can

cause

a

WWW

server

to

execute

on

the

server

machine

a

program

or

a

shell

script

which

in

turn

can

spawn

a

process

that

will

handle

the

pedagogical

interactions

with

the

distant

user

This

child

Distance

learning

with

the

World-Wide

Web

B.

Ibrahim

process

will

run

during

the

whole

instructional

session

and

its

output

will

be

sent

to

the

learner

's

viewer

as

virtual

HTML

documents

These

virtual

documents

contain

embedded

links

that

include

the

process-ID

of

the

educational

program

They

can

also

contain

fields

that

have

to

be

filled

by

the

user

and

that

will

be

transmitted

back

to

the

instructional

program

by

the

WWW

client

viewer

These

virtual

documents

can

as

well

contain

links

to

actual

documents

anywhere

on

the

Internet

thus

giving

the

learner

potential

-LRB-

guided

-RRB-

access

to

this

whole

world

of

information

References

-LSB-

-RSB-

Bruce

R.

Schatz

Joseph

B.

Hardin

NCSA

Mosaic

and

the

World

Wide

Web

Global

Hypermedia

Protocols

for

the

Internet

Science

magazine

Vol

August

pp

-LSB-

-RSB-

Bertrand

Ibrahim

World-Wide

Algorithm

Animation

Advance

Proceedings

of

the

First

World-Wide-Web

conference

Geneva

Switzerland

May

pp

-LSB-

-RSB-

Marcus

Speh

Carter

Butts

Colman

Reilly

Joseph

C.

Wang

WWW

for

the

Globewide

Network

Academy

Advance

Proceedings

of

the

First

World-Wide-Web

conference

Geneva

Switzerland

May

pp

-LSB-

-RSB-

Dimitri

A.

Dimitroyannis

Virtual

Classroom

A

Case

Study

Advance

Proceedings

of

the

First

World-Wide-Web

conference

Geneva

Switzerland

May

pp

WWW

client

-RRB-

sends

special

request

-LRB-

cgi-bin

script

-RRB-

to

initiate

WWW

server

spawns

instructional

process

then

communicates

with

it

Instructional

software

Sockets

instructional

software

-RRB-

sends

content

of

various

input

fields

in

document

received

from

instructional

HTTP

protocol

Request

with

name

of

instructional

program

process-ID

and

user

Next

instructional

page

entered

fields

user-entered

fields

analyzes

user

input

then

sends

next

instructional

page

based

on

this

input

Fig.

Interaction

scheme

between

the

remote

user

and

some

instructional

software

via

WWW

software

with

embedded

links

holding

name

of

program

and

process-ID

B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O