A

Safe

Efficient

Regression

Test

Selection

Gregg

Rothermel

and

Mary

Jean

Harrold

Department

of

Computer

and

Information

Science

The

Ohio

State

University

Dreese

Lab

Neil

Avenue

Columbus

OH

fgrother,harroldg@cis.ohio-state.edu

April

Abstract

Regression

testing

is

an

expensive

but

necessary

maintenance

activity

performed

on

modified

software

to

provide

confidence

that

changes

are

correct

and

do

not

adversely

affect

other

portions

of

the

software

A

regression

test

selection

technique

chooses

from

an

existing

test

set

tests

that

are

deemed

necessary

to

validate

modified

software

Most

regression

test

selection

techniques

depend

on

a

particular

test

adequacy

criterion

or

require

prior

knowledge

of

where

code

has

been

modified

We

present

a

new

technique

for

regression

test

selection

that

is

neither

adequacy-based

nor

requires

prior

knowledge

of

modifications

Our

algorithms

construct

control

flow

graphs

for

a

procedure

or

program

and

its

modified

version

and

use

these

graphs

to

select

tests

from

the

original

test

set

that

execute

changed

code

We

prove

that

under

certain

conditions

the

set

of

tests

our

algorithms

select

includes

every

test

from

the

original

test

suite

that

can

expose

faults

in

the

modified

procedure

or

program

Thus

under

these

conditions

the

algorithms

are

safe

Moreover

although

our

algorithms

may

select

some

tests

that

can

not

expose

faults

they

are

at

least

as

precise

as

other

safe

regression

test

selection

algorithms

Unlike

many

other

regression

test

selection

algorithms

our

algorithms

handle

all

language

constructs

and

all

types

of

program

modifications

We

have

implemented

our

algorithms

initial

empirical

studies

indicate

that

our

technique

can

significantly

reduce

the

cost

of

regression

testing

modified

software

Introduction

Software

maintenance

activities

may

account

for

as

much

as

two-thirds

of

the

overall

cost

of

software

production

-LSB-

-RSB-

One

necessary

maintenance

activity

regression

testing

is

performed

on

modified

software

to

provide

confidence

that

the

software

behaves

correctly

and

that

modifications

have

not

adversely

impacted

the

software

's

quality

Regression

testing

is

expensive

it

can

account

for

as

much

as

one-half

of

the

cost

of

software

maintenance

-LSB-

-RSB-

An

important

difference

between

regression

testing

and

development

testing

is

that

during

regression

testing

we

may

have

an

established

suite

of

tests

available

for

reuse

One

regression

testing

strategy

reruns

all

such

tests

but

this

retest

all

approach

may

consume

inordinate

time

and

resources

Selective

retest

techniques

in

contrast

attempt

to

reduce

the

time

required

to

retest

a

modified

program

by

selectively

reusing

tests

and

selectively

retesting

the

modified

program

Although

many

selective

retest

techniques

have

been

proposed

most

of

these

techniques

emphasize

the

use

of

structural

coverage

criteria

-LSB-

-RSB-

These

preliminary

version

of

this

work

appeared

in

Reference

-LSB-

-RSB-

This

work

was

partially

supported

by

a

grant

from

Microsoft

Inc.

and

by

NSF

under

Grant

to

Clemson

University

and

Ohio

State

University

techniques

identify

changed

or

affected

program

components

such

as

statements

or

paths

and

attempt

to

select

tests

that

exercise

these

components

Coverage

criteria

are

useful

because

they

offer

a

way

to

measure

test

adequacy

However

another

goal

of

regression

testing

is

to

provide

confidence

that

program

functionality

has

not

been

adversely

affected

by

modifications

In

practice

most

test

suites

contain

tests

designed

to

exercise

functionality

rather

than

tests

designed

to

cover

components

this

is

particularly

true

for

test

suites

designed

for

integration

and

system

testing

Where

selection

of

these

tests

from

an

existing

test

suite

is

concerned

we

are

often

more

concerned

with

finding

tests

that

reveal

faults

in

the

modified

program

than

we

are

with

finding

tests

that

cover

modified

components

The

coverage-based

selective

retest

techniques

cited

above

can

omit

such

fault-revealing

tests

in

significant

situations

-LSB-

-RSB-

Some

selective

retest

techniques

place

less

emphasis

on

coverage

criteria

-LSB-

-RSB-

One

drawback

of

these

techniques

is

that

they

require

prior

knowledge

of

where

the

modified

program

has

been

changed

This

knowledge

must

be

obtained

either

by

a

programming

environment

that

tracks

changes

as

they

are

made

or

by

algorithms

-LSB-

-RSB-

that

determine

corresponding

program

components

The

assumption

of

a

programming

environment

that

tracks

changes

may

be

unreasonable

given

current

practice

The

algorithms

for

determining

corresponding

components

may

be

unnecessarily

costly

for

this

application

Selective

retest

techniques

address

two

problems

-LRB-

-RRB-

the

problem

of

selecting

tests

from

an

existing

test

suite

and

-LRB-

-RRB-

the

problem

of

determining

where

additional

tests

may

be

required

Both

of

these

problems

are

important

however

in

this

paper

we

address

the

first

problem

the

regression

test

selection

problem

In

this

paper

we

present

a

new

technique

for

regression

test

selection

that

is

neither

coverage-criteria

based

nor

requires

complete

information

on

corresponding

program

components

Our

algorithms

construct

control

flow

graphs

for

a

procedure

or

program

and

its

modified

version

and

use

these

graphs

to

select

tests

from

the

original

test

suite

that

execute

changed

code

Our

technique

has

several

advantages

over

other

regression

test

selection

techniques

Unlike

many

techniques

our

algorithms

select

tests

that

may

now

execute

new

or

modified

statements

and

tests

that

formerly

executed

statements

that

have

been

deleted

from

the

original

program

We

prove

that

under

certain

conditions

our

technique

selects

every

test

from

the

original

test

suite

that

can

expose

faults

in

the

modified

program

Thus

under

those

conditions

our

algorithms

are

safe

Moreover

although

our

algorithms

may

select

some

tests

that

can

not

reveal

faults

they

are

more

precise

than

other

safe

algorithms

because

they

select

fewer

tests

that

can

not

execute

changed

code

Our

algorithms

are

also

more

efficient

than

most

existing

algorithms

The

algorithms

are

easy

to

implement

and

they

automate

an

important

portion

of

the

regression

testing

process

Finally

our

algorithms

are

more

general

than

most

previous

techniques

They

handle

both

interprocedural

and

intraprocedural

regression

test

selection

Also

they

handle

all

language

constructs

and

all

types

of

program

modifications

for

procedural

languages

We

have

implemented

our

algorithms

and

conducted

empirical

studies

on

several

subject

programs

and

modified

versions

Our

results

suggest

that

in

practice

our

algorithms

can

significantly

reduce

the

cost

of

regression

testing

a

modified

program

In

the

next

section

we

provide

background

information

We

present

our

test

selection

algorithms

in

Section

and

we

analyze

and

evaluate

our

algorithms

in

Section

Section

presents

our

conclusions

Background

We

adopt

the

following

notation

throughout

the

rest

of

this

paper

Let

P

be

a

procedure

or

program

let

P

be

a

modified

version

of

P

and

let

S

and

be

the

specifications

for

P

and

P

respectively

P

-LRB-

i

-RRB-

refers

to

the

output

of

P

on

input

i

P

-LRB-

i

-RRB-

refers

to

the

output

of

P

on

input

i

S

-LRB-

i

-RRB-

refers

to

the

specified

output

for

P

on

input

i

and

-LRB-

i

-RRB-

refers

to

the

specified

output

for

P

on

input

i.

Let

T

be

a

set

of

tests

-LRB-

a

test

suite

-RRB-

created

to

test

P

A

test

is

a

identifier

input

output

in

which

identifier

identifies

the

test

input

is

the

input

for

that

execution

of

the

program

and

output

is

the

specified

output

S

-LRB-

input

-RRB-

for

this

input

For

simplicity

in

the

sequel

we

refer

to

a

test

ht

i

S

-LRB-

i

-RRB-

i

by

its

identifier

t

and

refer

to

the

outputs

P

-LRB-

i

-RRB-

and

S

-LRB-

i

-RRB-

of

test

t

for

input

i

as

P

-LRB-

t

-RRB-

and

S

-LRB-

t

-RRB-

respectively

We

assume

that

all

statements

in

P

are

statically

reachable

Because

it

is

possible

to

eliminate

code

that

is

not

statically

reachable

from

a

program

this

is

not

an

unreasonable

restriction

Control

Flow

Graphs

A

control

flow

graph

-LRB-

CFG

-RRB-

for

procedure

P

contains

a

node

for

each

simple

or

conditional

statement

in

P

edges

between

nodes

represent

the

flow

of

control

between

statements

Figure

shows

procedure

avg

and

its

CFG

In

the

figure

statement

nodes

shown

as

ellipses

represent

simple

statements

Predicate

nodes

shown

as

squares

stand

for

conditional

statements

Labeled

edges

leaving

predicate

nodes

are

called

branches

-LCB-

they

represent

control

paths

taken

when

the

predicate

evaluates

to

the

value

of

the

edge

label

Statement

and

predicate

nodes

are

labeled

to

indicate

the

statements

in

P

to

which

they

correspond

In

the

figure

for

simplicity

we

use

statement

numbers

as

labels

but

we

could

instead

use

the

actual

code

of

the

associated

statements

as

labels

A

unique

entry

node

and

a

unique

exit

node

represent

entry

to

and

exit

from

P

The

CFG

for

a

procedure

P

has

size

and

can

be

constructed

in

time

linear

in

the

number

of

simple

and

conditional

statements

in

P

-LSB-

-RSB-

To

facilitate

the

presentation

and

discussion

of

our

algorithms

we

think

of

unlabeled

CFG

edges

as

having

the

label

ffl

we

also

assume

initially

that

case

statements

are

rendered

as

nested

if-else

statements

Given

these

assumptions

every

CFG

node

has

either

one

out-edge

labeled

ffl

or

two

out-edges

labeled

T

and

F

respectively

Assume

further

that

we

associate

all

declaration

and

nonexecutable

initialization

statements

that

pertain

to

a

procedure

with

a

declaration

node

that

we

insert

into

the

CFG

as

the

immediate

successor

of

the

entry

node

We

discuss

other

methods

for

handling

case

statements

declarations

and

other

types

of

nonexecutable

initialization

statements

in

Section

Code

Instrumentation

We

can

instrument

a

program

P

such

that

when

we

execute

the

instrumented

version

of

P

with

test

t

the

program

reports

the

branches

taken

in

the

program

for

t.

Using

this

branch

trace

information

we

can

easily

determine

for

t

which

edges

in

the

CFG

G

for

P

were

traversed

when

t

was

executed

We

call

the

simplicity

we

omit

these

nodes

from

our

figures

say

that

an

edge

-LRB-

-RRB-

in

G

was

traversed

by

test

t

if

and

only

if

when

t

was

run

on

P

the

statements

associated

with

and

were

executed

sequentially

at

least

once

during

the

execution

fread

-LRB-

fileptr

n

-RRB-

endif

count

numarray

-LSB-

count

-RSB-

n

elsereturn

-LRB-

error

-RRB-

if

-LRB-

n

-RRB-

exit

while

-LRB-

not

EOF

-RRB-

do

fread

-LRB-

fileptr

n

-RRB-

count

Procedure

AVG

endwhileavg

calcavg

-LRB-

numarray

count

-RRB-

return

-LRB-

avg

-RRB-

entry

Figure

Procedure

avg

and

its

CFG

information

thus

gathered

an

edge

trace

for

t

on

P

An

edge

trace

for

t

on

P

has

size

linear

in

the

number

of

edges

in

the

CFG

for

P

we

can

use

a

bit

vector

to

represent

such

a

trace

Given

program

P

and

test

suite

T

we

construct

a

test

history

for

P

with

respect

to

T

by

gathering

edge

trace

information

for

each

test

in

T

and

representing

it

such

that

for

each

edge

-LRB-

-RRB-

in

the

CFG

for

P

we

know

which

tests

traversed

that

edge

This

representation

requires

O

-LRB-

ejT

j

-RRB-

bits

where

e

is

the

number

of

edges

in

G

and

jT

j

is

the

number

of

tests

in

T

For

CFGs

of

the

form

we

employ

e

is

no

greater

than

twice

the

number

of

nodes

in

G

thus

e

is

linear

in

the

size

of

P

Figure

gives

a

table

that

reports

test

information

along

with

the

corresponding

test

history

for

program

avg

of

Figure

For

convenience

in

the

remainder

of

this

paper

we

assume

the

existence

of

a

function

TestsOnEdge

-LRB-

-RRB-

that

returns

a

bit

vector

v

of

size

jT

j

bits

such

that

the

ith

bit

in

v

is

set

if

and

only

if

test

i

in

T

traversed

edge

-LRB-

-RRB-

in

G.

Regression

Testing

Research

on

regression

testing

spans

a

wide

variety

of

topics

including

test

environments

and

automation

-LSB-

-RSB-

capture-playback

mechanisms

-LSB-

-RSB-

test

suite

management

-LSB-

-RSB-

program

size

reduction

-LSB-

-RSB-

and

regression

testability

-LSB-

-RSB-

Most

recent

research

on

regression

testing

however

concerns

selective

retest

techniques

Selective

retest

techniques

reduce

the

cost

of

regression

testing

by

reusing

existing

tests

and

identifying

portions

of

the

modified

program

or

its

specification

that

should

be

tested

Selective

retest

techniques

differ

from

the

retest-all

technique

which

runs

all

tests

in

the

existing

test

suite

Leung

and

White

-LSB-

-RSB-

show

that

a

selective

retest

technique

is

more

economical

than

the

retest-all

technique

if

the

cost

of

selecting

a

reduced

empty

-LRB-

entry

-RRB-

-LRB-

-RRB-

inputtest

edges

traversed

-LRB-

entry

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

exit

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

exit

-RRB-

-LRB-

entry

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

exit

-RRB-

edge

TEST

INFORMATION

TEST

HISTORYTestsOnEdge

-LRB-

edge

-RRB-

-LRB-

entry

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

exit

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

exit

-RRB-

Figure

Test

information

and

test

history

for

procedure

avg

subset

of

tests

to

run

is

less

than

the

cost

of

running

the

tests

that

the

selective

retest

technique

lets

us

omit

A

typical

selective

retest

technique

proceeds

as

follows

Select

T

T

a

set

of

tests

to

execute

on

P

Test

P

with

T

establishing

P

's

correctness

with

respect

to

T

If

necessary

create

T

a

set

of

new

functional

or

structural

tests

for

P

Test

P

with

T

establishing

P

's

correctness

with

respect

to

T

Create

T

a

new

test

suite

and

test

history

for

P

from

T

T

and

T

In

performing

these

steps

a

selective

retest

technique

addresses

several

problems

Step

involves

the

regression

test

selection

problem

the

problem

of

selecting

a

subset

T

of

T

with

which

to

test

P

This

problem

includes

the

subproblem

of

identifying

tests

in

T

that

are

obsolete

for

P

Test

t

is

obsolete

for

program

P

if

and

only

if

t

specifies

an

input

to

P

that

according

to

is

invalid

for

P

or

t

specifies

an

invalid

input-output

relation

for

P

Step

addresses

the

coverage

identification

problem

the

problem

of

identifying

portions

of

P

or

that

require

additional

testing

Steps

and

address

the

test

suite

execution

problem

the

problem

of

efficiently

executing

tests

and

checking

test

results

for

correctness

Step

addresses

the

test

suite

maintenance

problem

the

problem

of

updating

and

storing

test

information

Although

each

of

these

problems

is

significant

we

restrict

our

attention

to

the

regression

test

selection

problem

We

further

restrict

our

attention

to

code-based

regression

test

selection

techniques

which

rely

on

analysis

of

P

and

P

to

select

tests

We

distinguish

two

phases

of

regression

testing

a

preliminary

phase

and

a

critical

phase

The

preliminary

phase

of

regression

testing

begins

after

the

release

of

some

version

of

the

software

during

this

phase

examples

of

specification-based

techniques

see

References

-LSB-

-RSB-

and

-LSB-

-RSB-

programmers

enhance

and

correct

the

software

When

corrections

are

complete

the

critical

phase

of

regression

testing

begins

during

this

phase

regression

testing

is

the

dominating

activity

and

its

time

is

limited

by

the

deadline

for

product

release

It

is

in

the

critical

phase

that

cost

minimization

is

most

important

for

regression

testing

Regression

test

selection

techniques

can

exploit

these

phases

For

example

a

technique

that

requires

test

history

and

program

analysis

information

during

the

critical

phase

can

achieve

a

lower

critical

phase

cost

by

gathering

that

information

during

the

preliminary

phase

Regression

test

selection

algorithms

For

reasons

that

shall

become

clear

our

goal

is

to

identify

nonobsolete

tests

in

T

that

execute

changed

code

with

respect

to

P

and

P

In

other

words

we

wish

to

identify

tests

in

T

that

-LRB-

-RRB-

execute

code

that

is

new

or

modified

for

P

or

-LRB-

-RRB-

used

to

execute

code

in

P

that

is

no

longer

present

in

P

We

call

such

tests

modification-traversing

for

P

and

P

To

capture

the

notion

of

these

tests

more

formally

we

define

an

execution

trace

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

for

t

on

P

to

consist

of

the

sequence

of

statements

in

P

that

are

executed

when

P

is

tested

with

t

We

say

that

two

execution

traces

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

and

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

are

equivalent

if

they

have

different

lengths

or

if

when

we

compare

their

elements

from

first

to

last

we

find

some

pair

of

elements

that

are

lexically

nonidentical

We

say

that

t

is

modification-traversing

for

P

and

P

-LRB-

or

for

simplicity

that

t

is

modification-traversing

-RRB-

if

and

only

if

ET

-LRB-

P

-LRB-

i

-RRB-

-RRB-

is

not

equivalent

to

ET

-LRB-

P

-LRB-

i

-RRB-

-RRB-

To

identify

the

modification-traversing

tests

in

T

we

must

identify

nonobsolete

tests

in

T

that

have

different

execution

traces

in

P

and

P

Assume

henceforth

that

T

contains

no

obsolete

tests

either

because

it

contained

initially

or

because

we

have

removed

them

In

addition

assume

that

for

each

test

t

T

P

terminated

and

produced

its

specified

output

when

we

executed

it

with

t

Our

task

is

to

partition

T

as

nearly

as

possible

into

tests

that

have

different

execution

traces

in

P

and

P

and

tests

that

do

not

We

next

present

an

algorithm

that

identifies

modification-traversing

tests

for

intraprocedural

test

selection

Section

presents

our

interprocedural

test

selection

algorithm

Intraprocedural

test

selection

We

begin

the

presentation

of

our

intraprocedural

algorithm

by

informally

motivating

our

approach

Given

an

execution

trace

there

is

a

unique

mapping

between

that

trace

and

the

nodes

of

a

CFG

We

obtain

this

mapping

by

replacing

each

statement

in

the

execution

trace

by

its

corresponding

CFG

node

or

equivalently

by

the

label

of

that

node

We

call

such

a

mapping

a

traversal

trace

and

say

that

for

test

t

with

execution

trace

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

the

traversal

trace

for

t

on

G

is

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

If

N

is

a

node

in

traversal

trace

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

we

define

the

traversal

trace

prefix

for

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

with

respect

to

N

to

be

the

portion

of

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

beginning

with

the

first

node

in

the

trace

and

ending

at

N

are

various

ways

in

which

this

two-phase

process

may

fit

into

the

overall

software

maintenance

process

A

big

bang

process

performs

all

modifications

and

when

these

are

complete

turns

to

regression

testing

An

incremental

process

performs

regression

testing

at

intervals

throughout

the

maintenance

life

cycle

with

each

testing

session

aimed

at

the

product

in

its

current

state

of

evolution

Preliminary

phases

are

typically

shorter

for

the

incremental

model

than

with

the

big

bang

model

however

for

both

models

both

phases

exist

and

can

be

exploited

we

can

not

effectively

determine

test

obsolesence

we

can

not

effectively

judge

test

correctness

Thus

this

assumption

is

necessary

if

we

intend

to

reuse

tests

at

all

whether

selectively

or

not

Given

two

execution

traces

we

can

perform

a

pairwise

comparison

of

the

traces

by

comparing

the

first

statements

in

each

trace

to

each

other

then

comparing

the

second

statements

in

each

trace

to

each

other

and

so

forth

If

a

test

t

has

nonequivalent

execution

traces

in

P

and

P

a

pairwise

comparison

of

the

execution

traces

for

t

in

P

and

P

reaches

a

first

pair

of

statements

that

are

lexically

different

Assume

that

CFG

nodes

contain

as

labels

the

text

of

the

statements

to

which

they

correspond

In

this

case

we

can

perform

a

pairwise

comparison

of

the

nodes

in

two

traversal

traces

by

comparing

node

labels

If

t

has

different

traversal

traces

in

P

and

P

we

shall

reach

a

first

pair

of

nodes

that

have

lexically

different

labels

Suppose

t

has

nonequivalent

execution

traces

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

and

ET

-LRB-

P

-LRB-

t

-RRB-

-RRB-

in

P

and

P

and

let

N

and

N

be

the

first

pair

of

nodes

found

to

differ

during

a

pairwise

comparison

of

the

corresponding

traversal

traces

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

and

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

The

traversal

trace

prefixes

of

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

and

TR

-LRB-

P

-LRB-

t

-RRB-

-RRB-

with

respect

to

N

and

N

respectively

are

equivalent

up

to

but

not

including

N

and

N

In

other

words

if

t

is

modification-traversing

for

P

and

P

there

is

some

pair

of

nodes

N

and

N

in

G

and

such

that

N

and

N

differ

and

N

and

N

are

endpoints

of

traversal

trace

prefixes

that

are

identical

up

to

but

not

including

N

and

N

To

find

tests

that

are

modification-traversing

for

P

and

P

we

can

synchronously

traverse

CFG

paths

that

begin

with

the

entry

nodes

of

G

and

looking

for

pairs

of

nodes

N

and

N

that

differ

When

we

find

such

a

pair

we

use

TestsOnEdge

to

select

all

tests

known

to

have

reached

N

On

first

view

it

may

appear

expensive

to

compare

traversal

traces

because

such

traces

can

be

lengthy

-LCB-

particularly

when

procedures

contain

loops

However

three

facts

combine

to

render

the

approach

efficient

First

when

we

walk

a

pair

of

traversal

trace

prefixes

p

and

in

G

and

respectively

reach

a

pair

of

nodes

N

and

N

that

differ

and

select

the

tests

that

reach

N

we

need

not

walk

beyond

N

and

N

we

have

selected

all

tests

that

have

traversal

trace

prefixes

p

and

and

reach

additional

modifications

beyond

N

and

N

Tests

that

reach

modifications

accessible

along

other

traversal

trace

prefixes

are

selected

when

we

traverse

those

other

prefixes

Second

when

we

reach

in

our

walk

a

pair

of

nodes

N

and

N

that

have

already

been

pairwise

compared

we

do

not

need

to

compare

N

and

N

again

Third

when

we

attach

test

information

to

edges

in

G

a

single

walk

of

G

simultaneously

accounts

for

all

tests

attached

to

each

edge

as

it

crosses

the

edge

we

do

not

require

a

separate

graph

walk

for

each

test

in

T

For

these

reasons

the

graph

walks

required

to

select

modification-traversing

tests

on

CFGs

require

worst-case

time

quadratic

in

the

number

of

nodes

in

the

graphs

The

basic

test

selection

algorithm

Figure

presents

SelectTests

our

intraprocedural

regression

test

selection

algorithm

SelectTests

takes

a

procedure

P

its

modified

version

P

and

the

test

suite

T

for

P

and

returns

T

a

set

that

contains

tests

that

are

modification-traversing

for

P

and

P

SelectTests

first

initializes

T

to

OE

and

then

constructs

CFGs

G

-LRB-

with

entry

node

E

-RRB-

and

-LRB-

with

entry

node

-RRB-

for

P

and

P

respectively

Next

the

algorithm

calls

Compare

with

E

and

Compare

ultimately

places

tests

that

are

modification-traversing

for

P

and

P

into

T

SelectTests

returns

these

tests

earlier

version

of

this

algorithm

-LSB-

-RSB-

was

based

on

control

dependence

graphs

for

P

and

P

The

possibility

of

performing

that

algorithm

on

CFGs

was

suggested

by

Weibao

Wu

-LRB-

personal

communication

-RRB-

Both

approaches

select

safe

test

sets

but

the

CFG-based

approach

is

more

efficient

and

easier

to

implement

than

the

earlier

approach

algorithm

SelectTests

-LRB-

P

P

T

-RRB-

T

input

P

P

base

and

modified

versions

of

a

procedure

T

a

test

set

used

to

test

P

output

T

the

subset

of

T

selected

for

reuse

begin

T

OE

construct

G

and

CFGs

for

P

and

P

with

entry

nodes

E

and

Compare

-LRB-

E

-RRB-

return

T

end

procedure

Compare

-LRB-

N

N

-RRB-

input

N

and

N

nodes

in

G

and

begin

mark

N

N

visited

for

each

successor

C

of

N

in

G

do

L

the

label

on

edge

-LRB-

N

C

-RRB-

the

node

in

such

that

-LRB-

N

-RRB-

has

label

L

if

C

is

not

marked

if

Equivalent

-LRB-

C

-RRB-

T

T

-LSB-

TestsOnEdge

-LRB-

-LRB-

N

C

-RRB-

-RRB-

else

Compare

-LRB-

C

-RRB-

endif

endif

endfor

end

Figure

Algorithm

for

intraprocedural

test

selection

Compare

is

a

recursive

procedure

Compare

is

called

with

pairs

of

nodes

N

and

N

from

G

and

respectively

that

are

reached

simultaneously

during

our

algorithm

's

synchronous

walks

of

traversal

trace

prefixes

Given

two

such

nodes

N

and

N

Compare

determines

whether

N

and

N

have

successors

that

are

syntactically

nonequivalent

along

pairs

of

identically

labeled

edges

If

N

and

N

have

syntactically

nonequivalent

successors

along

some

pair

of

identically

labeled

edges

tests

that

traverse

the

edges

are

modification-traversing

due

to

changes

in

the

code

associated

with

those

successors

In

this

case

Compare

selects

those

tests

If

N

and

N

have

syntactically

equivalent

successors

along

a

pair

of

identically

labeled

edges

Compare

walks

forward

along

the

edges

in

G

and

by

invoking

itself

on

those

successors

Lines

of

Figure

describe

Compare

's

actions

more

precisely

When

Compare

is

called

with

CFG

nodes

N

and

N

Compare

first

marks

node

N

N

visited

-LRB-

line

-RRB-

After

Compare

has

been

called

once

with

N

and

N

it

does

not

need

to

consider

them

again

-LCB-

this

marking

step

lets

Compare

avoid

revisiting

a

pair

of

nodes

Next

in

the

for

loop

of

lines

Compare

considers

each

control

flow

successor

of

N

For

each

successor

C

Compare

locates

the

label

L

on

the

edge

from

N

to

C

then

seeks

the

node

C

in

C

such

that

-LRB-

N

C

-RRB-

has

label

L.

-LRB-

Such

a

C

necessarily

exists

because

Compare

is

called

only

with

pairs

of

nodes

N

and

N

that

are

syntactically

equivalent

thus

N

and

N

are

either

both

predicate

nodes

with

out-edges

labeled

T

and

F

or

N

and

N

are

both

statement

nodes

with

out-edges

labeled

ffl

-RRB-

Next

Compare

considers

C

and

C

If

C

is

marked

C

visited

Compare

has

already

been

called

with

C

and

C

so

Compare

does

not

take

any

action

with

C

and

C

If

C

is

not

marked

C

visited

Compare

calls

Equivalent

with

C

Procedure

if

-LRB-

n

-RRB-

'S

'S

F

TT

F

exit

entry

fread

-LRB-

fileptr

n

-RRB-

count

return

-LRB-

avg

-RRB-

while

-LRB-

not

EOF

-RRB-

do

avg

calcavg

-LRB-

numarray

count

-RRB-

endwhilefread

-LRB-

fileptr

n

-RRB-

endifnumarray

-LSB-

count

-RSB-

n

elsereturn

-LRB-

error

-RRB-

print

-LRB-

bad

input

-RRB-

Figure

Procedure

and

its

CFG

and

C

If

Equivalent

-LRB-

C

C

-RRB-

is

false

then

tests

that

traverse

edge

-LRB-

N

C

-RRB-

are

modification-traversing

for

P

and

P

so

Compare

uses

TestsOnEdge

to

identify

these

tests

and

adds

them

to

T

If

Equivalent

-LRB-

C

C

-RRB-

is

true

Compare

invokes

itself

on

C

and

C

to

continue

the

graph

traversals

beyond

these

nodes

The

Equivalent

function

takes

a

pair

of

nodes

N

and

N

and

determines

whether

the

statements

S

and

associated

with

N

and

N

may

be

semantically

nonequivalent

A

simple

version

of

Equivalent

answers

conservatively

the

question

of

whether

S

and

are

semantically

nonequivalent

by

comparing

the

two

statements

lexicographically

ignoring

extra

white-space

characters

when

not

included

in

character

constants

We

discuss

more

precise

albeit

more

complex

versions

of

Equivalent

in

Section

We

next

consider

several

examples

that

illustrate

how

SelectTests

works

Figure

presents

procedure

and

the

CFG

for

-LCB-

is

a

modified

version

of

procedure

avg

shown

in

Figure

In

statement

has

erroneously

been

deleted

and

statement

has

been

added

When

called

with

avg

and

and

with

test

suite

T

-LRB-

shown

in

Figure

-RRB-

SelectTests

initializes

T

to

OE

constructs

the

CFGs

for

the

two

procedures

and

calls

Compare

with

entry

and

Compare

marks

entry

and

then

considers

the

successor

of

entry

Compare

finds

that

is

the

corresponding

successor

of

and

because

is

not

marked

calls

Equivalent

with

and

Because

and

are

syntactically

equivalent

Equivalent

returns

true

and

Compare

invokes

itself

on

and

-LRB-

invocation

-RRB-

Recursive

calls

continue

in

this

manner

on

nodes

and

-LRB-

invocation

-RRB-

and

and

-LRB-

invocation

-RRB-

in

each

case

the

successors

of

the

nodes

are

equivalent

On

invocation

Compare

must

consider

two

successors

of

P

and

When

Compare

considers

it

calls

itself

with

and

and

subsequently

with

exit

and

and

selects

no

tests

When

Compare

considers

and

it

first

seeks

a

true

child

phrase

may

be

is

important

here

because

the

problem

of

determining

semantic

equivalence

of

statements

is

in

general

undecidable

-LSB-

-RSB-

of

to

compare

with

it

finds

and

calls

Equivalent

with

and

The

statement

associated

with

and

the

statement

associated

with

are

not

syntactically

equivalent

so

Equivalent

returns

false

Compare

uses

TestsOnEdge

-LRB-

-RRB-

to

locate

the

set

of

tests

-LRB-

-RRB-

that

reach

in

avg

and

adds

these

tests

to

T

When

Compare

seeks

a

false

successor

of

it

finds

and

calls

Equivalent

with

and

Equivalent

returns

true

for

these

nodes

so

Compare

invokes

itself

on

and

Compare

finds

the

successors

of

these

nodes

and

nonequivalent

and

adds

to

T

the

set

of

tests

that

traverse

edge

-LRB-

-RRB-

T

is

now

At

this

point

the

algorithm

has

walked

all

traversal

trace

prefixes

either

up

to

modifications

or

up

to

the

exit

node

so

no

further

traversal

is

necessary

recursive

Compare

calls

return

to

the

main

program

and

the

algorithm

returns

set

T

in

which

both

tests

are

modification-traversing

Many

other

regression

test

selection

techniques

-LSB-

-RSB-

omit

or

If

for

avg

and

the

deletion

of

had

been

the

only

change

SelectTests

would

have

returned

only

If

the

addition

of

had

been

the

only

change

SelectTests

would

have

returned

only

In

this

latter

case

observe

that

Compare

would

eventually

invoke

itself

with

and

but

find

the

successor

of

P

already

marked

thus

Compare

would

not

reinvoke

itself

with

and

To

see

how

SelectTests

handles

changes

in

predicate

statements

consider

the

result

when

line

in

procedure

avg

is

also

changed

-LRB-

erroneously

-RRB-

to

n

This

change

alters

only

the

text

associated

with

node

in

's

CFG

In

this

case

called

with

the

CFGs

for

avg

and

SelectTests

proceeds

as

in

the

previous

example

until

it

reaches

and

Here

it

finds

nonequivalent

successors

and

and

selects

The

procedure

does

not

need

to

analyze

successors

of

and

To

see

how

SelectTests

handles

large-scale

structural

changes

consider

the

result

when

new

error

handling

code

is

inserted

into

avg

such

that

the

procedure

checks

fileptr

as

its

first

action

and

executes

the

rest

of

its

statements

only

if

fileptr

is

not

SelectTests

detects

this

change

on

the

initial

invocation

of

Compare

when

Compare

detects

the

differences

between

successors

of

entry

and

The

procedure

does

not

need

to

analyze

successors

of

entry

and

it

returns

the

entire

test

set

T

because

all

tests

in

T

are

modification-traversing

To

understand

why

at

line

SelectTests

marks

C

C

visited

rather

than

just

visited

consider

Figure

The

figure

contains

procedure

twovisits

-LRB-

far

left

-RRB-

a

modified

version

of

that

procedure

-LRB-

far

right

-RRB-

and

the

CFGs

for

the

two

procedures

-LRB-

next

to

their

respective

procedures

-RRB-

The

two

versions

produce

identical

output

for

all

values

of

x

other

than

When

x

the

execution

trace

for

twovisits

is

h

entry

P

exit

i

and

the

procedure

prints

For

the

same

input

the

execution

trace

for

is

h

i

and

the

procedure

prints

When

SelectTests

runs

on

the

CFGs

for

twovisits

and

it

considers

entry

and

first

and

then

invokes

itself

on

and

Suppose

SelectTests

next

invokes

itself

on

and

-LRB-

we

have

not

required

the

algorithm

to

visit

the

successors

of

a

pair

of

nodes

in

any

particular

order

if

we

had

we

could

reverse

the

contents

of

the

if

and

else

clauses

in

this

example

and

still

make

the

point

that

we

are

about

to

make

-RRB-

SelectTests

marks

then

continues

with

invocations

on

and

and

exit

and

selecting

no

tests

because

tests

that

take

these

paths

in

the

two

versions

are

non-modification-traversing

Now

SelectTests

resumes

its

consideration

of

successors

of

and

and

'S

entry

'S

entry

-LCB-

-RCB-

twovisits

-LRB-

x

-RRB-

if

-LRB-

x

-RRB-

else

endif

print

-LRB-

-RRB-

goto

-LCB-

'S

'S

'S

'S

-RCB-

print

-LRB-

-RRB-

twovisits

-LRB-

x

-RRB-

if

-LRB-

x

-RRB-

else

goto

endifexit

print

-LRB-

-RRB-

exit

exit

Figure

Procedures

twovisits

and

and

their

CFGs

invokes

itself

on

and

SelectTests

marks

and

considers

the

successors

of

has

only

one

successor

the

corresponding

successor

of

is

Here

is

the

point

we

wish

to

make

If

on

visiting

SelectTests

had

simply

marked

visited

and

now

seeing

that

had

been

visited

declined

to

visit

it

again

SelectTests

would

not

compare

and

and

would

not

detect

the

need

to

select

tests

through

However

as

the

algorithm

is

written

SelectTests

sees

that

is

only

marked

not

and

thus

proceeds

to

check

for

equivalence

of

and

and

select

the

necessary

tests

In

certain

cases

it

is

possible

for

SelectTests

to

select

tests

that

are

not

modification-traversing

for

P

and

P

Figure

which

illustrates

this

possibility

depicts

a

C

function

pathological

and

a

modified

version

of

that

function

with

the

CFGs

for

the

versions

Each

while

construct

in

the

versions

first

increments

the

value

of

x

and

then

tests

the

incremented

value

to

determine

whether

to

enter

or

exit

its

loop

Suppose

test

suite

T

for

pathological

contains

tests

and

that

use

input

values

and

respectively

to

exercise

the

function

Figure

shows

the

inputs

outputs

and

traversal

traces

that

result

when

we

run

pathological

and

on

these

tests

When

we

run

pathological

on

tests

and

it

outputs

for

both

When

we

run

on

test

it

also

outputs

however

when

we

run

on

test

it

outputs

Tests

and

both

traverse

edge

-LRB-

-RRB-

in

the

CFG

for

pathological

Consider

the

actions

of

SelectTests

invoked

on

pathological

and

Called

with

entry

and

Compare

invokes

itself

with

and

then

with

and

When

invoked

with

and

Compare

considers

their

successors

and

respectively

finds

them

lexically

identical

and

invokes

itself

with

them

Compare

finds

their

successors

and

nonequivalent

and

selects

the

tests

on

edge

-LRB-

-RRB-

that

is

tests

and

The

problem

with

this

test

selection

is

that

whereas

is

modification-traversing

for

the

two

versions

of

pathological

is

not

modification-traversing

for

the

two

versions

Figure

shows

the

traversal

traces

is

not

surprising

because

the

problem

of

precisely

identifying

these

tests

in

general

is

PSPACE-hard

thus

unless

P

NP

no

efficient

algorithm

will

always

identify

precisely

the

tests

that

are

modification-traversing

for

P

and

P

-LSB-

-RSB-

while

-LRB-

x

-RRB-

printf

-LRB-

d

x

-RRB-

-LCB-

while

-LRB-

x

-RRB-

-LCB-

-RCB-

-RCB-

-LCB-

-LCB-

-RCB-

-RCB-

pathological

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

F

entry

exit

FT

TF

FT

T

entry'P

exit

F

F

T

FT

-LCB-

-RCB-

while

-LRB-

x

-RRB-

-LCB-

-LCB-

-RCB-

-RCB-

printf

-LRB-

d

x

-RRB-

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

-LCB-

-RCB-

-LCB-

-RCB-

pathological

-LRB-

x

-RRB-

Figure

Procedures

pathological

and

and

their

CFGs

input

entry

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

entry

while

-LRB-

x

-RRB-

printf

-LRB-

d'

x

-RRB-

exit

while

-LRB-

x

-RRB-

printf

-LRB-

d'

x

-RRB-

exit

pathological

traversal

traces

-LRB-

showing

node

labels

-RRB-

for

procedure

input

entry

while

-LRB-

x

-RRB-

printf

-LRB-

d'

x

-RRB-

exit

entry

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

while

-LRB-

x

-RRB-

printf

-LRB-

d'

x

-RRB-

exit

traversal

traces

-LRB-

showing

node

labels

-RRB-

for

procedure

test

pathological

Figure

Test

suites

and

traversal

traces

for

pathological

and

which

correspond

to

the

execution

traces

for

the

tests

on

the

two

versions

Test

has

equivalent

execution

traces

for

the

two

versions

whereas

test

does

not

the

traces

for

differ

in

their

fifth

terms

Thus

is

not

modification-traversing

for

the

two

versions

and

SelectTests

chooses

it

unnecessarily

Efficiency

effectiveness

and

correctness

of

the

algorithm

In

this

section

we

summarize

results

that

are

presented

in

detail

in

Reference

-LSB-

-RSB-

We

define

controlled

regression

testing

as

the

practice

of

testing

P

under

conditions

equivalent

to

those

that

were

used

to

test

P

As

such

controlled

regression

testing

applies

scientific

method

to

regression

testing

to

determine

whether

code

modifications

cause

errors

we

test

the

new

code

holding

all

other

factors

that

might

affect

program

behavior

constant

For

the

purpose

of

regression

test

selection

we

would

like

to

identify

all

tests

t

T

that

cause

P

to

produce

output

that

differs

from

its

specified

output

and

thus

reveal

faults

in

P

We

call

such

tests

faultrevealing

and

we

call

an

algorithm

that

selects

every

fault-revealing

test

in

T

safe

There

is

no

effective

procedure

that

in

general

precisely

identifies

the

fault-revealing

tests

in

T

However

under

controlled

regression

testing

the

modification-traversing

tests

are

a

superset

of

the

fault-revealing

tests

Thus

for

cases

where

controlled

regression

testing

is

possible

a

regression

test

selection

algorithm

that

selects

all

modification-traversing

tests

necessarily

selects

all

fault-revealing

tests

and

is

safe

This

result

is

significant

because

we

can

show

that

for

cases

where

controlled

regression

testing

is

practiced

if

t

T

is

modificationtraversing

then

SelectTests

selects

t.

Thus

the

following

theorem

holds

Theorem

SelectTests

is

safe

for

controlled

regression

testing

Proof

See

Reference

-LSB-

-RSB-

pages

Theorem

is

important

because

although

we

can

not

find

an

algorithm

that

is

safe

in

general

we

know

from

the

theorem

that

SelectTests

is

safe

in

situations

in

which

controlled

regression

testing

is

possible

Theorem

lets

us

draw

conclusions

about

the

ability

of

our

technique

to

select

tests

that

may

reveal

faults

However

we

are

also

interested

in

how

well

our

technique

does

at

omitting

tests

that

can

not

reveal

faults

For

controlled

regression

testing

tests

that

are

non-modification-traversing

can

not

be

fault-revealing

ideally

we

would

like

to

omit

all

such

In

the

previous

section

however

we

showed

that

there

are

cases

in

which

SelectTests

selects

tests

that

are

non-modification-traversing

The

theorem

identifies

a

necessary

condition

for

SelectTests

to

select

such

tests

the

multiply-visited-node

condition

The

multiply-visitednode

condition

holds

for

P

and

P

if

and

only

if

SelectTests

run

on

P

and

P

marks

some

node

in

P

X-visited

for

more

than

one

node

X

in

the

CFG

for

P

Our

next

theorem

is

as

follows

Theorem

Given

procedure

P

modified

version

P

and

test

suite

T

for

P

if

the

multiplyvisited-node

condition

does

not

hold

for

P

and

P

and

SelectTests

selects

test

set

T

for

P

then

every

test

in

T

is

modification-traversing

for

P

and

P

Proof

see

Reference

-LSB-

-RSB-

pages

Theorem

gives

us

a

way

to

characterize

the

class

of

programs

and

modified

versions

for

which

SelectTests

selects

non-modification-traversing

tests

The

theorem

is

significant

for

two

reasons

First

procedures

like

pathological

and

which

cause

the

multiply-visited-node

condition

to

hold

are

atypical

With

some

work

we

can

construct

other

examples

that

illustrate

similar

effects

however

all

examples

located

to

date

have

been

contrived

Second

in

our

empirical

studies

using

real

programs

we

have

never

found

a

case

in

which

the

multiply-visited-node

condition

holds

Thus

we

conjecture

that

in

practice

given

P

P

and

T

SelectTests

selects

exactly

the

tests

in

T

that

are

modification-traversing

for

P

and

P

earlier

work

-LSB-

-RSB-

we

used

the

phrase

proper

regression

testing

in

this

context

however

the

term

proper

has

a

pejorative

connotation

that

we

do

not

intend

to

imply

Hence

in

this

work

we

substitute

the

more

neutral

term

controlled

importance

of

controlled

regression

testing

is

stated

well

by

Beizer

-LSB-

-RSB-

who

writes

It

must

be

possible

to

precisely

recreate

the

entire

test

situation

or

else

it

may

be

impossible

to

resolve

some

of

the

nastiest

configuration

dependent

bugs

that

show

up

in

the

field

Our

final

theorem

is

as

follows

Theorem

SelectTests

terminates

Proof

The

proof

proceeds

by

showing

that

-LRB-

-RRB-

the

number

of

recursive

calls

made

to

Compare

is

bounded

and

-LRB-

-RRB-

the

work

required

by

a

call

to

Compare

is

bounded

See

Reference

-LSB-

-RSB-

page

for

details

Complexity

of

the

algorithm

The

running

time

of

SelectTests

is

bounded

by

the

time

required

to

construct

CFGs

for

P

and

P

plus

the

number

and

cost

of

calls

to

Compare

Let

n

be

the

number

of

statements

in

P

and

the

number

of

statements

in

P

CFG

construction

is

an

O

-LRB-

n

-RRB-

operation

-LSB-

-RSB-

We

obtain

an

upper

bound

on

the

number

of

calls

to

Compare

by

assuming

that

Compare

can

be

called

with

each

pair

of

nodes

N

and

N

in

G

and

respectively

Under

this

assumption

the

overall

cost

of

SelectTests

is

O

-LRB-

n

m

-LRB-

-RRB-

-RRB-

where

m

is

the

cost

of

a

call

to

Compare

Each

call

to

Compare

results

in

an

examination

of

at

most

two

edges

-LRB-

T

and

F

-RRB-

at

line

and

thus

two

calls

to

Equivalent

-LRB-

line

-RRB-

Depending

on

the

results

of

the

Equivalent

operation

the

call

to

Compare

results

in

either

a

set

union

operation

-LRB-

line

-RRB-

or

an

examination

of

at

most

two

successors

of

N

-LRB-

line

-RRB-

The

set

union

task

implemented

as

a

bit

vector

operation

has

a

worst-case

cost

proportional

to

the

number

of

tests

in

T

The

Equivalent

procedure

has

a

cost

that

is

linear

in

the

number

of

characters

in

the

statements

compared

for

practical

purposes

this

size

is

bounded

by

a

constant

-LRB-

the

maximum

line

length

present

in

the

procedure

-RRB-

Thus

m

in

the

above

equation

is

bounded

by

kjT

j

for

some

constant

k

It

follows

that

given

a

pair

of

procedures

for

which

CFGs

G

and

contain

n

and

nodes

respectively

and

given

a

test

suite

of

t

tests

if

Compare

is

called

for

each

pair

of

nodes

-LRB-

N

N

-RRB-

-LRB-

N

G

and

N

-RRB-

the

running

time

of

SelectTests

is

O

-LRB-

-RRB-

The

assumption

that

Compare

may

be

called

for

each

pair

of

nodes

N

and

N

from

G

and

applies

only

to

procedures

P

and

P

for

which

the

multiply-visited-node

condition

holds

Program

pathological

of

Figure

illustrates

a

case

in

which

that

condition

holds

in

that

example

for

graphs

of

and

nodes

respectively

the

algorithm

makes

calls

to

Compare

When

the

multiply-visited-node

condition

does

not

hold

however

Compare

is

called

at

most

minfn

times

Thus

if

our

conjecture

that

the

multiply-visitednode

condition

does

not

hold

in

practice

is

correct

then

for

practical

purposes

SelectTests

runs

in

time

O

-LRB-

t

-LRB-

minfn

-RRB-

-RRB-

Improvements

to

the

basic

algorithm

In

the

preceding

sections

to

facilitate

the

presentation

we

presented

a

deliberately

simplified

version

of

SelectTests

There

are

several

ways

in

which

to

increase

the

efficiency

or

precision

of

the

basic

algorithm

We

discuss

three

improvements

here

Reference

-LSB-

-RSB-

discusses

additional

improvements

Determining

statement

nonequivalence

more

precisely

The

Compare

procedure

that

we

presented

in

Figure

relies

on

a

function

Equivalent

that

takes

a

pair

of

nodes

N

and

N

and

determines

whether

the

statements

S

and

associated

with

N

and

N

may

be

semantically

nonequivalent

To

retain

safety

Equivalent

must

conservatively

estimate

whether

pairs

of

statements

are

semantically

nonequivalent

erring

only

on

the

side

of

claiming

nonequivalence

when

the

statements

are

in

fact

equivalent

A

simple

version

of

Equivalent

performs

a

lexicographic

comparison

of

S

and

ignoring

white-space

characters

except

when

they

are

part

of

character

constants

This

approach

is

safe

and

easy

to

implement

but

also

imprecise

for

several

obvious

cases

Consider

for

example

the

statements

if

-LRB-

x

-RRB-

and

if

-LRB-

x

-RRB-

Clearly

the

two

statements

are

lexicographically

nonidentical

but

semantically

equivalent

The

same

applies

to

the

statements

x

-LRB-

n

n

-RRB-

-LRB-

n

-RRB-

and

x

-LRB-

-RRB-

-LRB-

-RRB-

and

to

the

statements

x

comment

and

x

different

comment

A

more

sophisticated

implementation

of

Equivalent

can

detect

for

cases

like

these

whether

statements

are

semantically

nonequivalent

For

example

Equivalent

can

easily

ignore

comments

when

it

performs

lexicographic

comparisons

of

statements

Additionally

for

certain

classes

of

expressions

such

as

assignment

statements

Equivalent

can

determine

equivalence

using

techniques

such

as

polynomial

interpolation

or

symbolic

computation

-LSB-

-RSB-

At

further

expense

Equivalent

can

use

algorithms

such

as

those

described

in

References

-LSB-

-RSB-

that

conservatively

determine

statement

equivalence

Improving

Equivalent

increases

the

precision

of

our

test

selection

algorithms

but

also

increases

the

cost

of

the

algorithms

In

practice

we

expect

that

some

improvements

to

Equivalent

will

produce

precision

improvements

sufficient

to

justify

this

added

cost

while

others

will

not

Handling

variable

and

type

declarations

A

second

test

selection

issue

involves

variable

and

type

declarations

A

change

in

a

variable

or

type

declaration

may

render

a

test

fault-revealing

even

though

that

test

executes

no

changed

program

statements

other

than

the

declaration

For

example

in

a

Fortran

program

changing

the

type

of

a

variable

from

Real

to

Real

can

cause

the

program

to

fail

even

in

the

absence

of

direct

alterations

to

executable

code

A

simple

way

to

handle

variable

and

type

declarations

in

the

context

of

our

intraprocedural

test

selection

algorithm

is

to

associate

each

declaration

with

a

CFG

node

and

to

attach

every

test

that

enters

the

procedure

to

the

edge

that

enters

that

node

An

analogous

technique

handles

the

interprocedural

case

In

this

case

our

algorithms

detect

syntactic

differences

between

variable

and

type

declarations

and

flag

tests

that

encounter

these

differences

as

modification-traversing

This

approach

has

drawbacks

Just

as

a

modified

variable

initialization

statement

may

unnecessarily

force

selection

of

all

tests

through

a

procedure

so

a

new

modified

or

deleted

variable

or

type

declaration

may

unnecessarily

force

selection

of

all

tests

If

a

declaration

of

variable

v

changes

the

only

tests

that

can

be

fault-revealing

due

to

such

a

change

-LRB-

for

controlled

regression

testing

-RRB-

are

tests

that

reach

some

statement

in

the

executable

portion

of

the

procedure

or

program

that

contains

a

reference

to

the

memory

associated

with

v

If

a

declaration

of

type

t

changes

the

only

tests

that

can

be

fault-revealing

due

to

such

a

change

-LRB-

for

controlled

regression

testing

-RRB-

are

tests

that

reach

some

statement

in

the

executable

portion

of

the

procedure

or

program

that

contains

a

reference

to

the

memory

associated

with

some

variable

whose

type

is

based

on

t

Our

test

selection

algorithms

can

be

modified

to

reduce

imprecise

test

selection

at

declaration

changes

One

approach

requires

the

algorithm

to

determine

affected

variables

which

are

variables

whose

declarations

have

changed

or

whose

declarations

are

dependent

on

changed

type

definitions

and

keep

a

list

of

these

variables

The

Equivalent

procedure

uses

this

list

to

detect

occurrences

of

affected

variables

and

report

statements

that

contain

references

to

the

memory

locations

associated

with

those

variables

as

modified

The

modified

algorithm

postpones

test

selection

until

it

locates

statements

that

contain

affected

references

it

then

selects

only

tests

that

reach

those

statements

An

alternative

representation

for

case

statements

Instead

of

representing

case

statements

as

a

series

of

nested

if-else

statements

we

can

represent

them

as

a

set

of

choices

incident

on

a

single

predicate

node

that

has

a

labeled

out-edge

for

each

separate

case

This

representation

can

yield

more

precise

test

selection

than

the

nested

if-else

representation

A

principal

difference

in

this

representation

is

that

for

switch

predicates

the

set

of

labeled

out-edges

may

vary

from

P

to

P

if

cases

are

added

or

removed

If

a

case

is

added

to

a

switch

then

all

tests

in

T

that

took

the

default

case

edge

in

P

can

conceivably

take

the

branch

to

the

new

case

in

P

thus

Compare

looks

for

new

edges

and

when

it

detects

them

it

adds

tests

that

formerly

traversed

the

default

edge

to

T

If

a

case

is

removed

from

a

switch

Compare

detects

the

missing

labeled

edge

in

and

selects

all

tests

that

took

the

edge

in

G.

With

minor

modifications

our

Compare

routine

copes

with

these

differences

and

handles

the

alternative

representation

of

switches

Interprocedural

test

selection

Thus

far

we

have

considered

our

technique

in

an

intraprocedural

context

Our

examples

demonstrate

that

the

technique

reduces

the

number

of

tests

that

must

be

rerun

on

a

single

procedure

However

the

examples

also

suggest

that

when

procedures

are

small

and

uncomplicated

and

their

test

sets

are

small

it

may

be

more

cost

effective

to

rerun

all

tests

This

objection

is

mitigated

when

we

consider

interprocedural

testing

Test

sets

for

subsystems

and

complete

programs

are

typically

much

larger

than

test

sets

for

single

procedures

In

this

context

the

savings

that

can

result

from

selective

retest

increases

Moreover

in

the

interprocedural

context

the

problem

of

regression

test

selection

is

even

more

differentiable

from

the

problem

of

coverage

identification

than

in

the

intraprocedural

context

Structural

coverage

criteria

if

employed

at

all

are

typically

applied

only

during

intraprocedural

testing

Although

integration

and

system-level

coverage

criteria

have

been

proposed

-LSB-

-RSB-

in

practice

most

system

and

integration

testing

is

functional

or

specification-based

It

is

precisely

with

functional

and

specification-based

tests

that

we

are

most

interested

in

running

all

tests

that

may

reveal

faults

without

regard

to

coverage

considerations

It

is

precisely

with

functional

and

specification-based

tests

that

we

can

benefit

most

from

an

automatable

regression

test

selection

technique

that

selects

tests

based

on

program

structure

because

we

can

not

in

general

determine

tests

that

are

not

fault-revealing

simply

by

referring

to

program

specifications

Assume

that

we

can

obtain

a

mapping

of

procedure

names

in

P

to

procedure

names

in

P

that

tells

us

for

each

procedure

P

P

the

name

P

of

its

counterpart

in

P

A

simple

but

naive

approach

to

interprocedural

Bexit

main

enter

C

exit

A

enter

B

enter

C

call

A

call

B

enter

main

if

-LRB-

x

-RRB-

Figure

Program

sys

that

illustrates

interprocedural

test

selection

effects

regression

test

selection

runs

the

intraprocedural

SelectTests

algorithm

on

every

pair

of

procedures

-LRB-

P

P

-RRB-

such

that

P

is

the

counterpart

of

P

With

this

simple

approach

if

a

procedure

P

P

is

no

longer

present

in

P

tests

that

used

to

enter

P

are

selected

at

former

call

sites

to

P

Similarly

if

a

procedure

P

is

inserted

into

P

tests

that

enter

P

are

selected

at

the

call

sites

to

P

Notice

that

the

number

of

times

P

is

called

from

within

P

is

immaterial

trace

information

tells

us

precisely

which

tests

reach

which

edges

in

P

and

a

single

traversal

of

P

and

P

suffices

to

find

tests

that

are

modification-traversing

for

P

and

P

If

we

run

SelectTests

on

all

pairs

of

corresponding

procedures

in

P

and

P

we

obtain

a

safe

test

set

T

but

we

fail

to

take

advantage

of

several

opportunities

for

improvements

in

efficiency

To

describe

these

opportunities

we

refer

to

the

CFGs

for

the

procedures

in

a

program

sys

shown

in

Figure

Program

sys

contains

four

procedures

main

is

the

entry

point

to

the

program

and

A

B

and

C

are

invoked

when

the

program

runs

Notice

that

A

is

recursive

Suppose

statement

in

sys

is

modified

creating

a

new

version

of

sys

In

this

case

all

tests

in

T

are

modification-traversing

for

sys

and

and

are

selected

when

SelectTests

called

with

main

and

reaches

in

main

There

is

no

need

in

this

case

to

compare

procedures

A

B

and

C

to

their

counterparts

in

we

have

already

selected

all

tests

that

could

reach

those

procedures

and

become

modification-traversing

within

them

The

naive

algorithm

does

unnecessary

work

for

this

example

Alternatively

suppose

statement

-LRB-

and

no

other

statement

-RRB-

in

sys

is

modified

In

this

case

because

every

test

of

sys

that

enters

main

necessarily

reaches

the

call

to

A

in

main

then

reaches

the

call

to

C

in

A

every

test

of

sys

becomes

modification-traversing

in

C

and

There

is

no

need

in

this

case

to

compare

B

with

its

counterpart

in

and

no

need

to

traverse

portions

of

the

CFG

for

main

beyond

the

call

to

A

or

portions

of

the

CFG

for

A

beyond

the

call

to

C.

Here

too

the

naive

algorithm

does

unnecessary

work

These

observations

motivate

an

algorithm

for

interprocedural

test

selection

by

CFG

traversal

that

begins

by

examining

the

entry

procedures

for

P

and

P

When

the

algorithm

reaches

call

nodes

it

immediately

enters

the

graphs

for

the

called

procedures

if

their

entry

nodes

have

not

previously

been

visited

If

the

algorithm

has

previously

completed

its

walk

of

a

procedure

it

uses

the

knowledge

gained

during

that

walk

of

whether

tests

that

enter

the

procedure

can

exit

it

without

becoming

modification-traversing

the

algorithm

continues

its

graph

traversal

beyond

call

nodes

if

and

only

if

tests

can

pass

through

called

procedures

without

becoming

modification-traversing

The

basic

interprocedural

test

selection

algorithm

Figure

gives

our

algorithm

SelectInterTests

for

selecting

tests

for

subsystems

or

programs

The

algorithm

uses

procedures

and

which

are

similar

to

the

intraprocedural

test

selection

procedures

SelectTests

and

Compare

respectively

The

algorithm

also

keeps

data

structure

proctable

that

records

the

name

of

each

procedure

encountered

in

the

traversal

of

the

graph

for

P

in

a

name

field

and

keeps

a

status

flag

for

each

procedure

that

if

defined

can

have

value

visited

or

selectsall

SelectTests

first

initializes

T

and

proctable

to

OE

and

invokes

on

the

entry

procedures

PE

and

P

E

of

the

two

programs

Like

SelectTests

takes

two

procedures

P

and

P

as

input

and

locates

tests

that

are

modification-traversing

for

those

procedures

However

begins

by

inserting

P

into

proctable

and

setting

the

status

flag

for

P

to

visited

to

indicate

that

a

traversal

of

P

has

begun

The

procedure

then

creates

CFGs

for

P

and

P

and

calls

with

the

entry

nodes

of

those

CFGs

When

control

returns

from

to

checks

to

see

whether

the

exit

node

of

P

was

reached

during

the

traversal

of

If

not

tests

that

enter

P

become

modificationtraversing

on

every

path

through

P

thus

there

is

no

point

in

visiting

nodes

beyond

calls

to

P

To

note

this

fact

sets

the

status

flag

for

P

in

proctable

to

selectsall

is

similar

to

Compare

except

that

when

finds

two

nodes

C

and

C

equivalent

before

it

invokes

itself

on

those

nodes

it

checks

to

see

if

C

contains

any

calls

If

C

contains

calls

it

may

be

appropriate

to

invoke

on

the

called

procedures

-LRB-

It

is

not

necessary

to

check

C

for

calls

at

this

point

in

the

algorithm

C

and

C

have

already

been

compared

and

found

lexically

identical

thus

C

and

C

are

equivalent

in

terms

of

calls

-RRB-

Compare

examines

the

status

flag

for

each

procedure

O

called

in

C

if

status

is

visited

or

selectsall

then

O

has

been

-LRB-

or

in

the

former

case

may

in

the

case

of

recursive

calls

be

in

the

process

of

being

-RRB-

traversed

and

there

is

no

need

to

invoke

on

O

and

again

Finally

if

any

procedure

called

in

C

has

its

status

flag

set

to

selectsall

then

all

tests

through

that

procedure

-LRB-

and

thus

all

tests

through

C

-RRB-

have

been

selected

and

there

is

no

need

to

compare

successors

of

C

and

C

In

this

fashion

SelectInterTests

processes

pairs

of

procedures

from

base

and

modified

programs

By

beginning

with

the

entry

procedures

and

processing

called

procedures

only

when

it

reaches

calls

to

those

procedures

the

algorithm

avoids

analyzing

procedures

when

calls

to

those

procedures

occur

only

subsequent

to

code

changes

Furthermore

the

algorithm

avoids

traversing

portions

of

graphs

that

lie

beyond

calls

to

procedures

through

which

all

tests

are

modification-traversing

Unlike

the

naive

algorithm

that

processes

every

pair

of

procedures

in

P

and

P

SelectInterTests

requires

no

mapping

between

procedure

names

in

P

and

P

-LRB-

provided

P

and

P

compile

and

link

-RRB-

If

algorithm

SelectInterTests

-LRB-

P

P

PE

P

E

T

-RRB-

T

input

P

P

base

and

modified

versions

of

a

program

or

subsystem

input

PE

P

E

entry

procedures

to

P

and

P

T

a

test

set

used

previously

to

test

P

output

T

the

subset

of

T

selected

for

reuse

data

proctable

contains

fields

name

and

status

begin

T

OE

proctable

OE

-LRB-

PE

P

E

-RRB-

return

T

end

algorithm

-LRB-

P

P

-RRB-

input

P

P

base

and

modified

versions

of

a

procedure

begin

add

P

to

proctable

setting

its

status

to

visited

construct

G

and

CFGs

for

P

and

P

with

entry

nodes

E

and

-LRB-

E

-RRB-

if

the

exit

node

in

G

is

not

marked

S-visited

for

some

node

S

in

set

the

status

flag

for

P

to

selectsall

endif

end

procedure

-LRB-

N

N

-RRB-

input

N

and

N

nodes

in

G

and

begin

mark

N

N

visited

for

each

successor

C

of

N

in

G

do

L

the

label

on

edge

-LRB-

N

C

-RRB-

the

node

in

such

that

-LRB-

N

-RRB-

has

label

L

if

C

is

not

marked

if

Equivalent

-LRB-

C

-RRB-

T

T

-LSB-

TestsOnEdge

-LRB-

-LRB-

N

C

-RRB-

-RRB-

else

for

each

procedure

O

called

in

C

do

if

O

proctable

or

status

for

O

is

not

visited

or

selectsall

-LRB-

O

-RRB-

endif

endfor

if

any

procedures

called

in

C

do

not

have

status

flag

selectsall

-LRB-

C

-RRB-

endif

endif

endif

endfor

end

Figure

Algorithm

for

interprocedural

test

selection

procedure

foo

in

P

is

deleted

from

P

statements

in

P

that

contain

calls

to

foo

must

be

changed

for

P

In

this

case

when

reaches

a

node

in

G

that

corresponds

to

a

statement

in

which

foo

is

called

selects

all

tests

that

reach

the

node

and

does

not

invoke

on

foo

Thus

it

is

not

possible

for

in

line

to

be

unable

to

find

a

counterpart

for

foo

in

P

The

cases

where

foo

is

not

present

in

P

but

is

added

to

P

and

where

foo

is

renamed

for

P

are

handled

similarly

The

improvements

to

our

basic

intraprocedural

algorithm

discussed

in

Section

also

apply

to

our

interprocedural

algorithm

To

illustrate

the

use

of

SelectInterTests

we

offer

the

following

example

Suppose

we

change

program

sys

of

Figure

to

by

modifying

the

code

associated

with

node

in

procedure

B.

-LRB-

We

do

not

show

the

modified

graphs

to

discuss

the

example

we

distinguish

nodes

in

the

CFG

for

sys

from

nodes

in

the

CFG

for

by

adding

primes

to

them

-RRB-

Initially

SelectInterTests

calls

with

main

and

adds

main

to

proctable

with

status

visited

creates

the

CFGs

for

the

two

procedures

then

invokes

with

the

entry

nodes

of

those

graphs

begins

traversing

the

graphs

and

on

reaching

nodes

call

A

and

call

because

A

is

not

listed

in

proctable

invokes

on

A

and

adds

A

to

proctable

with

status

visited

then

builds

the

CFGs

for

the

two

procedures

and

begins

traversing

them

On

reaching

calls

to

C

the

algorithm

adds

C

to

to

proctable

with

status

visited

makes

CFGs

for

C

and

and

begins

traversing

them

The

algorithm

finds

no

calls

or

differences

in

C

and

and

thus

when

the

call

to

Compare

with

the

entry

nodes

of

C

and

terminates

the

algorithm

marks

the

exit

node

of

C

This

marking

means

that

tests

entering

C

and

pass

through

unaffected

so

does

not

set

the

status

flag

for

C

to

selectsall

On

returning

from

the

call

to

with

C

and

Compare

resumes

at

line

finds

that

the

status

flag

is

not

set

and

continues

traversing

A

and

by

invoking

itself

with

the

call

C

node

and

its

counterpart

in

the

graph

for

The

traversal

eventually

reaches

the

call

to

A

in

the

if

predicate

here

finds

that

A

has

status

visited

and

does

not

reinvoke

on

A

thus

handling

the

recursion

On

reaching

the

call

to

B

in

the

else

clause

of

the

predicate

invokes

with

B

and

This

invocation

ultimately

identifies

the

differences

in

B

and

and

selects

tests

that

reach

the

modified

code

Furthermore

because

the

code

difference

prevents

from

reaching

the

exit

B

node

when

the

call

to

Compare

with

the

entry

nodes

of

B

and

returns

sets

the

Status

flag

for

B

to

selectsall

all

tests

that

enter

B

are

modification-traversing

On

returning

from

the

call

to

for

A

and

resumes

at

line

with

the

call

A

node

in

main

and

its

corresponding

node

in

The

algorithm

traverses

the

graphs

through

node

and

its

counterpart

in

Here

when

it

examines

successors

of

the

nodes

Compare

notes

that

the

status

flag

for

B

is

set

to

selectsall

and

thus

does

not

reinvoke

on

B

and

Furthermore

sees

that

all

procedures

called

in

the

call

B

node

have

status

selectsall

and

thus

does

not

traverse

the

graph

for

main

any

further

Complexity

of

the

algorithm

SelectTests

and

SelectInterTests

are

of

comparable

complexity

To

see

this

suppose

P

contains

p

procedures

and

n

statements

of

which

c

contain

procedure

calls

and

suppose

P

contains

statements

Assume

that

the

number

of

procedure

calls

in

a

single

statement

and

the

length

of

a

statement

are

bounded

by

constants

and

respectively

We

obtain

an

upper

bound

on

the

running

time

of

SelectInterTests

by

considering

the

case

where

P

and

P

are

identical

in

this

case

the

algorithm

builds

and

walks

CFGs

for

every

procedure

in

P

and

its

corresponding

procedure

in

P

In

this

worst

case

regardless

of

the

value

of

p

the

time

required

to

build

CFGs

for

all

procedures

in

P

is

O

-LRB-

n

-RRB-

and

the

time

required

to

build

the

CFGs

for

all

procedures

in

P

is

O

-LRB-

-RRB-

We

obtain

an

upper

bound

on

the

number

of

calls

to

by

assuming

that

every

node

in

P

must

be

compared

to

every

node

in

P

as

might

happen

if

P

and

P

contain

single

procedures

in

this

case

the

number

of

calls

is

O

-LRB-

-RRB-

Excluding

for

the

moment

the

cost

of

the

P

rocT

able

lookups

in

lines

a

call

to

requires

the

same

amount

of

work

as

a

call

to

Compare

namely

j.

Regardless

of

the

number

of

calls

to

Compare

lines

over

the

course

of

a

complete

execution

of

SelectInterTests

require

at

most

table

lookups

on

a

table

that

contains

at

most

p

entries

using

a

naive

table

lookup

algorithm

the

lines

require

O

-LRB-

cp

-RRB-

string

comparisons

where

the

time

for

each

comparison

is

bounded

by

Thus

the

time

required

by

SelectInterTests

in

the

worst

case

is

O

-LRB-

n

jT

cp

-RRB-

where

cp

is

bounded

above

by

In

practice

however

we

expect

a

lower

bound

on

execution

time

When

the

multiply-visited-node

condition

does

not

hold

the

term

jT

becomes

jT

j

-LRB-

minfn

-RRB-

Furthermore

if

we

use

an

efficient

hashing

scheme

to

implement

P

rocT

able

we

reduce

the

O

-LRB-

cp

-RRB-

table

lookup

time

to

-LRB-

expected

time

-RRB-

O

-LRB-

c

-RRB-

which

is

O

-LRB-

n

-RRB-

Evaluations

of

the

Algorithms

In

this

section

we

evaluate

our

algorithms

and

compare

them

to

existing

test

selection

techniques

Section

presents

an

analytical

evaluation

and

comparison

Section

presents

empirical

results

Analytical

Evaluation

In

Reference

-LSB-

-RSB-

we

present

a

framework

for

analyzing

regression

test

selection

techniques

that

consists

of

four

categories

inclusiveness

precision

efficiency

and

generality

Inclusiveness

measures

the

extent

to

which

a

technique

selects

tests

that

reveal

faults

in

a

modified

program

a

inclusive

technique

is

safe

Precision

measures

the

extend

to

which

a

technique

omits

tests

that

can

not

reveal

faults

in

a

modified

program

Efficiency

measures

the

space

and

time

requirements

of

a

technique

focusing

in

particular

on

critical

phase

costs

Generality

measures

the

ability

of

a

technique

to

function

in

some

practical

and

sufficiently

wide

range

of

situations

We

use

our

framework

to

compare

and

evaluate

all

code-based

regression

test

selection

techniques

that

we

have

found

in

the

literature

We

also

use

our

framework

to

evaluate

our

technique

and

compare

it

to

existing

techniques

Here

we

summarize

the

results

reported

in

that

work

Inclusiveness

Our

test

selection

algorithms

are

safe

for

controlled

regression

testing

Only

three

other

techniques

-LSB-

-RSB-

can

also

make

this

claim

Moreover

all

of

these

safe

techniques

depend

for

their

safety

upon

the

same

assumptions

on

which

our

algorithms

depend

Thus

at

present

with

existing

regression

test

selection

techniques

safe

test

selection

is

possible

only

for

controlled

regression

testing

Precision

Our

test

selection

algorithms

are

not

precise

However

because

the

problem

of

precisely

identifying

the

tests

that

are

fault-revealing

for

a

program

and

its

modified

version

is

undecidable

we

know

that

we

can

not

have

an

efficient

algorithm

that

is

both

safe

and

precise

Thus

because

our

goal

is

safety

some

imprecision

is

unavoidable

Nevertheless

our

algorithms

are

the

most

precise

safe

algorithms

currently

available

For

cases

where

the

multiply-visited-node

condition

does

not

hold

-LRB-

which

we

expect

includes

all

practical

cases

-RRB-

our

technique

selects

exactly

the

modification-traversing

tests

whereas

other

safe

techniques

select

the

modification-traversing

tests

along

with

tests

that

are

not

modification-traversing

In

cases

where

the

multiply-visited-node

condition

does

hold

we

can

prove

that

SelectTests

and

SelectInterTests

are

more

precise

than

two

of

the

other

three

test

selection

techniques

and

we

have

strong

evidence

to

suggest

that

our

algorithms

are

more

precise

than

the

third

technique

-LSB-

-RSB-

Efficiency

As

we

discussed

previously

our

algorithms

run

in

time

O

-LRB-

jT

-RRB-

for

procedures

or

programs

of

n

statements

and

test

set

size

jT

j

This

is

an

improvement

over

the

efficiency

of

two

of

the

other

safe

techniques

Moreover

in

practice

we

expect

the

basic

algorithms

-LRB-

without

added

precision

-RRB-

to

run

in

time

O

-LRB-

jT

jn

-RRB-

a

bound

comparable

to

the

worst-case

run

time

of

the

third

safe

technique

Our

algorithms

are

fully

automatable

Furthermore

much

of

the

work

required

by

our

technique

such

as

construction

of

CFGs

for

P

and

collection

of

test

history

information

can

be

completed

during

the

preliminary

regression

testing

phase

Unlike

other

safe

algorithms

our

algorithms

do

not

require

prior

computation

of

a

mapping

between

components

of

programs

or

procedures

and

their

modified

versions

instead

they

locate

changed

code

as

they

proceed

and

in

the

presence

of

significant

changes

avoid

unnecessary

comparison

Generality

Our

algorithms

apply

to

procedural

languages

generally

because

we

can

obtain

the

required

graphs

and

test

history

information

for

all

such

languages

Our

technique

supports

both

intraprocedural

and

interprocedural

test

selection

Our

technique

handles

all

types

of

program

modifications

and

handles

multiple

modifications

in

a

single

application

of

the

algorithms

Empirical

Evaluation

Researchers

wishing

to

experiment

with

software

testing

techniques

face

several

difficulties

-LCB-

among

them

the

problem

of

locating

suitable

experimental

subjects

The

subjects

for

testing

experimentation

include

both

software

and

test

suites

for

regression

testing

experimentation

we

also

need

multiple

versions

of

the

software

Obtaining

such

subjects

is

a

non-trivial

task

Free

software

often

in

multiple

versions

is

readily

accessible

but

free

software

is

not

typically

equipped

with

test

suites

Commercial

software

vendors

who

are

more

likely

to

maintain

established

test

suites

are

often

reluctant

to

release

their

source

code

and

test

suites

to

researchers

Even

when

suitable

experimental

subjects

are

available

prototype

testing

tools

may

not

be

robust

enough

to

operate

on

those

subjects

and

the

time

required

to

ensure

adequate

robustness

may

be

prohibitive

Given

adequate

experimental

subjects

and

sufficiently

robust

prototypes

we

may

still

question

the

generalizability

of

experimental

results

derived

using

those

subjects

and

prototypes

Experimental

results

obtained

in

the

medical

sciences

generalize

due

to

the

fact

that

a

carefully

chosen

subset

of

a

population

of

subjects

typically

represents

a

fair

-LRB-

i.e.

normally

distributed

-RRB-

cross-section

of

that

population

As

Weyuker

states

however

when

our

subject

population

is

the

universe

of

software

systems

we

do

not

know

what

it

means

to

select

a

fair

cross-section

of

that

population

-LSB-

-RSB-

Nor

do

we

know

what

it

means

to

select

a

fair

cross-section

of

the

universe

of

modified

versions

or

test

suites

for

software

Weyuker

concludes

that

software

engineers

typically

perform

empirical

studies

rather

than

experiments

She

insists

however

that

such

studies

offer

insight

and

are

valuable

tools

in

understanding

the

topic

studied

We

agree

with

Weyuker

hence

this

section

outlines

the

results

of

empirical

studies

To

empirically

evaluate

our

regression

test

selection

technique

we

implemented

the

SelectTests

and

SelectInterTests

algorithms

as

tools

which

we

call

and

respectively

Our

implementations

select

tests

for

programs

written

in

C.

We

implemented

our

tools

and

conducted

our

empirical

studies

on

a

Sun

Microsystems

SPARCstation

with

of

virtual

memory

Study

Intraprocedural

and

Interprocedural

Test

Selection

Our

first

study

investigated

the

efficacy

of

and

on

a

set

of

small

but

non-trivial

real

subject

programs

Subjects

Hutchins

Foster

Goradia

and

Ostrand

-LSB-

-RSB-

report

the

results

of

an

experiment

on

the

effectiveness

of

dataflow

and

controlflow-based

test

adequacy

criteria

To

conduct

their

study

the

authors

obtained

seven

C

programs

that

ranged

in

size

from

to

lines

of

code

and

to

procedures

They

manufactured

versions

of

these

programs

and

created

large

test

pools

for

the

programs

The

authors

made

these

programs

versions

and

test

suites

available

to

us

We

refer

to

their

experiment

as

the

Siemens

study

and

to

the

experimental

programs

as

the

Siemens

programs

Table

describes

the

Siemens

programs

To

study

our

intraprocedural

test

selection

algorithms

we

considered

each

procedure

in

the

Siemens

programs

that

had

been

modified

for

one

or

more

versions

of

a

program

Table

lists

these

procedures

Because

the

Siemens

study

addressed

error

detection

capability

the

study

employed

faulty

versions

of

base

programs

For

our

purposes

we

think

of

these

faulty

versions

as

ill-fated

attempts

to

create

modified

versions

of

the

base

programs

The

use

of

faulty

versions

also

lets

us

make

observations

about

error

detection

during

regression

testing

Hutchins

et

al.

-LSB-

-RSB-

describe

the

process

used

by

the

Siemens

researchers

to

manufacture

test

suites

and

describe

the

requirements

for

our

tools

and

their

design

and

implementation

in

Reference

-LSB-

-RSB-

is

a

trademark

of

Sun

Microsystems

Inc.

are

a

few

differences

between

the

numbers

reported

in

Table

and

the

numbers

reported

in

Reference

-LSB-

-RSB-

Hutchins

et

al.

report

versions

of

tcas

their

distribution

to

us

contained

Also

the

numbers

of

tests

in

the

test

pools

we

obtained

from

their

distribution

differed

slightly

from

the

numbers

they

reported

in

the

two

most

extreme

cases

for

example

we

found

more

tests

-LRB-

tcas

-RRB-

and

fewer

tests

-LRB-

usl

-RRB-

In

some

cases

we

can

attribute

the

difference

in

numbers

of

tests

to

repetitions

of

equivalent

tests

in

their

test

plan

In

any

event

the

difference

amounts

to

less

than

of

the

total

test

pool

size

and

does

not

affect

the

results

of

this

study

Table

Subject

Programs

for

Study

Program

Procedures

LOC

Nodes

Edges

Versions

Tests

Description

replace

21 512 383 432

pattern

replacement

usl

20 472 303 364

lexical

analyzer

totinfo

16 440 249 271

information

measure

usl

21 399 355

lexical

analyzer

16 301 219 243

priority

scheduler

18 292 219 232

priority

scheduler

tcas

altitude

separation

Table

Subject

Procedures

for

Study

Procedure

LOC

Versions

Tests

Procedure

LOC

Versions

Tests

infotbl

locate

gettoken

gser

main

isnumconstant

isstrconstant

omatch

noncrossingbdescend

makepat

noncrossingbclimb

finish

dodash

findnth

gcf

change

getcommand

putend

numericcase

upgradeprio

upgradeprocessprio

getline

getprocess

enqueue

esc

initialize

lgamma

inpatset

altseptest

istokenend

ownbelowthreat

newjob

ownabovethreat

subline

inhibitbiasedclimb

alim

unblockprocess

faulty

program

versions

-LCB-

we

paraphrase

that

description

here

The

Siemens

researchers

created

faulty

versions

of

base

programs

by

manually

seeding

faults

into

those

programs

Most

faults

involve

single

line

changes

a

few

involve

multiple

changes

The

researchers

required

that

the

faults

be

neither

too

easy

nor

too

difficult

to

detect

-LRB-

a

requirement

that

was

quantified

by

insisting

that

each

fault

be

detectable

by

at

least

and

at

most

tests

in

the

test

pool

-RRB-

and

that

the

faults

model

realistic

faults

Ten

people

performed

the

fault

seeding

working

for

the

most

part

without

knowledge

of

each

other

's

work

The

Siemens

researchers

created

test

pools

according

to

good

testing

practices

based

on

the

tester

's

understanding

of

the

program

's

functionality

and

knowledge

of

the

code

The

researchers

initially

generated

tests

using

the

category

partition

method

and

the

Siemens

TSL

-LRB-

Test

Specification

Language

-RRB-

tool

-LSB-

-RSB-

they

then

added

additional

tests

to

the

test

suites

to

ensure

that

each

coverage

unit

-LRB-

statement

edge

and

du-pair

-RRB-

in

the

base

program

and

versions

was

exercised

by

at

least

tests

The

Siemens

subjects

present

some

disadvantages

for

our

study

because

they

employ

only

faulty

modifications

use

manufactured

faults

rather

than

real

ones

and

use

only

faults

that

yield

meaningful

detection

rates

However

the

Siemens

subjects

also

have

considerable

advantages

The

fact

that

the

Siemens

researchers

made

the

subjects

available

to

us

is

an

obvious

advantage

Also

the

Siemens

test

suites

are

enormous

and

the

seeded

faults

do

model

real

faults

Furthermore

the

source

code

for

the

base

programs

and

versions

is

standard

C

amenable

to

analysis

and

instrumentation

by

our

prototype

tools

Finally

the

Siemens

subjects

have

served

previously

as

a

basis

for

published

empirical

results

Empirical

Procedure

To

obtain

our

empirical

results

we

initially

used

an

analysis

tool

-LSB-

-RSB-

on

the

base

programs

and

modified

versions

to

create

control

flow

graphs

for

those

versions

We

then

ran

a

code

instrumentation

tool

to

generate

instrumented

versions

of

the

base

programs

For

each

base

program

we

ran

all

of

the

tests

for

that

program

on

the

instrumented

version

of

the

program

and

collected

test

history

information

for

those

tests

We

then

ran

our

implementation

of

SelectTests

on

each

procedure

from

a

Siemens

program

that

had

been

modified

in

one

or

more

modified

versions

with

each

modified

version

of

the

base

version

of

that

procedure

We

also

ran

our

implementation

of

SelectInterTests

on

each

Siemens

base

program

with

each

modified

version

of

that

base

program

We

obtained

execution

timings

during

off-peak

hours

on

a

restricted

machine

our

testing

processes

were

the

only

user

processes

active

on

the

machine

We

repeated

each

experiment

five

times

for

each

-LRB-

base

program

modified

program

-RRB-

or

-LRB-

base

procedure

modified

procedure

-RRB-

pair

and

averaged

our

results

over

these

runs

all

timings

that

we

report

for

this

study

list

these

average

results

In

our

experimentation

we

practiced

controlled

regression

testing

Results

Figure

depicts

the

test

selection

results

for

our

intraprocedural

test

selection

tool

in

Study

The

graph

depicts

for

each

of

the

base

versions

of

Siemens

procedures

the

percentage

of

tests

selected

by

on

average

over

the

set

of

modified

versions

of

that

base

procedure

As

the

graph

shows

for

this

study

intraprocedural

test

selection

reduced

the

size

of

selected

test

sets

in

some

cases

but

the

overall

savings

was

not

dramatic

In

fact

for

of

the

subject

procedures

always

selected

of

the

tests

for

those

procedures

reduced

test

sets

by

more

than

on

average

in

only

five

cases

We

discuss

these

results

in

greater

detail

later

in

this

section

Figure

depicts

the

test

selection

results

for

our

interprocedural

test

selection

tool

in

Study

The

graph

depicts

for

each

of

the

base

versions

of

Siemens

programs

the

percentage

of

tests

selected

by

on

average

over

the

set

of

modified

versions

of

that

base

program

As

the

graph

shows

in

Study

the

average

test

set

selected

by

for

a

modified

version

was

as

large

as

the

test

set

required

by

the

retest

all

approach

In

other

words

averaged

a

savings

in

test

set

size

of

Over

the

various

base

programs

the

test

sets

selected

by

ranged

from

-LRB-

on

replace

-RRB-

to

-LRB-

on

-RRB-

of

the

size

of

the

total

test

sets

for

those

programs

Selected

Procedure

Number

of

Pct.tests

Figure

Intraprocedural

test

selection

for

Study

Program

Name

tcas

Pct.oftests

Selected

-LRB-

-RRB-

replace

usl

usl

totinfo

Figure

Interprocedural

test

selection

for

Study

totinfousl

replace

tcas

-LRB-

mins

-RRB-

Time

Analysis

Retest

All

Retest

Selected

Program

and

pct

of

time

saved

Figure

Interprocedural

timings

for

Study

The

fact

that

our

test

selection

algorithms

reduce

the

number

of

tests

required

to

retest

modified

programs

does

not

by

itself

indicate

the

possible

worth

of

our

algorithms

If

we

required

ten

hours

of

analysis

to

determine

that

we

could

save

one

hour

of

testing

this

might

not

be

of

benefit

-LCB-

unless

the

ten

hours

were

fully

automated

the

hour

saved

was

an

hour

of

human

time

and

we

could

spare

the

ten

hours

We

would

like

to

show

that

the

time

we

save

in

not

having

to

rerun

tests

exceeds

the

time

we

spend

analyzing

programs

and

selecting

reduced

test

suites

Toward

this

end

Figure

depicts

some

timings

For

each

of

the

seven

base

programs

the

figure

shows

the

average

time

required

to

run

all

of

the

tests

on

the

modified

version

of

the

program

-LRB-

darkest

column

-RRB-

the

time

required

to

perform

analysis

on

average

of

the

base

and

modified

versions

of

the

program

-LRB-

lightest

column

-RRB-

and

the

time

required

on

average

to

run

the

selected

tests

on

the

modified

version

of

the

program

Times

are

shown

in

minutes

and

seconds

Note

that

in

this

study

we

were

able

to

fully

automate

both

the

execution

of

tests

and

the

validation

of

test

results

As

the

figure

indicates

the

cost

of

our

safe

test

selection

algorithms

in

terms

of

time

is

negligible

never

exceeding

seconds

Our

measurements

include

the

cost

of

building

CFGs

for

both

the

base

and

modified

program

version

however

the

CFG

for

the

base

version

could

have

been

computed

and

stored

like

test

history

information

during

the

preliminary

phase

of

testing

reducing

the

critical

period

cost

of

the

algorithm

The

figure

also

shows

that

in

all

cases

produced

a

savings

in

overall

regression

testing

time

In

the

worst

case

for

saved

only

seconds

or

of

total

effort

In

the

best

case

for

replace

using

saved

minutes

and

seconds

or

of

total

effort

By

absolute

measures

savings

of

a

few

minutes

and

seconds

such

as

this

study

illustrates

may

be

relatively

unimportant

However

in

practice

test

suites

may

take

hours

days

or

even

weeks

to

run

and

much

of

this

effort

may

be

human-intensive

Moreover

lengthy

testing

times

may

occur

despite

efforts

to

Table

Subject

Program

for

Study

Program

Procedures

LOC

Nodes

Edges

Versions

Tests

Description

player

transaction

manager

Table

Modified

Versions

of

player

Version

Functions

Modified

Lines

of

Code

Changed

automate

test

execution

and

results

validation

If

results

such

as

those

depicted

by

this

study

scale

up

to

such

larger-scale

cases

a

savings

of

even

may

matter

and

a

savings

of

may

offer

a

big

win

In

fact

we

conjecture

that

the

savings

obtainable

from

increase

on

average

as

larger

programs

are

used

as

subjects

Our

second

study

investigates

this

conjecture

Study

Interprocedural

Test

Selection

on

a

Larger

Scale

Our

second

study

investigated

the

efficacy

of

our

interprocedural

test

selection

algorithm

on

a

larger

subject

Subjects

For

our

second

study

we

obtained

a

program

player

that

is

one

of

the

executable

components

of

an

internet-based

game

called

Empire

The

player

executable

is

essentially

a

transaction

manager

its

main

routine

contains

initialization

code

followed

by

a

five-statement

event

loop

that

waits

for

receipt

of

a

user

command

and

on

receiving

one

calls

a

routine

that

processes

the

command

-LRB-

possibly

invoking

many

more

routines

to

do

so

-RRB-

then

waits

to

receive

the

next

command

The

loop

and

the

program

terminate

when

a

user

issues

a

quit

command

Since

its

initial

encoding

in

as

an

internet-based

game

Empire

has

been

rewritten

many

times

several

modified

versions

of

the

program

have

been

created

Most

of

these

versions

involve

modifications

to

player

For

our

study

we

located

a

base

version

of

player

for

which

five

distinct

modified

versions

were

available

Table

presents

some

statistics

about

the

base

version

As

the

table

indicates

the

version

contains

C

functions

and

lines

of

code

excluding

blank

lines

and

lines

that

contain

only

comments

The

CFGs

for

the

functions

contained

approximately

nodes

and

edges

in

total

-LRB-

these

numbers

are

approximate

for

reasons

explained

later

-RRB-

Table

describes

the

versions

of

player

that

we

used

for

our

study

There

were

no

test

suites

available

for

player

To

construct

a

realistic

test

suite

we

used

the

Empire

information

files

which

describe

the

commands

that

are

recognized

by

the

player

executable

and

discuss

parameters

and

special

side-effects

for

each

command

We

treated

the

information

files

as

informal

specifica

tions

for

each

command

we

used

its

information

file

to

construct

versions

of

the

command

that

exercise

all

parameters

and

special

features

and

test

erroneous

parameters

and

conditions

This

process

yielded

a

test

suite

of

functional

tests

We

believe

that

this

test

suite

is

typical

of

the

sorts

of

functional

test

suites

designed

in

practice

for

large

software

systems

for

which

test

suites

often

contain

only

functional

tests

The

player

program

is

a

reasonable

subject

for

several

reasons

First

the

program

is

part

of

an

existing

software

system

that

has

a

long

history

of

maintenance

at

the

hands

of

numerous

coders

in

this

respect

the

system

is

similar

to

many

existing

commercial

software

systems

Second

as

a

transaction

manager

the

player

program

is

representative

of

a

large

class

of

software

systems

that

receive

and

process

a

variety

of

user

commands

Third

we

were

able

to

locate

several

real

modified

versions

of

one

base

version

of

the

program

Fourth

although

the

absence

of

established

test

suites

for

the

program

was

a

disadvantage

the

user

documentation

provided

a

code-independent

means

for

generating

functional

tests

in

a

realistic

fashion

Finally

although

not

huge

the

program

is

reasonably

large

Empirical

Procedure

Due

to

limitations

in

our

prototype

analysis

tools

we

could

analyze

only

of

the

procedures

in

the

player

program

thus

we

could

not

instrument

or

run

our

DejaVu

implementations

on

of

the

procedures

However

we

were

able

to

simulate

the

test

selection

effects

of

DejaVu

on

player

Our

simulation

determines

exactly

the

numbers

of

tests

selected

and

omitted

by

our

algorithm

in

practice

we

were

also

able

to

determine

exactly

the

time

required

to

run

all

tests

or

all

selected

tests

We

could

not

obtain

precise

results

of

the

time

required

to

build

CFGs

for

the

versions

and

perform

test

selection

on

those

graphs

but

we

were

able

to

obtain

close

estimates

of

those

times

Results

Figure

depicts

the

test

selection

results

for

our

interprocedural

test

selection

algorithm

for

the

modified

versions

of

player

The

graph

depicts

for

each

of

the

five

modified

versions

the

percentage

of

tests

that

our

algorithm

selects

As

the

results

indicate

on

average

over

the

five

versions

our

algorithm

selects

of

the

tests

in

the

existing

test

suite

In

other

words

on

average

the

algorithm

reduces

the

number

of

tests

that

must

be

rerun

by

over

Figure

depicts

timings

for

this

study

The

graph

depicts

for

each

of

the

five

modified

versions

the

time

required

to

run

all

of

the

tests

on

the

modified

version

of

the

program

-LRB-

darkest

column

-RRB-

the

estimated

time

required

to

perform

analysis

of

the

base

and

modified

versions

of

the

program

-LRB-

lightest

column

-RRB-

and

the

time

required

to

run

the

selected

tests

on

the

modified

version

of

the

program

Times

are

shown

in

hours

minutes

and

seconds

The

graph

shows

that

in

all

cases

our

algorithm

produced

a

savings

in

overall

regression

testing

time

This

savings

ranged

from

hours

and

minutes

-LRB-

of

total

effort

-RRB-

to

hours

and

minutes

-LRB-

of

total

effort

-RRB-

examples

of

systems

in

this

class

include

database

management

systems

operating

systems

menu-driven

systems

and

computer-aided

drafting

systems

to

name

just

a

few

determined

the

time

required

to

build

CFGs

and

run

on

of

the

code

We

report

that

time

scaled

up

by

Work

is

ongoing

to

improve

the

robustness

of

our

analysis

tools

Version

Number

Selected

-LRB-

-RRB-

Pct.oftests

Figure

Test

selection

statistics

for

Study

-LRB-

hrs

-RRB-

Time

Analysis

Retest

All

Retest

Selected

Figure

Timings

for

Study

Our

estimate

of

analysis

time

projects

a

cost

of

at

most

minutes

however

this

estimate

computes

the

cost

of

building

CFGs

for

every

procedure

in

both

the

base

and

modified

program

version

during

the

critical

period

and

walking

all

of

those

CFGs

completely

In

practice

we

could

build

the

CFGs

for

procedures

in

the

base

version

during

the

preliminary

phase

and

build

CFGs

for

procedures

in

modified

versions

on

demand

lowering

the

analysis

cost

Furthermore

our

test

timings

consider

only

the

cost

of

running

the

tests

because

we

were

not

able

to

automate

the

validation

of

results

for

these

tests

In

practice

then

the

time

required

to

run

tests

would

be

much

larger

than

the

times

shown

and

the

resulting

savings

would

increase

Additional

discoveries

and

discussion

Our

studies

yielded

several

additional

discoveries

First

in

Section

we

saw

that

our

algorithms

can

select

tests

that

are

not

modification-traversing

for

P

and

P

but

only

when

the

multiply-visited-node

condition

holds

In

our

experiments

we

never

encountered

a

case

where

that

condition

held

The

results

support

our

conjecture

that

in

practice

our

algorithms

will

not

select

non-modification-traversing

tests

Second

although

we

have

reported

our

results

as

averages

over

sets

of

modified

programs

it

is

interesting

to

examine

the

behavior

of

our

algorithms

for

individual

cases

Consider

for

example

the

results

for

program

replace

The

test

pool

for

replace

contains

tests

of

which

selects

on

average

-LRB-

-RRB-

However

over

the

modified

versions

of

replace

the

selected

test

sets

ranged

in

size

from

to

tests

with

no

size

range

predominant

the

standard

deviation

in

the

sizes

of

the

selected

test

sets

was

In

contrast

for

selects

on

average

-LRB-

-RRB-

of

the

program

's

pool

of

tests

with

a

standard

deviation

of

On

eight

of

the

ten

modified

versions

of

selects

at

least

of

the

existing

tests

Given

this

range

of

variance

we

would

like

to

be

able

to

determine

the

factors

that

influence

the

success

of

test

selection

This

could

help

us

determine

when

test

selection

is

likely

to

be

successful

and

also

help

us

draw

conclusions

about

ways

in

which

to

build

programs

and

test

suites

that

are

regression

testable

On

examining

our

experimental

subjects

and

test

suites

we

determined

that

the

effectiveness

of

test

selection

was

influenced

by

three

factors

the

structure

of

P

the

location

of

the

modifications

in

P

and

the

extent

and

type

of

code

coverage

achieved

by

tests

in

T

Although

these

factors

can

interact

they

may

also

operate

independently

Finally

as

an

interesting

side-effect

of

this

study

our

results

support

some

hypotheses

about

the

faultrevealing

capabilities

of

regression

test

selection

techniques

Study

involved

faulty

modified

program

versions

for

these

versions

a

very

small

percentage

of

tests

are

fault-revealing

For

example

consider

program

replace

with

a

test

suite

of

tests

For

version

of

replace

of

the

tests

are

fault-revealing

finds

that

of

the

tests

are

modification-traversing

and

selects

them

A

minimization

test

selection

technique

that

selects

one

test

from

the

set

of

tests

that

cover

modified

code

has

only

a

chance

of

selecting

one

of

the

fault-revealing

tests

Next

consider

version

of

replace

Although

of

the

tests

of

replace

are

modification-traversing

for

this

version

only

of

these

tests

-LSB-

-RSB-

lists

results

for

all

programs

and

modified

versions

individually

are

fault-revealing

for

the

version

A

minimization

test

selection

technique

that

selects

only

one

of

the

tests

that

cover

this

modification

has

only

a

chance

of

selecting

a

test

that

exposes

the

fault

In

either

case

guarantees

that

the

fault

is

exposed

Every

case

considered

in

Study

supports

a

comparable

conclusion

It

would

not

be

fair

on

the

basis

of

Study

alone

to

draw

general

conclusions

about

the

relative

effectiveness

of

minimization

and

safe

test

selection

techniques

because

the

Siemens

study

deliberately

restricted

program

modifications

to

those

that

contained

faults

that

were

neither

too

easy

nor

too

difficult

to

detect

Nevertheless

the

Siemens

study

did

employ

faults

that

are

representative

of

real

faults

so

we

expect

that

cases

such

as

the

two

discussed

above

and

such

as

those

found

in

the

Siemens

study

in

general

arise

in

practice

Our

results

thus

give

us

good

reason

to

question

the

efficacy

of

minimization

test

selection

techniques

where

fault

detection

is

concerned

Summary

of

empirical

results

and

limitations

of

the

studies

We

summarize

the

major

conclusions

derived

from

our

empirical

studies

about

the

use

of

our

algorithms

in

practice

as

follows

ffl

Our

algorithms

can

reduce

the

time

required

to

regression

test

modified

software

even

when

the

cost

of

the

analysis

performed

to

select

tests

is

considered

ffl

We

expect

interprocedural

test

selection

to

offer

greater

savings

than

intraprocedural

test

selection

ffl

Regression

test

selection

algorithms

may

yield

greater

savings

when

applied

to

large

complex

programs

than

when

applied

to

small

simple

programs

ffl

There

exist

programs

modified

versions

and

test

suites

for

which

test

selection

offers

little

in

the

way

of

savings

ffl

The

factors

that

affect

the

effectiveness

of

test

selection

techniques

include

the

structure

of

programs

the

nature

of

the

modifications

made

to

those

programs

and

the

type

of

coverage

attained

by

tests

ffl

Our

conjecture

that

in

practice

programs

contain

no

multiply-visited-nodes

remains

plausible

ffl

Our

results

support

a

conjecture

that

minimization

techniques

for

test

selection

may

be

ineffective

at

revealing

faults

For

the

sake

of

fairness

we

mention

the

following

limitations

on

our

studies

ffl

We

have

studied

only

a

small

sample

of

the

universe

of

possible

programs

modified

programs

and

test

suites

We

have

no

hard

data

on

which

to

base

a

claim

that

this

sample

is

normally

distributed

We

believe

however

that

the

subjects

of

our

studies

are

representative

of

significant

classes

of

programs

that

occur

in

practice

ffl

Both

of

our

studies

required

some

manufactured

artifacts

study

used

manufactured

modified

versions

and

tests

and

study

two

used

manufactured

tests

In

both

cases

however

efforts

were

made

to

ensure

that

manufactured

artifacts

were

representative

of

real

counterparts

ffl

Due

to

limitations

in

our

program

analysis

tools

our

second

study

required

an

estimation

of

analysis

times

However

we

were

careful

to

estimate

fairly

and

conservatively

We

believe

that

our

estimates

understate

the

analysis

time

required

by

our

technique

Conclusions

and

Future

Work

We

have

presented

a

technique

for

selecting

tests

from

an

existing

test

suite

for

use

in

regression

testing

a

modified

procedure

or

program

We

have

shown

that

for

controlled

regression

testing

our

algorithms

are

safe

they

select

every

test

in

the

existing

test

suite

that

may

reveal

faults

in

the

modified

program

Our

algorithms

are

also

the

most

precise

safe

test

selection

algorithms

currently

available

The

algorithms

are

efficient

and

handle

both

intraprocedural

and

interprocedural

regression

test

selection

Our

empirical

studies

suggest

that

in

practice

our

technique

can

reduce

the

cost

of

regression

testing

a

modified

program

This

work

is

important

for

two

reasons

According

to

Pressman

-LSB-

-RSB-

the

cost

of

software

maintenance

dominates

the

overall

cost

of

software

Moreover

the

cost

of

maintenance

measured

in

terms

of

the

percentage

of

software

budget

spent

on

maintenance

is

increasing

In

the

typical

industry

expenditures

on

software

maintenance

comprised

to

percent

of

total

software

budget

In

the

expenditures

rose

to

between

and

percent

of

total

budget

and

in

the

the

figure

exceeds

percent

Beizer

presents

similar

statistics

-LSB-

-RSB-

Because

regression

testing

constitutes

a

significant

percentage

of

maintenance

costs

there

are

good

reasons

to

seek

improvements

in

regression

testing

processes

Both

development

testing

and

regression

testing

are

expensive

However

the

cost

of

development

testing

is

incurred

once

when

a

software

product

is

created

this

cost

is

then

amortized

over

the

lifetime

of

the

product

In

contrast

a

software

product

is

regression

tested

every

time

a

new

version

of

that

product

is

released

The

cost

of

regression

testing

is

thus

compounded

over

the

lifetime

of

the

product

Thus

improvements

in

regression

testing

may

have

greater

potential

to

reduce

the

overall

cost

of

software

than

improvements

in

development

testing

The

motivation

for

finding

better

regression

testing

techniques

is

not

just

economic

Both

developers

and

users

of

software

want

to

believe

that

their

software

works

correctly

Both

developers

and

users

of

software

want

the

probability

that

their

software

will

fail

to

be

reduced

Testing

is

the

method

of

choice

both

for

building

confidence

in

software

and

for

increasing

its

reliability

Regression

testing

plays

an

important

role

in

both

of

these

tasks

At

present

however

according

to

Beizer

regression

testing

is

often

overlooked

or

performed

inadequately

either

the

testing

of

new

features

or

the

revalidation

of

old

features

or

both

are

sacrificed

As

a

result

software

reliability

decreases

over

the

software

's

lifetime

-LSB-

-RSB-

Practical

effective

selective

retest

techniques

promote

software

quality

We

have

discovered

several

promising

directions

for

future

work

in

this

area

First

while

the

empirical

results

reported

in

this

paper

are

encouraging

they

are

also

preliminary

Additional

implementations

of

the

algorithms

and

enhancements

described

in

Section

and

further

empirical

studies

with

those

implementations

would

be

useful

Empirical

work

to

examine

the

cost

and

the

fault-revealing

capabilities

of

these

and

other

regression

test

selection

techniques

would

be

particularly

valuable

Second

our

work

has

focused

on

the

problem

of

selecting

tests

from

an

existing

test

suite

An

equally

important

problem

however

is

that

of

ensuring

that

code

modified

or

affected

by

modifications

is

adequately

tested

The

tests

that

we

select

from

an

existing

test

suite

can

help

establish

adequacy

but

new

tests

may

also

be

required

Future

work

should

consider

the

extension

of

this

technique

to

help

identify

the

need

for

new

tests

Third

our

work

considers

the

problem

of

code-based

test

selection

future

work

should

consider

the

application

of

the

technique

to

the

problem

of

specification-based

selection

Fourth

our

research

has

revealed

that

the

size

of

the

test

sets

our

algorithms

select

may

vary

significantly

This

variance

is

a

function

of

program

structure

modifications

and

test

suite

design

Further

research

could

investigate

correlations

between

these

three

factors

and

the

related

issues

of

regression

testability

and

test

suite

design

Fifth

our

technique

is

safe

only

for

controlled

regression

testing

In

this

respect

all

existing

safe

regression

test

selection

techniques

are

alike

In

the

absence

of

controlled

regression

testing

techniques

that

are

otherwise

safe

may

omit

fault-revealing

tests

This

raises

the

question

can

we

find

ways

to

increase

the

conditions

under

which

safe

test

selection

is

possible

There

are

several

ways

in

which

to

address

this

question

We

might

try

to

find

ways

to

make

controlled

regression

testing

possible

in

situations

wherein

it

is

currently

difficult

to

attain

Or

we

might

try

to

find

ways

to

identify

the

types

of

fault-revealing

tests

that

techniques

like

ours

might

omit

in

the

absence

of

controlled

regression

testing

and

extend

the

techniques

to

select

these

tests

Or

finally

we

might

find

ways

to

otherwise

identify

the

errors

that

could

be

exposed

by

tests

omitted

by

these

techniques

Finally

even

when

we

can

not

employ

controlled

regression

testing

and

thus

can

not

guarantee

safety

our

algorithms

may

still

select

test

suites

that

are

useful

It

is

not

the

case

that

our

algorithms

function

only

in

the

presence

of

controlled

regression

testing

it

is

simply

that

they

like

all

other

regression

test

selection

algorithms

are

not

safe

in

the

absence

of

controlled

regression

testing

Viewed

differently

our

algorithms

select

all

tests

in

T

that

if

we

had

practiced

controlled

regression

testing

could

have

exposed

faults

in

P

When

the

testing

budget

is

limited

and

we

must

choose

a

subset

of

T

modification-traversing

tests

such

as

those

selected

by

our

algorithm

may

be

better

candidates

for

execution

than

non-modification-traversing

tests

Thus

even

in

the

absence

of

controlled

regression

testing

we

may

find

our

algorithm

useful

Empirical

studies

could

investigate

this

conjecture

further

References

-LSB-

-RSB-

H.

Agrawal

J.

Horgan

E.

Krauser

and

S.

London

Incremental

regression

testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

September

-LSB-

-RSB-

A.V.

Aho

R.

Sethi

and

J.D.

Ullman

Compilers

Principles

Techniques

and

Tools

Addison-Wesley

Reading

MA

-LSB-

-RSB-

M.

Balcer

W.

Hasling

and

T.

Ostrand

Automatic

generation

of

test

scripts

from

formal

test

specifications

In

Proceedings

of

the

Third

Symposium

on

Software

Testing

Analysis

and

Verification

pages

-LCB-

December

-LSB-

-RSB-

S.

Bates

and

S.

Horwitz

Incremental

program

testing

using

program

dependence

graphs

In

Proceedings

of

the

ACM

Symposium

on

Principles

of

Programming

Languages

January

-LSB-

-RSB-

B.

Beizer

Software

Testing

Techniques

Van

Nostrand

Reinhold

New

York

NY

-LSB-

-RSB-

B.

Beizer

Black-Box

Testing

John

Wiley

and

Sons

New

York

NY

-LSB-

-RSB-

P.

Benedusi

A.

Cimitile

and

U.

De

Carlini

Post-maintenance

testing

based

on

path

change

analysis

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

D.

Binkley

Using

semantic

differencing

to

reduce

the

cost

of

regression

testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

November

-LSB-

-RSB-

D.

Binkley

Reducing

the

cost

of

regression

testing

by

semantics

guided

test

case

selection

In

Proceedings

of

the

Conference

on

Software

Maintenance

October

-LSB-

-RSB-

P.A.

Brown

and

D.

Hoffman

The

application

of

module

regression

testing

at

TRIUMF

Nuclear

Instruments

and

Methods

in

Physics

Research

Section

A

-LRB-

-RRB-

-LCB-

August

-LSB-

-RSB-

Y.F.

Chen

D.S.

Rosenblum

and

K.P.

Vo

TestTube

A

system

for

selective

regression

testing

In

Proceedings

of

the

International

Conference

on

Software

Engineering

pages

-LCB-

May

-LSB-

-RSB-

T.

Dogsa

and

I.

Rozman

CAMOTE

computer

aided

module

testing

and

design

environment

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

K.F.

Fischer

A

test

case

selection

method

for

the

validation

of

software

maintenance

modifications

In

Proceedings

of

COMPSAC

pages

-LCB-

November

-LSB-

-RSB-

K.F.

Fischer

F.

Raji

and

A.

Chruscicki

A

methodology

for

retesting

modified

software

In

Proceedings

of

the

National

Telecommunications

Conference

pages

-LCB-

November

-LSB-

-RSB-

R.

Gupta

M.J.

Harrold

and

M.L.

Soffa

An

approach

to

regression

testing

using

slicing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

November

-LSB-

-RSB-

M.J.

Harrold

R.

Gupta

and

M.L.

Soffa

A

methodology

for

controlling

the

size

of

a

test

suite

ACM

Transactions

on

Software

Engineering

and

Methodology

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

M.J.

Harrold

L.

Larsen

J.

Lloyd

D.

Nedved

M.

Page

G.

Rothermel

M.

Singh

and

M.

Smith

Aristotle

a

system

for

the

development

of

program-analysis-based

tools

In

Proceedings

of

the

Annual

Southeast

Conference

pages

-LCB-

March

-LSB-

-RSB-

M.J.

Harrold

and

M.L.

Soffa

An

incremental

approach

to

unit

testing

during

maintenance

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

M.J.

Harrold

and

M.L.

Soffa

An

incremental

data

flow

testing

tool

In

Proceedings

of

the

Sixth

International

Conference

on

Testing

Computer

Software

May

-LSB-

-RSB-

M.J.

Harrold

and

M.L.

Soffa

Interprocedural

data

flow

testing

In

Proceedings

of

the

Third

Testing

Analysis

and

Verification

Symposium

pages

-LCB-

December

-LSB-

-RSB-

J.

Hartmann

and

D.J.

Robson

Revalidation

during

the

software

maintenance

phase

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

J.

Hartmann

and

D.J.

Robson

RETEST

development

of

a

selective

revalidation

prototype

environment

for

use

in

software

maintenance

In

Proceedings

of

the

Twenty-Third

Hawaii

International

Conference

on

System

Sciences

pages

-LCB-

January

-LSB-

-RSB-

J.

Hartmann

and

D.J.

Robson

Techniques

for

selective

revalidation

IEEE

Software

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

D.

Hoffman

A

CASE

study

in

module

testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

D.

Hoffman

and

C.

Brealey

Module

test

case

generation

In

Proceedings

of

the

Third

Workshop

on

Software

Testing

Analysis

and

Verification

pages

-LCB-

December

-LSB-

-RSB-

S.

Horwitz

and

T.

Reps.

The

use

of

program

dependence

graphs

in

software

engineering

In

Proceedings

of

the

International

Conference

on

Software

Enginnering

pages

-LCB-

May

-LSB-

-RSB-

M.

Hutchins

H.

Foster

T.

Goradia

and

T.

Ostrand

Experiments

on

the

effectiveness

of

dataflow

and

controlflow-based

test

adequacy

criteria

In

Proceedings

of

the

International

Conference

on

Software

Engineering

pages

-LCB-

May

-LSB-

-RSB-

J.

Laski

and

W.

Szermer

Identification

of

program

modifications

and

its

applications

in

software

maintentance

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

November

-LSB-

-RSB-

J.A.N.

Lee

and

X.

He

A

Methodology

for

Test

Selection

The

Journal

of

Systems

and

Software

-LRB-

-RRB-

-LCB-

September

-LSB-

-RSB-

H.K.N.

Leung

and

L.

White

Insights

Into

Regression

Testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

H.K.N.

Leung

and

L.

White

Insights

into

testing

and

regression

testing

global

variables

Journal

of

Software

Maintenance

-LCB-

December

-LSB-

-RSB-

H.K.N.

Leung

and

L.J.

White

A

study

of

integration

testing

and

software

regression

at

the

integration

level

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

November

-LSB-

-RSB-

H.K.N.

Leung

and

L.J.

White

A

cost

model

to

compare

regression

test

strategies

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

-LSB-

-RSB-

R.

Lewis

D.W.

Beck

and

J.

Hartmann

Assay

a

tool

to

support

regression

testing

In

ESEC

European

Software

Engineering

Conference

Proceedings

pages

-LCB-

September

-LSB-

-RSB-

U.

Linnenkugel

and

M.

Mullerburg

Test

data

selection

criteria

for

-LRB-

software

-RRB-

integration

testing

In

Systems

Integration

Proceedings

of

the

First

International

Conference

on

Systems

Integration

pages

-LCB-

April

-LSB-

-RSB-

T.J.

Ostrand

and

M.J.

Balcer

The

category-partition

method

for

specifying

and

generating

functional

tests

Communications

of

the

ACM

-LRB-

-RRB-

June

-LSB-

-RSB-

T.J.

Ostrand

and

E.J.

Weyuker

Using

dataflow

analysis

for

regression

testing

In

Sixth

Annual

Pacific

Northwest

Software

Quality

Conference

pages

-LCB-

September

-LSB-

-RSB-

R.

Pressman

Software

Engineering

A

Practitioner

's

Approach

McGraw-Hill

New

York

NY

-LSB-

-RSB-

G.

Rothermel

Efficient

Effective

Regression

Testing

Using

Safe

Test

Selection

Techniques

Ph.D.

dissertation

Clemson

University

May

-LSB-

-RSB-

G.

Rothermel

and

M.J.

Harrold

A

safe

efficient

algorithm

for

regression

test

selection

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

September

-LSB-

-RSB-

G.

Rothermel

and

M.J.

Harrold

Selecting

regression

tests

for

object-oriented

software

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

September

-LSB-

-RSB-

G.

Rothermel

and

M.J.

Harrold

Analyzing

regression

test

selection

techniques

Technical

Report

OSU-CISRC

The

Ohio

State

University

February

-LSB-

-RSB-

S.

Schach

Software

Engineering

Aksen

Associates

Boston

MA

-LSB-

-RSB-

B.

Sherlund

and

B.

Korel

Modification

oriented

software

testing

In

Conference

Proceedings

Quality

Week

pages

-LCB-

-LSB-

-RSB-

A.B.

Taha

S.M.

Thebaut

and

S.S.

Liu

An

approach

to

software

fault

localization

and

revalidation

based

on

incremental

data

flow

analysis

In

Proceedings

of

the

Annual

International

Computer

Software

and

Applications

Conference

pages

-LCB-

September

-LSB-

-RSB-

A.

von

Mayrhauser

R.T.

Mraz

and

J.

Walls

Domain

based

regression

testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

September

-LSB-

-RSB-

E.J.

Weyuker

Empirical

techniques

for

assessing

testing

strategies

-LRB-

Panel

discussion

at

the

International

Symposium

on

Software

Testing

and

Analysis

-RRB-

August

-LSB-

-RSB-

L.J.

White

and

H.K.N.

Leung

A

firewall

concept

for

both

control-flow

and

data-flow

in

regression

integration

testing

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

November

-LSB-

-RSB-

L.J.

White

V.

Narayanswamy

T.

Friedman

M.

Kirschenbaum

P.

Piwowarski

and

M.

Oha

Test

Manager

a

regression

testing

tool

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

September

-LSB-

-RSB-

S.

Wolfram

Mathematica

A

System

for

Doing

Mathematics

on

a

Computer

Addison-Wesley

Reading

MA

second

edition

-LSB-

-RSB-

W.

E.

Wong

J.

R.

Horgan

S.

London

and

A.

P.

Mathur

Effect

of

test

set

minimization

on

fault

detection

effectiveness

In

International

Conference

on

Software

Engineering

pages

-LCB-

April

-LSB-

-RSB-

W.

Yang

Identifying

syntactic

differences

between

two

programs

Software

Practice

and

Experience

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

W.

Yang

S.

Horwitz

and

T.

Reps.

A

program

integration

algorithm

that

accomodates

semantics-preserving

transformations

ACM

Transactions

on

Software

Engineering

and

Methodology

-LRB-

-RRB-

-LCB-

July

-LSB-

-RSB-

S.S.

Yau

and

Z.

Kishimoto

A

method

for

revalidating

modified

programs

in

the

maintenance

phase

In

COMP

SAC

The

Eleventh

Annual

International

Computer

Software

and

Applications

Conference

pages

-LCB-

October

-LSB-

-RSB-

J.

Ziegler

J.M.

Grasso

and

L.G.

Burgermeister

An

Ada

based

real-time

closed-loop

integration

and

regression

test

tool

In

Proceedings

of

the

Conference

on

Software

Maintenance

pages

-LCB-

October

regression test selection
regression testing
selective retest
software maintenance