Satisfiability

Testing

with

More

Reasoning

and

Less

Guessing

Allen

Van

Gelder

Yumi

K.

Tsuji

Baskin

Center

for

Computer

Engineering

and

Information

Sciences

University

of

California

Santa

Cruz

avg@cs.ucsc.edu

tsuji@cs.ucsc.edu

April

Abstract

A

new

algorithm

for

testing

satisfiability

of

propositional

formulas

in

conjunctive

normal

form

-LRB-

CNF

-RRB-

is

described

It

applies

reasoning

in

the

form

of

certain

resolution

operations

and

identification

of

equivalent

literals

Resolution

produces

growth

in

the

size

of

the

formula

but

within

a

global

quadratic

bound

most

previous

methods

avoid

operations

that

produce

any

growth

and

generally

do

not

identify

equivalent

literals

Computational

experience

indicates

that

the

method

does

substantially

less

guessing

than

previously

reported

algorithms

while

keeping

a

polynomial

time

bound

on

the

work

done

between

guesses

Experiments

indicate

that

for

larger

problems

the

time

investment

in

reasoning

returns

a

profit

in

reduced

searching

and

the

profit

increases

with

increasing

problem

size

Experimental

data

compares

six

branching

strategies

of

the

proposed

algorithm

on

a

variety

of

problems

including

several

Dimacs

benchmarks

These

branching

strategies

were

shown

to

perform

differently

with

statistical

significance

A

new

scheme

based

on

Johnson

's

maximum

satisfiability

approximation

algorithm

was

found

to

be

the

best

overall

Both

satisfiable

and

unsatisfiable

random

formulas

with

-LCB-

variables

and

ratio

of

clauses

to

variables

have

been

tested

this

class

is

generally

acknowledged

to

be

relatively

hard

and

required

extensive

backtracking

by

other

algorithms

Unsatisfiable

random

problems

were

found

to

deviate

from

the

easy-hard-easy

pattern

The

new

algorithm

solves

random

formulas

with

surprisingly

little

backtracking

the

average

number

of

guesses

was

for

variables

at

this

ratio

and

for

variables

Larger

unsatisfiable

formulas

from

circuit-fault

analysis

with

-LCB-

variables

were

solved

with

no

backtracking

in

some

cases

Extensive

statistics

on

guesses

and

time

are

reported

Statistical

and

experimental

techniques

and

traps

are

discussed

An

exponential

growth

rate

for

random

formulas

is

estimated

To

appear

in

Cliques

Coloring

and

Satisfiability

Second

DIMACS

Implementation

Challenge

Johnson

D.

S.

and

Trick

M.

eds.

American

Mathematical

Society

Authors

were

supported

in

part

by

NSF

grant

Overview

The

problem

of

Boolean

or

propositional

satisfiability

is

the

fundamental

NP-complete

problem

and

has

many

practical

applications

as

well

We

assume

the

reader

is

generally

familiar

with

it

and

give

definitions

only

when

needed

for

clarity

In

Section

we

present

the

idea

of

k-closure

an

operation

on

a

propositional

formula

that

is

in

conjunctive

normal

form

-LRB-

CNF

-RRB-

The

method

involves

a

limited

use

of

resolution

called

k-limited

resolution

-LRB-

Definition

-RRB-

Starting

from

the

given

formula

derivation

of

clauses

by

k-limited

resolution

continues

until

no

further

such

derivations

are

possible

Since

the

number

of

such

clauses

in

a

formula

of

n

variables

is

O

-LRB-

nk

-RRB-

k-closure

takes

polynomial

time

We

describe

an

algorithm

for

satisfiability

based

on

k-closure

in

Section

The

k-closure

approach

is

compared

with

other

approaches

and

some

of

its

properties

are

described

An

earlier

version

of

this

algorithm

-LRB-

presented

at

the

AAAI

Spring

Symposium

-RRB-

used

but

some

operations

were

much

too

expensive

as

implemented

The

new

version

reported

here

called

uses

plus

certain

resolution

operations

on

longer

clauses

-LRB-

see

Section

-RRB-

Two-closure

is

achievable

in

quadratic

time

per

eliminated

variable

Section

presents

the

experimental

data

comparing

the

different

branching

strategies

used

by

the

proposed

algorithm

on

a

variety

of

problems

The

Student

's

paired

t-test

was

applied

to

the

number

of

guesses

for

these

branching

strategies

to

demonstrate

the

statistical

significance

of

their

differences

A

new

scheme

based

on

Johnson

's

maximum

satisfiability

approximation

algorithm

was

found

to

perform

the

best

on

random

formulas

and

was

always

close

to

the

best

on

other

varieties

Section

addresses

the

question

of

whether

the

added

time

for

reasoning

that

is

occurred

by

pays

dividends

in

terms

of

reduced

search

space

-LRB-

as

measured

by

the

number

of

guesses

-RRB-

The

time

performance

of

is

compared

to

the

traditional

satisfiability

algorithm

of

Davis

Putnam

Logemann

and

Loveland

-LSB-

-RSB-

on

families

of

increasingly

large

random

formulas

Above

literals

-LRB-

about

variables

-RRB-

emerges

as

the

clear

winner

Achieving

acceptable

performance

depends

strongly

on

efficient

data

structures

We

found

that

memory

was

often

more

of

a

problem

than

time

for

large

numbers

of

variables

for

an

n

variable

formula

it

was

not

feasible

to

use

an

n?n

array

to

remember

the

results

Space-efficient

data

structures

often

require

some

overhead

in

time

Keeping

that

overhead

low

was

the

primary

implementation

challenge

We

believe

there

is

room

for

significant

additional

improvement

without

changing

the

basic

structure

of

the

algorithm

Appendix

A

provides

further

discussion

on

the

statistical

traps

and

techniques

for

NP-hard

problems

Concerning

experimental

techniques

generally

it

was

found

that

variable

reordering

-LRB-

Section

-RRB-

has

a

significant

impact

on

the

performance

of

satisfiability

testing

of

Boolean

formulas

associated

with

structured

problems

We

investigated

the

effect

of

variable

reordering

by

shuffling

the

variables

in

the

input

formula

so

that

ties

would

be

broken

in

a

variety

of

ways

a

seed

for

the

random

number

generator

used

by

this

shuffling

procedure

can

be

changed

at

the

invocation

time

The

effectiveness

of

this

approach

was

experimentally

confirmed

for

many

of

the

Dimacs

benchmark

problems

Appendix

B

gives

the

results

on

the

benchmark

problems

for

the

Second

DIMACS

Challenge

CNF

Formulas

and

Closures

In

this

paper

we

regard

a

CNF

propositional

formula

in

the

standard

way

as

a

set

of

clauses

where

each

clause

is

regarded

as

a

set

of

literals

Whether

x

is

a

positive

or

negative

literal

x

denotes

its

complement

We

may

classify

CNF

formulas

by

their

widest

clause

as

etc.

Sometimes

the

shortest

clause

length

is

of

interest

too

The

empty

clause

is

written

as

and

represents

false

Recall

that

clause

C

is

said

to

subsume

clause

D

and

D

is

subsumed

by

C

if

the

literals

of

C

are

a

subset

of

those

of

D

A

tautologous

clause

is

one

that

contains

both

a

literal

and

its

complement

can

be

thought

of

as

true

and

is

considered

to

be

subsumed

by

any

clause

We

assume

familiarity

with

the

terms

resolution

unit

clause

rule

and

pure

literal

rule

A

Krom

formula

-LRB-

or

formula

-RRB-

has

an

associated

implication

graph

its

nodes

are

literals

of

the

formula

and

clause

fx

yg

induces

directed

edges

x

y

and

y

x

The

notation

a

b

means

that

there

is

a

path

from

literal

a

to

literal

b

in

the

implication

graph

We

shall

use

n

to

represent

the

number

of

variables

m

for

the

number

of

clauses

and

L

for

the

number

of

occurrences

of

literals

in

a

formula

Definition

Several

notions

of

closure

for

sets

of

clauses

will

be

used

A

set

of

clauses

S

is

said

to

be

closed

under

resolution

or

simply

closed

if

no

clause

of

S

is

subsumed

by

a

different

clause

of

S

and

the

resolvent

of

each

pair

of

resolvable

clauses

is

implied

by

-LRB-

subsumed

by

-RRB-

some

clause

in

S.

A

closure

of

a

CNF

formula

F

is

a

CNF

formula

that

derived

from

F

by

a

series

of

resolutions

-LRB-

which

add

clauses

-RRB-

and

subsumptions

-LRB-

which

delete

clauses

-RRB-

and

is

closed

The

operation

of

k-limited

resolution

is

defined

to

be

resolution

in

which

the

operands

and

the

resolvent

have

at

most

k

literals

each

A

set

of

clauses

S

is

said

to

be

k-closed

if

no

clause

of

S

is

subsumed

by

a

different

clause

of

S

and

the

resolvent

of

each

possible

k-limited

resolution

with

operands

in

S

is

implied

by

-LRB-

subsumed

by

-RRB-

some

clause

in

S.

A

k-closure

of

a

CNF

formula

F

is

a

CNF

formula

that

derived

from

F

by

a

series

of

k-limited

resolutions

-LRB-

which

add

clauses

-RRB-

and

subsumptions

-LRB-

which

delete

clauses

-RRB-

and

is

k-closed

It

is

easy

to

see

that

both

closure

and

k-closure

are

unique

By

the

completeness

of

resolution

a

set

of

clauses

that

is

closed

under

resolution

is

unsatisfiable

if

and

only

if

it

contains

the

empty

clause

The

size

of

the

closure

of

formula

F

with

n

propositional

variables

may

be

exponential

in

n.

However

the

k-closure

has

a

size

that

is

O

-LRB-

nk

-RRB-

and

so

can

be

computed

in

polynomial

time

if

k

is

a

constant

We

shall

be

interested

primarily

in

k

or

Two-Closure

For

a

formula

all

resolvents

have

at

most

two

literals

so

its

is

also

its

closure

It

follows

that

computing

the

provides

a

polynomial-time

decision

procedure

for

Using

the

technique

of

strongly

connected

components

it

is

possible

to

decide

formulas

more

efficiently

by

avoiding

the

explicit

construction

of

the

-LSB-

-RSB-

However

this

method

does

not

compute

all

available

inferences

on

satisfiable

such

inferences

are

valuable

when

the

is

embedded

in

a

larger

formula

Larrabee

's

algorithm

-LSB-

-RSB-

-LRB-

in

effect

-RRB-

uses

the

of

the

part

of

more

general

CNF

formulas

This

paper

extends

that

idea

by

adding

efficient

subsumption

resolution

and

by

incrementally

updating

the

and

backtracking

out

of

the

updates

when

necessary

Pretolani

-LSB-

-RSB-

as

well

as

Jaumard

et.al

-LSB-

-RSB-

presented

satisfiability

algorithms

that

exploit

relaxation

Combining

some

resolution

with

searching

was

also

reported

by

Billionnet

and

Sutter

-LSB-

-RSB-

but

they

do

not

give

enough

specifics

to

permit

a

detailed

comparison

Three-Closure

For

a

formula

F

it

might

happen

that

its

is

also

its

closure

in

which

case

absence

or

presence

of

the

empty

clause

immediately

decides

satisfiability

In

general

of

course

the

is

not

the

closure

as

the

resolvent

of

two

is

normally

a

which

is

not

in

the

However

the

might

contain

the

empty

clause

anyway

demonstrating

unsatisfiability

Definition

Suppose

a

set

of

clauses

S

contains

a

variable

v

such

that

some

resolvent

involving

v

as

the

annihilated

variable

is

not

subsumed

by

any

clause

in

S

Then

v

is

said

to

have

implicit

information

associated

with

it

If

no

variable

has

implicit

information

then

the

set

S

is

closed

and

the

satisfiability

question

is

trivial

-LRB-

look

for

;-RRB-

It

may

also

be

possible

to

exploit

the

fact

that

one

variable

has

no

implicit

information

Theorem

If

a

variable

x

contains

no

implicit

information

in

a

CNF

formula

then

all

clauses

containing

that

variable

can

be

deleted

without

changing

the

satisfiability

of

the

formula

Proof

A

valid

step

of

the

original

-LRB-

resolution-based

-RRB-

Davis

Putnam

procedure

is

to

perform

resolution

with

x

as

the

annihilated

variable

then

delete

all

clauses

containing

x.

But

by

Definition

all

resolvents

of

this

operation

are

already

in

the

formula

or

are

subsumed

by

clauses

already

in

the

formula

Unfortunately

testing

for

implicit

information

is

very

expensive

Nevertheless

the

concept

can

still

be

useful

A

set

of

binary

clauses

has

no

implicit

information

which

leads

to

one

of

our

principal

guidelines

Fact

It

is

never

necessary

to

guess

an

assignment

to

a

variable

that

occurs

only

in

binary

clauses

Although

we

do

not

want

to

pay

to

verify

the

presence

of

implicit

information

before

choosing

a

variable

to

branch

on

we

can

restrict

the

choice

to

those

that

most

probably

do

have

such

implicit

information

The

Ideal

Algorithm

The

above

considerations

suggest

an

algorithm

that

performs

on

the

current

formula

checks

for

then

checks

whether

there

is

any

variable

with

implicit

information

If

so

then

guess

an

assignment

to

that

variable

If

this

guess

leads

to

an

unsatisfiable

formula

then

backtrack

and

guess

the

complementary

assignment

In

both

cases

the

guessed

assignment

is

added

to

the

current

formula

as

a

new

unit

clause

and

the

algorithm

is

called

recursively

It

is

interesting

to

compare

the

above

approach

with

the

older

algorithms

for

CNF

satisfiability

The

first

algorithm

oriented

toward

CNF

formulas

was

due

to

Davis

and

Putnam

-LSB-

-RSB-

and

was

based

on

variable

elimination

through

resolution

It

used

the

unit

clause

rule

and

pure

literal

rule

as

heuristics

for

choosing

a

variable

but

when

those

were

not

applicable

it

specified

to

choose

a

variable

from

a

shortest

clause

All

possible

resolutions

with

this

variable

are

done

and

clauses

containing

this

variable

-LRB-

or

its

complement

-RRB-

are

eliminated

There

is

no

guessing

or

branching

but

longer

and

more

numerous

clauses

may

be

created

Subsumption

checking

is

an

option

to

reduce

the

size

of

the

formula

What

is

often

referred

to

in

current

literature

as

the

Davis-Putnam

algorithm

is

actually

a

modification

due

to

Davis

Logemann

and

Loveland

-LSB-

-RSB-

and

will

be

called

the

DPLL

algorithm

here

Neither

longer

clauses

nor

new

clauses

are

created

by

DPLL

The

DPLL

algorithm

uses

the

unit

clause

rule

-LRB-

a

special

case

of

resolution

-RRB-

and

the

pure

literal

rule

but

when

these

are

inapplicable

it

chooses

a

variable

from

a

shortest

clause

and

guesses

an

assignment

essentially

by

adding

that

literal

to

the

current

formula

as

a

new

unit

clause

and

recursively

calling

the

algorithm

If

this

guess

leads

to

an

unsatisfiable

formula

the

complement

is

guessed

and

the

algorithm

again

called

recursively

this

step

is

called

backtracking

If

both

guesses

create

unsatisfiable

formulas

then

the

current

formula

is

reported

as

unsatisfiable

Section

presents

performance

results

for

DPLL

However

the

chosen

variable

may

or

may

not

contain

implicit

information

For

example

if

the

current

formula

contains

-LRB-

v

a

-RRB-

-LRB-

v

b

c

-RRB-

-LRB-

a

b

c

-RRB-

and

other

clauses

but

no

more

occurrences

of

v

or

v

then

v

contains

no

implicit

information

but

might

be

chosen

by

DPLL

as

the

branching

or

guessing

variable

Several

variants

of

this

procedure

have

been

proposed

but

all

have

the

same

property

that

the

branching

variable

may

have

no

implicit

information

The

motivation

for

choosing

a

variable

in

a

shortest

clause

is

probably

to

create

more

unit

clauses

The

method

combines

the

ideas

of

resolution

and

assignment

guessing

By

only

doing

resolution

when

the

resolvent

is

three

or

fewer

literals

we

are

assured

of

a

polynomial

bound

-LRB-

per

guess

-RRB-

on

this

operation

By

always

choosing

a

variable

with

implicit

information

for

assignment

guessing

we

bring

the

formula

closer

to

one

whose

is

its

closure

We

add

two

further

optimizations

to

reduce

formula

size

The

pure

literal

rule

requires

no

comment

Equivalent

literal

recognition

if

-LRB-

a

b

-RRB-

and

-LRB-

b

a

-RRB-

are

present

then

all

occurrences

of

b

and

b

may

be

replaced

by

a

and

a

respectively

This

reduces

the

number

of

variables

and

usually

causes

many

clauses

to

be

subsumed

Definition

In

the

context

of

a

k-closure

algorithm

we

say

that

a

k-CNF

formula

is

k-stable

if

it

is

k-closed

does

not

contain

has

no

pure

literals

and

has

no

equivalent

literals

as

described

above

Casual

experimentation

shows

that

it

is

quite

difficult

to

construct

a

formula

that

also

contains

binary

clauses

One

example

is

shown

below

An

unusual

feature

of

our

ideal

algorithm

is

that

it

may

and

normally

does

detect

satisfiability

without

constructing

a

satisfying

assignment

Consider

this

formula

-LRB-

a

b

-RRB-

-LRB-

x

y

-RRB-

-LRB-

a

x

y

-RRB-

-LRB-

y

b

a

-RRB-

-LRB-

b

x

y

-RRB-

-LRB-

x

b

a

-RRB-

It

is

seen

to

be

and

is

in

fact

the

smallest

formula

containing

binary

clauses

we

have

been

able

to

construct

Because

it

is

closed

and

does

not

contain

the

empty

clause

it

must

be

satisfiable

Some

applications

require

an

explicit

satisfying

assignment

to

be

produced

Fortunately

this

can

be

done

efficiently

by

relying

on

the

following

observation

Theorem

If

literal

x

occurs

in

a

closed

formula

S

then

S

has

a

model

in

which

x

is

true

Proof

The

formula

S

has

no

implicit

information

so

x

can

not

be

a

logical

consequence

of

S

by

the

completeness

of

resolution

So

a

satisfying

assignment

for

a

closed

k-stable

formula

S

can

be

found

recursively

by

choosing

any

literal

x

in

a

non-unit

clause

of

S

and

then

finding

a

satisfying

assignment

for

the

closure

of

-LRB-

S

-LSB-

-LRB-

x

-RRB-

-RRB-

The

Practical

Algorithm

As

mentioned

before

to

determine

whether

a

variable

contains

implicit

information

it

is

necessary

first

that

the

formula

be

and

then

all

possible

resolutions

must

be

tried

to

see

if

any

produce

an

unsubsumed

We

programmed

a

short-cut

that

chooses

variable

that

is

likely

to

have

implicit

information

Once

we

abandon

strict

implicit

information

there

is

little

motivation

for

performing

complete

However

certain

of

these

operations

on

long

clauses

-LRB-

or

more

literals

-RRB-

are

efficient

and

very

valuable

Krom

subsumption

resolution

removes

one

literal

from

a

long

clause

fa

xg

and

f

a

x

y

g

resolve

to

fx

y

g

which

subsumes

the

original

f

a

x

y

g.

Simple

subsumption

removes

a

long

clause

when

a

binary

clause

or

unit

clause

implies

it

This

reduces

the

non-information

in

long

clauses

so

branching

choices

will

be

more

pertinent

In

addition

the

following

reduction

might

gain

efficiency

Corollary

If

a

variable

x

-LRB-

including

x

-RRB-

occurs

only

in

binary

clauses

in

a

CNF

formula

-LRB-

possibly

containing

longer

clauses

-RRB-

then

all

clauses

containing

that

variable

can

be

deleted

without

changing

the

satisfiability

of

the

formula

Proof

Variable

x

has

no

implicit

information

so

Theorem

applies

While

removing

such

clauses

is

an

option

in

our

implementation

the

situation

seems

to

arise

rarely

and

we

have

not

observed

clear-cut

benefits

in

practice

Our

algorithm

is

a

modification

of

DPLL

Between

guesses

DPLL

repeatedly

performs

unit

clause

simplification

and

pure

literal

simplification

until

no

further

simplifications

are

possible

Between

guesses

repeatedly

performs

-LRB-

which

includes

unit

clause

simplification

-RRB-

pure

literal

simplification

equivalent

literal

simplification

Krom

subsumption

resolution

and

simple

subsumption

until

a

formula

-LRB-

Definition

-RRB-

is

attained

The

implementation

for

which

we

report

experimental

results

maintains

explicit

that

is

every

derivable

unsubsumed

binary

clause

is

represented

explicitly

in

the

data

structure

Each

literal

is

associated

with

the

sorted

list

of

clauses

containing

it

so

that

it

is

reasonably

efficient

to

locate

all

long

clauses

containing

two

specified

literals

by

the

intersection

operation

This

secondary

index

supports

Krom

subsumption

resolution

and

simple

subsumption

Recall

that

the

set

of

can

be

regarded

as

edges

in

an

implication

graph

Explicit

amounts

to

maintaining

its

transitive

closure

The

important

information

that

may

be

present

in

the

transitive

closure

that

can

not

be

detected

from

the

strong

component

analysis

is

a

path

from

a

literal

to

its

complement

x

x

from

which

the

unit

clause

x

can

be

inferred

However

updating

the

transitive

closure

is

potentially

expensive

following

a

guess

-LRB-

a

backtrackable

variable

assignment

-RRB-

one

new

inferred

edge

can

generate

O

-LRB-

-RRB-

secondary

updates

Moreover

they

all

need

to

be

retracted

upon

backtracking

While

efficient

transitive

closure

has

been

much

studied

we

are

not

aware

of

any

work

that

considers

the

need

to

backtrack

out

of

updates

to

the

graph

We

use

a

straightforward

method

in

which

the

transitively

closed

graph

is

an

array

-LRB-

indexed

by

variable

-RRB-

of

adjacency

lists

Each

new

edge

that

is

added

is

recorded

in

a

journal

and

upon

backtracking

the

journal

is

used

to

roll

back

the

updates

as

is

common

in

database

systems

Experimental

Results

This

section

presents

experimental

data

on

the

proposed

algorithm

with

attention

to

variations

that

all

have

the

same

reasoning

component

See

Section

for

comparisons

with

DPLL

and

discussion

of

the

trade-off

between

reasoning

and

guessing

Six

branching

strategies

were

tested

on

a

variety

of

problems

including

several

Dimacs

benchmarks

These

branching

strategies

were

shown

to

perform

differently

with

statistical

significance

A

new

strategy

based

on

Johnson

's

maximum

satisfiability

approximation

algorithm

proved

the

best

for

the

class

of

random

formulas

However

for

the

circuit

fault-detection

formulas

the

performance

of

the

branching

strategies

displayed

a

less

clear

pattern

Student

's

paired

t-test

was

used

to

test

for

the

significantly

different

means

of

the

number

of

guesses

for

these

branching

strategies

Such

statistical

tests

must

be

applied

with

care

further

discussions

on

the

statistical

traps

and

techniques

for

NP-complete

problems

are

found

in

Appendix

A.

Typical

Questions

There

are

numerous

satisfiability

algorithms

and

corresponding

implementations

We

consider

the

following

typical

questions

that

arise

in

comparative

evaluations

of

satisfiability

programs

Are

there

significant

differences

among

the

observed

performances

of

the

given

set

of

programs

on

the

same

distribution

of

formulas

Is

the

relative

goodness

of

programs

affected

if

we

use

different

distributions

of

formulas

Is

performance

of

an

algorithm

sensitive

to

the

presentation

of

the

formula

By

presentation

we

mean

choice

of

variable

numbering

clause

order

etc.

Performance

Measures

Our

principal

measure

of

resource

usage

other

than

CPU

time

is

the

number

of

guesses

or

branches

A

guess

is

a

variable

assignment

that

may

change

the

satisfiability

of

the

formula

so

its

complement

may

have

to

be

considered

-LRB-

or

was

considered

earlier

-RRB-

The

number

of

guesses

has

the

advantage

of

being

a

reproducible

measure

so

it

is

used

for

most

presentations

See

Appendix

A

for

further

discussion

For

the

random

formulas

reported

upon

here

we

have

found

by

regression

analysis

that

CPU

time

-LRB-

for

-RRB-

for

is

modeled

quite

accurately

by

the

equation

cpusec

g

-LRB-

L

-RRB-

where

g

is

the

number

of

guesses

and

L

is

the

number

of

literal

occurrences

The

root

mean

square

error

of

the

model

was

on

data

whose

standard

deviation

was

so

informally

it

explains

of

the

variation

observed

The

regression

data

included

all

branching

strategies

all

tested

clause/variable

ratios

and

most

tested

formula

sizes

For

structured

formulas

no

adequate

regression

equation

was

found

Statistics

on

guesses

and

CPU

time

for

both

random

and

structured

formulas

are

given

in

various

tables

and

figures

as

discussed

throughout

the

paper

Branching

Strategies

To

provide

some

answers

to

the

above

questions

we

conducted

satisfiability

experiments

on

various

classes

of

formulas

we

also

compared

six

variants

of

our

implementation

of

the

algorithm

described

in

Section

These

variants

correspond

to

the

six

different

selection

criteria

for

the

branching

variables

For

all

criteria

a

positive

score

and

a

negative

score

are

computed

for

each

variable

The

variable

's

final

score

is

the

product

of

the

positive

and

negative

components

The

eligible

variable

-LRB-

eligibility

is

defined

by

each

branching

rule

-RRB-

with

the

maximum

final

score

is

chosen

for

branching

Our

use

of

the

product

instead

of

the

sum

appears

to

be

unique

in

the

literature

It

is

motivated

by

the

desire

to

reduce

the

size

of

each

subproblem

substantially

For

example

if

branching

on

x

achieves

reductions

Circuit

Number

of

Branches

Family

statistics

maxscore

minlen

maxlen

dsj

Mean

184 527 184 511

StdDev

sat

Min

unsat

Median

202 494 202 496

Max

272 920 272 858

Mean

1398 5066 1398 1853

StdDev

2641 8216 2641 1980

sat

Min

unsat

Median

250 1296 250 851

Max

6310 8756 8412

Mean

StdDev

sat

Min

unsat

Median

Max

Circuit

CPU

seconds

for

Family

statistics

maxscore

minlen

maxlen

dsj

Mean

Max

Mean

Max

Mean

Max

Figure

The

statistics

for

the

number

of

guesses

and

the

running

time

by

the

six

branching

strategies

of

on

circuit

fault-detection

formulas

More

information

on

the

number

of

variables

and

literals

for

these

groups

of

formulas

can

be

found

in

Figure

of

and

while

branching

on

y

achieves

reductions

of

and

we

prefer

y

with

the

higher

product

rather

than

higher

sum

maxscore

All

variables

are

eligible

score

is

sum

of

occurrences

in

all

clauses

minlen

Variables

that

occur

in

a

minimum

length

clause

are

eligible

score

is

sum

of

occurrences

in

minimum

length

clauses

Variables

that

occur

in

a

minimum

length

clause

are

eligible

score

is

sum

of

occurrences

in

all

clauses

maxlen

Variables

that

occur

in

a

clause

of

length

at

least

are

eligible

score

is

sum

of

occurrences

in

clauses

of

length

at

least

-LRB-

If

there

are

only

binary

clauses

then

this

reverts

to

minlen

-RRB-

Variables

that

occur

in

a

clause

of

length

at

least

are

eligible

score

is

sum

of

occurrences

in

all

clauses

-LRB-

If

there

are

only

binary

clauses

then

this

reverts

to

minlen

-RRB-

dsj

All

variables

are

eligible

score

is

the

weighted

sum

of

occurrences

in

all

clauses

Binary

clauses

count

twice

as

much

as

and

count

twice

as

much

as

the

clauses

of

longer

length

This

is

the

modified

version

of

the

weighting

used

in

D.

S.

Johnson

's

maximum

satisfiability

approximation

algorithm

-LSB-

-RSB-

clauses

samples

maxscore

minlen

maxlen

dsj

450 100 101

Figure

The

average

number

of

guesses

by

the

six

branching

strategies

of

on

random

formulas

-LRB-

constant

width

model

-RRB-

with

variables

and

varying

numbers

of

clauses

Standard

deviations

are

prefixed

by

Jeroslow

and

Wang

-LSB-

-RSB-

have

reported

on

a

satisfiability

algorithm

that

uses

a

somewhat

similar

branching

strategy

except

they

maximize

over

positive

and

negative

components

where

we

multiply

Their

motivation

is

most

likely

to

satisfy

Hooker

and

Vinay

have

challenged

the

most

likely

to

satisfy

explanation

-LRB-

see

also

Section

-RRB-

and

have

proposed

the

Jeroslow-Wang

rule

with

the

motivation

maximize

size

reduction

-LSB-

-RSB-

This

rule

is

like

dsj

except

that

it

adds

where

we

multiply

There

may

be

multiple

candidates

for

the

choice

of

a

branching

variable

A

tie

between

any

two

such

variables

is

broken

by

selecting

a

lower

or

higher

variable

number

depending

on

the

parity

of

their

sum

We

provide

further

randomization

by

shuffling

the

variable

numbers

at

the

preprocessing

stage

Observations

on

the

Branching

Strategies

The

six

branching

strategies

were

run

on

some

classes

of

circuit

fault-detection

formulas

Results

appear

in

Figure

We

observed

evident

weakness

of

minlen

strategy

on

the

bf

family

Due

to

the

small

sample

sizes

and

large

variance

for

these

classes

of

formulas

we

have

not

been

successful

in

establishing

significance

of

the

differences

in

the

average

number

of

guesses

Figure

presents

additional

results

for

dsj

only

on

a

larger

set

of

circuit

fault-detection

formulas

The

six

branching

strategies

were

also

run

on

random

formulas

which

were

generated

according

to

the

constant

width

model

each

triple

of

distinct

variables

is

equally

likely

to

be

selected

for

a

clause

each

variable

is

signed

plus

or

minus

with

equal

probability

and

all

clauses

are

drawn

independently

with

replacement

Results

appear

in

Figure

The

dsj

and

minlen

branching

strategies

had

the

lowest

overall

averages

Because

of

the

poor

performance

of

the

minlen

strategy

on

circuit

fault-detection

formulas

we

chose

dsj

as

the

best

candidate

To

determine

the

statistical

significance

of

the

results

the

dsj

strategy

was

compared

against

each

of

the

other

five

on

five

samples

of

varying

clause/variable

ratio

making

cases

in

all

The

application

of

the

Student

's

paired

t-test

-LSB-

-RSB-

using

a

published

C-language

implementation

-LSB-

-RSB-

-LRB-

reviewed

in

Appendix

A

-RRB-

yields

the

conclusion

that

dsj

branching

strategy

outperforms

others

with

statistical

significance

-LRB-

at

level

-RRB-

in

cases

underperforms

significantly

in

cases

and

is

not

significantly

different

in

cases

The

details

including

exact

probability

values

are

given

in

Figure

clauses

samples

maxscore

minlen

maxlen

Figure

Student

's

paired

t-test

significance

probabilities

of

the

mean

difference

from

the

dsj

branching

strategy

on

random

formulas

-LRB-

constant

width

model

-RRB-

with

variables

We

indicate

the

significance

level

of

for

the

paired

t-test

by

For

each

entry

with

a

significant

mean

difference

dsj

performed

better

unless

it

is

also

marked

with

satisfiable

unsatisfiable

clauses

variables

samples

branches

time

samples

branches

time

300 100 100

Figure

The

different

difficulty

patterns

of

satisfiable

and

unsatisfiable

random

formulas

Time

is

in

seconds

for

These

tests

convinced

us

that

the

new

dsj

strategy

was

the

best

among

the

alternatives

considered

for

Most

subsequently

reported

experiments

use

this

strategy

exclusively

Satisfiable

vs.

Unsatisfiable

Random

Formulas

We

have

tested

both

satisfiable

and

unsatisfiable

random

formulas

with

variables

and

from

to

clauses

including

clauses

believed

to

be

the

hardest

point

in

the

spectrum

-LSB-

-RSB-

The

result

for

the

dsj

branching

strategy

appears

in

Figure

This

figure

shows

Unsatisfiable

random

problems

do

not

follow

the

easy-hard-easy

pattern

Satisfiable

problems

are

much

easier

on

balance

Growth

Rate

Function

on

Random

Formulas

We

have

also

tested

a

range

from

to

variables

maintaining

the

clause-to-variable

ratio

The

algorithm

has

solved

them

with

surprisingly

little

backtracking

for

variables

the

average

number

of

guesses

was

for

variables

the

average

number

of

guesses

was

For

variables

the

number

of

guesses

ranged

from

to

for

satisfiable

and

from

to

for

unsatisfiable

CPU

times

are

reported

in

Section

Figure

summarizes

the

results

Growth

rate

was

analyzed

separately

for

unsatisfiable

and

satisfiable

formulas

since

the

satisfiable

are

to

times

easier

and

the

fraction

of

satisfiable

formulas

at

a

fixed

ratio

changes

with

the

number

of

variables

500 1000 1500 2000

2500 3000 3500 4000

number

of

guesses

-LRB-

log

scaled

-RRB-

number

of

literals

Unsatisfiable

Formulas

All

Formulas

Satisfiable

Formulas

Total

Sample

Branches

Result

Vars

literals

Size

Ave

Stderr

Stddev

Min

Median

Max

unsat

31 48 116 212

540 14 132 208

200 2562 110 4860

159 1671 2164 4454

sat

50 642 113

71 909 113

100 1281 109

141 1806 117 168

181 10 105 1151

2562 90 1320 149

1409 18 835 7002

Figure

Growth

rate

of

on

random

formulas

using

the

dsj

option

The

average

number

of

guesses

is

plotted

against

the

number

of

literals

on

a

logscale

The

formulas

generated

all

have

the

clausesto-variables

ratio

of

Cpu

secs

-LRB-

-RRB-

Branches

Family

Fmlas

Solved

Ave

Stddev

Min

Median

Max

170 170 21177

6634 67 142 26215

Figure

Performance

of

with

dsj

branching

strategy

on

inductive

inference

Dimacs

benchmark

formulas

of

Mean

Mean

Circuit

of

Sat

of

of

Number

of

Branches

Family

Fmlas

Fmlas

Vars

Lits

Ave

Stddev

Min

Median

Max

501 2481 268 137

80 320 400

12 1530 7835 52044

80 80 1626 8208

1183 8296 1407 2430

149 2829 19319

53 16 1531 8366

2428 4174892

CPU

seconds

for

Ave

Stddev

Min

Median

Max

12 1530 7835 1539

80 80 1626 8208

149 2829 19319

53 16 1531 8366

Figure

Performance

of

with

dsj

branching

strategy

on

the

Circuit

formulas

However

no

significant

difference

in

growth

rate

was

observed

between

these

two

groups

See

Figure

Our

experiments

for

the

dsj

branching

strategy

indicate

an

exponential

growth

rate

of

L

where

L

is

the

number

of

occurrences

of

literals

in

the

formula

The

estimated

coefficient

C

was

obtained

by

fitting

the

equation

mean

branches

to

the

observed

values

of

for

L

and

for

L

Based

on

the

standard

errors

in

Figure

the

standard

error

of

C

is

about

At

this

ratio

variables

occur

an

average

of

times

so

the

growth

rate

in

terms

of

n

variables

appears

to

be

n

this

coefficient

's

standard

error

is

about

Observations

on

the

Inductive

Inference

Formulas

The

Dimacs

inductive

inference

benchmark

formulas

were

attempted

with

spotty

results

Results

appear

in

Figure

With

a

limit

of

hour

CPU

time

imposed

the

algorithm

only

solved

out

of

problems

All

of

the

tests

given

in

the

result

were

done

with

the

same

variable

shuffling

algorithm

Without

the

variable

shuffling

we

observed

that

one

more

formula

was

solved

within

the

given

time

limit

however

one

formula

was

no

longer

solved

with

the

net

result

being

very

similar

Observations

on

the

Circuits

Formulas

Figure

shows

the

result

on

all

the

circuits

formulas

that

have

been

submitted

to

the

DIMACS

database

of

cnf

formulas

Perhaps

the

most

remarkable

result

in

this

group

is

that

the

three

unsatisfiable

formulas

with

an

average

of

variables

were

solved

with

no

guessing

at

all

We

are

not

aware

of

any

other

algorithm

that

has

succeeded

solving

these

formulas

All

formulas

in

the

group

were

solved

although

one

required

CPU

hours

Experimental

Techniques

This

section

discusses

two

techniques

used

in

our

experiments

the

first

one

assures

the

randomness

of

the

formulas

while

maintaining

the

repeatability

of

the

experiment

the

second

provides

information

about

the

sensitivity

of

an

algorithm

to

accidents

in

the

presentation

of

the

formula

Seed

Conversion

The

basic

assumption

in

the

experiments

on

the

randomly

generated

formulas

is

that

the

input

formulas

are

chosen

independently

at

random

from

a

specified

parent

population

and

distribution

Such

randomness

is

typically

simulated

by

a

formula

generator

that

uses

a

reliable

random

number

generator

whose

initial

seed

value

is

either

produced

internally

or

provided

at

invocation

time

by

the

user

In

either

case

unless

one

remembers

all

the

initial

seeds

the

repeatability

of

the

experiment

is

lost

This

becomes

an

unreasonable

burden

for

the

experimenter

since

easily

remembered

set

of

seeds

can

compromise

the

randomness

factor

For

example

if

the

same

initial

seed

was

used

to

generate

a

formula

of

clauses

and

one

with

clauses

both

with

the

same

number

of

variables

then

there

is

a

dependence

between

those

two

formulas

in

fact

one

is

a

prefix

of

the

other

To

avoid

this

problem

for

those

experiments

in

Sections

and

we

used

a

script

that

converts

a

user

specified

seed

in

an

easily

remembered

range

such

as

-LCB-

into

a

unique

actual

seed

to

be

used

by

the

formula

generator

via

the

following

formula

-LRB-

-LRB-

v

c

-RRB-

i

-RRB-

mod

where

v

c

and

i

are

the

parameter

values

corresponding

to

the

number

of

variables

the

number

of

clauses

and

the

user

specified

seed

respectively

The

same

range

say

-LCB-

can

be

used

for

a

variety

of

values

of

v

and

c

without

incurring

any

dependence

Internal

Variable

Shuffling

To

determine

sensitivity

to

the

formula

presentation

we

provide

the

option

to

shuffle

internal

variable

numbers

at

the

preprocessing

stage

By

default

variables

are

internally

numbered

in

order

of

first

appearance

and

these

numbers

may

determine

how

ties

are

broken

This

shuffling

does

not

change

the

formula

in

an

essential

way

It

is

also

referred

to

as

variable

reordering

The

programs

we

used

in

this

experiment

have

an

execution

time

option

to

shuffle

the

variable

numbers

in

the

input

formula

at

the

preprocessing

stage

The

default

seed

value

is

to

indicate

no

shuffling

but

500 1000 1500 2000

2500 3000 3500 4000

number

of

guesses

-LRB-

log

scaled

-RRB-

number

of

literals

Guesses

Growth

Comparison

DPLL

with

maxscore

with

maxscore

and

with

dsj

DPLL

maxscore

maxscore

dsj

number

number

Number

of

guesses

of

of

DPLL

maxscore

maxscore

dsj

variables

literals

Mean

StdErr

Mean

StdErr

Mean

StdErr

11 71 909 263

100 1281 1354

141 1806 10474

200 2562 322426

5216 282 3267 166

283 3624 23420708

Figure

Growth

rate

comparison

of

and

DPLL

on

random

formulas

The

average

number

of

guesses

is

plotted

against

the

number

of

literals

on

a

logscale

The

formulas

generated

all

have

the

clausesto-variables

ratio

of

user

may

specify

any

integer

as

the

seed

for

the

variable

shuffling

This

technique

was

used

as

a

tool

in

investigating

the

sensitivity

of

the

algorithmic

performance

to

the

formula

presentation

As

indicated

in

Section

and

in

Appendix

B

variable

shuffling

by

could

make

a

big

performance

difference

in

some

structured

formulas

However

it

made

very

little

difference

in

random

formulas

Time

Trade-Offs

for

Reasoning

vs.

Guessing

500 1000 1500 2000

2500 3000 3500 4000

number

of

seconds

-LRB-

log

scaled

-RRB-

number

of

literals

CPU

Time

Growth

Comparison

DPLL

with

maxscore

with

maxscore

and

with

dsj

DPLL

maxscore

maxscore

dsj

number

of

number

of

Mean

CPU

seconds

for

variables

literals

DPLL

maxscore

maxscore

dsj

200 2562 214

283 3624 21500

Figure

Growth

rate

comparison

of

and

DPLL

on

random

formulas

The

average

number

of

CPU

seconds

-LRB-

Sun

equivalent

as

explained

in

text

-RRB-

is

plotted

against

the

number

of

literals

on

a

logscale

The

formulas

generated

all

have

the

clauses-to-variables

ratio

of

The

theme

of

is

to

reduce

the

size

of

the

search

space

as

measured

by

the

number

of

guesses

by

incorporating

an

efficient

reasoning

component

into

the

search

A

natural

question

is

whether

the

time

invested

in

reasoning

pays

dividends

in

terms

of

reduced

search

To

address

this

question

we

compared

with

an

efficient

implementation

of

DPLL

The

same

samples

of

random

formulas

reported

in

Section

were

used

ranging

from

to

variables

at

the

ratio

all

sizes

had

samples

except

that

had

samples

Besides

relative

magnitudes

we

were

especially

interested

in

growth

rates

Because

the

theme

of

DPLL

is

to

guess

as

fast

as

possible

it

is

not

consistent

to

spend

time

maintaining

the

information

needed

to

implement

the

dsj

strategy

However

the

maxscore

strategy

can

be

implemented

with

little

overhead

and

proved

to

be

far

superior

to

the

published

strategy

of

choosing

any

variable

in

a

minimum-length

clause

-LRB-

on

random

formulas

at

least

-RRB-

Results

are

based

on

the

maxscore

strategy

for

DPLL

We

ran

with

both

the

maxscore

and

dsj

strategies

Figure

shows

the

growth

rates

in

terms

of

guesses

for

both

algorithms

Unsurprisingly

does

less

guessing

The

important

observation

is

that

exponential

growth

rate

for

guessing

is

much

lower

for

than

for

DPLL

as

shown

by

the

diverging

lines

Assuming

these

rates

persist

to

larger

problems

it

is

inevitable

that

will

eventually

require

less

time

because

it

spends

only

a

polynomial

factor

greater

time

per

guess

than

DPLL

Using

the

method

of

Section

where

dsj

was

found

to

grow

at

the

rate

L

here

we

find

that

with

maxscore

grows

at

L

while

DPLL

grows

at

L.

-LRB-

The

standard

errors

of

all

of

these

coefficients

is

about

-RRB-

Figure

shows

the

growth

rates

in

terms

of

CPU

time

for

both

algorithms

DPLL

was

executed

on

while

was

executed

on

a

slower

DECStation

The

CPU

times

measured

on

DECStation

were

transformed

to

the

equivalent

times

by

using

the

conversion

factor

of

This

graph

confirms

that

the

future

is

now

For

the

dsj

branching

strategy

the

extra

reasoning

begins

showing

a

profit

around

variables

-LRB-

literals

-RRB-

and

the

margin

widens

rapidly

for

larger

sizes

Roughly

speaking

by

considering

time

instead

of

guesses

the

DPLL

curve

translates

downward

relative

to

but

still

has

the

greater

slope

To

show

that

the

performance

difference

between

and

DPLL

is

attributable

to

reasoning

and

not

a

difference

in

branching

strategies

we

also

ran

with

the

less

effective

maxscore

strategy

which

is

the

strategy

that

worked

best

for

DPLL

We

see

the

similar

situation

in

this

case

although

the

cross-over

point

with

the

DPLL

is

higher

about

variables

-LRB-

literals

-RRB-

Conclusion

and

Future

Work

The

combination

of

reasoning

and

guessing

seems

to

be

beneficial

Careful

implementation

is

needed

to

keep

the

cost

of

reasoning

down

Further

investigation

is

needed

to

determine

whether

the

expensive

operations

can

be

done

more

efficiently

and

put

back

into

service

Analysis

of

the

asymptotic

complexity

both

of

the

worst

case

and

of

the

average

case

on

random

formulas

is

another

avenue

of

investigation

There

are

some

known

results

on

the

worst-case

performance

of

satisfiability

on

CNF

formulas

using

algorithms

that

add

various

forms

of

reasoning

to

the

basic

DPLL

framework

-LSB-

-RSB-

For

random

CNF

formulas

of

length

L

at

the

ratio

studied

here

the

smallest

known

worst

case

bound

is

L

This

is

still

far

above

our

observed

growth

rate

of

L.

Statistical

techniques

including

nonparametric

methods

-LSB-

-RSB-

for

the

comparative

evaluation

of

different

strategies

is

another

topic

that

can

be

further

pursued

Distributions

for

NP-hard

problems

are

typically

highly

skewed

For

most

of

the

structured

i.e.

nonrandom

formulas

we

observed

that

the

maximum

number

of

guesses

was

to

times

the

second

largest

number

of

guesses

in

the

same

sample

Consequently

it

is

important

to

report

a

variety

of

statistics

to

give

a

good

picture

of

the

algorithm

's

behavior

Besides

the

average

and

standard

deviation

we

have

also

reported

the

minimum

median

and

maximum

Hooker

has

advocated

an

empirical

approach

to

algorithm

analysis

that

involves

hypothesis

formation

and

prediction

in

analogy

with

natural

sciences

-LSB-

-RSB-

Many

heuristics

are

proposed

in

the

literature

with

informal

explanations

of

why

they

work

well

Hooker

believes

that

the

explanations

should

be

examined

more

critically

and

proposes

a

framework

for

doing

so

if

the

explanation

is

correct

then

we

should

be

able

to

make

new

predictions

and

verify

them

If

on

the

other

hand

the

predictions

are

not

borne

out

then

that

indicates

that

the

intuitive

explanation

has

somehow

not

really

captured

the

reason

for

the

heuristic

's

observed

success

In

the

latter

case

we

should

begin

looking

for

a

better

explanation

Hooker

and

Vinay

have

reported

on

an

application

of

this

technique

as

discussed

in

Section

-LSB-

-RSB-

Gent

and

Walsh

indirectly

applied

the

technique

to

arrive

at

negative

conclusions

about

the

importance

of

greediness

and

randomness

in

local

search

-LSB-

-RSB-

This

technique

seems

to

offer

great

promise

for

discovery

of

better

algorithms

for

intractable

problems

Acknowledgements

We

thank

the

anonymous

referee

for

many

suggestions

on

improving

the

paper

Many

of

the

experiments

were

facilitated

by

equipment

that

was

donated

by

Sun

Microsystems

Inc.

Both

authors

were

supported

in

part

by

NSF

grant

References

-LSB-

-RSB-

B.

Aspvall

M.

Plass

and

R.

Tarjan

A

linear-time

algorithm

for

testing

the

truth

of

certain

quantified

Boolean

formulas

Information

Processing

Letters

-LRB-

-RRB-

-LCB-

March

-LSB-

-RSB-

A.

Billionnet

and

A.

Sutter

An

efficient

algorithm

for

the

problem

Operations

Research

Letters

-LCB-

July

-LSB-

-RSB-

M.

Davis

G.

Logemann

and

D.

Loveland

A

machine

program

for

theorem-proving

Communications

of

the

ACM

-LCB-

-LSB-

-RSB-

M.

Davis

and

H.

Putnam

A

computing

procedure

for

quantification

theory

Journal

of

the

Association

for

Computing

Machinery

-LCB-

-LSB-

-RSB-

J.

D.

Gibbons

and

S.

Chakraborti

Nonparametric

Statistical

Inference

Marcel

Dekker

Inc.

-LSB-

-RSB-

I.

P.

Gent

and

T.

Walsh

Towards

an

understanding

of

hill-climbing

procedures

for

SAT

In

Proceedings

of

the

Eleventh

National

Conference

on

Artificial

Intelligence

and

IAAI

-LRB-

Washington

DC

USA

July

-RRB-

pages

-LCB-

Menlo

Park

CA

USA

AAAI

Press

-LSB-

-RSB-

J.

N.

Hooker

Needed

An

empirical

science

of

algorithms

Operations

Research

-LRB-

-RRB-

-LCB-

March-April

-LSB-

-RSB-

J.

N.

Hooker

and

V.

Vinay

Branching

rules

for

satisfiability

In

Third

International

Symposium

on

Artificial

Intelligence

and

Mathematics

Fort

Lauderdale

Florida

-LSB-

-RSB-

D.

S.

Johnson

Approximation

algorithms

for

combinatorial

problems

Journal

of

Computer

and

System

Sciences

-LCB-

-LSB-

-RSB-

B.

Jaumard

M.

Stan

and

J.

Desrosiers

Tabu

search

and

a

quadratic

relaxation

for

the

satisfiability

problem

In

D.

S.

Johnson

and

M.

Trick

editors

Cliques

Coloring

and

Satisfiability

Second

DIMACS

Implementation

Challenge.

DIMACS

Series

in

Discrete

Mathematics

and

Theoretical

Computer

Science

American

Mathematical

Society

-LSB-

-RSB-

R.

Jeroslow

and

J.

Wang

Solving

propositional

satisfiability

problems

Annals

of

Mathematics

and

Artificial

Intelligence

-LCB-

-LSB-

-RSB-

O.

Kullmann

and

H.

Luckhardt

Various

complexity

upper

bounds

for

decisions

on

propositional

tautology

Information

and

Computation

To

apppear

-LSB-

-RSB-

O.

Kullmann

Methods

for

in

less

than

n

steps

Technical

report

University

of

Frankfurt

-LSB-

-RSB-

T.

Larrabee

Test

pattern

generation

using

Boolean

satisfiability

IEEE

Transactions

on

Computer-Aided

Design

-LRB-

-RRB-

-LCB-

January

-LSB-

-RSB-

T.

Larrabee

and

Y.

Tsuji

Evidence

for

a

satisfiability

threshold

for

random

formulas

Technical

Report

UCSC

-LCB-

CRL

-LCB-

-LCB-

UC

Santa

Cruz

Santa

Cruz

CA.

October

-LSB-

-RSB-

B.

Monien

and

E.

Speckenmeyer

Solving

satisfiability

in

less

than

steps

Discrete

Applied

Mathematics

-LCB-

-LSB-

-RSB-

D.

Mitchell

B.

Selman

and

H.

Levesque

Hard

and

easy

distributions

of

SAT

problems

In

Proceedings

of

the

Tenth

National

Conference

on

Artificial

Intelligence

-LRB-

-RRB-

San

Jose

CA.

pages

-LCB-

July

-LSB-

-RSB-

D.

Pretolani

Efficiency

and

stability

of

hypergraph

SAT

algorithms

In

D.

S.

Johnson

and

M.

Trick

editors

Cliques

Coloring

and

Satisfiability

Second

DIMACS

Implementation

Challenge.

DIMACS

Series

in

Discrete

Mathematics

and

Theoretical

Computer

Science

American

Mathematical

Society

-LSB-

-RSB-

W.

H.

Press

S.

A.

Teukolsky

W.

T.

Vetterling

and

B.

P.

Flannery

Numerical

Recipes

in

C.

Cambridge

University

Press

second

edition

-LSB-

-RSB-

I.

Schiermeyer

Solving

in

less

than

n

steps

In

Computer

Science

Logic

Workshop

CSL

-LRB-

San

Miniato

Italy

Sept.

Oct.

-RRB-

LN

Comp

Sci

-LRB-

-RRB-

pages

-LCB-

Berlin

Germany

Springer-Verlag

-LSB-

-RSB-

A.

Van

Gelder

A

satisfiability

tester

for

non-clausal

propositional

calculus

Information

and

Control

-LRB-

-RRB-

-LCB-

October

-LSB-

-RSB-

J.

Welkowitz

R.

B.

Ewen

and

J.

Cohen

Introductory

Statistics

for

the

Behavioral

Sciences

Harcourt

Brace

Jovanovich

College

fourth

edition

A

Statistical

Traps

and

Techniques

for

NP-Complete

Problems

NP-hard

problems

can

produce

highly

skewed

distributions

of

certain

performance

measures

Care

is

needed

to

avoid

erroneous

interpretations

and

unjustified

conclusions

Consider

the

performance

of

algorithms

on

randomly

generated

formulas

from

the

same

family

The

table

below

shows

averages

for

independent

samples

of

runs

in

cpu

times

and

branching

Average

Average

Algorithm

CPU

Secs

Branches

A

B

C

D

Branches

CPU

Figure

Cumulative

distribution

of

independent

samples

of

the

Student

significance

statistic

S

Algorithms

B

and

C

look

pretty

indistinguishable

But

with

samples

clearly

A

is

better

than

D

right

Wrong

All

four

runs

were

made

with

the

same

deterministic

algorithm

with

independent

randomly

generated

formulas

This

data

is

not

cooked

up

It

is

the

summary

of

our

only

attempt

to

prove

our

point

not

the

worst

case

over

many

attempts

Trap

Look

only

at

the

averages

It

is

well

known

that

standard

deviations

or

other

measures

of

dispersion

are

necessary

to

know

the

accuracy

of

averages

but

we

see

a

great

many

tables

with

just

averages

and

no

measures

of

their

accuracy

In

the

above

data

the

standard

deviations

were

greater

than

the

averages

A

fairly

standard

way

to

test

for

a

significant

difference

between

two

average

values

is

the

Student

's

t-test

-LSB-

-RSB-

This

test

was

derived

assuming

that

both

samples

were

drawn

from

Gaussian

distributions

with

the

same

variance

but

possibly

different

means

-LRB-

average

values

-RRB-

Trap

Use

the

Student

t-test

without

knowing

whether

the

actual

distribution

is

approximately

Gaussian

The

Student

t-test

has

a

reputation

for

being

robust

that

is

not

overly

sensitive

to

the

distribution

assumptions

However

exponential

time

algorithms

can

stress

those

assumptions

beyond

the

limits

foreseen

by

traditional

statisticians

Is

the

Student

t-test

still

reliable

although

the

distribution

is

decidedly

non-Gaussian

Under

ideal

conditions

if

two

samples

are

drawn

from

the

same

distribution

the

end

result

of

their

Student

t-test

call

it

S

is

a

uniformly

distributed

random

variable

in

the

range

to

That

is

why

upon

observing

S

to

be

in

an

experiment

one

can

say

If

the

two

distributions

were

identical

-LRB-

the

hypothesis

-RRB-

the

probability

of

getting

an

S

this

small

or

smaller

is

So

construct

some

samples

where

you

know

the

hypothesis

holds

Calculate

the

Student

statistic

-LRB-

S

not

t

-RRB-

over

numerous

independent

pairs

of

such

samples

These

values

of

S

should

appear

to

be

uniformly

distributed

If

not

the

distribution

is

too

skewed

for

the

Student

t-test

to

be

accurate

We

carried

out

this

experiment

using

our

program

with

both

the

number

of

branches

and

the

CPU

time

in

seconds

as

the

measures

We

ran

pairs

of

samples

each

consisting

of

independent

formulas

-LRB-

runs

in

all

-RRB-

In

every

case

with

the

dsj

branching

rule

was

run

with

a

random

input

from

the

same

family

of

formulas

-LRB-

variables

clauses

-RRB-

This

yielded

independent

S

values

whose

cumulative

distribution

is

plotted

in

Figure

The

slightly

concave

appearance

suggests

that

the

distribution

is

shifted

a

little

to

the

high

end

making

it

conservative

Because

our

program

is

designed

to

reduce

branching

we

suspect

the

skewness

is

lower

than

might

be

exhibited

in

other

programs

As

mentioned

exponential

time

algorithms

can

stress

the

assumptions

on

underlying

distributions

beyond

the

limits

foreseen

by

traditional

statisticians

Trap

Fail

to

distinguish

between

random

variables

with

no

known

upper

bounds

or

known

variances

and

those

for

which

these

quantities

are

estimable

and

of

reasonable

size

Informally

the

problem

is

this

to

know

how

accurate

an

estimate

of

the

mean

is

one

must

know

the

standard

deviation

But

again

only

an

estimate

of

the

standard

deviation

is

available

so

how

does

one

know

how

accurate

that

is

Standard

statistical

methods

handle

this

question

by

assuming

that

the

underlying

distribution

is

Gaussian

This

is

not

realistic

in

this

setting

The

only

constraint

on

the

distribution

that

we

can

expect

in

practice

is

a

firm

upper

bound

Some

examples

having

firm

upper

bounds

include

ffl

Any

probability

is

bounded

between

and

and

has

standard

deviation

at

most

ffl

The

size

of

the

optimum

for

most

optimization-based

NP-complete

problems

is

easily

bounded

by

a

low

degree

polynomial

On

the

other

hand

the

CPU

time

required

to

solve

a

random

problem

in

a

given

family

normally

has

no

known

upper

limit

or

known

variance

As

mentioned

without

knowing

something

about

the

distribution

one

can

not

say

how

accurate

is

an

estimate

of

the

variance

calculated

from

a

sample

We

strongly

advocate

measuring

something

in

addition

to

CPU

time

for

performance

evaluation

The

alternate

measure

should

preferably

be

strongly

correlated

to

CPU

time

but

it

should

be

a

measure

for

which

an

upper

bound

is

known

for

the

whole

sample

space

a

priori

For

example

in

simple

DPLL

is

an

easy

upper

bound

on

the

number

of

branches

Closer

analysis

can

sharpen

that

Therefore

some

firm

statistical

statements

can

be

made

about

the

accuracy

of

an

estimate

of

the

average

number

of

branches

required

over

some

sample

space

Trap

Use

the

Student

t-test

when

the

pairs

test

is

applicable

To

compare

two

algorithms

performance

it

is

common

sense

to

eliminate

as

much

noise

as

possible

by

running

them

on

the

same

set

of

problems

But

it

is

a

common

mistake

to

try

to

work

with

averages

and

standard

deviations

of

the

two

samples

rather

than

all

the

data

The

pairs

test

compares

two

algorithms

input

by

input

and

computes

the

difference

in

the

performance

measure

for

each

input

Thus

the

noise

due

to

varying

difficulty

of

inputs

is

eliminated

The

standard

deviation

of

the

difference

may

well

be

an

order

of

magnitude

lower

than

the

standard

deviations

of

the

samples

themselves

and

much

higher

discrimination

results

clauses

samples

maxscore

minlen

maxlen

Figure

Student

's

t-test

significance

probabilities

of

the

mean

difference

from

the

dsj

branching

strategy

on

random

formulas

Only

those

entries

that

are

not

significant

at

level

are

shown

Those

marked

by

showed

significance

in

Figure

based

on

the

paired

t-test

for

same

data

The

phenomenon

is

illustrated

with

the

data

for

comparison

of

dsj

with

five

alternative

strategies

from

Section

Figure

shows

the

insignificant

entries

-LRB-

at

level

-RRB-

that

would

result

from

using

the

nonpaired

t-test

Of

these

those

that

were

significant

in

Figure

are

marked

with

a

For

this

data

and

significance

level

the

paired

t-test

finds

out

of

additional

significant

differences

B

Second

DIMACS

Challenge

Satisfiability

Benchmark

Results

GENERAL

INFORMATION

Authors

Allen

Van

Gelder

and

Yumi

K.

Tsuji

-LRB-

University

of

California

Santa

Cruz

-RRB-

Title

Satisfiability

Testing

with

More

Reasoning

and

Less

Guessing

Name

of

Algorithm

Brief

Description

of

Algorithm

Complete

Combination

of

branching

and

limited

resolution

Type

of

Machine

Compiler

and

flags

used

gcc

-LRB-

version

-RRB-

MACHINE

BENCHMARKS

User

time

for

instances

ALGORITHM

BENCHMARKS

Authors

Comments

ffl

Parameters

For

this

Appendix

the

following

invocation

parameter

values

were

used

-LCB-

Branching

Strategy

The

program

can

use

any

of

the

six

different

branching

strategies

we

selected

the

one

based

on

Johnson

's

maximum

satisfiability

approximation

algorithm

-LCB-

Randomization

The

program

shuffles

the

order

of

the

variables

there

is

an

invocation

parameter

to

set

a

seed

value

for

the

random

number

generator

used

by

this

shuffling

procedure

The

multiple

runs

presented

in

the

Appendix

were

produced

by

varying

this

initial

seed

ffl

Additional

columns

in

the

Table

We

have

two

additional

columns

in

the

Appendix

table

The

column

named

RelStdErr

refers

to

the

relative

standard

error

of

the

mean

It

is

calculated

as

the

percentage

of

the

standard

error

of

the

mean

with

respect

to

the

mean

-LRB-

Std

Dev

-RRB-

-LRB-

Mean?pn

-RRB-

The

last

column

Median

refers

to

the

int

-LRB-

-LRB-

n

-RRB-

-RRB-

th

value

in

the

ordered

list

of

the

cpu

values

of

the

runs

in

each

sample

ffl

Failed

Runs

The

failed

runs

given

in

the

table

had

the

maximum

time

limit

set

to

hours

For

the

and

formulas

we

have

also

tried

different

seeds

to

do

the

variable

scrambling

with

the

time

limit

set

to

minutes

for

each

they

had

all

failed

ffl

Note

on

the

Special

sequences

We

found

that

moving

the

first

clauses

to

the

end

of

the

formula

permitted

the

program

to

succeed

in

seconds

with

guesses

on

and

These

runs

are

not

included

in

the

table

Results

on

Benchmark

Instances

Time

Name

Runs

-LRB-

Fail

-RRB-

Min

Avg

-LRB-

StdDev

-RRB-

Max

Result

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

Yes

DNR

DNR

DNR

DNR

-LRB-

-RRB-

-LRB-

-RRB-

DNR

DNR

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

No

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

-LRB-

-RRB-

Yes

Benchmark

-LRB-

cont

-RRB-

Avg

Rel.Std

Median

Name

Time

Err

-LRB-

-RRB-

Time

propositional formula
resolution
satisfiability

2-satisfiability
boolean formula
k-closure