ON

LEARNING

HOW

TO

LEARN

LEARNING

STRATEGIES

Technical

Report

-LRB-

revised

-RRB-

Jurgen

Schmidhuber

Fakultat

fur

Informatik

Technische

Universitat

Munchen

Munchen

Germany

schmidhu@informatik.tu-muenchen.de

Revised

January

Abstract

This

paper

introduces

the

incremental

self-improvement

paradigm

Unlike

previous

methods

incremental

self-improvement

encourages

a

reinforcement

learning

system

to

improve

the

way

it

learns

and

to

improve

the

way

it

improves

the

way

it

learns

without

significant

theoretical

limitations

the

system

is

able

to

shift

its

inductive

bias

in

a

universal

way

Its

major

features

are

-LRB-

-RRB-

There

is

no

explicit

difference

between

learning

meta-learning

and

other

kinds

of

information

processing

Using

a

Turing

machine

equivalent

programming

language

the

system

itself

occasionally

executes

self-delimiting

initially

highly

random

self-modification

programs

which

modify

the

context-dependent

probabilities

of

future

action

sequences

-LRB-

including

future

self-modification

programs

-RRB-

-LRB-

-RRB-

The

system

keeps

only

those

probability

modifications

computed

by

useful

selfmodification

programs

those

which

bring

about

more

payoff

-LRB-

reward

reinforcement

-RRB-

per

time

than

all

previous

self-modification

programs

-LRB-

-RRB-

The

computation

of

payoff

per

time

takes

into

account

all

the

computation

time

required

for

learning

the

entire

system

life

is

considered

boundaries

between

learning

trials

are

ignored

-LRB-

if

there

are

any

-RRB-

A

particular

implementation

based

on

the

novel

paradigm

is

presented

It

is

designed

to

exploit

what

conventional

digital

machines

are

good

at

fast

storage

addressing

arithmetic

operations

etc.

Experiments

illustrate

the

system

's

mode

of

operation

Note

This

is

the

revised

and

extended

version

of

an

earlier

report

from

November

Introduction

A

recent

debate

in

the

machine-learning

community

highlighted

a

fact

that

appears

discouraging

at

first

glance

in

general

generalization

can

not

be

expected

inductive

inference

is

impossible

and

nothing

can

be

learned

See

e.g.

-LRB-

Dietterich

Schaffer

Wolpert

Schmidhuber

-RRB-

Paraphrasing

from

a

previous

argument

-LRB-

Schmidhuber

-RRB-

let

the

task

be

to

learn

some

relation

between

finite

bitstrings

and

finite

bitstrings

Somehow

a

training

set

is

chosen

In

almost

all

cases

the

shortest

algorithm

computing

a

-LRB-

non-overlapping

-RRB-

test

set

essentially

has

the

same

size

as

the

whole

test

set

This

is

because

most

computable

objects

are

irregular

and

incompressible

-LRB-

Kolmogorov

Chaitin

-RRB-

The

shortest

algorithm

computing

the

test

set

given

the

training

set

is

n't

any

shorter

In

other

words

the

relative

algorithmic

complexity

of

the

test

set

given

the

training

set

is

maximal

and

the

mutual

algorithmic

information

between

test

set

and

training

set

is

zero

-LRB-

ignoring

an

additive

constant

independent

of

the

problem

see

e.g.

Kolmogorov

Chaitin

Solomonoff

Li

and

Vit?anyi

-RRB-

Therefore

in

almost

all

cases

-LRB-

-RRB-

knowledge

of

the

training

set

does

not

provide

any

clues

about

the

test

set

-LRB-

-RRB-

there

is

no

hope

for

generalization

and

-LRB-

-RRB-

inductive

inference

does

not

make

any

sense

Atypical

real

world

Previous

learning

algorithms

Apparently

however

generalization

and

inductive

inference

do

make

sense

in

the

real

world

One

reason

for

this

may

be

that

the

real

world

is

run

by

a

short

algorithm

See

-LRB-

Schmidhuber

-RRB-

Anyway

problems

that

humans

consider

to

be

typical

are

atypical

when

compared

to

the

general

set

of

all

well-defined

problems

Otherwise

things

like

learning

by

analogy

learning

by

chunking

incremental

learning

continual

learning

learning

from

invariances

learning

by

knowledge

transfer

etc.

would

not

be

possible

and

experience

with

previous

problems

could

not

sensibly

adjust

the

prior

distribution

of

solution

candidates

in

the

search

space

for

a

new

problem

-LRB-

shift

of

inductive

bias

e.g.

Utgoff

-RRB-

In

fact

all

previous

learning

systems

are

implicitly

or

explicitly

designed

to

exploit

task-specific

regularities

of

some

kind

or

another

No

previous

learning

system

however

is

designed

to

make

optimal

use

of

its

computational

time/space

resources

by

exploiting

arbitrary

task-specific

regularities

-LRB-

if

there

are

any

-RRB-

Such

a

system

would

have

to

be

able

-LRB-

-RRB-

to

develop

arbitrary

problem-specific

representations

-LRB-

-RRB-

to

run

arbitrary

learning

algorithms

and

-LRB-

-RRB-

to

find

the

good

problem-specific

learning

algorithms

as

quickly

as

possible

In

particular

it

would

have

to

be

able

to

find

algorithms

for

finding

learning

algorithms

etc.

Self-improvement

Is

it

possible

to

build

such

a

system

A

system

that

can

tailor

its

learning

behavior

to

the

requirements

of

a

given

environment

with

arbitrary

initially

unknown

problem-specific

regularities

A

system

that

can

learn

to

improve

its

own

learning

strategy

in

a

universal

way

without

any

significant

theoretical

limitations

other

than

those

imposed

by

the

finiteness

of

the

hardware

In

principle

the

answer

is

yes

The

system

described

in

this

paper

uses

the

novel

incremental

selfimprovement

paradigm

to

exploit

benign

environments

in

a

more

general

way

than

previous

systems

Some

of

its

properties

are

-LRB-

-RRB-

Unlike

e.g.

hillclimbing/evolutionary/genetic

other

algorithms

it

potentially

can

evolve

its

own

smart

search

strategies

-LRB-

as

opposed

to

dumb

non-adaptive

strategies

like

the

ones

embodied

by

random

mutation

crossover

etc.

-RRB-

-LRB-

-RRB-

Unlike

with

previous

less

realistic

approaches

each

event

in

system

life

is

viewed

as

a

singular

event

learning

is

inductive

inference

from

non-repeatable

experiences

-LRB-

-RRB-

Unlike

with

previous

approaches

the

system

's

objective

function

takes

into

account

the

computation

time

required

for

learning

Outline

Section

lists

essential

ingredients

of

the

incremental

self-improvement

paradigm

Section

exemplifies

the

basic

principles

by

describing

and

justifying

a

concrete

-LRB-

working

-RRB-

implementation

Twenty

comments

on

both

general

and

implementation-specific

properties

of

incremental

selfimprovement

can

be

found

in

section

Illustrative

applications

to

toy

problems

-LRB-

including

a

simple

non-Markovian

maze

task

-RRB-

will

follow

in

section

Section

will

then

briefly

describe

the

history

of

related

ideas

The

Incremental

Self-Improvement

Paradigm

Incremental

self-improvement

is

a

machine-learning

paradigm

designed

for

a

system

executing

a

lifelong

sequence

of

actions

in

an

arbitrary

environment

The

system

's

goal

is

to

maximize

cumulative

payoff

-LRB-

reinforcement

reward

-RRB-

to

be

obtained

throughout

its

entire

span

of

life

To

achieve

its

goal

the

system

continually

attempts

to

create

action

subsequences

leading

to

faster

and

faster

payoff

intake

The

central

ideas

are

as

follows

Computing

self-modifications

The

initially

highly

random

actions

of

the

system

actually

are

primitive

instructions

of

a

Turing

machine

equivalent

programming

language

which

allows

for

implementing

arbitrary

learning

algorithms

Action

subsequences

represent

either

-LRB-

-RRB-

normal

interactions

with

the

environment

or

-LRB-

-RRB-

self-modification

programs

Self-modification

programs

can

modify

the

probabilities

of

future

action

subsequences

including

future

self-modification

programs

the

learning

system

is

able

to

modify

itself

in

a

universal

way

There

is

no

explicit

difference

between

learning

meta-learning

and

other

kinds

of

information

processing

Life

is

one-way

Each

action

of

the

learning

system

-LRB-

including

probability

modifying

actions

executed

by

self-modification

programs

-RRB-

is

viewed

as

a

singular

event

in

the

history

of

system

life

Unrealistic

concepts

such

as

exactly

repeatable

training

iterations

boundaries

between

trials

epochs

etc.

are

thrown

overboard

In

general

the

environment

can

not

be

reset

Life

is

one-way

Learning

is

inductive

inference

from

non-repeatable

experiences

Evaluations

of

self-modification

programs

The

system

has

a

time-varying

utility

value

which

is

the

average

payoff

per

time

since

system

start-up

Each

self-modification

program

also

has

a

time-varying

utility

value

This

value

is

the

average

amount

of

payoff

per

time

measured

since

the

program

began

execution

Evaluations

of

utility

take

into

account

all

the

computation

time

required

for

learning

including

the

time

required

for

evaluating

utility

Useful

self-modification

programs

accelerate

payoff

intake

The

system

keeps

track

of

probability

modifications

computed

by

self-modification

programs

that

it

considers

useful

Usefulness

is

defined

recursively

If

there

are

no

previous

useful

self-modification

programs

-LRB-

e.g.

at

system

start-up

-RRB-

a

new

self-modification

program

is

considered

useful

only

for

as

long

as

its

utility

value

exceeds

the

system

's

utility

value

More

recent

self-modification

programs

are

considered

useful

for

as

long

as

they

have

higher

utility

values

than

all

preceding

self-modification

programs

currently

considered

useful

Essentially

the

system

only

keeps

modifications

to

its

probability

values

that

originated

from

useful

self-modification

programs

The

result

is

that

payoff

intake

is

constantly

accelerated

Over

time

the

system

tends

to

make

better

and

better

use

of

its

computational

resources

A

Concrete

Working

Implementation

This

section

presents

one

of

many

possible

implementations

of

the

incremental

self-improvement

paradigm

The

implementation

makes

use

of

an

integer-based

programming

language

The

language

is

assemblerlike

and

has

primitive

instructions

designed

to

exploit

what

conventional

digital

machines

are

good

at

fast

storage

addressing

jumping

basic

arithmetic

operations

etc.

The

language

is

universal

-LRB-

i.e.

Turing

machine

equivalent

-RRB-

It

is

related

to

one

previously

published

-LRB-

Schmidhuber

-RRB-

but

there

are

significant

differences

and

extensions

In

particular

this

language

is

self-referential

in

a

manner

that

will

be

described

below

this

paper

when

referring

to

arbitrary

modifications

functions

etc.

there

is

only

one

essential

requirement

of

these

modifications

or

functions

they

must

be

computable

Overview

The

system

has

a

finite

amount

of

addressable

storage

broken

into

two

groups

work

cells

and

program

cells

The

system

exists

in

an

environment

that

occasionally

writes

inputs

into

certain

work

cells

No

assumptions

are

made

about

the

environment

-LRB-

it

may

or

may

not

be

non-deterministic

for

example

-RRB-

There

is

a

fixed

set

I

of

integer

instruction

values

and

integer

argument

values

There

is

also

an

internal

variable

-LRB-

not

stored

in

an

addressable

cell

-RRB-

called

the

Instruction

Pointer

-LRB-

IP

-RRB-

which

always

points

to

one

of

the

program

cells

-LRB-

initially

to

the

first

one

-RRB-

For

each

program

cell

and

for

every

possible

instruction

and

instruction

argument

there

is

a

probability

value

Pij

where

i

is

the

index

of

a

program

cell

and

j

I.

Pij

specifies

for

cell

i

the

conditional

probability

that

when

pointed

to

by

IP

its

contents

will

be

j

If

IP

i

and

the

contents

of

i

is

interpreted

as

an

instruction

then

the

cells

that

immediately

follow

i

will

be

interpreted

as

the

instruction

's

arguments

If

the

instruction

and

its

arguments

pass

a

syntax

check

the

instruction

is

executed

This

may

result

in

modifications

of

IP

and/or

environment

and/or

storage

If

the

test

is

not

passed

the

IP

is

reset

to

the

first

program

cell

If

the

instruction

does

not

itself

modify

IP

-LRB-

with

a

jump

instruction

-RRB-

then

IP

is

incremented

to

point

to

the

following

cell

This

instruction

cycle

is

repeated

over

and

over

again

and

represents

the

basic

operation

of

the

system

Some

instructions

are

special

self-referential

instructions

Two

of

them

can

address

and

modify

arbitrary

Pij

values

-LRB-

and

they

are

the

only

instructions

that

can

do

this

-RRB-

There

is

also

another

selfreferential

instruction

that

groups

a

sequence

of

probability-modifying

instructions

and

other

instructions

into

a

self-delimiting

self-modification

program

so

named

because

the

end

of

such

a

program

is

computed

by

itself

Some

of

the

-LRB-

initially

highly

random

-RRB-

action

subsequences

executed

during

system

life

will

indeed

be

self-delimiting

self-modification

programs

They

can

compute

arbitrary

sequences

of

modifications

of

Pij

values

resulting

in

arbitrary

modifications

of

context-dependent

probabilities

of

future

action

subsequences

including

future

self-modification

programs

Occasionally

the

environment

may

provide

payoff

a

real

number

indicating

how

well

the

system

has

done

The

system

's

goal

is

to

maximize

the

sum

of

all

payoffs

to

be

obtained

throughout

its

entire

-LRB-

initially

unknown

-RRB-

life

span

This

is

done

as

follows

The

payoff/time

ratio

is

constantly

monitored

and

updated

after

every

instruction

There

is

an

unmodifiable

top-level

credit

assignment

strategy

It

also

executes

after

each

instruction

except

when

a

self-modification

program

is

running

The

task

of

the

top-level

is

to

countermand

the

modifications

to

the

Pij

values

that

were

made

by

self-modification

programs

no

longer

considered

useful

-LRB-

see

recursive

description

in

section

-RRB-

This

countermanding

is

done

by

restoring

old

probability

distributions

saved

onto

a

stack

before

the

corresponding

self-modification

programs

modified

them

The

computation

time

required

for

top-level

management

is

taken

into

account

when

measuring

usefulness

It

will

be

seen

that

this

scheme

favors

sequences

of

useful

self-modification

programs

leading

to

faster

and

faster

payoff

intake

In

particular

self-modification

programs

can

prove

their

long

term

usefulness

by

setting

the

stage

for

additional

useful

self-modification

programs

which

potentially

include

programs

executing

known

-LRB-

and

not

yet

known

-RRB-

learning

algorithms

This

encourages

learning

how

to

learn

Technical

Details

Span

of

system

life

For

simplicity

we

assume

discrete

time

System

life

begins

at

birth

time

step

zero

It

ends

at

death

time

step

T

T

is

not

necessarily

known

in

advance

Goal

Payoff

Occasionally

the

environment

provides

payoff

Payoff

is

an

integer

number

depending

on

the

tasks

to

be

solved

The

sum

of

all

payoffs

obtained

between

birth

and

time

t

is

denoted

by

R

-LRB-

t

-RRB-

Throughout

its

lifetime

the

system

's

goal

is

to

maximize

R

-LRB-

T

-RRB-

the

cumulative

payoff

at

death

At

a

given

time

the

system

can

only

maximize

future

payoff

the

past

is

already

gone

Storage

The

system

's

storage

is

a

single

array

of

cells

Each

cell

has

an

integer

address

in

the

interval

-LSB-

Min

Max

-RSB-

Max

is

a

positive

integer

Min

is

a

negative

integer

ai

denotes

the

cell

with

address

i

The

variable

contents

of

ai

are

denoted

by

ci

-LSB-

Maxint

Maxint

-RSB-

and

are

of

type

integer

as

well

-LRB-

Maxint

Max

Maxint

abs

-LRB-

Min

-RRB-

-RRB-

Special

addresses

InputStart

InputEnd

RegisterStart

and

Primitive

Semantics

Stop

-LRB-

-RRB-

Halt

current

run

Jmp

-LRB-

-RRB-

IP

Jmpleq

-LRB-

-RRB-

If

IP

Jmpeq

-LRB-

-RRB-

If

IP

Add

-LRB-

-RRB-

Sub

-LRB-

-RRB-

Mul

-LRB-

-RRB-

Div

-LRB-

-RRB-

-LRB-

integer

division

-RRB-

Rem

-LRB-

-RRB-

remainder

-LRB-

-RRB-

Inc

-LRB-

-RRB-

Dec

-LRB-

-RRB-

Mov

-LRB-

-RRB-

Init

-LRB-

-RRB-

Outputi

-LRB-

-RRB-

i-th

problem

specific

primitive

for

influencing

the

environment

Inputi

-LRB-

-RRB-

i-th

primitive

for

perceiving

environmental

input

GetP

-LRB-

-RRB-

round

-LRB-

Maxint

-RRB-

IncP

-LRB-

-RRB-

k

k

-LRB-

-RRB-

DecP

-LRB-

-RRB-

k

k

EndSelfMod

-LRB-

-RRB-

Enable

top

level

strategy

for

restoring

probabilities

-LRB-

see

text

-RRB-

Table

Semantics

of

primitives

and

their

parameters

The

normal

primitives

are

shown

in

the

top

block

the

self-referential

primitives

are

shown

in

the

bottom

block

Note

the

extensive

use

of

doubleindexed

indirect

addressing

Results

of

arithmetic

operations

leading

to

underflow/overflow

are

replaced

by

Maxint/Maxint

respectively

The

same

holds

for

positive

and

negative

divisions

by

zero

DecP

and

IncP

have

no

effect

if

the

indirectly

addressed

cell

contents

are

not

an

integer

between

and

or

if

the

corresponding

probability

modification

would

lead

to

at

least

one

P

value

below

MinP

Rules

for

syntactic

correctness

IP

may

point

to

any

program

cell

ai

i

Max

-LRB-

enough

space

has

to

be

left

for

arguments

-RRB-

Operations

that

read

cell

contents

-LRB-

such

as

Add

Move

Jumpleq

etc.

-RRB-

may

read

only

from

existing

addresses

in

storage

Operations

that

write

cell

contents

-LRB-

such

as

Add

Move

GetP

etc.

-RRB-

may

write

only

into

work

area

addresses

in

-LSB-

Min

P

rogramStart

-RSB-

P

rogramStart

are

used

to

further

divide

storage

into

segments

Min

InputStart

InputEnd

RegisterStart

ProgramStart

Max

The

input

area

is

the

set

of

input

cells

fai

InputStart

i

InputEndg

The

register

area

is

the

set

of

register

cells

fai

i

P

rogramStartg

Registers

are

convenient

for

indirect-addressing

purposes

The

program

area

is

the

set

of

program

cells

fai

P

rogramStart

i

Maxg

Integer

sequences

in

the

program

area

are

interpreted

as

executable

code

The

work

area

is

the

set

of

work

cells

fai

Min

i

P

rogramStartg

Instructions

executed

in

the

program

area

may

read

from

and

write

to

the

work

area

Both

register

area

and

input

area

are

subsets

of

the

work

area

Environmental

inputs

At

every

time

step

new

inputs

from

the

environment

may

be

written

into

the

input

cells

Primitives

The

number

of

instructions

is

nops

-LRB-

nops

Maxint

-RRB-

Each

such

primitive

is

represented

by

a

unique

number

in

the

set

nops

-LRB-

due

to

the

code

being

written

in

C

-RRB-

The

primitive

with

number

j

is

denoted

by

pj

Primitives

may

have

from

zero

to

three

arguments

each

of

which

has

a

value

in

nops

The

semantics

of

the

primitives

and

their

corresponding

arguments

are

given

in

Table

The

non-self-referential

-LRB-

normal

-RRB-

primitives

include

actions

for

comparisons

and

conditional

jumps

for

copying

storage

contents

for

initializing

certain

storage

cells

with

small

integers

and

for

adding

multiplying

dividing

and

halting

They

also

include

output

actions

for

modifying

the

environment

and

input

actions

for

perceiving

environmental

states

The

self-referential

primitives

will

be

described

in

detail

below

Primitive

and

argument

probabilities

For

each

cell

ai

in

the

program

area

there

is

a

discrete

probability

distribution

Pi

over

the

set

of

possible

cell

contents

The

variable

InstructionPointer

-LRB-

IP

-RRB-

always

points

to

one

of

the

program

cells

If

IP

i

and

i

Max

then

Pij

denotes

the

probability

of

selecting

primitive

pj

as

the

next

instruction

The

restriction

i

Max

is

needed

to

leave

room

for

the

instruction

's

possible

arguments

should

it

require

any

Once

pj

is

selected

ci

j

If

pj

has

a

first

argument

then

Pi

k

is

the

probability

of

k

being

chosen

as

its

actual

value

for

k

nops

Once

some

k

is

selected

ci

k.

Analoguously

if

pj

has

a

second

argument

then

Pi

l

is

the

probability

of

l

being

chosen

as

its

actual

value

for

l

nops

Once

some

l

is

selected

ci

l.

And

finally

if

pj

has

a

third

argument

then

Pi

m

is

the

probability

of

m

being

chosen

as

its

actual

value

for

m

nops

Once

some

m

is

selected

ci

m.

Arguments

point

to

storage

addresses

To

reduce

the

number

of

probability

values

for

each

program

cell

primitive

arguments

are

restricted

to

only

nops

different

values

Therefore

to

allow

all

storage

cells

to

be

addressed

double

indexed

indirect

addressing

is

used

for

most

instructions

That

is

-LRB-

for

most

instructions

-RRB-

the

arguments

point

to

cells

in

the

register

area

which

in

turn

point

to

cells

in

storage

Recall

that

the

range

of

values

available

to

registers

-LRB-

and

other

work

cells

-RRB-

is

far

less

restricted

which

effectively

allows

all

storage

cells

to

be

addressed

Self-referential

primitives

Two

special

primitives

DecP

and

IncP

may

be

used

to

address

and

modify

the

current

probability

distribution

of

any

program

cell

-LRB-

see

Table

-RRB-

With

the

action

DecP

the

Pij

value

for

a

particular

cell/value

pair

-LRB-

ai

j

-RRB-

can

be

decreased

by

some

factor

in

g

The

probabilities

for

that

cell

are

then

normalized

Likewise

with

the

action

IncP

the

complement

-LRB-

Pij

-RRB-

of

the

Pij

value

for

a

particular

cell

ai

and

value

j

can

be

decreased

by

a

factor

in

g

-LRB-

and

the

cell

probabilities

are

again

renormalized

-RRB-

DecP

and

IncP

have

no

effect

if

the

indirectly

addressed

cell

contents

-LRB-

see

Table

-RRB-

are

not

an

integer

between

and

or

if

the

corresponding

probability

modification

would

lead

to

at

least

one

P

value

below

MinP

-LRB-

a

small

positive

constant

-RRB-

The

primitive

GetP

can

be

used

to

write

scaled

versions

of

current

probability

values

into

work

cells

GetP

is

potentially

useful

for

purposes

of

introspection

Instruction

cycle

A

single

step

of

the

interpreter

works

as

follows

if

IP

points

to

program

cell

ai

a

primitive

and

the

corresponding

arguments

are

chosen

randomly

according

to

the

current

probability

distributions

as

already

described

They

are

sequentially

written

onto

the

program

area

starting

from

ai

Syntax

checks

are

performed

Rules

for

syntactic

correctness

are

given

in

the

caption

of

Table

If

syntactically

correct

the

instruction

gets

executed

Otherwise

the

current

run

-LRB-

see

next

paragraph

-RRB-

is

halted

If

the

program

did

not

halt

nor

change

the

value

of

IP

-LRB-

e.g.

by

causing

a

jump

-RRB-

IP

is

set

to

the

address

of

the

cell

following

the

last

argument

of

the

current

instruction

Runs

In

the

beginning

of

a

run

IP

is

set

equal

to

P

rogramStart

and

the

instruction

cycle

is

repeated

until

a

halt

situation

-LRB-

e.g.

syntax

error

-RRB-

is

encountered

Due

to

MinP

being

positive

there

is

always

a

non-vanishing

halting

probability

System

life

At

time

step

storage

is

initialized

with

zeros

The

probability

distributions

of

all

program

cells

are

initialized

with

maximum

entropy

distributions

-LRB-

Shannon

-RRB-

That

is

all

Pij

values

are

initialized

to

the

same

value

so

that

there

is

no

bias

for

a

particular

value

in

any

cell

After

initialization

runs

are

repeated

over

and

over

again

until

time

T

Recall

that

T

does

not

have

to

be

known

in

advance

Work

area

as

part

of

the

environment

Neither

storage

nor

environment

are

re-initialized

after

each

run

The

system

might

use

the

environment

to

store

representations

of

previous

events

by

executing

actions

that

modify

the

Likewise

the

program

area

may

use

the

work

area

to

store

representations

of

previous

events

Thus

the

work

area

may

be

viewed

as

part

of

the

total

environment

of

the

program

area

Kaelbling

sometimes

refers

to

this

as

writing

on

the

walls

but

says

that

the

real

name

is

stigmergy

-LRB-

personal

communication

-RRB-

Programs

Each

subsequence

of

primitives

executed

during

system

life

is

called

a

program

Primitives

can

be

combined

to

form

programs

for

performing

arbitrary

computations

The

only

limitations

are

those

imposed

by

the

necessarily

finite

hardware

Self-delimiting

self-modification

programs

Occasionally

the

system

will

modify

one

of

its

probability

distributions

by

using

IncP

or

DecP

Occasionally

it

will

execute

the

EndSelfMod

primitive

The

first

probability

modification

after

an

EndSelfMod

action

or

after

system

birth

begins

a

selfmodification

program

The

self-modification

program

ends

itself

by

executing

the

EndSelfMod

action

Due

to

the

universality

of

the

underlying

programming

language

self-modification

programs

may

result

in

specific

arbitrary

modifications

of

context-dependent

probabilities

of

future

programs

However

due

to

MinP

being

positive

the

probability

of

selecting

and

executing

a

particular

instruction

at

a

particular

time

can

not

entirely

vanish

The

remainder

of

this

section

is

devoted

to

basic

concepts

required

to

ensure

that

the

system

keeps

only

probability

modifications

computed

by

useful

self-modification

programs

essentially

those

which

bring

about

more

payoff

per

time

than

all

previous

self-modification

programs

Payoff/time

ratios

Suppose

a

self-modification

program

s

started

execution

at

time

and

completed

itself

at

time

For

t

and

t

T

the

payoff/time

ratio

Q

-LRB-

s

t

-RRB-

is

defined

as

Q

-LRB-

s

t

-RRB-

R

-LRB-

t

-RRB-

R

-LRB-

-RRB-

t

DEFINITION

useful

self-modification

programs

The

usefulness

of

a

self-modification

program

is

defined

recursively

at

birth

there

are

no

useful

self-modification

programs

At

some

later

point

t

in

system

life

we

consider

two

cases

a

self-modification

program

s

that

ended

itself

at

time

is

considered

useful

if

-LRB-

-RRB-

-LRB-

a

-RRB-

there

are

no

previous

self-modification

programs

that

are

considered

useful

and

-LRB-

b

-RRB-

for

all

tx

tx

t

Q

-LRB-

s

tx

-RRB-

R

-LRB-

tx

-RRB-

tx

-LRB-

the

total

payoff/time

ratio

at

time

tx

-RRB-

Or

-LRB-

-RRB-

-LRB-

a

-RRB-

there

are

previous

self-modification

programs

considered

useful

and

-LRB-

b

-RRB-

for

all

tx

tx

t

Q

-LRB-

s

tx

-RRB-

Q

-LRB-

tx

-RRB-

where

is

the

most

recent

useful

self-modification

program

preceding

s

The

computation

of

payoff/time

ratios

always

takes

into

account

all

computation

time

including

time

required

for

learning

FACT

At

a

given

time

to

decide

whether

the

most

recent

useful

self-modification

program

remains

useful

one

needs

to

compare

its

current

payoff/time

ratio

only

to

the

current

payoff/time

ratio

of

the

most

recent

previous

useful

self-modification

program

FACT

A

completed

self-modification

program

is

considered

useful

as

long

as

average

payoff

intake

since

its

beginning

occurred

faster

than

with

all

previous

self-modification

programs

still

considered

useful

Proof

See

definition

of

usefulness

above

Keeping

track

of

useful

self-modifications

In

order

to

evaluate

whether

life

after

the

beginning

of

some

self-modification

program

was

better

or

worse

than

before

and

to

be

able

to

countermand

probability

modifications

computed

by

self-modification

programs

that

seem

useless

or

even

harmful

a

stack

S

is

introduced

The

k-th

entry

of

S

k

StackSizeg

is

denoted

S

-LRB-

k

-RRB-

S

-LRB-

k

-RRB-

consists

of

the

following

variables

S

-LRB-

k

-RRB-

t

S

-LRB-

k

-RRB-

R

S

-LRB-

k

-RRB-

address

S

-LRB-

k

-RRB-

first

and

S

-LRB-

k

-RRB-

oldP

-LRB-

a

vector

of

nops

variables

-RRB-

The

variable

sp

points

to

the

current

topmost

stack

entry

If

sp

-LRB-

the

value

at

system

startup

-RRB-

the

stack

is

empty

If

the

system

modifies

some

probability

distribution

Pi

at

time

t

-LRB-

by

using

IncP

or

DecP

-RRB-

sp

is

incremented

and

the

following

values

are

pushed

onto

the

stack

the

current

time

the

total

payoff

so

far

the

address

of

the

modified

program

cell

its

old

probability

distribution

right

before

the

current

modification

-LRB-

represented

by

nops

real

values

-RRB-

and

a

pointer

to

the

stack

entry

corresponding

to

the

first

probability

modification

computed

by

the

current

self-modification

program

More

formally

S

-LRB-

sp

-RRB-

t

t

S

-LRB-

sp

-RRB-

R

R

-LRB-

t

-RRB-

S

-LRB-

sp

-RRB-

address

i

S

-LRB-

sp

-RRB-

oldP

Pi

before

the

modification

-LRB-

represented

by

nops

real

values

-RRB-

If

t

marks

the

beginning

of

a

self-modification

program

then

S

-LRB-

sp

-RRB-

first

sp

Otherwise

-LRB-

in

the

middle

of

a

self-modification

program

-RRB-

S

-LRB-

sp

-RRB-

first

S

-LRB-

sp

-RRB-

first

The

zeroth

stack

entry

which

can

not

be

popped

is

initialized

as

follows

S

-LRB-

-RRB-

t

S

-LRB-

-RRB-

R

S

-LRB-

-RRB-

first

The

remaining

values

are

undefined

UNMODIFIABLE

TOP-LEVEL

CREDIT

ASSIGNMENT

STRATEGY

inductive

inference

based

on

single

experiences

It

is

very

simple

If

there

is

no

self-modification

program

running

then

after

each

completed

instruction

cycle

a

decision

is

made

as

to

whether

probability

distributions

modified

by

previous

self-modification

programs

should

be

kept

or

replaced

by

their

old

values

from

the

stack

The

process

works

as

follows

After

each

completed

instruction

cycle

do

-LRB-

-RRB-

If

there

currently

is

an

incomplete

self-modification

program

-LRB-

not

yet

ended

by

an

EndSelfMod

action

-RRB-

do

nothing

-LRB-

-RRB-

Else

successively

pop

and

restore

saved

probability

distributions

until

the

payoff/time

ratio

since

the

beginning

of

the

most

recent

self-modification

program

that

computed

modifications

of

probability

distributions

currently

saved

on

the

stack

exceeds

the

payoff/time

ratio

of

the

preceding

such

self-modification

program

-LRB-

if

there

is

any

-LCB-

otherwise

compare

with

the

total

payoff/time

ratio

-RRB-

More

formally

-LRB-

t

denotes

the

current

time

-RRB-

While

sp

and

R

-LRB-

t

-RRB-

S

-LRB-

S

-LRB-

sp

-RRB-

first

-RRB-

R

t

S

-LRB-

S

-LRB-

sp

-RRB-

first

-RRB-

t

R

-LRB-

t

-RRB-

S

-LRB-

S

-LRB-

S

-LRB-

sp

-RRB-

first

-RRB-

first

-RRB-

R

t

S

-LRB-

S

-LRB-

S

-LRB-

sp

-RRB-

first

-RRB-

first

-RRB-

t

do

PS

-LRB-

sp

-RRB-

i

S

-LRB-

sp

-RRB-

oldP

sp

sp

FACT

above

says

that

at

a

given

time

the

top-level

strategy

needs

to

consider

only

the

two

most

recent

self-modification

programs

whose

direct

effects

have

not

yet

been

countermanded

in

order

to

decide

whether

to

pop

the

stack

Then

why

do

we

need

a

while

loop

as

above

The

reason

is

that

popping

and

restoring

probability

distributions

takes

time

-LRB-

t

increases

during

execution

of

the

while

loop

-RRB-

possibly

causing

utility

values

to

drop

Therefore

in

the

process

of

popping

and

restoring

distributions

modified

by

one

program

the

payoff/time

ratio

of

the

preceding

program

may

fall

enough

so

that

the

distributions

it

modified

must

be

restored

too

The

process

can

potentially

continue

until

the

stack

is

completely

empty

FACT

After

each

instruction

-LRB-

except

during

the

execution

of

a

self-modification

program

-RRB-

the

top-level

ensures

that

the

beginning

of

each

completed

self-modification

program

that

computed

valid

probability

modifications

has

been

followed

by

faster

payoff

intake

than

the

beginnings

of

all

previous

such

self-modification

programs

All

currently

valid

probability

modifications

were

computed

by

currently

useful

self-modification

programs

The

nature

of

the

environment

does

not

matter

Proof

See

formal

top-level

description

and

FACT

Twenty

Comments

The

experiments

are

described

in

section

If

you

are

in

a

hurry

you

can

skip

these

-LRB-

mostly

rather

obvious

-RRB-

comments

Why

self-delimiting

self-modification

programs

The

EndSelfMod

primitive

allows

the

system

to

delay

top-level

evaluations

of

probability

modifications

arbitrarily

The

expectation

of

the

delay

remains

finite

however

due

to

MinP

being

positive

The

system

's

delaying

capabilities

are

important

for

two

reasons

-LRB-

-RRB-

In

general

payoff

events

will

be

separated

by

long

-LRB-

unknown

-RRB-

time

lags

Hence

novel

probability

modifications

are

not

necessarily

bad

if

they

do

not

lead

to

immediate

payoff

The

system

itself

should

be

able

to

learn

how

much

time

to

spend

on

waiting

for

first

payoff

events

-LRB-

-RRB-

Two

successive

modifications

of

two

particular

probability

distributions

may

turn

out

to

be

beneficial

while

each

by

itself

may

be

harmful

Therefore

the

system

should

be

able

to

compute

arbitrary

sequences

of

probability

modifications

before

facing

top-level

evaluations

Delaying

top-level

evaluations

does

cost

time

though

which

is

taken

into

account

when

usefulness

is

measured

In

the

long

run

the

system

is

encouraged

to

create

useful

self-modification

programs

of

the

appropriate

size

Non-decreasing

search

space

Due

to

MinP

being

positive

there

will

always

be

a

non-vanishing

-LRB-

possibly

tiny

-RRB-

probability

of

executing

any

program

at

any

time

Thus

the

space

of

possible

action

subsequences

will

never

really

decrease

Only

the

probability

distribution

on

this

space

-LRB-

the

bias

-RRB-

can

change

But

there

can

not

be

total

determinism

corresponding

to

total

lack

of

exploration

Speeding

up

payoff

intake

Learning

how

to

learn

The

top-level

takes

the

entire

learning

history

into

account

note

that

at

time

t

the

value

t

S

-LRB-

S

-LRB-

sp

-RRB-

first

-RRB-

t

stands

for

all

the

time

since

the

beginning

of

the

most

recent

self-modification

program

whose

effects

have

not

yet

been

countermanded

The

utility

value

of

a

self-modification

program

is

based

on

total

elapsed

time

since

the

program

began

This

includes

the

computation

time

required

for

learning

Over

time

the

system

tends

to

make

better

and

better

use

of

its

limited

temporal

and

spatial

resources

due

to

FACT

self-modifications

that

speed

up

payoff

intake

in

the

long

run

are

preferred

So

are

self-modifications

speeding

up

the

search

for

self-modifications

speeding

up

payoff

intake

This

encourages

learning

how

to

learn

and

learning

how

to

learn

how

to

learn

and

represents

an

essential

difference

to

previous

approaches

to

continual

learning

see

-LRB-

Ring

-RRB-

Directed

mutations

as

opposed

to

random

mutations

Unlike

evolutionary

and

genetic

algorithms

-LRB-

Rechenberg

Schwefel

Holland

Hoffmeister

and

Back

Koza

-RRB-

self-modification

programs

may

lead

to

very

specific

directed

sequences

of

strategy

mutations

as

opposed

to

undirected

totally

random

mutations

The

system

can

arbitrarily

modify

its

prior

distribution

on

the

space

of

solution

candidates

Just

as

evolution

discovered

that

having

the

genetic

crossover

operator

was

a

good

thing

the

system

is

potentially

able

to

discover

that

various

more

directed

search

strategies

are

good

things

Life

is

one-way

Note

that

only

direct

effects

of

self-modification

programs

on

primitive

probability

distributions

can

be

countermanded

by

the

top-level

strategy

In

realistic

environments

it

is

not

possible

to

countermand

all

indirect

effects

and

effects

of

the

system

behavior

on

the

unknown

environment

life

is

one-way

However

the

top

level

may

encourage

the

development

of

environment-specific

strategies

for

countermanding

certain

indirect

effects

Such

strategies

will

be

kept

as

long

as

they

appear

to

be

more

useful

than

previous

strategies

By

focusing

on

the

observation

and

control

of

changes

of

probability

distributions

-LRB-

as

opposed

to

general

changes

involving

internal

state

and

environment

-RRB-

the

top

level

attempts

to

control

a

complex

world

by

controlling

a

small

part

of

it

namely

the

variable

probability

distributions

The

latter

however

may

have

an

arbitrary

influence

on

themselves

and

the

rest

of

the

world

Usefulness

and

true

usefulness

Incremental

self-improvement

keeps

useful

self-modifications

only

in

the

sense

that

useful

was

defined

above

However

the

system

will

never

have

a

proof

that

a

particular

self-modification

program

was

the

true

reason

for

more

payoff

In

fact

what

the

system

actually

does

is

inductive

inference

based

on

single

experiences

at

one

point

in

its

life

it

did

something

and

at

some

later

point

it

measures

apparent

overall

effects

on

its

performance

What

appeared

to

be

useful

up

until

now

is

assumed

to

remain

useful

in

the

future

though

it

may

have

been

just

a

fluke

that

might

later

turn

out

actually

to

have

harmful

consequences

shifts

of

inductive

bias

generated

by

the

system

itself

may

be

evaluated

as

harmful

in

the

eyes

of

a

god-like

external

observer

with

additional

prior

knowledge

But

without

access

to

complete

knowledge

of

the

environment

the

system

is

forced

to

rely

upon

its

previous

experience

to

decide

what

's

harmful

and

what

's

not

This

is

what

inductive

inference

is

all

What

about

self-modifications

useful

just

by

chance

This

question

is

related

to

the

last

comment

For

the

sake

of

the

argument

suppose

a

self-modification

program

is

considered

useful

by

the

system

but

not

by

a

god-like

external

observer

This

does

not

at

all

imply

a

fatal

catastrophy

typically

the

reason

for

the

apparent

usefulness

of

the

actually

useless

-LRB-

or

even

harmful

-RRB-

self-modification

program

will

be

that

its

long-term

effects

were

overcompensated

-LRB-

before

the

corresponding

probability

modifications

were

cancelled

-RRB-

by

later

truly

useful

selfmodification

programs

And

note

that

the

system

will

always

have

a

chance

to

undo

previous

probability

modifications

by

executing

appropriate

additional

self-modifications

Universality

Learning

to

remember

It

is

not

difficult

to

show

that

the

primitives

in

Table

form

a

universal

set

in

the

following

sense

they

can

be

composed

to

form

programs

writing

any

computable

integer

sequence

onto

the

work

area

within

the

hardwired

size

and

range

limitations

Note

that

the

primitives

make

it

easy

to

create

action

sequences

for

handling

stacks

recursion

etc.

The

scheme

allows

for

very

general

sequential

interaction

with

the

environment

-LRB-

given

appropriate

problem-specific

actions

that

translate

storage

contents

into

output

actions

and

environmental

changes

-RRB-

The

self-referential

primitives

are

designed

to

allow

for

specific

changes

of

probability

distributions

of

all

program

cells

-LRB-

possibly

done

very

quickly

making

things

like

one-shot

learning

possible

-RRB-

Universality

implies

that

the

system

is

in

principle

capable

of

creating

programs

for

storing

representations

of

environmental

events

Unlike

with

most

previous

reinforcement-learning

algorithms

see

e.g.

-LRB-

Barto

Watkins

Dayan

and

Sejnowski

Williams

Sutton

-RRB-

there

is

no

need

for

a

Markovian

interface

-LRB-

Schmidhuber

-RRB-

between

the

environment

and

the

learning

system

Also

there

is

no

need

for

a

discount

factor

discounting

the

system

's

expectation

of

future

payoff

in

case

of

potentially

infinite

life

spans

Does

n't

the

system

start

with

a

huge

disadvantage

Conventional

learning

systems

have

a

fixed

learning

strategy

for

selecting

and

testing

solution

candidates

from

some

non-universal

search

space

Incremental

self-improvement

however

does

not

only

search

for

solutions

to

some

specific

task

but

also

for

learning

strategies

for

finding

solutions

Does

n't

the

system

's

universality

increase

its

search

space

In

general

it

does

With

many

toy

tasks

an

external

user

will

be

able

to

provide

a

conventional

learning

algorithm

with

enough

problem-specific

bias

to

solve

a

certain

task

more

quickly

than

-LRB-

initially

less

informed

-RRB-

search

based

on

incremental

self-improvement

On

the

other

hand

however

unlike

previous

learning

methods

incremental

self-improvement

can

use

experience

to

modify

its

search

in

a

universal

way

by

exploiting

arbitrary

task-specific

regularities

if

there

are

any

and

by

creating

its

own

problem-specific

bias

In

the

long

run

this

advantage

may

outweigh

initial

disadvantages

due

to

universality

tomorrow

you

will

be

punished

for

scratching

your

ear

years

ago

maybe

this

is

in

the

nature

of

the

algorithm

running

the

universe

There

is

no

proof

that

this

is

not

going

to

happen

-LRB-

though

our

environment

appears

to

be

somewhat

more

benign

than

this

-RRB-

In

general

you

would

not

have

a

chance

to

discover

the

true

reason

for

the

punishment

Inserting

prior

bias

Efficiency

considerations

Primitive

instructions

need

not

be

low-level

instructions

like

those

in

Table

They

may

correspond

to

complex

submodules

reflecting

the

user

's

prior

knowledge

Informally

there

is

one

general

constraint

to

obey

-LRB-

Schmidhuber

-RRB-

whatever

is

computable

on

the

used

hardware

should

be

computable

just

as

efficiently

-LRB-

up

to

a

small

constant

factor

-RRB-

by

a

program

written

in

the

programming

language

For

instance

on

a

typical

serial

digital

machine

we

would

like

to

have

instructions

exploiting

fast

storage

addressing

mechanisms

We

would

not

want

to

limit

ourselves

to

the

simulation

of

say

a

slow

one

tape

Turing

machine

Likewise

on

a

machine

with

many

parallel

processors

we

would

like

to

use

a

set

of

instructions

allowing

for

processes

with

maximal

parallelism

Bias

towards

short

runs

Unlike

Levin

's

universal

search

algorithm

-LRB-

which

is

optimal

for

a

wide

variety

of

non-incremental

search

problems

based

on

trials

with

exactly

repeatable

initial

conditions

see

Levin

-RRB-

the

system

presented

here

has

no

explicit

bias

towards

runs

with

low

Kolmogorov

complexity

or

low

Levin

complexity

-LRB-

Kolmogorov

Chaitin

Solomonoff

Levin

-RRB-

e.g.

runs

based

on

only

few

instructions

repeated

over

and

over

again

In

principle

however

it

may

create/strengthen

such

a

bias

and

the

bias

will

stick

if

it

appears

to

be

useful

Of

course

a

priori

bias

of

this

kind

can

be

explicitly

introduced

by

the

programmer

One

possibility

is

to

reward

low-complexity

runs

more

than

others

-LRB-

by

providing

more

external

payoff

-RRB-

Another

possibility

is

this

instead

of

selecting

primitives

randomly

-LRB-

according

to

the

current

probability

distributions

-RRB-

at

each

time

step

of

each

run

make

random

selections

only

if

IP

points

to

a

program

cell

that

has

not

yet

been

used

during

the

current

run

Otherwise

use

the

instruction

executed

during

the

most

recent

visit

of

the

program

cell

This

leads

to

an

explicit

bias

towards

low

algorithmic

probability

-LRB-

Solomonoff

-RRB-

and

has

been

done

previously

in

-LRB-

Schmidhuber

-RRB-

Occasionally

this

will

lead

to

non-halting

programs

For

such

cases

upper

runtime

bounds

need

to

be

introduced

In

the

spirit

of

the

incremental

self-improvement

paradigm

such

time

bounds

should

be

computed

by

the

system

itself

-LRB-

using

appropriate

special

primitives

-RRB-

For

an

additional

comment

on

inserting

prior

bias

see

section

Exploration/exploitation

tradeoff

The

system

itself

can

decide

how

much

time

it

wants

to

spend

on

exploring

effects

of

new

action

sequences

and

how

much

time

it

wants

to

spend

on

exploiting

beneficial

effects

of

action

sequences

that

it

tried

before

In

the

long

run

the

system

will

prefer

those

strategies

that

led

to

the

best

-LRB-

environment-specific

-RRB-

balance

between

exploration

and

exploitation

One

task

many

tasks

An

external

user

may

choose

a

way

of

translating

tasks

and

system

performance

into

payoff

From

the

user

's

point

of

view

there

may

be

many

tasks

and

the

system

itself

may

choose

which

to

attack

first

From

the

system

's

point

of

view

there

is

only

one

task

namely

to

maximize

cumulative

payoff

Note

that

every

task

that

requires

the

maximization

of

some

kind

of

reward

may

be

viewed

as

being

decomposable

into

many

tasks

the

first

task

is

to

generate

actions

leading

to

a

little

bit

of

reward

The

next

task

is

to

generate

actions

leading

to

more

reward

etc.

What

if

the

task

changes

In

the

light

of

what

has

been

said

above

this

is

actually

a

misleading

question

It

is

tinged

by

the

idea

of

exactly

repeatable

training

events

suddenly

being

replaced

by

different

exactly

repeatable

training

events

But

in

this

paper

there

is

no

unrealistic

a

priori

assumption

of

exactly

repeatable

training

events

From

the

system

's

point

of

view

there

is

only

one

task

namely

to

maximize

cumulative

payoff

-LRB-

see

previous

comment

-RRB-

The

system

always

tends

to

keep

the

strategy

that

led

to

the

best

overall

results

so

far

Without

additional

prior

knowledge

there

are

no

alternatives

the

system

can

not

know

whether

payoff

changes

are

due

to

external

task

changes

or

whether

they

are

due

to

long

term

effects

of

its

own

previous

actions

-LRB-

as

discussed

in

comments

and

above

-RRB-

Life

is

one-way

and

change

is

in

the

nature

of

a

dynamic

environment

For

the

sake

of

the

argument

however

suppose

a

particular

sequence

of

probability

modifications

appears

justified

at

a

certain

point

and

the

external

observer

decides

that

the

first

task

is

solved

At

time

however

the

system

fails

to

keep

its

old

payoff/time

ratio

because

the

task

has

changed

in

the

eyes

of

the

external

observer

Then

over

time

direct

effects

of

previously

useful

self-modification

programs

will

tend

to

be

countermanded

in

inverse

order

of

their

occurrence

-LRB-

unless

they

do

n't

get

protected

by

additional

useful

self-modification

programs

-RRB-

until

the

current

probability

distributions

reflect

knowledge

useful

for

solving

both

tasks

All

probability

modifications

will

be

countermanded

only

if

the

initial

strategies

developed

for

solving

the

first

task

are

useless

for

solving

the

second

task

But

without

additional

prior

knowledge

this

does

make

sense

from

the

learning

system

's

point

of

view

Teacher

as

part

of

the

environment

-LRB-

-RRB-

Of

course

an

external

teacher

may

provide

taskspecific

inputs

conveying

information

about

task

changes

But

the

system

has

first

to

learn

to

make

use

of

these

inputs

Analoguously

children

first

have

to

learn

to

interprete

sound

waves

emitted

by

their

parents

as

teacher

signals

They

will

learn

this

if

it

turns

out

to

be

useful

in

the

long

run

-LRB-

-RRB-

To

achieve

his

teaching

goals

the

teacher

may

directly

influence

the

way

payoff

is

generated

thus

influencing

the

context

sensitivity

of

the

reward

In

both

cases

it

is

natural

to

view

the

teacher

as

part

of

the

environment

Direct

teacher

forcing

The

teacher

may

decide

that

the

current

strategy

of

the

system

-LRB-

at

time

-RRB-

is

actually

a

valuable

one

and

should

not

be

countermanded

Instead

of

influencing

payoff

generation

-LRB-

see

previous

comment

-RRB-

he

may

decide

to

influence

the

learning

process

directly

by

preventing

the

top-level

strategy

from

countermanding

probability

modifications

generated

by

selfmodification

programs

considered

useful

at

time

This

would

be

one

way

to

insert

additional

prior

knowledge

Success

history

in

stack

At

a

given

time

the

current

history

of

useful

self-modifications

is

reflected

by

the

current

stack

entries

Each

self-modification

program

on

the

stack

was

followed

by

faster

payoff

intake

than

all

previous

self-modification

programs

on

the

stack

This

is

true

despite

the

fact

that

time

for

computing

and

testing

later

self-modification

programs

is

taken

into

account

Useful

self-modification

programs

are

rare

Each

self-modification

program

undergoes

a

test

which

may

last

for

the

entire

remaining

system

life

provided

the

program

is

considered

useful

for

such

a

long

time

Typically

only

few

self-modification

programs

will

be

followed

by

faster

payoff

intake

than

all

previous

useful

self-modification

programs

Therefore

the

costs

of

saving

old

probability

distributions

in

the

stack

typically

will

tend

to

remain

comparatively

small

This

is

borne

out

by

the

experiments

in

section

Limited

stacksize

circular

stack

In

practical

applications

the

stack

will

be

finite

A

circular

stack

that

overwrites

earlier

stack

entries

-LRB-

starting

from

the

bottom

entries

-RRB-

could

keep

track

of

self-modifications

after

stack

overflow

-LRB-

circular

stack

-RRB-

Only

the

StackSize

most

recent

probability

modifications

could

then

be

directly

restored

by

popping

However

every

probability

distribution

can

be

indirectly

restored

by

additional

self-modification

programs

executed

by

the

system

itself

In

the

experiments

conducted

so

far

there

never

was

a

danger

of

stack

overflow

See

section

It

is

intended

to

introduce

additional

introspective

primitives

for

addressing

and

examining

stack

entries

-LRB-

in

the

style

of

GetP

-RRB-

This

is

not

yet

implemented

however

When

to

apply

incremental

self-improvement

It

is

always

possible

to

construct

cruel

environments

where

previous

experiences

are

necessarily

useless

for

future

planning

Indeed

as

can

be

seen

from

what

has

been

said

in

the

introduction

almost

all

thinkable

environments

are

of

this

kind

-LRB-

except

those

which

we

generally

are

most

interested

in

those

with

regularities

-RRB-

The

incremental

self-improvement

paradigm

wo

n't

be

of

any

help

in

the

general

case

The

same

Primitive

Semantics

Write

-LRB-

-RRB-

Read

-LRB-

-RRB-

Table

Semantics

of

problem

specific

primitives

and

their

parameters

Again

double-indexed

indirect

addressing

is

employed

See

text

for

rules

for

syntactic

correctness

Compare

with

Table

holds

for

any

other

learning

paradigm

though

However

if

certain

aspects

of

the

environment

repeat

themselves

if

experiments

conducted

in

the

environment

do

not

change

it

such

that

previous

knowledge

becomes

totally

useless

if

the

tasks

to

be

solved

do

exhibit

regularities

then

the

incremental

self-improvement

paradigm

appears

to

be

a

very

general

way

of

exploiting

this

Incremental

self-improvement

should

be

of

interest

in

cases

where

the

user

's

bias

is

already

captured

by

the

choice

of

the

initial

programming

language

and

where

the

user

expects

additional

-LRB-

yet

unknown

-RRB-

problem-specific

regularities

Illustrative

Experiments

The

following

brief

case

studies

are

not

designed

to

impress

but

to

illustrate

basic

aspects

of

the

system

The

first

task

requires

to

compute

regular

integer

strings

The

second

task

is

a

maze

task

from

-LRB-

Sutton

-RRB-

With

both

tasks

the

system

uses

low-level

problem-specific

primitives

in

addition

to

the

general

primitives

from

Table

The

primitives

reflect

the

system

's

initial

-LRB-

weak

-RRB-

bias

Of

course

different

problem-specific

primitives

lead

to

different

initial

bias

and

performance

A

task

that

can

be

solved

within

a

few

minutes

using

one

set

of

primitives

may

require

a

day

of

computation

time

using

a

different

set

of

primitives

The

purpose

of

this

section

however

is

not

to

perform

a

statistically

significant

experimental

evaluation

of

the

system

's

initial

bias

or

to

study

effects

of

introducing

different

kinds

of

initial

bias

or

to

compare

the

system

to

other

learning

systems

with

different

initial

bias

Instead

this

section

's

purpose

is

to

describe

typical

aspects

of

system

lives

illustrating

the

system

's

basic

-LRB-

bias

independent

-RRB-

mode

of

operation

Writing

Variable

Sequences

Task

The

external

environment

consists

of

an

array

of

variables

The

i-th

variable

is

denoted

by

Vi

Its

current

contents

are

denoted

by

Ci

-LSB-

Maxint

Maxint

-RSB-

Time

is

measured

in

discrete

time

steps

At

time

step

all

variables

are

initialized

with

zeros

Every

time

steps

the

number

of

variables

whose

values

equal

their

addresses

is

written

into

a

special

input

cell

This

number

is

the

current

payoff

Then

all

variables

are

re-initialized

with

zeros

The

goal

is

to

maximize

cumulative

payoff

Details

In

addition

to

the

general

primitives

from

Table

-LRB-

not

counting

input/output

primitives

-RRB-

there

are

two

problem-specific

primitives

Each

has

two

integer

arguments

W

rite

-LRB-

-RRB-

writes

the

contents

of

the

storage

cell

indirectly

addressed

by

the

first

argument

into

the

variable

indirectly

addressed

by

the

second

argument

Read

-LRB-

-RRB-

writes

the

contents

of

the

variable

indirectly

addressed

by

the

second

argument

into

the

work

cell

indirectly

addressed

by

the

second

argument

See

Table

Write

and

read

operations

outside

the

valid

ranges

halt

the

current

run

Since

nops

all

initial

probabilities

of

all

possible

contents

of

all

program

cells

are

equal

to

Parameters

for

storage

size

etc.

are

Min

Max

P

rogramStart

bottom

-LRB-

nops

-RRB-

MinP

StackSize

Maxint

To

inform

the

system

about

what

is

going

on

the

following

values

are

written

into

special

input

cells

whenever

they

change

IP

sp

and

the

remainder

of

t

Maxint

-LRB-

integer

division

where

t

denotes

the

current

time

-RRB-

Measuring

time

By

definition

each

computation

that

requires

the

consideration

of

all

nops

probabilities

of

some

program

cell

-LRB-

such

as

selecting

an

instruction

selecting

a

parameter

pushing

or

popping

probability

distributions

during

top-level

management

etc.

-RRB-

costs

one

time

step

Other

computations

do

not

cost

anything

This

ensures

that

measured

time

is

of

the

order

of

total

cpu-time

The

somewhat

unelegant

way

of

measuring

time

was

introduced

because

measuring

cpu-time

directly

turned

out

to

be

somewhat

unreliable

How

difficult

is

this

task

For

a

number

of

reasons

the

task

is

non-trivial

the

system

does

not

appear

to

have

much

built-in

bias

towards

the

task

-LRB-

-RRB-

Only

one

of

the

primitives

-LRB-

W

rite

-RRB-

may

affect

variable

contents

at

all

But

initially

the

system

does

not

even

have

such

seemingly

trivial

knowledge

there

is

no

built-in

idea

about

which

actions

may

lead

to

payoff

Therefore

it

has

to

find

out

on

its

own

-LRB-

-RRB-

The

values

referred

to

by

the

two

arguments

of

W

rite

have

to

be

identical

and

within

the

required

ranges

to

lead

to

a

useful

result

-LRB-

-RRB-

There

are

different

variables

with

different

values

Only

one

of

them

namely

is

correctly

re-initialized

with

its

own

address

after

each

payoff

event

-LRB-

-RRB-

There

is

no

explicit

a

priori

bias

towards

short

programs

such

as

the

one

in

-LRB-

Schmidhuber

-RRB-

for

a

related

task

-LRB-

-RRB-

Finally

recall

that

the

work

area

is

never

re-initialized

after

system

birth

Hence

as

mentioned

above

it

may

be

viewed

as

part

of

the

environment

of

the

program

area

The

environment

is

changing

quite

unpredictably

due

to

actions

executed

by

the

system

itself

Performance

was

measured

with

and

without

self-modification

capabilities

In

the

latter

case

the

primitives

IncP

and

DecP

had

no

effect

Both

versions

were

run

for

time

steps

corresponding

to

payoff

events

or

about

half

a

day

of

computation

time

on

a

SUN

SPARC

Note

that

the

optimal

cumulative

payoff

is

This

value

can

be

achieved

only

by

a

system

with

optimal

prior

bias

starting

at

birth

such

a

system

keeps

executing

optimal

actions

without

having

to

learn

anything

Results

Without

Self-Modifications

At

system

death

total

payoff

equaled

about

Average

payoff

per

payoff

event

was

about

Most

of

the

total

payoff

-LRB-

about

-RRB-

could

be

attributed

to

the

fact

that

was

correctly

re-initialized

after

each

payoff

event

the

system

received

a

little

bit

of

payoff

even

in

cases

where

it

did

not

execute

any

write

operations

As

expected

average

payoff

intake

did

not

significantly

increase

or

decrease

during

the

lifetime

of

the

system

However

this

was

not

safely

predictable

in

advance

due

to

the

changing

environment

Results

With

Self-Modifications

At

system

death

total

payoff

was

about

To

find

out

whether

the

incremental

self-improvement

paradigm

did

indeed

lead

to

incremental

self-improvement

let

us

have

a

look

at

the

learning

history

Self-generated

reduction

of

numbers

of

probability

modifications

In

the

beginning

the

system

computed

a

lot

of

probability

modifications

but

soon

preferred

to

decrease

the

number

of

probability

modifications

per

time

interval

There

were

probability

modifications

during

the

first

time

steps

There

were

probability

modifications

during

the

second

time

steps

Almost

all

probability

changes

were

countermanded

by

the

top-level

strategy

by

this

time

the

stack

had

only

entries

corresponding

to

useful

self-modification

programs

Most

of

the

useful

self-modification

programs

computed

either

one

or

two

probability

modifications

After

time

steps

there

were

only

about

probability

modifications

per

time

steps

Speed-up

of

payoff

intake

By

then

the

system

behaved

much

more

deterministically

Average

payoff

per

payoff

event

had

increased

from

to

-LRB-

the

optimal

value

being

of

course

-RRB-

and

the

stack

had

entries

These

entries

corresponded

to

modifications

of

single

cell

probability

distributions

computed

by

self-modification

programs

each

being

more

useful

than

all

the

previous

ones

Storage

already

looked

very

messy

For

instance

almost

all

cells

in

the

work

area

were

filled

with

-LRB-

partly

big

-RRB-

integers

quite

different

from

the

initial

values

Recall

that

the

work

area

is

never

re-initialized

and

may

be

viewed

as

part

of

the

environment

of

the

program

area

First

maximal

payoff

After

payoff

events

the

system

correctly

had

written

all

variables

for

the

first

time

and

received

maximal

payoff

Due

to

remaining

non-determinism

in

the

system

the

current

average

payoff

per

payoff

event

-LRB-

measured

shortly

afterwards

at

time

step

-RRB-

was

about

After

payoff

events

current

average

payoff

per

payoff

event

was

By

now

the

number

of

probability

modifications

per

time

steps

was

up

to

again

But

the

stack

had

only

entries

-LRB-

corresponding

to

useful

self-modification

programs

-RRB-

After

payoff

events

current

average

payoff

per

payoff

event

was

-LRB-

stack

entries

-RRB-

After

payoff

events

-LRB-

at

system

death

-RRB-

it

was

about

with

tendency

to

increase

By

then

there

were

stack

entries

They

corresponded

to

self-modification

programs

each

being

more

useful

than

all

the

previous

ones

Temporary

speed-ups

of

performance

improvement

Performance

did

not

increase

smoothly

during

the

lifetime

of

the

system

For

instance

at

time

step

the

system

correctly

had

written

more

than

variables

for

the

first

time

-LRB-

namely

-RRB-

This

record

was

not

broken

for

a

long

time

for

nearly

additional

time

steps

This

time

interval

is

comparable

to

the

entire

previous

learning

time

Then

an

unexpected

sequence

of

rather

quick

improvements

began

At

time

the

new

record

was

correct

variables

At

time

the

new

record

was

At

time

the

new

record

was

Nearly

immediately

afterwards

at

time

the

new

record

was

Thus

within

less

than

time

steps

the

record

was

broken

times

Then

progress

slowed

down

again

Such

temporary

speed-ups

of

performance

improvement

indicate

useful

shifts

of

inductive

bias

which

may

later

be

replaced

by

inductive

bias

created

by

the

next

breakthrough

Automatic

fine-tuning

of

search

space

structure

A

look

at

the

stack

entries

revealed

that

many

-LRB-

but

far

from

all

-RRB-

useful

probability

modifications

focused

on

few

program

cells

Often

selfmodification

programs

directly

changing

the

probabilities

of

certain

additional

self-modification

programs

were

considered

useful

For

instance

of

the

stack

entries

at

time

step

corresponded

to

useful

probability

modifications

of

the

-LRB-

self-referential

-RRB-

IncP

action

of

the

program

cell

with

address

entries

corresponded

to

useful

modifications

of

the

EndSelfMod

probability

of

various

cells

Such

stack

entries

may

be

interpreted

as

results

of

adjusting

the

prior

on

the

space

of

solution

candidates

or

fine-tuning

search

space

structure

or

learning

to

create

directed

mutations

or

learning

how

to

learn

A

Navigation

Task

Task

-LRB-

following

Sutton

-RRB-

The

external

environment

consists

of

a

two-dimensional

grid

with

by

fields

Fi

j

denotes

the

field

in

the

i-th

row

and

the

j-th

column

The

following

fields

are

blocked

by

obstacles

In

the

beginning

an

artificial

agent

is

placed

on

-LRB-

the

start

field

-RRB-

In

addition

to

the

general

primitives

from

Table

-LRB-

not

counting

input/output

primitives

-RRB-

there

are

four

problem-specific

primitives

with

obvious

meaning

one-step-north

-LRB-

-RRB-

one-stepsouth

-LRB-

-RRB-

one-step-east

-LRB-

-RRB-

one-step-west

-LRB-

-RRB-

The

system

can

not

execute

actions

that

would

lead

outside

the

grid

or

into

an

obstacle

Again

the

following

values

are

written

into

special

cells

in

the

input

area

whenever

they

change

IP

sp

remainder

-LRB-

t

Maxint

-RRB-

Another

input

cell

is

filled

with

a

whenever

the

agent

is

on

the

goal

field

otherwise

it

is

filled

with

a

Four

additional

input

cells

are

rewritten

after

each

execution

of

some

problem-specific

primitive

the

first

-LRB-

second

third

fourth

-RRB-

cell

is

filled

with

Maxint

if

the

field

to

the

north

-LRB-

south

east

west

-RRB-

of

the

agent

is

blocked

or

does

not

exist

otherwise

the

cell

is

filled

with

Maxint

Whenever

the

agent

reaches

-LRB-

the

goal

field

-RRB-

the

system

receives

a

constant

payoff

-LRB-

-RRB-

and

the

agent

is

transferred

back

to

-LRB-

the

start

field

-RRB-

Parameters

for

storage

size

etc.

are

the

same

as

with

the

previous

task

and

time

is

measured

the

same

way

Clearly

to

maximize

cumulative

payoff

the

system

has

to

find

short

paths

from

start

to

goal

How

difficult

is

this

task

Again

the

system

does

not

appear

to

have

much

built-in

bias

towards

the

task

-LRB-

-RRB-

Unlike

with

previous

reinforcement

learning

algorithms

the

system

does

not

have

a

smart

initial

strategy

for

temporal

credit

assignment

it

has

to

develop

its

own

such

strategies

-LRB-

-RRB-

Unlike

with

Sutton

's

original

set-up

-LRB-

-RRB-

the

system

does

not

see

a

built-in

unique

representation

of

its

current

position

on

the

grid

From

the

system

's

point

of

view

its

interface

to

the

environment

is

nonMarkovian

-LRB-

Schmidhuber

-RRB-

the

current

input

does

not

provide

all

information

about

the

agent

's

current

position

-LRB-

-RRB-

To

make

use

of

the

few

inputs

it

gets

the

system

first

has

to

discover

that

certain

input

cells

may

be

relevant

for

solving

its

task

-LRB-

-RRB-

The

total

environment

-LRB-

including

the

work

area

-RRB-

is

changing

quite

unpredictably

due

to

actions

executed

by

the

system

itself

Results

without

Self-Modifications

As

with

the

previous

task

the

system

was

first

tested

with

self-referential

primitives

IncP

and

DecP

being

switched

off

At

system

death

at

time

total

payoff

was

about

Average

trial

length

-LRB-

number

of

time

steps

required

to

move

from

start

to

goal

-RRB-

was

The

shortest

trial

ever

occurred

around

time

step

and

took

time

steps

Results

with

Self-Modifications

At

system

death

-LRB-

at

time

-RRB-

total

payoff

was

about

By

then

average

trial

length

-LRB-

including

time

required

for

top-level

management

of

course

-RRB-

was

down

to

time

steps

-LRB-

as

opposed

to

more

than

time

steps

without

self-modifications

-RRB-

with

ongoing

tendency

to

decrease

As

with

the

previous

task

performance

did

not

improve

smoothly

The

history

of

broken

records

reflects

the

history

of

performance

improvements

First

there

was

a

rather

quick

sequence

of

improvements

which

lasted

until

time

By

then

-LRB-

after

payoff

events

-RRB-

the

shortest

trial

so

far

had

taken

time

steps

Then

the

current

record

did

not

improve

any

more

for

a

comparatively

long

time

interval

time

steps

the

length

of

this

boring

time

interval

by

far

exceeded

the

entire

previous

learning

time

Sudden

improvement

speed-up

Then

quite

unexpected

to

the

observer

the

system

started

to

create

a

new

sequence

of

additional

improvements

around

time

step

At

time

the

record

was

down

to

At

time

the

record

was

down

to

At

time

the

record

was

down

to

At

time

the

record

was

down

to

At

time

the

record

was

down

to

At

time

the

record

was

down

to

Thus

within

about

time

steps

the

record

was

broken

times

sometimes

dramatically

Then

performance

improvement

slowed

down

again

Throughout

this

flurry

of

broken

records

starting

at

time

the

number

of

stack

entries

increased

quite

steadily

from

-LRB-

corresponding

to

useful

self-modification

programs

-RRB-

to

-LRB-

corresponding

to

self-modification

programs

-RRB-

Apparently

around

time

step

the

system

made

a

revolutionary

discovery

that

permitted

a

sequence

of

more

evolutionary

additional

directed

selfmutations

At

system

death

-LRB-

time

step

-RRB-

the

record

was

down

to

The

system

's

average

payoff

intake

per

time

interval

still

had

a

tendency

to

increase

In

the

end

there

were

useful

self-modification

programs

each

leading

to

better

results

than

all

previous

ones

As

with

the

previous

task

many

useful

self-modification

programs

directly

modified

the

probabilities

of

additional

self-modification

programs

Compare

the

paragraph

entitled

automatic

fine-tuning

search

space

structure

in

section

Experiment

corrupted

inputs

In

another

experiment

the

system

was

applied

to

the

same

task

but

inputs

were

corrupted

and

unreliable

In

the

beginning

it

took

the

system

much

longer

to

come

up

with

short

trials

At

time

the

current

record

was

Then

not

much

happened

for

a

long

time

there

was

only

one

minor

improvement

-LRB-

-RRB-

during

the

next

time

steps

Again

the

length

of

this

boring

time

interval

by

far

exceeded

the

entire

previous

learning

time

Then

around

time

step

-LRB-

corresponding

to

half

a

day

of

computation

time

-RRB-

a

revolution

occurred

within

only

about

additional

time

steps

the

record

was

broken

times

at

time

the

record

was

down

to

Throughout

this

sudden

flurry

of

broken

records

starting

at

time

the

number

of

stack

entries

increased

quite

steadily

from

-LRB-

corresponding

to

useful

self-modification

programs

-RRB-

to

-LRB-

corresponding

to

self-modification

programs

-RRB-

Then

performance

improvement

slowed

down

again

System

life

ended

at

time

step

By

this

time

the

record

was

down

to

The

system

's

average

payoff

intake

per

time

interval

still

had

a

tendency

to

increase

Three

Comments

Stability

of

probability

modifications

With

the

experiments

conducted

so

far

the

top

level

hardly

ever

countermanded

probability

modifications

other

than

those

computed

by

the

most

recent

useful

self-modification

programs

For

instance

once

there

were

stack

entries

the

oldest

stack

entries

appeared

extremely

safe

and

had

a

good

chance

to

survive

the

entire

system

life

This

empirically

justifies

the

method

suggested

in

the

comment

on

limited

circular

stacks

in

section

Revolutions

and

evolutions

In

the

tasks

above

unexpected

temporary

speed-ups

of

performance

improvements

were

observed

Even

if

the

system

appears

to

be

stuck

for

a

long

time

the

external

observer

never

can

be

sure

that

it

will

not

suddenly

discover

a

new

revolutionary

shift

of

bias

that

builds

the

basis

for

additonal

smoother

evolutionary

performance

improvements

This

is

analoguous

to

the

history

of

science

itself

One

nice

thing

about

open-ended

incremental

self-improvement

is

that

there

is

no

significant

theoretical

limit

to

what

the

system

may

learn

This

is

of

course

due

to

the

universal

nature

of

the

underlying

programming

language

Informally

a

revolution

corresponds

to

a

self-improvement

with

high

conceptual

jump

size

-LRB-

an

expression

coined

by

Solomonoff

-RRB-

while

evolution

corresponds

to

a

sequence

of

selfimprovements

with

low

conceptual

jump

sizes

Inserting

prior

bias

The

experiments

above

certainly

are

not

meant

to

convince

the

reader

that

from

now

on

he

should

combine

the

incremental

self-improvement

paradigm

with

the

lowlevel

programming

language

from

section

and

apply

it

to

real

world

problems

Instead

the

experiments

are

meant

to

illustrate

basic

principles

of

the

paradigm

Of

course

with

large

scale

problems

it

is

desirable

to

insert

prior

knowledge

into

the

system

-LRB-

if

such

knowledge

is

indeed

available

-RRB-

With

incremental

self-improvement

a

priori

knowledge

resides

in

the

programmer

's

selection

of

primitives

with

problem-specific

built-in

bias

-LRB-

and

in

the

payoff

function

he

chooses

-RRB-

There

is

no

reason

why

certain

primitives

should

not

be

complex

time

consuming

programs

by

themselves

such

as

statistic

classifiers

neural

net

learning

algorithms

logic

programs

etc.

For

instance

using

different

primitives

for

the

navigation

task

from

section

can

greatly

reduce

the

time

required

to

achieve

near-optimal

trials

This

paper

however

is

not

a

study

of

the

effects

of

different

kinds

of

initial

bias

History

of

Ideas

Previous

Work

In

what

follows

I

will

briefly

describe

earlier

work

and

the

train

of

thought

leading

to

this

paper

Meta-evolution

My

first

attempts

to

come

up

with

schemes

for

true

self-referential

learning

based

on

universal

languages

date

back

to

They

were

partly

inspired

by

a

collaboration

with

Dickmanns

and

Winklhofer

-LRB-

-RRB-

We

used

a

genetic

algorithm

-LRB-

GA

-RRB-

to

evolve

variable

length

Prolog

programs

for

solving

simple

Soon

there

was

a

desire

to

improve

the

trivial

mutation

and

crossover

strategies

used

to

construct

new

programs

from

old

ones

This

led

to

an

algorithmic

scheme

-LRB-

called

meta-evolution

-RRB-

for

letting

more

sophisticated

strategies

be

learned

by

a

potentially

infinite

hierarchy

am

not

talking

about

fixed

learning

algorithms

for

adjusting

the

parameters

of

others

For

instance

GAs

are

sometimes

used

to

adjust

learning

rates

of

gradient

based

neural

nets

etc.

Or

a

neural

net

is

used

to

compute

the

weights

of

another

neural

net

In

the

literature

one

can

find

quite

a

few

approaches

of

this

kind

-LRB-

too

many

to

cite

them

all

I

settle

by

citing

not

even

my

own

-RRB-

Although

such

approaches

sometimes

may

have

their

merits

they

do

not

deserve

the

attribute

self-referential

the

additional

level

typically

just

defers

the

credit

assignment

problem

There

were

a

few

apparently

more

general

approaches

For

instance

Lenat

-LRB-

-RRB-

reports

that

his

Eurisko

system

was

able

to

discover

certain

heuristics

for

discovering

heuristics

However

his

approach

as

well

as

all

other

previous

approaches

I

am

aware

of

were

either

quite

limited

-LRB-

many

essential

aspects

of

system

behavior

being

unmodifiable

-RRB-

and/or

lacked

a

convincing

global

credit

assignment

strategy

-LRB-

as

embodied

by

the

top-level

strategy

of

the

incremental

self-improvement

paradigm

-RRB-

this

approach

would

be

classified

as

Genetic

Programming

e.g.

-LRB-

Koza

-RRB-

of

higher

level

GAs

whose

domains

were

to

construct

construction

strategies

-LRB-

Schmidhuber

-RRB-

Metaevolution

recursively

creates

a

growing

hierarchy

of

pools

of

programs

higher-level

pools

containing

program

modifying

programs

being

applied

to

lower-level

programs

and

being

rewarded

based

on

lowerlevel

performance

Collapsing

meta-levels

The

explicit

creation

of

meta-levels

and

meta-meta-levels

seemed

unnatural

however

For

this

reason

alternative

systems

based

on

self-referential

languages

were

explored

the

goal

being

to

collapse

all

meta-levels

into

one

-LRB-

Schmidhuber

-RRB-

At

that

time

however

no

convincing

global

credit

assignment

strategy

was

provided

Self-referential

neural

nets

Later

work

presented

a

neural

network

with

the

potential

to

run

its

own

weight

change

algorithm

-LRB-

Schmidhuber

-RRB-

With

this

system

top-level

credit

assignment

is

performed

by

gradient

descent

This

is

unsatisfactory

however

due

to

problems

with

local

minima

and

because

repeatable

training

sequences

are

required

In

general

this

makes

it

impossible

to

take

the

entire

learning

history

into

account

Algorithmic

probability

Universal

search

Levin

's

universal

search

algorithm

is

theoretically

optimal

for

certain

non-incremental

search

tasks

with

exactly

repeatable

initial

conditions

See

Levin

-LRB-

-RRB-

see

also

Adleman

-LRB-

-RRB-

There

were

a

few

attempts

to

extend

universal

search

to

incremental

learning

situations

where

previous

trials

may

provide

information

about

how

to

speed

up

further

learning

see

e.g.

-LRB-

Solomonoff

Paul

and

Solomonoff

Schmidhuber

-RRB-

For

instance

to

improve

future

performance

Solomonoff

-LRB-

-RRB-

describes

more

traditional

-LRB-

as

opposed

to

self-improving

-RRB-

methods

for

assigning

probabilities

to

successful

subprograms

Alternatively

one

of

the

actually

implemented

systems

in

-LRB-

Schmidhuber

-RRB-

simply

keeps

successful

code

in

its

program

area

This

system

was

a

conceptual

starting

point

for

the

one

in

the

current

paper

With

first

attempts

-LRB-

in

September

-RRB-

the

probability

distributions

underlying

the

Turing

machine

equivalent

language

required

for

universal

search

were

modified

heuristically

One

strategy

was

to

slightly

increase

the

context-dependent

probabilities

of

program

cell

contents

used

in

successful

programs

and

then

continue

universal

search

based

on

the

new

probability

distributions

With

a

number

of

experiments

this

actually

led

to

good

results

-LRB-

at

first

glance

more

impressive

results

than

those

in

the

current

paper

at

least

if

one

does

not

take

the

lack

of

bias

into

account

as

one

should

always

do

-RRB-

The

system

however

was

unsatisfactory

precisely

because

there

was

no

principled

way

of

adjusting

probability

distributions

This

criticism

led

to

the

ideas

expressed

in

the

current

paper

Meta-version

of

universal

search

Without

going

into

details

Solomonoff

-LRB-

-RRB-

mentions

that

self-improvement

may

be

formulated

as

a

time-limited

optimization

problem

thus

being

solvable

by

universal

search

However

the

straight-forward

meta-version

of

universal

search

-LRB-

generating

and

evaluating

probability

distributions

in

order

of

their

Levin

complexities

see

Levin

-RRB-

just

defers

the

credit

assignment

problem

to

the

meta-level

and

does

not

necessarily

make

optimal

incremental

use

of

computational

resources

and

previous

Note

that

incremental

self-improvement

is

not

a

meta-version

of

universal

search

In

fact

incremental

self-improvement

does

not

make

a

difference

between

search

and

meta-search

Ongoing/future

work

The

concrete

implementation

described

in

section

represents

only

one

out

of

many

ways

of

implementing

the

incremental

self-improvement

paradigm

It

is

intended

to

apply

incremental

self-improvement

to

more

complex

tasks

including

prediction

and

control

tasks

using

a

variety

of

universal

self-referential

sets

of

primitives

including

sets

designed

to

exploit

the

benefits

of

parallel

neural

net-like

hardware

appears

to

be

well

aware

of

problems

with

the

meta-version

at

the

end

of

his

paper

he

refers

to

self-improvement

as

a

more

distant

goal

The

kind

of

training

needed

involves

more

mathematics

and

work

on

various

kinds

of

optimization

problems

ultimately

problems

of

improving

computer

programs

Another

more

distant

goal

mentioned

by

Solomonoff

is

to

let

the

system

work

on

an

unordered

batch

of

problems

deciding

itself

which

are

the

easiest

and

solving

them

first

Note

that

the

incremental

self-improvement

paradigm

addresses

both

goals

without

depending

on

a

meta-version

of

universal

search

See

e.g.

comment

in

section

Acknowledgements

I

am

grateful

to

Ray

Solomonoff

Peter

Dayan

Mike

Mozer

Don

Matthis

Clayton

McMillan

and

various

NIPS

participants

for

valuable

comments/discussions

on

the

first

version

of

this

report

-LRB-

from

November

-RRB-

Many

thanks

to

Sepp

Hochreiter

Gerhard

Weiss

Martin

Eldracher

Margit

Kinder

and

Daniel

Prelinger

for

critical

remarks

on

earlier

drafts

and

to

Leslie

Kaelbling

David

Cohn

and

Tommi

Jaakkola

for

useful

comments

on

later

versions

I

am

particularly

indebted

to

Mark

Ring

for

extensive

and

constructive

criticism

References

Adleman

L.

-LRB-

-RRB-

Time

space

and

randomness

Technical

Report

Laboratory

for

Computer

Science

MIT

Barto

A.

G.

-LRB-

-RRB-

Connectionist

approaches

for

control

Technical

Report

COINS

Technical

Report

University

of

Massachusetts

Amherst

MA

Chaitin

G.

-LRB-

-RRB-

On

the

length

of

programs

for

computing

finite

binary

sequences

statistical

considerations

Journal

of

the

ACM

-LCB-

Dayan

P.

and

Sejnowski

T.

-LRB-

-RRB-

TD

-LRB->=-RRB-

Convergence

with

probability

Machine

Learning

-LCB-

Dickmanns

D.

Schmidhuber

J.

and

Winklhofer

A.

-LRB-

-RRB-

Der

genetische

Algorithmus

Eine

Implementierung

in

Prolog

Fortgeschrittenenpraktikum

Institut

fur

Informatik

Lehrstuhl

Prof.

Radig

Technische

Universitat

Munchen

Dietterich

T.

G.

-LRB-

-RRB-

Limitations

of

inductive

learning

In

Proceedings

of

the

Sixth

International

Workshop

on

Machine

Learning

Ithaca

NY

pages

-LCB-

San

Francisco

CA

Morgan

Kaufmann

Hoffmeister

F.

and

Back

T.

-LRB-

-RRB-

Genetic

algorithms

and

evolution

strategies

Similarities

and

differences

In

Manner

R.

and

Schwefel

H.

P.

editors

Proc

of

International

Conference

on

Parallel

Problem

Solving

from

Nature

Berlin

Springer

Holland

J.

H.

-LRB-

-RRB-

Adaptation

in

Natural

and

Artificial

Systems

University

of

Michigan

Press

Ann

Arbor

Kolmogorov

A.

-LRB-

-RRB-

Three

approaches

to

the

quantitative

definition

of

information

Problems

of

Information

Transmission

-LCB-

Koza

J.

R.

-LRB-

-RRB-

Genetic

evolution

and

co-evolution

of

computer

programs

In

Langton

C.

Taylor

C.

Farmer

J.

D.

and

Rasmussen

S.

editors

Artificial

Life

II

pages

-LCB-

Addison

Wesley

Publishing

Company

Lenat

D.

-LRB-

-RRB-

Theory

formation

by

heuristic

search

Machine

Learning

Levin

L.

A.

-LRB-

-RRB-

Laws

of

information

-LRB-

nongrowth

-RRB-

and

aspects

of

the

foundation

of

probability

theory

Problems

of

Information

Transmission

-LRB-

-RRB-

-LCB-

Levin

L.

A.

-LRB-

-RRB-

Randomness

conservation

inequalities

Information

and

independence

in

mathematical

theories

Information

and

Control

-LCB-

Li

M.

and

Vit?anyi

P.

M.

B.

-LRB-

-RRB-

An

Introduction

to

Kolmogorov

Complexity

and

its

Applications

Springer

Paul

W.

and

Solomonoff

R.

J.

-LRB-

-RRB-

Autonomous

theory

building

systems

Manuscript

revised

Rechenberg

I.

-LRB-

-RRB-

Evolutionsstrategie

Optimierung

technischer

Systeme

nach

Prinzipien

der

biologischen

Evolution

Dissertation

Published

by

Fromman-Holzboog

Ring

M.

B.

-LRB-

-RRB-

Continual

Learning

in

Reinforcement

Environments

PhD

thesis

University

of

Texas

at

Austin

Austin

Texas

Schaffer

C.

-LRB-

-RRB-

Overfitting

avoidance

as

bias

Machine

Learning

-LCB-

Schmidhuber

J.

H.

-LRB-

-RRB-

Evolutionary

principles

in

self-referential

learning

or

on

learning

how

to

learn

the

meta-meta

hook

Institut

fur

Informatik

Technische

Universitat

Munchen

Schmidhuber

J.

H.

-LRB-

-RRB-

Reinforcement

learning

in

Markovian

and

non-Markovian

environments

In

Lippman

D.

S.

Moody

J.

E.

and

Touretzky

D.

S.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Schmidhuber

J.

H.

-LRB-

-RRB-

Steps

towards

self-referential

learning

Technical

Report

Dept.

of

Comp

Sci.

University

of

Colorado

at

Boulder

Schmidhuber

J.

H.

-LRB-

-RRB-

A

neural

network

that

embeds

its

own

meta-levels

In

Proc

of

the

International

Conference

on

Neural

Networks

San

Francisco

IEEE

Schmidhuber

J.

H.

-LRB-

-RRB-

A

self-referential

weight

matrix

In

Proceedings

of

the

International

Conference

on

Artificial

Neural

Networks

Amsterdam

pages

-LCB-

Springer

Schmidhuber

J.

H.

-LRB-

-RRB-

Discovering

problem

solutions

with

low

Kolmogorov

complexity

and

high

generalization

capability

Technical

Report

Fakultat

fur

Informatik

Technische

Universitat

Munchen

Schwefel

H.

P.

-LRB-

-RRB-

Numerische

Optimierung

von

Computer-Modellen

Dissertation

Published

by

Birkhauser

Basel

Shannon

C.

E.

-LRB-

-RRB-

A

mathematical

theory

of

communication

-LRB-

parts

I

and

II

-RRB-

Bell

System

Technical

Journal

XXVII

-LCB-

Solomonoff

R.

-LRB-

-RRB-

A

formal

theory

of

inductive

inference

Part

I.

Information

and

Control

-LCB-

Solomonoff

R.

-LRB-

-RRB-

A

system

for

incremental

learning

based

on

algorithmic

probability

In

Pednault

E.

P.

D.

editor

The

Theory

and

Application

of

Minimal-Length

Encoding

-LRB-

Preprint

of

Symposium

papers

of

AAAI

Spring

Symposium

-RRB-

Sutton

R.

S.

-LRB-

-RRB-

Integrated

modeling

and

control

based

on

reinforcement

learning

and

dynamic

programming

In

Lippman

D.

S.

Moody

J.

E.

and

Touretzky

D.

S.

editors

Advances

in

Neural

Information

Processing

Systems

pages

-LCB-

San

Mateo

CA

Morgan

Kaufmann

Utgoff

P.

-LRB-

-RRB-

Shift

of

bias

for

inductive

concept

learning

In

Machine

Learning

volume

Morgan

Kaufmann

Los

Altos

CA

Watkins

C.

-LRB-

-RRB-

Learning

from

Delayed

Rewards

PhD

thesis

King

's

College

Williams

R.

J.

-LRB-

-RRB-

Simple

statistical

gradient-following

algorithms

for

connectionist

reinforcement

learning

Machine

Learning

-LCB-

Wolpert

D.

H.

-LRB-

-RRB-

On

overfitting

avoidance

as

bias

Technical

Report

SFI

TR

Santa

Fe

Institute

NM