Using

Recurrent

Neural

Networks

to

Learn

the

Structure

of

Interconnection

Mark

W.

Goudreau

Department

of

Computer

Science

University

of

Central

Florida

and

C.

Lee

Giles

NEC

Research

Institute

Inc.

and

UMIACS

University

of

Maryland

Published

in

Neural

Networks

-LRB-

-RRB-

p.

Copyright

Elsevier

paper

is

adapted

from

Chapter

of

M.

W.

Goudreau

Neural

Network

Applications

for

Interconnection

Networks

PhD

thesis

Princeton

University

Princeton

NJ

June

A

shortened

version

of

this

paper

appeared

in

M.

W.

Goudreau

and

C.

L.

Giles

Discovering

the

structure

of

a

self-routing

interconnection

network

with

a

recurrent

neural

network

in

Proceedings

of

the

International

Workshop

on

Applications

of

Neural

Networks

to

Telecommunications

-LRB-

J.

Alspector

R.

Goodman

and

T.

X.

Brown

eds

-RRB-

-LRB-

Hillsdale

NJ

-RRB-

pp

-LCB-

Lawrence

Erlbaum

Associates

Inc.

Using

Recurrent

Neural

Networks

to

Learn

the

Structure

of

Interconnection

Networksy

Mark

W.

Goudreau

Department

of

Computer

Science

University

of

Central

Florida

C.

Lee

Giles

NEC

Research

Institute

Inc.

and

UMIACS

University

of

Maryland

Abstract

A

modified

Recurrent

Neural

Network

-LRB-

RNN

-RRB-

is

used

to

learn

a

Self-Routing

Interconnection

Network

-LRB-

SRIN

-RRB-

from

a

set

of

routing

examples

The

RNN

is

modified

so

that

it

has

several

distinct

initial

states

This

is

equivalent

to

a

single

RNN

learning

multiple

different

synchronous

sequential

machines

We

define

such

a

sequential

machine

structure

as

augmented

and

show

that

a

SRIN

is

essentially

an

Augmented

Synchronous

Sequential

Machine

-LRB-

ASSM

-RRB-

As

an

example

we

learn

a

small

six-switch

SRIN

After

training

we

extract

the

network

's

internal

representation

of

the

ASSM

and

corresponding

SRIN

Introduction

The

use

of

Recurrent

Neural

Networks

-LRB-

RNNs

-RRB-

to

learn

Synchronous

Sequential

Machines

-LRB-

SSMs

-RRB-

from

examples

is

a

problem

which

has

been

studied

extensively

A

related

topic

that

to

the

authors

knowledge

has

not

been

studied

previously

is

the

use

of

RNNs

to

learn

SSMs

for

which

several

distinct

initial

states

are

possible

This

problem

is

interesting

because

it

maps

directly

into

the

problem

of

learning

the

structure

of

an

Interconnection

Network

-LRB-

IN

-RRB-

from

examples

Learning

an

IN

from

examples

is

an

unusual

approach

Traditionally

INs

have

been

designed

-LRB-

and

not

learned

-RRB-

based

on

several

criteria

including

speed

complexity

ease

of

route

calculation

and

fault

tolerance

Numerous

different

types

of

INs

have

been

proposed

A

yThis

paper

is

adapted

from

-LRB-

Goudreau

Chapter

-RRB-

A

shortened

version

of

this

paper

was

published

in

-LRB-

Goudreau

Giles

-RRB-

detailed

description

of

many

of

the

INs

that

have

been

applied

to

parallel

computing

can

be

found

in

Siegel

's

book

-LRB-

Siegel

-RRB-

In

this

paper

the

learning

of

Self-Routing

Interconnection

Networks

-LRB-

SRINs

-RRB-

is

discussed

SRINs

are

described

in

detail

in

Section

They

can

be

used

to

describe

many

commonly

used

INs

If

one

considers

a

parallel

computing

system

the

idea

is

that

the

processors

have

certain

communication

requirements

with

other

processors

and

certain

message

headers

-LRB-

also

described

in

Section

-RRB-

must

be

used

that

allow

the

message

to

pass

through

the

SRIN

and

reach

the

desired

destination

processor

The

message

headers

provide

routing

information

to

the

switches

in

the

SRIN

The

method

that

is

proposed

makes

use

of

a

second-order

Single-Layer

Recurrent

Neural

Network

-LRB-

SLRNN

-RRB-

to

learn

the

training

data

The

training

data

is

a

table

of

source

processors

message

headers

and

destination

processors

Once

the

training

data

has

been

learned

the

structure

of

the

SRIN

can

be

extracted

from

the

SLRNN

One

topic

that

is

related

to

the

learning

of

INs

was

presented

by

Hillis

-LRB-

Hillis

-RRB-

In

that

paper

Hillis

makes

use

of

simulated

evolution

to

construct

sorting

networks

It

should

be

also

be

mentioned

that

neural

networks

have

been

previously

used

for

interconnection

network

routing

for

example

see

-LRB-

Brown

Brown

Liu

Funabiki

et

al.

Funabiki

et

al.

Goudreau

Giles

Hakim

Meadows

Lee

Chang

Marrakchi

Troudet

Melsa

et

al.

Melsa

et

al.

Takefuji

Lee

Thomopoulos

et

al.

Troudet

Walters

-RRB-

However

of

these

methods

learned

the

structure

of

the

interconnection

networks

the

structure

of

the

interconnection

network

was

always

directly

mapped

into

the

neural

network

The

learning

of

interconnection

networks

is

a

new

idea

as

such

there

are

no

existing

applications

That

said

this

paper

can

be

viewed

as

an

attempt

to

look

at

interconnection

network

design

in

a

different

light

Rather

than

start

with

the

design

of

an

interconnection

network

and

have

the

structure

of

the

interconnection

network

determine

the

routes

it

is

possible

to

start

with

a

set

of

desired

routes

and

use

them

to

determine

the

structure

of

an

interconnection

network

The

possibility

that

this

technique

can

be

useful

has

been

made

more

likely

by

automata

rule

encoding

and

extraction

methods

recently

developed

for

recurrent

neural

networks

-LRB-

Andrews

et

al.

Giles

Omlin

Maclin

Shavlik

-RRB-

Self-Routing

Interconnection

Networks

In

this

section

we

describe

SRINs

The

purpose

of

a

SRIN

is

to

allow

a

set

of

processors

to

communicate

amongst

themselves

using

a

store-and-forward

methodology

For

store-and-forward

routing

a

message

travels

along

the

path

towards

its

destination

one

switch

at

a

time

A

switch

can

be

thought

of

as

a

simple

processor

eheader

body

end-symbol

Figure

A

message

for

a

SRIN

The

header

is

a

string

of

symbols

The

body

is

also

a

string

of

symbols

The

header

is

separated

from

the

body

by

the

end-symbol

ffl

nn

n

m

Figure

A

N

self-routing

switch

If

the

heading

symbol

is

a

ij

the

message

is

routed

from

input

to

output

nj

Messages

are

buffered

until

they

can

be

routed

Messages

are

routed

in

a

First-In

First-Out

-LRB-

FIFO

-RRB-

basis

that

accepts

a

message

and

then

routes

the

message

to

the

appropriate

output

line

Once

a

switch

has

sent

a

message

to

the

next

switch

it

is

free

to

be

used

to

route

a

different

message

A

SRIN

does

not

use

an

external

controller

to

route

messages

Rather

the

switches

in

a

SRIN

are

smart

they

examine

the

message

that

is

being

sent

and

decide

which

way

to

route

it

A

message

as

it

is

defined

in

this

paper

consists

of

two

parts

a

header

and

a

body

The

header

and

the

body

are

separated

by

an

end-symbol

which

will

be

denoted

by

ffl

A

schematic

diagram

of

a

message

is

shown

in

Figure

The

SRIN

uses

self-routing

switches

to

route

messages

A

self-routing

switch

with

M

inputs

and

N

outputs

will

be

called

a

M

N

self-routing

switch

A

drawing

of

a

N

self-routing

switch

is

shown

in

Figure

Figure

is

a

drawing

of

a

M

N

self-routing

switch

Intuitively

the

routing

works

in

the

following

manner

When

a

message

arrives

at

a

switch

the

switch

strips

off

the

first

symbol

of

the

message

header

and

examines

it

If

the

symbol

is

not

the

end-symbol

ffl

the

switch

sends

the

message

-LRB-

minus

the

header

symbol

that

it

just

examined

-RRB-

through

the

appropriate

output

port

If

the

nn

n

mm

m

Figure

An

M

N

self-routing

switch

Again

input

messages

are

routed

on

a

First-In

First-Out

-LRB-

FIFO

-RRB-

basis

symbol

is

the

end-symbol

then

the

message

should

be

given

to

the

processor

that

is

associated

with

the

switch

In

other

words

once

the

header

has

been

stripped

down

so

that

only

the

end-symbol

is

left

the

message

does

not

get

passed

through

the

SRIN

any

longer

Although

it

is

not

shown

in

Figures

and

it

must

be

remembered

that

there

is

a

connection

from

each

switch

to

its

associated

processor

This

can

be

thought

of

as

another

output

port

for

the

end-symbol

ffl

The

switches

work

in

a

First-In

First-Out

-LRB-

FIFO

-RRB-

manner

If

a

message

can

not

be

routed

immediately

it

is

buffered

until

it

can

be

routed

A

Formal

Description

of

a

Self-Routing

Interconnection

Network

We

will

now

present

a

more

formal

description

of

an

SRIN

The

SRIN

will

have

a

set

of

M

processors

P

pM

Each

processor

pj

will

be

associated

with

a

set

of

switches

Qj

Each

set

Qj

must

contain

at

least

one

switch

-LRB-

Otherwise

the

processor

would

have

no

way

to

communicate

with

the

other

processors

-RRB-

Note

that

not

all

of

the

switches

in

the

SRIN

need

to

be

associated

with

a

processor

Some

switches

in

an

SRIN

might

never

be

used

to

connect

to

a

processor

Such

switches

are

called

do

n't

care

switches

or

free

switches

A

message

can

be

routed

through

a

free

switch

but

a

free

switch

should

never

be

the

first

switch

nor

last

switch

in

a

route

to

do

so

would

imply

that

the

free

switch

is

associated

with

some

processor

For

the

sake

of

convenience

we

will

associate

some

processor

with

each

free

switch

even

though

such

an

association

is

meaningless

since

it

is

never

used

Now

the

SRIN

has

the

set

of

switches

Q

-LSB-

-LSB-

-LSB-

QM

The

processor

function

performs

the

mapping

Q

P

That

is

if

q

is

a

switch

then

-LRB-

q

-RRB-

is

the

processor

associated

with

that

switch

We

will

let

R

be

the

finite

input

alphabet

for

the

header

and

the

body

The

end

symbol

ffl

is

not

a

member

of

R

that

is

ffl

R

The

end-symbol

is

only

used

to

separate

the

header

from

the

body

One

typical

alphabet

would

be

R

In

general

however

the

magnitude

of

the

alphabet

can

be

greater

than

two

Since

most

computing

environments

are

binary

the

situation

becomes

more

complicated

when

the

magnitude

of

the

alphabet

is

greater

than

two

In

such

cases

the

members

of

the

alphabet

must

be

encoded

in

some

way

There

does

not

need

to

be

a

size

limitation

for

the

header

nor

the

body

In

a

binary

system

the

end-symbol

might

consist

of

a

string

of

zeros

and

ones

that

is

illegal

in

the

header

Alternatively

one

might

send

the

header

and

the

body

separately

in

which

case

the

position

of

the

end-symbol

will

be

understood

by

the

receiving

switch

Another

approach

would

be

to

designate

the

first

byte

of

the

header

to

represent

the

length

of

the

header

There

are

many

different

ways

to

implement

the

end-symbol

but

for

our

purposes

here

we

will

assume

the

end-symbol

is

just

a

symbol

that

can

be

transmitted

in

one

time

step

We

now

define

the

switch

transition

function

OE

which

performs

the

mapping

OE

Q

R

Q

If

q

is

a

switch

and

r

is

the

input

symbol

that

is

taken

from

the

front

of

the

header

then

OE

-LRB-

q

r

-RRB-

is

the

next

switch

that

the

message

will

be

sent

to

Finally

when

processor

pj

sends

a

message

it

starts

the

message

off

from

one

of

the

switches

in

the

set

Qj

Each

processor

will

have

a

switch

that

is

designated

for

this

purpose

We

define

the

switch

function

which

performs

the

mapping

P

Q

If

p

is

a

processor

then

-LRB-

p

-RRB-

is

the

switch

that

performs

the

first

stage

of

the

routing

for

any

messages

that

p

sends

We

will

call

the

switch

-LRB-

p

-RRB-

the

designated

switch

for

processor

p

The

SRIN

can

now

be

defined

formally

Definition

A

self-routing

interconnection

network

is

a

-LRB-

P

Q

R

OE

ffl

-RRB-

where

ffl

P

is

a

finite

nonempty

set

of

processors

ffl

Q

is

a

finite

nonempty

set

of

switches

ffl

R

is

a

finite

nonempty

set

of

input

symbols

ffl

OE

Q?R

Q

is

the

switch

transition

function

ffl

Q

P

is

the

processor

function

ffl

P

Q

is

the

switch

function

ffl

ffl

is

the

end-symbol

An

Example

of

a

Self-Routing

Interconnection

Network

Figure

shows

an

example

of

a

SRIN

This

SRIN

has

the

set

of

switches

Q

In

Figure

the

switches

are

shown

as

white

boxes

with

their

labels

in

the

upper

left

corner

The

outputs

labeled

and

are

on

the

right

side

of

each

switch

The

inputs

come

to

the

left

side

of

the

switch

A

switch

with

a

in

its

lower

left

corner

is

a

designated

switch

The

set

of

processors

is

P

In

Figure

the

processors

are

represented

by

shaded

areas

For

this

example

we

have

the

input

alphabet

R

Thus

each

switch

has

two

output

ports

In

practice

not

all

of

the

output

ports

need

to

be

connected

some

can

be

do

n't

cares

if

they

are

never

used

for

routing

The

number

of

input

ports

for

each

switch

can

be

zero

or

any

larger

integer

If

a

switch

has

zero

input

ports

it

must

be

a

designated

switch

or

it

will

have

no

purpose

in

the

SRIN

The

processor

function

is

shown

here

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

The

switch

transition

function

OE

is

shown

here

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

OE

-LRB-

-RRB-

-LRB-

-RRB-

Finally

the

switch

function

is

shown

here

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

-LRB-

-RRB-

Suppose

processor

has

data

to

send

to

processor

One

possible

way

to

send

the

data

there

is

to

use

the

header

The

message

starts

in

switch

-LRB-

-RRB-

The

q

q

q

q

q

q

p

p

p

p

Figure

A

sample

self-routing

interconnection

network

switch

strips

off

the

left-most

symbol

in

the

header

in

this

case

and

routes

the

message

to

OE

-LRB-

-RRB-

The

message

then

goes

to

switch

OE

-LRB-

-RRB-

and

at

last

to

switch

OE

-LRB-

-RRB-

At

this

point

the

header

has

been

spent

and

the

message

is

led

by

the

end-symbol

Switch

therefore

delivers

the

message

to

processor

-LRB-

-RRB-

Synchronous

Sequential

Machines

-LRB-

SMMs

-RRB-

In

this

section

we

discuss

the

relationship

between

SSMs

and

SRINs

SSMs

are

thoroughly

described

in

-LRB-

Hopcroft

Ullman

Kohavi

-RRB-

We

will

use

the

definition

of

SSMs

that

is

provided

in

-LRB-

Kohavi

-RRB-

-LRB-

A

finite

state

automata

is

a

restricted

case

of

a

sequential

machine

that

has

reduced

output

alphabet

of

accept

or

reject

of

input

sequences

-RRB-

Definition

A

synchronous

sequential

machine

is

a

quintuple

-LRB-

O

S

I

ffi

>=-RRB-

where

ffl

O

is

a

finite

nonempty

set

of

outputs

symbols

ffl

S

is

a

finite

nonempty

set

of

states

ffl

I

is

a

finite

nonempty

set

of

inputs

symbols

ffl

ffi

S

I

S

is

the

state

transition

function

ffl

S

O

is

the

output

function

From

Definitions

and

it

is

clear

that

SRINs

and

SSMs

are

very

similar

In

fact

it

only

takes

a

slight

expansion

of

the

definition

of

SSMs

to

make

them

directly

equivalent

to

SRINs

We

will

describe

how

SRINs

are

equivalent

to

Augmented

SSMs

-LRB-

ASSMs

-RRB-

which

will

be

defined

below

Let

each

processor

in

P

be

an

output

symbol

in

O.

Similarly

let

switch

in

Q

be

a

state

in

S

and

each

input

symbol

in

R

be

an

input

symbol

in

I

The

switch

transition

function

OE

becomes

the

state

transition

function

ffi

The

processor

function

becomes

the

output

function

Now

the

only

components

of

the

SRIN

that

are

not

equivalent

to

components

in

the

SSM

are

the

end-symbol

ffl

and

the

switch

function

The

ASSM

will

have

an

end-symbol

The

meaning

of

the

end-symbol

in

this

context

is

merely

that

the

input

string

has

reached

its

conclusion

and

the

ASSM

can

now

output

the

value

our

definition

is

for

a

Moore

machine

corresponding

to

the

input

string

The

ASSM

will

also

have

a

state

function

ae

The

state

function

ae

performs

the

mapping

ae

O

S

In

this

context

the

state

function

allows

for

some

set

of

initial

states

in

the

ASSM

Thus

each

input

string

that

is

to

be

entered

into

the

ASSM

must

have

an

output

symbol

associated

with

it

This

output

symbol

allows

the

ASSM

to

choose

the

correct

starting

state

The

ASSM

can

now

be

defined

formally

Definition

As

augmented

synchronous

sequential

machine

is

an

-LRB-

O

S

I

ffi

ae

<=-RRB-

where

ffl

O

is

a

finite

nonempty

set

of

outputs

ffl

S

is

a

finite

nonempty

set

of

states

ffl

I

is

a

finite

nonempty

set

of

inputs

ffl

ffi

S

I

S

is

the

state

transition

function

ffl

S

O

is

the

output

function

ffl

ae

O

S

is

the

state

function

ffl

is

the

end-symbol

It

is

now

clear

from

Definitions

and

that

SRINs

and

ASSMs

are

equivalent

Machine

Inference

Since

SRINs

and

ASSMs

are

equivalent

there

are

many

issues

that

have

been

explored

for

ASSMs

that

can

now

be

used

for

SRINs

For

example

just

as

one

can

minimize

the

size

of

an

ASSM

by

merging

equivalent

states

-LRB-

Kohavi

-RRB-

one

can

minimize

the

size

of

a

SRIN

by

merging

equivalent

switches

What

we

are

interested

in

is

the

inference

of

a

SRIN

from

examples

A

great

deal

of

work

has

been

done

on

the

problem

of

machine

inference

It

has

been

shown

that

in

the

worst

case

inferring

a

SSM

from

sparse

data

is

an

intractable

problem

-LRB-

Angluin

Gold

Kearns

Valiant

Pitt

Warmuth

-RRB-

Approaches

that

can

be

used

to

infer

SSMs

will

now

be

examined

Recurrent

Neural

Network

Approaches

The

literature

on

the

use

of

neural

networks

for

grammatical

inference

and

finitestate

machine

learning

is

now

well-established

-LRB-

Cleeremans

et

al.

Giles

et

al.

Giles

et

al.

Mozer

Bachrach

Pollack

Watrous

Kuhn

Zeng

et

al.

-RRB-

These

approaches

use

RNNs

to

represent

SSMs

For

the

work

done

in

this

paper

the

approach

described

in

-LRB-

Giles

et

al.

Giles

et

al.

-RRB-

will

be

used

-LRB-

see

Section

-RRB-

We

refer

readers

who

are

interested

in

the

details

to

those

references

In

Section

there

is

a

qualitative

explanation

of

the

RNN

approach

to

learning

SRINs

Until

recently

the

RNN

approach

for

SSM

inference

that

is

used

in

this

paper

had

only

been

possible

for

unknown

SSMs

with

a

small

number

of

states

-LRB-

approximately

-RRB-

It

should

be

pointed

out

that

the

limited

success

of

this

approaches

is

due

to

the

learning

algorithms

Generally

the

RNNs

have

rich

representational

capabilities

However

recent

work

has

shown

that

certain

types

of

large

SSMs

with

thousands

of

states

are

learnable

-LRB-

Clouse

et

al.

Giles

et

al.

-RRB-

Furthermore

the

performance

of

the

RNNs

can

sometimes

be

improved

by

using

hints

if

partial

information

about

the

structure

of

the

SSM

is

known

-LRB-

Giles

Omlin

-RRB-

Other

approaches

that

use

neural

networks

for

grammatical

inference

exist

that

will

not

be

used

in

this

paper

For

example

the

use

of

update

graphs

has

been

proposed

by

Rivest

and

Schapire

-LRB-

Rivest

Schapire

Rivest

Schapire

Schapire

-RRB-

An

update

graph

is

an

alternate

representation

of

a

SSM

that

can

be

much

smaller

than

the

SSM

for

certain

environments

that

often

arise

in

practice

Update

graphs

can

be

mapped

to

a

connectionist

system

that

can

learn

the

environment

from

examples

-LRB-

Mozer

Bachrach

Mozer

Bachrach

-RRB-

Traditional

Approaches

Other

methods

for

grammatical

inference

which

do

not

use

neural

networks

have

demonstrated

some

promising

results

In

fact

a

polynomial

time

algorithm

proposed

by

Trakhtenbrot

and

Barzdin

-LRB-

Trakhtenbrot

Barzdin

-RRB-

has

been

shown

to

be

able

to

infer

some

very

large

finite

automata

The

algorithm

produces

a

machine

that

is

consistent

with

a

sparsely

labeled

tree

but

the

machine

that

is

produced

is

not

necessarily

the

minimum

machine

that

is

consistent

with

the

data

Lang

-LRB-

Lang

-RRB-

performed

several

experiments

using

this

algorithm

for

random

finite

automata

with

states

and

transitions

Given

enough

training

examples

the

algorithm

was

almost

always

able

to

construct

a

machine

that

was

similar

to

the

correct

machine

source

header

destination

processor

processor

Table

An

example

of

training

data

for

a

SRIN

The

data

is

consistent

with

the

SRIN

in

Figure

Recurrent

Neural

Networks

to

Learn

Interconnec

tion

Networks

The

problem

that

we

are

trying

to

solve

is

posed

in

the

following

form

We

have

a

training

list

of

source

processor

header

and

destination

processor

combinations

that

must

be

implemented

by

a

SRIN

For

example

Table

contains

data

for

some

such

problem

The

data

in

Table

is

consistent

with

the

SRIN

in

Figure

We

must

infer

a

SRIN

that

can

accomplish

all

of

the

routings

described

in

the

training

list

Hopefully

the

SRIN

will

also

be

able

to

generalize

That

is

we

would

like

the

SRIN

to

perform

correct

routings

even

for

examples

that

are

not

on

the

training

list

In

Section

the

recurrent

neural

network

that

is

used

to

learn

the

interconnection

network

is

described

The

training

algorithm

is

also

discussed

Section

contains

a

training

example

and

explains

the

specific

encodings

used

Finally

extracting

the

SRIN

from

the

trained

recurrent

neural

network

is

described

in

Section

Recurrent

Neural

Network

with

Several

Distinct

Initial

States

The

structure

of

a

general

SLRNN

is

shown

in

Figure

There

are

M

inputs

lines

xM

The

value

of

input

xi

-LRB-

i

M

-RRB-

at

time

t

is

xti

There

is

a

single

layer

of

N

neurons

yN

The

output

value

of

neuron

yi

-LRB-

i

N

-RRB-

at

time

t

is

yti

At

each

time

step

these

output

values

are

stored

in

a

bank

of

latches

to

act

as

the

state

of

the

network

The

state

is

fed

back

as

an

input

to

the

layer

of

neurons

on

the

subsequent

time

step

In

general

all

of

the

state

values

can

be

considered

as

output

values

but

it

might

be

that

the

problem

domain

only

requires

some

number

K

output

values

where

K

N

In

this

case

only

the

first

K

neurons

are

considered

to

provide

output

values

although

all

N

neurons

are

Nt

Mt

N

yy

yy

y

y

xx

Figure

A

Single-Layer

Recurrent

Neural

Network

-LRB-

SLRNN

-RRB-

There

are

M

input

bits

N

state

bits

and

-LRB-

up

to

-RRB-

N

output

bits

The

bank

of

N

latches

is

shown

on

the

right

providing

state

values

For

a

second-order

SLRNN

neuron

j

and

input

k

have

a

combined

effect

on

neuron

i

that

is

quantified

by

the

parameter

wijk

which

is

called

a

weight

The

output

of

neuron

i

is

defined

by

the

following

equation

yti

g

-LRB-

NX

j

M

X

k

wijkyt

j

xtk

-RRB-

-LRB-

-RRB-

This

multiplication

and

summing

occurs

inside

the

neurons

shown

in

Figure

The

activation

function

g

-LRB-

x

-RRB-

is

the

sigmoid

function

shown

here

g

-LRB-

x

-RRB-

e?ax

-LRB-

-RRB-

The

second-order

SLRNN

is

used

to

infer

the

ASSM

that

is

equivalent

to

the

unknown

SRIN

Again

the

approach

used

in

-LRB-

Giles

et

al.

Giles

et

al.

-RRB-

will

be

used

here

The

SLRNN

will

learn

the

training

data

and

the

ASSM

will

be

extracted

from

the

SLRNN

The

training

algorithm

that

is

used

is

a

variation

of

the

Real-Time

RecurrentTraining

-LRB-

RTRL

-RRB-

algorithm

proposed

by

Williams

and

Zipser

-LRB-

Williams

Zipser

-RRB-

The

original

RTRL

algorithm

was

proposed

for

first-order

SLRNNs

but

the

version

used

here

is

for

second-order

SLRNNs

The

RTRL

algorithm

is

an

online

gradient-descent-based

algorithm

Other

recurrent

training

algorithms

could

be

used

for

this

application

e.g.

backpropagation

through

time

or

the

extended

Kalman

estimator

Recall

that

the

training

data

is

in

the

form

of

Table

Each

line

in

the

table

is

called

a

table

entry

In

order

to

use

the

SLRNN

it

is

necessary

to

encode

the

symbols

of

a

table

entry

into

binary

vectors

that

can

be

recognized

by

the

SLRNN

The

source

processor

defines

the

initial

state

vector

of

the

SLRNN

-LRB-

i.e.

the

values

of

for

i

N

-RRB-

Thus

each

source

processor

must

be

assigned

a

distinct

N-bit

binary

vector

The

input

symbols

in

the

header

-LRB-

along

with

the

end-symbol

-RRB-

correspond

to

the

input

vectors

of

the

SLRNN

-LRB-

i.e.

the

values

of

xti

for

i

M

-RRB-

Thus

each

input

symbol

-LRB-

and

the

end-symbol

-RRB-

must

be

assigned

a

distinct

M-bit

binary

vector

Note

that

the

inputs

to

the

SLRNN

change

over

time

with

the

binary

vector

of

the

first

input

symbol

applied

at

t

the

binary

vector

of

the

second

input

symbol

applied

at

t

etc.

After

the

binary

vectors

for

the

sequence

of

input

symbols

have

been

applied

the

binary

vector

for

the

end-symbol

is

applied

as

the

final

input

vector

At

this

point

it

is

possible

to

check

the

resulting

output

vector

of

the

SLRNN

-LRB-

i.e.

the

values

of

yTi

for

i

K

where

T

is

the

final

time

step

-RRB-

against

the

desired

result

The

desired

result

is

a

K-bit

binary

vector

that

represents

the

destination

processor

so

each

destination

processor

must

be

assigned

a

distinct

K-bit

binary

vector

Intuitively

the

input

vectors

of

the

SLRNN

represent

the

inputs

and

the

end-symbol

of

the

ASSM

-LRB-

and

therefore

the

input

symbols

and

the

end-symbol

of

the

SRIN

-RRB-

The

state

vectors

of

the

SLRNN

represent

the

states

of

the

ASSM

-LRB-

and

the

switches

of

the

SRIN

-RRB-

And

the

output

vectors

of

the

SLRNN

represent

the

outputs

of

the

the

ASSM

-LRB-

and

the

processors

of

the

SRIN

-RRB-

It

is

important

to

note

that

the

binary

vectors

that

are

chosen

to

represent

the

source

processors

the

input

symbols

the

end

symbol

and

the

destination

processors

are

arbitrary

However

we

will

use

simple

one-hot

encodings

for

all

of

the

necessary

binary

vectors

Recall

that

a

one-hot

code

is

a

code

for

which

each

symbol

is

represented

by

a

vector

that

has

one

element

equal

to

one

while

all

of

the

other

elements

are

equal

to

zero

This

structure

is

chosen

because

it

is

known

that

-LRB-

given

enough

neurons

-RRB-

a

solution

will

exist

to

map

the

SLRNN

to

the

desired

ASSM

-LRB-

Goudreau

et

al.

-RRB-

The

solution

that

is

known

to

exist

requires

the

use

of

one-hot

codes

for

the

states

and

the

inputs

The

representation

that

the

SLRNN

actually

learns

however

can

have

states

that

are

not

in

a

one-hot

code

The

SLRNN

might

construct

a

solution

that

is

different

from

the

one-hot

solution

Clearly

there

must

be

enough

neurons

to

represent

the

processors

-LRB-

outputs

-RRB-

with

a

one-hot

code

Therefore

the

number

of

neurons

must

at

least

be

equal

to

the

number

of

processors

For

the

one-hot

solution

to

exist

however

there

must

be

one

neuron

for

each

switch

as

well

Unfortunately

one

does

not

generally

know

the

number

of

switches

beforehand

It

is

necessary

to

estimate

the

number

of

switches

and

provide

at

least

that

many

neurons

This

is

one

of

the

weaknesses

that

is

common

to

many

neural

network

approaches

often

it

is

not

clear

what

size

neural

network

would

be

best

One

approach

is

to

start

with

as

many

switch

neurons

as

reasonably

possible

if

training

is

successful

then

reduce

the

number

of

neurons

using

a

destructive

heuristic

-LRB-

Giles

Omlin

-RRB-

Assume

that

the

values

for

i

K

constitute

the

binary

vector

that

represents

the

desired

output

vector

Training

occurs

for

this

table

entry

if

for

any

i

-LRB-

i

K

-RRB-

we

have

yTi

where

for

our

simulations

If

it

is

determined

that

training

should

occur

for

this

table

entry

we

use

the

learning

algorithm

to

incrementally

change

the

weights

in

the

direction

opposite

that

of

the

gradient

of

the

mean-square-error

cost

function

ET

ET

K

X

i

-LRB-

yTi

-RRB-

-LRB-

-RRB-

The

weights

of

the

SLRNN

will

therefore

potentially

be

updated

after

each

table

entry

is

examined

As

a

result

the

learning

algorithm

does

not

perform

true

gradient

descent

for

the

entire

table

of

data

rather

it

approximates

true

gradient

descent

The

table

of

training

data

is

cycled

through

repeatedly

until

of

the

table

entries

require

updating

of

the

weights

Once

the

entire

table

is

learned

the

structure

of

the

underlying

ASSM

is

extracted

from

the

SLRNN

This

procedure

is

described

in

Section

The

state

vectors

that

are

created

through

the

use

of

the

learning

algorithm

are

not

examined

during

training

but

they

are

examined

after

training

to

describe

the

structure

of

the

underlying

ASSM

As

mentioned

before

these

states

actually

represent

switches

in

the

SRIN

Since

the

neurons

have

a

continuous

activation

function

it

is

generally

necessary

to

use

some

partitioning

and

clustering

techniques

to

make

a

group

of

states

equivalent

After

this

is

done

an

ASSM

can

be

extracted

from

the

SLRNN

The

extracted

ASSM

might

not

be

minimal

but

in

this

case

standard

SSM

minimization

techniques

can

be

used

It

is

instructive

to

compare

learning

an

ASSM

with

learning

a

SSM

which

has

been

extensively

studied

by

the

neural

network

community

The

problem

of

inferring

an

ASSM

as

opposed

to

inferring

an

SSM

is

that

in

an

ASSM

there

will

generally

be

several

possible

initial

states

In

an

SSM

a

single

initial

state

is

generally

assumed

With

this

in

mind

the

reader

should

be

able

to

see

that

the

SLRNN

that

is

trying

to

learn

an

ASSM

will

in

fact

be

trying

to

learn

a

SSM

for

each

possible

initial

state

However

all

of

these

SSMs

will

have

the

same

structure

the

only

difference

is

that

the

initial

state

varies

Since

all

of

the

SSMs

have

the

same

structure

it

can

reasonably

be

hoped

that

the

SLRNN

will

try

to

merge

these

multiple

SSMs

into

a

single

ASSM

In

fact

this

turns

out

to

be

the

case

Rather

than

learning

several

different

SSMs

in

different

sections

of

the

state

space

the

SLRNN

will

take

advantage

of

the

identical

structures

of

the

SSMs

and

merge

them

A

Training

Example

Perhaps

the

easiest

way

to

describe

the

mapping

to

the

SLRNN

would

be

to

describe

the

structure

of

the

SLRNN

given

data

for

the

SRIN

from

Figure

We

will

choose

to

have

N

neurons

in

the

SLRNN

The

choice

of

N

is

somewhat

arbitrary

however

we

did

want

to

give

the

recurrent

network

enough

neurons

to

easily

learn

the

interconnection

network

Recall

that

we

do

not

know

the

structure

of

the

SRIN

and

are

trying

to

learn

it

from

routing

strings

Since

one-hot

codes

will

be

used

four

initial

states

vectors

are

necessary

corresponding

to

the

four

source

processors

that

are

present

in

the

data

table

In

reality

however

it

should

be

remembered

that

the

state

vectors

in

the

SLRNN

will

actually

correspond

to

switches

from

the

SRIN

rather

than

processors

Thus

when

we

choose

state

vectors

for

the

four

source

processors

we

are

really

choosing

state

vectors

for

their

respective

designated

switches

-LRB-

as

shown

in

Equation

-RRB-

Let

hn

m

be

a

vector

of

dimension

n

that

has

a

value

in

position

m

and

a

value

in

all

of

the

other

positions

These

are

the

initial

values

of

the

yi

vectors

For

example

-LSB-

-RSB-

T

When

the

source

processor

is

processor

-LRB-

which

is

equivalent

to

saying

that

the

designated

switch

is

see

Equation

-RRB-

the

initial

state

vector

will

be

Similarly

source

processor

will

make

the

initial

state

vector

source

processor

will

make

the

initial

state

vector

and

source

processor

will

make

the

initial

state

vector

Messages

always

start

at

these

source

processors

This

mapping

of

source

processors

to

initial

state

vectors

can

be

made

arbitrarily

There

is

no

reason

not

to

use

vector

for

example

The

only

reason

that

this

vector

was

not

used

was

because

of

the

way

the

training

data

was

generated

Now

the

input

symbols

and

as

well

as

the

end-symbol

ffl

can

be

mapped

to

input

vectors

There

are

three

symbols

that

must

be

mapped

to

input

vectors

so

we

will

have

three

vectors

of

length

three

for

this

purpose

Let

correspond

to

input

vector

correspond

to

input

vector

and

ffl

correspond

to

input

vector

Finally

the

processors

can

now

be

mapped

to

output

vectors

There

are

eight

neurons

in

the

SLRNN

but

only

the

outputs

of

four

of

them

are

needed

for

the

output

vectors

Therefore

the

output

values

will

only

be

taken

from

neurons

and

Destination

processor

will

correspond

to

output

vector

destination

processor

will

correspond

to

output

vector

destination

processor

will

correspond

to

output

vector

and

destination

processor

will

correspond

to

output

vector

In

summary

the

second-order

SLRNN

network

that

is

used

for

this

example

is

shown

in

Figure

and

has

N

neurons

and

M

input

bits

When

the

output

values

are

examined

only

the

first

four

neurons

are

used

-LRB-

K

-RRB-

Now

that

the

specifics

of

the

encoding

have

been

explained

we

can

use

the

training

data

to

train

the

SLRNN

The

data

table

contained

an

entry

for

every

header

up

to

length

-LRB-

including

the

end-symbol

-RRB-

for

every

processor

The

data

table

starts

with

strings

of

length

one

and

concludes

with

strings

of

length

The

SLRNN

does

not

try

to

learn

all

of

the

data

in

this

table

simultaneously

Rather

it

first

learns

the

first

lines

of

the

data

table

for

each

possible

starting

state

Then

the

resulting

SLRNN

is

checked

against

the

rest

of

the

data

table

to

see

how

it

generalizes

If

perfect

generalization

does

not

occur

the

SLRNN

adds

more

lines

to

its

training

data

Once

these

lines

are

learned

generalization

is

checked

again

This

process

is

repeated

until

all

of

the

lines

in

the

data

table

have

been

learned

This

heuristic

approach

which

involves

incremental

expansion

of

the

training

data

has

proven

to

be

quite

successful

in

practice

For

our

experiment

by

the

time

the

SLRNN

learned

the

first

lines

of

the

table

the

SLRNN

was

able

to

generalize

for

all

of

the

remaining

strings

in

the

table

The

full

table

had

lines

Extraction

of

the

Interconnection

Network

from

the

Trained

Neural

Network

It

was

mentioned

in

Section

that

this

problem

can

be

thought

of

as

the

problem

of

learning

several

separate

SSMs

in

this

case

four

Given

this

fact

one

can

examine

the

four

separate

SSMs

that

are

generated

from

the

four

different

initial

states

Table

shows

the

unminimized

SSM

with

initial

state

corresponding

to

switch

that

was

extracted

from

the

SLRNN

This

machine

shall

be

called

One

of

the

advantages

of

using

second-order

SLRNNs

is

the

ease

with

which

automata

can

be

extracted

from

the

the

trained

or

training

networks

However

first-order

SLRNNs

could

also

be

used

-LRB-

Manolios

Fanelli

Miller

Giles

-RRB-

Details

on

the

method

of

SSM

extraction

that

we

used

can

be

found

in

-LRB-

Giles

et

al.

Giles

et

al.

-RRB-

The

left

column

-LRB-

S

-RRB-

contains

the

number

of

each

state

State

in

Tables

to

corresponds

to

the

initial

state

The

next

column

-LRB-

O

-RRB-

contains

the

output

value

associated

with

that

state

The

following

two

columns

contain

the

next

state

given

input

zero

-LRB-

-RRB-

and

given

input

one

-LRB-

-RRB-

The

final

column

-LRB-

QR

-RRB-

contains

the

quantized

representation

for

the

state

in

the

SLRNN

It

should

be

kept

in

mind

that

the

SLRNN

actually

uses

real

valued

state

vectors

as

does

the

clustering

algorithm

that

was

used

for

SSM

extraction

The

SSM

extraction

algorithm

makes

use

of

a

real

valued

representative

vector

for

each

state

In

the

SLRNN

whenever

a

state

vector

is

close

to

a

representative

state

vector

it

is

assumed

that

the

two

vectors

implement

the

same

state

To

simplify

our

analysis

the

real

valued

representative

state

vectors

are

quantized

Any

value

greater

than

or

equal

to

in

a

representative

state

vector

is

set

to

after

quantization

while

any

value

less

than

is

set

to

Henceforth

when

state

vectors

are

mentioned

we

will

actually

be

talking

about

these

quantized

representative

state

vectors

Thus

S

O

QR

Table

Machine

the

unminimized

SSM

with

initial

state

corresponding

to

switch

Column

S

contains

the

state

Column

O

contains

the

output

Column

contains

the

next

state

given

input

Column

contains

the

next

state

given

input

Column

QR

contains

the

quantized

representation

for

the

state

in

the

SLRNN

in

general

the

state

vectors

in

Table

actually

represent

some

volume

of

the

state

space

The

unminimized

SSMs

given

designated

switches

and

are

shown

in

Tables

-LRB-

machine

-RRB-

-LRB-

machine

-RRB-

and

-LRB-

machine

-RRB-

respectively

Once

the

unminimized

SSMs

and

their

state

vector

representations

are

known

the

SSMs

can

be

minimized

and

merged

Table

contains

the

minimized

SSM

that

is

extracted

from

the

SLRNN

when

the

initial

state

vector

corresponds

to

designated

switch

Each

state

in

the

minimized

machine

can

be

associated

with

one

or

more

state

vectors

For

example

state

of

the

SSM

in

Table

is

associated

with

two

state

vectors

and

The

minimal

SSMs

with

initial

states

corresponding

to

designated

switches

and

are

shown

in

Tables

and

respectively

With

the

information

in

Tables

to

we

can

merge

the

SSMs

into

an

ASSM

States

in

the

SSMs

with

any

overlapping

vector

representations

are

assumed

to

be

equivalent

The

results

show

that

the

SLRNN

has

indeed

merged

the

SSMs

In

fact

if

we

ignore

the

quantized

representations

of

the

states

the

four

minimal

SSMs

are

equivalent

except

for

the

fact

that

they

have

different

initial

states

That

is

if

we

ignore

the

initial

states

the

four

SSMs

can

be

relabeled

so

that

they

are

identical

It

should

be

clear

how

Table

corresponds

to

the

SRIN

in

Figure

State

S

O

QR

10 10000101

11 12 11001101

11 12 11011101

13 00100101

Table

Machine

the

unminimized

SSM

with

initial

state

corresponding

to

switch

S

O

QR

10 00010101

11 12 11001101

13 00100101

Table

Machine

the

unminimized

SSM

with

initial

state

corresponding

to

switch

S

O

QR

10 11001101

11 01100101

11 12 10000101

13 01010100

10 11011101

14 00100101

13 01011100

11 00100001

13 01111100

Table

Machine

the

unminimized

SSM

with

initial

state

corresponding

to

switch

S

O

QR

Table

The

minimal

SSM

with

initial

state

corresponding

to

switch

This

SSM

is

equivalent

to

machine

in

Table

S

O

QR

Table

The

minimal

SSM

with

initial

state

corresponding

to

switch

This

SSM

is

equivalent

to

machine

in

Table

corresponds

to

switch

state

corresponds

to

switch

state

corresponds

to

switch

state

corresponds

to

switch

state

corresponds

to

switch

and

state

corresponds

to

switch

Simple

observation

will

show

that

the

other

three

SSMs

also

correspond

to

the

SRIN

in

Figure

Table

merges

this

information

to

show

how

the

SLRNN

represents

the

switches

of

Figure

State

vectors

are

shown

and

the

machines

that

utilized

them

are

presented

Examination

of

Table

shows

that

for

the

most

part

machines

and

make

use

of

state

vectors

that

are

approximately

equal

For

example

all

four

machines

use

the

state

region

to

represent

switch

from

Figure

Another

interesting

fact

is

that

the

state

vectors

that

represent

equivalent

states

tend

to

be

close

to

each

other

in

the

state

space

That

is

the

state

vectors

for

equivalent

states

tend

to

have

small

Hamming

distances

from

one

another

For

example

machine

uses

two

state

regions

to

represent

switch

and

This

fact

leads

one

to

believe

that

the

SLRNN

actually

uses

something

like

a

cohesive

sub-space

for

each

state

It

seems

likely

that

the

unminimized

SSMs

that

were

extracted

have

equivalent

states

due

to

the

nature

of

the

SSM

extraction

algorithm

that

is

used

If

other

clustering

approaches

were

used

-LRB-

Watrous

Kuhn

Zeng

et

al.

-RRB-

it

is

possible

that

minimal

SSMs

could

have

been

extracted

directly

from

the

SLRNN

Furthermore

values

such

as

and

will

have

different

values

after

quantization

although

they

are

in

fact

quite

close

in

the

state

space

Thus

two

state

vectors

that

are

quite

close

in

terms

of

Euclidean

distance

-LRB-

before

quantization

-RRB-

might

be

quite

far

from

each

other

after

quantization

This

is

S

O

QR

Table

The

minimal

SSM

with

initial

state

corresponding

to

switch

This

SSM

is

equivalent

to

machine

in

Table

S

O

QR

Table

The

minimal

SSM

with

initial

state

corresponding

to

switch

This

SSM

is

equivalent

to

machine

in

Table

switch

QR

machine

Table

How

the

SLRNN

represents

the

switches

from

the

SRIN

in

Figure

another

possible

explanation

for

the

fact

that

the

extracted

SSM

was

not

minimal

Another

observation

is

that

the

SLRNN

did

not

learn

anything

approaching

the

one-hot

solution

that

was

described

in

Section

In

fact

the

only

states

vectors

that

were

one-hot

after

quantization

were

the

initial

state

vectors

that

were

forced

upon

the

SLRNN

However

while

these

initial

state

vectors

were

not

returned

to

they

did

seem

to

give

the

SLRNN

a

bias

on

its

state

representation

Finally

the

fact

that

the

correct

SRIN

was

extracted

means

that

good

generalization

for

strings

longer

than

those

in

the

training

data

has

obviously

been

achieved

Conclusions

A

radical

approach

to

the

construction

of

interconnection

networks

has

been

presented

This

approach

uses

training

data

from

an

unknown

interconnection

network

to

teach

a

recurrent

neural

network

-LRB-

RNN

-RRB-

to

generate

an

interconnection

network

that

is

capable

of

routing

the

training

data

It

was

shown

that

this

problem

maps

directly

to

the

problem

of

learning

a

SSM

with

several

distinct

initial

states

The

proposed

approach

took

advantage

of

previous

work

on

the

use

of

RNNs

to

inference

synchronous

sequential

machines

-LRB-

SSMs

-RRB-

However

it

should

be

noted

that

the

relationship

between

interconnection

networks

and

SSMs

might

also

allow

for

some

non-neural

network

approaches

to

be

used

for

the

same

problem

It

seems

likely

that

such

methods

could

be

varied

slightly

to

accommodate

the

interconnection

network

inference

problem

just

as

the

RNN

method

for

SSM

inference

can

be

varied

slightly

to

perform

interconnection

network

inference

It

was

demonstrated

that

given

a

table

of

training

data

it

is

possible

to

use

a

secondorder

single-layer

RNN

to

generate

the

structure

of

an

interconnection

network

that

is

capable

of

routing

the

training

data

Furthermore

the

interconnection

network

that

is

generated

might

be

able

to

generalize

for

inputs

that

are

not

in

the

training

data

A

sample

problem

was

used

to

illustrate

the

methodology

It

is

an

open

question

as

to

whether

other

RNN

models

and/or

training

methods

can

outperform

these

results

This

work

clearly

pointed

out

the

need

for

further

research

into

the

use

of

RNNs

to

inference

larger

SSMs

To

date

RNNs

have

had

limited

success

for

large

problems

in

grammatical

inference

but

some

recent

results

are

promising

-LRB-

Giles

et

al.

Clouse

et

al.

-RRB-

The

concept

of

learning

interconnection

networks

is

an

unusual

one

for

the

interconnection

network

community

It

remains

to

be

seen

whether

such

learning

approaches

will

become

a

useful

method

for

interconnection

network

design

or

analysis

Acknowledgement

The

authors

would

like

to

acknowledge

useful

discussions

with

Sanjeev

Kulkarni

and

Cliff

B.

Miller

The

authors

would

also

like

to

thank

an

anonymous

reviewer

whose

comments

led

to

several

improvements

in

the

paper

References

Andrews

R.

Diederich

J.

Tickle

A.

-LRB-

-RRB-

A

survey

and

critique

of

techniques

for

extracting

rules

from

trained

artificial

neural

networks

Technical

Report

Neurocomputing

Research

Centre

Queensland

University

of

Technology

Brisbane

Australia

Angluin

D.

-LRB-

-RRB-

On

the

complexity

of

minimum

inference

of

regular

sets

Information

and

Control

-LCB-

Brown

T.

X.

-LRB-

-RRB-

Neural

networks

for

switching

IEEE

Communications

Magazine

-LRB-

-RRB-

-LCB-

Brown

T.

X.

Liu

K.-H

-LRB-

-RRB-

Neural

network

design

of

a

Banyan

network

controller

IEEE

Journal

on

Selected

Areas

of

Communication

-LRB-

-RRB-

-LCB-

Cleeremans

A.

Servan-Schreiber

D.

McClelland

J.

L.

-LRB-

-RRB-

Finite

state

automata

and

simple

recurrent

networks

Neural

Computation

-LCB-

Clouse

D.

Giles

C.

Horne

B.

Cottrell

G.

-LRB-

-RRB-

Learning

large

debruijn

automata

with

feed-forward

neural

networks

Technical

Report

Computer

Science

and

Engineering

University

of

California

at

San

Diego

La

Jolla

CA

Funabiki

N.

Takefuji

Y.

Lee

K.

C.

-LRB-

-RRB-

A

neural

network

model

for

traffic

controls

in

multistage

interconnection

networks

In

Proceedings

of

the

International

Joint

Conference

on

Neural

Networks

-LRB-

pp

-RRB-

Funabiki

N.

Takefuji

Y.

Lee

K.

C.

-LRB-

-RRB-

Comparisons

of

seven

neural

network

models

on

traffic

control

problems

in

multistage

interconnection

networks

IEEE

Transactions

on

Computers

-LRB-

-RRB-

-LCB-

Giles

C.

Horne

B.

Lin

T.

-LRB-

-RRB-

Learning

a

class

of

large

finite

state

machines

with

a

recurrent

neural

network

Technical

Report

and

CS

Institute

for

Advanced

Computer

Studies

University

of

Maryland

College

Park

Maryland

Giles

C.

Omlin

C.

-LRB-

-RRB-

Extraction

insertion

and

refinement

of

symbolic

rules

in

dynamically-driven

recurrent

neural

networks

Connection

Science

-LRB-

-RRB-

-LCB-

Special

Issue

on

Architectures

for

Integrating

Symbolic

and

Neural

Processes

Giles

C.

Omlin

C.

-LRB-

-RRB-

Pruning

recurrent

neural

networks

for

improved

generalization

performance

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

Giles

C.

L.

Miller

C.

B.

Chen

D.

Chen

H.

H.

Sun

G.

Z.

Lee

Y.

C.

-LRB-

-RRB-

Learning

and

extracting

finite

state

automata

with

second-order

recurrent

neural

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

Giles

C.

L.

Miller

C.

B.

Chen

D.

Sun

G.

Z.

Chen

H.

H.

Lee

Y.

C.

-LRB-

-RRB-

Extracting

and

learning

an

unknown

grammar

with

recurrent

neural

networks

In

Moody

J.

Hanson

S.

Lippmann

R.

-LRB-

Eds

-RRB-

Advances

in

Neural

Information

Processing

Systems

-LRB-

pp

-LCB-

-RRB-

San

Mateo

CA

Morgan

Kaufmann

Publishers

Gold

E.

M.

-LRB-

-RRB-

Complexity

of

automaton

identification

from

given

data

Information

and

Control

-LCB-

Goudreau

M.

Giles

C.

-LRB-

-RRB-

Discovering

the

structure

of

a

self-routing

interconnection

network

with

a

recurrent

neural

network

In

Alspector

J.

Goodman

R.

Brown

T.

-LRB-

Eds

-RRB-

International

Workshop

on

Applications

of

Neural

Networks

to

Telecommunications

-LRB-

pp.

-LCB-

-RRB-

Hillsdate

NJ

Lawrence

Erlbaum

Goudreau

M.

Giles

C.

Chakradhar

S.

Chen

D.

-LRB-

-RRB-

First-order

vs.

secondorder

single

layer

recurrent

neural

networks

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

Goudreau

M.

W.

-LRB-

-RRB-

Neural

Network

Applications

for

Interconnection

Networks

PhD

thesis

Princeton

University

Princeton

NJ

Goudreau

M.

W.

Giles

C.

L.

-LRB-

-RRB-

Routing

in

random

multistage

interconnection

networks

Comparing

exhaustive

search

greedy

and

neural

network

approaches

International

Journal

of

Neural

Systems

-LRB-

-RRB-

-LCB-

Hakim

N.

Z.

Meadows

H.

E.

-LRB-

-RRB-

A

neural

network

approach

to

the

setup

of

the

Benes

switch

In

Infocom

-LRB-

pp.

-LCB-

-RRB-

Hillis

W.

D.

-LRB-

-RRB-

Co-evolving

parasites

improve

simulated

evolution

as

an

optimization

procedure

Physics

D

-LCB-

Hopcroft

J.

E.

Ullman

J.

D.

-LRB-

-RRB-

Introduction

to

Automata

Theory

Languages

and

Computation

Reading

MA

Addison-Wesley

Publishing

Company

Inc.

Kearns

M.

Valiant

L.

-LRB-

-RRB-

Cryptographic

limitations

on

learning

boolean

formulae

and

finite

automata

In

Proceedings

of

the

Annual

ACM

Symposium

on

Theory

of

Computing

ACM

Press

Kohavi

Z.

-LRB-

-RRB-

Switching

and

Finite

Automata

Theory

-LRB-

second

ed

-RRB-

New

York

NY

McGraw-Hill

Inc.

Lang

K.

-LRB-

-RRB-

Random

DFA

's

can

be

approximately

learned

from

sparse

uniform

examples

In

Proceedings

of

the

Fifth

ACM

Workshop

on

Computational

Learning

Theory

-LRB-

pp.

-LCB-

-RRB-

ACM

Press

Lee

S.-L

Chang

S.

-LRB-

-RRB-

Neural

networks

for

routing

of

communication

networks

with

unreliable

components

IEEE

Transactions

on

Neural

Networks

-LRB-

-RRB-

-LCB-

Maclin

R.

Shavlik

J.

-LRB-

-RRB-

Using

knowledge-based

neural

networks

to

improve

algorithms

Refining

the

Chou-Fasman

algorithm

for

protein

folding

Machine

Learning

-LCB-

Manolios

P.

Fanelli

R.

-LRB-

-RRB-

First

order

recurrent

neural

networks

and

deterministic

finite

state

automata

Neural

Computation

-LRB-

-RRB-

-LCB-

Marrakchi

A.

M.

Troudet

T.

-LRB-

-RRB-

A

neural

net

arbitrator

for

large

crossbar

packet-switches

IEEE

Transactions

on

Circuits

and

Systems

-LRB-

-RRB-

-LCB-

Melsa

P.

J.

W.

Kenney

J.

B.

Rohrs

C.

E.

-LRB-

-RRB-

A

neural

network

solution

for

call

routing

with

preferential

call

placement

In

Proceedings

of

the

Global

Telecommunications

Conference

-LRB-

pp

-LCB-

-RRB-

Melsa

P.

J.

W.

Kenney

J.

B.

Rohrs

C.

E.

-LRB-

-RRB-

A

neural

network

solution

for

routing

in

three

stage

interconnection

networks

In

Proceedings

of

the

International

Symposium

on

Circuits

and

Systems

-LRB-

pp.

-LCB-

-RRB-

Miller

C.

Giles

C.

-LRB-

-RRB-

Experimental

comparison

of

the

effect

of

order

in

recurrent

neural

networks

International

Journal

of

Pattern

Recognition

and

Artificial

Intelligence

-LRB-

-RRB-

-LCB-

Special

Issue

on

Neural

Networks

and

Pattern

Recognition

editors

I.

Guyon

P.S.P.

Wang

Mozer

M.

C.

Bachrach

J.

-LRB-

-RRB-

Discovering

the

structure

of

a

reactive

environment

by

exploration

Neural

Computation

-LRB-

-RRB-

-LCB-

Mozer

M.

C.

Bachrach

J.

-LRB-

-RRB-

SLUG

A

connectionist

architecture

for

inferring

the

structure

of

finite-state

environments

Machine

Learning

-LRB-

-RRB-

-LCB-

Pitt

L.

Warmuth

M.

-LRB-

-RRB-

The

minimum

DFA

consistency

problem

can

not

be

approximated

within

any

polynomial

In

Proceedings

of

the

Annual

ACM

Symposium

on

Theory

of

Computing

ACM

Press

Pollack

J.

B.

-LRB-

-RRB-

The

induction

of

dynamical

recognizers

Machine

Learning

-LRB-

-RRB-

-LCB-

Rivest

R.

L.

Schapire

R.

E.

-LRB-

-RRB-

Diversity-based

inference

of

finite

automata

In

Proceedings

of

the

Twenty-Eighth

Annual

Symposium

on

Foundations

of

Computer

Science

-LRB-

pp.

-LCB-

-RRB-

Rivest

R.

L.

Schapire

R.

E.

-LRB-

-RRB-

A

new

approach

to

unsupervised

learning

in

deterministic

environments

In

Langley

P.

-LRB-

Ed

-RRB-

Proceedings

of

the

Fourth

International

Workshop

on

Machine

Learning

Schapire

R.

E.

-LRB-

-RRB-

Diversity-based

inference

of

finite

automata

Master

's

thesis

Massachusetts

Institute

of

Technology

Cambridge

MA

Siegel

H.

J.

-LRB-

-RRB-

Interconnection

Networks

for

Large

Scale

Parallel

Processing

New

York

McGraw-Hill

Takefuji

Y.

Lee

K.

C.

-LRB-

-RRB-

An

artificial

hysteresis

binary

neuron

A

model

suppressing

the

oscillatory

behavior

of

neural

dynamics

Biological

Cybernetics

-LCB-

Thomopoulos

S.

C.

A.

Zhang

L.

Wann

C.

D.

-LRB-

-RRB-

Neural

network

implementation

of

the

shortest

path

algorithm

for

traffic

routing

in

communication

networks

In

Proceedings

of

the

International

Joint

Conference

on

Neural

Networks

-LRB-

pp.

-LCB-

-RRB-

Singapore

Trakhtenbrot

B.

Barzdin

Y.

-LRB-

-RRB-

Finite

Automata

Behavior

and

Synthesis

Amsterdam

North-Holland

Publishing

Company

Troudet

T.

P.

Walters

S.

M.

-LRB-

-RRB-

Neural

network

architecture

for

crossbar

switch

control

IEEE

Transactions

on

Circuits

and

Systems

-LRB-

-RRB-

-LCB-

Watrous

R.

L.

Kuhn

G.

M.

-LRB-

-RRB-

Induction

of

finite-state

languages

using

second-order

recurrent

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

Williams

R.

J.

Zipser

D.

-LRB-

-RRB-

A

learning

algorithm

for

continually

running

fully

recurrent

neural

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

Zeng

Z.

Goodman

R.

M.

Smyth

P.

-LRB-

-RRB-

Learning

finite

state

machines

with

self-clustering

recurrent

networks

Neural

Computation

-LRB-

-RRB-

-LCB-

interconnection networks
recurrent networks
sequential machines

finite-state automata
knowledge extraction
real-time training