OEFAI Reports 1990 1 OEFAI-TR-90-1 Extended version of a paper that appeared in Proc . European Working Session on Learning , EWSL-89 , Montpellier , France , 1989 , Pitman . Incremental Knowledge-Intensive Learning : A Case Study Based on an Extension to Bergadano & Giordana 's Integrated Learning Strategy Gerhard Widmer Bergadano & Giordana 's learning algorithm is an attractive method for knowledge-intensive learning in domains with background theories that are both incomplete and inconsistent . The paper presents an algorithm that extends this method by enabling it to learn incrementally . The algorithm has been compared to three other incremental variants of the Bergadano & Giordana method and has been applied to two test domains . Preliminary performance data show that the algorithm behaves quite well and still retains most of the advantages of the original -LRB- non-incremental -RRB- method . In the appendix , detailed results of some recent experiments are presented ; they provide further evidence for the utility and robustness of the incremental algorithm . OEFAI-TR-90-2 Metastructures as a Basis for the Implementation of Constraint Logic Programming Techniques Christian Holzbaur The aim of this paper is to show that a simple extension to ordinary unification is sufficient to implement a variety of constraint programming techniques . The nonstandard part of the extended unification algorithm is formulated in PROLOG as opposed to most of the current implementations of extensions - which are either emulated by metainterpreters or are coded in the implementation language of the corresponding PROLOG system . The proposed extension is conservative in the sense that , given the hook into unification is installed properly , it entails virtually no costs to programs that do n't use the extension . The extension does not disable standard optimization techniques like clause indexing and last call optimization . A definition of a metainterpreter is presented to be used as a specification of the extension . Unfortunately it can not be used directly or folded into application programs , because it rests so deep in the bowels of the computation and has to be applied to each and every predicate . As the proposed implementation provides just a hook into unification , the efforts to provide it should be modest on a variety of PROLOG implementations . The extensions to syntactical unification are specified in the language the user is already familiar with . The implementation of fully general nondeterministic unification and/or the implementation of CLP solvers should become more portable , more reliable and easier to debug and maintain . Keywords : Constraints , Unification , Metaprogramming . OEFAI-TR-90-3 International Journal of Man-Machine Studies 35 -LRB- 3 -RRB- , pp. 329-362 , 1991 . Also in Readings in Model-based Diagnosis -LRB- W.C. Hamscher , L. Console , J. de Kleer , Eds . -RRB- , pp. 354-372 , Morgan Kaufmann , San Mateo , CA , 1992 . Hierarchical Model-Based Diagnosis Igor Mozeti < = c Model-based reasoning about a system requires an explicit representation of the system 's components and their connections . Diagnosing such a system consists of locating those components whose abnormal behavior accounts for the faulty system behavior . In order to increase the efficiency of model-based diagnosis , we propose a model representation at several levels of detail , and define three refinement -LRB- abstraction -RRB- operators . We specify formal conditions that have to be satisfied by the hierarchical representation , and emphasize that the multi-level scheme is independent of any particular single-level model representation . The hierarchical diagnostic algorithm which we define turns out to be very general . We show that it emulates the bisection method , and can be used for hierarchical constraint satisfaction . We apply the hierarchical modeling principle and diagnostic algorithm to a medium-scale medical problem . The performance of a four-level qualitative model of the heart is compared to other representations in terms of diagnostic efficiency and space requirements . The hierarchical model does not reach the time/space performance of dedicated diagnostic rules , but it speeds up the diagnostic efficiency of a one-level model for a factor of 20 . OEFAI-TR-90-4 Proc . Automatic Generation of Approximations and Abstractions , AAAI-90 Workshop , pp. 64-75 , Boston , July 30 , 1990 . Abstractions in Model-Based Diagnosis Igor Mozeti < = c Most of the research on abstractions with a sound formal basis was applied to theorem proving and planning , and only a few results which show some computational advantage of abstractions are known . The paper presents an application of abstractions to model-based reasoning , with the goal to improve the efficiency of diagnosis . Model-based reasoning about a system requires an explicit representation -LRB- a model -RRB- of the system 's components and their connections . Diagnosing such a system consists of finding abnormal states of components , so that the faulty system behavior is accounted for . Our approach is not restricted to qualitative models , but requires an explicit fault model . We define three abstraction operators which map a detailed , complex model to an abstract , simpler one . A formal consistency condition which must be satisfied by any pair of adjacent mod - OEFAI Reports 1990 2 els in a hierarchy is specified . We define a diagnostic algorithm and prove its correctness and completeness . Abstractions and the algorithm were applied to a complex medical problem : ECG interpretation based on the model of the heart 's electrical activity . Results show that hierarchical diagnosis has logarithmic time complexity when compared to a one-level diagnosis | specifically , a speedup of a factor of 20 was achieved . OEFAI-TR-90-5 Artificial Intelligence in Medicine 3 -LRB- 1 -RRB- , pp. 21-39 , 1991 . Utilizing Detailed Anatomical Knowledge for Hypotheses Formation and Hypotheses Testing in Rheumatological Decision Support Werner Horn Diagnostic decisions in rheumatology are based to a large extent on a good understanding of the anatomy of the human body . A decision support system for rheumatology has to represent this fundamental anatomical knowledge in order to be able to reason about causal relationships between disturbances affecting the musculoskeletal system . We have built a knowledge-based system incorporating a detailed representation of the anatomy . This yields two main advantages : 1 -RRB- it enabled us to build disease descriptions of generic format . Instantiation automatically constructs specific disease descriptions by filling in the anatomical details which describe the situation of the patient ; 2 -RRB- the system provides a user interface showing all the anatomical details within the context of the patient 's problem . This is essential for the intended field of application , namely , primary medical care . This paper concentrates on the usage of the anatomical knowledge during hypotheses formation and during hypotheses testing . Keywords : Anatomical Knowledge , Causal Reasoning , Deep Knowledge , Rheumatology , Diagnostic Decision Support , Medical Expert System . OEFAI-TR-90-6 Artificial Intelligence in Medicine 2 -LRB- 2 -RRB- , pp. 67-83 , 1990 . Diagnostic Efficiency of Deep and Surface Knowledge in KARDIO Igor Mozeti < = c The KARDIO system deals with the problem of diagnosing cardiac arrhythmias from symbolic descriptions of electrocardiograms . The system incorporates a qualitative model which simulates the electrical activity of the heart . In the paper we outline two methods for an efficient application of a simulation model to diagnosis . First , through abstractions and refinements , the model is represented at several levels of detail . Second , the model is ` compiled ' into surface diagnostic rules . Through simulation , a relational table is generated and subsequently compressed into efficient diagnostic rules by inductive learning . A novel contribution to the KAR - DIO , presented here , includes a comparison of diagnostic efficiency and space complexity of four types of knowledge : a simulation model of the heart , a hierarchical four-level model , a relational table , and compressed diagnostic rules . Keywords : Qualitative Modeling , Diagnosis , Abstractions , Machine Learning . OEFAI-TR-90-7 Proc . 13th Intl. . Conf . on Computational Linguistics , COLING-90 , Univ. . of Helsinki , Finland , 1990 . Integrating Stress and Intonation into a Conceptto-Speech System Georg Dorffner , Ernst Buchberger , Markus Kommenda The paper deals with the integration of intonation algorithms into a concept-to-speech system for German . The algorithm for computing the stress hierarchy of a sentence introduced by Kiparski 's `` Ueber den deutschen Akzent '' -LRB- in : Studia Grammatica VII , Berlin , 3rd ed. , 1973 -RRB- and the theory of syntactic grouping for intonation patterns developed by Bierwisch -LRB- `` Regeln fuer die Intonation deutscher Saetze '' , in : Studia Grammatica VII , Berlin , 3rd ed. , 1973 -RRB- have been studied extensively , but they have never been implemented in a concept-tospeech system like the one presented here . We describe the back end of this concept-to-speech system : The surface generator transfers a hierarchical dependency structure of a sentence into a phoneme string by traversing it in a recursive-descent manner . Surface structures unfold while generation proceeds , which means that at no point of the process does the full syntactic tree structure exist . As they depend on syntactic features , both the indices introduced by the Kiparski -LRB- degrees of stress -RRB- and the Bierwisch -LRB- indexed border markers -RRB- formalism have to be inserted by the generator . This implies some changes to the original algorithms , which are demonstrated in this paper . The generator has been tested in the domain of an expert system that helps to debug electronic circuits . The synthesized utterances of the test domain show significant improvements over monotonous forms of speech produced by systems not making use of intonation information . Keywords : Natural Language Generation , Speech Synthesis , Intonation . OEFAI-TR-90-8 Implementation of Constraint Based Inference Mechanisms through Semantic Unification Christian Holzbaur We present an extension to unification as a general means to smoothly integrate a variety of control and constraint-based inference mechanisms into PROLOG : freeze , forward checking , CLP -LRB- B -RRB- , CLP -LRB- R -RRB- , ... The proposed extension entails virtually no overhead to programs which do not use it , provided the hook into unification is installed properly . The extension does not OEFAI Reports 1990 3 disable the standard optimization techniques like clause indexing and last call optimization . A distinguishing feature of our approach is that extended theories themselves are implemented in PROLOG - in contrast to the majority of current implementations in C . The advantages are obvious - the time needed for implementation and modification is reduced , and the resulting programs are in a high-level , transparent language . On the other hand , the efficiency reduction is much lower then expected . We outline our implementation of the extended unification in a standard PROLOG interpreter , and an implementation of two extended theories : forward checking and CLP -LRB- R -RRB- . Our experiments with a partial implementation of CLP -LRB- R -RRB- indicate that a modern compiler with decent garbage collection would achieve almost the same efficiency as a dedicated implementation in C. OEFAI-TR-90-9 Integration of Extended Unification into a Prolog Interpreter Christian Holzbaur This paper reports on the practical experiences collected during the implementation of constraint logic programming techniques via metaterms and extended unification . During the actual implementation of extended unification in a concrete PROLOG implementation some minor refinements to a specification proposed earlier -LRB- Holzbaur 1990 -RRB- had to be taken care of . Further , a convention related to the interpretation of the meaning of metaterms has been moved into the specification . This paper is also thought as a supplement to the C-Prolog manual , describing the additional predicates and conventions for the implementation of extended unification . OEFAI-TR-90-10 Proc . First Intl. . Workshop on Principles of Diagnosis , pp. 102-111 , Stanford University , Palo Alto , CA , July 23-25 , 1990 . Reduction of Diagnostic Complexity through Model Abstractions Igor Mozeti < = c The paper presents an application of abstractions to model-based reasoning with the goal to improve the efficiency of diagnosis . We define three abstraction operators which map a detailed , complex model to an abstract , simpler one . Our approach is not restricted to discrete , qualitative models . We define a provably correct and complete diagnostic algorithm and analyse its complexity . Abstractions and the algorithm were applied to a complex medical problem : ECG interpretation based on the model of the heart 's electrical activity . Results show that hierarchical diagnosis has logarithmic time complexity when compared to a one-level diagnosis | in particular , a speedup of a factor of 20 was achieved . OEFAI-TR-90-11 Realization of Forward Checking in Logic Programming via Extended Unification Christian Holzbaur Forward checking is one of the most promising a priori pruning techniques for Constraint Satisfaction Problems -LRB- Mackworth 1977 , Mackworth and Freuder 1985 -RRB- . In the past some schemes for the incorporation of forward checking and other consistency techniques over finite domains into Prolog have been introduced in -LRB- Hentenryck 1989 -RRB- . This work resulted in the implementation of a new prototype Prolog interpreter . In this paper we present an implementation of forward checking in Prolog . This implementation is based on a very general scheme for the incorporation of semantic unification into Prolog -LRB- Holzbaur 1990 , Neumerkel 1990 -RRB- . Constraint Satisfaction Problems can be formulated declaratively in Prolog . The problem with the standard backtracking evaluation strategy however is that it does not execute this specification efficiently . Therefore , the Prolog implementation of forward checking serves at least two purposes : the set of effectively executable declarative specifications in Prolog is enlarged ; the choice of Prolog as implementation language results in more declarative , reliable , maintainable , accessible , and extendible code . OEFAI-TR-90-12 A Qualitative Model of the Perception of Melodies and Harmonizations Gerhard Widmer The paper presents a qualitative model that identifies the most important factors underlying the perception of harmonized melodies . Relationships among these factors are described via qualitative dependency relations . The model is stated in a general , declarative , purpose-free form . The objective of this research is the development of a general computational model of music perception , which will allow a computer to ` understand ' music in a similar way as experienced listeners understand , i.e. , perceive it . This will enable the computer to reason about music , to communicate its deliberations to musicians in intelligible , ` common-sense ' terms , and to learn musical concepts effectively . This paper describes the structure of the qualitative model , discusses the underlying assumptions , and demonstrates the usefulness of the model by identifying the types of inference supported by the model . Keywords : Music , Artificial Intelligence , Qualitative Modelling , Cognitive Modelling , Plausible Reasoning , Learning . OEFAI Reports 1990 4 OEFAI-TR-90-13 Using a Qualitative Perception Model to Learn Harmonization Rules for Melodies Gerhard Widmer The paper presents a learning strategy and an implemented computer program that can learn to harmonize -LRB- i.e. , place chord symbols underneath -RRB- given melodies . The system learns from specific harmonization examples provided by a teacher , but also brings its own general knowledge of music to bear on this task . The musical knowledge is a general qualitative model of the perception of harmonized melodies . It is meant to capture roughly what we believe ordinary listeners ` know ' about tonal music . The system presented here demonstrates the beneficial effect of general musical knowledge for learning systems . The learning scenario and algorithm are described , and an extended example of the system at work is presented . In the appendix , an experiment with the system is described . Keywords : Music , Artificial Intelligence , Learning , Qualitative Modelling , Plausible Reasoning . OEFAI-TR-90-14 Art/ificial Intelligence : A Short Bibliography on AI and the Arts Richard K. Zach , Gerhard Widmer , Robert Trappl This report is the first -LRB- printed -RRB- result of a seminar entitled \ Art/ificial Intelligence : Die Beziehung zwischen Kunst und AI -LRB- The relation between Art and AI -RRB- '' , held in the spring term of 1990 as a joint endeavour by the Department of Medical Cybernetics and Artificial Intelligence of the University of Vienna , and the Department of Visual Media of the Academy of Applied Arts , Vienna . The bibliography presented in this report lists publications that are situated in the intersection of Artificial Intelligence and Art . They are very broadly grouped into four classes : 1 -RRB- papers and books dealing with general considerations and ideas for combining Art and Artificial Intelligence ; 2 -RRB- publications describing research in Artificial Intelligence and Music -LRB- this is the area that has seen the most interest so far -RRB- ; 3 -RRB- a somewhat inhomogeneous collection centering around literature and the performing arts ; this includes such endeavours as poetry , theater and drama , dance , etc. ; 4 -RRB- and finally , a substantial number of publications dealing with the use of Artificial Intelligence methods in the visual arts . Some of the references include short descriptions or abstracts . OEFAI-TR-90-15 To appear in CSLI Lecture Notes , Stanford , CA , Proc . of the Workshop on Semantic-Pragmatic Processing of Plurals and Quantors in NLP at GWAI-90 . A Framework for Treating Non-singular Terms in a Natural Language Consulting System John Matiasek , Wolfgang Heinz For use in an NLU system the domain of individuals has to be structured to encompass concepts like plurality or group formation . We present an analysis of non-singular terms in the framework of situation semantics . Our treatment also covers contextual aspects and accounts for phenomena linking groups and their members . The semantics is coupled with syntactic structures that split the traditional noun phrase into a determiner phrase and its complement -LRB- the DP-analysis -RRB- . The approach facilitates implementation in a unification-based system . OEFAI-TR-90-16 Proc . European Working Session on Learning , EWSL - 91 , pp. 282-297 , Porto , Portugal , March 6-8 , 1991 , Springer-Verlag . Extending Explanation-Based Generalization by Abstraction Operators Igor Mozeti < = c , Christian Holzbaur We present two contributions to the explanation-based generalization techniques . First , the operationality criterion is extended by abstraction operators . These allow for the goal concept to be reformulated not only in terms of operational predicates , but also allow to delete irrelevant arguments , and to collapse indistinguishable constants . The abstraction algorithm is presented and illustrated by an example . Second , the domain theory is not restricted to variables with finite -LRB- discrete -RRB- domains , but can deal with infinite -LRB- e.g. , real-valued -RRB- domains as well . The interpretation and abstraction are effectively handled through constraint logic programming mechanisms . In the paper we concentrate on the role of CLP -LRB- R -RRB- - a solver for systems of linear equations and inequalities over reals . OEFAI-TR-90-17 Applied Artificial Intelligence 5 -LRB- 1 -RRB- , pp. 47-56 , 1991 . The Challenge of Deep Models , Inference Structures , and Abstract Tasks Werner Horn The paper discusses methodological achievements which have been incorporated into second generation expert systems . The key ideas are -LRB- 1 -RRB- to incorporate more principled knowledge about the domain into a knowledge based system and to reason from these -LRB- first -RRB- principles , -LRB- 2 -RRB- to define the conceptual model explicitly , and -LRB- 3 -RRB- to do some abstraction . Abstraction is done on three levels : the factual knowledge level , the level of inference steps , and the task level . The challenges of these methods are discussed from the viewpoint of medical and technical applications . Keywords : Second Generation Expert Systems , Deep Models , Inference Structures , Generic Tasks . OEFAI Reports 1990 5 OEFAI-TR-90-18 Proc . European Working Session on Learning , EWSL - 91 , pp. 33-43 , Porto , Portugal , March 6-8 , 1991 , Springer-Verlag . Using Plausible Explanations to Bias Empirical Generalization in Weak Theory Domains Gerhard Widmer The paper argues for the usefulness of plausible explanations not just for analytical learning , but also for empirical generalization . The larger context is an implemented system that learns complex rules -LRB- for a musical task -RRB- on the basis of a qualitative theory of the domain . It learns by generalizing and compiling plausible explanations , but it can also incrementally modify learned rules in reaction to new evidence . The paper shows how this incremental modification -LRB- generalization -RRB- becomes more effective if it is based on an analysis of the explanations underlying learned rules ; these explanations support a notion of ` deep ' similarity and can provide substantial bias on the empirical modification of concepts . Several criteria that implement this bias are described , and an extended example illustrates how they lead to highly intelligent generalization behaviour . OEFAI-TR-90-19 Proc . AISB-91 , pp. 129-135 , Leeds , UK , April 16-19 , 1991 , Springer-Verlag . Constraintpropagation in Qualitative Modelling : Domain Variables Improve Diagnostic Efficiency Bernhard Pfahringer This paper shows how a specific constraint propagation technique - namely domain variables - can speed up qualitative diagnosis considerably . We are using the KAR - DIO system , a qualitative simulation model of the electrical activity of the heart , to exemplify our points . Furthermore we describe how the domain handling mechanism itself can be implemented in PROLOG efficiently . For a class of applications , where the constraint solver only performs a minor part of the computation our approach is comparable to or better than specialised constraint logic programming systems with regard to overall runtime . Additionally we gain the benefit of being able to specify all of the system in a single language . Keywords : Constraint Propagation , Unification , Qualitative Modelling , Implementation .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

incremental learning
induction
knowledge-intensive learning


Submitted to International Conference on Machine Learning -LRB- ICML ? 97 -RRB- , 1997 . Instance Pruning Techniques D. Randall Wilson Tony R. Martinez Computer Science Department Brigham Young University Provo , UT 84058 Abstract . The nearest neighbor algorithm and its derivatives are often quite successful at learning a concept from a training set and providing good generalization on subsequent input vectors . However , these techniques often retain the entire training set in memory , resulting in large memory requirements and slow execution speed , as well as a sensitivity to noise . This paper provides a discussion of issues related to reducing the number of instances retained in memory while maintaining -LRB- and sometimes improving -RRB- generalization accuracy , and mentions algorithms other researchers have used to address this problem . It presents three intuitive noise-tolerant algorithms that can be used to prune instances from the training set . In experiments on 29 applications , the algorithm that achieves the highest reduction in storage also results in the highest generalization accuracy of the three methods . Email : randy@axon.cs.byu.edu, martinez@cs.byu.edu Phone : 1-801-378-6464 Fax : 1-801-378-7555 Instance Pruning Techniques Abstract . The nearest neighbor algorithm and its derivatives are often quite successful at learning a concept from a training set and providing good generalization on subsequent input vectors . However , these techniques often retain the entire training set in memory , resulting in large memory requirements and slow execution speed , as well as a sensitivity to noise . This paper provides a discussion of issues related to reducing the number of instances retained in memory while maintaining -LRB- and sometimes improving -RRB- generalization accuracy , and mentions algorithms other researchers have used to address this problem . It presents three intuitive noise-tolerant algorithms that can be used to prune instances from the training set . In experiments on 29 applications , the algorithm that achieves the highest reduction in storage also results in the highest generalization accuracy of the three methods . 1 . Introduction The nearest neighbor algorithm -LSB- Cover & Hart , 1967 ; Dasarathy , 1991 -RSB- has been used successfully for pattern classification on many applications . Each pattern has an input vector with one value for each of several input attributes . An instance has an input vector and an output class . A training set T is a collection of instances with known output classes . A new pattern x is classified in the basic nearest neighbor algorithm by finding the instance y in T that is closest to x , and using the output class of y as the predicted classification for x . The instance that is closest to x is taken to be the one with minimum distance , using some distance function . The distance function can have a significant impact on the ability of the classifier to generalize correctly . The nearest neighbor algorithm learns very quickly -LRB- O -LRB- n -RRB- -RRB- because it need only read in the training set without further processing . It can also generalize accurately for many applications because it can learn complex concept descriptions , and provides an appropriate bias for many applications . The nearest neighbor algorithm also has several shortcomings . Since it stores all training instances , it has large -LRB- O -LRB- n -RRB- -RRB- memory requirements . Since it must search through all available instances to classify a new input vector , it is slow -LRB- O -LRB- n -RRB- -RRB- during classification . Since it stores every instance in the training set , noisy instances -LRB- i.e. , those with errors in the input vector or output class , or those not representative of typical cases -RRB- are stored as well , which can degrade generalization accuracy . Techniques such as k-d trees -LSB- Sproull , 1991 -RSB- and projection -LSB- Papadimitriou & Bentley , 1980 -RSB- can reduce the time required to find the nearest neighbor -LRB- s -RRB- of an input vector , but they do not reduce storage requirements , do not address the problem of noise , and often become much less effective as the dimensionality of the problem -LRB- i.e. , the number of input attributes -RRB- grows . On the other hand , when some of the instances are removed -LRB- or pruned -RRB- from the training set , the storage requirements and time necessary for generalization are correspondingly reduced . This paper focuses on the problem of reducing the size of the stored set of instances while trying to maintain or even improve generalization accuracy . Much research has been done in this area , and an overview of this research is given in Section 2 . Section 3 discusses several issues related to the problem of instance set reduction , and provides motivation for further algorithms . Section 4 presents four new instance reduction techniques that were hypothesized to provide substantial instance reduction while continuing to generalize accurately , even in the presence of noise . The first is similar to the Reduced Nearest Neighbor -LRB- RNN -RRB- algorithm -LSB- Gates 1972 -RSB- . The second changes the order in which instances are considered for removal , and the third adds a noise-reduction step similar to that done by Wilson -LSB- 1972 -RSB- before proceeding with the main reduction algorithm . Section 5 describes experiments on 29 datasets that compare the performance of each of these three reduction techniques to a basic k-nearest neighbor algorithm . The results indicate that these algorithms achieve substantial storage reduction , while maintaining good generalization accuracy . Section 6 provides conclusions and future research directions . 2 . Related Work Several researchers have addressed the problem of training set size reduction . This section reviews previous work in reduction algorithms . 2.1 . Nearest Neighbor Editing Rules Hart -LSB- 1968 -RSB- made one of the first attempts to reduce the size of the training set with his Condensed Nearest Neighbor Rule -LRB- CNN -RRB- . This algorithm begins by randomly selecting one instance belonging to each output class from the training set T and putting it in a subset S . Then each instance in T is classified using only the instances in S . If an instance is misclassified , it is added to S , thus ensuring that it will be classified correctly . This algorithm is especially sensitive to noise , because noisy instances will usually be misclassified by their neighbors , and thus will be retained . Rittler et . al. -LSB- 1975 -RSB- extended the condensed NN method in their Selective Nearest Neighbor Rule -LRB- SNN -RRB- such that every member of T must be closer to a member of S of the same class than to a member of T -LRB- instead of S -RRB- of a different class . Further , the method ensures a minimal subset satisfying these conditions . This algorithm resulted in greater reduction in the training set size as well as slightly higher accuracy than CNN in their experiments , though it is still sensitive to noise . Gates -LSB- 1972 -RSB- modified this algorithm in his Reduced Nearest Neighbor Rule -LRB- RNN -RRB- . The reduced NN algorithm starts with S ? = ? T and removes each instance from S if such a removal does not cause any other instances in T to be misclassified . Since the instance being removed is not guaranteed to be classified correctly , this algorithm is able to remove noisy instances and internal instances while retaining border points . Wilson -LSB- 1972 -RSB- developed an algorithm which removes instances that do not agree with the majority of their k nearest neighbors -LRB- with k = 3 , typically -RRB- . This edits out noisy instances as well as close border cases , leaving smoother decision boundaries . It also retains all internal points , which keeps it from reducing the storage requirements as much as many other algorithms . Tomek -LSB- 1976 -RSB- extended Wilson?s algorithm with his ? all k-NN ? method of editing by calling Wilson?s algorithm repeatedly with k = 1 . . k , though this still retains internal points . 2.2 . ? Instance-Based ? Learning Algorithms Aha et . al. -LSB- 1991 -RSB- presented a series of instance-based learning algorithms that reduce storage . IB2 is quite similar to the Condensed Nearest Neighbor -LRB- CNN -RRB- rule -LSB- Hart , 1968 -RSB- , and suffers from the same sensitivity to noise . IB3 -LSB- Aha et al. 1991 -RSB- addresses IB2?s problem of keeping noisy instances by using a statistical test to retain only acceptable misclassified instances . In their experiments , IB3 was able to achieve greater reduction in the number of instances stored and also achieved higher accuracy than IB2 , due to its reduced sensitivity to noise on the applications on which it was tested . Zhang -LSB- 1992 -RSB- used a different approach called the Typical Instance Based Learning -LRB- TIBL -RRB- algorithm , which attempted to save instances near the center of clusters rather than on the border . Cameron-Jones -LSB- 1995 -RSB- used an encoding length heuristic to determine how good the subset S is in describing T . After a growing phase similar to IB3 , a random mutation hill climbing method called Explore is used to search for a better subset of instances , using the encoding length heuristic as a guide . 2.3 . Prototypes and other Modifications of the Instances Some algorithms seek to reduce storage requirements and speed up classification by modifying the instances themselves , instead of just deciding which ones to keep . Chang -LSB- 1974 -RSB- introduced an algorithm in which each instance in T is initially treated as a prototype . The nearest two instances that have the same class are merged into a single prototype -LRB- using a weighted averaging scheme -RRB- that is located somewhere between the two prototypes . This process is repeated until classification accuracy starts to suffer . Domingos -LSB- 1995 -RSB- introduced the RISE 2.0 system which treats each instance in T as a rule in R , and then generalizes rules until classification accuracy starts to suffer . Salzberg -LSB- 1991 -RSB- introduced the nested generalized exemplar -LRB- NGE -RRB- theory , in which hyperrectangles are used to take the place of one or more instances , thus reducing storage requirements . Wettschereck & Dietterich , -LSB- 1995 -RSB- introduced a hybrid nearest-neighbor and nearest-hyperrectangle algorithm that used hyperrectangles to classify input vectors if they fell inside the hyperrectangle , and kNN to classify inputs that were not covered by any hyperrectangle . This algorithm must therefore store the entire training set T , but accelerates classification by using relatively few hyperrectangles whenever possible . 3 . Instance Reduction Issues From the above learning models , several observations can be made regarding the issues involved in training set reduction . This section covers the issues of instance representation , the order of the search , the choice of distance function , the general intuition of which instances to keep , and how to evaluate the different strategies . 3.1 . Representation One choice in designing a training set reduction algorithm is to decide whether to retain a subset of the original instances or whether to modify the instances using a new representation . For example , NGE -LSB- Salzberg , 1991 -RSB- and its derivatives -LSB- Wettschereck & Dietterich , 1995 -RSB- use hyperrectangles to represent collections of instances ; RISE -LSB- Domingos , 1995 -RSB- generalizes instances into rules ; and prototypes -LSB- Chang 1974 -RSB- can be used to represent a cluster of instances , even if no original instance occurred at the point where the prototype is located . On the other hand , many models seek to retain a subset of the original instances , including the Condensed NN rule -LRB- CNN -RRB- -LSB- Hart , 1968 -RSB- , the Reduced NN rule -LRB- RNN -RRB- -LSB- Gates 1972 -RSB- , the Selective NN rule -LRB- SNN -RRB- -LSB- Rittler et . al. , 1975 -RSB- , Wilson?s rule -LSB- Wilson , 1972 -RSB- , the ? all k-NN ? method -LSB- Tomek , 1976 -RSB- , InstanceBased -LRB- IBL -RRB- Algorithms -LSB- Aha et . al. . 1991 -RSB- , and the Typical Instance Based Learning -LRB- TIBL -RRB- algorithm -LSB- Zhang , 1992 -RSB- . Another decision that affects the concept description for many algorithms is the choice of k , which is the number of neighbors used to decide the output class of an input vector . The value of k is typically a small integer -LRB- e.g. , 1 , 3 or 5 -RRB- that is odd so as to avoid ? ties ? in the voting of neighbors . The value of k is often determined from cross-validation . 3.2 . Direction of Search When searching for a subset S of instances to keep from training set T , there are different directions the search can proceed . We call these search directions incremental , decremental , and batch . Incremental . The incremental search begins with an empty subset S , and adds each instance in T to S if it fulfills the criteria . Using such a search direction , the order of presentation of instances can be very important . In particular , the first few instances may have a very different probability of being included in S than they would if they were visited later . For example , CNN begins by selecting one instance from each class at random , which gives these instances a 100 % chance of being included . The next instances visited are classified only by the few instances that are already in S , while instances chosen near the end of the algorithm are classified by a much larger number of instances that have been included in S . Other incremental schemes include IB2 and IB3 . One advantage to an incremental scheme is that if instances are made available later , after training is complete , they can continue to be added to S according to the same criteria . Decremental . Decremental searches begins with S = T , and then search for instances to remove from S. Again the order of presentation can be important , but unlike the incremental process , all of the training examples are available for examination at any time , so a search can be made to determine which instance would be best to remove during each step of the algorithm . Decremental algorithms include RNN , SNN , and Wilson?s -LSB- 1972 -RSB- rule . NGE and RISE can also be viewed as decremental algorithms , except that instead of simply removing instances from S , they are instead generalized into hyperrectangles or rules . Similarly , Chang?s prototype rule operates in a decremental order , but prototypes are merged into each other instead of being simply removed . One disadvantage with the decremental rule is that it is often computationally more expensive than incremental algorithms . For example , in order to find the nearest neighbor in T of an instance , n distance calculations must be made . On the other hand , there are fewer than n instances in S -LRB- zero initially , and some fraction of T eventually -RRB- , so finding the nearest neighbor in S of an instance takes less computation . However , if the application of a decremental algorithm can result in greater storage reduction and/or increased generalization accuracy , then the extra computation during learning -LRB- which is done just once -RRB- can be well worth the computational savings during execution thereafter . Batch . A final way to apply a training set reduction rule is in batch mode . This involves deciding if each instance meets the removal criteria before removing any , and then removing all of them at once . For example , the ? allkNN ? rule operates this way . This can relieve the algorithm from having to constantly update lists of nearest neighbors and other information when instances are individually removed , but there are also dangers in batch processing . For example , assume we apply a rule such as ? Remove an instance if it has the same output class as its k nearest neighbors ? to a training set . This could result in entire clusters disappearing if there are no instances of a different class nearby . If done in decremental mode , however , some instances would remain , because eventually enough neighbors would be removed that one of the k nearest neighbors of an instance would have to be of another class , even if it was originally surrounded by those of its own class . 3.3 . Border points vs. central points Another factor that distinguishes instance reduction techniques is whether they seek to retain border points , central points , or some other set of points . The intuition behind retaining border points is that ? internal ? points do not affect the decision boundaries as much as the border points , and thus can be removed with relatively little effect on classification . Algorithms which tend to retain border points include CNN , RNN , IB2 , and IB3 . On the other hand , some algorithms instead seek to remove border points . Wilson?s rule and the ? all kNN ? rule are examples of this . They remove points that are noisy or do not agree with their neighbors . This removes close border points , leaving smoother decision boundaries behind . This may help generalization in some cases , but typically keeps most of the instances . Some algorithms retain center points instead of border points . For example , the Typical Instance Based Learning algorithm attempts to retain center points and can achieve very dramatic reduction -LRB- ideally one instance per class -RRB- when conditions are right . However , when decision boundaries are complex , it may take nearly as many instances to define the boundaries using center points as it would using boundary points . Current research is addressing this question . 3.4 . Distance Function The nearest neighbor algorithm and its derivatives usually use the Euclidean distance function , which is defined as : E -LRB- x , y -RRB- = -LRB- x i - y i -RRB- 2 i = 1 m ? -LRB- 1 -RRB- where x and y are the two input vectors , m is the number of input attributes , and xi and yi are the input values for input attribute i . This function is appropriate when all the input attributes are numeric and have ranges of approximately equal width . When the attributes have substantially different ranges , the attributes can be normalized by dividing the individual attribute distances by the range or standard deviation of the attribute . When nominal -LRB- discrete , unordered -RRB- attributes are included in an application , a distance metric is needed that can handle them . We use a distance function based upon the Value Difference Metric -LRB- VDM -RRB- -LSB- Stanfill & Waltz , 1986 -RSB- for nominal attributes . A simplified version of the VDM defines the distance between two values x and y of a single attribute a as : vdm a -LRB- x , y -RRB- = N a , x , c N a , x - N a , y , c N a , y ? ? ? ? ? ? c = 1 C ? 2 -LRB- 2 -RRB- where Na , x is the number of times attribute a had value x ; Na , x , c is the number of times attribute a had value x and the output class was c ; and C is the number of output classes . Using this distance measure , two values are considered to be closer if they have more similar classifications , regardless of the order of the values . In order to handle heterogeneous applications?those with both numeric and nominal attributes?we use the heterogeneous distance function HVDM -LSB- Wilson & Martinez , 1997 -RSB- , which is defined as : HVDM -LRB- x , y -RRB- = d a 2 -LRB- x a , y a -RRB- a = 1 m ? -LRB- 3 -RRB- where the function da -LRB- x , y -RRB- is the distance for attribute a and is defined as : d a -LRB- x , y -RRB- = vdm a -LRB- x , y -RRB- , if a is nominal x - y 4s a , if a is numeric ? ? ? ? ? -LRB- 4 -RRB- where vdma -LRB- x , y -RRB- is the function given in -LRB- 2 -RRB- , and sa is the standard deviation of the values occurring for attribute a in the instances in the training set T . This function handles unknown input values by assigning them a large distance -LRB- 1.0 -RRB- , and provides appropriate normalization between numeric and nominal attributes , as well as between numeric attributes of different scales . 4 . New Instance Set Reduction Algorithms This section presents a collection of new heuristics or rules used to decide which instances to keep and which instances to remove from a training set . In order to avoid repeating lengthy definitions , some notation is introduced here : A training set T consists of n instances -LRB- or prototypes -RRB- P1 . . n. Each instance P has k nearest neighbors P.N1 . . k -LRB- ordered from nearest to furthest -RRB- , where k is a small odd integer such as 1 , 3 or 5 . P also has a nearest enemy , P.E , which is the nearest instance with a different output class . Those instances that have P as one of their k nearest neighbors are called associates of P , and are notated as P.A1 . . a -LRB- sorted from nearest to furthest -RRB- where a is the number of associates that P has . Given the issues in Section 3 to consider , our research is directed towards finding instance reduction techniques that provide noise tolerance , high generalization accuracy , insensitivity to the order of presentation of instances , and significant storage reduction , which in turn improves generalization speed . Sections 4.1-4 .3 present three rules , called RT1-RT3 , respectively , that seek to meet these goals . 4.1 . Reduction Technique 1 -LRB- RT1 -RRB- The first reduction technique we present , RT1 , uses the following basic rule to decide if it is safe to remove an instance from the instance set S -LRB- where S ? = ? T originally -RRB- : Remove P if at least as many of its associates in S would be classified correctly without it . To see if an instance can be removed using this rule , each associate is checked to see what effect the removal of P would have on it . Removing P causes each associate P.Ai to use its k +1 st nearest neighbor -LRB- P.Ai.Nk +1 -RRB- in place of P . If P has the same class as P.Ai , and P.Ai.Nk +1 has a different class than P.Ai , this weakens its classification , and could cause P.Ai to be misclassified by its neighbors . On the other hand , if P is a different class than P.Ai and the P.Ai.Nk +1 is the same class as P.Ai , the removal of P?could cause a previously misclassified instance to be classified correctly . In essence , this rule tests to see if removing P would degrade leave-one-out cross-validation generalization accuracy , which is an estimate of the true generalization ability of the resulting classifier . An instance is removed when it results in the same level of generalization with lower storage requirements . The algorithm for RT1 proceeds as shown in Figure 1 . 1 RT1 -LRB- Training set T -RRB- : Instance set S. 2 Let S = T. 3 For each instance P in S : 4 Find P.N1 . . k +1 , the k +1 nearest neighbors of P?in S. 5 Add P?to each of its neighbors ? lists of associates . 6 For each instance P in S : 7 Let with = # of associates of P classified correctly with P as a neighbor . 8 Let without = # of associates of P classified correctly without P. 9 If -LRB- without - with -RRB- ? 0 10 Remove P from S. 11 Remove P from its associates ? lists of nearest neighbors , and find 12 the next nearest neighbor for each of these associates . 13 Remove P from its neighbors ? lists of associates . 14 Endif 15 Return S. Figure 1 . Pseudo-code for RT1 . This algorithm begins by building a list of nearest neighbors for each instance , as well as a list of associates . Then each instance in S is removed if its removal does not hurt the classification of the instances remaining in S . This algorithm removes noisy instances , because a noisy instance P usually has associates that are mostly of a different class , and such associates will be at least as likely to be classified correctly without P. RT1 also removes instances in the center of clusters , because associates there are not near their enemies , and thus continue to be classified correctly without P. Near the border , the removal of some instances sometimes causes others to be classified incorrectly because the majority of their neighbors can become enemies . Thus this algorithm tends to keep non-noisy border points . At the limit , there is typically a collection of border instances such that the majority of the k nearest neighbors of each of these instances is the correct class . 4.2 . Reduction Technique 2 -LRB- RT2 -RRB- There is a potential problem that can arise in RT1 with regards to noisy instances . A noisy instance will typically have associates of a different class , and will thus be contained to a somewhat small portion of the input space . However , if its associates are removed by the above criteria , the noisy instance may cover more and more of the input space . Eventually it is hoped that the noisy instance itself will be removed . However , if many of its neighbors are removed first , its associates may eventually include instances of the same class from the other side of the original decision boundary , and it is possible that removing the noisy instance at that point could cause some of its distant associates to be classified incorrectly . RT2 solves this problem by considering the effect of the removal of an instance on all the instances in the original training set T instead of considering only those instances remaining in S . In other words , an instance P is removed from S only if at least as many of its associates -LRB- including those that may have already been pruned -RRB- are classified correctly without it . Using this modification , each instance in the original training set T continues to maintain a list of its k ? + ? 1 nearest neighbors in S , even after it is removed from S . This in turn means that instances in S have associates that are both in and out of S , while instances that have been removed from S have no associates -LRB- because they are no longer a neighbor of any instance -RRB- . This modification makes use of additional information that is available for estimating generalization accuracy , and also avoids some problems that can occur with RT1 such as removing entire clusters . This change is made by removing line 13 from the pseudo-code for RT1 so that pruned instances will still be associates of their nearest neighbors in S. RT2 also changes the order of removal of instances . It initially sorts the instances in S by the distance to their nearest enemy . Instances are then checked for removal beginning at the instance furthest from its nearest enemy . This tends to remove instances furthest from the decision boundary first , which in turn increases the chance of retaining border points . 4.3 . Reduction Technique 3 -LRB- RT3 -RRB- RT2 sorts S in an attempt to remove center points before border points . One problem with this method is that noisy instances are also ? border ? points , and cause the order of removal to be drastically changed . One noisy point in the center of a cluster causes many points in that cluster to be considered border points , and some of these can remain in S even after the noisy point is removed . Two passes through S can remove the dangling center points , but unfortunately , by that time some border points may have already been removed that should have been kept . RT3 therefore uses a noise-filtering pass before sorting the instances in S . This is done using a rule similar to Wilson?s Rule -LSB- Wilson , 1972 -RSB- : Any instance misclassified by its k nearest neighbors is removed . This removes noisy instances , as well as close border points , which can in turn smooth the decision boundary slightly . This helps to avoid ? overfitting ? the data , i.e. , using a decision surface that goes beyond modeling the underlying function and starts to model the data sampling distribution as well . After removing noisy instances from S in this manner , the instances are sorted by distance to their nearest enemy remaining in S , and thus points far from the real decision boundary are removed first . This allows points internal to clusters to be removed early in the process , even if there were noisy points nearby . 5 . Experimental Results The reduction algorithms RT1 , RT2 and RT3 were implemented using k ? = ? 3 , and using the HVDM distance function described in Section 3.4 . These algorithms were tested on 29 data sets from the University of California , Irvine Machine Learning Database Repository -LSB- Merz & Murphy , 1996 -RSB- and compared to a k-nearest neighbor classifier that was identical to RT1 except that it does not remove any instances from the instance set -LRB- i.e. , S = T -RRB- . Database Anneal Australian Breast Cancer WI Bridges Crx Echocardiogram Flag Glass Heart Heart.Cleveland Heart.Hungarian Heart.Long Beach VA Heart.More Heart.Swiss Hepatitis Horse-Colic Image.Segmentation Ionosphere Iris LED-Creator +17 LED-Creator Liver.Bupa Pima Indians Diabetes Promoters Sonar Soybean-Large Vehicle Vowel Wine Average kNN 93.11 84.78 96.28 66.09 83.62 94.82 61.34 73.83 81.48 81.19 79.22 70.00 74.17 92.69 80.62 57.84 93.10 84.62 94.00 67.10 60.60 65.57 73.56 93.45 87.55 88.59 71.76 96.57 94.93 80.78 -LRB- size -RRB- 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 100 RT1 87.85 82.61 94.00 55.64 81.01 93.39 58.13 60.30 79.26 77.85 78.92 73.00 73.20 91.15 76.21 65.09 84.76 84.91 89.33 68.80 64.00 59.66 70.96 85.00 69.81 79.81 66.21 88.98 91.05 76.93 -LRB- size -RRB- 9.11 7.67 2.56 20.86 6.70 9.01 24.51 26.11 12.96 14.26 11.38 11.78 11.20 2.08 8.67 10.89 10.21 5.67 11.70 17.50 7.78 27.28 20.12 8.18 24.04 24.03 24.00 43.14 8.55 14.55 RT2 95.36 84.64 96.14 59.18 84.93 85.18 62.34 64.98 81.11 79.87 79.22 72.00 74.50 93.46 82.00 66.17 92.38 88.32 95.33 70.50 67.00 65.80 73.31 87.91 81.86 84.99 68.91 91.46 93.79 80.09 -LRB- size -RRB- 11.42 15.41 5.79 24.11 14.11 7.51 32.30 31.52 21.60 20.61 15.98 16.33 16.98 2.89 13.98 17.98 13.76 12.09 16.89 26.23 11.20 37.55 28.11 17.82 30.82 27.94 31.60 46.91 15.23 20.16 RT3 93.49 84.35 96.14 58.27 85.80 93.39 61.29 65.02 83.33 80.84 79.95 73.50 76.25 93.46 81.87 71.08 92.62 87.75 95.33 70.40 68.20 60.84 75.01 86.82 78.00 85.62 65.85 89.56 94.93 80.31 -LRB- size -RRB- 8.63 5.93 3.58 18.66 5.46 9.01 20.45 23.88 13.62 12.76 10.43 4.22 9.10 1.81 7.81 7.42 10.98 7.06 14.81 12.66 11.57 24.99 16.90 16.67 26.87 25.73 23.00 45.22 16.11 14.32 Table 1 . Generalization accuracy and storage requirements of kNN , RT1 , RT2 , and RT3 . Each test consisted of ten trials , each using one of ten partitions of the data randomly selected from the data sets , i.e. , 10-fold cross-validation . For each trial , 90 % of the training instances were used for T , the subset S was determined using each reduction technique -LRB- except for the kNN algorithm , which keeps all the instances -RRB- , and the remaining 10 % of the instances were classified using only the instances remaining in S . The average generalization accuracy over the ten trials for each test is given in Table 1 . The average percentage of instances retained in S is shown in the table as well . From the results in this table , several observations can be made . RT1 had very good storage reduction on average , but also dropped in generalization accuracy by an average of almost 4 % . This is likely due to the fact that this algorithm ignores those instances that have already been pruned from S when deciding whether to remove additional instances . It therefore prunes too many instances , thus reducing generalization accuracy . Also , it prunes them in a random order , thus causing some border points to be removed too early . RT2 had better generalization accuracy than RT1 because of its use of the additional information provided by pruned instances in determining whether to remove others . However , it also has higher storage requirements , due at least in part to the fact that noisy instances sometimes cause nearby instances to be retained even after the noisy instance is removed . RT3 had a higher average generalization accuracy than RT1 or RT2 , and also had the lowest storage requirements of the three . Its generalization accuracy was within one-half of a percent of the kNN algorithm that retained 100 % of the instances , and yet it retained on average only 14.32 % of the instances . Some datasets seem to be especially well suited for these reduction techniques . For example , RT3 required less than 2 % storage for the Heart.Swiss dataset , yet it achieved even higher generalization accuracy than the kNN algorithm . On the other hand , some datasets were not so appropriate . On the Vowel dataset , for example , RT3 required over 45 % of the data , and dropped in generalization accuracy by 7 % , suggesting that RT3 is inappropriate for this particular dataset . Empirical results were not available for all of the reduction algorithms mentioned in Section 2 , but comparisons with IB3 -LSB- Aha , Kibler & Albert , 1991 -RSB- were possible on some datasets . Results reported by Zarndt -LSB- 1995 -RSB- on IB3 include 21 datasets overlapping with those presented in Table 1 . On these datasets , IB3 had an average accuracy of 74 % while RT3 had an accuracy of 82 % . Unfortunately , storage requirements were not reported by Zarndt . However , results reported by Cameron-Jones for IB3 included three datasets that were used in our experiments . On these datasets -LRB- Heart.Cleveland , Heart.Hungarian , Iris -RRB- , IB3 used an average of about half the storage -LRB- 5.9 % compared to 12.7 % -RRB- required by RT3 , but had a slightly lower average accuracy . Additional comparisons are necessary to draw any substantial conclusions on storage requirements . One reason for the larger storage requirements of RT3 in our experiments is due to the use of k ? = ? 3 . This causes some instances to be retained that could be removed with k ? = ? 1 . Initial experiments suggest that the smaller value of k results in more dramatic storage reduction , but may reduce generalization accuracy slightly in RT3 . One area of future research is in the use of a dynamic value of k , where k starts out with a value of 3 or 5 and is reduced as pruning continues until it is eventually reduced to a value of 1 . We were interested to see how fast generalization accuracy drops as instances are removed from S , and to see if it drops off more slowly when using an intelligent reduction technique than it does when removing instances randomly . 65 70 75 80 60 50 100 % Accuracy % Storage Random RT3 Figure 2 . Generalization Accuracy for RT3 vs. Randomly removing instances . In order to test this , the experiments described above were modified as follows . When instances were removed from S , the order of their removal was recorded so that the training set T could be sorted by order of removal -LRB- with the instances in S at the beginning in random order , since they were not removed -RRB- . Then the instances in the test set were classified using only the first 1 % of the data , then 2 % , and so on up to 100 % of the data . Figure 2 shows the average generalization accuracy -LRB- over all 10 trials on all 29 datasets -RRB- as a function of the percentage of the training set that was used for generalization . As can be seen from the figure , the average generalization accuracy drops off more quickly if instances are randomly removed than if they are removed using RT3 , indicating that the order of removal is improved by RT3 . This can be useful if more storage is available than that required by RT3 . For example , if an instance set of 10 million instances is reduced to just 100 , but there is sufficient storage and computational resources to handle 1000 instances , then by using the sorted list of pruned instances , the best 1000 instances can be used . The user is thus allowed to manually trade off storage for accuracy without the higher loss in accuracy that random removal would cause . 6 . Conclusions and Future Research Directions Nearest neighbor algorithms and their derivatives are often appropriate and can provide high generalization accuracy for real-world applications , but often the storage and computational requirements are restrictive when the size of the training set is large . This paper introduced three new instance reduction techniques which are intuitive and provide good storage reduction . In experiments on 29 datasets , the third technique , RT3 , provided higher generalization accuracy and lower storage requirements than the other two methods , and its accuracy was within 0.5 % of that of a nearest neighbor classifier that retained all of the instances . On average it retained under 15 % of the original training set . RT3 -LRB- and to a lesser extent the other algorithms -RRB- is designed to be robust in the presence of noise and use an estimate of generalization accuracy in making decisions , which helps it avoid removing instances that would be helpful in generalization . Since RT3 makes use of all the instances in the training set in making its decision , it is not sensitive to the order of presentation of the instances -LRB- as are incremental approaches -RRB- , and is able to choose a good order of removal regardless of how the instances were ordered in the original training set . These reduction algorithms were also among the first to use heterogeneous distance functions appropriate for applications with both nominal and continuous attributes -LSB- Wilson & Martinez , 1997 -RSB- . Future research will focus on determining the conditions under which these algorithms are not appropriate -LRB- such as the Vowel dataset -RRB- , and will seek to overcome weaknesses in such areas . These reduction algorithms will also be integrated with feature selection algorithms and weighting techniques in order to produce comprehensive instance-based learning systems that are robust in the presence of irrelevant attributes and other difficult circumstances , thus providing more accurate learning algorithms for a wide variety of problems . References Aha , David W. , Dennis Kibler , Marc K. Albert , -LRB- 1991 -RRB- . ? Instance-Based Learning Algorithms , ? Machine Learning , vol . 6 , pp. 37-66 . Cameron-Jones , R. M. , -LRB- 1995 -RRB- . Instance Selection by Encoding Length Heuristic with Random Mutation Hill Climbing . In Proceedings of the Eighth Australian Joint Conference on Artificial Intelligence , pp . 99-106 . Chang , Chin-Liang , -LRB- 1974 -RRB- . ? Finding Prototypes for Nearest Neighbor Classifiers , ? IEEE Transactions on Computers , vol . 23 , no. 11 , November 1974 , pp. 1179-1184 . Cover , T. M. , and P. E. Hart , -LRB- 1967 -RRB- . ? Nearest Neighbor Pattern Classification , ? Institute of Electrical and Electronics Engineers Transactions on Information Theory , vol . 13 , no. 1 , January 1967 , pp . 21-27 . Dasarathy , Belur V. , -LRB- 1991 -RRB- . Nearest Neighbor -LRB- NN -RRB- Norms : NN Pattern Classification Techniques . Los Alamitos , CA : IEEE Computer Society Press . Domingos , Pedro , -LRB- 1995 -RRB- . ? Rule Induction and Instance-Based Learning : A Unified Approach , ? to appear in The 1995 International Joint Conference on Artificial Intelligence -LRB- IJCAI-95 -RRB- . Gates , G. W. -LRB- 1972 -RRB- . ? The Reduced Nearest Neighbor Rule , ? IEEE Transactions on Information Theory , vol . IT-18 , no. 3 , pp. 431-433 . Hart , P. E. , -LRB- 1968 -RRB- . ? The Condensed Nearest Neighbor Rule , ? Institute of Electrical and Electronics Engineers Transactions on Information Theory , vol . 14 , pp. 515-516 . Merz , C. J. , and P. M. Murphy , -LRB- 1996 -RRB- . UCI Repository of Machine Learning Databases . Irvine , CA : University of California Irvine , Department of Information and Computer Science . Internet : http://www.ics.uci.edu/~mlearn/ MLRepository.html . Papadimitriou , Christos H. , and Jon Louis Bentley , -LRB- 1980 -RRB- . A Worst-Case Analysis of Nearest Neighbor Searching by Projection . Lecture Notes in Computer Science , Vol . 85 , Automata Languages and Programming , pp. 470 - 482 . Rittler , G. L. , H. B. Woodruff , S. R. Lowry , and T. L. Isenhour , -LRB- 1975 -RRB- . ? An Algorithm for a Selective Nearest Neighbor Decision Rule , ? IEEE Transactions on Information Theory , vol . 21 , no. 6 , November 1975 , pp . 665 - 669 . Salzberg , Steven , -LRB- 1991 -RRB- . ? A Nearest Hyperrectangle Learning Method , ? Machine Learning , vol . 6 , pp. 277-309 . Sproull , Robert F. , -LRB- 1991 -RRB- . Refinements to Nearest-Neighbor Searching in kDimensional Trees . Algorithmica , Vol . ? 6 , pp. 579-589 . Stanfill , C. , and D. Waltz , -LRB- 1986 -RRB- . ? Toward memory-based reasoning , ? Communications of the ACM , vol . 29 , December 1986 , pp. 1213-1228 . Tomek , Ivan , -LRB- 1976 -RRB- . ? An Experiment with the Edited Nearest-Neighbor Rule , ? IEEE Transactions on Systems , Man , and Cybernetics , vol . 6 , no . 6 , June 1976 , pp. 448-452 . Wettschereck , Dietrich , -LRB- 1994 -RRB- . ? A Hybrid Nearest-Neighbor and NearestHyperrectangle Algorithm ? , To appear in the Proceedings of the 7th European Conference on Machine Learning . Wettschereck , Dietrich , and Thomas G. Dietterich , -LRB- 1995 -RRB- . ? An Experimental Comparison of Nearest-Neighbor and Nearest-Hyperrectangle Algorithms , ? Machine Learning , vol . 19 , no. 1 , pp. 5-28 . Wilson , D. Randall , and Tony R. Martinez , -LRB- 1997 -RRB- . ? Improved Heterogeneous Distance Functions , ? Journal of Artificial Intelligence Research -LRB- JAIR -RRB- , vol . 6 , no. 1 , pp. 1-34 . Wilson , Dennis L. , -LRB- 1972 -RRB- . ? Asymptotic Properties of Nearest Neighbor Rules Using Edited Data , ? IEEE Transactions on Systems , Man , and Cybernetics , vol . 2 , no. 3 , pp. 408-421 . Zarndt , Frederick , -LRB- 1995 -RRB- . A Comprehensive Case Study : An Examination of Connectionist and Machine Learning Algorithms , Master?s Thesis , Brigham Young University . Zhang , Jianping , -LRB- 1992 -RRB- . ? Selecting Typical Instances in Instance-Based Learning , ? Proceedings of the Ninth International Conference on Machine Learning .

O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B I O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O B O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O B O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O B I O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O O

editing
generalization
instance reduction
nearest neighbor

inductive learning
instance-based learning