Incremental Learning: Areas and Methods - A Survey
---
paper_title: A novel kernel method for clustering

paper_content:

Kernel methods are algorithms that, by replacing the inner product with an appropriate positive definite function, implicitly perform a nonlinear mapping of the input data into a high-dimensional feature space. In this paper, we present a kernel method for clustering inspired by the classical k-means algorithm in which each cluster is iteratively refined using a one-class support vector machine. Our method, which can be easily implemented, compares favorably with respect to popular clustering algorithms, like k-means, neural gas, and self-organizing maps, on a synthetic data set and three UCI real data benchmarks (IRIS data, Wisconsin breast cancer database, Spam database).
---
paper_title: Clustering Text Data Streams

paper_content:

Clustering text data streams is an important issue in data mining community and has a number of applications such as news group filtering, text crawling, document organization and topic detection and tracing etc. However, most methods are similarity-based approaches and only use the TF*IDF scheme to represent the semantics of text data and often lead to poor clustering quality. Recently, researchers argue that semantic smoothing model is more efficient than the existing TF*IDF scheme for improving text clustering quality. However, the existing semantic smoothing model is not suitable for dynamic text data context. In this paper, we extend the semantic smoothing model into text data streams context firstly. Based on the extended model, we then present two online clustering algorithms OCTS and OCTSM for the clustering of massive text data streams. In both algorithms, we also present a new cluster statistics structure named cluster profile which can capture the semantics of text data streams dynamically and at the same time speed up the clustering process. Some efficient implementations for our algorithms are also given. Finally, we present a series of experimental results illustrating the effectiveness of our technique.
---
paper_title: Enhancing an Incremental Clustering Algorithm for Web Page Collections

paper_content:

With the size and state of the Internet today, a good quality approach to organizing this mass of information is of great importance. Clustering web pages into groups of similar documents is one approach, but relies heavily on good feature extraction and document representation as well as a good clustering approach and algorithm. Due to the changing nature of the Internet, resulting in a dynamic dataset, an incremental approach is preferred. In this work we propose an enhanced incremental clustering approach to develop a better clustering algorithm that can help to better organize the information available on the Internet in an incremental fashion. Experiments show that the enhanced algorithm outperforms the original histogram based algorithm by up to 7.5%.
---
paper_title: Incremental Clustering for Mining in a Data Warehousing Environment

paper_content:

Data warehouses provide a great deal of opportunities for performing data mining tasks such as classification and clustering. Typically, updates are collected and applied to the data warehouse periodically in a batch mode, e.g., during the night. Then, all patterns derived from the warehouse by some data mining algorithm have to be updated as well. Due to the very large size of the databases, it is highly desirable to perform these updates incrementally. In this paper, we present the first incremental clustering algorithm. Our algorithm is based on the clustering algorithm DBSCAN which is applicable to any database containing data from a metric space, e.g., to a spatial database or to a WWW-log database. Due to the density-based nature of DBSCAN, the insertion or deletion of an object affects the current clustering only in the neighborhood of this object. Thus, efficient algorithms can be given for incremental insertions and deletions to an existing clustering. Based on the formal definition of clusters, it can be proven that the incremental algorithm yields the same result as DBSCAN. A performance evaluation of IncrementalDBSCAN on a spatial database as well as on a WWW-log database is presented, demonstrating the efficiency of the proposed algorithm. IncrementalDBSCAN yields significant speed-up factors over DBSCAN even for large numbers of daily updates in a data warehouse.
---
paper_title: Incremental machine learning techniques for document layout understanding

paper_content:

In real-world digital libraries, artificial intelligence techniques are essential for tackling the automatic document processing task with sufficient flexibility. The great variability in document kind, content and shape requires powerful representation formalisms to catch all the domain complexity. The continuous flow of new documents requires adaptable techniques that can progressively adjust the acquired knowledge on documents as long as new evidence becomes available, even extending if needed the set of recognized document types. Both these issues have not yet been thoroughly studied. This paper presents an incremental first-order logic learning framework for automatically dealing with various kinds of evolution in digital repositories content: evolution in the definition of class definitions, evolution in the set of known classes and evolution by addition of new unknown classes. Experiments show that the approach can be applied to real-world.
---
paper_title: Rough set based incremental clustering of interval data

paper_content:

This paper introduces a novel incremental approach to clustering interval data. The method employs rough set theory to capture the inherent uncertainty involved in cluster analysis. Our experimental results show that it produces meaningful cluster abstractions for interval data at a minimal computational expense.
---
paper_title: Incremental Clustering of Mobile Objects

paper_content:

Moving objects are becoming increasingly attractive to the data mining community due to continuous advances in technologies like GPS, mobile computers, and wireless communication devices. Mining spatio-temporal data can benefit many different functions: marketing team managers for identifying the right customers at the right time, cellular companies for optimizing the resources allocation, web site administrators for data allocation matters, animal migration researchers for understanding migration patterns, and meteorology experts for weather forecasting. In this research we use a compact representation of a mobile trajectory and define a new similarity measure between trajectories. We also propose an incremental clustering algorithm for finding evolving groups of similar mobile objects in spatio-temporal data. The algorithm is evaluated empirically by the quality of object clusters (using Dunn and Rand indexes), memory space efficiency, execution times, and scalability (run time vs. number of objects).
---
paper_title: A Fast and Stable Incremental Clustering Algorithm

paper_content:

Clustering is a pivotal building block in many data mining applications and in machine learning in general. Most clustering algorithms in the literature pertain to off-line (or batch) processing, in which the clustering process repeatedly sweeps through a set of data samples in an attempt to capture its underlying structure in a compact and efficient way. However, many recent applications require that the clustering algorithm be online, or incremental, in the that there is no a priori set of samples to process but rather samples are provided one iteration at a time. Accordingly, the clustering algorithm is expected to gradually improve its prototype (or centroid) constructs. Several problems emerge in this context, particularly relating to the stability of the process and its speed of convergence. In this paper, we present a fast and stable incremental clustering algorithm, which is computationally modest and imposes minimal memory requirements. Simulation results clearly demonstrate the advantages of the proposed framework in a variety of practical scenarios.
---
paper_title: Yuming “A Fast Incremental Clustering Algorithm

paper_content:

Clustering has played a very important role in data mining. In this paper, a fast incremental clustering algorithm is proposed by changing the radius threshold value dynamically. The algorithm restricts the number of the final clusters and reads the original dataset only once. At the same time an inter-cluster dissimilarity measure taking into account the frequency information of the attribute values is introduced. It can be used for the categorical data. The experimental results on the mushroom dataset show that the proposed algorithm is feasible and effective. It can be used for the large-scale data set.
---
paper_title: An online incremental learning pattern-based reasoning system.

paper_content:

An architecture for reasoning with pattern-based if-then rules is proposed. By processing patterns as real-valued vectors and classifying similar if-then rules into clusters in long-term memory, the proposed system can store pattern-based if-then rules of propositional logic, including conjunctions, disjunctions, and negations. Moreover, it achieves some important properties for intelligent systems such as incremental learning, generalization, avoidance of duplicate results, and robustness to noise. Results of experiments demonstrate that the proposed method is effective for intelligent systems for solving various tasks autonomously in a real environment.
---
paper_title: Incremental clustering of mixed data based on distance hierarchy

paper_content:

Clustering is an important function in data mining. Its typical application includes the analysis of consumer's materials. Adaptive resonance theory network (ART) is very popular in the unsupervised neural network. Type I adaptive resonance theory network (ART1) deals with the binary numerical data, whereas type II adaptive resonance theory network (ART2) deals with the general numerical data. Several information systems collect the mixing type attitudes, which included numeric attributes and categorical attributes. However, ART1 and ART2 do not deal with mixed data. If the categorical data attributes are transferred to the binary data format, the binary data do not reflect the similar degree. It influences the clustering quality. Therefore, this paper proposes a modified adaptive resonance theory network (M-ART) and the conceptual hierarchy tree to solve similar degrees of mixed data. This paper utilizes artificial simulation materials and collects a piece of actual data about the family income to do experiments. The results show that the M-ART algorithm can process the mixed data and has a great effect on clustering.
---
paper_title: A semi-supervised incremental learning framework for sports video view classification

paper_content:

Sports videos have special characteristics such as well-defined video structure, specialized sports syntax, and typically having some canonical view types. In this paper, we propose a semi-supervised incremental learning framework for sports video view classification. Baseball is selected as an example to explain the main ideas. In order to obtain an optimal model based on a small number of pre-labeled training samples, the semi-supervised incremental learning framework explores the local distributed properties of the video sequences and sufficiently utilizes the information of a positive model pool and a negative model pool. After each round of online optimization process for the under-investigating video, a locally-optimized positive model and a set of negative models are added into the positive model pool and the negative model pool according to some heuristic criteria, respectively. Experiments results on real sports video data show that the proposed system is effective and promising.
---
paper_title: Semi-supervised Incremental Learning of Manipulative Tasks

paper_content:

For a social robot, the ability of learning tasks via human demonstration is very crucial. But most current approaches suffer from either the demanding of the huge amount of labeled training data, or the limited recognition cabability caused by very domain-specific modeling. This paper puts forward a semi-supervised incremental strategy for the robot to learn the manipulative tasks performed by the user. The task models are extended Markov models, taking a set of pre-learned object-specific manipulative primitives as basic states. They can be initialized with few labeled data, and updated continously when new unlabeled data is available. Furthermore, the system also has the capability to reject unlabeled observation as unseen tasks and detect a new task model from a group of them. Thus, using this strategy, the robot only needs human teaching at every beginning, then elaborate the learned tasks, and even extend task knowledge by its own observation. The experimental results in an office environment show the applicability of this approach.
---
paper_title: Semi-supervised incremental learning

paper_content:

The paper introduces a hybrid evolving architecture for dealing with incremental learning. It consists of two components: resource allocating neural network (RAN) and growing Gaussian mixture model (GGMM). The architecture is motivated by incrementality on one hand and on the other hand by the possibility to handle unlabeled data along with the labeled one, given that the architecture is dedicated to classification problems. The empirical evaluation shows the efficiency of the proposed hybrid learning architecture.
---
paper_title: Incremental learning for online face recognition

paper_content:

In this paper, a new approach to face recognition is presented in which not only a classifier but also a feature space of input variables is learned incrementally to adapt to incoming training samples. A benefit of this type of incremental learning is that the search for useful features and the learning of an optimal decision boundary are carried out in an online fashion. To implement this idea, an extended version of incremental principal component analysis (IPCA) and resource allocating network with long-term memory (RAN-LTM) are effectively combined. Using IPCA, a feature space is updated by rotating its eigen-axes and increasing the dimensions to adapt to a new training sample. In RAN-LTM, a small number of training samples called memory items are selected and they are utilized for retraining a classifier to realize an excellent incremental ability. To accommodate the classifier to the evolution of the feature space, we present a way to reconstruct the neural classifier without keeping all of the training samples given previously. In the experiments, the proposed incremental learning model is evaluated over a self-compiled face image database. As the result, we verify that the proposed model works well without serious forgetting and the test performance is improved as the learning stages proceed.
---
paper_title: Incremental Learning for Text Document Classification

paper_content:

This paper presents our research in incremental learning for text document classification. Incremental learning is important in text document classification since many applications have huge amount of training data, and training documents become available through time. We propose an incremental learning framework, ILTC(Incremental Learning of Text Classification) that involves the learning of features of text classes followed by an incremental Perceptron learning process. ILTC has the capabilities of incremental learning of new feature dimensions as well as new document classes. We applied the ILTC to a classification system of diagnostic text documents. The experiment results demonstrate that ILTC was able to incrementally learn new knowledge from newly available training data without either referring to the older training data or forgetting the already learnt knowledge.
---
paper_title: Incremental Learning From Stream Data

paper_content:

Recent years have witnessed an incredibly increasing interest in the topic of incremental learning. Unlike conventional machine learning situations, data flow targeted by incremental learning becomes available continuously over time. Accordingly, it is desirable to be able to abandon the traditional assumption of the availability of representative training data during the training period to develop decision boundaries. Under scenarios of continuous data flow, the challenge is how to transform the vast amount of stream raw data into information and knowledge representation, and accumulate experience over time to support future decision-making process. In this paper, we propose a general adaptive incremental learning framework named ADAIN that is capable of learning from continuous raw data, accumulating experience over time, and using such knowledge to improve future learning and prediction performance. Detailed system level architecture and design strategies are presented in this paper. Simulation results over several real-world data sets are used to validate the effectiveness of this method.
---
paper_title: Incremental learning for segmentation in medical images

paper_content:

Hand-coded vision systems are problematic in complex medical domains and are hard to change as new information emerges. Knowledge-engineering and machine learning approaches to intelligent vision systems also face learning bottlenecks. We have developed an approach to engineering vision systems, which allowed the user to make incremental changes to refine the performance of the system and address these limitations. A medical image segmentation system was built using this approach. In only a few hours of training, the system was able to exceed the performance of a similar hand-coded system built over a period of three months.
---
paper_title: Dynamic Weighting Ensembles for Incremental Learning

paper_content:

This paper investigates an interesting question of solving incremental learning problems using ensemble algorithms. The motivation is to help classifiers learn additional information from new batches of data incrementally while preserving previously acquired knowledge. Experimental results show that the proposed dynamic weighting scheme can achieve better performance compared to the fixed weighting scheme on a variety of standard UCI benchmark datasets.
---
paper_title: Incremental Learning of Chunk Data for Online Pattern Classification Systems

paper_content:

This paper presents a pattern classification system in which feature extraction and classifier learning are simultaneously carried out not only online but also in one pass where training samples are presented only once. For this purpose, we have extended incremental principal component analysis (IPCA) and some classifier models were effectively combined with it. However, there was a drawback in this approach that training samples must be learned one by one due to the limitation of IPCA. To overcome this problem, we propose another extension of IPCA called chunk IPCA in which a chunk of training samples is processed at a time. In the experiments, we evaluate the classification performance for several large-scale data sets to discuss the scalability of chunk IPCA under one-pass incremental learning environments. The experimental results suggest that chunk IPCA can reduce the training time effectively as compared with IPCA unless the number of input attributes is too large. We study the influence of the size of initial training data and the size of given chunk data on classification accuracy and learning time. We also show that chunk IPCA can obtain major eigenvectors with fairly good approximation.
---
paper_title: Incremental Learning of Concept Drift in Nonstationary Environments

paper_content:

We introduce an ensemble of classifiers-based approach for incremental learning of concept drift, characterized by nonstationary environments (NSEs), where the underlying data distributions change over time. The proposed algorithm, named Learn++.NSE, learns from consecutive batches of data without making any assumptions on the nature or rate of drift; it can learn from such environments that experience constant or variable rate of drift, addition or deletion of concept classes, as well as cyclical drift. The algorithm learns incrementally, as other members of the Learn++ family of algorithms, that is, without requiring access to previously seen data. Learn++.NSE trains one new classifier for each batch of data it receives, and combines these classifiers using a dynamically weighted majority voting. The novelty of the approach is in determining the voting weights, based on each classifier's time-adjusted accuracy on current and past environments. This approach allows the algorithm to recognize, and act accordingly, to the changes in underlying data distributions, as well as to a possible reoccurrence of an earlier distribution. We evaluate the algorithm on several synthetic datasets designed to simulate a variety of nonstationary environments, as well as a real-world weather prediction dataset. Comparisons with several other approaches are also included. Results indicate that Learn++.NSE can track the changing environments very closely, regardless of the type of concept drift. To allow future use, comparison and benchmarking by interested researchers, we also release our data used in this paper.
---
paper_title: A Probabilistic Scheme for Keyword-Based Incremental Query Construction

paper_content:

Databases enable users to precisely express their informational needs using structured queries. However, database query construction is a laborious and error-prone process, which cannot be performed well by most end users. Keyword search alleviates the usability problem at the price of query expressiveness. As keyword search algorithms do not differentiate between the possible informational needs represented by a keyword query, users may not receive adequate results. This paper presents IQP - a novel approach to bridge the gap between usability of keyword search and expressiveness of database queries. IQP enables a user to start with an arbitrary keyword query and incrementally refine it into a structured query through an interactive interface. The enabling techniques of IQP include: 1) a probabilistic framework for incremental query construction; 2) a probabilistic model to assess the possible informational needs represented by a keyword query; 3) an algorithm to obtain the optimal query construction process. This paper presents the detailed design of IQP, and demonstrates its effectiveness and scalability through experiments over real-world data and a user study.
---
paper_title: A survey of techniques for incremental learning of HMM parameters

paper_content:

The performance of Hidden Markov Models (HMMs) targeted for complex real-world applications are often degraded because they are designed a priori using limited training data and prior knowledge, and because the classification environment changes during operations. Incremental learning of new data sequences allows to adapt HMM parameters as new data becomes available, without having to retrain from the start on all accumulated training data. This paper presents a survey of techniques found in literature that are suitable for incremental learning of HMM parameters. These techniques are classified according to the objective function, optimization technique and target application, involving block-wise and symbol-wise learning of parameters. Convergence properties of these techniques are presented along with an analysis of time and memory complexity. In addition, the challenges faced when these techniques are applied to incremental learning is assessed for scenarios in which the new training data is limited and abundant. While the convergence rate and resource requirements are critical factors when incremental learning is performed through one pass over abundant stream of data, effective stopping criteria and management of validation sets are important when learning is performed through several iterations over limited data. In both cases managing the learning rate to integrate pre-existing knowledge and new data is crucial for maintaining a high level of performance. Finally, this paper underscores the need for empirical benchmarking studies among techniques presented in literature, and proposes several evaluation criteria based on non-parametric statistical testing to facilitate the selection of techniques given a particular application domain.
---

```
Title: Incremental Learning: Areas and Methods - A Survey

Section 1: INTRODUCTION
Description 1: This section provides an introduction to the problem of continuously evolving new data in data mining, the necessity of incremental learning, and an overview of its importance and current research trends.

Section 2: EXISTING LEARNING METHODS AND INCREMENTAL LEARNING
Description 2: This section discusses traditional learning methods, the evolution of incremental learning in these methods, and how incremental learning adapts to different applications and data.

Section 3: MATHEMATICAL REPRESENTATION AND ALGORITHM
Description 3: This section presents the mathematical representation of incremental learning and summarizes the process in an algorithm.

Section 4: CLUSTERING AND INCREMENTAL LEARNING
Description 4: This section explores the role of incremental learning in clustering, examining various algorithms and methods that handle evolving data without re-clustering.

Section 5: SUPERVISED AND SEMI-SUPERVISED INCREMENTAL LEARNING
Description 5: This section analyzes incremental learning in supervised and semi-supervised environments, detailing various methods, applications, and techniques that accommodate new training data over time.

Section 6: CURRENT SCENARIO
Description 6: This section reviews recent advancements in incremental learning, including techniques for handling data distributions, query formulation, and reinforcement learning in multi-agent scenarios.

Section 7: DISCUSSION AND CONCLUSION
Description 7: This section discusses the highlighted methodologies, areas for future research, the importance of feedback mechanisms, and the need for accuracy and effective decision-making in incremental learning.
```