Data Mining - Montefiore

desertcockatooData Management

Nov 20, 2013 (4 years and 3 months ago)



IEEE Computer Applications in Power, Volume 12, Number 3, July 1999, pages 19


Cristina Olaru and Louis Wehenkel

University of Liège, Belgium

‘Data Mining’ (DM) is a folkloric denomination of a complex activity which aims at extracting
hesized and previously unknown information from large databases. It denotes also a multidisciplinary field
of research and development of algorithms and software environments to support this activity in the context of
real life problems, where often huge a
mounts of data are available for mining. There is a lot of publicity in this
field and also different ways to see the things. Hence, depending on the viewpoints, DM is sometimes considered
as just a step in a broader overall process called Knowledge Discov
ery in Databases (KDD), or as a synonym of
the latter as we do in this paper. Thus, according to this less purist definition DM software includes tools of
automatic learning from data, such as machine learning and artificial neural networks, plus the tradi
approaches to data analysis such as query
reporting, on
line analytical processing or relational calculus, so
as to deliver the maximum benefit from data.

The concept was born about ten years ago. The interest in the data mining field and its ex
ploitation in
different domains (marketing, finances, banking, engineering, health care, power systems, meteorology,…) has
increased recently due to a combination of factors. They include:

the emergence of very large amount of data (terabytes



of data) due to computer automated
data measurement and/or collection, digital recording, centralized data archives and software and hardware

the dramatic cost decrease of mass storage devices

the emergence and quick growt
h of fielded data base management systems

the advances in computer technology such as faster computers and parallel architectures

the continuous developments in automatic learning techniques

the possible presence of uncertainty in data (noise, outliers, m
issing information).


The general purpose of data mining is to process the information from the enormous stock of data we
have or that we may generate, so as to develop better ways to handle data and support future decision making.
Sometimes, the patterns t
o be searched for, and the models to be extracted from data are subtle, and require
complex calculus and/or significant specific domain knowledge. Or even worse, there are situations where one
would like to search for patterns that humans are not well suit
ed to find, even if they are good experts in the
field. For example, in many power system related problems one is faced with high dimensional data sets that can
not be easily modeled and controlled on the whole, and therefore automatic methods capable of s
structures from such data become a necessity.

This article presents the concept of data mining and aims at providing an understanding of the overall
process and tools involved: how the process turns out, what can be done with it, what are the
main techniques
behind it, which are the operational aspects. We aim also at describing a few examples of data mining
applications, so as to motivate the power system field as a very opportune data mining application. For a more in
depth presentation of da
ta mining tools and their possible applications in power systems we invite the reader to
have a look at the references indicated for further reading.

Data Mining process

By definition, data mining is the nontrivial process of extracting valid, previously
comprehensible, and useful information from large databases and using it. It is an exploratory data analysis,
trying to discover useful patterns in data that are not obvious to the data user.

What is a database (DB)? It is a collection of objects
(called tuples in the DB jargon, examples in
machine learning, or transactions in some application fields), each one of which is described by a certain number
of attributes, which provide detailed information about each object. Certain attributes are selec
ted as input
attributes for a problem, certain ones as outputs (i.e. the desired objective: a class, a continuous value…). Table
1 shows some examples of hourly energy transactions recorded in a data base for a power market analysis
application (each line
of the table corresponds to an object and each column indicates one attribute, e.g. buyer,
quantity, price). In such an application, the power system is considered as a price
based market with bilateral
contracts (i.e. direct contracts between the power pr
oducers and users or broker outside of a centralized power


pool), where the two parties, the buyer and the seller, could be utility distribution companies, utility and non
utility retailers (i.e. energy service providers), independent generators (i.e. inde
pendent power producers),
generation companies, or end customers (as single customers or as parts of aggregated loads). The example will
be used further in the presentation in order to exemplify the techniques implied in data mining.

Usually, one of the f
irst tasks of a data mining process consists of summarizing the information stored in
the database, in order to understand well its content. This is done by means of statistical analysis or query
reporting techniques. Then more complex operations are i
nvolved such as to identify models which may be used
to predict information about future objects. The term supervised learning (known as “learning with a teacher”) is
implied in mining data in which for each input of the learning objects, the desired outpu
t objective is known and
implicated in learning. In unsupervised learning approaches (“learning by observation”) the output is not
provided or not considered at all, and the method learns by itself only from input attribute values.






Product / Service


Unitary Price

(price units)

Transaction Price

(price units)



23 Feb. 1998

9 a.m.


20 MWh





23 Feb. 1998

11 a.m


50 MWh





5 Apr. 1998

9 a.m.


30 MWh





9 Apr.


2 p.m.

Spinning Reserve

10 MW





15 May 1998

4 a.m


30 MWh





15 May 1998

5 a.m.

Spinning Reserve

20 MW





31 July 1998

8 a.m.

Spinning Reserve

10 MW



Table 1. Example of a data base.

that generally only about 10% of the total collected data is ever analyzed (not only by means of
data mining). Many companies realize the poor quality of their data collection only when a data mining analysis
is started on it. The databases are usually ver
y expensive to create and expensive to maintain, and for a small
additional investment in mining them, highly profitable information may be discovered hidden in the data. Thus,
the classical scenario is as follows: a company realizing that there might be “
nuggets” of information in the data
they process starts by building a long term repository (a data warehouse) to store as much as possible data (e.g.
by recording systematically all purchases by individual customers of a supermarket); then they would launc
h a
pilot DM study in order to identify actual opportunities; finally some of the applications identified as interesting
would be selected for actual implementation.


However, apart from the “cheaply” collected or already available data, there are some app
lications of
data mining where the data is produced by computer simulations or expensive real experiments. For example, in
the case where future yet unknown situations have to be forecasted, or in fields where security aspects are
analyzed for a system (co
mputer system, power system, or banking system) when the history does not provide
fortunately negative examples, one may use Monte
Carlo simulations in order to generate a DB automatically.
Note that this is itself a non trivial task, but we will not furth
er elaborate on it in this paper.

Figure 1. Data mining process

The usual film when a company or a holder of a big amount of data decides that the information he has
collected is worth to be analyzed unfolds like this: he comes with the data to the data
miner (e.g. a consultant),
the data miner first gets familiar with the field of application and with the application specifics, then depending
on the data mining software he has, he will select a portion of the available data and apply those techniques he
expects to give him more knowledge in terms of some established objectives. In the case the results of this
combination of tools does not give to the interested one any improvement in the existing knowledge about the
subject, either the miner gives up (it
is indeed possible that this process yields only uninteresting results), or he
tries to go further by implementing new customized methods for mining the specific data (e.g. for a temporal
problem of early anomalies detection, a temporal decision tree may o
ffer more valuable results than a decision

What is a data miner?

some person, usually with background in computer science or in statistics

in the
domain of interest, or a couple of two specialists, one in data mining, one in the domain of int
erest, able to
perform the steps of the data mining process. The miner is able to decide how much iterative to be the whole
process and to interpret the visual information he gets at every sub

In general the data mining process iterates through fiv
e basic steps:

Data selection.

This step consists of choosing the goal and the tools of the data mining process, identifying
the data to be mined, then choosing appropriate input attributes and output information to represent the task.


Data transformation.

Transformation operations include organizing data in desired ways, converting one
type of data to another (e.g. from symbolic to numerical) defining new attributes, reducing the
dimensionality of the data, removing noise, “outliers”, normalizing, if appro
priate, deciding strategies for
handling missing data.

Data mining step per se.

The transformed data is subsequently mined, using one or more techniques to
extract patterns of interest. The user can significantly aid the data mining method by correctly per
the preceding steps.

Result interpretation and validation.

For understanding the meaning of the synthesized knowledge and its
range of validity, the data mining application tests its robustness, using established estimation methods and
unseen data

from the data base. The extracted information is also assessed (more subjectively) by comparing
it with prior expertise in the application domain.

Incorporation of the discovered knowledge.

This consists of presenting the results to the decision maker

may check/resolve potential conflicts with previously believed or extracted knowledge and apply the
new discovered patterns.

Figure 2. Software structures: a) DM in place; b) DM offline.

Figure 1 is presenting schematically the whole process, by showing

what is happening with the data: it is
processed, mined and post
processed, the result being a refinement in the knowledge about the application.
The data mining process is iterative, interactive, and very much a trial and error activity.

on plays an important role. Because we find it difficult to emulate human intuition and
making on a machine, the idea is to transform the derived knowledge into a format that is easy for
humans to digest, such as images or graphs. Then, we rely on

the speed and capability of the human user visual
system to spot what is interesting, at every step of the data mining process: preliminary representation of data,
domain specific visualization or result presentation.

From the point of view of software st
ructure, there are two types of possible implementations:


the one represented in figure 2a, called data mining “in place”: the learning system accesses the data through
a data base management system (DBMS) and the user is able to interact with both the dat
abase (by means of
queries) and the data mining tools. The advantage is that the approach may handle very large databases and
may exploit the DBMS facilities (e.g. the handling of distributed data).

the one called data mining “offline” shown in figure 2b:
the objects are first loaded in the data mining
software, with a translation into a particular form, outside the data base, and the user is interacting mainly
with the data mining software. They allow to use the existing machine learning systems with only
modifications in implementation, and may be faster but are generally limited to handle medium sized data
sets which can be represented in main memory (up to several hundred Mbytes).

What can be done at the Data Mining step?

Depending mainly on the a
pplication domain and on the interest of the miner, one can identify several
types of data mining tasks for which data mining offers possible answers. We present them in the order they are
usually implied in the process. Possible results for each one of th
ese tasks are provided by considering the
example in table 1 as the database to be mined:


It aims at producing compact and characteristic descriptions for a given set of data. It can take
multiple forms: numerical (simple descriptive statist
ical measures like means, standard deviations…), graphical
forms (histograms, scatter plots…), or the form of “if
then” rules. It may provide descriptions about objects in
the whole data base or in selected subsets of it.
Example of summarization: “the min
imum unitary price for all
the transactions with energy is 70 price units”
(see table 1).


A clustering problem is an unsupervised learning problem which aims at finding in the data clusters
of similar objects sharing a number of interesting pro
perties. It may be used in data mining to evaluate
similarities among data, to build a set of representative prototypes, to analyze correlations between attributes, or
to automatically represent a data set by a small number of regions, preserving the topol
ogical properties of the
original input space.
Example of a clustering result: “from the seller B point of view, buyers A and E are similar
customers in terms of total price of the transactions done in 1998”.



A classification problem is a s
upervised learning problem where the output information is a
discrete classification, i.e. given an object and its input attributes, the classification output is one of the possible
mutually exclusive classes of the problem. The aim of the classification t
ask is to discover some kind of
relationship between the input attributes and the output class, so that the discovered knowledge can be used to
predict the class of a new unknown object.
Example of a derived rule, which classifies sales made early in the
ay (a sale is said to be early if it was made between 6 a.m. and 12 a.m.): “if the product is energy then the sale
is likely to be early (confidence 0.75)”.

A regression problem is a supervised learning problem of building a more or less trans
parent model,
where the output information is a continuous numerical value or a vector of such values rather than a discrete
class. Then given an object, it is possible to predict one of its attributes by means of the other attributes, by using
the built m
odel. The prediction of numeric values may be done by classical or more advanced statistical methods
and by “symbolic” methods often used in the classification task.
Example of a model derived in a regression
problem: “when buyer A buys energy, there exist
s a linear dependence between the established unitary price
and the quantity he buys:


Dependency modeling.

A dependency modeling problem consists in discovering a model which describes
significant dependencies among attributes. Th
ese dependencies are usually expressed as “if
then” rules in the
form “if antecedent is true then consequent is true”, where both the antecedent and the consequent of the rule
may be any combination of attributes, rather than having the same output in the
consequent like in the case of the
classification rules.
Example: such a rule might be “if product is energys then transaction price is larger than
2000 price units”.

Deviation detection.

This is the task focusing on discovering the most significant change
s or deviations in the
data between the actual content of the data and its expected content (previously measured) or normative values.
It includes searching for temporal deviations (important changes in data with time), and searching for group
deviations (
unexpected differences between two subsets of data). In our example, deviation detection could be
used in order to find main differences between sales patterns in different periods of the year.

Temporal problems.

In certain applications it is useful to pro
duce rules which take into account explicitly the
role of time. There are data bases containing temporal information which may be exploited by searching for


similar temporal patterns in data or learn to anticipate some abnormal situations in data.
: “a customer
buying energy will buy spinning reserve later on (confidence 0.66)”, or “if total quantity of daily transactions is
less than 100 price units during at least 1 month for a client, the client is likely to be lost”.

Causation modeling.

This is
a problem of discovering relationships of cause and effect among attributes. A
causal rule of type “if
then” indicates not only that there is a correlation between the antecedent and the
consequent of the rule, but also that the antecedent causes the conse
Example: “decreasing energy price
will result in more sold energy daily”.

What techniques are behind all these tasks?

The enumerated types of data mining tasks are based on a set of important techniques originating in artificial
intelligence paradi
gms, statistics, information theory, machine learning, reasoning with uncertainty (fuzzy sets),
pattern recognition, or visualization. Thus, a data mining software package is supported to varying degrees by a
set of technologies, which nearly always includ

Tree and rule induction.
Machine learning (ML) is the center of the data mining concept, due to its
capabilities to gain physical insight into a problem, and participates directly in data selection and model
search steps. To address problems like class
ification (crisp and fuzzy decision trees), regression (regression
tree), time
dependent prediction (temporal trees), ML field is basically concerned with the automatic design
of “if
then” rules similar to those used by human experts. Decision tree inducti
on, the best known ML
framework, was found to be able to handle large scale problems due to its computational efficiency, to
provide interpretable results and in particular, able to identify the most representative attributes for a given

Association rule generators are a powerful data mining technique used to search through
an entire data set, for rules revealing the nature and frequency of relationships or associations between data
entities. The resulting associations can be used t
o filter the information for human analysis and possibly to
define a prediction model based on observed behavior.

Clustering methods.
They are used often in the data selection pre
processing step, due to the property of
learning unsupervised similarities
between objects and reducing the search space to a set of most important


attributes for the application, or to a finite set of objects “alike”. The most frequently used clustering method
is the
method which identifies a certain number of groups of
similar objects; it may be used in
combination with the
neighbor rule

which classifies any new object in the group most similar (most
near) to it. This method may also be used in order to identify outliers in a data base. For example, using this
echnique it might be possible in our example to identify groups of similar sales (large quantity & cheap
unitary price versus small quantity & expensive unitary price) and to find out that some of the sales are
outliers (e.g. small quantity & cheap). Then
a supervised learning technique might be used in order to find a
rule to characterize these abnormal sales, in terms of attributes (seller, buyer, product, date…).

Artificial neural networks.

They are recognized in the automatic learning framework as “univ
approximators”, with massively parallel computing character and good generalization capabilities, but also
as black
boxes due to the difficulty to obtain insight into the relationship learned. They are used within the
data mining step: to generate a
regression model that can predict future behavior, on the basis of a data base
with input

output pairs of continuous numerical historical information (the neural network acts like a
mapping, associating numerical outputs to any new object of known attrib
utes values), and to automatically
represent a data set by a small number of representative prototypes, preserving the topological properties of
the original attribute space (unsupervised learning).

Statistical techniques
such as linear regression, discri
minant analysis, or statistical summarization…

Visualization techniques:
histograms (estimate the probability distribution for a certain numerical attribute
given a set of objects), scatter plots

(provide information on the relation between two numerical
and a discrete one), three
dimensional maps, dendrograms (a correlation analyses between attributes or

In addition, some DM packages include:
genetic algorithms

(optimization techniques based on the concepts
of genetic combination, mut
ation and natural selection),
sequential patterns

discovery (group objects with the
same succession of given attribute values over a time period),
series similarity

(detect similar time
over a period of time),
Bayesian belief networks
l models that encode probabilistic relationships among
variables of interest, systems able to learn causal relationships),

systems (fuzzy inference systems
that incorporate the learning and generalization abilities of neural networks).


Even if w
e like to consider data mining tools like toolboxes of multiple techniques able to perform a
complete data analysis, the reality is not yet so, the market offering presently only partially equipped products.

DM techniques are different one from another in

terms of problem representation, parameters to optimize,
accuracy, complexity, run time, transparency, and interpretability. Making a compromise between accuracy and
complexity (by means of pruning techniques), enhancing the comprehensibility of derived p
atterns, and fighting
to avoid the overfitting (a problem which appears when the model to be extracted is too complex with respect to
the information provided in the learning set) are common features for all the techniques.

Operational aspects

The succes
s of mining some data is induced by a list of factors:

The right tools.
A distinctive feature of a data mining software is the quality of its algorithms, the effectiveness
of the techniques, and sometimes their speed. In addition, the efficiency of using
the hardware, the operating
system, the database resources and the parallel computing are influencing the process. Moreover, it turns out that
the particular set of tools useful in a given application are highly dependent on the practical problem. Thus at

the prototyping step, it is useful to have available a broad enough set of techniques so as to identify interesting
applications. However, in the final product used for actual field implementation it is often possible to use only a
small subset of the lat
ter tools. Customizing data mining techniques to the application domain and using methods
that are reliable means to the proposed goal may enhance the process of extracting useful information.

The right data.
The data to be mined should contain information

worth mining: consistent, cleaned,
representative for the application. Of course, it is useless to apply data mining to an invalid data base with high
measurement or estimation data errors, or to try to precisely estimate numerical outputs which present a

high noise. A data mining tool ideally explains as much information as is stored in the data which is mined (a
derived model is strongly dependent on the learning set used), and sometimes it is not what is in the data that
matters for an applicatio
n (wrong attributes, wrong selected sample).

An important part of data mining result errors are due to uncertainties in modeling and generation of
objects in certain data bases in discordance with the real probabilities of phenomena appearances in the syst


That is why the data mining errors often do not have a meaning by themselves, they just provide a practical
means to compare efficiencies of different criteria applied to the same data base.

The right people.
Regardless of what many producers of data m
ining tools claim, data mining is not (yet) an
“automatic” operation with little or no human intervention. On the contrary, the human analyst plays an
important role, mostly in the areas of data selection and data / knowledge interpretation. The data miner

have an understanding of the data under analysis and the domain or industry to which it pertains. It is more
important for the mining process to embrace the problems of the application meant to solve, than to incorporate
in the data mining software

the hottest technologies.

The right application.
Almost always a problem well posed is already a partially solved problem. It is
important to clearly define the goals and choose the appropriate objectives so as to yield a significant impact on
the under
lying decision making process.

The right questions.
An important issue: how does the data miner structure a data analysis problem so that the
right question can be asked, knowing how easy and useless it is to give the right answer to the wrong question?

The right sense of uncertainty.
Data miners are more interested in understandability than accuracy or
predictability per se. Often, even the best methods of search will leave the data miner with a range of
uncertainties about the correct model or the corre
ct prediction.

Common applications of data mining

Data mining approach has a major advantage from the point of view of its applicability: almost all the
domains of human activity may benefit from it, both the ones where a lot of data is already available
and the
ones where the data have to be simulated in order to extract some more profitable knowledge concerning the
field. We mention further some particular broad domains of interest in present data mining applications.

Market basket analysis

refers to the

process of examining point
sale data to identify affinities between
products and services purchased by a customer. Data mining must deal in these applications with large volumes
of transactional and spread data and must be performed in a time interval
that will allow an organization to
respond to market opportunity before competition does. Data mining techniques like association rules and


sequential patterns discovery are involved in the automatic identification of important buying patterns, types of
nsumers exhibiting such patterns, customer characteristics that may be correlated to the consumer’s choices.

Customer segmentation

is the process of analyzing data about customers or general consumers to identify
characteristics and behaviors that can be e
xploited in the market place. Clustering, statistical analysis, deviation
detection and modeling are implicated in reducing the customer attrition phenomenon, i.e. the loss of customers
(searching for customers that exhibit characteristics typical of som
eone who is likely to leave for a competing
company), or in target marketing (attraction of others customers, identification of the risk associated with

Fraud detection.

Data mining applications have demonstrated their benefits in the areas
where many actions
(transactions) are undertaken, making the respective system vulnerable to fraud: credit card services,
telecommunications, computer systems…

Detection of patterns in text, image, on the world wide web

are broadly extensive areas of DM ap
due to the impressive amount of information available: finding associations amongst the keywords labeling items
in a collection of textual documents, recognizing actions in video image sequences, helping users locate desired
information in the w

Medical diagnosis

through means of data mining are intended to be helpful tools that can improve the
physicians’ performance and make the diagnosis process more objective and more reliable. From the
descriptions of the patients treated in the past for
which the final diagnosis were verified, diagnosis rules may be
automatically derived by means of clustering, machine learning, association rules, although the technology is not
widely accepted in medical practice, encountering a resistance of the physicia
ns to new diagnostic technology.

What about data mining in power systems?

Why would data mining tools be useful in the power system field? Like many other application areas,
the power system field is presently facing an explosive growth of data. In power
systems, irrespectively of the
particular application, there are three main sources of data: (i) field data, collected by various devices distributed
throughout the system, such as digital records; (ii) centralized data archives, such as those maintained b
y control
center SCADA systems; (iii) data from simulations, carried out in planning or operation environments.


In a power system there are a few DM related aspects: large scale character of power systems (thousands
of state variables), temporal (from mill
iseconds to minutes, hours, weeks, years) and statistical nature of data,
existence of a discrete (e.g. events such as topology changes or protection arming) and continuous (analog state
variables) information mixture, necessity of communication with exper
ts through means of visualization, on
operation time restrictions for fast decision making, existence of uncertainty (noise, outliers, missing

Engineers trying to solve power system related problems should look at the whole tool
box of d
mining methods, and not hesitate to combine different techniques to yield a full, practical solution. Data
selection step may be performed with a decision tree, a clustering approach, or a correlation analysis and later
on, the result may serve as inpu
t for other supervised techniques, possibly with the problem decomposed into
simpler sub

There are three dimensions along which data mining may complement classical system theory oriented
methods for power systems:

Computational efficiency.

By us
ing synthetic information extracted by DM, instead of numerical methods,
much higher speed may be reached for real
time decision making. Further, in terms of data requirements,
DM may require only significant and/or available input parameters data base, in
stead of a “full” description
of the system model.

Anticipative physical insight
. The present day practice generally handles new problems

that some
undesirable consequences have already been observed on the system. Carrying out DM studies will allow
engineers to have a more anticipative view on potential problems.

Management of uncertainties.

The behavior of a power system will always have some unexpected
experiences (e.g. a relay which mis
operated, an operator which did not behave as expected, a

which was different from prescriptions, a load which were modeled inappropriately…). DM copes with this
problem by making use of more simulations carried out by relaxing assumptions on the dynamic models

For Further Reading


U. M. Fayyad,
G. Piatetsky
Shapiro, P. Smyth, and R. Uthurusamy (editors),

Advances in knowledge
discovery and data mining
, Menlo Park / Cambridge, AAAI Press / MIT Press, 1996.

U. M. Fayyad, Data Mining and Knowledge Discovery: Making Sense Out of Data
, IEEE Expert,
telligent Systems and their Applications
, pages 20
25, October 1996.

C. Glymour, D. Madigan, D. Pregibon, P. Smyth, Statistical Themes and Lessons for Data Mining,
Mining and Knowledge Discovery journal
, volume , number 1, pages 11
28, Kluwer Academic
, 1997.

R. S. Michalski, I Bratko and M. Kubat,
Machine Learning and Data Mining. Methods and
, Chichester, Willey, 1998.

E. Simoudis, Reality check for data mining,
IEEE Expert, Intelligent Systems and their Applications
pages 26
33, October

L. Wehenkel,
Automatic learning techniques in power systems
, Boston, Kluwer Academic, 1998.


Cristina Olaru
received her diploma in power system engineering and her MS from the Politehnica University
of Bucharest, Romania. She is pursuing P
h.D. studies at the University of Liège, Belgium, in the Department of
Electrical and Computer Engineering. Her main research interests are in fuzzy decision tree induction and its
application in power systems operation and control.

L. Wehenkel

was born in

Nürnberg, Germany, in 1961. He received the Electrical (Electronics) engineering
degree in 1986 and the Ph.D. degree in 1990 both from the University of Liège, Belgium, where he is presently
professor of electrical engineering. His research interests lie
mainly in artificial intelligence and stochastic
methods, in particular automatic learning and data mining, and their applications to complex and uncertain
systems like electric power systems.