Last modified on 22 October 2013, at 10:39

Data Science: An Introduction/Exploratory Analysis


Chapter 25: Exploratory Analysis

CC-BY-SA icon.svg

Loupe light.svg








Sixthly, some syntax rules:

  • Please bold key terms and phrases the student should learn.


Chapter SummaryEdit

Wikipedia defines Exploratory data analysis(EDA) as an approach to analyzing data sets to summarize their main characteristics, often with visual methods. During EDA the data scientist is looking for patterns in the data with an open mind and is often described as 'digging into the data' or 'getting your hands dirty'. The results of this analysis can lead to the formulation of new hypotheses and to further data collection activities. It can also highlight outliers in the data that can inform data cleansing activities or even demonstrate systemic flaws in the data that may make a data set unusable. This chapter describes some of the more common techniques used in EDA


The American mathematician John Tukey coined the term EDA to the approach of analysing data for the purpose of formulating hypotheses worth testing as opposed to confirmatory data analysis where conventional statistical methods are used to test hypotheses. By getting insights from the data, EDA is able to suggest hypotheses about the causes of observed phenomena and allows the data scientist to assess their assumptions and select appropriate tools and techniques for further analysis. Essentially EDA is an approach to searching for patterns in the data with an open mind. Or as John Tukey put it: “If we need a short suggestion of what exploratory data analysis is, I would suggest that It is an attitude and a flexibility and some graph paper” (although these days a spreadsheet or R is an easier alternative)

With powerful computers and an arsenal of statistical tests it can be tempting to dive straight into a dataset and start crunching numbers without taking the time to pose the right question. An example of this was provided by the statistician Francis Anscombe (coincidentally the brother in- law of John Tukey) where 4 data sets (now known as Anscombe's quartet) have very nearly identical statistical properties yet appear very different when graphed:

Since EDA is exploratory by definition, it is to a certain extent a method of trial and error, and the particular analyses that prove useful will depend on the specific data set being investigated. Wikipedia provides a list of useful graphical techniques for EDA including, some of the most commonly used are outlined below

A box plot shows the quartiles of a numeric data series. The spacing between these quartiles indicate the spread and skewness in the data. Plotting several series on the same chart shows differences between series without making any assumptions about their underlying statistical distribution.

A Histogram is a graphical representation of the distribution of data, obtained by placing each data point into a set of discrete intervals (or bins) of equal size and calculating the total number of data points in each bin. There is no “best” number of bins so the data scientist may have to experiment with each data set to find the most appropriate bin width.

A Scatter Plot is used to explore potential relationships between two variables by plotting one variable on the horizontal X-axis and the other on the vertical Y-axis. This can suggest correlation between the two variables. A pattern of dots sloping from lower left to upper right suggests a positive correlation, while a pattern sloping from upper left to lower right suggest a negative correlation. A line of best fit (or ‘trendline’) can be calculated to assess this correlation. For a linear correlation this is known as linear regression

The Pareto Chart, named after the Italian economist Vilfredo Pareto, is a combined bar and line chart where individual values are shown in descending order by bars with the cumulative total overlaid as a line on top. It is used to identify the most important factors in the data

Letting Data Speak for ThemselvesEdit


More ReadingEdit


Copyright NoticeEdit


You are free:

  • to Remix — to adapt or make derivative works

Under the following conditions:

  • The author's moral rights;