Data Mining

Discovering hidden value in your data warehouse

Overview

Data mining, the extraction of hidden predictive information from large databases, is a powerful
new technology with great potential to help companies focus on the most important information
in their data warehouses. Data mining tools predict future trends and behaviors, allowing
businesses to make proactive, knowledge-driven decisions. The automated, prospective analyses
offered by data mining move beyond the analyses of past events provided by retrospective tools
typical of decision support systems. Data mining tools can answer business questions that
traditionally were too time consuming to resolve. They scour databases for hidden patterns,
finding predictive information that experts may miss because it lies outside their expectations.

The Foundations of Data Mining

Data mining techniques are the result of a long process of research and product development.
This evolution began when business data was first stored on computers, continued with
improvements in data access, and more recently, generated technologies that allow users to
navigate through their data in real time. Data mining takes this evolutionary process beyond
retrospective data access and navigation to prospective and proactive information delivery. Data
mining is ready for application in the business community because it is supported by three
technologies that are now sufficiently mature:

 Massive data collection

 Powerful multiprocessor computers
 Data mining algorithms

Commercial databases are growing at unprecedented rates. A recent META Group survey of data
warehouse projects found that 19% of respondents are beyond the 50 gigabyte level, while 59%
expect to be there by second quarter of 1996.1 In some industries, such as retail, these numbers
can be much larger. The accompanying need for improved computational engines can now be
met in a cost-effective manner with parallel multiprocessor computer technology. Data mining
algorithms embody techniques that have existed for at least 10 years, but have only recently been
implemented as mature, reliable, understandable tools that consistently outperform older
statistical methods.

In the evolution from business data to business information, each new step has built upon the
previous one. For example, dynamic data access is critical for drill-through in data navigation
applications, and the ability to store large databases is critical to data mining. From the user’s
point of view, the four steps listed in Table 1 were revolutionary because they allowed new
business questions to be answered accurately and quickly.
Evolutionary Business Question Enabling Product Characteristics
Step Technologies Providers

Data Collection "What was my total Computers, tapes, IBM, CDC Retrospective,
revenue in the last five disks static data
(1960s) years?" delivery

Data Access "What were unit sales Relational databases Oracle, Retrospective,
in New England last (RDBMS), Sybase, dynamic data
(1980s) March?" Structured Query Informix, delivery at
Language (SQL), IBM, record level
ODBC Microsoft

Data "What were unit sales On-line analytic Pilot, Retrospective,

Warehousing & in New England last processing (OLAP), Comshare, dynamic data
March? Drill down to multidimensional Arbor, delivery at
Decision Boston." databases, data Cognos, multiple levels
Support warehouses Microstrategy

(1990s)

Data Mining "What’s likely to Advanced Pilot, Prospective,

happen to Boston unit algorithms, Lockheed, proactive
(Emerging sales next month? multiprocessor IBM, SGI, information
Today) Why?" computers, massive numerous delivery
databases startups
(nascent
industry)

Table 1. Steps in the Evolution of Data Mining.

The core components of data mining technology have been under development for decades, in
research areas such as statistics, artificial intelligence, and machine learning. Today, the maturity
of these techniques, coupled with high-performance relational database engines and broad data
integration efforts, make these technologies practical for current data warehouse environments.

The Scope of Data Mining

Data mining derives its name from the similarities between searching for valuable business
information in a large database — for example, finding linked products in gigabytes of store
scanner data — and mining a mountain for a vein of valuable ore. Both processes require either
sifting through an immense amount of material, or intelligently probing it to find exactly where
the value resides. Given databases of sufficient size and quality, data mining technology can
generate new business opportunities by providing these capabilities:

 Automated prediction of trends and behaviors. Data mining automates the process of
finding predictive information in large databases. Questions that traditionally required
extensive hands-on analysis can now be answered directly from the data — quickly. A
typical example of a predictive problem is targeted marketing. Data mining uses data on
 past promotional mailings to identify the targets most likely to maximize return on
investment in future mailings. Other predictive problems include forecasting bankruptcy
and other forms of default, and identifying segments of a population likely to respond
similarly to given events.

 Automated discovery of previously unknown patterns. Data mining tools sweep

through databases and identify previously hidden patterns in one step. An example of
pattern discovery is the analysis of retail sales data to identify seemingly unrelated
products that are often purchased together. Other pattern discovery problems include
detecting fraudulent credit card transactions and identifying anomalous data that could
represent data entry keying errors.

Data mining techniques can yield the benefits of automation on existing software and hardware
platforms, and can be implemented on new systems as existing platforms are upgraded and new
products developed. When data mining tools are implemented on high performance parallel
processing systems, they can analyze massive databases in minutes. Faster processing means that
users can automatically experiment with more models to understand complex data. High speed
makes it practical for users to analyze huge quantities of data. Larger databases, in turn, yield
improved predictions.

The most commonly used techniques in data mining are:

 Artificial neural networks: Non-linear predictive models that learn through training and
resemble biological neural networks in structure.

 Decision trees: Tree-shaped structures that represent sets of decisions. These decisions
generate rules for the classification of a dataset. Specific decision tree methods include
Classification and Regression Trees (CART) and Chi Square Automatic Interaction
Detection (CHAID) .

 Genetic algorithms: Optimization techniques that use processes such as genetic

combination, mutation, and natural selection in a design based on the concepts of
evolution.

 Nearest neighbor method: A technique that classifies each record in a dataset based on a
combination of the classes of the k record(s) most similar to it in a historical dataset
(where k ³ 1). Sometimes called the k-nearest neighbor technique.

 Rule induction: The extraction of useful if-then rules from data based on statistical
significance.

Glossary of Data Mining Terms

analytical model A structure and process for analyzing a dataset. For example, a decision tree
is a model for the classification of a dataset.

anomalous data Data that result from errors (for example, data entry keying errors) or that
represent unusual events. Anomalous data should be examined carefully
because it may carry important information.
artificial neural Non-linear predictive models that learn through training and resemble
networks biological neural networks in structure.

CART Classification and Regression Trees. A decision tree technique used for
classification of a dataset. Provides a set of rules that you can apply to a new
(unclassified) dataset to predict which records will have a given outcome.
Segments a dataset by creating 2-way splits. Requires less data preparation
than CHAID.

CHAID Chi Square Automatic Interaction Detection. A decision tree technique used
for classification of a dataset. Provides a set of rules that you can apply to a
new (unclassified) dataset to predict which records will have a given
outcome. Segments a dataset by using chi square tests to create multi-way
splits. Preceded, and requires more data preparation than, CART.

classification The process of dividing a dataset into mutually exclusive groups such that
the members of each group are as "close" as possible to one another, and
different groups are as "far" as possible from one another, where distance is
measured with respect to specific variable(s) you are trying to predict. For
example, a typical classification problem is to divide a database of
companies into groups that are as homogeneous as possible with respect to a
creditworthiness variable with values "Good" and "Bad."

clustering The process of dividing a dataset into mutually exclusive groups such that
the members of each group are as "close" as possible to one another, and
different groups are as "far" as possible from one another, where distance is
measured with respect to all available variables.

data cleansing The process of ensuring that all values in a dataset are consistent and
correctly recorded.

data mining The extraction of hidden predictive information from large databases.

data navigation The process of viewing different dimensions, slices, and levels of detail of a
multidimensional database. See OLAP.

data The visual interpretation of complex relationships in multidimensional data.

visualization

data warehouse A system for storing and delivering massive quantities of data.

decision tree A tree-shaped structure that represents a set of decisions. These decisions
generate rules for the classification of a dataset. See CART and CHAID.

dimension In a flat or relational database, each field in a record represents a dimension.

In a multidimensional database, a dimension is a set of similar entities; for
example, a multidimensional sales database might include the dimensions
Product, Time, and City.
exploratory data The use of graphical and descriptive statistical techniques to learn about the
analysis structure of a dataset.

genetic Optimization techniques that use processes such as genetic combination,

algorithms mutation, and natural selection in a design based on the concepts of natural
evolution.

linear model An analytical model that assumes linear relationships in the coefficients of
the variables being studied.

linear regression A statistical technique used to find the best-fitting linear relationship
between a target (dependent) variable and its predictors (independent
variables).

logistic A linear regression that predicts the proportions of a categorical target

regression variable, such as type of customer, in a population.

multidimensional A database designed for on-line analytical processing. Structured as a

database multidimensional hypercube with one axis per dimension.

multiprocessor A computer that includes multiple processors connected by a network. See

computer parallel processing.

nearest neighbor A technique that classifies each record in a dataset based on a combination
of the classes of the k record(s) most similar to it in a historical dataset
(where k ³ 1). Sometimes called a k-nearest neighbor technique.

non-linear model An analytical model that does not assume linear relationships in the
coefficients of the variables being studied.

OLAP On-line analytical processing. Refers to array-oriented database applications

that allow users to view, navigate through, manipulate, and analyze
multidimensional databases.

outlier A data item whose value falls outside the bounds enclosing most of the other
corresponding values in the sample. May indicate anomalous data. Should
be examined carefully; may carry important information.

parallel The coordinated use of multiple processors to perform computational tasks.

processing Parallel processing can occur on a multiprocessor computer or on a network
of workstations or PCs.

predictive model A structure and process for predicting the values of specified variables in a
dataset.

prospective data Data analysis that predicts future trends, behaviors, or events based on
analysis historical data.

RAID Redundant Array of Inexpensive Disks. A technology for the efficient

 parallel storage of data for high-performance computer systems.

retrospective Data analysis that provides insights into trends, behaviors, or events that
data analysis have already occurred.

rule induction The extraction of useful if-then rules from data based on statistical
significance.

SMP Symmetric multiprocessor. A type of multiprocessor computer in which

memory is shared among the processors.

terabyte One trillion bytes.

time series The analysis of a sequence of measurements made at specified time

analysis intervals. Time is usually the dominating dimension of the data.