Saturday, November 12, 2022
AI and Software Testing - ppt download
AI and Software Testing - ppt download: ABSTRACT This presentation gives an Introduction to Artificial Intelligence; and how AI touches our lives on day to day basis. It talks about the usage of AI in QA. It explains that how to deploy AI Systems for QA, and what benefits can be expected out of this. This will help Test Managers to decide if using AI techniques for software testing; will be helpful to their organization
Wednesday, January 25, 2012
Introduction to RFM Analysis(Part - I)
RFM is a method used for analyzing customer behavior and defining value based market segments. It is commonly used in database marketing and direct marketing.
– RFM stands for : Recency, frequency, monetary
– RFM is a most commonly used method for analyzing customer behavior and defining market segments.
– Has been used for over 50 years by direct marketers to target a subset of their customers, save mailing costs, and improve profits.
– Based on three simple customer attributes:
• Recency of Purchase: When was the last customer interaction?
• Frequency of purchase: How frequent was the customer in its interactions with the business?
• Monetary value of Purchase: How much the customer is spending?
– Each one of these variables is having relationship with response to an offer
– Customers who have recently made a purchase are likely to purchase again.
– Customers who make frequent purchases are likely to purchase again.
– Customers who have spent a lot of money in the past are likely to spend more money now
– The three attributes: Recency, Frequency, Monetary are easy to comprehend and quite powerful in their predictive ability
RFM Methodology
Saturday, July 9, 2011
Friday, July 8, 2011
Thursday, June 16, 2011
What is the difference between linear regression and logistic regression?
Linear regression analyzes the relationship between two/more variables. For each subject (or experimental unit), one knows independent variables(X’s) and dependent variable(Y) and one wants to find the best straight line through the data points. In some situations, the slope and/or intercept have a scientific meaning(for example, in a log – linear model like lnY=β0+ β 1(lnX1)+ β2(lnx2) + ……….+ βn(lnxn) + Error, β 1 gives the elasticity of Y with respect to X, i.e., the percentage change in Y with respect to percentage change in X. This equation is also known as the constant elasticity form as in this equation, the elasticity of y with respect to changes in x as δ lny/δ lnxn = βn, which does not vary with xn . This log-linear form is often used in models of demands and production.). In other cases, one generally uses the linear regression line as a standard curve to find the values of Y from X’s.
Generally, a linear regression line has an equation of the form Y = a + bX, where X is the explanatory variable and Y is the dependent variable. The slope of the line is b, and a is the intercept (the value of y when x = 0).
Logistic regression is quite different than linear regression, the predictor is continuous, but the response is categorical or dichotomous (only 2 options). The logistic regression provides the probability of an event occurring. For every one unit increase in the predictor/s, changes the probability of the occurrence of an event.
For example, one might be interested to look at the relationship between body size and weight of different mammals, so one would use linear regression (please note that the relationship might not actually be linear). However, one might want to look at the relationship between body size of different birds and their chance of surviving a winter storm, so one predicts the body size of birds that do and don't survive the winter storm, and generate a logistic regression that gives the probability that a bird will survive if it has a certain body size. One can also have multiple logistics regression, so one can see what the probability is that bird will survive at a given body size and given wing span.
Logistic regression belongs to the class of 'discrete analysis' techniques. Predicting outcomes that are yes/no or one-of (a,b or c) is very different from predicting outcomes that are simple measures, such as revenue.
For modeling percentage data, Logistic regression is a better option. This is true because percentage is a simple and convenient way to represent binomial data, and logistic regression (not linear regression) should be used for binomial data.
When linear regression is used for binary data, there are three problems:
ü The variance of the error term is not constant,ü The error term is not normally distributed,
The first problem can be handled by using weighted least-square regression. When the sample size is very large, the method of least squares provides estimators that are asymptotically normal under fairly general regulations, even when the distribution of the error term is far from normal. But the third problem is insurmountable.
Logistic regression may seem much more complicated than its linear counterpart. Though most of the statistical software packages can do logistic regression with no more effort than linear regression, it is not as easy and straightforward to interpret the coefficients and test for goodness of fit of logistic models.
References:
Friday, June 10, 2011
Wednesday, June 8, 2011
Thursday, June 2, 2011
Tuesday, May 31, 2011
Wednesday, May 25, 2011
Saturday, January 1, 2011
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part –VI)
Computing scores and posterior probabilities using the linear discriminant function coefficients
t = subscript to distinguish the groups
n = number of quantitative variables
C = constant
L = n-by-1 vector of coefficients for the linear discriminant function
X = n-by-1 observation vector
The equation to obtain the score for Group t on a new observation X using METHOD=NORMAL
For example, here are the linear discriminant function coefficients(please refer the link: http://support.sas.com/onlinedoc/913/getDoc/en/statug.hlp/discrim_sect29.htm)
Linear Discriminant Function for Corp:
For example, say a new set of values for x1 though x4 is (16 30 35 31). To classify this observation into a group, you compute the posterior probability of this observation belonging to each group. The observation is classified into the group for which it has the largest probability. To compute the probabilities, you first need to computes the score on each group.
For example, the score on the Clover group is
Clover_score = -10.98457 + 0.08907*16 + 0.17379*30 + 0.11899*35 + 0.15637*31
PROC DISCRIM doesn't report the scores anywhere because their magnitudes are meaningless as well as not interpretable. The LIST and TESTLIST options print the posterior probabilities in the printed output, the OUT= and TESTOUT= options save them to a data set.
The posterior probabilities are computed as
where denominator = eclover_score + eCorn_score + eCotton_score + eSoybeans_score + eSugarbeets_score
Computing scores and posterior probabilities using the quadratic discriminant function coefficients
t = subscript to distinguish the groups
n = number of quantitative variables
C = constant
L = n-by-1 vector of coefficients for the linear discriminant function
Q = n-by-n matrix of coefficients for the quadratic discrim function
X = n-by-1 observation vector
The equation to obtain the score for Group t on a new observation X using METHOD=NORMAL POOL=NO or POOL=TEST with a significant p-value is:
For example, here is how to obtain the score on one group(Please refer the link: http://support.sas.com/onlinedoc/913/getDoc/en/statug.hlp/discrim_sect28.htm)
For example, say there is new set of observation
SepalLength = 80
SepalWidth = 30
PetalLength = 25
PetalWidth = 16
In PROC IML, it would look like
proc iml;
reset print;
const=-75.8208;
linear = {0.737 1.325 0.623 0.966};
quadratic = {-0.053 0.017 0.050 -0.009,
0.017 -0.079 -0.006 0.042,
0.050 -0.006 -0.067 0.014,
-0.009 0.042 0.014 -0.097};
obs = {80 30 25 16};
score = const + linear*obs` + obs*quadratic*obs`;
quit;
Note: in case you don’t have IML, you need to write the equation manually to compute the scores
For more details, please refer the link: http://support.sas.com/onlinedoc/913/getDoc/en/statug.hlp/discrim_sect17.htm
The posterior probability of x belonging to group t is then equal to
The discriminant scores are -0.5 Du2(x).
So to compute the 3 posterior probabilities for this new observation on the three groups(please refer the example mentioned at: http://support.sas.com/onlinedoc/913/getDoc/en/statug.hlp/discrim_sect28.htm), Setosa, Versicolor, and Virginica, first compute the score for each group as above. Then each posterior probability is computed from dividing the exponentiated score by the sum of all the three exponentiated scores.
Reference:
Wednesday, December 29, 2010
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part – V)
Data preparation for the Analysis
Preparing data for Discriminant Analysis is an important step.
Missing Imputation and normality of dependent variable
All the independent variables and the dependent variable that go into the model should not have any missing values. If the dependent variable has a missing value for an observation, which is rarely the case, then it should be discarded.
For the independent variables, the missing values are replaced / imputed. Some of the commonly used imputation techniques include
- Replacing by median,
- Replaced by mode,
- Replaced by 0
- Replaced by other logical values.
Median is preferred to mean because it is not impacted by extreme values. Mode is used when the variable is discrete or categorical, where the mode is the most frequently occurring value. 0 is mostly used for indicator/dummy/binary variables. Other logical values can also be used based on their business implications.
Outlier Treatment
A single observation that is substantially different from all other observations can make a large difference in the results of the Discriminant analysis. If a single observation (or small group of observations) substantially changes the results, one would want to know about this and investigate further.
High values are known as upper outliers and low ones are known as lower outliers. Such values should be modified or else they would bias the estimation of the model parameters. The most simple and commonly used outlier treatment technique is by capping the values which are above 99% or below 1% of the population. This means that if a value is above the 99th percentile then it is replaced by the value corresponding to the 99th percentile. Similar capping is done for values below 1st percentile.
Use the entire data for identifying attributes pertaining to group differences and comparison.
Training & Validation data
Once the data preparation is done, the entire population is split into modeling and validation population. This split is done in a random way, so that the distribution of dependent/class variable for both these samples is roughly the same. It is assumed that the characteristics of the independent variables would be similar in the two samples, as it is a random split. The modeling population is used to build the model and then it is implemented on the validation population. The performance of a model should be similar in both modeling and validation samples.
- Split the Sample data into two data sets viz. Training sample and validation/Holdout sample
- Split ratio can be 50-50 or 60-40 depending on the sample size.
- Ensure that equal proportion of the grouping variable (categorical variable) has been put under Training and Holdout samples.
Training sample: The data set used to compute the Discriminant function.
Validation/Holdout sample: The data set used to validate the accuracy of classification (prediction) based on the function computed using the training data set.
Selection of variables for analysis(reduction of variables)
- Dependent variable: The grouping variable could be >= 2 group
- Independent variable: Selected based on
a) Apriori Business understanding or logic.
b) Ensure there is no multicollinearity among the selected independent variables.
c) Stepwise Discriminant Analysis
a. Could be performed to understand the set independent variables (discriminators) having high discriminating power among the set of large independent variables.
b. The selection of the independent variables in the function is based on the individual F-value. Hence, check for the independent variable(s) that have been excluded from the Discriminant function as the data was not supporting but, the variable still makes logical sense.
c. SAS Code:
PROC STEPDISC data = <libname.data set-name> ;
CLASS < grouping variable>;
VAR;
RUN;
CLASS < grouping variable>;
VAR
RUN;
For details about STEPDISC Procedure Options, please see the link below:
Discriminant Analysis (Direct Method)
Once the first cut variable selection has been done using Stepdisc, one can use proc discrim for Discriminant function
PROC DISCRIM DATA= <libname.data set-name> POOL= YES CANONICAL CROSSVALIDATE ANOVA MANOVA LIST OUTSTAT = <filename>;
CLASS < grouping variable>;
VAR <Independent variables>;
PRIORS PROP;
RUN;
CLASS < grouping variable>;
VAR <Independent variables>;
PRIORS PROP;
RUN;
Some useful options and their meaning:
- PROC DISCRIM = Procedure for Linear Discriminant Analysis
- PROC STEPDISC = Procedure for Stepwise Discriminant Analysis
- DATA = Data set
- POOL = YES : To use the pooled covariance matrix in calculating the Discriminant function
ü POOL=NO | TEST | YES
Determines whether the pooled or within-group covariance matrix is the basis of the measure of the squared distance. If you specify POOL=YES, PROC DISCRIM uses the pooled covariance matrix in calculating the (generalized) squared distances. Linear discriminant functions are computed. If you specify POOL=NO, the procedure uses the individual within-group covariance matrices in calculating the distances. Quadratic discriminant functions are computed. The default is POOL=YES. When you specify METHOD=NORMAL, the option POOL=TEST requests Bartlett’s modification of the likelihood ratio test (Morrison 1976; Anderson 1984) of the homogeneity of the within-group covariance matrices. The test is unbiased (Perlman 1980). However, it is not robust to non-normality. If the test statistic is significant at the level specified by the SLPOOL= option, the within-group covariance matrices are used. Otherwise, the pooled covariance matrix is used. The discriminant function coefficients are displayed only when the pooled covariance matrix is used.
- CANONICAL : Performs canonical Discriminant analysis
- CROSSVALIDATE: Classifies each observation in the data set by computing a Discriminant function from the other observation in the data set, excluding the observation being classified (i.e. leave one-out classification)
- ANOVA: Displays univariate statistics
- MANOVA: Displays multivariate statistics
- LIST: Displays the Hit ratio (correct classification) based on Discriminant function computed using the entire data set.
- OUTSTAT: To save the calibration information (Discriminant function)
- CLASS: Grouping variable
- VAR: Set of independent variables
- PRIORS PROP: specifies the prior probabilities of group membership based on probabilities proportional to the sample size.
Reference:
Thursday, December 2, 2010
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part – IV)
Two Goals for Discriminant Analysis
Interpretation: “How are the groups different?” Find and interpret linear combinations of variables that optimally predict group differences
Classification: “How accurately can observations be classified into groups?” Using functions of variables to predict group membership for a data set and evaluate expected error rates
Steps involved in Discriminant Analysis Process
Specify the dependent & the predictor variables
Test the model’s assumptions a priori
Determine the method for selection and criteria for entering the predictor variables into the model
Estimate the parameters of the model
Determine the goodness-of-fit of the model and examine the residuals
Determine the significance of the predictors
Test the assumptions
Validate the results
Interpretation: “How are the groups different?” Find and interpret linear combinations of variables that optimally predict group differences
Classification: “How accurately can observations be classified into groups?” Using functions of variables to predict group membership for a data set and evaluate expected error rates
Steps involved in Discriminant Analysis Process
Specify the dependent & the predictor variables
Test the model’s assumptions a priori
Determine the method for selection and criteria for entering the predictor variables into the model
Estimate the parameters of the model
Determine the goodness-of-fit of the model and examine the residuals
Determine the significance of the predictors
Test the assumptions
Validate the results
Sunday, November 21, 2010
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part – III)
Terminologies:
• F values : Is the ratio of the between sum of squares to the within sum of squares of variable.
• Wilks’ Lambda: Is the ratio of the within sum of squares to the total sum of squares for the entire set of variables in analysis. Wilks’ Lambda varies between 0 to 1. Also called U statistics.
• Classification matrix: Is a matrix that contains the number of correctly classified and misclassified cases.
• Hit Ratio: Percentage of cases correctly classified by the discriminant function.
The DISCRIM Procedure
• PROC DISCRIM can be used for many different types of analysis including
• canonical discriminant analysis
• assessing and confirming the usefulness of the functions (empirical validation and crossvalidation)
• predicting group membership on new data using the functions (scoring)
• linear and quadratic discriminant analysis
• nonparametric discriminant analysis
Discriminant Function
Where,
Z = Discriminant score, a number used to predict group membership of a case
a = Discriminant constant
Wk = Discriminant weight or coefficient, a measure of the extent to which variable Xk discriminates among the groups of the DV
Xk = An Independent Variable or Predictor variable. Can be metric or non-metric.
Number of discriminant functions = min (number of groups – 1, k).
k = Number of predictor variables.
Discriminant Function : Interpretation
• The weights are chosen so that one will be able to compute a discriminant score for each subject and then do an ANOVA on Z.
• More precisely, the weights of the discriminant function are calculated in such a way, that the ratio (between groups SS)/(within groups SS) is as large as possible.
• The value of this ratio is the eigenvalue
• First discriminant function Z1 distinguishes first group from groups 2,3,..N.
• Second discriminant function Z2 distinguishes second group from groups 3, 4…,N. etc
Note : Discriminant analysis uses OLS to estimate the values of the parameters (a) and Wk that minimize the Within Group SS.
Partitioning Sums of Squares in Discriminant Analysis
In Linear Regression:
• Total sums of squares are partitioned into Regression sums of squares and Residual sums of squares.
• And Goal is to estimate parameters that minimize the Residual SS.
In Discriminant Analysis:
• The Total sums of squares is partitioned into Between Group sums of squares and Within Groups sums of squares
Where,
i = an individual case,
j = group j
Zi = individual discriminant score
Z = grand mean of the discriminant scores
Zj = mean discriminant score for group j
Here, Goal is to estimate parameters that minimize the Within Group Sums of Squares
• F values : Is the ratio of the between sum of squares to the within sum of squares of variable.
• Wilks’ Lambda: Is the ratio of the within sum of squares to the total sum of squares for the entire set of variables in analysis. Wilks’ Lambda varies between 0 to 1. Also called U statistics.
• Classification matrix: Is a matrix that contains the number of correctly classified and misclassified cases.
• Hit Ratio: Percentage of cases correctly classified by the discriminant function.
The DISCRIM Procedure
• PROC DISCRIM can be used for many different types of analysis including
• canonical discriminant analysis
• assessing and confirming the usefulness of the functions (empirical validation and crossvalidation)
• predicting group membership on new data using the functions (scoring)
• linear and quadratic discriminant analysis
• nonparametric discriminant analysis
Discriminant Function
Linear discriminant analysis constructs one or more discriminant equations (linear combinations of the predictor variables Xk) such that the different groups differ as much as possible on Z.
Z = Discriminant score, a number used to predict group membership of a case
a = Discriminant constant
Wk = Discriminant weight or coefficient, a measure of the extent to which variable Xk discriminates among the groups of the DV
Xk = An Independent Variable or Predictor variable. Can be metric or non-metric.
Number of discriminant functions = min (number of groups – 1, k).
k = Number of predictor variables.
Discriminant Function : Interpretation
• The weights are chosen so that one will be able to compute a discriminant score for each subject and then do an ANOVA on Z.
• More precisely, the weights of the discriminant function are calculated in such a way, that the ratio (between groups SS)/(within groups SS) is as large as possible.
• The value of this ratio is the eigenvalue
• First discriminant function Z1 distinguishes first group from groups 2,3,..N.
• Second discriminant function Z2 distinguishes second group from groups 3, 4…,N. etc
Note : Discriminant analysis uses OLS to estimate the values of the parameters (a) and Wk that minimize the Within Group SS.
Partitioning Sums of Squares in Discriminant Analysis
In Linear Regression:
• Total sums of squares are partitioned into Regression sums of squares and Residual sums of squares.
• And Goal is to estimate parameters that minimize the Residual SS.
In Discriminant Analysis:
• The Total sums of squares is partitioned into Between Group sums of squares and Within Groups sums of squares
Where,
i = an individual case,
j = group j
Zi = individual discriminant score
Z = grand mean of the discriminant scores
Zj = mean discriminant score for group j
Here, Goal is to estimate parameters that minimize the Within Group Sums of Squares
Thursday, November 11, 2010
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part – II)
Discriminant Analysis attempts to find a rule that separates clusters to the maximum possible extent.
Discriminant Analysis - Assumptions
The underlying assumptions of Discriminant Analysis (DA) are:
– Each group is normally distributed, Discriminant Analysis is relatively robust to departures from normality.
– The groups defined by the dependent variable exist a priori.
– The Predictor variable, Xk are multivariate normally distributed, independent, and non-collinear
– The variance/covariance matrix of the predictor variable across the various groups are the same in the population, (i.e. Homogeneous)
– The relationship is linear in its parameters
– Absence of leverage point outliers
– The sample is large enough: Unequal sample sizes are acceptable. The sample size of the smallest group needs to exceed the number of predictor variables. As a “rule of thumb”, the smallest sample size should be at least 20 for a few (4 or 5) predictors. The maximum number of independent variables is n - 2, where n is the sample size. While this low sample size may work, it is not encouraged, and generally it is best to have 4 or 5 times as many observations and independent variables
– Errors are randomly distributed
Drawback of Discriminant Analysis
– An important drawback of discriminant analysis is its dependence on a relatively equal distribution of group membership. If one group within the population is substantially larger than the other group, as is often the case in real life, Discriminant analysis might classify all observations in only one group. An equal good-bad sample should be chosen for building the discriminant analysis model.
– Another significant restriction of discriminant analysis is that it can’t handle categorical independent variables.
– Discriminant analysis is more rigid than logistic regression in its assumptions. In contrast to ordinary linear regression, discriminant analysis does not have unique coefficients. Each of the coefficients depends on the other coefficients in the estimation and therefore there is no way of determining the absolute value of any coefficient.
Discriminant Analysis Vs Logistic Regression
– Similarity: Both techniques examine an entire set of interdependent relationships
Discriminant Analysis Vs ANOVA
– Similarity: Both techniques examine an entire set of interdependent relationships
– Difference: In Discriminant analysis, Independent variables are metric where as in ANOVA it is categorical.
Reference:
http://userwww.sfsu.edu/~efc/classes/biol710/discrim/discrim.pdf
www.shsu.edu/~icc_cmf/cj_742/stats7.doc
Discriminant Analysis - Assumptions
The underlying assumptions of Discriminant Analysis (DA) are:
– Each group is normally distributed, Discriminant Analysis is relatively robust to departures from normality.
– The groups defined by the dependent variable exist a priori.
– The Predictor variable, Xk are multivariate normally distributed, independent, and non-collinear
– The variance/covariance matrix of the predictor variable across the various groups are the same in the population, (i.e. Homogeneous)
– The relationship is linear in its parameters
– Absence of leverage point outliers
– The sample is large enough: Unequal sample sizes are acceptable. The sample size of the smallest group needs to exceed the number of predictor variables. As a “rule of thumb”, the smallest sample size should be at least 20 for a few (4 or 5) predictors. The maximum number of independent variables is n - 2, where n is the sample size. While this low sample size may work, it is not encouraged, and generally it is best to have 4 or 5 times as many observations and independent variables
– Errors are randomly distributed
Drawback of Discriminant Analysis
– An important drawback of discriminant analysis is its dependence on a relatively equal distribution of group membership. If one group within the population is substantially larger than the other group, as is often the case in real life, Discriminant analysis might classify all observations in only one group. An equal good-bad sample should be chosen for building the discriminant analysis model.
– Another significant restriction of discriminant analysis is that it can’t handle categorical independent variables.
– Discriminant analysis is more rigid than logistic regression in its assumptions. In contrast to ordinary linear regression, discriminant analysis does not have unique coefficients. Each of the coefficients depends on the other coefficients in the estimation and therefore there is no way of determining the absolute value of any coefficient.
Discriminant Analysis Vs Logistic Regression
– Similarity: Both techniques examine an entire set of interdependent relationships
Discriminant Analysis Vs ANOVA
– Similarity: Both techniques examine an entire set of interdependent relationships
– Difference: In Discriminant analysis, Independent variables are metric where as in ANOVA it is categorical.
Reference:
http://userwww.sfsu.edu/~efc/classes/biol710/discrim/discrim.pdf
www.shsu.edu/~icc_cmf/cj_742/stats7.doc
Saturday, November 6, 2010
A Powerful Classification Technique in Data Mining - Discriminant Analysis(part – I)
Classification is a data mining technique used to predict group membership for data instances. In predictive customer analytics, classification techniques are deployed frequently and are true across most applications including acquisition, cross-sell, attrition, credit scoring, collections and classifying first time buyer etc. The objective of any classification model is to classify customers in two or more groups based on a predicted outcome associated with each customer e.g. responder or non-responder, defaulter or non-defaulter, churner or non-churner, valuable customers or non valuable customers etc. Businesses are interested in predicting likelihood of each customer behaving in a particular fashion, and classification techniques provide them with predictive models for the same.
Various parametric and non-parametric methods are used to solve classification related problems. Traditional statistical methods are parametric in nature based on the assumptions about the nature of the distributions and estimate the parameters of the distributions to solve the problem. Non-parametric methods, on the other hand, make no assumptions about the specific distributions involved, and are therefore distribution-free.
Discriminant analysis is a technique for classifying a set of observations into two or more predefined classes. The purpose is to determine the class of an observation based on a set of variables known as predictors or input variables (analogous to independent variables in regression). The model is built based on a set of observations for which the classes are known. This set of observations is sometimes referred to as the training set. Based on the training set, the technique constructs a set of linear functions of the predictors, known as discriminant functions, such that
L = b1x1 + b2x2 + …… + bnxn + c , where the b's are discriminant coefficients, the x's are the input variables or predictors and c is a constant.
These discriminant functions are used to predict the class of a new observation with unknown class. For a k class problem k discriminant functions are constructed. Given a new observation, all the k discriminant functions are evaluated and the observation is assigned to class i if the ith discriminant function has the highest value.
Discriminant Analyis (DA), a multivariate statistical technique is commonly used to build a predictive / descriptive model of group discrimination based on observed predictor variables and to classify each observation into one of the groups. In DA multiple quantitative attributes are used to discriminate single classification variable. DA is different from the cluster analysis because prior knowledge of the classes, usually in the form of a sample from each class is required.
The common objectives of DA are
i. To investigate differences between groups
ii. To discriminate groups effectively;
iii. To identify important discriminating variables;
iv. To perform hypothesis testing on the differences between the expected groupings
v. To classify new customers into pre-existing groups.
Commonly used DA techniques available in the SAS systems are :
DISCRIM: Computes various discriminant functions for classifying observations. Linear or quadratic discriminant functions can be used for data with approximately multivariate normal within-class distributions. Nonparametric methods can be used without making any assumptions about these distributions.
CANDISC: Performs a canonical analysis to find linear combinations of the quantitative variables that best summarize the differences among the classes.
STEPDISC: It uses forward selection, backward elimination, or stepwise selection to try to find a subset of quantitative variables that best reveals differences among the classes.
Reference:
http://www2.sas.com/proceedings/sugi27/p247-27.pdf
Various parametric and non-parametric methods are used to solve classification related problems. Traditional statistical methods are parametric in nature based on the assumptions about the nature of the distributions and estimate the parameters of the distributions to solve the problem. Non-parametric methods, on the other hand, make no assumptions about the specific distributions involved, and are therefore distribution-free.
Discriminant analysis is a technique for classifying a set of observations into two or more predefined classes. The purpose is to determine the class of an observation based on a set of variables known as predictors or input variables (analogous to independent variables in regression). The model is built based on a set of observations for which the classes are known. This set of observations is sometimes referred to as the training set. Based on the training set, the technique constructs a set of linear functions of the predictors, known as discriminant functions, such that
L = b1x1 + b2x2 + …… + bnxn + c , where the b's are discriminant coefficients, the x's are the input variables or predictors and c is a constant.
These discriminant functions are used to predict the class of a new observation with unknown class. For a k class problem k discriminant functions are constructed. Given a new observation, all the k discriminant functions are evaluated and the observation is assigned to class i if the ith discriminant function has the highest value.
Discriminant Analyis (DA), a multivariate statistical technique is commonly used to build a predictive / descriptive model of group discrimination based on observed predictor variables and to classify each observation into one of the groups. In DA multiple quantitative attributes are used to discriminate single classification variable. DA is different from the cluster analysis because prior knowledge of the classes, usually in the form of a sample from each class is required.
The common objectives of DA are
i. To investigate differences between groups
ii. To discriminate groups effectively;
iii. To identify important discriminating variables;
iv. To perform hypothesis testing on the differences between the expected groupings
v. To classify new customers into pre-existing groups.
Commonly used DA techniques available in the SAS systems are :
DISCRIM: Computes various discriminant functions for classifying observations. Linear or quadratic discriminant functions can be used for data with approximately multivariate normal within-class distributions. Nonparametric methods can be used without making any assumptions about these distributions.
CANDISC: Performs a canonical analysis to find linear combinations of the quantitative variables that best summarize the differences among the classes.
STEPDISC: It uses forward selection, backward elimination, or stepwise selection to try to find a subset of quantitative variables that best reveals differences among the classes.
Reference:
http://www2.sas.com/proceedings/sugi27/p247-27.pdf
Saturday, October 16, 2010
My Understanding about Linear Regression - Part V
Model Validation Statistics
Once a linear regression model is built, it is necessary to validate the performance of the model. There are various validation techniques. Some of the widely used techniques are listed below:
R-Square
Any model is only as good as it is able to predict the actual outcome with accuracy. R-Square is a measure of how well the model is able to predict the changes in the actual data. R-Square ranges between 0 and 1, generally with values over 0.7 indicating a good fit between the predictions and actual data.
Mean Absolute Percent Error (MAPE)
MAPE is a measure of how high or low are the differences between the predictions and actual data. For e.g. 10% MAPE means on average the predictions from a model will be 10% higher or lower than actual.
Mape is defined by the formula:
defined by the formula:
1/n (Sigma(a-f)/a*100)
Where a is the Actual value and f is the predicted value.
Rank Ordering
This is an initial validation process, used once the predicted values are created. The predicted values are sorted in descending order and are grouped into deciles. Then the number of actual numbers is computed for each decile. Generally, the mean actual values are used for Rank ordering. The model is said to have rank ordering if these numbers follow a monotonically decreasing order, i.e., the average in1st decile should be strictly higher than the average number in 2nd decile, and so on.
The equation created in training data, the same equation should be used in validation data for creating deciles.
Sign Check
This is more from a business point of view. The signs (+/-) of the parameters corresponding to each independent variable are checked to see if it makes business sense. For example, is a variable is supposed to have a positive relation with the predicted value then its sign should be positive.
Once we finalize the model in training data, we used to fit the same model in the validation data and check the sign of the coefficient of the independent variables.
Lift Curve
A lift curve is a graphical representation of the % of cumulative dependent variable (for example if the dependent variable is revenue, then cumulative percentage of revenue in each decile) captured at a specific cut-off. The cut-off can be a particular decile or a percentile. Similar, to rank ordering procedure, the data is in descending order of the predicted value and is then grouped into deciles/percentiles. The cumulative sum of dependent variable is then computed for each decile/percentile. Taking the example from rank ordering, a lift curve for the same would be as follows:
One of the assumptions of linear regression analysis is that the residuals are normally distributed. This assumption assures that the p-values for the t-tests will be valid. As before, we will generate the residuals (called r) and predicted values (called fv) and put them in a dataset (called elem1res). We will also keep the other independent variables in that dataset.
proc reg data=best_model;
model ln_gross_rev =
Tot_unit_AnyProd
Flag_PC
Flag_other
hh_size
online_ordr_amt_avg
age;
output out=elem1res (keep= ln_gross_rev Tot_unit_AnyProd
Flag_PC
Flag_other
hh_size
online_ordr_amt_avg
age r fv) residual=r predicted=fv;
run;
quit;
proc kde data=elem1res out=den;
var r ;
run;
proc sort data=den;
by r;
run;
goptions reset=all;
symbol1 c=blue i=join v=none height=1;
proc gplot data=den;
plot density*r=1;
run;
quit;
qqplot of residual
Proc univariate will produce a normal quantile graph. qqplot plots the quantiles of a variable against the quantiles of a normal distribution. qqplotis most sensitive to non-normality near two tails.
goptions reset=all;
proc univariate data=elem1res normal;
var r;
qqplot r / normal(mu=est sigma=est);
run;
Some cosmetic treatment of a model – I learnt from my experiences
Sometimes I have seen the model is not rank ordering, there are several methods to check/correct the rank ordering problem:
a. For this we may exclude one or more variables as a time, and build the linear regression model to see when we get a rank ordering in the modeling and validation samples.
b. One can plot each of the independent variables against the decile created for rank ordering. Each of the independent variable should follow a trend.
i. One can verify the sign of the coefficient of each of the independent variable
ii. In case the independent variable is not following a trend, one can use different transformation like:
1. Linear transformation(for example , for independent continuous variable , one can create categorical variable like if recency is between 4.33 years to 3.1 years then recency_1 = 3, and if recency is <3.1 but >=2.5 then recency_1 = 2, else recency_1 = 1) .
2. Quadratic transformations are also useful sometimes
3. One can use parabolic and hyperbolic transformations too but those are difficult to explain.
Autocorrelation
Another way in which the assumption of independence can be broken is when data are collected on the same variables over time. Let's say that we collect medicine usage data every quarter for 12 years. In this situation it is likely that the errors for observation between adjacent quarters will be more highly correlated than for observations more separated in time. This is known as autocorrelation. When you have data that can be considered to be time-series, you should use the dw option that performs a Durbin-Watson test for correlated residuals.
Durbin-Watson Statistic:
One peculiar feature of data recorded over time, like monthly sales, is that it tends to be correlated over time. For e.g. high sales months may be tend to be followed by high sales months and low sales months by more low sales months. This may be caused either by seasonal/cyclical trends or seasonal promotion, marketing or competitive effects. Whatever the factor causing this correlation, correlated errors violate one of the fundamental assumptions needed for least squares regression- independence of errors or in other words random errors. Durbin-Watson Statistic is a measure used to detect such correlations. Every model has one measure for Durbin-Watson statistic. Durbin-Watson Statistic, ranges in value from 0 to 4 with an ideal value of 2 indicating that errors are not correlated (although values from 1.75 to 2.25 may be considered acceptable). A value significantly below 2 indicates a positive correlation and a value significantly greater than 2 suggests negative correlation. In either case the model specification needs to be reviewed to identify variables potentially omitted or redundant variables.
References:
http://www.ats.ucla.edu/stat/sas/library/SASReg_mf.htm
http://www.ats.ucla.edu/stat/sas/webbooks/reg/chapter2/sasreg2.htm
http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#/documentation/cdl/en/statug/63033/HTML/default/statug_reg_sect007.htm
http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm
http://www.sfu.ca/sasdoc/sashtml/stat/chap55/sect38.htm
Once a linear regression model is built, it is necessary to validate the performance of the model. There are various validation techniques. Some of the widely used techniques are listed below:
R-Square
Any model is only as good as it is able to predict the actual outcome with accuracy. R-Square is a measure of how well the model is able to predict the changes in the actual data. R-Square ranges between 0 and 1, generally with values over 0.7 indicating a good fit between the predictions and actual data.
Mean Absolute Percent Error (MAPE)
MAPE is a measure of how high or low are the differences between the predictions and actual data. For e.g. 10% MAPE means on average the predictions from a model will be 10% higher or lower than actual.
Mape is defined by the formula:
defined by the formula:
1/n (Sigma(a-f)/a*100)
Where a is the Actual value and f is the predicted value.
Rank Ordering
This is an initial validation process, used once the predicted values are created. The predicted values are sorted in descending order and are grouped into deciles. Then the number of actual numbers is computed for each decile. Generally, the mean actual values are used for Rank ordering. The model is said to have rank ordering if these numbers follow a monotonically decreasing order, i.e., the average in1st decile should be strictly higher than the average number in 2nd decile, and so on.
The equation created in training data, the same equation should be used in validation data for creating deciles.
Sign Check
This is more from a business point of view. The signs (+/-) of the parameters corresponding to each independent variable are checked to see if it makes business sense. For example, is a variable is supposed to have a positive relation with the predicted value then its sign should be positive.
Once we finalize the model in training data, we used to fit the same model in the validation data and check the sign of the coefficient of the independent variables.
Lift Curve
A lift curve is a graphical representation of the % of cumulative dependent variable (for example if the dependent variable is revenue, then cumulative percentage of revenue in each decile) captured at a specific cut-off. The cut-off can be a particular decile or a percentile. Similar, to rank ordering procedure, the data is in descending order of the predicted value and is then grouped into deciles/percentiles. The cumulative sum of dependent variable is then computed for each decile/percentile. Taking the example from rank ordering, a lift curve for the same would be as follows:
Tests for Normality of Residuals
One of the assumptions of linear regression analysis is that the residuals are normally distributed. This assumption assures that the p-values for the t-tests will be valid. As before, we will generate the residuals (called r) and predicted values (called fv) and put them in a dataset (called elem1res). We will also keep the other independent variables in that dataset.
proc reg data=best_model;
model ln_gross_rev =
Tot_unit_AnyProd
Flag_PC
Flag_other
hh_size
online_ordr_amt_avg
age;
output out=elem1res (keep= ln_gross_rev Tot_unit_AnyProd
Flag_PC
Flag_other
hh_size
online_ordr_amt_avg
age r fv) residual=r predicted=fv;
run;
quit;
proc kde data=elem1res out=den;
var r ;
run;
proc sort data=den;
by r;
run;
goptions reset=all;
symbol1 c=blue i=join v=none height=1;
proc gplot data=den;
plot density*r=1;
run;
quit;
qqplot of residual
Proc univariate will produce a normal quantile graph. qqplot plots the quantiles of a variable against the quantiles of a normal distribution. qqplotis most sensitive to non-normality near two tails.
goptions reset=all;
proc univariate data=elem1res normal;
var r;
qqplot r / normal(mu=est sigma=est);
run;
Some cosmetic treatment of a model – I learnt from my experiences
Sometimes I have seen the model is not rank ordering, there are several methods to check/correct the rank ordering problem:
a. For this we may exclude one or more variables as a time, and build the linear regression model to see when we get a rank ordering in the modeling and validation samples.
b. One can plot each of the independent variables against the decile created for rank ordering. Each of the independent variable should follow a trend.
i. One can verify the sign of the coefficient of each of the independent variable
ii. In case the independent variable is not following a trend, one can use different transformation like:
1. Linear transformation(for example , for independent continuous variable , one can create categorical variable like if recency is between 4.33 years to 3.1 years then recency_1 = 3, and if recency is <3.1 but >=2.5 then recency_1 = 2, else recency_1 = 1) .
2. Quadratic transformations are also useful sometimes
3. One can use parabolic and hyperbolic transformations too but those are difficult to explain.
Autocorrelation
Another way in which the assumption of independence can be broken is when data are collected on the same variables over time. Let's say that we collect medicine usage data every quarter for 12 years. In this situation it is likely that the errors for observation between adjacent quarters will be more highly correlated than for observations more separated in time. This is known as autocorrelation. When you have data that can be considered to be time-series, you should use the dw option that performs a Durbin-Watson test for correlated residuals.
Durbin-Watson Statistic:
One peculiar feature of data recorded over time, like monthly sales, is that it tends to be correlated over time. For e.g. high sales months may be tend to be followed by high sales months and low sales months by more low sales months. This may be caused either by seasonal/cyclical trends or seasonal promotion, marketing or competitive effects. Whatever the factor causing this correlation, correlated errors violate one of the fundamental assumptions needed for least squares regression- independence of errors or in other words random errors. Durbin-Watson Statistic is a measure used to detect such correlations. Every model has one measure for Durbin-Watson statistic. Durbin-Watson Statistic, ranges in value from 0 to 4 with an ideal value of 2 indicating that errors are not correlated (although values from 1.75 to 2.25 may be considered acceptable). A value significantly below 2 indicates a positive correlation and a value significantly greater than 2 suggests negative correlation. In either case the model specification needs to be reviewed to identify variables potentially omitted or redundant variables.
References:
http://www.ats.ucla.edu/stat/sas/library/SASReg_mf.htm
http://www.ats.ucla.edu/stat/sas/webbooks/reg/chapter2/sasreg2.htm
http://support.sas.com/documentation/cdl/en/statug/63033/HTML/default/viewer.htm#/documentation/cdl/en/statug/63033/HTML/default/statug_reg_sect007.htm
http://www.stat.yale.edu/Courses/1997-98/101/linreg.htm
http://www.sfu.ca/sasdoc/sashtml/stat/chap55/sect38.htm
Sunday, October 10, 2010
Why use multiple linear regression?
To investigate a collection of factors for their potential association with the outcome of interest
To investigate a collection of known relevant factors for their ability to predict the outcome of interest.
The GOAL
To obtain a parsimonious set of variables that efficiently predicts the response variable of interest.
Model Selection
PROC REG supports a variety of model selection methods but does not support a CLASS statement.
PROC GLM supports the CLASS statement but does not include the model selection methods
PROC GLMSELECT supports the CLASS statement and includes model selection methods but does not include regression diagnostics or hypothesis testing, LS-means etc.
o Only available in SAS 9.2
o Can download from SAS website for 9.1
o http://support.sas.com/rnd/app/da/glmselect.html
To investigate a collection of known relevant factors for their ability to predict the outcome of interest.
The GOAL
To obtain a parsimonious set of variables that efficiently predicts the response variable of interest.
Model Selection
PROC REG supports a variety of model selection methods but does not support a CLASS statement.
PROC GLM supports the CLASS statement but does not include the model selection methods
PROC GLMSELECT supports the CLASS statement and includes model selection methods but does not include regression diagnostics or hypothesis testing, LS-means etc.
o Only available in SAS 9.2
o Can download from SAS website for 9.1
o http://support.sas.com/rnd/app/da/glmselect.html
Subscribe to:
Posts (Atom)