Before continuing, we will briefly review the assumptions for ordinary least squares (OLS) regression. Below are the common assumptions of OLS. One point to make is that these assumptions are filled with caveats, all of which are not possible to discuss in detail here. We point out important considerations when appropriate.
- Linearity - this suggests that the dependent and independent variables can be modeled in a linear fashion. The linearity idea here is that a linear function (a regression function) can be built that models the assumed linear relationship in the dependent and independent variables. This does not imply that all relationships are exclusive to x and y. In some cases, the values of x or y need to be transformed in some fashion (e.g., log(x + 1)) to account for some artifact in the data.
- Independence of Observations/Errors - any observation should not be dependent upon any other observation. If the observations are related in some fashion, then there is a structure that the regression will not account for and will therefore not produce a valid model. In other words, the error associated with any prediction should not be biased with the error associated with any other prediction. The errors should be randomly distributed across all predictions.
- Zero conditional mean - a predictor should not be systematically or mechanically related to an omitted factor contained in the error term. In other words, if there is a variable that affects the dependent variable y, but is omitted and correlated with one of the independent variables x, then the regression coefficients can be biased.
- Multicolinearity - ideally, all of the independent variables uniquely contribute to the prediction of y. In cases where more than one independent variable correlates with another independent variable, then the regression cannot determine which contribution was unique. As a simple example, assume I used two variables to predict human performance in a bicycle race (y): weight in pounds and kilograms. In this case, pounds and kilos are perfectly correlated with each other. Because of this, the regression process cannot determine which variable uniquely contributes to the overall prediction of the dependent variable. One of these variables needs to be removed from the regression due to their redundancy in correlation. Tests exists that can help you determine if multicolinearity is present and which variable(s) to remove.
- Homoscedasticity - the variance in the error terms needs to be constant across all predictions. To check for the presence of homoscedasticity, you can look at scatterplots of the standardized residuals against the standardized fitted (predicted) values.
- Normality of errors - one examines the normality of the residuals to determine how well the regression model performs and whether a different form of regression is needed.