Project 5: Explore the Data: Descriptive Statistics and Histograms

Before you start working with any dataset, it is important to explore the data using descriptive statistics and view the data’s distribution using histograms (or another graphical summary method). Descriptive statistics enable you to compare various measures across the different variables. These include mean, mode, standard deviation, etc. There are many kinds of graphical summary methods, such as histograms and boxplots. For this part of the assignment, we will use histograms to examine the distribution of the variables.

Before we get too far along with this discussion, lets define a few terms here so that we are on the same page with their usage.

Variation - is the extent to which a dataset is spread out or dispersed with respect to its center. The variation of a dataset can be spread out or very concentrated about its center. The variation of a dataset has nothing to do with skewness or normality. Variation can be quantified according to the variance, standard deviation, or interquartile range.

Skewness - is the departure from symmetry. A symmetrical dataset has a distribution that is equally shaped on both side of its center. Departures from this symmetry are describes according to the direction of the departure. Positive skewness suggests that the dataset extends to a greater degree in the positive direction while negative skewness suggests that the dataset extends to a greater degree in the negative direction. 

Measures of skewness provide quantitative evidence by which the distribution departs from symmetry. 
- values that are + suggest a positive skew 
- values that are - skewness suggest a negative skew
- values closer to 0 suggest the data are symmetrical

Normality - measures the symmetry of a dataset. Unlike measures of skewness, a normality test statistically indicates whether a distribution is or is not symmetrical. One can look at the visual appearance of a histogram, for example, an see a departure from symmetry, but a normality test may conclude that the distribution is statistically normally distributed according to the returned p-value against common α levels of .10, .05, .01. 

Two of the more common test for normality include the Kolmogorov - Smirnov (KS) test and the Shapiro-Wilk test.

For the KS test, conventional guidance suggests that this test is applied when your number of elements is 50 or greater...

• Small D-statistic: suggests that the two distributions are very close together, so your data fits the model well.
• Large D-statistic: suggests that the gap is wide, suggesting your data does not match the expected distribution.
• P-value: Tells you if the difference is statistically significant. If the p-value is smaller than your chosen α level (usually .05), you reject the idea that the distributions are the same

For the Shapiro-Wilk test, conventional guidance suggests that this test is applied when your number of elements is less than 50...

• p-value > .05: Fail to reject the null hypothesis. You can assume the data are normal.
• p-value ≤ .05: Reject the null hypothesis. The data significantly depart from normality

The basic hypotheses for testing for normality are as follows: 
H0: The data are ≈ normally distributed
HA: The data are not ≈ normally distributed

Note that the use of the ≈ symbol is deliberate, as the symbol means “approximately.” We reject the null hypothesis when the p-value is ≤ a specified α-level. Common α-levels include .10, .05, and .01.

It is important to note that the magical "50" number was born out of historical table constraints rather than a strict mathematical failure. Tables reporting outcomes of statistical tests were often printed in the back of statistic books. The page size often limited the sample sizes and corresponding values that were reported. In addition, the original 1965 paper by Shapiro and Wilk only provided critical value tables up to a sample size of 50. Statistical programming environments like R use advanced numerical approximations to compute the test for larger samples (up to n = 5000). For most applications, the Shaprio-Wilk test is preferred because it has higher statistical power to detect non-normal data (meaning that the test rejects a false null more often).

### Chunk 3: Descriptive statistics and plot histograms
```{r}

# Summarize the Poverty Dataset
# describe() returns the following descriptive statistics
# Number of variables, nvalid, mean, standard deviation (sd), median, trimmed median, mad (median absolute deviation from the median), minimum value, maximum value, skewness, kurtosis, and standard error of the mean.

attach(Poverty_Data)
descriptives <- cbind(Poverty_Data$PctFamsPov, Poverty_Data$PctNoHealth, Poverty_Data$MedHHIncom, Poverty_Data$PCTUnemp)
describe(descriptives)

# Create Histograms of the Dependent and Independent Variables

# Percent families below poverty level histogram (dependent variable)
# Note that the c(0,70) sets the y-axis to have the same range so each histogram can be directly compared to one another
hist(Poverty_Data$PctFamsPov, col = "Lightblue", main = "Histogram of Percent of Families in Poverty", ylim = c(0,70))

# Percent families without health insurance histogram
hist(Poverty_Data$PctNoHealth, col = "Lightblue", main = "Histogram of Percent of People over 18 with No Health Insurance", ylim = c(0,70))

# Median household income histogram
hist(Poverty_Data$MedHHIncom, col = "Lightblue", main = "Histogram of Median Household Income", ylim = c(0,70))

# Percent unemployed histogram
hist(Poverty_Data$PCTUnemp, col = "Lightblue", main = "Histogram of Percent Unemployed", ylim = c(0,70))

```

Figure 5.4 shows a summary of the various descriptive statistics that are provided by the describe() function. In Figure 5.4, X1, X2, X3, and X4 represent the percent of families below the poverty level, the percent of individuals without health insurance, the median household income, and the percent of unemployed individuals, respectively.

We begin our examination of the descriptive statistics by comparing the mean and median values of the variables. In cases where the mean and median values are similar, the data’s distribution can be considered approximately normal. Note that a similarity in mean and median values can be seen in rows X1 and X4. For X1, the difference between the mean and median is 0.35 percent and for X4 the difference is 0.23 percent. There is a larger difference between the mean and median for the variables in rows X2 and X3. The difference between the mean and median for X2 and X3 is 0.57 and $48,189, respectively. Based on this comparison, variables X1 and X4 would seem to be more normally distributed than X2 and X3.

We also examine the skewness values to see what they report about a given variable’s departure from normality and whether outliers may be present. Skewness values that are “+” suggest a positive skew (outliers are on located on the higher range of the data values and are pulling the mean in the positive direction). Skewness values that are “–“ suggest a negative skew (outliers are located on the lower end of the range of data values and are pulling the mean in the negative direction). A skewness value close to 0.0 suggests a distribution that is approximately normal. As skewness values increase, the severity of the skew also increases. Skewness values close to ±0.5 are considered to possess a moderate skew while values above ±1.0 suggests the data are severely skewed. From Figure 5.4, X2 and X3 have skewness values of 6.04 and 1.82, respectively. Both variables are severely positively skewed. Variables X1 and X4 (reporting skewness of 0.00 and 0.41, respectively) appear to be more normal; although X4 appears to have a moderate level of positive skewness. 

The histograms in Figures 5.5 and 5.6 both reflect what was observed from the mean and median comparison and the skewness values. Figure 5.5 shows a distribution that appears rather symmetrical while Figure 5.6 shows a distribution that is distinctively positively skewed (note the outlier data value located on the far right-hand side of the distribution).

Figure 5.5: The percent of families below the poverty level appears to have a rather normal appearing distribution.

Credit: Kessler, 2019

Figure 5.6: The percent of people with no health insurance appears to have a significant positive skew. Note the potential outlier above 40% pulling the mean of the dataset in the positive direction, creating a positive skew.

Credit: Kessler, 2019

Now that you have visually examined the distribution of the variables and noted some concerns about skewness, we can statistically explore that data with respect to the presence of outliers and their impact on the overall data distribution in the fourth chunk of RMarkdown. 

If outliers are present, you should think about their impact on the regression model, especially the regression coefficients and certain diagnostic tests. Some authors argue that outliers can/should be removed from a dataset if they are deemed to negatively impact the regression model. Recall that outliers can be spurious data values, data entry errors, or atypical data values. You should review each data distribution to determine whether any outliers in your dataset should remain in your analysis or be removed. For example, data entry errors should be addressed as they impact the accuracy of your dataset and any tests that are run on that dataset.

A graphical way to examine a dataset with respect to the presence of outliers is to generate Q-Q plots. Q-Q plots represent the pairing of two probability distributions by plotting their quantiles against each other. If the two distributions being compared are similar (e.g., linearly related), the points in the Q-Q plot will approximately lie on the line y = x. If the distributions are not similar (demonstrating non-normality), then the points in the tails of the plot will deviate from the overall trend of the points. 

Below is the code example for a Q-Q plot.

## Chunk 4: Generate Q-Q plots of the variables

```{r}

# Examine the data using QQ-plots
qqnorm(Poverty_Data$PctFamsPov);qqline(Poverty_Data$PctFamsPov, col = 2)
qqnorm(Poverty_Data$PctNoHealth);qqline(Poverty_Data$PctNoHealth, col = 2)
qqnorm(Poverty_Data$MedHHIncom);qqline(Poverty_Data$MedHHIncom, col = 2)
qqnorm(Poverty_Data$PCTUnemp);qqline(Poverty_Data$PCTUnemp, col = 2)

```

A good overview of how to interpret Q-Q plots can be seen here.

For example, the data points representing the median household income shown in Figure 5.7 do not completely align with the red trend line. Most of the middle portion of the data do align with the trend line but the data “tails” deviate and in some cases, those tails deviate considerable. The data “tails” signal that the data may posses outliers. While the presence of outliers does not automatically invalidate the utility of OLS, we should still be aware that outliers may be present in the data and that their inclusion in the regression may be problematic.

Figure 5.7. A Q-Q plot of the median household income variable.

© Penn State is licensed under CC BY-NC-SA 4.0

What should I do if data include outliers?

We see that two variables are highly skewed and not normal: percent families with no health insurance and median household income. Based on the evidence supplied by the descriptive statistics, histograms, and Q-Q plot, we suspect that outliers are the reason why these two datasets are skewed. Generally speaking, an outlier is an atypical data value. Since we determined that these two datasets may contain outliers, we should review the data and determine what kind of outliers are present and whether their removal is prudent.

There are several approaches that are used to identify and remove outliers. To add a bit of confusion to the mix, there isn’t a single quantitative definition that describes what is atypical in a dataset. Moreover, there isn’t a single method to detect outliers – the chosen method depends on your preferences and needs. For an overview of how outliers are defined and some of the methods used to identify those outliers, check out the following two websites:

1. A simple discussion of outliers and their detection
2. The outlier package in r

### Chunk 5: Conduct outlier test and clean the data
```{r}

# Check for Outliers and Clean those Variables of Outliers
outlier <- boxplot.stats(Poverty_Data$PctFamsPov)$out

outlier # Report any outliers

# Check for Outliers and Clean those Variables of Outliers
outlier <- boxplot.stats(Poverty_Data$PctNoHealth)$out

outlier # Report any outliers
# An outlier was detected. 
# Find the row number.
row_to_be_deleted <- which(Poverty_Data$PctNoHealth == outlier)

# Delete the entire row of data
Poverty_Data_Cleaned <- Poverty_Data[-c(row_to_be_deleted),] 

# Redraw the histogram to check the distribution
hist(Poverty_Data_Cleaned$PctNoHealth, col = "Lightblue", main = "Histogram of Percent of People over 18 with No Health Insurance", ylim = c(0,70))

# Check for Outliers and Clean those Variables of Outliers
outlier <- boxplot.stats(Poverty_Data_Cleaned$MedHHIncom)$out

outlier # Report any outliers
# An outlier was detected.
# Use another method to remove rows of data. Find the minimum outlier value
outlier <- min(outlier)
# Subset the data values that are less than the minimum outlier value.
Poverty_Data_Cleaned <- subset(Poverty_Data_Cleaned, Poverty_Data_Cleaned$MedHHIncom < outlier)

# Median household income histogram
hist(Poverty_Data_Cleaned$MedHHIncom, col = "Lightblue", main = "Histogram of Median Household Income", ylim = c(0,70))

# View a new boxplot
outlier <- boxplot.stats(Poverty_Data_Cleaned$PCTUnemp)$out

# Report any outliers
outlier

# Find the row number.
row_to_be_deleted <- which(Poverty_Data_Cleaned$PCTUnemp == outlier)

# Delete the entire row of data
Poverty_Data_Cleaned <- Poverty_Data_Cleaned[-c(row_to_be_deleted),] 

# Redraw the histogram to confirm
hist(Poverty_Data_Cleaned$PCTUnemp, col = "Lightblue", main = "Histogram of Percent of People over 18 with No Health Insurance", ylim = c(0,70))

```

You should realize that removing outliers is a rather subjective decision. Even after removing any outliers and the appearance of the distribution does not seem appropriate, then you can attempt to transform the data. A data transformation applies a specific mathematical function to the data, attempting to change the appearance of the data distribution. For example, data that exhibit extremely strong positive skewness can be transformed by applying the log10 transformation to the data. Applying a transformation has the effect of changing the scale of the data and therefore the distribution’s “shape” and hopefully skewness. There are several data transformations available, but the more common include log10(x), x2, sqrt(x), and ln(x).

Here is a brief but useful site that covers most of the major data transformations.

It is important to note that in most cases, your data are not going to be without any skewness. The extent to which a dataset is skewed can be extremely challenging to deal with especially as skewness can impact the results of a statistical. This consideration has a lot to do with making the regression model residuals more symmetric since one assumption in linear regression is that the errors are Gaussian (normally distributed). Not meeting the normality assumption can have negative consequences in the ability of the regression analysis to produce meaningful and statistically valid results.