12 Correlation and Regression
12.01: Correlation
We are often interested in the relationship between two variables. This chapter determines whether a linear relationship exists between sets of quantitative data and making predictions for a population—for instance, the relationship between the number of hours of study time and an exam score, or smoking and heart disease.
A predictor variable (also called the independent or explanatory variable; usually we use the letter \(x\)) explains or causes changes in the response variable. The predictor variable can be manipulated or changed by the researcher.
A response variable (also called the dependent variable; usually we use the letter \(y\)) measures the outcome of a study. The different outcomes for a dependent variable are measured or observed by the researcher. For instance, suppose we are interested in how much time spent studying affects the scores on an exam. In this study, study time is the predictor variable, and exam score is the response variable.
In data from an experiment, it is much easier to know which variable we should use for the independent and dependent variables. This can be harder to distinguish in observational data. Think of the dependent variable as the variable that you are trying to learn about.
If we were observing the relationship between unemployment rate and economic growth rate, it may not be clear which variable should be \(x\) and \(y\). Do we want to predict the unemployment rate or the economic growth rate? One should never jump to a cause and effect reasoning with observational data. Just because there is a strong relationship between unemployment rate and economic growth rate does not mean that one causes the other to change directly. There may be many other contributing factors to both of these rates changing at the same time, such as retirements or pandemics.
12.01: Scatterplots
A scatterplot shows the relationship between two quantitative variables measured on the same individuals.
- The predictor variable is labeled on the horizontal or \(x\)-axis.
- The response variable is labeled on the vertical or \(y\)-axis.
How to Interpret a Scatterplot:
- Look for the overall pattern and for deviations from that pattern.
- Look for outliers, individual values that fall outside the overall pattern of the relationship.
- A positive linear relation results when larger values of one variable are associated with larger values of the other.
- A negative linear relation results when larger values of one variable are associated with smaller values of the other.
- A scatterplot has no association if no obvious linear pattern is present.
Use technology to make a scatterplot for the following sample data set:
.png?revision=1)
.png?revision=1)
.png?revision=1)
.jpg?revision=1&size=bestfit&width=995&height=741)
Correlation Coefficient
The sample correlation coefficient measures the direction and strength of the linear relationship between two quantitative variables. There are several different types of correlations. We will be using the Pearson Product Moment Correlation Coefficient (PPMCC). The PPMCC is named after biostatistician Karl Pearson. We will just use the lower-case \(r\) for short when we want to find the correlation coefficient, and the Greek letter \(\rho\), pronounced “rho,” (rhymes with sew) when referring to the population correlation coefficient.
Interpreting the Correlation:
- A positive \(r\) indicates a positive association (positive linear slope).
- A negative \(r\) indicates a negative association (negative linear slope).
- \(r\) is always between \(-1\) and \(1\), inclusive.
- If \(r\) is close to \(1\) or \(-1\), there is a strong linear relationship between \(x\) and \(y\).
- If \(r\) is close to \(0\), there is a weak linear relationship between \(x\) and \(y\). There may be a non-linear relation or there may be no relation at all.
- Like the mean, \(r\) is strongly affected by outliers. Figure 12-1 gives examples of correlations with their corresponding scatterplots.
.png?revision=1)
When you have a correlation that is very close to \(-1\) or \(1\), then the points on the scatter plot will line up in an almost perfect line. The closer \(r\) gets to \(0\), the more scattered your points become.
Take a moment and see if you can guess the approximate value of \(r\) for the scatter plots below.
.png?revision=1)
Solution
Scatterplot A: \(r = 0.98\), Scatterplot B: \(r = 0.85\), Scatterplot C: \(r = -0.85\).
When \(r\) is equal to \(-1\) or \(1\) all the dots in the scatterplot line up in a straight line. As the points disperse, \(r\) gets closer to zero. The correlation tells the direction of a linear relationship only. It does not tell you what the slope of the line is, nor does it recognize nonlinear relationships. For instance, in Figure 12-2, there are three scatterplots overlaid on the same set of axes. All three data sets would have \(r = 1\) even though they all have different slopes.
.png?revision=1)
For the next example in Figure 12-3, \(r = 0\) would indicate no linear relationship; however, there is clearly a non-linear pattern with the data.
.png?revision=1)
Figure 12-4 shows a correlation \(r = 0.874\), which is pretty close to one, indicating a strong linear relationship. However, there is an outlier, called a leverage point, which is inflating the value of the slope. If you remove the outlier then \(r = 0\), and there is no up or down trend to the data.
.png?revision=1)
Calculating Correlation
To calculate the correlation coefficient by hand we would use the following formula.
\[r = \frac{\sum \left( \left(x_{i} - \bar{x}\right) \left(y_{i} - \bar{y}\right) \right)}{\sqrt{ \left( \left(\sum \left(x_{i} - \bar{x}\right)^{2}\right) \left(\sum \left(y_{i} - \bar{y}\right)^{2}\right) \right)} } = \frac{SS_{xy}}{\sqrt{ \left(SS_{xx} \cdot SS_{yy}\right) }}\]
Instead of doing all of these sums by hand we can use the output from summary statistics. Recall that the formula for a variance of a sample is \(s_{x}^{2} = \frac{\sum \left(x_{i} - \bar{x}\right)^{2}}{n-1}\). If we were to multiply both sides by the degrees of freedom, we would get \(\sum \left(x_{i} - \bar{x}\right)^{2} = (n-1) s_{x}^{2}\).
We use these sums of squares \(\sum \left(x_{i} - \bar{x}\right)^{2}\) frequently, so for shorthand we will use the notation \(SS_{xx} = \sum \left(x_{i} - \bar{x}\right)^{2}\). The same would hold true for the \(y\) variable; just changing the letter, the variance of \(y\) would be \(s_{y}^{2} = \frac{\sum \left(y_{i} - \bar{y}\right)^{2}}{n-1}\), therefore \(SS_{yy} = (n-1) s_{y}^{2}\).
The numerator of the correlation formula is taking in the horizontal distance of each data point from the mean of the \(x\) values, times the vertical distance of each point from the mean of the \(y\) values. This is time-consuming to find so we will use an algebraically equivalent formula \(\sum \left(\left(x_{i} - \bar{x}\right) \left(y_{i} - \bar{y}\right) \right) = \sum (xy) - n \cdot \bar{x} \bar{y}\), and for short we will use the notation \(SS_{xy} = \sum (xy) - n \cdot \bar{x} \bar{y}\).
To start each problem, use descriptive statistics to find the sum of squares.
| \(SS_{xx} = (n-1) s_{x}^{2}\) | \(SS_{yy} = (n-1) s_{y}^{2}\) | \(SS_{xy} = sum (xy) - n \cdot \bar{x} \bar{y}\) |
Use the following data to calculate the correlation coefficient.
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
When is a correlation statistically significant? The next subsection shows how to run a hypothesis test for correlations.
12.01: Hypothesis Test for a Correlation
One should perform a hypothesis test to determine if there is a statistically significant correlation between the independent and the dependent variables. The population correlation coefficient \(\rho\) (this is the Greek letter rho, which sounds like “row” and is not a \(p\)) is the correlation among all possible pairs of data values \((x, y)\) taken from a population.
We will only be using the two-tailed test for a population correlation coefficient \(\rho\). The hypotheses are:
\(H_{0}: \rho = 0\)
\(H_{1}: \rho \neq 0\)
The null-hypothesis of a two-tailed test states that there is no correlation (there is not a linear relation) between \(x\) and \(y\). The alternative-hypothesis states that there is a significant correlation (there is a linear relation) between \(x\) and \(y\).
The t-test is a statistical test for the correlation coefficient. It can be used when \(x\) and \(y\) are linearly related, the variables are random variables, and when the population of the variable \(y\) is normally distributed.
The formula for the t-test statistic is \(t = r \sqrt{\left( \dfrac{n-2}{1-r^{2}} \right)}\).
Use the t-distribution with degrees of freedom equal to \(df = n - 2\).
Note the \(df = n - 2\) since we have two variables, \(x\) and \(y\).
Test to see if the correlation for hours studied on the exam and grade on the exam is statistically significant. Use \(\alpha\) = 0.05.
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.jpg?revision=1)
Correlation is Not Causation
Just because two variables are significantly correlated does not imply a cause and effect relationship. There are several relationships that are possible. It could be that \(x\) causes \(y\) to change. You can actually swap \(x\) and \(y\) in the fields and get the same \(r\) value and \(y\) could be causing \(x\) to change. There could be other variables that are affecting the two variables of interest. For instance, you can usually show a high correlation between ice cream sales and home burglaries. Selling more ice cream does not “cause” burglars to rob homes. More home burglaries do not cause more ice cream sales. We would probably notice that the temperature outside may be causing both ice cream sales to increase and more people to leave their windows open. This third variable is called a lurking variable and causes both \(x\) and \(y\) to change, making it look like the relationship is just between \(x\) and \(y\).
There are also highly correlated variables that seemingly have nothing to do with one another. These seemingly unrelated variables are called spurious correlations.
The following website has some examples of spurious correlations (a slight caution that the author has some gloomy examples): http://www.tylervigen.com/spurious-correlations. Figure 12-7 is one of their examples:
.png?revision=1)
If we were to take out each pair of measurements by year from the time-series plot in Figure 12-7, we would get the following data.
| Year | Engineering Doctorates | Mozzarella Cheese Consumption |
|---|---|---|
| 2000 | 480 | 9.3 |
| 2001 | 501 | 9.7 |
| 2002 | 540 | 9.7 |
| 2003 | 552 | 9.7 |
| 2004 | 547 | 9.9 |
| 2005 | 622 | 10.2 |
| 2006 | 655 | 10.5 |
| 2007 | 701 | 11 |
| 2008 | 712 | 10.6 |
| 2009 | 708 | 10.6 |
Using Excel to find a scatterplot and compute a correlation coefficient, we get the scatterplot shown in Figure 12-8 and a correlation of \(r = 0.9586\).
.png?revision=1)
With \(r = 0.9586\), there is strong correlation between the number of engineering doctorate degrees earned and mozzarella cheese consumption over time, but earning your doctorate degree does not cause one to go eat more cheese. Nor does eating more cheese cause people to earn a doctorate degree. Most likely these items are both increasing over time and therefore show a spurious correlation to one another.
When two variables are correlated, it does not imply that one variable causes the other variable to change.
“Correlation is causation” is an incorrect assumption that because something correlates, there is a causal relationship. Causality is the area of statistics that is most commonly misused, and misinterpreted, by people. Media, advertising, politicians and lobby groups often leap upon a perceived correlation and use it to “prove” their own agenda. They fail to understand that, just because results show a correlation, there is no proof of an underlying causality. Many people assume that because a poll, or a statistic, contains many numbers, it must be scientific, and therefore correct. The human brain is built to try and subconsciously establish links between many pieces of information at once. The brain often tries to construct patterns from randomness, and may jump to conclusions, and assume that a cause and effect relationship exists. Relationships may be accidental or due to other unmeasured variables. Overcoming this tendency to jump to a cause and effect relationship is part of academic training for students and in most fields, from statistics to the arts.
Summary
When looking at correlations, start with a scatterplot to see if there is a linear relationship prior to finding a correlation coefficient. If there is a linear relationship in the scatterplot, then we can find the correlation coefficient to tell the strength and direction of the relationship. Clusters of dots forming a linear uphill pattern from left to right will have a positive correlation. The closer the dots in the scatterplot are to a straight line, the closer \(r\) will be to \(1\). If the cluster of dots in the scatterplots go downhill from left to right in linear pattern, then there is a negative relationship. The closer those dots in the scatterplot are to a straight line going downhill, the closer \(r\) will be to \(-1\). Use a t-test to see if the correlation is statistically significant. As sample sizes get larger, smaller values of \(r\) become statistically significant. Be careful with outliers, which can heavily influence correlations. Most importantly, correlation is not causation. When \(x\) and \(y\) are significantly correlated, this does not mean that \(x\) causes \(y\) to change.
12.02: Simple Linear Regression
A linear regression is a straight line that describes how the values of a response variable \(y\) change as the predictor variable \(x\) changes. The equation of a line, relating \(x\) to \(y\) uses the slope-intercept form of a line, but with different letters than what you may be used to in a math class. We let \(b_{0}\) represent the sample \(y\)-intercept (the value of \(y\) when \(x = 0\)), \(b_{1}\) the sample slope (rise over run), and \(\hat{y}\) the predicted value of \(y\) for a specific value of \(x\). The equation is written as \(\hat{y} = b_{0} + b_{1}x\).
Some textbooks and the TI calculators use the letter \(a\) to represent the \(y\)-intercept and \(b\) to represent the slope, and the equation is written as \(\hat{y} = a + bx\). These letters are just symbols representing the placeholders for the numeric values for the \(y\)-intercept and slope.
If we were to fit the best line that was closest to all the points on the scatterplot we would get what we call the “line of best fit,” also known as the “regression equation” or “least squares regression line.” Figure 12-9 is a scatterplot with just five points.
Figure 12-10 shows the least-squares regression line of \(y\) on \(x\), which is the line that minimizes the squared vertical distance from all of the data. If we were to fit the line that best fits through the points, we would get the line pictured below.
.png?revision=1)
What we want to look for is the minimum of the squared vertical distance between each point and the regression equation, called a residual. This is where the name of the least squares regression line comes from. Figure 12-11 shows the squared residuals.
To find the slope and \(y\)-intercept for the equation of the least-squares regression line \(\hat{y} = b_{0} + b_{1} x\) we use the following formulas: slope \(= b_{1} = \frac{SS_{xy}}{SS_{xx}}\), \(y\)-intercept: \(b_{0} = \bar{y} - b_{1} \bar{x}\).
To compute the least squares regression line, you will need to first find the slope. Then substitute the slope into the following equation of the \(y\)-intercept: \(b_{0} = \bar{y} - b_{1} \bar{x}\), where \(\bar{x}\) = the sample mean of the \(x\)’s and \(\bar{y}\) = the sample mean of the \(y\)’s.
Once we find the equation for the regression line, we can use it to estimate the response variable \(y\) for a specific value of the predictor variable \(x\).
Note: we would only want to use the regression equation for prediction if we reject \(H_{0}\) and find that there is a significant correlation between \(x\) and \(y\). Alternatively, we could start with the regression equation and then test to see if the slope is significantly different from zero.
Use the following data to find the line of best fit.
.png?revision=1)
.png?revision=1)
.png?revision=1)
12.02: Hypothesis Test for Linear Regression
To test to see if the slope is significant we will be doing a two-tailed test with hypotheses. The population least squares regression line would be \(y = \beta_{0} + \beta_{1} + \varepsilon\) where \(\beta_{0}\) (pronounced “beta-naught”) is the population \(y\)-intercept, \(\beta_{1}\) (pronounced “beta-one”) is the population slope and \(\varepsilon\) is called the error term.
If the slope were horizontal (equal to zero), the regression line would give the same \(y\)-value for every input of \(x\) and would be of no use. If there is a statistically significant linear relationship then the slope needs to be different from zero. We will only do the two-tailed test, but the same rules for hypothesis testing apply for a one-tailed test.
We will only be using the two-tailed test for a population slope.
The hypotheses are:
\(H_{0}: \beta_{1} = 0\)
\(H_{1}: \beta_{1} \neq 0\)
The null hypothesis of a two-tailed test states that there is not a linear relationship between \(x\) and \(y\). The alternative hypothesis of a two-tailed test states that there is a significant linear relationship between \(x\) and \(y\).
Either a t-test or an F-test may be used to see if the slope is significantly different from zero. The population of the variable \(y\) must be normally distributed.
F-Test for Regression
An F-test can be used instead of a t-test. Both tests will yield the same results, so it is a matter of preference and what technology is available. Figure 12-12 is a template for a regression ANOVA table,
.png?revision=1)
where \(n\) is the number of pairs in the sample and \(p\) is the number of predictor (independent) variables; for now this is just \(p = 1\). Use the F-distribution with degrees of freedom for regression = \(df_{R} = p\), and degrees of freedom for error = \(df_{E} = n - p - 1\). This F-test is always a right-tailed test since ANOVA is testing the variation in the regression model is larger than the variation in the error.
Use an F-test to see if there is a significant relationship between hours studied and grade on the exam. Use \(\alpha\) = 0.05.
.png?revision=1)
.png?revision=1&size=bestfit&width=504&height=420)
.png?revision=1&size=bestfit&width=711&height=600)
.png?revision=1&size=bestfit&width=1068&height=194)
.png?revision=1&size=bestfit&width=1065&height=316)
T-Test for Regression
If the regression equation has a slope of zero, then every \(x\) value will give the same \(y\) value and the regression equation would be useless for prediction. We should perform a t-test to see if the slope is significantly different from zero before using the regression equation for prediction. The numeric value of t will be the same as the t-test for a correlation. The two test statistic formulas are algebraically equal; however, the formulas are different and we use a different parameter in the hypotheses.
The formula for the t-test statistic is \(t = \frac{b_{1}}{\sqrt{ \left(\frac{MSE}{SS_{xx}}\right) }}\)
Use the t-distribution with degrees of freedom equal to \(n - p - 1\).
The t-test for slope has the same hypotheses as the F-test:
\(H_{0}: \beta_{1} = 0\)
\(H_{1}: \beta_{1} \neq 0\)
Use a t-test to see if there is a significant relationship between hours studied and grade on the exam, use \(\alpha\) = 0.05.
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
12.02: Residuals
When we overlay the regression equation on a scatterplot, most of the time, the points do not lie on the line itself. The vertical distance between the actual value of \(y\) and the predicted value of \(\hat{y}\) is called the residual. The numeric value of the residual is found by subtracting the predicted value of \(y\) from the actual value of \(y\): \(y - \hat{y}\). When we find the line of best fit using least squares regression, this finds the regression equation with the smallest sum of the residuals \(\sum y - \hat{y}\).
When your residual is positive, then your data point is above the regression line, when the residual is negative, your data point is below the regression line. If you were to find the residuals for all the sample points and add them up you would get zero. The expected value of the residuals will always be zero. The regression equation is found so that there is just as much distance for the residuals above the line as there is below the line.
Find the residual for the point \((15, 80)\) for the exam data.
Standard Error of Estimate
The standard deviation of the residuals is called the standard error of estimate or \(s\). Some texts will use a subscript \(s_{e}\) or \(s_{est}\) to distinguish the different standard deviations from one another. When all of your data points line up in a perfectly straight line, \(s = 0\) since none of your points deviate from the regression line. As your data points get more scattered away from a regression line, \(s\) gets larger. When you are analyzing a regression model, you want \(s\) to be as small as possible.
Standard Error of Estimate
\[s_{est} = s = \sqrt{\frac{\sum \left(y_{i} - \hat{y}_{i}\right)^{2}}{n-2}} = \sqrt{MSE} \nonumber\]
The standard error of estimate is the standard deviation of the residuals. The standard error of estimate measures the deviation in the vertical distance from data points to the regression equation. The units of \(s\) are the same as the units of \(y\).
Use the exam data to find the standard error of estimate.
Solution
To find the \(\sum \left(y_{i} - \hat{y}_{i}\right)^{2}\) you would need to find the residual for every data point, square the residuals, and sum them up. This is a lot of math. Recall the regression ANOVA table found earlier. The MSE = 15.4912.
.png?revision=1)
The mean square error is the variance of the residuals, if we take the square root of the MSE we find the standard deviation of the residuals, which is the standard error of estimate.
\(s = \sqrt{MSE} = \sqrt{15.4912} = 3.9359\)
You can also use the technology to find \(s\).
.png?revision=1)
12.02: Coefficient of Determination
The coefficient of determination \(R^{2}\) (or \(r^{2}\)) is the fraction (or percent) of the variation in the values of \(y\) that is explained by the least-squares regression of \(y\) on \(x\). \(R^{2}\) is a measure of how well the values of \(y\) are explained by \(x\).
For example, there is some variability in the dependent variable values, such as grade. Some of the variation in student’s grades is due to hours studied and some is due to other factors. How much of the variation in a student’s grade is due to hours studied?
When considering this question, you want to look at how much of the variation in a student’s grade is explained by the number of hours they studied and how much is explained by other variables. Realize that some of the changes in grades have to do with other factors. You can have two students who study the same number of hours, but one student may have a higher grade. Some variability is explained by the model and some variability is not explained. Together, both of these give the total variability.
\[\begin{aligned} \text{(Total Variation)} &= \text{(Explained Variation)} & + \quad \text{(Unexplained Variation)} \\ \quad \sum \left(y - \bar{y}\right)^{2} &= \quad \sum \left(\hat{y} - \bar{y}\right)^{2} & + \sum \left(y - \bar{y}\right)^{2} \end{aligned}\]
Coefficient of Determination
The proportion of the variation that is explained by the model is \[R^{2} = \frac{\text{Explained Variation}}{\text{Total Variation}} = \frac{SSR}{SST} \nonumber\]
Find and interpret the coefficient of determination for the hours studied and exam grade data.
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1&size=bestfit&width=488&height=293)
.png?revision=1&size=bestfit&width=495&height=293)
12.02: Prediction
We can now use the least squares regression equation for prediction.
Use the regression equation to predict the grade for a student who has studied for 18 hours for their exam using the previous data.
Prediction Interval
We can find a special type of confidence interval to estimate the true value of \(y\) called the prediction interval.
The prediction interval is the confidence interval for the actual value of \(y\): \[\hat{y} \pm t_{\alpha / 2} \cdot s \sqrt{\left( 1 + \frac{1}{n} + \frac{(x - \bar{x})^{2}}{SS_{xx}}\right)} \nonumber\]
where \(\hat{y}\) is the predicted value of \(y\) for the given value of \(x\).
Using the previous data, find and interpret the 95% prediction interval for a student who studies 18 hours.
Solution
From the question, \(x = 18\). From previous examples we found that \(\hat{y} = 26.742 + 3.216346 \cdot 18 = 84.636228\) and \(s = 3.935892\). Find the critical value from the invT using \(df = n - 2 = 13\); we get \(t_{\alpha / 2} = 2.160369\). Make sure to go out at least 6 decimal places in between steps. Ideally, never round between steps. Use the 2-Var Stats from your calculator to find the sums and then substitute values back into the equation to get
\[\begin{aligned} & 84.636228 \pm 2.1600369 \cdot 3.935892 \sqrt{\left( 1 + \frac{1}{15} + \frac{(18 - 16.6)^{2}}{41.6} \right)} \\ & \Rightarrow \quad 84.636228 \pm 8.9723 \\ & \Rightarrow \quad 75.6639 < y < 93.6085 \end{aligned}\]
We are 95% confident that the predicted exam grade for a student that studies 18 hours is between 75.6639 and 93.6085.
A confidence interval can be more accurate (narrower) when you increase the sample size. Note that in the last example, the predicted grade for an individual student could have been anywhere from a C to an A grade. If you wanted to predict \(y\) with more accuracy, then you would want to sample more than 15 students to get a smaller margin of error. The confidence interval for a mean will have a smaller margin of error than for an individual’s predicted value.
Excel, the TI-83 and 84 do not have built in prediction intervals.
TI-89: Enter the \(x\)-values in list1 and the \(y\)-values in list2, select [F7] Intervals, then select option 7:LinRegTInt… Use the Var-Link button to enter in list1 and list2 for the X List and Y List. Select Response in the drop-down menu for Interval. Enter in the \(x\)-value given in the question. Change the confidence level (C-Level) to match what was in the question, the [Enter]. Scroll down to Pred Int for the prediction interval. The calculator does not round between steps so if you rounded \(b_{0}\) and \(b_{1}\), for instance, when doing hand calculations, your answer may be slightly different than the calculator results.
.png?revision=1)
Extrapolation is the use of a regression line for prediction far outside the range of values of the independent variable \(x\). As a general rule, one should not use linear regression to estimate values too far from the given data values. The further away you move from the center of the data set, the more variable results become. For instance, we would not want to estimate a student’s grade for someone that studied way less than 14 hours or more than 20 hours.
12.02: Outliers
A scatter plot should be checked for outliers. An outlier is a point that seems out of place when compared with the other points. Some of these points can affect the equation of the regression line.
Should linear regression be used with this data set?
.png?revision=1)
.png?revision=1)
.png?revision=1)
There is another type of outlier called an influential point. Influential points are positioned far away from the main cluster of data points on the \(y\)-axis. There is an option in most software packages to get the “standardized” residuals. Standardized residuals are z-scores of the residuals. Any standardized residual that is not between \(-2\) and \(2\) may be an outlier. If it is not between \(-3\) and \(3\) then the point is an outlier. When this happens, the points are called influential points or influential observations.
Use technology to compute the standardized residuals. Should linear regression be used with this data set?
.png?revision=1)
.png?revision=1)
12.02: Conclusion - Simple Linear Regression
A lurking variable is a variable other than the independent or dependent variables that may influence the regression line. For instance, the highly correlated ice cream sales and home burglary rates probably have to do with the season. Hence, linear regression does not imply cause and effect.
Two variables are confounded when their effects on the dependent variable cannot be distinguished from each other. For instance, if we are looking at diet predicting weight, a confounding variable would be age. As a person gets older, they can gain more weight with fewer calories compared to when they were younger. Another example would be predicting someone’s midterm score from hours studied for the exam. Some confounding variables would be GPA, IQ score, and teacher’s difficultly level.
Assumptions for Linear Regression
There are assumptions that need to be met when running simple linear regression. If these assumptions are not met, then one should use more advanced regression techniques.
The assumptions for simple linear regression are:
- The data need to follow a linear pattern.
- The observations of the dependent variable y are independent of one another.
- Residuals are approximately normally distributed.
- The variance of the residuals is constant.
Most software packages will plot the residuals for each \(x\) on the \(y\)-axis against either the \(x\)-variable or \(\hat{y}\) along the \(x\)-axis. This plot is called a residual plot. Residual plots help determine some of these assumptions.
Use technology to compute the residuals and make a residual plot for the hours studied and exam grade data.
.jpg?revision=2)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
Putting It All Together
High levels of hydrogen sulfide \((\mathrm{H}_{2} \mathrm{S})\) in the ocean can be harmful to animal life. It is expensive to run tests to detect these levels. A scientist would like to see if there is a relationship between sulfate \((\mathrm{SO}_{4})\) and \(\mathrm{H}_{2} \mathrm{S}\) levels, since \(\mathrm{SO}_{4}\) is much easier and less expensive to test in ocean water. A sample of \(\mathrm{SO}_{4}\) and \(\mathrm{H}_{2} \mathrm{S}\) were recorded together at different depths in the ocean. The sample is reported below in millimolar (mM). If there were a significant relationship, the scientist would like to predict the \(\mathrm{H}_{2} \mathrm{S}\) level when the ocean has an \(\mathrm{SO}_{4}\) level of 25 mM. Run a complete regression analysis and check the assumptions. If the model is significant, then find the 95% prediction interval to predict the sulfide level in the ocean when the sulfate level is 25 mM.
.png?revision=1)
.png?revision=2)
.png?revision=1)
.png?revision=1)
.png?revision=1)
.png?revision=1)
Summary
A simple linear regression should only be performed if you observe visually that there is a linear pattern in the scatterplot and that there is a statistically significant correlation between the independent and dependent variables. Use technology to find the numeric values for the \(y\)-intercept = \(a = b_{0}\) and slope = \(b = b_{1}\), then make sure to use the correct notation when substituting your numbers back in the regression equation \(\hat{y} = b_{0} + b_{1} x\). Another measure of how well the line fits the data is called the coefficient of determination \(R^{2}\). When \(R^{2}\) is close to 1 (or 100%), then the line fits the data very closely. The advantage over using \(R^{2}\) over \(r\) is that we can use \(R^{2}\) for nonlinear regression, whereas \(r\) is only for linear regression.
One should always check the assumptions for regression before using the regression equation for prediction. Make sure that the residual plots have a completely random horizontal band around zero. There should be no patterns in the residual plots such as a sideways V that may indicate a non-constant variance. A pattern like a slanted line, a U, or an upside-down U shape would suggest a non-linear model. Check that the residuals are normally distributed; this is not the same as the population being normally distributed. Check to make sure that there are no outliers. Be careful with lurking and confounding variables.
12.03: Multiple Linear Regression
A multiple linear regression line describes how two or more predictor variables affect the response variable \(y\). An equation of a line relating \(p\) independent variables to \(y\) is of the form for the population as: \(y = \beta_{0} + \beta_{1} x_{1} + \beta_{2} x_{2} + \cdots + \beta_{p} x_{p} + \varepsilon\), where \(\beta_{1}, \beta_{2}, \ldots, \beta_{p}\) are the slopes, \(\beta_{0}\) is the \(y\)-intercept and \(\varepsilon\) is called the error term.
We use sample data to estimate this equation using the predicted value of \(y\) as \(\hat{y}\) with the regression equation (also called the line of best fit or least squares regression line) as: \[y = b_{0} + b_{1} x_{1} + b_{2} x_{2} + \cdots + b_{p} x_{p} \nonumber\]
where \(b_{1}, b_{2}, \ldots, b_{p}\) are the slopes, and \(b_{0}\) is the \(y\)-intercept
For example, if we had two independent variables, we would have a 3-dimensional space as in Figure 12-25 where the red dots represent the sample data points and the equation would be a plane in the space represented by \(y = b_{0} + b_{1} x_{1} + b_{2} x_{2}\).
.png?revision=1)
The calculations use matrix algebra, which is not a prerequisite for this course. We will instead rely on a computer to calculate the multiple regression model.
If all the population slopes were equal to zero, the model \(y = \beta_{0} + \beta_{1} x_{1} + \beta_{2} x_{2} + \cdots + \beta_{p} x_{p} + \varepsilon\) would not be significant and should not be used for prediction. If one or more of the population slopes are not equal to zero then the model will be significant, meaning there is a significant relationship between the independent variables and the dependent variable and we may want to use this model for prediction. There are other statistics to look at to decide if this would be the best model to use. Those methods are discussed in more advanced courses.
The hypotheses will always have an equal sign in the null hypotheses.
The hypotheses are:
\(H_{0}: \beta_{1} = \beta_{2} = \cdots = \beta_{p} = 0\)
\(H_{1}:\) At least one slope is not zero.
Note that the alternative hypothesis is not written as \(H_{1}: \beta_{1} \neq \beta_{2} \neq \cdots \neq \beta_{p} \neq 0\). This is because we just want one or more of the independent variables to be significantly different from zero, not necessarily all the slopes unequal to zero.
Use the F-distribution with degrees of freedom for regression = \(df_{R} = p\), where \(p\) = the number of independent variables (predictors), and degrees of freedom for error = \(df_{E} = n - p - 1\), where \(n\) is the number of pairs. This is always a right-tailed ANOVA test, since we are testing if the variation in the regression model is larger than the variation in the error.
The test statistic and p-value are the last two values on the right in the ANOVA table. The p-value rule is easiest to use since the p-value is part of the outcome, but a critical value can be found using the invF program on your calculator or in Excel using =F.INV.RT(\(\alpha, df_{R}, df_{E}\)) We can also single out one independent variable at a time and use a t-test to see if the variable is significant by itself in predicting \(y\).
This would have hypotheses:
\(H_{0}: \beta_{i} = 0\)
\(H_{1}: \beta_{i} \neq 0\)
where \(i\) is a placeholder for whichever independent variable is being tested.
This t-test is found in the same row as the coefficient that you are testing.
Assumptions for Multiple Linear Regression
When doing multiple regression, the following assumptions need to be met:
- The residuals of the model are approximately normally distributed.
- The residuals of the model are independent (not autocorrelated) and have a constant variance (homoscedasticity).
- There is a liner relationship between the dependent variable and each independent variable.
- Independent variables are uncorrelated with each other (no multicollinearity).
The following is a schematic for the regression output for Microsoft Excel. Other software usually has a similar output but may have numbers in slightly different places. The blue spaces have the descriptions of the corresponding numbers.
.png?revision=1)
The coefficients column gives the numeric values to find the regression equation \(y = b_{0} + b_{1} x_{1} + b_{2} x_{2} + \cdots + b_{p} x_{p}\). The p-values for \(b_{i}\) should be investigated to see if the variable is statistically significant. One should also be careful that the independent variables are not significantly correlated amongst themselves. Correlated independent variables may give unexpected outcomes in the overall regression model and actually flip the sign on a coefficient.
A sample of 30 homes that were recently on the market were selected. The listing price in $1,000’s of the home, the livable square feet of the home, the lot size in 1,000’s of square feet and the number of bathrooms in the home were recorded. A multiple linear regression was done in Excel with the following output. Test to see if there is a significant relationship between the listing price of a home with the livable square feet, lot size, and number of bathrooms. If there is a relationship, then use the regression model to predict the listing price for a home that has 2,350 square feet, 3 bathrooms and has a 5,000 square foot lot. Use \(\alpha\) = 0.05.
.png?revision=1)
Solution
First, we need to test to see if the overall model is significant.
The hypotheses are:
\(H_{0}: \beta_{1} = \beta_{2} = \beta_{3} = 0\)
\(H_{1}:\) At least one slope is not zero.
The test statistic is \(F = 187.9217\) and the p-value = \(9.74E-18 \sim 0\)
.png?revision=1)
We reject \(H_{0}\), since the p-value is less than \(\alpha\) = 0.05. There is enough evidence to support the claim that there is a significant relationship between the number of bathrooms and lot size of a home with its listing price. Since we reject \(H_{0}\), we can use the regression model for prediction.
The question asked to predict the listing price for a home that has 2,350 square feet, 3 bathrooms and has a 5,000 square foot lot. This gives us \(x_{1} = 2350\), \(x_{2} = 5\) (5,000 square feet), and \(x_{3} = 3\).
The coefficients column has the values that correspond to the y-intercept and slopes gives the regression equation: \(\hat{y} = -28.8477 + 0.170908 \cdot x_{1} + 6.7705 \cdot x_{2} + 15.5347 \cdot x_{3}\).
.png?revision=1)
Substitute the three given x values into the equation in the correct order and you get \(\hat{y} = -28.8477 + 0.170908 \cdot 2350 + 6.7705 \cdot 5 + 15.5347 \cdot 3 = 453.2787\).
This then gives a predicted listing price of $453,278.
Note that our sample size is very small and we really need to check assumptions in order to use this predicted value with any reliability.
Is this the best model to use? Note that not all the p-values for each of the individual slope coefficients are significant. The number of bathrooms has a t-test statistic = 1.687038 and p-value = 0.10356, which is not statistically significant at the 5% level of significance.
.png?revision=1)
We may want to rerun the regression model without the number of bathroom variables and see if we get a higher \(R^{2}\) and a lower standard error of estimate. Ideally, we would try all the different combinations of independent variables and see which combination gives the best model. This is a lot of work to do if you have many independent variables. Most statistical software packages have built in functions that find the best fit.
Adjusted Coefficient of Determination
When we add more predictor variables into the model, this inflates the coefficient of variation, \(R^{2}\). In multiple regression, we adjust for this inflation using the following formula for adjusted coefficient of variation.
Adjusted Coefficient of Determination
\[R_{adj}^{2} = 1 - \left( \frac{\left(1 - R^{2}\right) (n-1)}{(n - p - 1)} \right) \nonumber\]
Use the previous example to verify the value of the adjusted coefficient of determination starting with the regular coefficient of determination \(R^{2} = 0.955915\).
Solution
First identify in the Excel output \(R^{2} = 0.955915\), \(n - 1 = df_{T} = 29\), and \(n - p - 1 = df_{E} = 26\).
.png?revision=1)
Substitute these values in and we get \(R_{adj}^{2} = 1 - \left(\frac{(1-0.955915)(29)}{(26)}\right) = 0.950828\). This is the same value as the adjusted \(R^{2}\) reported in the Excel output.
.png?revision=1)
The Excel output has both the adjusted coefficient of determination and the regular coefficient of determination. However, you may need the equation for the adjusted coefficient of determination depending on what information is given in a problem.
There are more types of regression models and more that should be done for a complete regression analysis. Ideally, you would find several models and pick the one with no outliers, the smallest standard error of estimate, a good residual plot, and the highest adjusted \(R^{2}\) and check the assumptions behind each model before using for prediction. More advanced techniques are discussed in a regression course.
“Well, I was in fact, I was moving backwards in time. Hmmm. Well, I think we've sorted all that out now. If you'd like to know, I can tell you that in your universe you move freely in three dimensions that you call space. You move in a straight line in a fourth, which you call time, and stay rooted to one place in a fifth, which is the first fundamental of probability. After that it gets a bit complicated, and there's all sorts of stuff going on in dimensions 13 to 22 that you really wouldn't want to know about. All you really need to know for the moment is that the universe is a lot more complicated than you might think…”
(Adams, 2002)
12.04: Chapter 12 Formulas
| \(SS_{xx} = (n-1) s_{rx}^{2}\) \(SS_{yy} = (n-1) s_{y}^{2}\) \(SS_{xy} = \sum (xy) - n \cdot \bar{x} \cdot \bar{y}\) |
Correlation Coefficient \(r = \frac{SS_{xy}}{\sqrt{\left(SS_{xx} \cdot SS_{yy}\right)}}\) |
| Correlation t-test \(H_{0}: \rho = 0\) \(H_{1}: \rho \neq 0\) \(t = r \sqrt{\left( \frac{n-2}{1-r^{2}} \right)}\) \(df = n-2\) |
Regression Equation (Line of Best Fit) \(\hat{y} = b_{0} + b_{1}x\) |
| Slope \(b_{1} = \frac{SS_{xy}}{SS_{xx}}\) |
y-Intercept \(b_{0} = \bar{y} - b_{1} \bar{x}\) |
| Slope t-test \(H_{0}: \beta_{1} = 0\) \(H_{1}: \beta_{1} \neq 0\) \(t = \frac{b_{1}}{\sqrt{ \left( \frac{MSE}{SS_{xx}} \right)}}\) \(df = n - p - 1 = n - 2\) |
Slope/Model F-test \(H_{0}: \beta_{1} = 0\) \(H_{1}: \beta_{1} \neq 0\) |
| Standard Error of Estimate \(s_{est} = \sqrt{ \frac{\sum \left(y_{i} - \hat{y}_{i}\right)^{2}}{n-2}} = \sqrt{MSE}\) |
Residual \(e_{i} = y_{i} - \hat{y}_{i}\) |
| Prediction Interval \(\hat{y} \pm t_{\alpha / 2} \cdot s_{est} \sqrt{\left( 1 + \frac{1}{n} + \frac{\left(x - \bar{x}\right)^{2}}{SS_{xx}} \right)}\) |
Coefficient of Determination \(R^{2} = (r)^{2} = \frac{SSR}{SST}\) |
| Multiple Linear Regression Equation \(\hat{y} = b_{0} + b_{1} x_{1} + b_{2} x_{2} + \cdot + b_{p} x_{p}\) |
Model F-Test for Multiple Regression \(H_{0}: \beta_{1} = \beta_{2} = \cdots = \beta_{p} = 0\) \(H_{1}:\) At least one slope is not zero. |
| Adjusted Coefficient of Determination \(R_{adj}^{2} = 1 - \left( \frac{\left(1 - R^{2}\right) (n-1)}{(n - p - 1)} \right)\) |
.png?revision=1)
12.05: Chapter 12 Exercises
1. The correlation coefficient, \(r\), is a number between _______________.
a) -1 and 1
b) -10 and 10
c) 0 and 10
d) 0 and \(\infty\)
e) 0 and 1
f) \(-\infty\) and \(\infty\)
2. To test the significance of the correlation coefficient, we use the t-distribution with how many degrees of freedom?
a) \(n - 1\)
b) \(n\)
c) \(n + 1\)
d) \(n - 2\)
e) \(n_{1} + n_{2} - 2\)
3. What are the hypotheses for testing to see if a correlation is statistically significant?
a) \(H_{0}: r = 0 \quad \ \ \ H_{1}: r \neq 0\)
b) \(H_{0}: \rho = 0 \quad \ \ \ H_{1}: \rho \neq 0\)
c) \(H_{0}: \rho = \pm 1 \quad H_{1}: \rho \neq \pm 1\)
d) \(H_{0}: r = \pm 1 \quad H_{1}: r \neq \pm 1\)
e) \(H_{0}: \rho = 0 \quad \ \ \ H_{1}: \rho = 1\)
4. The coefficient of determination is a number between _______________.
a) -1 and 1
b) -10 and 10
c) 0 and 10
d) 0 and \(\infty\)
e) 0 and 1
f) \(-\infty\) and \(\infty\)
5. Which of the following is not a valid linear regression equation?
a) \(\hat{y} = -5 + \frac{2}{9} x\)
b) \(\hat{y} = 3x + 2\)
c) \(\hat{y} = \frac{2}{9} - 5x\)
d) \(\hat{y} = 5 + 0.4 x\)
6. Body frame size is determined by a person's wrist circumference in relation to height. A researcher measures the wrist circumference and height of a random sample of individuals. The data is displayed below. Use \(\alpha\) = 0.05.
.png?revision=1)
.png?revision=1)
a) What is the value of the test statistic to see if the correlation is statistically significant?
i. 6.0205
ii. 1.16E-06
iii. 3.55E-08
iv. 5.2538
v. 7.2673
vi. 0.7499
vii. 0.7938
b) What is the correct p-value and conclusion for testing if there is a significant correlation?
i. 1.16E-06; there is a significant correlation.
ii. 3.55E-08; there is a significant correlation.
iii. 1.16E-06; There is not a significant correlation.
iv. 3.55E-08, There is not a significant correlation.
v. 0.7938, There is a significant correlation.
vi. 0.7938, There is not a significant correlation.
7. Bone mineral density and cola consumption have been recorded for a sample of patients. Let \(x\) represent the number of colas consumed per week and \(y\) the bone mineral density in grams per cubic centimeter. Assume the data is normally distributed. Calculate the correlation coefficient.
| \(x\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(y\) | 0.883 | 0.8734 | 0.8898 | 0.8852 | 0.8816 | 0.863 | 0.8634 | 0.8648 | 0.8552 | 0.8546 | 0.862 |
8. A teacher believes that the third homework assignment is a key predictor of how well students will do on the midterm. Let \(x\) represent the third homework score and y the midterm exam score. A random sample of last term’s students were selected and their grades are shown below. Assume scores are normally distributed. Use \(\alpha\) = 0.05.
| HW3 | Midterm | HW3 | Midterm | HW3 | Midterm | ||
|---|---|---|---|---|---|---|---|
| 13.1 | 59 | 6.4 | 43 | 20 | 86 | ||
| 21.9 | 87 | 20.2 | 79 | 15.4 | 73 | ||
| 8.8 | 53 | 21.8 | 84 | 25 | 93 | ||
| 24.3 | 95 | 23.1 | 92 | 9.7 | 52 | ||
| 5.4 | 39 | 22 | 87 | 15.1 | 70 | ||
| 13.2 | 66 | 11.4 | 54 | 15 | 65 | ||
| 20.9 | 89 | 14.9 | 71 | 16.8 | 77 | ||
| 18.5 | 78 | 18.4 | 76 | 20.1 | 78 |
a) State the hypotheses to test for a significant correlation.
b) Compute the correlation coefficient.
c) Compute the p-value to see if there is a significant correlation.
d) State the correct decision.
e) Is there a significant correlation?
f) Compute the coefficient of determination.
g) Write a sentence interpreting \(R^{2}\).
h) Does doing poorly on homework 3 cause a student to do poorly on the midterm exam? Explain.
i) Find the standard error of estimate.
9. The sum of the residuals should be ________.
a) a
b) b
c) 0
d) 1
e) \(r\)
10. An object is thrown from the top of a building. The following data measure the height of the object from the ground for a five-second period. Calculate the correlation coefficient.
| Seconds | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 |
|---|---|---|---|---|---|---|---|---|---|---|
| Height | 112.5 | 110.875 | 106.8 | 100.275 | 91.3 | 79.875 | 70.083 | 59.83 | 30.65 | 0 |
11. A teacher believes that the third homework assignment is a key predictor of how well students will do on the midterm. Let \(x\) represent the third homework score and \(y\) the midterm exam score. A random sample of last term’s students were selected and their grades are shown below. Assume scores are normally distributed. Use \(\alpha\) = 0.05.
| HW3 | Midterm | HW3 | Midterm | HW3 | Midterm | ||
|---|---|---|---|---|---|---|---|
| 13.1 | 59 | 6.4 | 43 | 20 | 86 | ||
| 21.9 | 87 | 20.2 | 79 | 15.4 | 73 | ||
| 8.8 | 53 | 21.8 | 84 | 25 | 93 | ||
| 24.3 | 95 | 23.1 | 92 | 9.7 | 52 | ||
| 5.4 | 39 | 22 | 87 | 15.1 | 70 | ||
| 13.2 | 66 | 11.4 | 54 | 15 | 65 | ||
| 20.9 | 89 | 14.9 | 71 | 16.8 | 77 | ||
| 18.5 | 78 | 18.4 | 76 | 20.1 | 78 |
a) Compute the regression equation.
b) Compute the predicted midterm score when the homework 3 score is 15.
c) Compute the residual for the point \((15, 65)\).
d) Find the 95% prediction interval for the midterm score when the homework 3 score is 15.
12. Body frame size is determined by a person's wrist circumference in relation to height. A researcher measures the wrist circumference and height of a random sample of individuals. The data is displayed below.
a) Which is the correct regression equation?
i. \(\hat{y} = 31.6304 + 5.4496\)
ii. \(\hat{y} = 31.6304 + 5.4496 x\)
iii. \(\hat{y} = 5.4496 + 31.6304 x\)
iv. \(\hat{y} = 31.6304 + 5.2538 x\)
v. \(y = 31.6304 + 5.4496 x\)
b) What is the predicted height (in inches) for a person with a wrist circumference of 7 inches?
c) Which number is the standard error of estimate?
d) Which number is the coefficient of determination?
e) Compute the correlation coefficient.
f) What is the correct test statistic for testing if the slope is significant \(H_{1}: \beta_{1} \neq 0\)?
g) What is the correct p-value for testing if the slope is significant \(H_{1}: \beta_{1} \neq 0\)?
h) At the 5% level of significance, is there a significant relationship between wrist circumference and height?
13. Bone mineral density and cola consumption have been recorded for a sample of patients. Let \(x\) represent the number of colas consumed per week and \(y\) the bone mineral density in grams per cubic centimeter. Assume the data is normally distributed. Calculate the coefficient of determination.
| \(x\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(y\) | 0.883 | 0.8734 | 0.8898 | 0.8852 | 0.8816 | 0.863 | 0.8634 | 0.8648 | 0.8552 | 0.8546 | 0.862 |
14. Bone mineral density and cola consumption have been recorded for a sample of patients. Let \(x\) represent the number of colas consumed per week and \(y\) the bone mineral density in grams per cubic centimeter. Assume the data is normally distributed. A regression equation for the following data is \(\hat{y} = 0.8893 - 0.0031x\). Which is the best interpretation of the slope coefficient?
| \(x\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(y\) | 0.883 | 0.8734 | 0.8898 | 0.8852 | 0.8816 | 0.863 | 0.8634 | 0.8648 | 0.8552 | 0.8546 | 0.862 |
a) For every additional average weekly soda consumption, a person’s bone density increases by 0.0031 grams per cubic centimeter.
b) For every additional average weekly soda consumption, a person’s bone density decreases by 0.0031 grams per cubic centimeter.
c) For an increase of 0.8893 in the average weekly soda consumption, a person’s bone density decreases by 0.0031 grams per cubic centimeter.
d) For every additional average weekly soda consumption, a person’s bone density decreases by 0.8893 grams per cubic centimeter.
15. Which residual plot has the best linear regression model?
.png?revision=1)
a) a
b) b
c) c
d) d
e) e
f) f
16. An object is thrown from the top of a building. The following data measure the height of the object from the ground for a five-second period.
| Seconds | 0.5 | 1 | 1.5 | 2 | 2.5 | 3 | 3.5 | 4 | 4.5 | 5 |
|---|---|---|---|---|---|---|---|---|---|---|
| Height | 112.5 | 110.875 | 106.8 | 100.275 | 91.3 | 79.875 | 70.083 | 59.83 | 30.65 | 0 |
The following four plots were part of the regression analysis.
.png?revision=1)
There is a statistically significant correlation between time and height, \(r = -0.942\), p-value = 0.0000454. Should linear regression be used for this data? Why or why not? Choose the correct answer.
a) Yes, the p-value indicates that there is a significant correlation so we can use linear regression.
b) Yes, the normal probability plot has a nice curve to it.
c) Yes, there is a nice straight line in the line fit plot.
d) No, there is a curve in the residual plot, normal plot and the scatterplot.
17. The following data represent the leaching rates (percent of lead extracted vs. time in minutes) for lead in solutions of magnesium chloride \((\mathrm{MgCl}_{2})\). Use \(\alpha = 0.05\).
| Time \((x)\) | 4 | 8 | 16 | 30 | 60 | 120 |
|---|---|---|---|---|---|---|
| Percent Extracted \((y)\) | 1.2 | 1.6 | 2.3 | 2.8 | 3.6 | 4.4 |
a) State the hypotheses to test for a significant correlation.
b) Compute the correlation coefficient.
c) Compute the p-value to see if there is a significant correlation.
d) State the correct decision.
e) Is there a significant correlation?
f) Compute the coefficient of determination.
g) Compute the regression equation.
h) Compute the 95% prediction interval for 100 minutes.
i) Write a sentence interpreting this interval using units and context.
18. Bone mineral density and cola consumption have been recorded for a sample of patients. Let x represent the number of colas consumed per week and y the bone mineral density in grams per cubic centimeter. Assume the data is normally distributed. What is the residual for the observed point \((7, 0.8634)\).
| \(x\) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 | 11 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| \(y\) | 0.883 | 0.8734 | 0.8898 | 0.8852 | 0.8816 | 0.863 | 0.8634 | 0.8648 | 0.8552 | 0.8546 | 0.862 |
19. A study was conducted to determine if there was a linear relationship between a person's age and their peak heart rate. Use \(\alpha = 0.05\).
| Age \((x)\) | 16 | 26 | 32 | 37 | 42 | 53 | 48 | 21 |
|---|---|---|---|---|---|---|---|---|
| Peak Heart Rate \((y)\) | 220 | 194 | 193 | 178 | 172 | 160 | 174 | 214 |
a) What is the estimated regression equation that relates number of hours worked and test scores for high school students.
b) Interpret the slope coefficient for this problem.
c) Compute and interpret the coefficient of determination.
d) Compute the coefficient of nondetermination.
e) Compute the standard error of estimate.
f) Compute the correlation coefficient.
g) Compute the 95% Prediction Interval for peak heart rate for someone who is 25 years old.
20. The following data represent the weight of a person riding a bike and the rolling distance achieved after going down a hill without pedaling.
| Weight (lbs) | 59 | 84 | 97 | 56 | 103 | 87 | 88 | 92 | 53 | 66 | 71 | 100 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Rolling distance (m) | 26 | 43 | 48 | 20 | 59 | 44 | 48 | 46 | 28 | 32 | 39 | 49 |
a) Can it be concluded at a 0.05 level of significance that there is a linear correlation between the two variables?
b) Using the regression line for this problem, find the predicted bike rolling distance for a person that weighs 110 lbs.
c) Find the 99% prediction interval for bike rolling distance for a person that weighs 110 lbs.
21. Body frame size is determined by a person's wrist circumference in relation to height. A researcher measures the wrist circumference and height of a random sample of individuals. The Excel output and scatterplot are displayed below. Find the regression equation and predict the height (in inches) for a person with a wrist circumference of 7 inches. Then, compute the residual for the point \((7, 75)\).
.png?revision=1)
22. It has long been thought that the length of one’s femur is positively correlated to the length of one’s tibia. The following are data for a classroom of students who measured each (approximately) in inches. A significant linear correlation was found between the two variables. Find the 90% prediction interval for the length of someone’s tibia when it is known that their femur is 23 inches long.
| Femur Length | 18.7 | 20.5 | 16.2 | 15.0 | 19.0 | 21.3 | 21.0 | 14.3 | 15.8 | 18.8 | 18.7 |
|---|---|---|---|---|---|---|---|---|---|---|---|
| Tibia Length | 14.2 | 15.9 | 13.1 | 12.4 | 16.2 | 15.8 | 16.2 | 12.1 | 13.0 | 14.3 | 13.8 |
23. The data below represent the driving speed (mph) of a vehicle and the corresponding gas mileage (mpg) for several recorded instances.
| Driving Speed | Gas Mileage | Driving Speed | Gas Mileage | |
|---|---|---|---|---|
| 57 | 21.8 | 62 | 21.5 | |
| 66 | 20.9 | 66 | 20.5 | |
| 42 | 25.0 | 67 | 23.0 | |
| 34 | 26.2 | 52 | 19.4 | |
| 44 | 24.3 | 49 | 25.3 | |
| 44 | 26.3 | 48 | 24.3 | |
| 25 | 26.1 | 41 | 28.4 | |
| 20 | 27.2 | 38 | 29.6 | |
| 24 | 23.5 | 26 | 32.5 | |
| 42 | 22.6 | 24 | 30.8 | |
| 52 | 19.4 | 21 | 28.8 | |
| 54 | 23.9 | 19 | 33.5 | |
| 60 | 24.8 | 24 | 25.1 |
a) Do a hypothesis test to see if there is a significant correlation. Use \(\alpha = 0.10\).
b) Compute the standard error of estimate.
c) Compute the regression equation and use it to find the predicted gas mileage when a vehicle is driving at 77 mph.
d) Compute the 90% prediction interval for gas mileage when a vehicle is driving at 77 mph.
24. The following data represent the age of a car and the average monthly cost for repairs. A significant linear correlation is found between the two variables. Use the data to find a 95% prediction interval for the monthly cost of repairs for a vehicle that is 15 years old.
| Age of Car (yrs) | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Monthly Cost ($) | 25 | 34 | 42 | 45 | 55 | 71 | 82 | 88 | 87 | 90 |
25. In a sample of 20 football players for a college team, their weight and 40-yard-dash time in minutes were recorded.
| Weight (lbs) | 40-Yard Dash (min) | Weight (lbs) | 40-Yard Dash (min) | |
|---|---|---|---|---|
| 285 | 5.95 | 195 | 4.85 | |
| 185 | 4.99 | 254 | 5.12 | |
| 165 | 4.92 | 140 | 4.87 | |
| 188 | 4.77 | 212 | 5.05 | |
| 160 | 4.52 | 158 | 4.75 | |
| 156 | 4.67 | 188 | 4.87 | |
| 256 | 5.22 | 134 | 4.53 | |
| 169 | 4.95 | 205 | 4.92 | |
| 210 | 5.06 | 178 | 4.88 | |
| 165 | 4.83 | 159 | 4.79 |
a) Do a hypothesis test to see if there is a significant correlation. Use \(\alpha = 0.01\).
b) Compute the standard error of estimate.
c) Compute the regression equation and use it to find the predicted 40-yard-dash time for a football player that is 200 lbs.
d) Compute the 99% prediction interval for a football player that is 200 lbs.
e) Write a sentence interpreting the prediction interval.
26. The following data represent the enrollment at a small college during its first ten years of existence. A significant linear relationship is found between the two variables. Find a 90% prediction interval for the enrollment after the college has been open for 14 years.
| Years | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | 10 |
|---|---|---|---|---|---|---|---|---|---|---|
| Enrollment | 856 | 842 | 923 | 956 | 940 | 981 | 1025 | 996 | 1057 | 1088 |
27. A new fad diet called Trim-to-the-MAX is running some tests that they can use in advertisements. They sample 25 of their users and record the number of days each has been on the diet along with how much weight they have lost in pounds. The data is below. A significant linear correlation was found between the two variables. Find the 95% prediction interval for the weight lost when a person has been on the diet for 60 days.
| Days on Diet | 7 | 12 | 16 | 19 | 25 | 34 | 39 | 43 | 44 | 49 |
|---|---|---|---|---|---|---|---|---|---|---|
| Weight Loss | 5 | 7 | 12 | 15 | 20 | 25 | 24 | 29 | 33 | 35 |
28. An elementary school uses the same system to test math skills at their school throughout the course of the 5 grades at their school. The age and score (out of 100) of several students is displayed below. A significant linear relationship is found between the student’s age and their math score. Find a 90% prediction interval for the score a student would earn given that they are 5 years old.
| Student Age | 6 | 6 | 7 | 8 | 8 | 9 | 10 | 11 | 11 |
|---|---|---|---|---|---|---|---|---|---|
| Math Score | 54 | 42 | 50 | 61 | 67 | 65 | 71 | 72 | 79 |
29. The intensity (in candelas) of a 100-watt light bulb was measured by a sensing device at various distances (in meters) from the light source. A linear regression was run and the following residual plot was found.
.png?revision=1)
a) Is linear regression a good model to use?
b) Write a sentence explaining your answer.
30. The table below shows the percentage of adults in the United States who were married before age 24 for the years shown. A significant linear relationship was found between the two variables.
| Year | 1960 | 1965 | 1970 | 1975 | 1980 | 1985 | 1990 | 1995 | 2000 | 2005 |
|---|---|---|---|---|---|---|---|---|---|---|
| % Married Before 24 Years Old | 52.1 | 51.3 | 45.9 | 46.3 | 40.8 | 38.1 | 34.0 | 32.6 | 28.1 | 25.5 |
a) Compute the regression equation.
b) Predict the percentage of adults who married before age 24 in the United States in 2015.
c) Compute the 95% prediction interval for the percentage of adults who married before age 24 in the United States in 2015.
31. A nutritionist feels that what mothers eat during the months they are nursing their babies is important for healthy weight gain of their babies. She samples several of her clients and records their average daily caloric intake for the first three months of their babies’ lives and also records the amount of weight the babies gained in those three months. The data are below.
| Daily Calories | 1523 | 1649 | 1677 | 1780 | 1852 | 2065 | 2096 | 2145 | 2378 |
|---|---|---|---|---|---|---|---|---|---|
| Baby's Weight Gain (lbs) | 4.62 | 4.77 | 4.62 | 5.12 | 5.81 | 5.34 | 5.89 | 5.96 | 6.05 |
a) Compute the regression equation.
b) Test to see if the slope is significantly different from zero, using \(\alpha = 0.05\).
c) Predict the weight gain of a baby whose mother gets 2,500 calories per day.
d) Compute the 95% prediction interval for the weight gain of a baby whose mother gets 2,500 calories per day.
32. The data below show the predicted average high temperature \(({}^{\circ} \mathrm{F})\) per month by the Farmer’s Almanac in Portland, Oregon alongside the actual high temperature per month that occurred.
| Farmer's Almanac | 45 | 50 | 57 | 62 | 69 | 72 | 81 | 90 | 78 | 64 | 51 | 48 |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Actual High | 46 | 52 | 60 | 61 | 72 | 78 | 82 | 95 | 85 | 68 | 52 | 49 |
a) Compute the regression equation.
b) Test to see if the slope is significantly different from zero, using \(\alpha = 0.01\).
c) Predict the high temperature in the coming year, given that the Farmer’s Almanac is predicting the high to be \(58^{\circ} \mathrm{F}\).
d) Compute the 99% prediction interval for the actual high temperature in the coming year, given that the Farmer’s Almanac is predicting the high to be \(58^{\circ} \mathrm{F}\).
33. In a multiple linear regression problem, p represents:
a) The number of dependent variables in the problem.
b) The probability of success.
c) The number of independent variables in the problem.
d) The probability of failure.
e) The population proportion.
34. The manager of a warehouse found a significant relationship exists between the number of weekly hours clocked \((x_{1}\)), the age of the employee \((x_{2})\), and the productivity of the employee \((y)\) (measured in weekly orders assembled). A multiple regression was run and the following line of best fit was found: \(\hat{y} = 66.238 + 2.7048 x_{1} - 0.7275 x_{2}\). Approximate the productivity level of an employee who clocked 61 hours in a given week and is 59 years old.
35. A multiple regression test concludes that there is a linear relationship and finds the following line of best fit: \(\hat{y} = -53.247 + 12.594 x_{1} - 0.648 x_{2} + 4.677 x_{3}\). Use the line of best fit to approximate \(y\) when \(x_{1} = 5\), \(x_{2} = 12\), \(x_{3} = 2\).
36. A career counselor feels that the strongest predictors of a student’s success after college are class attendance \((x_{1})\) (recorded as a percent) and GPA \((x_{2})\). To test this, she samples clients after they have found a job and lets the dependent variable be the number of weeks it took them to find a job. A multiple regression was run and the following line of best fit was found \(\hat{y} = 38.6609 - 0.3345 x_{1} - 1.0743 x_{2}\). Predict the number of weeks it will take a client to find a job, given that she had a 94% \((x_{1} = 94)\) attendance rate in college and a 3.82 GPA.
37. A study conducted by the American Heart Association provided data on how age, blood pressure and smoking relate to the risk of strokes. The following data is the SPSS output with Age in years \((x_{1})\), Blood Pressure in mmHg \((x_{2})\), Smoker (0 = Nonsmoker, 1 = Smoker) \((x_{3})\) and the Risk of a Stroke as a percent \((y)\).
.png?revision=1)
.png?revision=1)
a) Use \(\alpha = 0.05\) to test the claim that the regression model is significant. State the hypotheses, test statistic, p-value, decision and summary.
b) Use the estimated regression equation to predict the stroke risk for a 70-year-old smoker with a blood pressure of 180.
c) Interpret the slope coefficient for age.
d) Find the adjusted coefficient of determination.
38. Use technology to run a multiple linear regression with a dependent variable of annual salary for adults with full-time jobs in San Francisco and independent variables of years of education completed and age. The data for the sample used is below.
| Annual Salary | Years of Education | Age |
|---|---|---|
| 82,640 | 17 | 34 |
| 95,854 | 18 | 42 |
| 152,320 | 21 | 39 |
| 49,165 | 13 | 25 |
| 31,120 | 12 | 31 |
| 67,500 | 16 | 32 |
| 42,590 | 12 | 28 |
| 57,245 | 14 | 55 |
| 58,940 | 16 | 45 |
| 67,250 | 18 | 40 |
| 56,120 | 16 | 39 |
| 38,955 | 12 | 34 |
| 74,650 | 16 | 33 |
| 53,495 | 16 | 29 |
| 67,210 | 16 | 30 |
| 79,365 | 16 | 50 |
| 96,045 | 18 | 51 |
| 78,472 | 14 | 60 |
| 124,975 | 21 | 52 |
| 43,125 | 12 | 36 |
a) Test to see if the overall model is significant using \(\alpha = 0.05\).
b) Compute the predicted salary for a 35-year-old with 16 years of education.
c) Compute the adjusted coefficient of determination.
d) Are all the independent variables statistically significant? Explain.
39. A study was conducted to determine if there was a linear relationship between a person's weight in pounds with their gender, height and activity level. A person’s gender was recorded as a 0 for anyone who identified as male and 1 for those who did not identify as male. Height was measured in inches. Activity level was coded as 1, 2, or 3; the more active the person was, the higher the value.
.png?revision=1)
a) Interpret the slope coefficient for height.
b) Predict the weight for a male who is 70 inches tall and has an activity level of 2.
c) Calculate the adjusted coefficient of determination.
40. A professor claims that the number of hours a student studies for their final paired with the number of homework assignments completed throughout the semester (out of 20 total assignment opportunities) is a good predictor of a student’s final exam grade. She collects a sample of students and records the following data.
| Final Exam Score | # Hours Studied | HW Completed |
|---|---|---|
| 85 | 5.2 | 19 |
| 74 | 5.1 | 18 |
| 79 | 4.9 | 16 |
| 62 | 3.7 | 12 |
| 96 | 9 | 17 |
| 52 | 1 | 12 |
| 73 | 2 | 15 |
| 81 | 2 | 16 |
| 90 | 7.25 | 18 |
| 79 | 4.5 | 14 |
| 83 | 3 | 20 |
| 76 | 3.2 | 17 |
| 92 | 6.8 | 20 |
| 84 | 6.2 | 18 |
| 50 | 4.1 | 10 |
a) Test the professor’s claim using \(\alpha = 0.05\).
b) Compute the multiple correlation coefficient.
c) Interpret the slope coefficient for number of hours studied.
d) Compute the adjusted coefficient of determination.
- Solutions to Odd-Numbered Exercises
-
1. a
3. b
5. d
7. \(r = 0.8241\)
9. c
11. a) \(\hat{y} = 25.6472 + 2.8212 x\)
b) \(67.9657\)
c) \(-2.9657\)
d) \(61.3248 < y < 74.6065\)13. \(R^{2} = 0.679\)
15. a
17. a) \(H_{0}: \rho = 0; H_{1}: \rho \neq 0\)
b) \(r = 0.9403\)
c) \(0.0052\)
d) Reject \(H_{0}\)
e) Yes
f) \(R^{2} = 0.8842\)
g) \(\hat{y} = 1.6307 - 0.0257x\)
h) \(2.6154 < y < 5.7852\)
i) We can be 95% confident that the predicted percent of lead extracted in solutions of magnesium chloride at 100 minutes is anywhere between 2.6154 and 5.7852.19. a) \(\hat{y} = 241.8127 - 1.5618x\)
b) Every year a person ages, their peak heart rate decreases by an average of 1.5618.
c) \(R^{2} = 0.93722\)
d) \(0.06278\)
e) \(5.6924\)
f) \(r = -0.9681\)21. 5.2224
23. a) \(H_{0}: \rho = 0; H_{1}: \rho \neq 0; t = -5.2514;\) p-value = \(0.000022\).
Reject \(H_{0}\). There is a significant correlation between driving speed of a vehicle and the corresponding gas mileage.
b) 2.49433
c) \(\hat{y} = 32.40313 - 0.1662x\)
d) \(14.8697 < y < 24.3424\)25. a) \(H_{0}: \rho = 0; H_{1}: \rho \neq 0; t = 6.9083;\) p-value = \(0.000007\).
Reject \(H_{0}\). There is a significant correlation between a football player’s weight and 40-yard-dash time.
b) \(s = 0.160595\)
c) \(\hat{y} = 3.722331 - 0.006396x; 5.0016\)
d) \(4.527 < y < 5.476\)
e) We can be 99% confident that the predicted time for all 200-pound football player’s running time for the 40-yard-dash is between 4.527 and 5.476 minutes.27. \(36.605 < y < 47.740\)
29. a) No
b) The p-value = 0.00000304 suggests that there is a significant linear relationship between intensity (in candelas) of a 100-watt light bulb was measured by a sensing device at various distances (in meters) from the light source. However, the residual plot clearly shows a nonlinear relationship. Even though we can fit a straight line through the points, we would get a better fit with a curve.31. a) \(\hat{y} = 1.76267 - 0.001883x\)
b) \(H_{0}: \beta_{1} = 0; H_{1}: \beta_{1} \neq 0; t = 5.0668;\) p-value = \(0.0015\)
Reject \(H_{0}\). There is significant linear relationship between a nursing baby’s weight gain and the calorie intake of the mother.
c) \(6.4693 \mathrm{~lbs}\)
d) \(5.571 < y < 7.368\)33. c
35. 11.301
37. a) \(H_{0}: \beta_{1} = \beta_{2} = \beta_{3} = 0; H_{1}:\) At least one slope is not zero\(; F = 36.823;\) p-value = \(0.000000204\)
Reject \(H_{0}\). There is a significant relationship between a person’s age, blood pressure and smoking status and the risk of a stroke.
b) \(37.731%\)
c) For each year a person ages, their chance of a stroke increases by 1.077%.
d) \(84.9%\)39. a) For each additional inch in height, the predicted weight would increase 3.7 pounds.
b) \(171.34 \mathrm{~lbs}\)
c) \(65.56%\)
From Mostly Harmless Statistics by Rachel L. Webb, adapted from LibreTexts. Licensed CC BY-SA 4.0. XYZ Homework OER web edition, adapted with 7 verified corrections (see errata).