12.3 Linear Regression

Watch demo video

We can use the variables x and y to model a linear relationship between x and y with a linear equation of the form $y=b+mx$ .

The Least-Squares method can be used to obtain the predicted y values $\hat{y}$ as follows:

$\hat y=a+bx$

Where

$b=slope=r\times\frac{s_y}{s_x}$ and $a = \bar{y}-b\bar{x}$

Where $s_y$ and $s_x$ are std for x and y. The following code can be used to obtain the slope and intercept.

from scipy import stats
stats.linregress(x, y)

Example: The midterm exam out of a total of 100 points, x, and final exam out of a total of 200 points, y, scores for a random sample of 11 students is listed below. Find the slope and intercept of the regression line. Test if the correlation between x and y is significant at 0.05 level of significance.

x	65	67	71	71	66	75	67	70	71	69	69
y	145	133	185	163	126	198	153	163	159	151	159

x = [65, 67, 71, 71, 66, 75, 67, 70, 71, 69, 69]
y = [145, 133, 185, 163, 126, 198, 153, 163, 159, 151, 159]
from scipy import stats
stats.linregress(x, y)

Show expected output

LinregressResult(slope=6.364142538975501, intercept=-282.5556792873051, rvalue=0.8797451071731136, pvalue=0.00035593568153235875, stderr=1.1464698617902889)

Interpretation: slope=6.364142538975501, intercept=-282.5556792873051, rvalue=0.8797451071731136 (this is Pearson correlation coefficient), pvalue=0.00035593568153235875 (this p-value is for testing the significance of the correlation coefficient), stderr=1.1464698617902889

Since the p-value is less 0.05, there is significant correlation between x and y.

slope=6.364142538975501, intercept=-282.5556792873051.

Thus, $\hat{y}=-282.56+6.36x$ is the regression line with values rounded to two decimal places.

Example: he number of hours studied, x, and test scores, y, for 10 students are listed below. Find the the correlation coefficient (r-value), estimated slope and y-intercept of the regression line and the p-value for testing the significance of correlation.

A scatter of ten green dots — hours studied against exam score for ten students — with two straight lines through the cloud: the least-squares line as a dashed red guide, and an adjustable dark blue line set at a slope of 6 points per hour and an intercept of 47. The vertical axis is plotted in tens of points, so a score of 70 sits at 7.0. The blue line is the slightly steeper of the two: it starts below the dashed line and crosses it about five hours in. No setting of it makes the total squared distance to the ten dots smaller than the dashed line's — which is what least squares means. Adjustable parameters: Slope b1 (m) = 6 points per hour, Intercept b0 (b) = 47 points. Viewing window: x from -4.59 to 11.59, y from -0.4 to 9.6. — The ten students of this example, with the least-squares line (dashed) fixed and a second line whose slope and intercept are sliders. Both sliders read in the book's own units - points per hour and points - but the vertical axis is plotted in TENS of points (a score of 70 sits at 7.0), because the viewer derives both axes from one scale and a 0-to-7 hour axis against a 51-to-90 point axis is otherwise unplottable. Start where it opens, at slope 6.00 and intercept 47.0: a perfectly reasonable line through the cloud, with a sum of squared residuals of 164.00. Now drag toward the dashed line at slope 5.5646 and intercept 49.2109 and the residual sum bottoms out at 152.04. It cannot be beaten - that is the entire content of 'least squares', and it is a claim about every OTHER line that a picture of the winning line alone cannot make.

x	2	0	3	5	4	7	4	6	5	1
y	60	55	65	69	70	90	71	88	79	51

from scipy import stats

x=[2,0,3,5,4,7,4,6,5,1]
y=[60,55,65,69,70,90,71,88,79,51]

stats.linregress(x,y)

Show expected output

LinregressResult(slope=5.564625850340137, intercept=49.21088435374149, rvalue=0.9485857685252971, pvalue=2.8725087369610658e-05, stderr=0.6564713967897522)

Interpretation:

slope=5.564625850340137, intercept=49.21088435374149, r-value=0.9485857685252971, pvalue=2.8725087369610658e-05

import matplotlib.pyplot as plt
import numpy as np


x=[2,0,3,5,4,7,4,6,5,1]
y=[60,55,65,69,70,90,71,88,79,51]

x = np.array(x)
y = np.array(y)

y_hat = 5.564625850340137*x + 49.21088435374149

plt.scatter(x,y)
plt.plot(x,y_hat)

Show expected output

[<matplotlib.lines.Line2D at 0x7fa3d937f8d0>]

The hours-studied scatter plot with the fitted regression line ŷ = 5.56x + 49.21 drawn through the ten points.

Adapted from Python for Introductory Statistics, by Simon Aman (Truman College, City Colleges of Chicago), licensed under CC BY 4.0. Changes were made: reformatted as an accessible XYZ web edition with live in-browser code cells. License: CC-BY-4.0.