14.5 Regression

Multiple Linear Regression

The General Linear Model $y = β_{0} + β_{1} x_{1} + β_{2} x_{2} + . . . + β_{k} x_{k} + ϵ$ for multiple regression analysis where $y$ is the response variable we want to predict, $β sub 0, β sub 1,..., β sub k$ are unknown constants, $x sub 1, x sub 2,..., x sub k$ are independent predictor variables that are measured without error, and $ϵ$ is the random error.

$\hat{y} = b_{0} + b_{1} x_{1} + b_{2} x_{2} + . . . + b_{k} x_{k}$ can be determined using the method of Least Squares as follows.

import statsmodels.formula.api as smf
model = smf.ols(formula='y ~ x1 + x2 +...+ xk', data=x_df)
results = model.fit()

print(results.summary())

Example: The dataset in the website (URL below) includes 20 columns about home sale, including price, number of bedrooms and bathrooms, areas etc. The data includes 21,613 entries.

Write a program to display basic information and the first five rows. x_df.info and x_df.head.

kc_house_data.csv

import pandas as pd

url = 'kc_house_data.csv'

x_df = pd.read_csv(url)

Interpretation: The above code uses pandas to read and save the dataset as pandas dataframe.

x_df.info()

Show expected output

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 21613 entries, 0 to 21612
Data columns (total 21 columns):
 #   Column         Non-Null Count  Dtype  
---  ------         --------------  -----  
 0   id             21613 non-null  int64  
 1   date           21613 non-null  object 
 2   price          21613 non-null  float64
 3   bedrooms       21613 non-null  int64  
 4   bathrooms      21613 non-null  float64
 5   sqft_living    21613 non-null  int64  
 6   sqft_lot       21613 non-null  int64  
 7   floors         21613 non-null  float64
 8   waterfront     21613 non-null  int64  
 9   view           21613 non-null  int64  
 10  condition      21613 non-null  int64  
 11  grade          21613 non-null  int64  
 12  sqft_above     21613 non-null  int64  
 13  sqft_basement  21613 non-null  int64  
 14  yr_built       21613 non-null  int64  
 15  yr_renovated   21613 non-null  int64  
 16  zipcode        21613 non-null  int64  
 17  lat            21613 non-null  float64
 18  long           21613 non-null  float64
 19  sqft_living15  21613 non-null  int64  
 20  sqft_lot15     21613 non-null  int64  
dtypes: float64(5), int64(15), object(1)
memory usage: 3.5+ MB

Interpretation: There are 20 columns and 21,613 entries.

x_df.head()

	id	date	price	bedrooms	bathrooms	sqft_living	sqft_lot	floors	condition	grade	sqft_above	sqft_basement	yr_built	yr_renovated	zipcode	lat	long	sqft_living15	sqft_lot15
0	7129300520	20141013T000000	221900.0	3	1.00	1180	5650	1.0	3	7	1180	0	1955	0	98178	47.5112	-122.257	1340	5650
1	6414100192	20141209T000000	538000.0	3	2.25	2570	7242	2.0	3	7	2170	400	1951	1991	98125	47.7210	-122.319	1690	7639
2	5631500400	20150225T000000	180000.0	2	1.00	770	10000	1.0	3	6	770	0	1933	0	98028	47.7379	-122.233	2720	8062
3	2487200875	20141209T000000	604000.0	4	3.00	1960	5000	1.0	5	7	1050	910	1965	0	98136	47.5208	-122.393	1360	5000
4	1954400510	20150218T000000	510000.0	3	2.00	1680	8080	1.0	3	8	1680	0	1987	0	98074	47.6168	-122.045	1800	7503

Example: The dataset in the website (URL below) includes 20 columns about home sale, including price, number of bedrooms and bathrooms, areas etc. The data includes 21,613 entries.

Write a program to find the least-squares prediction equation, $y = β_{0} + β_{1} x_{1} + β_{2} x_{2} + β_{3} x_{3} + ϵ$ for these data where, $y$ is price, $x sub 1$ is bedrooms, $x sub 2$ bathrooms and $x sub 3$ is number of floors.
Use the overall F test to determine whether the model contributes significant information to the prediction of $y$ with 1% level of significance.
What is the value of R-squared. That is, the coefficient of determination.
Does the number of floors, $x sub 3$ , contribute significant information for the prediction of the price, $y$ , given that $x sub 1$ and $x sub 2$ are already in the model? (use $α equals 0.05$ )

kc_house_data.csv

import statsmodels.formula.api as smf
model = smf.ols(formula='price ~ bedrooms+bathrooms+floors', data=x_df)
results = model.fit()

print(results.summary())

Show expected output

                            OLS Regression Results                            
==============================================================================
Dep. Variable:                  price   R-squared:                       0.278
Model:                            OLS   Adj. R-squared:                  0.278
Method:                 Least Squares   F-statistic:                     2769.
Date:                Tue, 07 Sep 2021   Prob (F-statistic):               0.00
Time:                        21:32:40   Log-Likelihood:            -3.0409e+05
No. Observations:               21613   AIC:                         6.082e+05
Df Residuals:                   21609   BIC:                         6.082e+05
Df Model:                           3                                         
Covariance Type:            nonrobust                                         
==============================================================================
                 coef    std err          t      P>|t|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept   -2.91e+04   9209.213     -3.160      0.002   -4.72e+04   -1.11e+04
bedrooms    2.002e+04   2680.838      7.469      0.000    1.48e+04    2.53e+04
bathrooms   2.385e+05   3681.888     64.766      0.000    2.31e+05    2.46e+05
floors     -1737.4719   4569.452     -0.380      0.704   -1.07e+04    7218.991
==============================================================================
Omnibus:                    17359.610   Durbin-Watson:                   1.961
Prob(Omnibus):                  0.000   Jarque-Bera (JB):           905901.534
Skew:                           3.476   Prob(JB):                         0.00
Kurtosis:                      33.946   Cond. No.                         20.4
==============================================================================

Warnings:
[1] Standard Errors assume that the covariance matrix of the errors is correctly specified.

Interpretation:

See x_df.info and x_df.head above
$\hat{y} = b_{0} + b_{1} x_{1} + b_{2} x_{2} + b_{3} x_{3}$ where $b_{0} = - 2.91 e + 04$ , $b_{1} = 2.002 e + 04, b_{2} = 2.385 e + 05, b_{3} = - 1737.4719$
The F-statistic is 2769 with p-value of Prob (F-statistic) 0.00. Thus, the model has a significant contiribution to the prediction of price.
The coefficient of determination (R-squared:) is 0.278.
The p-value (P>|t|) for the floors is 0.704 which is not less than $α equals 0.05$ . It is not significant.

Logistic Regression

Logistic regression is the type of regression analysis used to find the probability of a certain event occurring. Best suited for cases where we have a categorical dichotomous dependent variable.

$p = \frac{1}{1 + e^{β_{0} + β_{1} x_{1} + . . . + β_{k} x_{k}}}$ where $p$ is probability that $y equals 1$ and $y$ is the response variable with values 1s and 0s.

Example: The data set below (URL) contains the occurrence of choronary heart disease (chd), tobacco, obesity, age, alcohol, etc. For chd, 0 means patient has no chd and 1 means patient has chd.

Write a program to read the data set and display the first five rows.
Write a program to fit a logistic regression model where the dependent variable is chd (coronary heart disease) and the predictor variables are tobacco, obesity and age.
Use the LLR p-value test to determine whether the model contributes significant information to the prediction of $y$ , chd, with 1% level of significance.

heart.csv this dataset came from https://web.stanford.edu/~hastie/ElemStatLearn//datasets/SAheart.data

import pandas as pd
x_df = pd.read_csv('heart.csv')
x_df.head()

	row.names	sbp	tobacco	ldl	adiposity	famhist	typea	obesity	alcohol	age	chd
0	1	160	12.00	5.73	23.11	Present	49	25.30	97.20	52	1
1	2	144	0.01	4.41	28.61	Absent	55	28.87	2.06	63	1
2	3	118	0.08	3.48	32.28	Present	52	29.14	3.81	46	0
3	4	170	7.50	6.41	38.03	Present	51	31.99	24.26	58	1
4	5	134	13.60	3.50	27.78	Present	60	25.99	57.34	49	1

x_df.info()

Show expected output

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 462 entries, 0 to 461
Data columns (total 11 columns):
 #   Column     Non-Null Count  Dtype  
---  ------     --------------  -----  
 0   row.names  462 non-null    int64  
 1   sbp        462 non-null    int64  
 2   tobacco    462 non-null    float64
 3   ldl        462 non-null    float64
 4   adiposity  462 non-null    float64
 5   famhist    462 non-null    object 
 6   typea      462 non-null    int64  
 7   obesity    462 non-null    float64
 8   alcohol    462 non-null    float64
 9   age        462 non-null    int64  
 10  chd        462 non-null    int64  
dtypes: float64(5), int64(5), object(1)
memory usage: 39.8+ KB

Example: The data set below (URL) contains the occurrence of coronary heart disease (chd), tobacco, obesity, age, alcohol, etc. For chd, 0 means patient has no chd and 1 means patient has chd.

Write a program to fit a logistic regression model where the dependent variable is chd (coronary heart disease) and the predictor variables are tobacco, obesity and age.
Use the LLR p-value test to determine whether the model contributes significant information to the prediction of $y$ , chd, with 1% level of significance.

heart.csv this dataset came from https://web.stanford.edu/~hastie/ElemStatLearn//datasets/SAheart.data

import pandas as pd
x_df = pd.read_csv('heart.csv')

import statsmodels.formula.api as smf
model = smf.logit(formula='chd ~ tobacco + obesity + age', data=x_df)
results = model.fit()

print(results.summary())

Show expected output

Optimization terminated successfully.
         Current function value: 0.557771
         Iterations 6
                           Logit Regression Results                           
==============================================================================
Dep. Variable:                    chd   No. Observations:                  462
Model:                          Logit   Df Residuals:                      458
Method:                           MLE   Df Model:                            3
Date:                Tue, 07 Sep 2021   Pseudo R-squ.:                  0.1354
Time:                        21:32:40   Log-Likelihood:                -257.69
converged:                       True   LL-Null:                       -298.05
Covariance Type:            nonrobust   LLR p-value:                 2.142e-17
==============================================================================
                 coef    std err          z      P>|z|      [0.025      0.975]
------------------------------------------------------------------------------
Intercept     -3.4213      0.753     -4.541      0.000      -4.898      -1.945
tobacco        0.0788      0.026      3.061      0.002       0.028       0.129
obesity        0.0019      0.026      0.074      0.941      -0.050       0.054
age            0.0538      0.009      5.808      0.000       0.036       0.072
==============================================================================

Interpretation:

$\hat{p} = \frac{1}{1 + e^{- 3.4213 + 0.0788 x_{1} + 0.0019 x_{2} + 0.0538 x_{3}}}$
LLR p-value 2.142e-17 is less than $α$ = 0.01. Thus, the model has a significant contribution to the prediction of price.

Adapted from Python for Introductory Statistics, by Simon Aman (Truman College, City Colleges of Chicago), licensed under CC BY 4.0. Changes were made: reformatted as an accessible XYZ web edition with live in-browser code cells. License: CC-BY-4.0.