#set document(title: "4.3 Correlation and Linear Regression Analysis", author: "OpenStax / XYZ Homework") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 4.3#h(0.6em)Correlation and Linear Regression Analysis === Learning Outcomes By the end of this section, you should be able to: - Create scatterplots and calculate and interpret correlation coefficients. - Perform linear regression and determine the best-fit linear equation. - Use Python to calculate correlation coefficients and determine equations of linear regression models. We briefly introduced #strong[correlation analysis] at the beginning of this chapter, but now we want to dig in a little deeper. Data scientists are often interested in knowing if there are relationships, or a #emph[correlation], between two numeric quantities. For example, a medical researcher might be interested in investigating if there is a relationship between age and blood pressure. A college professor might be interested to know if there is a relationship between time spent on social media by students and corresponding academic grades. Correlation analysis allows for the determination of a statistical relationship between two numeric quantities, or variables—an independent variable and a dependent variable. The #strong[independent variable] is the variable that you can change or control, while the #strong[dependent variable] is what you measure or observe to see how it responds to those changes. Regression analysis takes this a step further by quantifying the relationship between the two variables, and it can be used to predict one quantity based on a second quantity, assuming there is a significant correlation between the two quantities. For example, the value of a car can be predicted based on the age of the car. Earlier we noted many business-related applications where correlation and regression analysis are used. For instance, regression analysis can be used to establish a mathematical equation that relates a dependent variable (such as sales) to an independent variable (such as advertising expenditure). But it has many applications apart from business as well. An automotive engineer is interested in the correlation between outside temperature and battery life for an electric vehicle, for instance. Our discussion here will focus on linear regression—analyzing the relationship between one dependent variable and one independent variable, where the relationship can be modeled using a linear equation. === Scatterplots and Correlation In correlation analysis, we study the relationship between #strong[bivariate data], which is data collected on two variables where the data values are paired with one another. Correlation measures the association between two numeric variables. We may be interested in knowing if there is a correlation between bond prices and interest rates or between the age of a car and the value of the car. To investigate the correlation between two numeric quantities, the first step is to collect #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data for the two numeric quantities of interest and then create a scatterplot that will graph the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] ordered pairs. The independent, or explanatory, quantity is labeled the #math.equation(block: false, alt: "x")[$x$]-variable, and the dependent, or response, quantity is labeled the #math.equation(block: false, alt: "y")[$y$]-variable. For example, let’s say that a financial analyst wants to know if the price of Nike stock is correlated with the value of the #strong[S&P 500] (Standard & Poor’s 500 stock market index). To investigate this, monthly data can be collected for Nike stock prices and value of the S&P 500 for a period of time, and a scatterplot can be created and examined. A #strong[scatterplot, or scatter diagram], is a graphical display intended to show the relationship between two variables. The setup of the scatterplot is that one variable is plotted on the horizontal axis and the other variable is plotted on the vertical axis. Each pair of data values is considered as an #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] point, and the various points are plotted on the diagram. A visual inspection of the plot is then made to detect any patterns or trends on the scatter diagram. shows the relationship between the Nike stock price and its S&P value on a monthly basis over a one-year time period. #figure(table( columns: 3, align: left, inset: 6pt, table.header([#strong[Date]], [#strong[S&P 500 (#math.equation(block: false, alt: "x")[$x$])]], [#strong[Nike Stock Price (#math.equation(block: false, alt: "y")[$y$])]]), [Month 1], [2912.43], [87.18], [Month 2], [3044.31], [98.58], [Month 3], [3100.29], [98.05], [Month 4], [3271.12], [97.61], [Month 5], [3500.31], [111.89], [Month 6], [3363.00], [125.54], [Month 7], [3269.96], [120.08], [Month 8], [3621.63], [134.70], [Month 9], [3756.07], [141.47], [Month 10], [3714.24], [133.59], [Month 11], [3811.15], [134.78], [Month 12], [3943.34], [140.45], )) To assess #strong[linear correlation], examine the graphical trend of the data points on the scatterplot to determine if a straight-line pattern exists. If a linear pattern exists, the correlation may indicate either a positive or a negative correlation. A positive correlation indicates that as the independent variable increases, the dependent variable tends to increase as well, or as the independent variable decreases, the dependent variable tends to decrease (the two quantities move in the same direction). A negative correlation indicates that as the independent variable increases, the dependent variable decreases, or as the independent variable decreases, the dependent variable increases (the two quantities move in opposite directions). If there is no relationship or association between the two quantities, where one quantity changing does not affect the other quantity, we conclude that there is no correlation between the two variables. #figure(figph[A scatter plot for Nike stock against S&P 500. The diagram shows that the Nike stock price rises from approximately 85 to 140 as the S&P 500 rises from approximately 2900 to 4000. The data points generally align along an upwardly sloping line.], alt: "A scatter plot for Nike stock against S&P 500. The diagram shows that the Nike stock price rises from approximately 85 to 140 as the S&P 500 rises from approximately 2900 to 4000. The data points generally align along an upwardly sloping line.", caption: [Scatterplot of Nike Stock Price (\$) and Value of S&P 500 #linebreak() (source: Yahoo! Finance)]) From the scatterplot in the Nike stock versus S&P 500 example, we note that the trend reflects a positive correlation in that as the value of the S&P 500 increases, the price of Nike stock tends to increase as well. When inspecting a scatterplot, it may be difficult to assess a correlation based on a visual inspection of the graph alone. A more precise assessment of the correlation between the two quantities can be obtained by calculating the numeric correlation coefficient (referred to using the symbol #math.equation(block: false, alt: "r")[$r$]). The #strong[correlation coefficient] is a measure of the strength and direction of the correlation between the independent variable #math.equation(block: false, alt: "x")[$x$] and the dependent variable #math.equation(block: false, alt: "y")[$y$]. The formula for #math.equation(block: false, alt: "r")[$r$] is shown; however, software is typically used to calculate the correlation coefficient. #math.equation(block: true, alt: "r equals the fraction n Σ x y minus open parenthesis Σ x close parenthesis open parenthesis Σ y close parenthesis over the square root of n Σ x squared minus open parenthesis Σ x close parenthesis squared the square root of n Σ y squared minus open parenthesis Σ y close parenthesis squared")[$r = frac(n Σ x y − ( Σ x ) ( Σ y ), sqrt(n Σ x^(2) − attach(( Σ x ), t: 2)) sqrt(n Σ y^(2) − attach(( Σ y ), t: 2)))$] where #math.equation(block: false, alt: "n")[$n$] refers to the number of data pairs and the symbol #math.equation(block: false, alt: "Σ x")[$Σ x$] indicates to sum the #emph[x]-values. Note that this value of “#math.equation(block: false, alt: "r")[$r$]” is sometimes referred to as the “Pearson correlation coefficient.” provides a step-by-step procedure on how to calculate the correlation coefficient #math.equation(block: false, alt: "r")[$r$]. #figure(table( columns: 2, align: left, inset: 6pt, table.header([#strong[Step]], [#strong[Representation in Symbols]]), [1. Calculate the sum of the #emph[x]-values.], [#math.equation(block: false, alt: "Σ x")[$Σ x$]], [2. Calculate the sum of the #emph[y]-values.], [#math.equation(block: false, alt: "Σ y")[$Σ y$]], [3. Multiply each #emph[x]-value by the corresponding #emph[y]-value and calculate the sum of these #math.equation(block: false, alt: "x y")[$x y$] products.], [#math.equation(block: false, alt: "Σ x y")[$Σ x y$]], [4. Square each #emph[x]-value and then calculate the sum of these squared values.], [#math.equation(block: false, alt: "Σ x squared")[$Σ x^(2)$]], [5. Square each #emph[y]-value and then calculate the sum of these squared values.], [#math.equation(block: false, alt: "Σ y squared")[$Σ y^(2)$]], [6. Determine the value of #math.equation(block: false, alt: "n")[$n$], which is the number of data pairs.], [#emph[n]], [7. Use these results to then substitute into the formula for the correlation coefficient.], [#math.equation(block: false, alt: "r equals the fraction n Σ x y minus open parenthesis Σ x close parenthesis open parenthesis Σ y close parenthesis over the square root of n Σ x squared minus open parenthesis Σ x close parenthesis squared the square root of n Σ y squared minus open parenthesis Σ y close parenthesis squared")[$r = frac(n Σ x y − ( Σ x ) ( Σ y ), sqrt(n Σ x^(2) − attach(( Σ x ), t: 2)) sqrt(n Σ y^(2) − attach(( Σ y ), t: 2)))$]], )) Note that since #math.equation(block: false, alt: "r")[$r$] is calculated using sample data, #math.equation(block: false, alt: "r")[$r$] is considered a #emph[sample statistic] and is used to measure the strength of the correlation for the two population variables. Recall that sample data indicates data based on a subset of the entire population. As mentioned, given the complexity of this calculation, software is typically used to calculate the correlation coefficient. There are several options for calculating the correlation coefficient in Python. The example shown uses the #strong[scipy.stats library] in Python which includes the function pearsonr() for calculating the Pearson correlation coefficient (#math.equation(block: false, alt: "r")[$r$]). We will create two numerical arrays in Python using the #strong[np.array() function] and then use the pearsonr() function to calculate the Pearson correlation coefficient for the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data shown. The following Python code provides the correlation coefficient for the Nike Stock Price dataset as 0.923. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import libraries import numpy as np from scipy.stats  import pearsonr   \# establish x and y data, x-data is S&P500 value, y-data is Nike Stock Price SP500 = np.array(\[2912.43, 3044.31, 3100.29, 3271.12, 3500.31, 3363.00, 3269.96, 3621.63, 3756.07, 3714.24, 3811.15, 3943.34\]) Nike = np.array(\[87.18, 98.58, 98.05, 97.61, 111.89, 125.54, 120.08, 134.70, 141.47, 133.59, 134.78, 140.45\]) \#pearson r returns both value of r and corresponding p-value r, p = pearsonr(SP500, Nike) \#print value of r , rounded to 3 decimal places print("Correlation coefficient: ", round(r, 3)) The resulting output will look like this: Correlation coefficient: 0.923 ] ==== Interpret a Correlation Coefficient Once the value of #math.equation(block: false, alt: "r")[$r$] is calculated, this measurement provides two indicators for the correlation: + the strength of the correlation based on the #emph[value] of #math.equation(block: false, alt: "r")[$r$] + the direction of the correlation based on the #emph[sign] of #math.equation(block: false, alt: "r")[$r$] The #emph[value] of #math.equation(block: false, alt: "r")[$r$] gives us this information: - The value of #math.equation(block: false, alt: "r")[$r$] is always between #math.equation(block: false, alt: "minus 1")[$− 1$] and #math.equation(block: false, alt: "plus 1")[$+ 1$]: #math.equation(block: false, alt: "minus 1 less than or equal to r less than or equal to 1")[$− 1 ≤ r ≤ 1$]. - The size of the correlation #math.equation(block: false, alt: "r")[$r$] indicates the strength of the linear relationship between the two variables. Values of #math.equation(block: false, alt: "r")[$r$] close to #math.equation(block: false, alt: "minus 1")[$− 1$] or to #math.equation(block: false, alt: "plus 1")[$+ 1$] indicate a stronger linear relationship. - If #math.equation(block: false, alt: "r equals 0")[$r = 0$], there is no linear relationship between the two variables (#emph[no linear correlation]). - If #math.equation(block: false, alt: "r equals 1")[$r = 1$], there is perfect positive correlation. If #math.equation(block: false, alt: "r equals minus 1")[$r = − 1$], there is perfect negative correlation. In both of these cases, all the original data points lie on a straight line. The sign of #math.equation(block: false, alt: "r")[$r$] gives us this information: - A positive value of #math.equation(block: false, alt: "r")[$r$] means that when #math.equation(block: false, alt: "x")[$x$] increases, #math.equation(block: false, alt: "y")[$y$] tends to increase, and when #math.equation(block: false, alt: "x")[$x$] decreases, #math.equation(block: false, alt: "y")[$y$] tends to decrease (#emph[positive correlation]). - A negative value of #math.equation(block: false, alt: "r")[$r$] means that when #math.equation(block: false, alt: "x")[$x$] increases, #math.equation(block: false, alt: "y")[$y$] tends to decrease, and when #math.equation(block: false, alt: "x")[$x$] decreases, #math.equation(block: false, alt: "y")[$y$] tends to increase (#emph[negative correlation]). From the statistical results shown in the Python output, the correlation coefficient #math.equation(block: false, alt: "r")[$r$] is 0.923, which indicates that the relationship between Nike stock and the value of the S&P 500 over this time period represents a strong, positive correlation. ==== Test a Correlation Coefficient for Significance The correlation coefficient, #math.equation(block: false, alt: "r")[$r$], tells us about the strength and direction of the linear relationship between #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$]. The sample data are used to compute #math.equation(block: false, alt: "r")[$r$], the correlation coefficient for the sample. If we had data for the entire population (that is, all measurements of interest), we could find the population correlation coefficient, which is labeled as the Greek letter #math.equation(block: false, alt: "ρ")[$ρ$] (pronounced “rho”). But because we have only sample data, we cannot calculate the population correlation coefficient. The sample correlation coefficient, #math.equation(block: false, alt: "r")[$r$], is our estimate of the unknown population correlation coefficient. - #math.equation(block: false, alt: "ρ equals population correlation coefficient (unknown)")[$ρ = "population correlation coefficient (unknown)"$] - #math.equation(block: false, alt: "r equals sample correlation coefficient (known; calculated from sample data)")[$r = "sample correlation coefficient (known; calculated from sample data)"$] An important step in the correlation analysis is to determine if the correlation is significant. By this, we are asking if the correlation is strong enough to allow meaningful #strong[predictions] for #math.equation(block: false, alt: "y")[$y$] based on values of #math.equation(block: false, alt: "x")[$x$]. One method to test the significance of the correlation is to employ a hypothesis test. The hypothesis test lets us decide whether the value of the population correlation coefficient #math.equation(block: false, alt: "ρ")[$ρ$] is close to zero or significantly different from zero. We decide this based on the sample correlation coefficient #math.equation(block: false, alt: "r")[$r$] and the sample size n. If the test concludes that the correlation coefficient is significantly different from zero, we say that the correlation coefficient is significant. - Conclusion: There is sufficient evidence to conclude that there is a significant linear relationship between the two variables because the correlation coefficient is significantly different from zero. - What the conclusion means: There is a significant linear relationship between the two variables. If the test concludes that the correlation coefficient is not significantly different from zero (it is close to zero), we say that correlation coefficient is not significant. A hypothesis test can be performed to test if the correlation is significant. A hypothesis test is a statistical method that uses sample data to test a claim regarding the value of a population parameter. In this case, the hypothesis test will be used to test the claim that the population correlation coefficient #math.equation(block: false, alt: "ρ")[$ρ$] is equal to zero. Use these hypotheses when performing the hypothesis test: - Null hypothesis: #math.equation(block: false, alt: "H sub 0 : ρ equals 0")[$H_(0) : ρ = 0$] - Alternate hypothesis: #math.equation(block: false, alt: "H sub a : ρ not equal to 0")[$H_(a) : ρ ≠ 0$] The hypotheses can be stated in words as follows: - Null hypothesis #math.equation(block: false, alt: "H sub 0")[$H_(0)$]: The population correlation coefficient #emph[is not] significantly different from zero. There #emph[is not] a significant linear relationship (correlation) between #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$] in the population. - Alternate hypothesis #math.equation(block: false, alt: "H sub a")[$H_(a)$]: The population correlation coefficient is significantly different from zero. There #emph[is] a significant linear relationship (correlation) between #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$] in the population. A quick shorthand way to test correlations is the relationship between the sample size and the correlation. If #math.equation(block: false, alt: "| r | greater than or equal to the fraction 2 over the square root of n")[$| r | ≥ frac(2, sqrt(n))$], then this implies that the correlation between the two variables demonstrates that a linear relationship exists and is statistically significant at approximately the 0.05 level of significance. As the formula indicates, there is an inverse relationship between the sample size and the required correlation for significance of a linear relationship. With only 10 observations, the required correlation for significance is 0.6325; for 30 observations, the required correlation for significance decreases to 0.3651; and at 100 observations, the required level is only 0.2000. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Interpreting the Significance of the Sample Correlation Coefficient] - If #math.equation(block: false, alt: "r")[$r$] is significant and the scatterplot shows a linear trend, the line can be used to predict the value of #math.equation(block: false, alt: "y")[$y$] for values of #math.equation(block: false, alt: "x")[$x$] that are within the domain of observed #emph[x]-values. - If #math.equation(block: false, alt: "r")[$r$] is not significant OR if the scatterplot does not show a linear trend, the line should #emph[not] be used for prediction. - If #math.equation(block: false, alt: "r")[$r$] is significant and the scatterplot shows a linear trend, the line may #emph[not] be appropriate or reliable for prediction #emph[outside] the domain of observed #emph[x]-values in the data. ] #examplebox("Example 1")[][ Suppose that the chief financial officer (CFO) of a corporation is investigating the correlation between stock prices and unemployment rate over a period of 10 years and finds the correlation coefficient to be #math.equation(block: false, alt: "minus 0.68")[$− 0.68$]. There are 10 #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data points in the dataset. Should the CFO conclude that the correlation is significant for the relationship between stock prices and unemployment rate based on a level of significance of 0.05? #solutionbox[ When using a level of significance of 0.05, if #math.equation(block: false, alt: "| r | greater than or equal to the fraction 2 over the square root of n")[$| r | ≥ frac(2, sqrt(n))$], then this implies that the correlation between the two variables demonstrates a linear relationship that is statistically significant at approximately the 0.05 level of significance. In this example, we check if #math.equation(block: false, alt: "| minus 0.68 |")[$| − 0.68 |$] is greater than or equal to #math.equation(block: false, alt: "the fraction 2 over the square root of n")[$frac(2, sqrt(n))$] where #math.equation(block: false, alt: "n equals 10")[$n = 10$]. Since #math.equation(block: false, alt: "the fraction 2 over the square root of 10")[$frac(2, sqrt(10))$] is approximately 0.632, this indicates that the #emph[r]-value of #math.equation(block: false, alt: "minus 0.68")[$− 0.68$] is greater than #math.equation(block: false, alt: "the fraction 2 over the square root of n")[$frac(2, sqrt(n))$], and thus the correlation is deemed significant. ] ] === Linear Regression A regression model is typically developed once the correlation coefficient has been calculated and a determination has been made that the correlation is significant. The regression model can then be used to predict one quantity (the dependent variable) based on the other (the independent variable). For example, a business may want to establish a correlation between the amount the company spent on #strong[advertising] (the independent variable) versus its recorded sales (the dependent variable). If a strong enough correlation is established, then the business manager can predict sales based on the amount spent on advertising for a given period. In this discussion we will focus on linear regression, where a straight line is used to model the relationship between the two variables. Once a straight-line model is developed, this model can then be used to predict the value of the dependent variable for a specific value of the independent variable. ==== Method of Least Squares and Residuals To create a linear model that best fits the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] points on a scatterplot, researchers generally use the #strong[least squares method] based on a mathematical algorithm designed to minimize the squared distances from the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data points to the fitted line. Recall from algebra that the equation of a straight line is given by: #math.equation(block: true, alt: "y equals m x plus b")[$y = m x + b$] where #math.equation(block: false, alt: "m")[$m$] is the slope of the line and #math.equation(block: false, alt: "b")[$b$] is the #emph[y]-intercept of the line. The slope measures the steepness of the line, and the #emph[y]-intercept is that point on the #math.equation(block: false, alt: "y")[$y$]-axis where the graph crosses, or intercepts, the #math.equation(block: false, alt: "y")[$y$]-axis. In linear regression analysis, the equation of the straight line is written in a slightly different way using the model #math.equation(block: true, alt: "y ^ equals a plus b x")[$hat(y) = a + b x$] In this format, #math.equation(block: false, alt: "b")[$b$] is the slope of the line, and #math.equation(block: false, alt: "a")[$a$] is the #emph[y]-intercept of the line. The notation #math.equation(block: false, alt: "y ^")[$hat(y)$] is called #math.equation(block: false, alt: "y")[$y$]-hat and is used to indicate a predicted value of the dependent variable #math.equation(block: false, alt: "y")[$y$] for a certain value of the independent variable #math.equation(block: false, alt: "x")[$x$]. If a line extends uphill from left to right, the slope is a positive value (see ; if the line extends downhill from left to right, the slope is a negative value). #figure(figph[Three separate boxes show line graphs labeled a, b, and c, respectively. Graph a shows a line sloping upward to the right. Graph b shows a horizontal line. Graph c shows a line sloping downward to the right.], alt: "Three separate boxes show line graphs labeled a, b, and c, respectively. Graph a shows a line sloping upward to the right. Graph b shows a horizontal line. Graph c shows a line sloping downward to the right.", caption: [Three examples of graphs of #math.equation(block: false, alt: "y ^ equals a plus b x")[$hat(y) = a + b x$]. (a) If #math.equation(block: false, alt: "b greater than 0")[$b > 0$], the line slopes upward to the right. (b) If #math.equation(block: false, alt: "b equals 0")[$b = 0$], the line is horizontal. (c) If #math.equation(block: false, alt: "b less than 0")[$b < 0$], the line slopes downward to the right.]) When generating the equation of a line in algebra using #math.equation(block: false, alt: "y equals m x plus b")[$y = m x + b$], two #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] points were required to generate the equation. However, in regression analysis, all the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] points in the dataset will be utilized to develop the linear regression model. The first step in any regression analysis is to create the scatterplot. Then proceed to calculate the correlation coefficient #math.equation(block: false, alt: "r")[$r$] and check this value for significance. If we think that the points show a linear relationship, we would like to draw a line on the scatterplot. This line can be calculated through a process called linear regression. However, we only calculate a regression line if one of the variables helps to explain or predict the other variable. If #math.equation(block: false, alt: "x")[$x$] is the independent variable and #math.equation(block: false, alt: "y")[$y$] the dependent variable, then we can use a regression line to predict #math.equation(block: false, alt: "y")[$y$] for a given value of #math.equation(block: false, alt: "x")[$x$]. As an example of a regression equation, assume that a correlation exists between the monthly amount spent on advertising and the monthly #strong[revenue] for a Fortune 500 company. After collecting #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data for a certain time period, the company determines the regression equation is of the form #math.equation(block: true, alt: "y ^ equals 9376.7 plus 61.8 x")[$hat(y) = 9376.7 + 61.8 x$] where #math.equation(block: false, alt: "x")[$x$] represents the monthly amount spent on advertising (in thousands of dollars), #math.equation(block: false, alt: "y ^")[$hat(y)$] represents the monthly revenues for the company (in thousands of dollars), the slope is 61.8, and the #emph[y]-intercept is 9376.7. A scatterplot of the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data is shown. #figure(figph[A scatter plot of revenue against ad spend. The diagram shows revenue increasing from approximately 12 million dollars to 19 million dollars as advertising spend increases from approximately 50,000 dollars to 150,000 dollars.], alt: "A scatter plot of revenue against ad spend. The diagram shows revenue increasing from approximately 12 million dollars to 19 million dollars as advertising spend increases from approximately 50,000 dollars to 150,000 dollars.", caption: [Scatterplot of Revenue versus Advertising for a Fortune 500 Company (\$000s)]) The company would like to predict the monthly revenue if its executives decide to spend \$150,000 on advertising next month. To determine the estimate of monthly revenue, let #math.equation(block: false, alt: "x equals 150")[$x = 150$] in the regression equation and calculate a corresponding value for #math.equation(block: false, alt: "y ^")[$hat(y)$]: #math.equation(block: true, alt: "y ^, equals, 9376.7 plus 61.8 x; y ^, equals, 9376.7 plus 61.8 open parenthesis 150 close parenthesis; y ^, equals, 18646.7")[$hat(y) & = & 9376.7 + 61.8 x \ hat(y) & = & 9376.7 + 61.8 ( 150 ) \ hat(y) & = & 18646.7$] This predicted value of #math.equation(block: false, alt: "y")[$y$] indicates that the anticipated revenue would be \$18,646,700, given the advertising spend of \$150,000. Notice that from past data, there may have been a month where the company actually did spend \$150,000 on advertising, and thus the company may have an actual result for the monthly revenue. This actual, or observed, amount can be compared to the prediction from the linear regression model to calculate what is called a residual. A #strong[residual] is the difference between an observed #emph[y]-value and the predicted #emph[y]-value obtained from the linear regression equation. As an example, assume that in a previous month, the actual monthly revenue for an advertising spend of \$150,000 was \$19,200,000, and thus #math.equation(block: false, alt: "y equals 19,200")[$y = 19,200$]. The residual for this data point can be calculated as follows: #math.equation(block: true, alt: "Residual, equals, open parenthesis observed y -value close parenthesis minus open parenthesis predicted y -value close parenthesis; Residual, equals, y minus y ^; Residual, equals, 19200 minus 18646.7 equals 553.3")[$"Residual" & = & ( "observed " y "-value" ) − ( "predicted " y "-value" ) \ "Residual" & = & y − hat(y) \ "Residual" & = & 19200 − 18646.7 = 553.3$] Notice that residuals can be positive, negative, or zero. If the observed #emph[y]-value exactly matches the predicted #emph[y]-value, then the residual will be zero. If the observed #emph[y]-value is greater than the predicted #emph[y]-value, then the residual will be a positive value. If the observed #emph[y]-value is less than the predicted #emph[y]-value, then the residual will be a negative value. When formulating the best-fit linear regression line to the points on the scatterplot, the mathematical analysis generates a linear equation where the sum of the squared residuals is minimized. This analysis is referred to as the #strong[method of least squares]. The result is that the analysis generates a linear equation that is the “best fit” to the points on the scatterplot, in the sense that the line minimizes the differences between the predicted values and observed values for #math.equation(block: false, alt: "y")[$y$]; this is generally referred to as the #strong[best-fit linear equation]. #examplebox("Example 2")[][ Suppose that the chief financial officer of a corporation has created a linear model for the relationship between the company stock and interest rates. When interest rates are at 5%, the company stock has a value of \$94. Using the linear model, when interest rates are at 5%, the model predicts the value of the company stock to be \$99. Calculate the residual for this data point. #solutionbox[ A residual is the difference between an observed #emph[y]-value and the predicted #emph[y]-value obtained from the linear regression equation: #math.equation(block: true, alt: "Residual, equals, open parenthesis observed y -value close parenthesis minus open parenthesis predicted y -value close parenthesis; Residual, equals, y minus y ^; Residual, equals, 94 minus 99 equals minus 5")[$"Residual" & = & ( "observed " y "-value" ) − ( "predicted " y "-value" ) \ "Residual" & = & y − hat(y) \ "Residual" & = & 94 − 99 = − 5$] ] ] The goal in regression analysis is to determine the coefficients #math.equation(block: false, alt: "a")[$a$] and #math.equation(block: false, alt: "b")[$b$] in the following regression equation: #math.equation(block: true, alt: "y ^ equals a plus b x")[$hat(y) = a + b x$] Once the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] has been collected, the slope (#math.equation(block: false, alt: "b")[$b$]) and #emph[y]-intercept (#math.equation(block: false, alt: "a")[$a$]) can be calculated using the following formulas: #math.equation(block: true, alt: "b, equals, the fraction n Σ x y minus open parenthesis Σ x close parenthesis open parenthesis Σ y close parenthesis over n Σ x squared minus open parenthesis Σ x close parenthesis squared; a, equals, the fraction Σ y over n minus b the fraction Σ x over n")[$b & = & frac(n Σ x y − ( Σ x ) ( Σ y ), n Σ x^(2) − attach(( Σ x ), t: 2)) \ a & = & frac(Σ y, n) − b frac(Σ x, n)$] where #math.equation(block: false, alt: "n")[$n$] refers to the number of data pairs and #math.equation(block: false, alt: "Σ x")[$Σ x$] indicates sum of the #emph[x]-values. Notice that the formula for the #emph[y]-intercept requires the use of the slope result (b), and thus the slope should be calculated first, and the #emph[y]-intercept should be calculated second. When making predictions for #math.equation(block: false, alt: "y")[$y$], it is always important to plot a scatter diagram first. If the scatterplot indicates that there is a linear relationship between the variables, then it is reasonable to use a best-fit line to make predictions for #math.equation(block: false, alt: "y")[$y$], given #math.equation(block: false, alt: "x")[$x$] is within the domain of #emph[x]-values in the sample data, #emph[but not necessarily for x-values outside that domain]. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Using Technology for Linear Regression] Typically, technology is used to calculate the best-fit linear model as well as calculate correlation coefficients and scatterplot. Details of using Python for these calculations are provided in Using Python for Correlation and Linear Regression. ] ==== Assumptions for Linear Regression Testing the significance of the correlation coefficient requires that certain assumptions about the data are satisfied. The premise of this test is that the data are a sample of observed points taken from a larger population. We have not examined the entire population because it is not possible or feasible to do so. We are examining the sample to draw a conclusion about whether the linear relationship that we see between #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$] data in the sample data provides strong enough evidence that we can conclude that there actually is a linear relationship between #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$] data in the population. The regression line equation that we calculate from the sample data gives the best-fit line for our particular sample. We want to use this best-fit line for the sample as an estimate of the best-fit line for the population . Examining the scatterplot and testing the significance of the correlation coefficient helps us determine if it is appropriate to do this. These are the assumptions underlying the test of significance: + There is a linear relationship in the population that models the average value of #math.equation(block: false, alt: "y")[$y$] for varying values of #math.equation(block: false, alt: "x")[$x$]. In other words, the expected value of #math.equation(block: false, alt: "y")[$y$] for each particular value lies on a straight line in the population. (We do not know the equation for the line for the population. Our regression line from the sample is our best estimate of this line in the population.) + The #emph[y]-values for any particular #emph[x]-value are normally distributed about the line. This implies that there are more #emph[y]-values scattered closer to the line than are scattered farther away. Assumption (1) implies that these normal distributions are centered on the line: the means of these normal distributions of #emph[y]-values lie on the line. + The standard deviations of the population #emph[y]-values about the line are equal for each value of #math.equation(block: false, alt: "x")[$x$]. In other words, each of these normal distributions of #emph[y]-values has the same shape and spread about the line. + The residual errors are mutually independent (no pattern). + The data are produced from a well-designed random sample or randomized experiment. #figure(figph[Two diagrams of a best fit line. The first diagram labeled a shows a linearly descending line running through the center of three vertical sets of scattered points. The second diagram labeled b shows a linearly descending line running through the mean of three tilted bell curves. The bottom of each bell curve aligns with the position of the three vertical scattered points in diagram a.], alt: "Two diagrams of a best fit line. The first diagram labeled a shows a linearly descending line running through the center of three vertical sets of scattered points. The second diagram labeled b shows a linearly descending line running through the mean of three tilted bell curves. The bottom of each bell curve aligns with the position of the three vertical scattered points in diagram a.", caption: [Best-Fit Line. Note that the #emph[y]-values for each #emph[x]-value are normally distributed about the line with the same standard deviation. For each #emph[x]-value, the mean of the #emph[y]-values lies on the regression line. More #emph[y]-values lie near the line than are scattered farther away from the line]) === Using Python for Correlation and Linear Regression Once a correlation has been deemed significant, a linear regression model is developed. The goal in the regression analysis is to determine the coefficients #math.equation(block: false, alt: "a")[$a$] and #math.equation(block: false, alt: "b")[$b$] in the following regression equation: #math.equation(block: true, alt: "y ^ equals a plus b x")[$hat(y) = a + b x$] These formulas can be quite cumbersome, especially for a significant number of data pairs, and thus software is often used (such as Excel, Python, or R). ==== Develop a Linear Regression Model In the following example, Python will be used for the following analysis based on a given dataset of #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data. - Create a scatterplot. - Calculate the correlation coefficient. - Construct the best-fit linear equation. - Predict values of the dependent variable for given values of the independent variable. There are various ways to accomplish these tasks in Python, but we will use several functions available within the scipy library, such as: #strong[Pearsonr()] for correlation coefficient #strong[linregress()] for linear regression model #strong[plt.scatter()] for scatterplot generation #examplebox("Example 3")[][ A marketing manager is interested in studying the correlation between the amount spent on advertising and revenue at a certain company. Twelve months of data were collected as shown. (Dollar amounts are in thousands of dollars.) Use Python to perform the following analysis: + Generate a scatterplot. + Calculate the correlation coefficient #math.equation(block: false, alt: "r")[$r$]. + Determine if the correlation is significant (use level of significance of 0.05). + Construct a linear regression model. + Use the model to predict the revenue for a certain month where the amount spent on advertising is \$100. #figure(table( columns: 3, align: left, inset: 6pt, table.header([#strong[Month]], [#strong[Advertising Spend]], [#strong[Revenue]]), [Jan], [49], [12210], [Feb], [145], [17590], [Mar], [57], [13215], [Apr], [153], [19200], [May], [92], [14600], [Jun], [83], [14100], [Jul], [117], [17100], [Aug], [142], [18400], [Sep], [69], [14100], [Oct], [106], [15500], [Nov], [109], [16300], [Dec], [121], [17020], )) #solutionbox[ + The first step in the analysis is to generate a scatterplot. (Recall that scatterplots were also discussed in Scatterplots and Correlation.) #linebreak() The Python function plt.scatter() can be used to generate the scatterplot for the #math.equation(block: false, alt: "open parenthesis x , y close parenthesis")[$( x , y )$] data. Note that we consider advertising to be the independent variable (#math.equation(block: false, alt: "x")[$x$]-data) and revenue to be the dependent variable (#math.equation(block: false, alt: "y")[$y$]-data) since revenue depends on the amount of advertising. Here is the Python code to generate the scatterplot: #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import the matplotlib library import matplotlib.pyplot as plt \# define the x-data, which is amount spent on advertising x = \[49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121\] \# define the y-data, which is revenue y = \[12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020\] \# use the scatter function to generate a time series graph plt.scatter(x, y) \# Add a title using the title function plt.title("Revenue versus Advertising for a Company") \# Add labels to the x and y axes by using xlabel and ylabel functions plt.xlabel("Advertising \$000") plt.ylabel ("Revenue \$000") The resulting output will look like this: #figure(figph[A scatter plot of revenue against ad spend. The diagram shows revenue increasing from approximately 12 million dollars to 19 million dollars as advertising spend increases from approximately 50,000 dollars to 150,000 dollars.], alt: "A scatter plot of revenue against ad spend. The diagram shows revenue increasing from approximately 12 million dollars to 19 million dollars as advertising spend increases from approximately 50,000 dollars to 150,000 dollars.", caption: none) ] + The Python function #strong[pearsonr()] can be used to generate the correlation coefficient, #math.equation(block: false, alt: "r")[$r$]. Here is the Python code to calculate the correlation coefficient: #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import libraries import numpy as np from scipy.stats  import pearsonr \# define the x-data, which is amount spent on advertising x = \[49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121\] \# define the y-data, which is revenue y = \[12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020\] \#pearson r returns both value of r and corresponding p-value r, p = pearsonr(x, y) \#print value of r, rounded to 3 decimal places print("Correlation coefficient: ", round(r, 3)) The resulting output will look like this: Correlation coefficient: 0.981 #linebreak() This value of #math.equation(block: false, alt: "r equals 0.981")[$r = 0.981$] indicates a strong, positive correlation between advertising spend and revenue. ] + When using a level of significance of 0.05, if #math.equation(block: false, alt: "| r | greater than or equal to the fraction 2 over the square root of n")[$| r | ≥ frac(2, sqrt(n))$], then this implies that the correlation between the two variables demonstrates a linear relationship that is statistically significant at approximately the 0.05 level of significance. In this example, we check if #math.equation(block: false, alt: "| 0.981 |")[$| 0.981 |$] is greater than or equal to #math.equation(block: false, alt: "the fraction 2 over the square root of n")[$frac(2, sqrt(n))$] where #math.equation(block: false, alt: "n equals 12")[$n = 12$]. Since #math.equation(block: false, alt: "the fraction 2 over the square root of 12")[$frac(2, sqrt(12))$] is approximately 0.577, this indicates that the #emph[r]-value of 0.981 is greater than #math.equation(block: false, alt: "the fraction 2 over the square root of n")[$frac(2, sqrt(n))$], and thus the correlation is deemed significant. + The linear regression model will be of the form: #linebreak() #math.equation(block: true, alt: "y ^ equals a plus b x")[$hat(y) = a + b x$] To determine the slope and #emph[y]-intercept of the regression model, the Python function #strong[linregress()] can be used. #linebreak() This function takes the #math.equation(block: false, alt: "x")[$x$] and #math.equation(block: false, alt: "y")[$y$] data as inputs and provides the slope and intercept as well as other regression-related output. Here is the Python code to generate the slope and #emph[y]-intercept of the best-fit line: #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import libraries import numpy as np from scipy.stats  import linregress \# define the x-data, which is amount spent on advertising x = \[49, 145, 57, 153, 92, 83, 117, 142, 69, 106, 109, 121\] \# define the y-data, which is revenue y = \[12210, 17590, 13215, 19200, 14600, 14100, 17100, 18400, 14100, 15500, 16300, 17020\] \#pearson r returns both value of r and corresponding p-value slope, intercept, r\_value, p\_value, std\_err  = linregress(x, y) print ("slope  =", slope) print("intercept = ", intercept) The resulting output will look like this: slope = 61.79775818816664 #linebreak() intercept = 9376.698881009072 ] Based on this Python equation, the regression equation can be written as #math.equation(block: true, alt: "y ^, equals, a plus b x; y ^, equals, 9376.7 plus 61.8 x")[$hat(y) & = & a + b x \ hat(y) & = & 9376.7 + 61.8 x$] where #math.equation(block: false, alt: "x")[$x$] represents the amount spent on advertising (in thousands of dollars) and #math.equation(block: false, alt: "y")[$y$] represents the amount of revenue (in thousands of dollars). It is also useful to plot this best-fit line superimposed on the scatterplot to show the strong correlation between advertising spend and revenue, as follows: #figure(figph[Scatterplot showing a positive linear relationship between advertising spending and revenue. Advertising spending ranges from \$40,000 to \$160,000, and revenue ranges from \$12,000 to \$20,000. A regression line is fitted to the data, suggesting a strong correlation between the two variables.], alt: "Scatterplot showing a positive linear relationship between advertising spending and revenue. Advertising spending ranges from $40,000 to $160,000, and revenue ranges from $12,000 to $20,000. A regression line is fitted to the data, suggesting a strong correlation between the two variables.", caption: [Best-Fit Line Superimposed on the Scatterplot]) + The regression equation is: #math.equation(block: false, alt: "y ^ equals 9,376.7 plus 61.8 x")[$hat(y) = 9,376.7 + 61.8 x$]. #linebreak() The given amount of advertising spend of \$100 corresponds to a given #emph[x]-value, so replace “#math.equation(block: false, alt: "x")[$x$]” with 100 in this regression equation to generate a predicted revenue: #linebreak() #math.equation(block: true, alt: "y ^, equals, 9376.7 plus 61.8 x; y ^, equals, 9376.7 plus 61.8 open parenthesis 100 close parenthesis; y ^, equals, 15556.7")[$hat(y) & = & 9376.7 + 61.8 x \ hat(y) & = & 9376.7 + 61.8 ( 100 ) \ hat(y) & = & 15556.7$] Since both advertising and revenue are expressed in thousands of dollars, the conclusion is that if advertising spend is \$100,000, the predicted revenue for the company will be \$15,556,700. ] ] ==== Interpret and Apply the Slope and #emph[y]-Intercept The slope of the line, #math.equation(block: false, alt: "b")[$b$], describes how changes in the variables are related. It is important to interpret the slope of the line in the context of the situation represented by the data. You should be able to write a sentence interpreting the slope in plain English. Interpretation of the Slope The slope of the best-fit line tells us how the dependent variable (#math.equation(block: false, alt: "y")[$y$]) changes for every one-unit increase in the independent (#math.equation(block: false, alt: "x")[$x$]) variable, on average. In the previous example, the linear regression model for the monthly amount spent on advertising and the monthly revenue for a company for 12 months was generated as follows: #math.equation(block: true, alt: "y ^, equals, a plus b x; y ^, equals, 9376.7 plus 61.8 x")[$hat(y) & = & a + b x \ hat(y) & = & 9376.7 + 61.8 x$] Since the slope was determined to be 61.8, the company can interpret this to mean that for every \$1,000 spent on advertising, on average, this will result in an increase in revenues of \$61,800. The intercept of the regression equation is the corresponding #emph[y]-value when #math.equation(block: false, alt: "x equals 0")[$x = 0$]. Interpretation of the Intercept The intercept of the best-fit line tells us the expected mean value of #math.equation(block: false, alt: "y")[$y$] in the case where the #math.equation(block: false, alt: "x")[$x$]-variable is equal to zero. However, in many scenarios it may not make sense to have the #math.equation(block: false, alt: "x")[$x$]-variable equal zero, and in these cases, the intercept does not have any meaning in the context of the problem. In other examples, the #emph[x]-value of zero is outside the range of the #math.equation(block: false, alt: "x")[$x$]-data that was collected. In this case, we should not assign any interpretation to the #emph[y]-intercept. In the previous example, the range of data collected for the #math.equation(block: false, alt: "x")[$x$]-variable was from \$49,000 to \$153,000 spent per month on advertising. Since this interval does not include an #emph[x]-value of zero, we would not provide any interpretation for the intercept.