#set document(title: "6.1 What Is Machine Learning?", author: "OpenStax / XYZ Homework") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 6.1#h(0.6em)What Is Machine Learning? === Learning Outcomes By the end of this section, you should be able to: - Summarize the differences between supervised and unsupervised learning in machine learning. - Describe the roles of training and testing datasets. - Use common measures of accuracy and error to determine the fitness of a model. - Explain the concepts of overfitting and underfitting and how these problems can adversely affect the model. A #strong[machine learning (ML) model] is a mathematical and computational model that attempts to find a relationship between input variables and output (response) variables of a dataset. The way that the model “learns” is by adjusting internal parameters until the model meets a certain level of #strong[accuracy] (defined as a measure of how correct the model is when used to predict. We will have more to say on accuracy later). What sets apart a machine learning model from a typical mathematical model is that the user does not have full control over all the parameters used by the model. Moreover, there may be hundreds, thousands, or even millions of parameters that are used and adjusted within a given ML model—way more than a human could reasonably track. The part of that dataset that is used for initial learning is the #strong[training set (or data)]. There is often a #strong[testing set (or data)] as well, which (as the name implies) can be used to determine whether the model is accurate enough. As a general rule of thumb, about 60–80% of available data will be used for training and the remaining data used for testing. After a machine learning model is trained, it can be used to make predictions as new input data is fed in and the responses produced. There may be multiple rounds of training and testing before the ML model is ready to make predictions or decisions. The #strong[machine learning life cycle] typically follows these steps: + #strong[Problem formulation/identification:] Clearly state the problem you want the model to solve. + #strong[Data collection and preparation:] Collect relevant data and clean it for analysis. + #strong[Feature selection/engineering:] Choose important aspects of the data for the model. + #strong[Model/algorithm selection:] Select a suitable machine learning algorithm for the task. + #strong[Model/algorithm training:] Feed in the training data to teach the model patterns in the data. + #strong[Model validation:] Test how well the model predicts outcomes. + #strong[Model implementation:] Put the model to work in real-world scenarios. + #strong[Performance monitoring and enhancement:] Keep an eye on the model’s accuracy and update as needed. + #strong[Continuous improvement and refinement:] Repeat and refine steps based on feedback and changes. Throughout the process, be sure to keep ethical considerations in mind to ensure fairness, privacy, and transparency. For example, were the data collected with the proper permissions? Does the training data accurately reflect the same diversity of data that the algorithm will ultimately be used on? Currently there are serious questions as to the ethical implications of using facial recognition algorithms in law enforcement, as training images may have been gathered from people without their explicit consent, and the models created often reflect biases that can result in discriminatory outcomes, disproportionately affecting certain demographic groups, such as people of color or women. #strong[Bias]—error introduced by overall simplistic or overly rigid models that do not capture important features of the data—can be hard to identify! One example that is famous in data science circles (though it may be apocryphal) is the “#strong[Russian tank problem]” (Emspak, 2016). As the story goes, the United States military trained an early ML model to distinguish between Russian and US tanks. The training set consisted of photos of many different kinds of tanks used by both countries. Although the ML model produced very accurate results on the training and test sets, it performed terribly out in the field. Apparently, most of the pictures of Russian tanks were low quality and blurry, while those of the US tanks were high quality. They had inadvertently trained their ML model to distinguish between low- and high-quality photos regardless which county’s tanks were depicted in them! === Supervised vs. Unsupervised Learning Data may be labeled or unlabeled. #strong[Labeled data] has specific labels, categories, or classes associated with each data point. For example, a dataset containing the height, weight, cholesterol levels, and various other health-related measures for patients at a clinic, along with an indication as to whether each is susceptible to heart disease or not, would be labeled data. The label is the value (yes or no) of the variable “susceptible to heart disease.” #strong[Unlabeled data] does not have a specific label, category, or class associated with its data points. For example, in product marketing, we might collect data representing customer attributes such as age, income, and purchase history, and we may want to identify distinct groups of customers with similar characteristics without knowing ahead of time what those groups (labels) will be. #strong[Supervised learning] uses labeled data in its training and testing sets. Parameters are adjusted in the model based on the correctness of the results as compared with the known labels. #strong[Unsupervised learning] trains on unlabeled data. Often, both methods will be used in the same project. Unsupervised learning might be employed as an initial step to find patterns in the dataset, allowing the data to be labeled in some way. Then supervised learning techniques may be applied to the labeled data as a training step. The model created in this way may then be used to determine the best labels for future (unlabeled) data. ==== Supervised Learning Supervised learning is analogous to a student learning from an instructor. At first, the student may answer many questions incorrectly, but the instructor is there to correct the student each time. When the student gives (mostly) correct responses, there is evidence that the student has sufficiently learned the material and would be able to answer similar questions correctly out in the real world where it really matters. The goal of supervised learning is to produce a model that gives a mapping from the set of inputs or features to a set of output values or labels. The model is function #math.equation(block: false, alt: "y equals f open parenthesis X close parenthesis")[$y = f ( X )$], where #math.equation(block: false, alt: "X")[$X$] is the input array and #math.equation(block: false, alt: "y")[$y$] is the output value or label. The way that it does this is through training and testing. When training, the algorithm creates a correspondence between input features and the output value or label, often going through many iterations of trial and error until a desired level of accuracy is achieved. The model is then tested on additional data, and if the accuracy remains high enough, then the model may be deployed for use in real-world applications. Otherwise, the model may need to be adjusted substantially or abandoned altogether. #figure(figph[A flowchart illustrating the machine learning model development process. The flowchart has four steps: Step 1: Gather Data: This is the starting point, where data is collected for training the model. Step 2: Train: The collected data is used to train the model. Step 3: Test: The trained model is evaluated on a separate dataset to assess its accuracy. Step 4: Deploy Model: If the model's accuracy is satisfactory, it is deployed for use. The flowchart also includes a feedback loop, indicating that the model may need to be retrained and retested if its performance on new data is not satisfactory.], alt: "A flowchart illustrating the machine learning model development process. The flowchart has four steps: Step 1: Gather Data: This is the starting point, where data is collected for training the model. Step 2: Train: The collected data is used to train the model. Step 3: Test: The trained model is evaluated on a separate dataset to assess its accuracy. Step 4: Deploy Model: If the model's accuracy is satisfactory, it is deployed for use. The flowchart also includes a feedback loop, indicating that the model may need to be retrained and retested if its performance on new data is not satisfactory.", caption: [The Supervised Learning Cycle. The #strong[supervised learning cycle] consists of gathering data, training on some part of the data, testing on another part of the data, and deployment to solve problems about new data.]) Examples of supervised learning models include linear regression (discussed in Correlation and Linear Regression Analysis), logistic regression, naïve Bayes classification, decision trees, random forests, and many kinds of neural networks. Linear regression may not seem like a machine learning algorithm because there are no #emph[correction] steps. A formula simply produces the line of best fit. However, the regression formula itself represents an optimization that reduces the error in the values predicted by the regression line versus the actual data. If the linear regression model is not accurate enough, additional data could be collected to improve the accuracy of the model, and once the regression line is found, it may then be used to predict values that were not in the initial dataset. Linear regression has been developed in previous chapters, and you will see logistic regression, Bayes methods, decision trees, and random forests later in this chapter. ==== Unsupervised Learning Unsupervised learning is like building a puzzle without having the benefit of a picture to compare with. It is not impossible to build a puzzle without the picture, but it may be more difficult and time-consuming. The main advantage of unsupervised learning is that the training data does not need to be labeled. All the data is considered input, and the output would typically be information about the inherent structure of the data. Common tasks are to determine clusters, patterns, and shapes within the dataset. For example, if the current locations in latitude and longitude of every person in the world were somehow known and collected into a dataset, then an algorithm may discover locations of high densities of people—hence #emph[learn] of the existence of cities! Unsupervised learning algorithms include k-means clustering, DBScan, and some kinds of neural networks. More generally, most clustering algorithms and dimension-reduction methods such as #strong[principal component analysis (PCA)] and #strong[topological data analysis (TDA)] fall into the category of unsupervised learning. You will encounter the k-means and DBScan algorithms in this chapter. The more advanced topics of PCA and TDA are outside the scope of this text. ==== Variations and Hybrid Models Although we will only cover supervised and unsupervised models, it is important to note that there are some additional models of machine learning. When some of the data in a dataset has labels and some does not, a #emph[semi-supervised learning algorithm] may be appropriate. The labeled data can be run through a supervised learning algorithm to generate a predictive model that can then be used to label the unlabeled data, called #emph[pseudo-data]. From this point, the dataset can be regarded as completely labeled data, which can then be analyzed using supervised learning. #emph[Reinforcement learning] introduces rewards for accuracy while also penalizing incorrectness. This model most closely follows the analogy of a student-teacher relationship. Many algorithms in artificial intelligence use reinforcement learning. While reinforcement generally improves accuracy, there can be issues in locking the model into a rigid state. For example, have you ever noticed that when you search online for a type of product, you begin to see advertisements for those kinds of products everywhere—even after you’ve purchased it? It’s as if the algorithm has decided that the only product you ever wish to purchase is what you have just searched for! === Training and Testing a Model How do we know how good a model is? The key is to compare its output against known output. In supervised learning models, there is always a metric that measures efficacy of the model. During the training phase, this metric helps to determine how to adjust the model to make it more accurate. During the testing phase, the metric will be used to evaluate the model and determine if it is good enough to use on other datasets to make predictions. In this section, we will explore the supervised learning cycle in more detail. ==== Model Building Suppose we are given a dataset with inputs or features #math.equation(block: false, alt: "X")[$X$] and labels or values #math.equation(block: false, alt: "y")[$y$]. Our goal is to produce a function that maps #math.equation(block: false, alt: "y equals f open parenthesis X close parenthesis")[$y = f ( X )$]. In practice, we do not expect that our model can capture the function #math.equation(block: false, alt: "f")[$f$] with 100% accuracy. Instead, we should find a model #math.equation(block: false, alt: "f ^")[$hat(f)$] such that predicted values #math.equation(block: false, alt: "y ^ equals f ^ open parenthesis X close parenthesis")[$hat(y) = hat(f) ( X )$] are #emph[close enough to] #math.equation(block: false, alt: "y")[$y$]. In most cases, the metric we use to determine the fitness of our model #math.equation(block: false, alt: "f ^")[$hat(f)$] is some measure of error between #math.equation(block: false, alt: "y ^")[$hat(y)$] and #math.equation(block: false, alt: "y")[$y$] (e.g., see the calculation of residuals of a linear regression in Correlation and Linear Regression Analysis). The way that error is measured is specific to the type of model that we are building and the nature of the outputs. When the outputs are labels, we may use a count or percentage of misclassifications. When the outputs are numerical, we may find the total distances (absolute, squared, etc.) between predicted and actual output values. First, the data are separated into a training set and testing set. Usually about 60–80% of the data is used for training with the remaining amount used for testing. While training, parameters may be adjusted to reduce error. The result of the training phase will be the model #math.equation(block: false, alt: "f ^")[$hat(f)$]. Then, the model is tested on the testing set and error computed between predicted values and known outputs. If the model meets some predefined threshold of accuracy (low enough error), then the model can be used with some confidence to make predictions about new data. ==== Measures of Accuracy As mentioned previously, it is important to measure how good our machine learning models are. The choice of a specific measure of accuracy depends on the nature of the problem. Here are some common accuracy measures for different machine learning tasks: + Classification - Accuracy is the ratio of correct predictions to the total number of predictions. - #strong[Precision] #math.equation(block: false, alt: "open parenthesis p close parenthesis")[$( p )$] is the ratio of true positive predictions #math.equation(block: false, alt: "open parenthesis T P close parenthesis")[$( T P )$] to the total number of positive (true positive plus false positive) predictions #math.equation(block: false, alt: "open parenthesis T P plus F P close parenthesis")[$( T P + F P )$]. This is useful when the response variable is true/false and we want to minimize false positives. #math.equation(block: false, alt: "p equals the fraction T P over T P plus F P")[$p = frac(T P, T P + F P)$] - #strong[Recall] #math.equation(block: false, alt: "open parenthesis r close parenthesis")[$( r )$] is the ratio of true positive predictions #math.equation(block: false, alt: "open parenthesis T P close parenthesis")[$( T P )$] to the total number of actual positives (true positive plus false negative #math.equation(block: false, alt: "open parenthesis T P plus F N close parenthesis")[$( T P + F N )$] predictions. This is useful when the response variable is true/false and we want to minimize false negatives. #math.equation(block: false, alt: "r equals the fraction T P over T P plus F N")[$r = frac(T P, T P + F N)$] - #strong[F1 Score] is a combination of precision #math.equation(block: false, alt: "open parenthesis p close parenthesis")[$( p )$] and recall #math.equation(block: false, alt: "open parenthesis r close parenthesis")[$( r )$]. #math.equation(block: false, alt: "F 1 equals the fraction 2 open parenthesis p close parenthesis open parenthesis r close parenthesis over p plus r equals the fraction 2 T P over 2 T P plus F P plus F N")[$F 1 = frac(2 ( p ) ( r ), p + r) = frac(2 T P, 2 T P + F P + F N)$] + Regression (comparison of the predicted values, #math.equation(block: false, alt: "y ^ sub i")[$hat(y)_(i)$], and the actual values, #math.equation(block: false, alt: "y sub i")[$y_(i)$]) - #strong[Mean absolute error (MAE)]: #math.equation(block: false, alt: "the fraction 1 over n ∑ i equals 1 n | y sub i minus y ^ sub i |")[$frac(1, n) limits(∑)^(n)_(i = 1) | y_(i) − hat(y)_(i) |$] - #strong[Mean absolute percentage error (MAPE)]: #math.equation(block: false, alt: "the fraction 1 over n ∑ i equals 1 n | the fraction y sub i minus y ^ sub i over y sub i |")[$frac(1, n) limits(∑)^(n)_(i = 1) | frac(y_(i) − hat(y)_(i), y_(i)) |$] - #strong[Mean squared error (MSE)]: #math.equation(block: false, alt: "the fraction 1 over n ∑ i equals 1 n open parenthesis y sub i minus y ^ sub i close parenthesis squared")[$frac(1, n) limits(∑)^(n)_(i = 1) ( y_(i) − hat(y)_(i) )^(2)$] - #strong[Root mean squared error (RMSE)]: #math.equation(block: false, alt: "the square root of the fraction 1 over n ∑ i equals 1 n open parenthesis y sub i minus y ^ sub i close parenthesis squared")[$sqrt(frac(1, n) limits(∑)^(n)_(i = 1) ( y_(i) − hat(y)_(i) )^(2))$] - #emph[R-squared] is the proportion of the variance in the response variable that is predictable from the input variable(s). Note that MAE and RMSE, also discussed in Forecast Evaluation Methods, have the same units as the data itself. MSE and RMSE can be very sensitive to outliers because the differences are squared. MAPE (also introduced in Forecast Evaluation Methods) is often represented in percent form, which you can easily find by multiplying the raw MAPE score by 100%. The R-squared measure, which is discussed in Inferential Statistics and Regression Analysis in the context of linear regression, can be computed by statistical software packages, though we will not make significant use of it in this chapter. In all cases, the smaller the value of the error term, the more accurate the modeling would be. Due to the tedious calculations involved in these measures, statistical software packages are typically used to compute them in practice. But it is important for every data scientist to understand how the formulas work so that results can be evaluated and interpreted correctly. Next, we will work out a small example by hand so you can have experience with the formulas. #examplebox("Example 1")[][ A test for COVID-19 was applied to 1,000 patients. Two hundred thirty-five tested positive for COVID-19, while the remaining 765 tested negative. Of those that tested positive, 198 turned out to carry the COVID-19 virus, and 37 did not have the virus. Of those that tested negative, 63 patients did in fact have the virus, while the remaining 702 did not. Compute the accuracy, precision, recall, and F1 scores for this experiment. #solutionbox[ First, the total number of correct predictions were: 198 true positive predictions + 702 true negative predictions, for a total of #math.equation(block: false, alt: "T P plus T N equals 900")[$T P + T N = 900$]. So the accuracy is #math.equation(block: false, alt: "the fraction 900 over 1,000 equals 0.9 open parenthesis 90 % close parenthesis")[$frac(900, 1","000) = 0.9 ( 90 % )$]. For precision, we use #math.equation(block: false, alt: "T P equals 198")[$T P = 198$] and #math.equation(block: false, alt: "F P")[$F P$] (false positives) #math.equation(block: false, alt: "equals 37")[$= 37$], and we have: #math.equation(block: false, alt: "p equals the fraction T P over T P plus F P equals the fraction 198 over 198 plus 37 equals 0.843")[$p = frac(T P, T P + F P) = frac(198, 198 + 37) = 0.843$] (84.3%). For recall, we use #math.equation(block: false, alt: "T P equals 198")[$T P = 198$] and #math.equation(block: false, alt: "F N")[$F N$] (false negatives) #math.equation(block: false, alt: "equals 63")[$= 63$], and we have: #math.equation(block: false, alt: "r equals the fraction T P over T P plus F N equals the fraction 198 over 198 plus 63 equals 0.759")[$r = frac(T P, T P + F N) = frac(198, 198 + 63) = 0.759$] (75.9%). Finally, #math.equation(block: false, alt: "F 1 equals the fraction 2 open parenthesis p close parenthesis open parenthesis r close parenthesis over p plus r equals the fraction 2 open parenthesis 0.843 close parenthesis open parenthesis 0.759 close parenthesis over 0.843 plus 0.759 equals 0.799")[$F 1 = frac(2 ( p ) ( r ), p + r) = frac(2 ( 0.843 ) ( 0.759 ), 0.843 + 0.759) = 0.799$] (79.9%). ] ] #examplebox("Example 2")[][ The score on an exam is positively correlated with the time spent studying for the exam. You have collected the following data (pairs of study time in hours and exam scores): (9.8, 83.0), (8.2, 87.7), (5.2, 61.6), (8.0, 77.8), (2.1, 42.2), (6.8, 62.1), (2.3, 30.9), (9.5, 94.4), (6.6, 76.2), (9.5, 93.1) Use 80% of the dataset to create a linear model and discuss the accuracy of the model on the remaining testing set. #solutionbox[ Since there are 10 data points, we use 8 of them for training, as 8 is 80% of 10. Training data: (9.8, 83.0), (8.2, 87.7), (5.2, 61.6), (8.0, 77.8), (2.1, 42.2), (6.8, 62.1), (2.3, 30.9), (9.5, 94.4) The linear regression model that best fits the training data is #math.equation(block: false, alt: "y equals 21.7 plus 7.1 x")[$y = 21.7 + 7.1 x$]. (See Correlation and Linear Regression Analysis for details on linear regression.) We will compute MAE, MAPE, MSE, and RMSE for the remaining two data points (test set) as in . #figure(table( columns: 4, align: left, inset: 6pt, table.header([#strong[#emph[x]]], [#strong[Prediction]], [#strong[Actual Value]], [#strong[Difference (Error)]]), [2.3], [#math.equation(block: false, alt: "21.7 plus 7.1 open parenthesis 2.3 close parenthesis equals 38.03")[$21.7 + 7.1 ( 2.3 ) = 38.03$]], [30.9], [-7.13], [9.5], [#math.equation(block: false, alt: "21.7 plus 7.1 open parenthesis 9.5 close parenthesis equals 89.15")[$21.7 + 7.1 ( 9.5 ) = 89.15$]], [94.4], [5.25], )) - #math.equation(block: false, alt: "MAE equals the fraction 1 over 2 open parenthesis | minus 7.13 | plus | 5.25 | close parenthesis equals 6.19")[$"MAE" #h(0.2em) = frac(1, 2) ( | − 7.13 | + | 5.25 | ) = 6.19$] - #math.equation(block: false, alt: "MAPE equals the fraction 1 over 2 open parenthesis | the fraction minus 7.13 over 30.9 | plus | the fraction 5.25 over 94.4 | close parenthesis equals 0.143")[$"MAPE" #h(0.2em) = frac(1, 2) ( | frac(− 7.13, 30.9) | + | frac(5.25, 94.4) | ) = 0.143$], or 14.3% - MSE for cubic model: #math.equation(block: false, alt: "the fraction 1 over 2 open parenthesis open parenthesis minus 7.13 close parenthesis squared plus 5.25 squared close parenthesis equals 39.2")[$frac(1, 2) ( attach(( − 7.13 ), t: 2) + 5.25^(2) ) = 39.2$] - RMSE for cubic model: #math.equation(block: false, alt: "the square root of 39.2 equals 6.26")[$sqrt(39.2) = 6.26$] The MAE and RMSE both show that the model’s predictions are off by an average of about 6.2 points from actual values. The MAPE suggests that the model has an error of about 14.3% on average. ] ] === Overfitting and Underfitting Can 100% accuracy be attained from a machine learning algorithm? In all but the most trivial exercises, the answer is a resounding #strong[#emph[no]]. Real-world data is noisy, often incomplete, and rarely amenable to a simple model. (Recall from Collecting and Preparing Data that we defined noisy data as data that contains errors, inconsistencies, or random fluctuations that can negatively impact the accuracy and reliability of data analysis and interpretation.) Two of the most common issues with machine learning algorithms are overfitting and underfitting. #strong[Overfitting] happens when the model is adjusted to fit the training data too closely, resulting in a complex model that is too specific to the training data. When such a model is given data outside of the training set, it may perform worse, a phenomenon known as high #strong[variance]. #strong[Underfitting] occurs when the model is not robust enough to capture essential features of the data. This could be because of too small a dataset to train on, choosing a machine learning algorithm that is too rigid, or a combination of both. A model that suffers from underfitting may be said to have #emph[high bias]. Often, a model that has low variance will have high bias, and a model that has low bias will have high variance. This is known as the #emph[bias-variance trade-off]. ==== Overfitting It might at first seem desirable to produce a model that makes no mistakes on the training data; however, such “perfect” models are usually terrible at making predictions about data not in the training set. It is best to explain this idea by example. Suppose we want to create a model for the relationship between #emph[x] and #emph[y] based on the given dataset, #math.equation(block: true, alt: "S equals open parenthesis x , y close parenthesis equals open parenthesis 2 , 10 close parenthesis , open parenthesis 4 , 9 close parenthesis , open parenthesis 6 , 6 close parenthesis , open parenthesis 8 , 7 close parenthesis , open parenthesis 10 , 4 close parenthesis , open parenthesis 12 , 5 close parenthesis")[$S = ( x , y ) = ( 2 , 10 ) , ( 4 , 9 ) , ( 6 , 6 ) , ( 8 , 7 ) , ( 10 , 4 ) , ( 12 , 5 )$] How would we go about creating the model? We might first split S into a training set, #math.equation(block: false, alt: "S sub train equals { open parenthesis 2 , 10 close parenthesis , open parenthesis 4 , 9 close parenthesis , open parenthesis 6 , 6 close parenthesis , open parenthesis 8 , 7 close parenthesis }")[$S_("train") = \{ ( 2 , 10 ) , ( 4 , 9 ) , ( 6 , 6 ) , ( 8 , 7 ) \}$], and testing set, #math.equation(block: false, alt: "S sub test equals { open parenthesis 10 , 4 close parenthesis , open parenthesis 12 , 5 close parenthesis }")[$S_("test") = \{ ( 10 , 4 ) , ( 12 , 5 ) \}$]. It can be shown that the cubic equation, #math.equation(block: false, alt: "y ^ equals the fraction 1 over 8 x cubed minus the fraction 7 over 4 x squared plus the fraction 13 over 2 x plus 3")[$hat(y) = frac(1, 8) x^(3) − frac(7, 4) x^(2) + frac(13, 2) x + 3$], fits the training data perfectly. However, there is no reason to expect that the relationship between #emph[x] and #emph[y] is truly cubic. Indeed, the predictions of the cubic model on #math.equation(block: false, alt: "S sub test")[$S_("test")$] are terrible! #figure(table( columns: 3, align: left, inset: 6pt, table.header([#strong[#emph[x]]], [#strong[Prediction, Cubic Model]], [#strong[Actual Value, #emph[y]]]), [10], [18], [4], [12], [45], [5], )) The linear model (linear regression) that represents a best fit for the first four data points is #math.equation(block: false, alt: "y ^ equals minus 0. 6 x plus 11")[$hat(y) = − 0. 6 x + 11$]. shows the data points along with the two models. #figure(figph[A line graph of cubic and linear models for a set of points. The Y axis ranges from 0 to 20. The X axis ranges from 0 to 14. A blue linear line declines from left to right from about 11 to 2.5. An orange line forming organic cubic curve, intersects blue linear line at four data points (2,10), (4, 9), (6, 6), (8, 7) .Two additional data points (10, 4) and (12, 5) exists on the linear line.], alt: "A line graph of cubic and linear models for a set of points. The Y axis ranges from 0 to 20. The X axis ranges from 0 to 14. A blue linear line declines from left to right from about 11 to 2.5. An orange line forming organic cubic curve, intersects blue linear line at four data points (2,10), (4, 9), (6, 6), (8, 7) .Two additional data points (10, 4) and (12, 5) exists on the linear line.", caption: [Data Points along a Cubic Model and a Linear Model. The four data points in #math.equation(block: false, alt: "S sub train")[$S_("train")$] can be fitted onto a cubic curve (0% error), but it does poorly on additional data from the testing data, #math.equation(block: false, alt: "S sub test")[$S_("test")$]. The cubic curve overfits the training data. The regression line does not fit the training data exactly; however, it does a better job predicting additional data.]) To get a good sense of how much better the linear model is compared to the cubic model in this example, we will compute the MAPE and RMSE for each model, as shown. #figure(table( columns: 6, align: left, inset: 6pt, table.header([#math.equation(block: false, alt: "x sub i")[$x_(i)$]], [#strong[Actual Value,] #math.equation(block: false, alt: "y sub i")[$y_(i)$]], [#strong[Prediction,] #math.equation(block: false, alt: "y sub i")[$y_(i)$]#strong[, Cubic Model]], [#math.equation(block: false, alt: "| y sub i minus y ^ sub i |")[$| y_(i) − hat(y)_(i) |$]#strong[, Cubic Model]], [#strong[Prediction,] #math.equation(block: false, alt: "y ^ sub i")[$hat(y)_(i)$]#strong[, Linear Model]], [#math.equation(block: false, alt: "| y sub i minus y ^ sub i |")[$| y_(i) − hat(y)_(i) |$]#strong[, Linear Model]]), [2], [#strong[10]], [10], [0], [9.8], [0.2], [4], [#strong[9]], [9], [0], [8.6], [0.4], [6], [#strong[6]], [6], [0], [7.4], [1.4], [8], [#strong[7]], [7], [0], [6.2], [0.8], [10], [#strong[4]], [18], [14], [5], [1], [12], [#strong[5]], [45], [40], [3.8], [1.2], )) - MAPE for cubic model: #math.equation(block: false, alt: "the fraction 1 over 6 open parenthesis 0 / 10 plus 0 / 9 plus 0 / 6 plus 0 / 7 plus 14 / 4 plus 40 / 5 close parenthesis equals 1.917")[$frac(1, 6) ( 0 / 10 + 0 / 9 + 0 / 6 + 0 / 7 + 14 / 4 + 40 / 5 ) = 1.917$], or #math.equation(block: false, alt: "191.7 %")[$191.7 %$] - RMSE for cubic model: #math.equation(block: false, alt: "the square root of the fraction 1 over 6 open parenthesis 0 squared plus 0 squared plus 0 squared plus 0 squared plus 14 squared plus 40 squared close parenthesis equals 17.3")[$sqrt(frac(1, 6) ( 0^(2) + 0^(2) + 0^(2) + 0^(2) + 14^(2) + 40^(2) )) = 17.3$] - MAPE for linear model: #math.equation(block: false, alt: "the fraction 1 over 6 open parenthesis 0.2 / 10 plus 0.4 / 9 plus 1.4 / 6 plus 0.8 / 7 plus 1 / 4 plus 1.2 / 5 close parenthesis equals 0.15")[$frac(1, 6) ( 0.2 / 10 + 0.4 / 9 + 1.4 / 6 + 0.8 / 7 + 1 / 4 + 1.2 / 5 ) = 0.15$], or #math.equation(block: false, alt: "15 %")[$15 %$] - RMSE for linear model: #math.equation(block: false, alt: "the square root of the fraction 1 over 6 open parenthesis 0.2 squared plus 0.4 squared plus 1.4 squared plus 0.8 squared plus 1 squared plus 1.2 squared close parenthesis equals 0.93")[$sqrt(frac(1, 6) ( 0.2^(2) + 0.4^(2) + 1.4^(2) + 0.8^(2) + 1^(2) + 1.2^(2) )) = 0.93$] The MAPE and RMSE values for the linear model are much lower than their respective values for the cubic model even though the latter predicts four values with 100% accuracy. ==== Underfitting A model that is underfit may perform well on some input data and poorly on other data. This is because an underfit model fails to detect some important feature(s) of the dataset. #examplebox("Example 3")[][ A student recorded their time spent on each homework set in their statistics course. Each HW set was roughly similar in length, but the student noticed that they spent less and less time completing them as the semester went along. Find the linear regression model for the data in and discuss its effectiveness in making further predictions. #figure(table( columns: 7, align: left, inset: 6pt, table.header([#strong[HW Set]], [#strong[1]], [#strong[2]], [#strong[3]], [#strong[4]], [#strong[5]], [#strong[6]]), [Time (hr)], [9.8], [5.3], [3.4], [2.6], [1.9], [1.7], )) #solutionbox[ There is a definitive downward trend in the data. A simple linear regression model for this data is: #math.equation(block: true, alt: "y ^ equals minus 1.47143 x plus 9.26667")[$hat(y) = − 1.47143 x + 9.26667$] However, the line of best fit does not seem to fit the data all that well. There is no way that a linear function can #emph[bend] the way the data points seem to do in this example. #figure(figph[A line graph labeled “time spend on HW problem sets.” The X axis is labeled “Problem Set” and ranges from 0 to 8. The Y axis is labeled “time spent (hours)” and ranges from -2 to 12. The graph contains a scatterplot of six data points with the linear model that best fits the data—a straight blue line that declines from 9 to -1. A blue line is fitted to the data, and the point (7, -1) is highlighted in orange.], alt: "A line graph labeled “time spend on HW problem sets.” The X axis is labeled “Problem Set” and ranges from 0 to 8. The Y axis is labeled “time spent (hours)” and ranges from -2 to 12. The graph contains a scatterplot of six data points with the linear model that best fits the data—a straight blue line that declines from 9 to -1. A blue line is fitted to the data, and the point (7, -1) is highlighted in orange.", caption: [Linear Model That Best Fits the Data in Example 3. This graph contains a scatterplot of the six data points together with the linear model that best fits the data. This is a clear case of underfitting.]) Another issue with using the linear model for the data in is that eventually the values of #math.equation(block: false, alt: "y ^")[$hat(y)$] become negative! According to this model, it will take about #math.equation(block: false, alt: "minus 1")[$− 1$] hours to complete the 7th homework set, which is absurd. Perhaps a decreasing exponential model, shown in , would work better here. Proper model selection is the most effective way of dealing with underfitting. #figure(figph[A line graph labeled “time spend on HW problem sets (exponential model).” The X axis is labeled “Problem Set” and ranges from 0 to 8. The Y axis is labeled “time spent (hours)” and ranges from -2 to 12. The graph contains a scatterplot of six data points with the linear model that best fits the data—a curved blue line that declines from 11 to 1. The point (7, -1) is highlighted in orange], alt: "A line graph labeled “time spend on HW problem sets (exponential model).” The X axis is labeled “Problem Set” and ranges from 0 to 8. The Y axis is labeled “time spent (hours)” and ranges from -2 to 12. The graph contains a scatterplot of six data points with the linear model that best fits the data—a curved blue line that declines from 11 to 1. The point (7, -1) is highlighted in orange", caption: [Exponential Model That Best Fits the Data in Example 3. This graph contains a scatterplot of the six data points together with the exponential model that best fits the data. Under this model, the prediction of #math.equation(block: false, alt: "y ^ equals 1")[$hat(y) = 1$] when #math.equation(block: false, alt: "x equals 7")[$x = 7$] seems much more reasonable.]) ] ]