#set document(title: "15.4 Exploratory data analysis", author: "OpenStax / XYZ Homework") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 15.4#h(0.6em)Exploratory data analysis === Learning objectives By the end of this section you should be able to - Describe exploratory data analysis. - Inspect DataFrame entries through appropriate indexing. - Use filtering and slicing to obtain a subset of a DataFrame. - Identify Null values in a DataFrame. - Remove or replace Null values in a DataFrame. === Exploratory data analysis #strong[Exploratory Data Analysis (EDA)] is the task of analyzing data to gain insights, identify patterns, and understand the underlying structure of the data. During EDA, data scientists visually and statistically examine data to uncover relationships, anomalies, and trends, and to generate hypotheses for further analysis. The main goal of EDA is to become familiar with the data and assess the quality of the data. Once data are understood and cleaned, data scientists may perform feature creation and hypothesis formation. A #strong[feature] is an individual variable or attribute that is calculated from the raw data in the dataset. #strong[Data indexing] can be used to select and access specific rows and columns. Data indexing is essential in examining a dataset. In Pandas, two types of indexing methods exist: - Label-based indexing using loc\[\]: #strong[loc\[\]] allows you to access data in a DataFrame using row/column labels. Ex: df.loc\[row\_label, column\_label\] returns specific data at the intersection of row\_label and column\_label. - Integer-based indexing using iloc\[\]: #strong[iloc\[\]] allows you to access data in a DataFrame using integer-based indexes. Integer indexes can be passed to retrieve specific data. Ex: df.iloc\[row\_index, column\_index\] returns specific data at the index row\_index and column\_index. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Indexing a DataFrame] #link("https://www.openstax.org/r/indexing-DataFrame")[Indexing a DataFrame; ch 15, video 5] ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[DataFrame indexing] Given the following code, respond to the questions below. import pandas as pd \# Create sample data data = {   "A": \["a", "b", "c", "d"\],   "B": \[12, 20, 5, -10\],   "C": \["C", "C", "C", "C"\] } df = pd.DataFrame(data) ] === Data slicing and filtering Data slicing and filtering involve selecting specific subsets of data based on certain conditions or index/label ranges. #strong[Data slicing] refers to selecting a subset of rows and/or columns from a DataFrame. Slicing can be performed using ranges, lists, or Boolean conditions. - Slicing using ranges: Ex: df.loc\[start\_row:end\_row, start\_column:end\_column\] selects rows and columns within the specified ranges. - Slicing using a list: Ex: df.loc\[\[label1, label2, ...\], :\] selects rows that are in the list \[label1, label2, ...\] and includes all columns since all columns are selected by the colon operator. - Slicing based on a condition: df\[condition\] selects only the rows that meet the given condition. #strong[Data filtering] involves selecting rows or columns based on certain conditions. Ex: In the expression df\[df\['column\_name'\] \> threshold\], the DataFrame df is filtered using the selection operator (\[\]) and the condition(df\['column\_name'\] \> threshold) that is passed. All entries in the DataFrame df where the corresponding value in the DataFrame is True will be returned. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Indexing on a flight dataset] #link("https://www.openstax.org/r/indexing-flight-dataset")[Indexing on a flight dataset; ch 15, video 6] ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[DataFrame slicing and filtering] Given the following code, respond to the questions. import pandas as pd \# Create sample data data = {   "A": \[1, 2, 3, 4\],   "B": \[5, 6, 7, 8\],   "C": \[9, 10, 11, 12\] } df = pd.DataFrame(data) ] === Handling missing data Missing values in a dataset can occur when data are not available or are not recorded properly. Identifying and removing missing values is an important step in data cleaning and preprocessing. A data scientist should consider ethical considerations throughout the EDA process, especially when handling missing data. They might consider answering questions such as "Why are the data missing?", "Whose data are missing?", and "Considering the missing data, is the dataset still a representative sample of the population under study?". The functions below are useful in understanding and analyzing missing data. - isnull(): The isnull() function can be used to identify Null entries in a DataFrame. The return value of the function is a Boolean DataFrame, with the same dimensions as the original DataFrame with True values where missing values exist. - dropna(): The dropna() function can be used to drop rows with Null values. - fillna(): The fillna() function can be used to replace Null values with a provided substitute value. Ex: df.fillna(df.mean()) replaces all Null values with the average value of the specific column. To define a Null value in a DataFrame, you can use the np.nan value from the NumPy library. Functions that aid in identifying and removing null entries are described in the table below the following code. import pandas as pd import numpy as np \# Create sample data data = {   "Column 1": \["A", "B", "C", "D", "E"\],   "Column 2": \[np.NAN, 200, 500, 0, -10\],   "Column 3": \[True, True, False, np.NaN, np.NaN\] } df = pd.DataFrame(data) #figure(table( columns: 4, align: left, inset: 6pt, table.header([], [Column 1], [Column 2], [Column 3]), [0], [A], [NaN], [True], [1], [B], [200.0], [True], [2], [C], [500.0], [False], [3], [D], [0.0], [NaN], [4], [E], [-10.0], [NaN], )) #figure(table( columns: 4, align: left, inset: 6pt, table.header([Function], [Example], [Output], [Explanation]), [isnull()], [df.isnull()], [Column 1 Column 2 Column 3 0 False True False 1 False False False 2 False False False 3 False False True 4 False False True], [The df.isnull() function returns a Boolean array with Boolean values representing whether each entry is Null.], [fillna()], [df\["Column 2"\] =\\ df\["Column 2"\].fillna(df\["Column 2"\]   .mean())], [Column 1 Column 2 Column 3 0 A 172.5 True 1 B 200.0 True 2 C 500.0 False 3 D 0.0 NaN 4 E -10.0 NaN], [Null values in Column 2 are replaced with the mean of non-Null values in the column.], [dropna()], [\# Applied after the run \# of the previous row df = df.dropna()], [Column 1 Column 2 Column 3 0 A 172.5 True 1 B 200.0 True 2 C 500.0 False], [All rows containing a Null value are removed from the DataFrame.], )) #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Missing value treatment] ] === Programming practice with Google Use the Google Colaboratory document below to practice EDA on a given dataset. #link("https://openstax.org/r/100edapractice")[Google Colaboratory document]