#set document(title: "1.5 Data Science with Python", author: "OpenStax / XYZ Homework") #set page(width: 8.5in, height: auto, margin: 1in) #import "@preview/cetz:0.5.2" #set text(font: ("STIX Two Text", "Libertinus Serif", "New Computer Modern"), size: 10.5pt, lang: "en") #show math.equation: set text(font: ("STIX Two Math", "New Computer Modern Math")) #set par(justify: true, leading: 0.62em, spacing: 0.9em) #set enum(spacing: 1.1em) // room between list items so tall inline fractions don't collide #set list(spacing: 1.1em) #set table(stroke: 0.5pt + rgb("#c7ccd3")) #let BLUE = rgb("#183B6F") // brand navy — section bars + example/solution labels (white on navy 11.09:1) #let ORANGE = rgb("#A94509") // brand primary-700 — AA-safe deep orange for TEXT (5.93:1 on white; raw brand #F37021 is 2.94:1 and must never carry text) #let RED = rgb("#DC2626") // brand error-600 #let GREEN = rgb("#059669") // brand success-600 (decoration only; small green text uses green-text #007942) #show heading.where(level: 1): it => block(width: 100%, above: 0pt, below: 16pt, fill: gradient.linear(BLUE, rgb("#2C5AA0")), inset: (x: 14pt, y: 12pt), radius: 3pt, text(fill: white, weight: "bold", size: 19pt, it.body)) #show heading.where(level: 2): it => block(width: 100%, above: 18pt, below: 10pt, fill: BLUE, inset: (x: 10pt, y: 6pt), radius: 2pt, text(fill: white, weight: "bold", size: 12pt, it.body)) #show heading.where(level: 3): it => text(fill: ORANGE, weight: "bold", size: 12.5pt, it.body) #show heading.where(level: 4): it => text(fill: BLUE, weight: "bold", size: 10.5pt, it.body) #let examplebox(label, title, body) = block(width: 100%, breakable: true, fill: rgb("#EFF1F5"), stroke: 0.5pt + rgb("#CFDDF0"), radius: 4pt, inset: 10pt, above: 12pt, below: 12pt)[ #block(below: 6pt)[#box(fill: BLUE, inset: (x: 6pt, y: 2pt), radius: 2pt, text(fill: white, weight: "bold", size: 8.5pt, label)) #h(0.4em) #strong[#title]] #body] // rail = decorative left rule (raw brand token); labelcolor = AA-safe label text shade #let notebox(label, rail, labelcolor, tint, body) = block(width: 100%, breakable: true, fill: tint, stroke: (left: 3pt + rail), inset: (left: 10pt, rest: 8pt), radius: (right: 4pt), above: 11pt, below: 11pt)[ #text(fill: labelcolor, weight: "bold", size: 7.5pt, tracking: 0.5pt)[#upper(label)] #linebreak() #body] #let solutionbox(body) = block(above: 4pt, below: 8pt)[ #text(fill: BLUE, weight: "bold", size: 8.5pt)[Solution] #linebreak() #body] #let figph(msg) = block(width: 100%, height: 60pt, fill: rgb("#f6f7f9"), stroke: (paint: rgb("#c7ccd3"), dash: "dashed"), radius: 4pt, inset: 10pt)[ #align(center + horizon, text(fill: rgb("#889"), style: "italic", size: 9pt, msg))] // Standardize inlined figure sizes: measure the natural CeTZ canvas, then scale to a // consistent envelope (aspect-aware; see build_typst.py FIG_* constants). Unlike the // print preamble, dimensions are FLOORED: in an editor a user can trim a figure to a // degenerate 1-D shape (a bare line), and w/h or tw/w would then divide by zero. #let _STD_W = 3.5 #let _WIDE_W = 5.6 #let _MAX_H = 3.4 #let _ASPECT_WIDE = 2.2 #let _UPSCALE_MAX = 1.15 #let stdfig(body) = context { let m = measure(body) let w = calc.max(m.width / 1in, 0.01) let h = calc.max(m.height / 1in, 0.01) let tw = if w / h > _ASPECT_WIDE { _WIDE_W } else { _STD_W } let s = calc.min(tw / w, _MAX_H / h, _UPSCALE_MAX) align(center, box(scale(x: s * 100%, y: s * 100%, reflow: true, body))) } #show figure: set block(breakable: false) #set figure(gap: 8pt) #show figure.caption: set text(size: 8.5pt, fill: rgb("#555")) == 1.5#h(0.6em)Data Science with Python === Learning Outcomes By the end of this section, you should be able to - Load data to Python. - Perform basic data analysis using Python. - Use visualization principles to graphically plot data using Python. Multiple tools are available for writing and executing Python programs. Jupyter Notebook is one convenient and user-friendly tool. The next section explains how to set up the Jupyter Notebook environment using Google Colaboratory (Colab) and then provides the basics of two open-source Python libraries named Pandas and Matplotlib. These libraries are specialized for data analysis and data visualization, respectively. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Python Programming] In the discussion below, we assume you are familiar with basic Python syntax and know how to write a simple program using Python. If you need a refresher on the basics, please refer to Das, U., Lawson, A., Mayfield, C., & Norouzi, N. (2024). #emph[Introduction to Python Programming]. OpenStax. #link("https://openstax.org/r/page1")[https://openstax.org/books/introduction-python-programming/pages/1-introduction]. ] === Jupyter Notebook on Google Colaboratory #strong[Jupyter Notebook] is a web-based environment that allows you to run a Python program more interactively, using programming code, math equations, visualizations, and plain texts. There are multiple web applications or software you could use to edit a Jupyter Notebook, but in this textbook we will use Google’s free application named #link("https://openstax.org/r/colab1")[#strong[Google Colaboratory (Colab)]], often abbreviated as Colab. It is a cloud-based platform, which means that you can open, edit, run, and save a Jupyter Notebook on your Google Drive. Setting up Colab is simple. On your Google Drive, click New \> More. If your Google Drive has already installed Colab before, you will see Colaboratory under More. If not, click “Connect more apps” and install Colab by searching “Colaboratory” on the app store . For further information, see the #link("https://openstax.org/r/1pp")[Google Colaboratory Ecosystem] animation. #figure(figph[A screenshot of the Google Drive icon with an add new button. Below is the Google Workspace Marketplace menu with a magnifying glass and Colaboratory highlighted.], alt: "A screenshot of the Google Drive icon with an add new button. Below is the Google Workspace Marketplace menu with a magnifying glass and Colaboratory highlighted.", caption: [Install Google Colaboratory (Colab)]) Now click New \> More \> Google Laboratory. A new, empty Jupyter Notebook will show up as in . #figure(figph[A screenshot of an empty Jupyter Notebook within Google Colaboratory. The notebook is named Untitled.jpynb.], alt: "A screenshot of an empty Jupyter Notebook within Google Colaboratory. The notebook is named Untitled.jpynb.", caption: [Google Colaboratory Notebook]) The gray area with the play button is called a cell. A #strong[cell] is a block where you can type either code or plain text. Notice that there are two buttons on top of the first cell—“+ Code” and “+ Text.” These two buttons add a code or text cell, respectively. A code cell is for the code you want to run; a text cell is to add any text description or note. Let’s run a Python program on Colab. Type the following code in a code cell. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ print ("hello world!") The resulting output will look like this: hello world! ] You can write a Python program across multiple cells and put text cells in between. Colab would treat all the code cells as part of a single program, running from the top to bottom of the current Jupyter Notebook. For example, the two code cells below run as if it is a single program. When running one cell at a time from the top, we see the following outputs under each cell. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ a = 1 print ("The a value in the first cell:", a) The resulting output will look like this: The a value in the first cell: 1 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ b = 3 print ("a in the second cell:", a) print ("b in the second cell:", b) a + b The resulting output will look like this: a in the second cell: 1 #linebreak() b in the second cell: 3 #linebreak() 4 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Conventional Python versus Jupyter Notebook Syntax] While conventional Python syntax requires print() syntax to print something to the program console, Jupyter Notebook does not require print(). On Jupyter Notebook, the line a+b instead of print(a+b) also prints the value of a+b as an output. But keep in mind that if there are multiple lines of code that trigger printing some values, #emph[only] the output from the last line will show. ] You can also run multiple cells in bulk. Click Runtime on the menu, and you will see there are multiple ways of running multiple cells at once . The two commonly used ones are “Run all” and “Run before.” “Run all” runs all the cells in order from the top; “Run before” runs all the cells before the currently selected one. #figure(figph[A screenshot of the Runtime menu in Google Colab with options to Run all, Run before, Run the focused cell, Run selection, and Run after with the keyboard shortcuts for each.], alt: "A screenshot of the Runtime menu in Google Colab with options to Run all, Run before, Run the focused cell, Run selection, and Run after with the keyboard shortcuts for each.", caption: [Multiple Ways of Running Cells on Colab]) One thing to keep in mind is that being able to split a long program into multiple blocks and run one block at a time raises chances of user error. Let’s look at a modified code from the previous example. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ a = 1 print ("the value in the first cell:", a) The resulting output will look like this: the value in the first cell: 1 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ b = 3 print ("a in the second cell:", a) print ("b in the second cell:", b) a + b The resulting output will look like this: a in the second cell: 1 #linebreak() b in the second cell: 3 #linebreak() 4 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ a = 2 a + b The resulting output will look like this: 5 ] The modified code has an additional cell at the end, updating a from 1 to 2. Notice that now a+b returns 5 as a has been changed to 2. Now suppose you need to run the second cell for some reason, so you run the second cell again. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ a = 1 print ("the a value in the first cell:", a) The resulting output will look like this: the a value in the first cell: 1 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ b = 3 print ("a in the second cell:", a) print ("b in the second cell:", b) a + b The resulting output will look like this: a in the second cell: 2 #linebreak() b in the second cell: 3 #linebreak() 5 ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ a = 2 a + b The resulting output will look like this: 5 ] The value of a has changed to 2. This implies that the execution order of each cell matters! If you have run the third cell before the second cell, the value of a will have the value from the third one even though the third cell is located below the second cell. Therefore, it is recommended to use “Run all” or “Run before” after you make changes across multiple cells of code. This way your code is guaranteed to run sequentially from the top. === Python Pandas One of the strengths of Python is that it includes a variety of free, open-source libraries. Libraries are a set of already-implemented methods that a programmer can refer to, allowing a programmer to avoid building common functions from scratch. Pandas is a Python library specialized for data manipulation and analysis, and it is very commonly used among data scientists. It offers a variety of methods, which allows data scientists to quickly use them for data analysis. You will learn how to analyze data using Pandas throughout this textbook. #strong[Colab] already has Pandas installed, so you just need to import Pandas and you are set to use all the methods in Pandas. Note that it is convention to abbreviate pandas to pd so that when you call a method from Pandas, you can do so by using pd instead of having to type out Pandas every time. It offers a bit of convenience for a programmer! #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import Pandas and assign an abbreviated identifier "pd" import pandas as pd ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Installing Pandas on Your Computer] If you wish to install Pandas on your own computer, refer to the #link("https://openstax.org/r/pyd")[installation page of the Pandas website]. ] ==== Load Data Using Python Pandas The first step for data analysis is to load the data of your interest to your Notebook. Let’s create a folder on Google Drive where you can keep a CSV file for the dataset and a Notebook for data analysis. Download a public dataset, #link("https://openstax.org/r/movieprofit")[ch1-movieprofit.csv], and store it in a Google Drive folder. Then open a new Notebook in that folder by entering that folder and clicking New \> More \> Google Colaboratory. Open the Notebook and allow it to access files in your Google Drive by following these steps: First, click the Files icon on the side tab . #figure(figph[A screenshot of the side tab of Google Colab showing the following icons: hamburger menu, magnifying glass, x in brackets, key, and folder. The folder icon is highlighted and the word “Files” has popped up.], alt: "A screenshot of the side tab of Google Colab showing the following icons: hamburger menu, magnifying glass, x in brackets, key, and folder. The folder icon is highlighted and the word “Files” has popped up.", caption: [Side Tab of Colab]) Then click the Mount Drive icon and select “Connect to Google Drive” on the pop-up window. #figure(figph[A screenshot of the Files popup menu on Google Colab. The menu includes four icons, and the Mount Drive icon is selected.], alt: "A screenshot of the Files popup menu on Google Colab. The menu includes four icons, and the Mount Drive icon is selected.", caption: [Features under Files on Colab]) Notice that a new cell has been inserted on the Notebook as a result . #figure(figph[Code snippet in Google Colab displaying a Python command to mount Google Drive. The command imports the 'drive' module and uses the 'mount' function with the path '/content/drive'.], alt: "Code snippet in Google Colab displaying a Python command to mount Google Drive. The command imports the 'drive' module and uses the 'mount' function with the path '/content/drive'.", caption: [An Inserted Cell to Mount Your Google Drive]) Connect your Google Drive by running the cell, and now your Notebook file can access all the files under content/drive. Navigate folders under drive to find your Notebook and #link("https://openstax.org/r/movieprofit")[ch1-movieprofit.csv] files. Then click “…” \> Copy Path . #figure(figph[A screenshot showing how to copy the path of a C S V file located in a Google Drive folder.], alt: "A screenshot showing how to copy the path of a C S V file located in a Google Drive folder.", caption: [Copying the Path of a CSV File Located in a Google Drive Folder]) Now replace \[Path\] with the copied path in the below code. Run the code and you will see the dataset has been loaded as a table and stored as a Python variable data. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# import Pandas and assign an abbreviated identifier "pd" import pandas as pd data = pd.read\_csv("\[Path\]") data The resulting output will look like this: #figure(figph[A Python output table displaying movie data, including title, year, genre, rating, duration, US gross, worldwide gross, and votes. The table is sorted by worldwide gross in descending order.], alt: "A Python output table displaying movie data, including title, year, genre, rating, duration, US gross, worldwide gross, and votes. The table is sorted by worldwide gross in descending order.", caption: none) ] The read\_csv() method in Pandas loads a CSV file and stores it as a DataFrame. A #strong[DataFrame] is a data type that Pandas uses to store multi-column tabular data. Therefore, the variable data holds the table in #link("https://openstax.org/r/movieprofit")[ch1-movieprofit.csv] in the form of a Pandas DataFrame. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[DataFrame versus Series] Pandas defines two data types for tabular data—DataFrame and Series. While DataFrame is used for multi-column tabular data, Series is used for single-column data. Many methods in Pandas support both DataFrame and Series, but some are only for one or the other. It is always good to check if the method you are using works as you expect. For more information, refer to the #link("https://openstax.org/r/docs")[Pandas documentation] or Das, U., Lawson, A., Mayfield, C., & Norouzi, N. (2024). #emph[Introduction to Python Programming]. OpenStax. #link("https://openstax.org/r/page1")[https://openstax.org/books/introduction-python-programming/pages/1-introduction]. ] #examplebox("Example 1")[][ Remember the Iris dataset we used in Data and Datasets? Load the dataset #link("https://openstax.org/r/iris")[ch1-iris.csv] to a Python program using Pandas. #solutionbox[ The following code loads the #link("https://openstax.org/r/iris")[ch1-iris.csv] that is stored in a Google Drive. Make sure to replace the path with the actual path to #link("https://openstax.org/r/iris")[ch1-iris.csv] on your Google Drive. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ import pandas as pd data = pd.read\_csv("\[Path to ch1-iris.csv\]") \# Replace the path data The resulting output will look like this: #figure(figph[A Python output table displaying a portion of the Iris dataset. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.], alt: "A Python output table displaying a portion of the Iris dataset. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.", caption: none) ] ] ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ #emph[Can I load a file that is uploaded to someone else’s Google Drive and shared with me?] Yes! This is useful especially when your Google Drive runs out of space. Simply add the shortcut of the shared file to your own drive. Right-click \> Organize \> Add Shortcut will let you select where to store the shortcut. Once done, you can call pd.read\_csv() using the path of the shortcut. ] ==== Summarize Data Using Python Pandas You can compute basic statistics for data quite quickly by using the DataFrame.describe() method. Add and run the following code in a new cell. It calls the describe() method upon data, the DataFrame we defined earlier with #link("https://openstax.org/r/movieprofit")[ch1-movieprofit.csv]. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-movieprofit.csv\]") data.describe() like this: #figure(figph[A Python output table displaying descriptive statistics for movie data, including count, mean, standard deviation, minimum, 25th percentile, median, 75th percentile, and maximum values for rating, duration, US gross, and worldwide gross in millions.], alt: "A Python output table displaying descriptive statistics for movie data, including count, mean, standard deviation, minimum, 25th percentile, median, 75th percentile, and maximum values for rating, duration, US gross, and worldwide gross in millions.", caption: none) ] describe() returns a table whose columns are a subset of the columns in the entire dataset and whose rows are different statistics. The statistics include the number of unique values in a column (count), mean (mean), standard deviation (std), minimum and maximum values (min/max), and different quartiles (25%/50%/75%), which you will learn about in Measures of Variation. Using this representation, you can compute such statistics of different columns easily. #examplebox("Example 2")[][ Summarize the IRIS dataset using describe() of #link("https://openstax.org/r/iris")[ch1-iris.csv] you loaded in the previous example. #solutionbox[ The following code in a new cell returns the summary of the dataset. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-iriscsv\]") data.describe() The resulting output will look like this: #figure(figph[A Python output table displaying descriptive statistics for four Iris flower measurements: sepal length, sepal width, petal length, and petal width, including count, mean, standard deviation, minimum, quartiles, and maximum values.], alt: "A Python output table displaying descriptive statistics for four Iris flower measurements: sepal length, sepal width, petal length, and petal width, including count, mean, standard deviation, minimum, quartiles, and maximum values.", caption: none) ] ] ] ==== Select Data Using Python Pandas The Pandas DataFrame allows a programmer to use the column name itself when selecting a column. For example, the following code prints all the values in the “US\_Gross\_Million” column in the form of a Series (remember the data from a single column is stored in the Series type in Pandas). #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-movieprofit.csv\]") data\["US\_Gross\_Million"\] like this: 0   760.51 #linebreak() 1   858.37 #linebreak() 2   659.33 #linebreak() 3   936.66 #linebreak() 4   678.82 #linebreak()     ... #linebreak() 961   77.22 #linebreak() 962  177.20 #linebreak() 963  102.31 #linebreak() 964  106.89 #linebreak() 965   75.47 #linebreak() Name: US\_Gross\_Million, Length: 966, dtype: float64 ] DataFrame.iloc\[\] enables a more powerful selection—it lets a programmer select by both column and row, using column and row indices. Let’s look at some code examples below. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.iloc\[:, 2\] \# select all values in the second column The resulting output will look like this: 0   2009 #linebreak() 1   2019 #linebreak() 2   1997 #linebreak() 3   2015 #linebreak() 4   2018 #linebreak()     ... #linebreak() 961  2010 #linebreak() 962  1982 #linebreak() 963  1993 #linebreak() 964  1999 #linebreak() 965  2017 #linebreak() Name: Year, Length: 966, dtype: object ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.iloc\[2,:\] \# select all values in the third row The resulting output will look like this: Unnamed: 0             3 #linebreak() Title            Titanic #linebreak() Year              1997 #linebreak() Genre             Drama #linebreak() Rating              7.9 #linebreak() Duration             194 #linebreak() US\_Gross\_Million       659.33 #linebreak() Worldwide\_Gross\_Million   2201.65 #linebreak() Votes           1,162,142 #linebreak() Name: 2, dtype: object ] To pinpoint a specific value within the “US\_Gross\_Million” column, you can use an index number. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ print (data\["US\_Gross\_Million"\]\[0\]) \# index 0 refers to the top row print (data\["US\_Gross\_Million"\]\[2\]) \# index 2 refers to the third row The resulting output will look like this: 760.51 #linebreak() 659.33 ] You can also use DataFrame.iloc\[\] to select a specific group of cells on the table. The example code below shows different ways of using iloc\[\]. There are multiple ways of using iloc\[\], but this chapter introduces a couple of common ones. You will learn more techniques for working with data throughout this textbook. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.iloc\[:, 1\] \# select all values in the second column (index 1) The resulting output will look like this: 0                     Avatar #linebreak() 1                Avengers: Endgame #linebreak() 2                     Titanic #linebreak() 3   Star Wars: Episode VII - The Force Awakens #linebreak() 4             Avengers: Infinity War #linebreak()              ...           #linebreak() 961                  The A-Team #linebreak() 962                    Tootsie #linebreak() 963              In the Line of Fire #linebreak() 964                 Analyze This #linebreak() 965            The Hitman's Bodyguard #linebreak() Name: Title, Length: 966, dtype: object ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.iloc\[\[1, 3\], \[2, 3\]\]   \# select the rows at index 1 and 3, the columns at index 2 and 3 The resulting output will look like this: #figure(figph[A Python output table with two columns and two rows. The first column is labeled “Year,” and the second column is labeled “Genre.” The first row contains the values “2019” and “Action,” and the second row contains the values “2015” and “Action.” There are two icons to the right of the table, one that looks like a calendar and one that looks like a bar chart.], alt: "A Python output table with two columns and two rows. The first column is labeled “Year,” and the second column is labeled “Genre.” The first row contains the values “2019” and “Action,” and the second row contains the values “2015” and “Action.” There are two icons to the right of the table, one that looks like a calendar and one that looks like a bar chart.", caption: none) ] #examplebox("Example 3")[][ Select a “sepal\_width” column of the IRIS dataset using the column name. #solutionbox[ The following code in a new cell returns the “sepal\_width” column. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-iris.csv\]") data\["sepal\_width"\] The resulting output will look like this: 0   3.5 #linebreak() 1   3.0 #linebreak() 2   3.2 #linebreak() 3   3.1 #linebreak() 4   3.6 #linebreak()    ... #linebreak() 145  3.0 #linebreak() 146  2.5 #linebreak() 147  3.0 #linebreak() 148  3.4 #linebreak() 149  3.0 #linebreak() Name: sepal\_width, Length: 150, dtype: float64 ] ] ] #examplebox("Example 4")[][ Select a “petal\_length” column of the IRIS dataset using iloc\[\]. #solutionbox[ The following code in a new cell returns the “petal\_length” column. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.iloc\[:, 2\] The resulting output will look like this: 0   1.4 #linebreak() 1   1.4 #linebreak() 2   1.3 #linebreak() 3   1.5 #linebreak() 4   1.4 #linebreak()    ... #linebreak() 145  5.2 #linebreak() 146  5.0 #linebreak() 147  5.2 #linebreak() 148  5.4 #linebreak() 149  5.1 #linebreak() Name: petal\_length, Length: 150, dtype: float64 ] ] ] ==== Search Data Using Python Pandas To search for some data entries that fulfill specific criteria (i.e., filter), you can use DataFrame.loc\[\] of Pandas. When you indicate the filtering criteria inside the brackets, \[\], the output returns the filtered rows within the DataFrame. For example, the code below filters out the rows whose genre is comedy. Notice that the output only has 307 out of the full 3,400 rows. You can check the output on your own, and you will see their Genre values are all “Comedy.” #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-movieprofit.csv\]") data.loc\[data\['Genre'\] == 'Comedy'\] The resulting output will look like this: #figure(figph[A Python output table displaying movie data, including title, year, genre, rating, duration, US gross, worldwide gross, and votes. The table is sorted by worldwide gross in descending order.], alt: "A Python output table displaying movie data, including title, year, genre, rating, duration, US gross, worldwide gross, and votes. The table is sorted by worldwide gross in descending order.", caption: none) ] #examplebox("Example 5")[][ Using DataFrame.loc\[\], search for all the items of #emph[Iris-virginica] species in the IRIS dataset. #solutionbox[ The following code returns a filtered DataFrame whose species are #emph[Iris-virginica]. All such rows show up as an output. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-iris.csv\]") data.loc\[data\['species'\] == 'Iris-virginica'\] The resulting figure will look like this: #figure(figph[A Python output table displaying a portion of the Iris dataset, specifically Iris-virginica species. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.], alt: "A Python output table displaying a portion of the Iris dataset, specifically Iris-virginica species. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.", caption: none) (Rows 109 through 149 not shown.) ] ] ] #examplebox("Example 6")[][ This time, search for all the items whose species is #emph[Iris-virginica] and whose sepal width is wider than 3.2. #solutionbox[ You can use a Boolean expression—in other words, an expression that evaluates as either True or False—inside data.loc\[\]. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data.loc\[(data\['species'\] == 'Iris-virginica') & (data\['sepal\_width'\] \> 3.2)\] The resulting output will look like this: #figure(figph[A Python output table displaying a portion of the Iris dataset, specifically Iris-virginica species. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.], alt: "A Python output table displaying a portion of the Iris dataset, specifically Iris-virginica species. Columns include sepal length, sepal width, petal length, petal width, and species. Rows show data for individual Iris flowers.", caption: none) ] ] ] === Visualize Data Using Python Matplotlib There are multiple ways to draw plots of data in Python. The most common and straightforward way is to import another library, Matplotlib, which is specialized for data visualization. Matplotlib is a huge library, and to draw the plots you only need to import a submodule named pyplot. Type the following import statement in a new cell. Note it is convention to denote matplotlib.pyplot with plt, similarly to denoting Pandas with pd. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ import matplotlib.pyplot as plt ] Matplotlib offers a method for each type of plot, and you will learn the Matplotlib methods for all of the commonly used types throughout this textbook. In this chapter, however, let’s briefly look at how to draw a plot using Matplotlib in general. Suppose you want to draw a scatterplot between “US\_Gross\_Million” and “Worldwide\_Gross\_Million” of the movie profit dataset (#link("https://openstax.org/r/movieprofit")[ch1-movieprofit.csv]). You will investigate scatterplots in more detail in Correlation and Linear Regression Analysis. The example code below draws such a scatterplot using the method scatter(). scatter() takes the two columns of your interest—data\["US\_Gross\_Million"\] and data\["Worldwide\_Gross\_Million"\]—as the inputs and assigns them for the #emph[x]- and #emph[y]-axes, respectively. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ data = pd.read\_csv("\[Path to ch1-movieprofit.csv\]") \# draw a scatterplot using matplotlib’s scatter() plt.scatter(data\["US\_Gross\_Million"\], data\["Worldwide\_Gross\_Million"\]) The resulting output will look like this: #figure(figph[An unlabeled scatter plot. The X axis ranges from 0 to 1,000. The Y axis ranges from 0 to 3,000. Data points are clustered toward the lower left corner, with a general upward trend indicating that a higher value on the X axis tends to correlate with a higher value on the Y axis.], alt: "An unlabeled scatter plot. The X axis ranges from 0 to 1,000. The Y axis ranges from 0 to 3,000. Data points are clustered toward the lower left corner, with a general upward trend indicating that a higher value on the X axis tends to correlate with a higher value on the Y axis.", caption: none) ] Notice that it simply has a set of dots on a white plane. The plot itself does not show what each axis represents, what this plot is about, etc. Without them, it is difficult to capture what the plot shows. You can set these with the following code. The resulting plot below indicates that there is a positive correlation between domestic gross and worldwide gross. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# draw a scatterplot plt.scatter(data\["US\_Gross\_Million"\], data\["Worldwide\_Gross\_Million"\]) \# set the title plt.title("Domestic vs. Worldwide Gross") \# set the x-axis label plt.xlabel("Domestic") \# set the y-axis label plt.ylabel("Worldwide") The resulting output will look like this: #figure(figph[A scatter plot comparing the domestic gross versus worldwide gross of movies. The x-axis represents domestic gross and ranges from 0 to 1,000, and the y-axis represents worldwide gross and ranges from 0 to 3,000. Each data point is a blue dot representing a movie. The plot shows a general positive correlation between domestic and worldwide gross, indicating that movies with higher domestic gross tend to also have higher worldwide gross. Data points are clustered toward the lower left corner, with a general upward trend.], alt: "A scatter plot comparing the domestic gross versus worldwide gross of movies. The x-axis represents domestic gross and ranges from 0 to 1,000, and the y-axis represents worldwide gross and ranges from 0 to 3,000. Each data point is a blue dot representing a movie. The plot shows a general positive correlation between domestic and worldwide gross, indicating that movies with higher domestic gross tend to also have higher worldwide gross. Data points are clustered toward the lower left corner, with a general upward trend.", caption: none) ] You can also change the range of numbers along the #emph[x]- and #emph[y]-axes with plt.xlim() and plt.ylim(). Add the following two lines of code to the cell in the previous Python code example, which plots the scatterplot. #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ \# draw a scatterplot plt.scatter(data\["US\_Gross\_Million"\], data\["Worldwide\_Gross\_Million"\]) \# set the title plt.title("Domestic vs. Worldwide Gross") \# set the x-axis label plt.xlabel("Domestic") \# set the y-axis label plt.ylabel("Worldwide") \# set the range of values of the x- and y-axes plt.xlim(1\*10\*\*2, 3\*10\*\*2) \# x axis: 100 to 300 plt.ylim(1\*10\*\*2, 1\*10\*\*3) \# y axis: 100 to 1,000 The resulting output will look like this: #figure(figph[A scatter plot comparing the domestic gross versus worldwide gross of movies. The x-axis represents domestic gross and ranges from 100 to 300, and the y-axis represents worldwide gross and ranges from 100 to 1,000. Each data point is a blue dot representing a movie. The plot shows a general positive correlation between domestic and worldwide gross, indicating that movies with higher domestic gross tend to also have higher worldwide gross. Data points are clustered toward the lower left corner, with a general upward trend.], alt: "A scatter plot comparing the domestic gross versus worldwide gross of movies. The x-axis represents domestic gross and ranges from 100 to 300, and the y-axis represents worldwide gross and ranges from 100 to 1,000. Each data point is a blue dot representing a movie. The plot shows a general positive correlation between domestic and worldwide gross, indicating that movies with higher domestic gross tend to also have higher worldwide gross. Data points are clustered toward the lower left corner, with a general upward trend.", caption: none) ] The resulting plot with the additional lines of code has a narrower range of values along the #emph[x]- and #emph[y]-axes. #examplebox("Example 7")[][ Using the iris dataset, draw a scatterplot between petal length and height of #emph[Setosa Iris]. Set the title, #emph[x]-axis label, and #emph[y]-axis label properly as well. #solutionbox[ #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ import matplotlib.pyplot as plt data = pd.read\_csv("\[Path to ch1-iris.csv\]") \# select the rows whose species are Setosa Iris setosa = data.loc\[(data\['species'\] == 'Iris-setosa')\] \# draw a scatterplot plt.scatter(setosa\["petal\_length"\], setosa\["petal\_width"\]) \# set the title plt.title("Petal Length vs. Petal Width of Setosa Iris") \# set the x-axis label plt.xlabel("Petal Length") \# set the y-axis label plt.ylabel("Petal Width") The resulting output will look like this: #figure(figph[A scatter plot showing the relationship between petal length and petal width of setosa iris flowers. The x-axis represents petal length, and the y-axis represents petal width. Each data point is represented by a blue dot. Data points are clustered within specific ranges of petal length and width, with some overlap between clusters.], alt: "A scatter plot showing the relationship between petal length and petal width of setosa iris flowers. The x-axis represents petal length, and the y-axis represents petal width. Each data point is represented by a blue dot. Data points are clustered within specific ranges of petal length and width, with some overlap between clusters.", caption: none) ] ] ] #notebox("Note", rgb("#8a94a6"), rgb("#556666"), rgb("#f7f8fa"))[ Datasets #emph[Note: The primary datasets referenced in the chapter code may also be #link("https://openstax.org/r/spreadsheetsd1")[downloaded here].] ] ==== Project A: Data Source Quality As a student of, or a new professional working in, data science, you will not always be collecting new primary data. It’s just as important to be able to locate, critically evaluate, and properly clean existing sources of secondary data. (Collecting and Preparing Data will cover the topic of data collection and cleaning in more detail.) Some reputable government data sources are: #linebreak() #link("https://openstax.org/r/datagov")[#strong[Data.gov]] #link("https://openstax.org/r/blsgov1")[#strong[Bureau of Labor Statistics (BLS)]] #linebreak() #link("https://openstax.org/r/noaagov")[#strong[National Oceanic and Atmospheric Administration (NOAA)]] Some reputable nongovernment data sources are: #linebreak() #link("https://openstax.org/r/kaggle1")[#strong[Kaggle]] #linebreak() #link("https://openstax.org/r/statista")[#strong[Statista]] #linebreak() #link("https://openstax.org/r/pewresearch")[#strong[Pew Research Center]] Using the suggested sources or similar-quality sources that you research on the Internet, find two to three datasets about the field or industry in which you intend to work. (You might also try to determine whether similar data sets are available at the national, state/province, and local/city levels.) In a group, formulate a specific, typical policy issue or business decision that managers in these organizations might make. For the datasets you found, compare and contrast their size, collection methods, types of data, update frequency and recency, and relevance to the decision question you have identified. ==== Project B: Data Visualization Using one of the data sources mentioned in the previous project, find a dataset that interests you. Download it as a CSV file. Use Python to read in the CSV file as a Pandas DataFrame. As a group, think of a specific question that might be addressed using this dataset, discuss which features of the data seem most important to answer your question, and then use the Python libraries Pandas and Matplotlib to select the features and make graphs that might help to answer your question about the data. Note, you will learn many sophisticated techniques for doing data analysis in later chapters, but for this project, you should stick to simply isolating some data and visualizing it using the tools present in this chapter. Write a brief report on your findings. ==== Project C: Privacy, Ethics, and Bias Identify at least one example from recent current events or news articles that is related to each of the following themes (starting references given in parentheses): + Privacy concerns related to data collection (See the #link("https://openstax.org/r/gaog")[Protecting Personal Privacy] website of the U.S. Government Accountability Office.) + Ethics concerns related to data collection, including fair use of copyrighted materials (See the #link("https://openstax.org/r/fairuse")[U.S. Copyright Office guidelines].) + Bias concerns related to data collection (See the #link("https://openstax.org/r/bias")[National Cancer Institute (NCI) article] on data bias.) Suppose that you are part of a data science team working for an organization on data collection for a major project or product. Discuss as a team how the issues of privacy, ethics, and equity (avoiding bias) could be addressed, depending on your position in the organization and the type of project or product. Anaconda. (2020). #emph[2020 state of data science]. https://www.anaconda.com/resources/whitepapers/state-of-data-science-2020 Clewlow, A. (2024, January 26). #emph[Three smart cities that failed within five years of launch]. Intelligent Build.tech. https://www.intelligentbuild.tech/2024/01/26/three-smart-cities-that-failed-within-five-years-of-launch/ Hays, C. L. (2004, November 14). What Wal-Mart knows about customers’ habits.#emph[New York Times]. https://www.nytimes.com/2004/11/14/business/yourmoney/what-walmart-knows-about-customers-habits.html Herrington, D. (2023, July 31). #emph[Amazon is delivering its largest selection of products to U.S. Prime members at the fastest speeds ever]. Amazon News. https://www.aboutamazon.com/news/operations/doug-herrington-amazon-prime-delivery-speed Hitachi. (2023, February 22). #emph[Ag Automation and Hitachi drive farming efficiency with sustainable digital solutions]. Hitachi Social Innovation. https://social-innovation.hitachi/en-us/case\_studies/digital-solutions-in-agriculture-drive-sustainability-in-farming/ IABAC. (2023, September 20). #emph[Fraud detection through data analytics: Identifying anomalies and patterns]. International Association of Business Analytics Certification. https://iabac.org/blog/fraud-detection-through-data-analytics-identifying-anomalies-and-patterns Statista. (2024, May 10). #emph[Walmart: weekly customer visits to stores worldwide FY2017-FY2024]. https://www.statista.com/statistics/818929/number-of-weekly-customer-visits-to-walmart-stores-worldwide/ Van Bocxlaer, A. (2020, 20 August). #emph[Sensors for a smart city]. RFID & Wireless IoT. https://www.rfid-wiot-search.com/rfid-wiot-global-sensors-for-a-smart-city