14.3 Descriptive Statistics (multiple columns)

Demo Vidoe Link

Sample Dataset

We will use the students test scores dataset found at my_csv_file.csv. See below on how to upload it. This is "clean" dataset.

1. This is simulated data (not real)
2. 180 rows (observations) and 8 columns (variables)
3. time is completion time for test1_score
4. id, gender, section and level are categorical data
5. time, test1_score, test2_score, and test3_score are quantitative data

Basic Info about the Dataset

x_df.info gives basic information about the dataset stored in x_df: column names, non-empty data values, and data types.

Example: Write a program to display basic information about the dataset (URL below).

my_csv_file.csv

import pandas as pd

url = 'my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.info()

<class 'pandas.core.frame.DataFrame'>
RangeIndex: 180 entries, 0 to 179
Data columns (total 8 columns):
 #   Column       Non-Null Count  Dtype  
---  ------       --------------  -----  
 0   id           180 non-null    int64  
 1   gender       180 non-null    object 
 2   section      179 non-null    object 
 3   time         180 non-null    float64
 4   level        180 non-null    object 
 5   test1_score  180 non-null    float64
 6   test2_score  180 non-null    float64
 7   test3_score  180 non-null    float64
dtypes: float64(4), int64(1), object(3)
memory usage: 11.4+ KB

Interpretation:

1. The first row tells us the object x_df is of class DataFrame from pandas
2. The second, RangeIndex, tells us there are 180 entries 0 to 179

Descriptive Stats: mean, std, quartiles, frequencies, correlations etc

Example: Write a program to read the dataset (URL below) and display the descriptive statistics - count, mean, std, 1st quartile, 2nd quartile, 3rd quartile and max of 'test1_score`.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df['test1_score'].describe()

count    180.000000
mean      74.410000
std        5.890378
min       58.800000
25%       70.700000
50%       74.150000
75%       78.625000
max       90.300000
Name: test1_score, dtype: float64

Example: Write a program to read the dataset (URL below) and display the descriptive statistics - count, mean, std, 1st quartile, 2nd quartile, 3rd quartile and max of test1_score and test2_score.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df[['test1_score','test2_score']].describe()

Descriptive Statistics: All quantitative variables

Example: Write a program to read the dataset (URL below) and display the descriptive statistics - count, mean, std, 1st quartile, 2nd quartile, 3rd quartile and max of all variables(columns).

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.describe()   # summary for all quantitative columns 

The summary for the variable id is meaningless and should be ignored.

Grouped Frequencies

Example: Write a program to read the dataset (URL below) and display the frequency distribution by 'gender`. Change the frequencies to percent.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.value_counts(subset=['gender'])  # value counts by the variable(column) gender

gender
male      98
female    82
dtype: int64

x_df.value_counts(subset=['gender'],normalize=True).mul(100).round(1).astype(str)+'%'

gender
male      54.4%
female    45.6%
dtype: object

Summary Statistics Grouped by Gender

Example: Write a program to read the dataset (URL below) and display a table of test1_score mean, min, max, std, size grouped by gender.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.groupby(['gender']).agg({'test1_score': ['mean', 'min', 'max','std','size']})

Example: Write a program to read the dataset (URL below) and display a table of test1_score mean, min, max, std, size grouped by level.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.groupby(['level']).agg({'test1_score': ['mean', 'min', 'max','std','size']})

Example: Write a program to read the dataset (URL below) and display a table of test2_score mean, min, max, std, size grouped by section.

my_csv_file.csv

x_df.groupby(['section']).agg({'test1_score': ['mean', 'min', 'max','std','size']})

Grouped by Two Categorical Variables

Example: Write a program to read the dataset (URL below) and display a table of test1_score mean, min, max, std, size grouped by level and gender.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.groupby(['level','gender']).agg({'test1_score': ['mean', 'min', 'max','std','size']})

Grouped by Three Categorical Variables

Example: Write a program to read the dataset (URL below) and display a table of test1_score mean, min, max, std, size grouped by level, gender and section.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.groupby(['level','section','gender']).agg({'test1_score': ['mean', 'min', 'max','std','size']})

Correlation Coefficient (Pearson)

Example: Write a program to read the dataset (URL below) and display the correlation-coefficient between time and test1_score

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

# correlation coefficient between time and test1_score
x_df['time'].corr(x_df['test1_score'])

0.040726134954403075

Example: Write a program to read the dataset (URL below) and use x_df.corr to display the correlation-coefficient table between all columns.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

x_df.corr()

Correlation Coefficient (Spearman)

Example: Write a program to read the dataset (URL below) and display the Spearman correlation-coefficient between test1_score and test1_score.

my_csv_file.csv

import pandas as pd

url='my_csv_file.csv'

x_df = pd.read_csv(url)

from scipy import stats
stats.spearmanr(x_df["test1_score"],x_df["test2_score"] )   # Spearman's rho - ranked no normality assumed

SpearmanrResult(correlation=0.008971805516759307, pvalue=0.9048529194438715)

Adapted from Python for Introductory Statistics, by Simon Aman (Truman College, City Colleges of Chicago), licensed under CC BY 4.0. Changes were made: reformatted as an accessible XYZ web edition with live in-browser code cells. License: CC-BY-4.0.