14.1 Descriptive Statistics with Pandas

Watch demo video

Pandas Library

Pandas is another Python software library that is very useful for data analysis. Pandas is typically imported with alias np.

Pandas DataFrame

The Pandas DataFrame function is used for entering the data values in a spreadsheet like format wtih rows and columns labeled 0, 1, 2,... by default. Pandas output is typically displayed as DataFrame too.

Note: We will use the variable name x to store dataset in Python List.We will use the variable name x_df to store dataset in Pandas DataFrame.

To convert the Python list x to pandas dataframe x_df and display the results:

1. Click + Code
2. Type import pandas as pd
3. Type x = [x1, x2,...]
4. Type x_df = pd.DataFrame({'column_name':x})
5. Type x_df
6. Click play button

Warning: Like all Python variables x_df will only contain dataset that was stored last. Old values are replaced(updated) by the last data values stored.

Example: The dataset below is the waiting time, in minutes, of customers at a fast food restaurant. Write a pandas program to store and display the dataset in DataFrame.

5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5

1. Click + Code
2. Type import pandas as pd
3. Type x = [5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5]
4. Type x_df = pd.DataFrame({'waiting_time':x})
5. Type print(x_df)
6. Click play button

import pandas as pd
x = [5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5]
x_df = pd.DataFrame({'waiting_time':x})
print(x_df)

    waiting_time
0              5
1              3
2              4
3              1
4              5
5              3
6              5
7              2
8              3
9              2
10             5
11             4
12             4
13             6
14             4
15             3
16             4
17             1
18             2
19             5

Note: Unlike Numpy, Pandas DataFrame output is given in a spreadsheet like format. It has both column(s) and row(s). The first column is called the index column and it is automatically generated by Python. In the output above, the first column 0, 1, 2,..., 19 is the index column. Make a special note that it does NOT start at 1. It starts at 0. Waiting_time is the name of the second column after the index column. We only have one column with dataset here: waiting_time. The variable name for the whole DataFrame, including the columns and rows, is x_df.

Numpy vs. Pandas

Both numpy and pandas can be used for most computations in descriptive statistics. Unlike numpy, Pandas requires converting the dataset list to Pandas dataframe before any computation or analysis. But Pandas has more statistical computing and plotting capabilities.

Value Counts and Relative Frequency using Pandas

1. Pandas syntax for frequency

import pandas as pd  # pd alias for pandas
x=[x1,x2,x3,...]
x_df=pd.DataFrame({'column_name':x})
x_df.value_counts()

The above returns counts of unique values (frequencies).

1. Pandas syntax for a relative frequency

import pandas as pd
x=[x1,x2,x3,...]
x_df=pd.DataFrame({'column_name':x})
x_df.value_counts(normalize=True)

The above returns a relative frequency of the unique values (Frequency ÷ Total Frequency)

Example: The dataset below is the waiting time, in minutes, of customers at a fast food restaurant. Write a pandas program to construct a frequency distribution.

5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5

1. Click + Code
2. Type import pandas as pd
3. Type x= [5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5]
4. Type x_df = pd.DataFrame(x,columns = ['waiting_time'])
5. Type x_df.value_counts(sort=False)
6. Click play button

import pandas as pd
x= [5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5]
x_df = pd.DataFrame(x,columns = ['waiting_time'])
x_df.value_counts(sort=False)    # if sort is True it will sort by frequency

waiting_time
1               2
2               3
3               4
4               5
5               5
6               1
dtype: int64

Interpretation from the above output:

Example: The dataset below is the waiting time, in minutes, of customers at a fast food restaurant. Write a pandas program to construct a relative frequency distribution.

5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5

import pandas as pd

x = [5, 3, 4, 1, 5, 3, 5, 2, 3, 2, 5, 4, 4, 6, 4, 3, 4, 1, 2, 5]
x_df = pd.DataFrame({'waiting_time':x})
x_df.value_counts(normalize=True, sort=False)    # add normalize=True for relative freq

waiting_time
1               0.10
2               0.15
3               0.20
4               0.25
5               0.25
6               0.05
dtype: float64

Interpretation from the above output

Example: 20 students were asked if they were registered to vote. Their responses are given below. Write a program to construct a frequency and relative frequency distribution using pandas library.

NO, NO, YES, YES, NO, YES, NO, NO, YES, NO, YES, NO, NO, YES, YES, YES, YES, NO, YES, YES

# non-numerical values should be in 'quotes' 
x = ['NO', 'NO', 'YES', 'YES', 'NO', 'YES', 'NO', 'NO', 'YES', 'NO', 
             'YES', 'NO', 'NO', 'YES', 'YES', 'YES', 'YES', 'NO', 'YES', 'YES']
x_df = pd.DataFrame({'vote':x})
x_df.value_counts()

vote
YES     11
NO       9
dtype: int64

Interpretation from the above output

vote

# non-numerical values should be in 'quotes' 
x = ['NO', 'NO', 'YES', 'YES', 'NO', 'YES', 'NO', 'NO', 'YES', 'NO', 
             'YES', 'NO', 'NO', 'YES', 'YES', 'YES', 'YES', 'NO', 'YES', 'YES']
x_df = pd.DataFrame({'vote':x})
x_df.value_counts(normalize=True)

vote
YES     0.55
NO      0.45
dtype: float64

Interpretation from the above output

Descriptive Statistics with Pandas

Pandas syntax for the mean:

import pandas as pd
x = [x1,x2,x3,...,xn]
x_df = pd.DataFrame({'column_name':x})
x_df.mean()

Example: Find the mean of the data set 1,2,3 using pandas.

import pandas as pd
x = [1,2,3]
x_df = pd.DataFrame({'x':x})
x_df.mean()

x    2.0
dtype: float64

Interpretation: the mean is 2.0

Pandas syntax for the median:

import pandas as pd
x = [x1,x2,x3,...,xn]
x_df = pd.DataFrame(x)
x_df.median()

Example: find the median of the data set 1,2,3,4

import pandas as pd
x = [1,2,3,4]
x_df = pd.DataFrame({'x':x})
x_df.median()

x    2.5
dtype: float64

Interpretation: the median is 2.5

Mode

The mode(s) of a data set is the most frequent value(s).

pandas syntax for mode

import pandas as pd
x = [x1,x2,x3,...]
x_df = pd.DataFrame(x)
x_df.mode()

Example: find the mode of the data set 1,2,3,3,4,4

import pandas as pd
x = [1,2,3,3,4,4]
x_df = pd.DataFrame({'x':x})
x_df.mode()

Interpretation: There are two modes here, 3 and 4. It is bimodal. Recall that the columns and rows in pandas are 0, 1, 2,3,...

In this case, there is one column x and two rows 0 and 1.

Example: The dataset below is scores of 20 students in a Math class. Find the mean, median and mode.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

import pandas as pd
x = [81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 
             32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'score':x})

print("The mean",x_df.mean())
print('the median',x_df.median())
print('the mode',x_df.mode())

The mean score    42.866667
dtype: float64
the median score    36.0
dtype: float64
the mode    score
0     36

Interpretation: The mean is 42.866667, the median is 36.0 and the mode is 36

Example: The dataset below is scores of 21 students in a Math class. Find mode.

17, 81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

import pandas as pd
x = [17, 81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 
             32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'score':x})

print("Mode",x_df.mode())

Mode    score
0     17
1     36

There are two modes here: 17 and 36.

pandas syntax for variance and std:

import pandas as pd
x = [x1,x2,x3,...]
x_df = pd.DataFrame({'column_name':x})
x_df.var(ddof=0)      #pop variance
x_df.var()            #sample variance
x_df.std(ddpf=0)      #pop std
x_df.std()            #sample std 

Note: ddof=1 by default for pandas

Example: find the population variance of the data set 1,2,3,4,5

1. Click + Code
2. Type import pandas as pd
3. Type x = [1,2,3,4,5]
4. Type x_df =pd.DataFrame(x)
5. Type x_df.var(ddof=0)
6. Click play

import pandas as pd
x = [1,2,3,4,5]
x_df =pd.DataFrame(x)
x_df.var(ddof=0)              #in Pandas ddof=1 by default

0    2.0
dtype: float64

Interpretation: the population variance is 2.0

Example: find the sample variance of the data set 1,2,3,4,5

import pandas as pd
x = [1,2,3,4,5]
x_df =pd.DataFrame(x)
x_df.var()

0    2.5
dtype: float64

Interpretation: the sample variance is 2.5

Example: find the population standard deviation of the data set 1,2,3,4,5

import pandas as pd
x = [1,2,3,4,5]
x_df =pd.DataFrame(x)
x_df.std(ddof=0)

0    1.414214
dtype: float64

Interpretation: the population standard deviation is 1.414214

Example: find the sample standard deviation of the data set 1,2,3,4,5

import pandas as pd
x = [1,2,3,4,5]
x_df =pd.DataFrame(x)
x_df.std()

0    1.581139
dtype: float64

Interpretation: the sample standard deviation is 1.581139

Example: The dataset below is scores of 20 students in a Math class. Find the sample standard deviation and variance.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

import pandas as pd

x = [81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32,
             52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]

x_df = pd.DataFrame({'score':x})

x_df.std()        #sample standard deviation

score    23.555852
dtype: float64

In the above output, 23.555852 is the sample standard deviation.

import pandas as pd

x = [81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32,
             52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]

x_df = pd.DataFrame({'score':x})

x_df.var()        #sample variance

score    554.878161
dtype: float64

Interpretation: the sample variance is 554.878161

Example: The dataset below is scores of 20 students in a Math class. Find the population standard deviation and variance.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

import pandas as pd

x = [81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32,
             52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]

x_df = pd.DataFrame({'score':x})

x_df.std(ddof=0)     # population standard deviation ddof degrees of freedom

score    23.159927
dtype: float64

In the above output, 23.159927 is the population standard deviation.

import pandas as pd

x = [81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32,
             52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]

x_df = pd.DataFrame({'score':x})

x_df.var(ddof=0)     # population variance

score    536.382222
dtype: float64

Interpretation: The population variance is 536.382222

Percentiles

pandas syntax for percentile

import pandas as pd
x_df.quantile(p) 

Example: The dataset below is scores of 20 students in a Math class.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

• Find the 70th percentile.

Note: These results by pandas could be different from other results that use a different method of computing percentiles.

import pandas as pd
x=[81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 
           52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'score':x})
x_df.quantile(.70)   

score    50.6
Name: 0.7, dtype: float64

Interpretation: The 70th percentile is approximately 50.6

Note: In the output above, score is the name of the variable/column, 50.6 is the calculated 70th percentile, and dytype: float64 tells us that this result 50.6 is of type float64 is a double precision float.

Example: The dataset below is scores of 20 students in a Math class.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

• Find the 83rd percentile.

Note: These results by pandas could be different from other results that use a different method of computing percentiles.

import pandas as pd
x=[81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 
           52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'score':x})
x_df.quantile(.83)   

score    79.07
Name: 0.83, dtype: float64

Interpretation: The 83rd percentile is 79.07

Example: The dataset below is scores of 20 students in a Math class.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

1. Find the 70th percentile.
2. Find the 83rd percentile.
3. Find the 1st quartile.
4. Find the 3rd quartile.

Note: These results by pandas could be different from other results that use a different method of computing percentiles.

import pandas as pd
x=[81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 
           52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'score':x})
x_df.quantile([.7,.83,.25,.75])  # this will print all of them at once

Interpretation: In the output above, the first column is the percentiles given in decimals and the second column is the corresponding values. For example, 0.70 50.60 tells us the 70th percentile is 50.60.

Summary Statistics, Skewness and Kurtosis

use x_df.describe to get summary stats of a data. This code returns: size(count), mean, std, min, 1st, 2nd, 3rd quartile, max.

use x_df.skew for skewness and x_df.kurtosis for kurtosis.

Skewness is a measure of symmetry/asymmetry of the shape and kurtosis is a measure of heavy/light tail.

Example: The dataset below is scores of 20 students in a Math class. Use the x_df.describe to find summary stats.

81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32, 52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19

import pandas as pd
x=[81, 21, 53, 32, 80, 13, 26, 42, 35, 41, 14, 25, 87, 62, 46, 40, 32,
           52, 85, 28, 17, 36, 83, 79, 18, 50, 17, 36, 36, 19]
x_df = pd.DataFrame({'data':x})

x_df.describe()

In the above output, count is the number of data values, std is sample standard deviation, 25% is the 1st quartile or 25th percentile.

x_df.skew()

data    0.694929
dtype: float64

Interpretation: The skewness index is 0.694929. If the value of skew is positive, the shape of the data is skewed to the right. Negative implies skewed to the left and zero indicates no skewness.

x_df.kurtosis()

data   -0.71888
dtype: float64

Interpretation: The kurtosis index is -0.71888. For kurtosis, the general guideline is that if the value is greater than 3, the distribution has heavier tails and if it is less than 3, it indicates a distribution that has lighter tails as compared to the normal distribution.

Adapted from Python for Introductory Statistics, by Simon Aman (Truman College, City Colleges of Chicago), licensed under CC BY 4.0. Changes were made: reformatted as an accessible XYZ web edition with live in-browser code cells. License: CC-BY-4.0.