Exploratory Data Analysis (EDA) is the process of analyzing datasets to summarize their main characteristics, often using visual methods. It’s a crucial first step in any data science project that helps you understand the structure, patterns, relationships, and anomalies in your data. In this post, I’ll walk through the essential techniques of EDA and demonstrate them using Python.
Why is EDA Important?
Before diving into complex modeling, EDA helps you:
- Understand your data: Get familiar with variables, their distributions, and relationships.
- Clean your data: Identify missing values, outliers, and inconsistencies.
- Generate hypotheses: Discover patterns and relationships that lead to research questions.
- Choose appropriate models: The insights from EDA inform your modeling approach.
- Communicate findings: Visualizations from EDA help explain your data to stakeholders.
The EDA Process
A typical EDA workflow involves:
- Understanding the data structure
- Examining basic statistics
- Visualizing univariate distributions
- Analyzing relationships between variables
- Identifying patterns, trends, and outliers
Let’s go through each step with examples.
Setting Up the Environment
First, let’s import the necessary libraries:
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from scipy import stats
# Set seaborn style for better visualizations
sns.set_style("whitegrid")
plt.rcParams["figure.figsize"] = (12, 8)
# For reproducibility
np.random.seed(42)
1. Understanding Data Structure
Let’s load a dataset and examine its structure. We’ll use the Titanic dataset, a common beginner dataset in data science:
# Load the dataset
titanic = sns.load_dataset('titanic')
# Display the first few rows
print(titanic.head())
# Check the shape (rows, columns)
print(f"Dataset shape: {titanic.shape}")
# Get column names and data types
print(titanic.info())
# Check for missing values
print(titanic.isnull().sum())
This gives us an overview of the dataset, including the number of rows and columns, data types, and missing values.
2. Examining Basic Statistics
Next, let’s calculate summary statistics for our numeric variables:
# Summary statistics for numeric columns
print(titanic.describe())
# For categorical columns
print(titanic.describe(include=['O'])) # 'O' stands for object (string) type
# Unique values in categorical columns
for col in titanic.select_dtypes(include=['object']).columns:
print(f"\n{col}:")
print(titanic[col].value_counts())
print(f"Proportion of most common value: {titanic[col].value_counts(normalize=True).max():.2f}")
These statistics give us a quick numerical summary, including measures of central tendency (mean, median), dispersion (std, min, max), and distribution shape (25th, 50th, 75th percentiles).
3. Visualizing Univariate Distributions
Now, let’s visualize the distribution of each variable:
Numeric Variables
# Create histograms for numeric variables
numeric_cols = titanic.select_dtypes(include=['int64', 'float64']).columns
plt.figure(figsize=(15, 10))
for i, col in enumerate(numeric_cols, 1):
plt.subplot(2, 3, i)
sns.histplot(titanic[col], kde=True)
plt.title(f'Distribution of {col}')
plt.tight_layout()
plt.show()
# Box plots for the same variables
plt.figure(figsize=(15, 10))
for i, col in enumerate(numeric_cols, 1):
plt.subplot(2, 3, i)
sns.boxplot(y=titanic[col])
plt.title(f'Box plot of {col}')
plt.tight_layout()
plt.show()
Histograms show the frequency distribution of values, while box plots highlight the median, quartiles, and potential outliers.
Categorical Variables
# Bar plots for categorical variables
cat_cols = titanic.select_dtypes(include=['object']).columns
plt.figure(figsize=(15, 10))
for i, col in enumerate(cat_cols, 1):
plt.subplot(2, 2, i)
sns.countplot(y=titanic[col], order=titanic[col].value_counts().index)
plt.title(f'Count of {col}')
plt.tight_layout()
plt.show()
Bar plots show the frequency of each category in categorical variables.
4. Analyzing Relationships Between Variables
Next, let’s examine relationships between variables:
Correlations Between Numeric Variables
# Correlation matrix
corr_matrix = titanic.select_dtypes(include=['int64', 'float64']).corr()
# Heatmap of correlation matrix
plt.figure(figsize=(10, 8))
sns.heatmap(corr_matrix, annot=True, cmap='coolwarm', vmin=-1, vmax=1)
plt.title('Correlation Matrix')
plt.show()
# Pair plot for numeric variables
sns.pairplot(titanic[numeric_cols])
plt.suptitle('Pair Plot of Numeric Variables', y=1.02)
plt.show()
Correlation matrices and pair plots help visualize the strength and direction of relationships between numeric variables.
Relationships Between Categorical and Numeric Variables
# Box plots for categorical vs. numeric variables
plt.figure(figsize=(15, 10))
sns.boxplot(x='sex', y='age', data=titanic)
plt.title('Age Distribution by Sex')
plt.show()
plt.figure(figsize=(15, 10))
sns.boxplot(x='class', y='fare', data=titanic)
plt.title('Fare Distribution by Class')
plt.show()
# Violin plots provide more detail about the distribution
plt.figure(figsize=(15, 10))
sns.violinplot(x='survived', y='age', data=titanic)
plt.title('Age Distribution by Survival')
plt.show()
Box plots and violin plots show how a numeric variable’s distribution varies across categories.
Relationships Between Categorical Variables
# Contingency tables
contingency_table = pd.crosstab(titanic['sex'], titanic['survived'])
print(contingency_table)
# Visualize with a heatmap
plt.figure(figsize=(10, 8))
sns.heatmap(contingency_table, annot=True, cmap='Blues', fmt='d')
plt.title('Survival Count by Sex')
plt.show()
# Proportions
prop_table = pd.crosstab(titanic['class'], titanic['survived'], normalize='index')
print(prop_table)
plt.figure(figsize=(10, 8))
sns.heatmap(prop_table, annot=True, cmap='YlGnBu')
plt.title('Survival Rate by Class')
plt.show()
Contingency tables and heatmaps help visualize the relationship between categorical variables.
5. Identifying Patterns, Trends, and Outliers
Now, let’s dig deeper to find patterns and anomalies:
Detecting Outliers
# Z-score method
z_scores = stats.zscore(titanic['fare'].dropna())
outliers = (abs(z_scores) > 3)
print(f"Number of outliers: {sum(outliers)}")
# Visualize with a box plot
plt.figure(figsize=(10, 6))
sns.boxplot(x=titanic['fare'])
plt.title('Box Plot of Fare with Potential Outliers')
plt.show()
# IQR method
Q1 = titanic['fare'].quantile(0.25)
Q3 = titanic['fare'].quantile(0.75)
IQR = Q3 - Q1
lower_bound = Q1 - 1.5 * IQR
upper_bound = Q3 + 1.5 * IQR
outliers_iqr = ((titanic['fare'] < lower_bound) | (titanic['fare'] > upper_bound))
print(f"Number of outliers (IQR method): {sum(outliers_iqr)}")
The Z-score and IQR methods are common approaches for detecting outliers.
Finding Patterns in Survival
# Survival rate by passenger class and sex
survival_by_class_sex = titanic.pivot_table(
values='survived', index='sex', columns='class', aggfunc='mean'
)
plt.figure(figsize=(12, 6))
sns.heatmap(survival_by_class_sex, annot=True, cmap='YlGnBu', vmin=0, vmax=1)
plt.title('Survival Rate by Class and Sex')
plt.show()
# Age distribution of survivors vs. non-survivors
plt.figure(figsize=(12, 6))
sns.histplot(data=titanic, x='age', hue='survived', element='step', stat='density', common_norm=False)
plt.title('Age Distribution by Survival Status')
plt.show()
These visualizations help identify which groups had higher survival rates and how age distributions differed between survivors and non-survivors.
6. Advanced EDA Techniques
Let’s look at some more advanced techniques:
Kernel Density Estimation (KDE)
# KDE for fare distribution by survival status
plt.figure(figsize=(12, 6))
sns.kdeplot(data=titanic, x='fare', hue='survived', fill=True, common_norm=False)
plt.title('Fare Distribution by Survival Status')
plt.show()
KDE provides a smooth estimate of the probability density function.
Multivariate Analysis
# Scatter plot with multiple dimensions
plt.figure(figsize=(12, 8))
sns.scatterplot(data=titanic, x='age', y='fare', hue='survived', size='pclass', sizes=(50, 200), alpha=0.8)
plt.title('Age vs. Fare by Survival Status and Class')
plt.show()
This scatter plot visualizes the relationship between age and fare, with survival and class as additional dimensions.
Feature Engineering and Analysis
# Create a new feature for family size
titanic['family_size'] = titanic['sibsp'] + titanic['parch'] + 1
# Analyze survival by family size
plt.figure(figsize=(12, 6))
sns.barplot(x='family_size', y='survived', data=titanic, ci=None)
plt.title('Survival Rate by Family Size')
plt.show()
Feature engineering helps extract more insights from the data by creating new variables.
Common EDA Challenges and Solutions
Dealing with Missing Data
# Visualize missing data
plt.figure(figsize=(12, 6))
sns.heatmap(titanic.isnull(), cbar=False, yticklabels=False, cmap='viridis')
plt.title('Missing Data in Titanic Dataset')
plt.show()
# Filling missing values (imputation)
# For numeric columns, fill with median
for col in numeric_cols:
if titanic[col].isnull().sum() > 0:
titanic[col] = titanic[col].fillna(titanic[col].median())
# For categorical columns, fill with mode
for col in cat_cols:
if titanic[col].isnull().sum() > 0:
titanic[col] = titanic[col].fillna(titanic[col].mode()[0])
Visualizing missing data helps understand the pattern of missingness, and imputation is a common approach to handle missing values.
Handling Skewed Data
# Check skewness
for col in numeric_cols:
skewness = titanic[col].skew()
print(f"Skewness of {col}: {skewness:.2f}")
# Apply log transformation to fare (positive skew)
titanic['log_fare'] = np.log1p(titanic['fare'])
# Compare original and transformed distributions
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)
sns.histplot(titanic['fare'], kde=True)
plt.title('Original Fare Distribution')
plt.subplot(1, 2, 2)
sns.histplot(titanic['log_fare'], kde=True)
plt.title('Log-transformed Fare Distribution')
plt.tight_layout()
plt.show()
Log transformations can help normalize skewed data.
EDA Best Practices
- Start simple: Begin with basic statistics and univariate analysis before moving to more complex relationships.
- Be systematic: Follow a structured approach to ensure you don’t miss important insights.
- Iterate: EDA is an iterative process. New insights often lead to new questions and analyses.
- Document findings: Keep track of your discoveries and decisions made during EDA.
- Combine visualization with statistics: Visual methods are powerful, but they should be complemented with statistical tests.
- Consider domain knowledge: Context matters in interpreting data patterns.
Conclusion
Exploratory Data Analysis is not just a preliminary step—it’s a fundamental component of the data science workflow. By understanding your data through EDA, you can make informed decisions about preprocessing, feature engineering, and modeling approaches.
In this post, we’ve covered the essential techniques of EDA, from basic data examination to advanced visualization methods, using the Titanic dataset as an example. Remember that the specific techniques you use will depend on your data and research questions, but the structured approach outlined here provides a solid foundation for any data analysis project.
In future posts, we’ll explore more advanced EDA techniques and how they can be applied to different types of datasets and problems.