Pandas is one of the most powerful and widely used libraries for data manipulation and analysis in Python. If you’re working with data in Python, knowing Pandas is essential. In this post, I’ll introduce you to the basics of Pandas and show you how to get started.
What is Pandas?
Pandas is an open-source Python library that provides high-performance, easy-to-use data structures and data analysis tools. The name “Pandas” is derived from the term “panel data,” which refers to multidimensional structured data sets.
The two primary data structures in Pandas are:
- Series: A one-dimensional labeled array capable of holding any data type
- DataFrame: A two-dimensional labeled data structure with columns of potentially different types
Installing Pandas
Before you can start using Pandas, you need to install it. The easiest way is using pip:
pip install pandas
Or with conda:
conda install pandas
Getting Started with DataFrames
Let’s dive right in and see how to create and manipulate DataFrames:
import pandas as pd
import numpy as np
# Creating a DataFrame from a dictionary
data = {
'Name': ['John', 'Anna', 'Peter', 'Linda'],
'Age': [28, 34, 29, 42],
'City': ['New York', 'Paris', 'Berlin', 'London'],
'Salary': [65000, 70000, 62000, 85000]
}
df = pd.DataFrame(data)
print(df)
Output:
Name Age City Salary
0 John 28 New York 65000
1 Anna 34 Paris 70000
2 Peter 29 Berlin 62000
3 Linda 42 London 85000
Basic Operations with DataFrames
Viewing Data
# Display the first few rows
print(df.head())
# Display the last few rows
print(df.tail())
# Get a statistical summary
print(df.describe())
# Info about the DataFrame
print(df.info())
Selecting Data
# Select a single column (returns a Series)
ages = df['Age']
print(ages)
# Select multiple columns
subset = df[['Name', 'Salary']]
print(subset)
# Select by row index
print(df.iloc[0]) # First row
print(df.iloc[1:3]) # Rows 1 to 2
# Select by label
print(df.loc[0, 'Name']) # Value at row 0, column 'Name'
# Boolean indexing
high_salary = df[df['Salary'] > 70000]
print(high_salary)
Adding and Modifying Data
# Add a new column
df['Experience'] = [3, 8, 5, 15]
print(df)
# Modify values
df.loc[0, 'Salary'] = 67000
print(df)
# Apply a function to a column
df['Salary'] = df['Salary'].apply(lambda x: x * 1.1) # 10% raise for everyone
print(df)
Data Cleaning with Pandas
Data cleaning is a crucial part of data analysis. Here are some common operations:
Handling Missing Values
# Create some missing values
df.loc[1, 'Experience'] = None
df.loc[3, 'Age'] = None
# Check for missing values
print(df.isnull().sum())
# Fill missing values
df['Experience'] = df['Experience'].fillna(0)
df['Age'] = df['Age'].fillna(df['Age'].mean())
# Drop rows with any missing values
df_clean = df.dropna()
Removing Duplicates
# Add a duplicate row
df = pd.concat([df, df.iloc[[0]]])
print(df)
# Identify duplicates
print(df.duplicated())
# Remove duplicates
df = df.drop_duplicates()
print(df)
Data Aggregation and Grouping
Pandas excels at aggregating and grouping data:
# Create a larger dataset
data = {
'Department': ['IT', 'IT', 'HR', 'HR', 'Finance', 'Finance'],
'Employee': ['Alice', 'Bob', 'Charlie', 'David', 'Eva', 'Frank'],
'Salary': [72000, 68000, 65000, 62000, 85000, 82000],
'Experience': [5, 3, 7, 2, 10, 8]
}
df = pd.DataFrame(data)
print(df)
# Group by department and calculate means
dept_means = df.groupby('Department').mean()
print(dept_means)
# Group by department and aggregate multiple stats
aggregated = df.groupby('Department').agg({
'Salary': ['mean', 'min', 'max', 'sum'],
'Experience': ['mean', 'min', 'max']
})
print(aggregated)
Reading and Writing Data
Pandas can read and write data in various formats:
# CSV
df.to_csv('employees.csv', index=False)
df_from_csv = pd.read_csv('employees.csv')
# Excel
df.to_excel('employees.xlsx', index=False)
df_from_excel = pd.read_excel('employees.xlsx')
# JSON
df.to_json('employees.json')
df_from_json = pd.read_json('employees.json')
# SQL
from sqlalchemy import create_engine
engine = create_engine('sqlite:///employees.db')
df.to_sql('employees', engine, if_exists='replace')
df_from_sql = pd.read_sql('SELECT * FROM employees', engine)
Data Visualization with Pandas
Pandas integrates well with matplotlib for quick visualizations:
import matplotlib.pyplot as plt
# Bar plot
df.groupby('Department')['Salary'].mean().plot(kind='bar')
plt.title('Average Salary by Department')
plt.tight_layout()
plt.show()
# Histogram
df['Experience'].plot(kind='hist', bins=5)
plt.title('Experience Distribution')
plt.show()
# Scatter plot
df.plot(kind='scatter', x='Experience', y='Salary')
plt.title('Salary vs. Experience')
plt.show()
Conclusion
Pandas is an incredibly powerful library that simplifies data manipulation and analysis in Python. This introduction covered just the basics, but even these fundamentals can help you perform complex data operations with just a few lines of code.
In future posts, we’ll explore more advanced Pandas features like time series analysis, merging datasets, and advanced data transformations.
Happy data wrangling!