Chapter 4 — Data Handling Using Pandas Ii

What this topic is about
Pandas is a Python library for fast, flexible and expressive data manipulation. "Introduction and Recap" reviews core concepts and common operations you need before moving to advanced data handling in Class 11 Informatics Practices: creating Series and DataFrame objects, importing/exporting data, inspecting and summarizing data, indexing and selection, handling missing values, basic transformations, grouping/aggregation, merging/concatenating, reshaping and simple plotting.

Key building blocks

• Series — one-dimensional labeled array.
• DataFrame — two-dimensional labeled table (rows and columns).
• Import/export — read and write CSV/Excel/JSON (e.g. pd.read_csv, df.to_csv).
• Inspection — df.head(), df.tail(), df.info(), df.describe(), df.shape.
• Indexing & selection — df['col'], df.loc[row_label, col_label], df.iloc[row_idx, col_idx], boolean masks.
• Missing values — detect with df.isnull(), count with .sum(), remove with df.dropna(), replace with df.fillna().
• Type conversion — df.dtypes, df.astype('int'), pd.to_datetime() for dates.
• Aggregation & grouping — df.groupby('col').agg({'val':'mean'}), useful for summaries by category.
• Merging & concatenation — pd.concat([...]), pd.merge(df1, df2, on='key', how='inner').
• Reshaping — df.pivot_table(...), pd.melt(...).
• Apply/map — df['col'].apply(func), df.apply(func, axis=1) for elementwise or rowwise transforms.
• Plotting — basic plots via df.plot() or seaborn for quick visual checks.

Why recap matters
Before applying advanced functions (multi-indexing, time-series resampling, advanced joins or groupby complex aggregations) you must be comfortable with basic data ingestion, cleaning, indexing and simple summaries. Most data-analysis workflows are iterative: load → inspect → clean → transform → analyze → visualize.

Small code examples (recap)

# import and read
import pandas as pd
df = pd.read_csv('data.csv')

# inspect
df.head()
df.info()
df.describe()

# select rows where score > 50
passed = df[df['score'] > 50]

# handle missing
df['age'].fillna(df['age'].mean(), inplace=True)

# group and aggregate
summary = df.groupby('class')['score'].mean()

# merge datasets
merged = pd.merge(df1, df2, on='student_id', how='left')

# simple plot
df['score'].hist()

Study tips
Practice on small real datasets (student marks, sales records). After each operation inspect the result (shape, head, unique values). Use chaining where it improves readability but break complicated steps into named intermediate variables while learning.

What it is: Reading and writing data in Pandas means loading data from external sources (CSV, Excel, JSON, SQL, HTML, Parquet, URLs, compressed files) into DataFrame objects and saving DataFrames back to those formats. Pandas provides pd.read_... functions to read and DataFrame.to_... methods to write.

Key concepts:

• Readers: pd.read_csv, pd.read_excel, pd.read_json, pd.read_html, pd.read_sql, pd.read_parquet, etc.
• Writers: df.to_csv, df.to_excel, df.to_json, df.to_sql, df.to_parquet, etc.
• Important parameters: filepath_or_buffer, sep, header, index_col, usecols, dtype, parse_dates, na_values, skiprows, nrows, chunksize, encoding, compression, sheet_name, if_exists.
• Large files / streaming: Use chunksize to iterate over file pieces; use usecols and explicit dtype to reduce memory; consider reading compressed or parquet for space and speed.
• Data cleaning on read: Handle missing values (na_values), parse dates (parse_dates), convert dtypes, skip bad rows (error_bad_lines / on_bad_lines), and specify encodings.

Best practices:

• Pass explicit dtypes for large datasets to reduce memory and parsing time.
• Use parse_dates for time columns so you get datetime dtype.
• Use index_col when a column represents a natural index (e.g., id or timestamp).
• For multiple sheets in Excel use sheet_name=None to read all sheets into a dict of DataFrames.
• When writing, choose index=False unless the index is meaningful; pick appropriate compression to save disk space.

How it works (short workflow):

1. Choose the reader matching the source format.
2. Tune parameters (columns to read, dtypes, date parsing, encoding).
3. Optionally read in chunks and process each chunk to avoid high memory use.
4. After transformations, write out using the writer with desired options (index, header, compression).

Simple code pattern for chunked processing:

for chunk in pd.read_csv('bigfile.csv', chunksize=10000, usecols=['A','B'], parse_dates=['date']):
    # process chunk (clean, aggregate, write or append to DB/file)

Overview: Selecting, slicing and indexing in pandas are the basic operations used to access subsets of data from a DataFrame or Series. Indexing identifies rows and columns (labels or integer positions). Selecting extracts specific columns or rows. Slicing returns a range of rows/columns.

Indexing types:

• Label-based (use .loc): select using row/column labels. Note: label-based .loc slice endpoints are inclusive.
• Position-based (use .iloc): select using integer positions. .iloc slicing follows Python convention: end is exclusive.
• Scalar fast-access: .at (label) and .iat (integer position) for single value access, faster than .loc/.iloc for single elements.

Selecting columns and rows:

• Select a column: df['ColumnName'] or df.ColumnName (first is preferred).
• Select multiple columns: df[['A','B']].
• Select rows by label: df.loc['row_label'] or many rows df.loc['r1':'r5'].
• Select rows by position: df.iloc[0] or df.iloc[0:5].

Slicing with both axes:

Use df.loc[row_labels, col_labels] or df.iloc[row_positions, col_positions]. Examples: df.loc[2:6, 'Name':'Marks'] (inclusive labels) or df.iloc[0:4, 1:3] (positions 0–3 rows and columns 1–2).

Boolean (conditional) indexing: Filter rows by condition(s): df[df['Marks'] >= 50]. Combine conditions with & (and), | (or) and wrap each condition in parentheses: df[(df['Age'] >= 18) & (df['City'] == 'Delhi')].

Advanced points & best practices:

• Avoid chained indexing like df[df['A'] > 0]['B'] = value because it can produce SettingWithCopyWarning. Use loc for assignment: df.loc[df['A'] > 0, 'B'] = value.
• Use set_index('col') to make a column the DataFrame index (useful for time-series or unique IDs). Use reset_index() to revert.
• Use head(), tail(), sample() for quick checks.

Why it matters (real-life context): Efficient selecting and slicing lets you extract the exact subset of records you need for analysis, plotting or reporting — e.g., retrieving last month’s sales, selecting students who failed, or sampling rows for a preview.

What is missing data? Missing data (or null values) appear when entries are absent in a dataset. Pandas represents them as NaN (Not a Number) or None. Handling missing data is essential because many operations and machine learning models require complete data and because improper handling can bias results.

Types of missingness:

• MCAR (Missing Completely At Random): missingness unrelated to any data values.
• MAR (Missing At Random): missingness related to observed data (e.g., older respondents skip an item).
• MNAR (Missing Not At Random): missingness related to the missing value itself (e.g., people with very high income not reporting it).

Common steps in Pandas:

1. Detect: find where data are missing using df.isnull(), df.isnull().sum(), or df.info().
2. Assess: compute percentage missing per column and decide whether to drop or impute (based on domain knowledge and amount missing).
3. Handle: choose a strategy — remove rows/columns, impute values, or use models that can handle missingness.

Pandas techniques:

• dropna(axis=0|1, how='any'|'all', thresh=... , subset=[...]) — remove rows or columns with missing values. Use thresh to require a minimum number of non-null values.
• fillna(value=..., method='ffill'|'bfill', inplace=False) — replace nulls with a value or forward/backward fill.
• interpolate(method='linear'|'time') — numeric interpolation (useful for time-series).
• Model-based imputers (scikit-learn): KNNImputer, IterativeImputer for more advanced filling.
• Create a missing indicator column (e.g., df['age_missing'] = df['age'].isnull().astype(int)) to preserve information about missingness.

Trade-offs and guidance:

• If a column has a very high missing rate (e.g., >50–80%) consider dropping it unless it's critical.
• Mean/median imputation is simple but reduces variance and can bias relationships. Use median for skewed distributions and mean for symmetric numeric data.
• Forward/backward fill suits time series where previous/next value is a reasonable estimate.
• Model-based imputation or multiple imputation is better when preserving correlations between variables is important.

Short pandas example:

import pandas as pd

# detect
missing_counts = df.isnull().sum()

# drop rows with any missing value
df_drop = df.dropna()

# fill numeric column with median
df['marks'] = df['marks'].fillna(df['marks'].median())

# forward fill time series
df['temperature'] = df['temperature'].ffill()

# interpolate numeric values
df['sensor'] = df['sensor'].interpolate(method='linear')

# flag missing
df['marks_missing'] = df['marks'].isnull().astype(int)

Always compare distributions before and after imputation and consider domain knowledge when choosing a method.

What is Data Cleaning and Preparation?

Data cleaning and preparation is the process of detecting, correcting, or removing errors and inconsistencies in data so it is accurate, complete, and ready for analysis or machine learning. In Pandas (Python), this typically means using functions to handle missing values, convert types, standardize formats, remove duplicates, detect outliers, and encode categorical data.

Why it matters

• Bad data gives misleading statistics and poor model performance.
• Preparation ensures reproducibility and correct downstream analysis.

Common steps in Pandas

• Inspect data: use df.head(), df.info(), df.describe(), df.isna().sum() to find missing values, wrong types and summary stats.
• Handle missing values: df.dropna(), df.fillna(), forward/backward fill (ffill/bfill), or impute with mean/median/mode depending on context.
• Remove duplicates: df.drop_duplicates() and find duplicates with df.duplicated().
• Fix data types: convert with df['col'].astype(), parse dates with pd.to_datetime().
• Clean text and categories: standardize strings (str.lower(), str.strip()), replace inconsistent labels (df.replace()), convert to categorical pd.Categorical or use pd.get_dummies() for one-hot encoding.
• Detect and treat outliers: visualize with boxplots, compute IQR or z-scores and decide to cap, remove, or transform outliers.
• Scale/Normalize: apply min-max normalization or standardization before many machine learning algorithms.
• Validate and save: check ranges, constraints and export cleaned data with df.to_csv() or df.to_pickle().

Practical tips

• Always make a copy before destructive operations: df_clean = df.copy().
• Decide handling strategy by column type: numeric, categorical, text, datetime.
• Keep a log of cleaning steps (or a notebook) so results are reproducible.
• Use visualization to guide decisions: histograms, boxplots and missingness heatmaps reveal issues quickly.

Example Pandas methods (quick reference)

• Missing values: df.isna(), df.dropna(), df.fillna(value)
• Duplicates: df.duplicated(), df.drop_duplicates()
• Type conversions: df['col'].astype('int'), pd.to_datetime(df['date'])
• String ops: df['name'].str.lower(), df['col'].str.replace()
• Categorical & encoding: df['cat'] = df['cat'].astype('category'), pd.get_dummies(df, columns=['cat'])
• Outliers: compute quartiles with df['x'].quantile(), z-scores with (x - x.mean())/x.std()

When to remove vs impute

• Remove rows if missingness is rare and random and dropping won’t bias results.
• Impute (mean/median/mode or model-based) when dropping would lose important data or when patterns exist.

Final checks

• Check summary statistics (df.describe()) make sense after cleaning.
• Confirm data types, unique values for categorical fields, and no unintended NaNs remain.

Overview

Grouping and aggregation in pandas is the process of splitting data into groups (based on values of one or more columns), applying summary computations to each group, and combining the results. This follows the "split-apply-combine" pattern:

• Split: separate rows into groups using df.groupby(...)
• Apply: compute an aggregation or transformation for each group (sum, mean, count, etc.)
• Combine: return the aggregated results as a new DataFrame or Series

Key object: df.groupby('col') returns a GroupBy object. You then call aggregation methods like sum(), mean(), agg(), or use transform()/filter()/apply().

Common operations

• Single-column grouping: df.groupby('Class')['Marks'].mean() — average marks by class.
• Multi-column grouping: df.groupby(['Store','Product'])['Sales'].sum() — sales summed per store-product pair.
• Multiple aggregations: df.groupby('Store')['Sales'].agg(['sum','mean','count']) — several statistics at once.
• Different aggregations per column: df.groupby('Class').agg({'Marks':'mean','Age':'max'})
• as_index: df.groupby('col', as_index=False).sum() keeps group labels as columns instead of index.
• reset_index: after grouping results often have group labels as index; use .reset_index() to convert them back to columns.

Transform vs Aggregate vs Filter

• agg/aggregate: returns reduced output per group (one row per group or per-group statistics).
• transform: returns an object indexed like the original; used to produce a column of group-wise values (e.g., group mean for each row). Example: df['dev'] = df['Marks'] - df.groupby('Class')['Marks'].transform('mean').
• filter: keeps or drops entire groups based on a group-wise test. Example: keep groups with size > 10: df.groupby('Dept').filter(lambda g: len(g) > 10).

Pivot tables and cross-tabulations

• pd.pivot_table(df, index='Date', columns='Product', values='Sales', aggfunc='sum') aggregates values into a matrix indexed by one column and with columns from another.
• pd.crosstab(df['A'], df['B']) counts occurrences (useful for frequency tables).
• value_counts() gives counts for a single column (like a quick group-by count).

Handling missing values and sorting

• Many aggregation functions ignore NaN by default (e.g., mean, sum). Use dropna or fillna on original data as needed.
• Sort grouped results with .sort_values(). For groupby order, use sort=False in groupby if you want to preserve original order.

Simple code examples

import pandas as pd
# example DataFrame
df = pd.DataFrame({'Student':['A','B','C','D','E'],
                   'Class':['X','X','Y','Y','X'],
                   'Marks':[78,85,90,62,70]})
# mean marks by class
df.groupby('Class')['Marks'].mean()
# multiple aggregations
df.groupby('Class')['Marks'].agg(['mean','max','min','count'])
# add column with deviation from class mean
df['Deviation'] = df['Marks'] - df.groupby('Class')['Marks'].transform('mean')

Tips for students

• Remember the difference between operations that reduce groups (agg) and those that return row-aligned outputs (transform).
• Use as_index=False or reset_index() to get group keys back as columns for easier viewing or exporting.
• Practice common aggregations (sum, mean, count, min, max, median, std) on small datasets to build intuition.

What are Pivot Tables and Crosstab?

Pivot tables and crosstab are tools to summarize and reshape tabular data by aggregating values along two or more categorical dimensions. In pandas, pivot_table and pd.crosstab produce compact summary tables that help answer questions like "total sales by region and product" or "count of students by class and grade".

pivot_table (pandas.DataFrame.pivot_table)

• Purpose: Aggregate numeric data using one or more aggregation functions (sum, mean, count etc.) across specified index and column categories.
• Key parameters: index (rows), columns (columns), values (numeric column(s) to aggregate), aggfunc (e.g., 'sum', 'mean', 'count', numpy functions), margins (include totals), fill_value (replace NaN), dropna.
• When to use: When you want aggregated numeric summaries (sum, average, count, etc.) laid out in a matrix form.

pd.crosstab

• Purpose: Build a contingency table (frequency table) that counts occurrences of combinations of categorical variables. It can also aggregate values using values and aggfunc.
• Key parameters: first and second arrays/series (rows and columns), normalize (proportions by 'index', 'columns', or 'all'), margins, values and aggfunc for aggregated metrics rather than plain counts.
• When to use: When you need counts or proportions of categorical combinations (e.g., survey responses by gender and age group).

Differences and relations

• pivot vs pivot_table: pivot requires a unique index/column combination (no aggregation). pivot_table allows aggregation when combinations are not unique.
• pivot_table is similar to groupby(...).unstack(): groupby produces grouped aggregates and unstack converts one grouping level into columns—same result as many pivot tables.
• crosstab is a convenient wrapper for frequency tables; you can extend it with values and aggfunc to compute sums or means instead of counts.

Handling missing values & totals

• Use fill_value to replace missing cells (NaN) after aggregation.
• Use margins=True to get row/column totals (added as 'All').
• Use dropna=False if you want categories with all-NaN columns preserved.

Multi-index and multiple aggregations

• You can pass lists to index and columns for multi-level pivot tables.
• Pass a list of functions to aggfunc or a dict mapping columns to aggfuncs to compute multiple summaries (e.g., sum and mean).

Small code templates (pandas)

import pandas as pd
import numpy as np

# pivot_table template
pd.pivot_table(df, index=['row_cat'], columns=['col_cat'], values='num_col', aggfunc='sum', margins=True, fill_value=0)

# crosstab template (counts)
pd.crosstab(df['cat1'], df['cat2'], margins=True)

# crosstab with aggregation
pd.crosstab(df['cat1'], df['cat2'], values=df['num_col'], aggfunc=np.mean, margins=True)

Practical tips for students

• Start with a subset of columns to avoid huge pivot tables.
• Sort or filter the dataframe before pivoting if you want specific order or categories.
• Use reset_index() on a pivot result to convert MultiIndex rows into columns for easier plotting or export.

Overview

Reshaping transforms a DataFrame between wide and long formats. Pandas provides melt, stack and unstack to perform these transformations. Understanding them is essential for cleaning data and preparing it for analysis or visualization.

1. pd.melt (wide -> long)

melt collapses multiple columns into two columns: one for variable names and one for values. Use it when you want to convert a wide table (many value-columns) into a tidy long table where each row is one observation.

# pattern
pd.melt(df, id_vars=[...], value_vars=[...], var_name='variable', value_name='value')

Parameters:

• id_vars: columns to keep as identifier columns (not melted)
• value_vars: columns to melt (if omitted, all non-id_vars are used)
• var_name and value_name: names for the new columns

Example explanation (melt)

Wide table: student exam scores by term (columns Term1, Term2). Use melt to produce one row per student-term-score.

df =
  Student  Term1  Term2
0   A       80     85
1   B       75     82

pd.melt(df, id_vars=['Student'], var_name='Term', value_name='Score')
# Result:
#  Student  Term   Score
#  A        Term1  80
#  A        Term2  85
#  B        Term1  75
#  B        Term2  82

2. stack and unstack (index <-> column level pivoting)

stack and unstack operate on MultiIndex objects (index or columns). They pivot a level of the columns into the row index (stack) or a level of the row index into columns (unstack).

• df.stack(level=-1): moves the specified column level to the innermost row index, returning a Series (or DataFrame if multiple levels stacked).
• df.unstack(level=-1): pivots the specified index level into columns, turning a Series with MultiIndex into a DataFrame.

Example explanation (stack/unstack)

# Suppose df has MultiIndex columns: ('Sales','Jan'), ('Sales','Feb'), ('Cost','Jan'), ('Cost','Feb')
# stack the outer column level -> inner index level
stacked = df.stack(level=0)
# unstack the last index level to columns
unstacked = stacked.unstack(level='Month')

Common workflow: set_index to create a MultiIndex (e.g., by ['Region','Product']), then unstack a level (e.g., 'Product') to get products as columns and regions as rows, or stack to revert to long form.

Key differences and when to use

• Use melt to convert wide column-based values into long rows when working with simple column names.
• Use stack/unstack when your DataFrame uses MultiIndex on rows or columns and you want to shift an index level between rows and columns.
• melt is column-oriented; stack is index/column-level oriented and preserves MultiIndex structure.

Practical tips

• After melt, you may want to dropna() or convert types (e.g., to numeric) with pd.to_numeric.
• Use reset_index() or set_index() to move columns into or out of the index when preparing for stack/unstack.
• unstack can create NaNs if some combinations are missing; use fill_value with pivot_table if needed.

Inverse operations

melt is often inverted by pivot or pivot_table. Similarly, stack is inverted by unstack (and vice versa) when levels line up.

Overview: Sorting arranges data in a specified order (ascending/descending) based on one or more columns or the index. Ranking assigns ordinal positions to values (1st, 2nd, ...) and handles ties according to a chosen method. In pandas these operations are essential for data cleaning, analysis and reporting (leaderboards, top-k selection, percentile calculations).

Sorting in pandas

Common methods:

df.sort_values(by='col', ascending=True, inplace=False, na_position='last')

df.sort_values(by=['col1','col2'], ascending=[False, True])  # multi-column sort

df.sort_index(ascending=True)  # sort by index

Options:

• by: column name or list of column names.
• ascending: bool or list of bools (one per column).
• inplace: modify original DataFrame if True.
• na_position: 'first' or 'last' for NaN placement.

Quick selection of top/bottom elements:

df.nlargest(n, 'sales')
df.nsmallest(n, 'price')

Ranking in pandas

Use Series.rank (also DataFrame.rank). It assigns ranks to values; smaller values get smaller ranks by default.

series.rank(ascending=True, method='average', pct=False)

Important parameters:

• method: how to handle ties: 'average', 'min', 'max', 'first', 'dense'.
  • 'average': assign average of tied positions (e.g., tied at positions 2 and 3 => rank 2.5).
  • 'min': all tied values get lowest position (2 in example).
  • 'max': all tied values get highest position (3 in example).
  • 'first': ranks assigned in order they appear in the data (no averaging).
  • 'dense': like 'min' but next distinct value gets next integer (no gaps).
• ascending: False to rank high values as rank 1 (useful for leaderboards).
• pct: if True returns fractional rank = rank / count_non_na (percentage-based rank).

Example: compute descending ranks of scores and convert to percentile:

df['rank'] = df['score'].rank(ascending=False, method='min')
df['pct_rank'] = df['score'].rank(ascending=False, pct=True)  # rank / n

Tie handling & choices: choose method based on context. For competitions where ties share places, use 'min' or 'max'. For continuous scoring where you want fractional positions, use 'average'. For strict ordering by appearance, use 'first'.

Practical tips:

• Chain sorting and selecting top-n: df.sort_values('sales', ascending=False).head(10) or df.nlargest(10,'sales').
• For grouped ranking (rank within groups): use groupby + rank: df['grp_rank'] = df.groupby('group')['value'].rank(ascending=False, method='dense').
• Rank across columns (rows): df.rank(axis=1) ranks values across columns for each row.

When to use sort vs rank: sort when you want ordered rows (presentation or slicing), rank when you need ordinal labels, percentiles, or positional metrics to feed into further calculation.

What is Hierarchical (Multi-)Indexing?

Hierarchical or multi-indexing in pandas allows a DataFrame or Series to have more than one index level on rows and/or columns. Instead of a single label per row, each row can be identified by a tuple of labels (level1, level2, ...). This lets you represent and operate on higher-dimensional data in a 2D table efficiently.

Why use it?

• Simplifies representation of grouped or panel data (e.g., city → store → product).
• Makes aggregation, slicing and reshaping across multiple keys easy and expressive.
• Reduces need for repeated columns and helps produce tidy, compact tables.

How to create a MultiIndex

• From columns: df.set_index(['Level1', 'Level2'])
• From arrays or tuples: pd.MultiIndex.from_tuples([('A',1),('A',2),('B',1)])
• During pivoting: df.pivot_table(...)

Basic operations and concepts

• Levels: each position in the tuple is a level (named or unnamed).
• Labels: member values in each level (e.g., city names).
• Accessing rows: use a tuple in .loc or chained .loc if levels are named.
• Slicing: IndexSlice helps with partial selection across levels.
• Stack / unstack: transform between row-level and column-level hierarchies.
• xs (cross-section): extract data at a particular level value quickly.
• swaplevel / sort_index: reorder levels to suit operations or display.

Short code examples

# create a DataFrame with a multi-index from columns
sales = pd.DataFrame({
  'City': ['Mumbai','Mumbai','Delhi','Delhi'],
  'Store': ['S1','S2','S1','S2'],
  'Sales': [100, 150, 80, 120]
})
sales = sales.set_index(['City','Store'])
# now sales.index is a MultiIndex with two levels

# selection examples
sales.loc[('Mumbai','S2')]          # select single row by full tuple
sales.loc['Mumbai']                # selects all stores in Mumbai
sales.xs('S1', level='Store')      # cross-section by store across all cities

# reshape
sales_unstacked = sales.unstack(level='Store')  # stores become columns

Tips for reading and writing

• When printing, pandas shows levels stacked vertically for readability.
• Name your levels (df.index.names = ['City','Store']) for clearer code.
• Keep indexes sorted (df.sort_index()) for fast, reliable slicing.

Example use-cases in real life

• Retail: sales by (Region, Store, Product) to analyze sales across hierarchies.
• Education: marks by (Class, Student, Subject) to compute averages per class and subject.
• Sensor networks: readings by (Date, SensorID) for time-series per sensor.
• Finance: stock prices with (Ticker, Date) or (Sector, Company, Quarter).

Hierarchical indexing is a powerful feature in pandas that helps Class 11 students organize, slice, aggregate and visualize grouped data with clarity and efficiency.

What is .str?
In pandas, .str is an accessor that provides vectorized string methods for Series (and Index). It allows you to perform string operations on all elements of a column efficiently (without explicit Python loops).

Why use .str?
It is fast, concise, and handles missing values (NaN) safely. Many methods accept regular expressions (regex) to match patterns and extract parts of strings.

Common categories of operations

• Case and trimming: str.lower(), str.upper(), str.title(), str.strip() (remove whitespace or specific chars).
• Searching and matching: str.contains(), str.startswith(), str.endswith(), str.match(), str.find().
• Splitting and extracting: str.split(), str.rsplit(), str.get(), str.extract(), str.findall().
• Replacing and cleaning: str.replace(), often with regex to remove punctuation or standardize formats.
• Length and slicing: str.len(), str.slice(), str[:] (slice using str.slice or str.get).
• Concatenation and joining: str.cat() to join strings from multiple columns or a list.

Notes on regex: Many .str methods have a regex parameter or accept regular expressions (e.g., extract, replace, contains). Use raw Python strings for patterns: r'\d{3}'.

Worked example (small DataFrame)

import pandas as pd
df = pd.DataFrame({
  'full_name': ['  Alice Brown', 'bob smith ', 'CHARLIE DAVIS', None],
  'email': ['alice@school.edu', 'bob99@gmail.com', 'charlie@org.gov', 'na']
})

# Trim and standardize case
df['name_clean'] = df['full_name'].str.strip().str.title()

# Extract username and domain from email
df[['username','domain']] = df['email'].str.split('@', expand=True)

# Check rows with gmail
df['is_gmail'] = df['email'].str.contains('gmail', na=False)

# Get length of username
df['user_len'] = df['username'].str.len()

Handling missing values: By default .str methods return NaN if the element is missing. Use na='' or fillna to substitute if needed (some methods accept na= argument, e.g. str.contains('x', na=False)).

Performance tip: .str is vectorized and much faster than iterating row-by-row. For very large text processing (NLP) consider specialized libraries, but for typical table cleaning .str is ideal.

Class 11 learning points (what students must practice): trimming and case normalization, splitting full name into first/last, extracting domain from email, validating phone formats (using regex), counting words/characters, filtering rows by substring.

Overview
Date and time handling is essential when working with time-stamped data (logs, sales by day, sensor readings, attendance). In Pandas, date/time values are represented using the datetime64[ns] dtype and Pandas/Timestamp/Timedelta/Period objects which allow fast vectorized operations, indexing, resampling and time-based grouping.

Parsing and converting
Use pd.to_datetime() to convert strings or numeric timestamps into datetimes. Important parameters: format (specify input format), errors='coerce' (invalid -> NaT), unit (e.g. 's' for seconds), and infer_datetime_format=True (faster when formats are consistent).

# example
df['date'] = pd.to_datetime(df['date_str'], format='%d-%m-%Y', errors='coerce')

DatetimeIndex and indexing
Setting a datetime column as the index (df.set_index('date')) creates a DatetimeIndex. This enables fast slicing by date ranges, e.g. df['2021-01'] or df['2021-01-01':'2021-01-31'].

Accessing components
Use the dt accessor to get parts: df['date'].dt.year, .dt.month, .dt.day, .dt.hour, .dt.weekday, .dt.day_name(), .dt.time. Also methods like .dt.normalize() (set time to midnight), .dt.floor(), .dt.ceil(), .dt.round().

Timedelta and arithmetic
Differences between datetimes produce Timedelta objects. You can add/subtract timedeltas, compute durations, and convert to units: td.days, td.total_seconds(). Create timedeltas with pd.to_timedelta() or pd.Timedelta().

Resampling and frequency conversion
Use resample() (requires DatetimeIndex) to change frequency and aggregate: df.resample('M').sum() for monthly totals. For grouping without index, use pd.Grouper(key='date', freq='M') in groupby. Common freq aliases: 'D' (daily), 'W' (weekly), 'M' (month-end), 'MS' (month-start), 'H' (hourly).

Periods and time spans
Use Period for closed intervals (e.g., month periods) and period_range() to build period indexes. Useful for handling business quarters, months, or custom intervals.

Time zones
Localize naive timestamps with tz_localize() and convert between time zones with tz_convert(). Be careful when localizing: df['date'] = df['date'].dt.tz_localize('UTC').dt.tz_convert('Asia/Kolkata').

Missing values and parsing errors
Invalid date strings become NaT. Handle with fillna(), interpolate() (for numeric series indexed by time), or drop with dropna(). Use errors='coerce' to avoid exceptions when parsing.

Performance tips
Keep date columns as datetime64[ns], avoid repeated string parsing, and prefer DatetimeIndex for time-series ops. Use vectorized dt methods instead of Python loops.

Common workflow summary
1) Parse strings to datetimes (pd.to_datetime). 2) Set as index for time series ops (set_index). 3) Extract components with .dt. 4) Resample/group/roll as needed. 5) Visualize trends and seasonality.

What are categorical data? Categorical data are variables that take values from a limited set of discrete labels (categories). Examples: gender ('Male', 'Female'), blood group ('A', 'B', 'AB', 'O'), sizes ('S', 'M', 'L'), grades ('A', 'B', 'C'). They are either nominal (no natural order) or ordinal (have a meaningful order).

Why treat them specially? In memory and computation, repeating strings are costly. Many operations (grouping, counting, plotting) are faster and more meaningful if categories are stored as integer codes with a small lookup of category names. Also ordinality matters for comparisons and sorting.

Pandas support: Pandas provides a categorical dtype and helper functions to convert and factorize categorical data. Main tools:

• Series.astype('category') — converts a column to categorical dtype.
• pd.Categorical(values) — constructs a categorical object (can specify order and categories).
• Series.cat.codes — shows the integer codes that represent categories.
• pd.factorize(series) — returns a tuple (codes, uniques) to map each value to an integer code and the unique category labels.
• Series.cat.reorder_categories([...], ordered=True) or CategoricalDtype(categories=[...], ordered=True) — to set an order for ordinal categories.

How factorization works (conceptually): choose a list of unique categories (the vocabulary) and replace each value by the index of that category. For example, if uniques = ['Red','Green','Blue'] then codes = [0,1,2,...]. This reduces memory and speeds up group operations.

Advantages:

• Lower memory usage: repeated strings stored once (metadata) + small integer array for values.
• Faster grouping, sorting, comparisons and joins.
• Explicit ordering for ordinal data allowing correct comparisons and plotting.

Typical workflow (pandas):

# convert column to category
df['grade'] = df['grade'].astype('category')
# view category labels and codes
df['grade'].cat.categories
df['grade'].cat.codes
# factorize a series directly
codes, uniques = pd.factorize(df['city'])

Notes on missing values: Missing values (NaN) are allowed; codes for NaN are often set to -1 when using Series.cat.codes or pd.factorize, so handle them appropriately before modelling or plotting.

Overview: In pandas, apply, map and applymap let you run a function over Series or DataFrame values to transform, clean or aggregate data. They let you use Python functions (including lambdas) to perform custom operations that vectorized methods don’t cover.

map: Works on a Series only. It maps each element to a new value using either a function, a dict/Series (for lookup), or a scalar replacement. Use map when you want elementwise substitution or lookup of categorical values.

# Example: map with a dict
s = pd.Series(['A','B','A','C'])
s.map({'A':'Apple','B':'Banana','C':'Cherry'})

apply (Series): Series.apply(func) applies a function to each element of the Series (elementwise), similar to map but more general (works with functions that return scalars or more complex objects).

# Example: apply on a Series
s = pd.Series([10,20,30])
s.apply(lambda x: x/10)  # returns 1.0,2.0,3.0

apply (DataFrame): DataFrame.apply(func, axis=0 or 1) applies func along rows (axis=1) or columns (axis=0). The function receives a Series (a row or a column) and should return either a scalar or a Sequence. Useful for row/column aggregates or deriving new columns from multiple columns.

# Example: apply along rows
df = pd.DataFrame({'eng':[80,70],'math':[90,85]})
df['total'] = df.apply(lambda row: row['eng'] + row['math'], axis=1)

applymap: DataFrame.applymap(func) applies func elementwise to every single cell of the DataFrame. Use it when you need to transform every value independently (e.g., format numbers, convert units).

# Example: applymap to convert units
df.applymap(lambda x: x*0.001)  # if all numeric and you want to scale

Key differences (quick):

• map: Series only, elementwise, accepts dict/Series/function.
• Series.apply: Series only, elementwise, more flexible than map.
• DataFrame.apply: Works on rows or columns (aggregations or row-wise computations).
• DataFrame.applymap: Elementwise on entire DataFrame (every cell).

Performance note: Vectorized pandas/numpy operations are fastest. apply/applymap/map are slower because they loop in Python. Use them when vectorized alternatives are not available.

Common patterns:

• Cleaning strings: df['name'] = df['name'].str.strip().map(lambda s: s.title())
• Category lookup: df['state_name'] = df['state_code'].map(code_to_name_dict)
• Row calculations: df['grade'] = df.apply(lambda r: compute_grade(r['marks1'], r['marks2']), axis=1)
• Elementwise numeric transform: df = df.applymap(lambda x: round(x,2))

What are window / rolling operations?
Window (or rolling) operations compute summary statistics over a moving window of consecutive rows in a series or DataFrame. Instead of a single global aggregation (like mean of entire column), a rolling operation returns a result for each position using values from a local neighborhood (the window).

Key concepts
- Window size (n): number of consecutive observations used for each calculation.
- Centered vs right-aligned: a centered window takes values before and after the current index; right-aligned (default) uses current and previous values.
- min_periods: minimum non-NA observations in window required to produce a value (prevents NaN outputs at start).
- Types: rolling (fixed-size), expanding (grows from start to current index), ewm/exponentially-weighted (gives more weight to recent values).

Pandas usage (simple examples)

import pandas as pd
# rolling mean with window 3
df['SMA_3'] = df['value'].rolling(window=3, min_periods=1).mean()
# rolling sum
df['roll_sum_7'] = df['rainfall'].rolling(window=7).sum()
# expanding (cumulative) mean
df['cum_mean'] = df['sales'].expanding(min_periods=1).mean()
# exponential moving average (EMA)
df['EMA_10'] = df['price'].ewm(span=10, adjust=False).mean()

Why useful (intuition)
Rolling windows smooth noisy data, reveal short-term trends, detect local peaks/valleys, compute local variability (std), and find correlations that vary over time. Expanding windows provide cumulative summary (like running total). Exponentially-weighted methods adapt faster to recent changes by giving higher weights to recent observations.

Practical notes
- Rolling results often produce NaNs at the start if min_periods < window size is not set.
- For time-series with irregular timestamps use rolling on a time window: df.rolling('7D', on='date').mean().
- Use .shift() together with rolling to avoid look-ahead bias (e.g., when computing features for modeling).

Overview: Basic visualization with pandas means using pandas' built-in plotting methods (which use matplotlib under the hood) to quickly turn Series and DataFrame data into charts. Visuals help summarize patterns, trends, distributions and relationships in tabular data.

Requirements

• import pandas as pd
• import matplotlib.pyplot as plt
• Data in a Series or DataFrame (columns numeric or categorical as appropriate)

Common plotting methods

• Series.plot(kind='line') or DataFrame.plot() — default: line plot for trends over index or time.
• plot(kind='bar') / plot(kind='barh') — bar charts for categorical comparisons.
• plot(kind='hist') or df.hist() — histograms for distribution of a numeric column.
• plot(kind='box') or df.boxplot() — boxplots for spread and outliers.
• plot(kind='scatter', x='col1', y='col2') — scatter for relationships between two numeric variables.
• plot(kind='pie') — pie chart for parts of a whole (single Series).
• plot(kind='kde') — kernel density estimate for smoothed distributions.

Typical arguments and customization

• kind: type of plot.
• x, y: column names for axes.
• title='...' , xlabel='...' , ylabel='...'
• color, style, legend, figsize=(w,h), grid=True
• subplots=True to draw each column in separate axes; layout=(r,c) to arrange them.
• Use plt.savefig('file.png') to save the figure.

How to build a simple plot (example)

import pandas as pd
import matplotlib.pyplot as plt

df = pd.read_csv('sales.csv')
df.groupby('month')['revenue'].sum().plot(kind='line', figsize=(8,4), title='Monthly Revenue')
plt.xlabel('Month')
plt.ylabel('Revenue')
plt.grid(True)
plt.show()

Working with grouped data and counts

Use df.groupby(...).sum(), .mean() or value_counts() as input to plotting. Example: df['region'].value_counts().plot(kind='bar').

Handling missing values

• Either drop (dropna()) or fill (fillna()) before plotting to avoid misleading charts.

Interpretation tips

• Choose chart type to match the question: trends (line), comparisons (bar), composition (pie), distribution (hist/box), relationships (scatter).
• Check summary statistics (mean, median, std) to complement visuals.

Basic visualization with pandas provides quick, readable charts directly from DataFrame and Series objects and is ideal for exploratory data analysis in Class 11 level projects.

Exporting and reporting are the final steps of a data-analysis workflow: after cleaning and analysing data with pandas, you save (export) results and create human-friendly reports so others can read, reuse or visualise the findings. Common targets are files (CSV, Excel, JSON, HTML), databases (SQL), binary serialisations (pickle, parquet) and visual files (PNG, SVG). Reporting includes summary tables, charts and formatted HTML/Excel reports.

Key pandas methods for exporting:

• DataFrame.to_csv() — save to comma/other‑delimited files.
• DataFrame.to_excel() / pd.ExcelWriter — write one or multiple sheets to Excel workbooks.
• DataFrame.to_json() — export to JSON for APIs or web apps.
• DataFrame.to_html() — create HTML tables for web reports or emails.
• DataFrame.to_sql() — write to SQL databases (needs SQLAlchemy/DB connector).
• DataFrame.to_parquet(), to_pickle() — efficient binary formats for large datasets.
• Save plots from matplotlib/seaborn using plt.savefig().

Important options and considerations:

• index (True/False) — whether to write the DataFrame index as a column.
• columns — list to restrict exported columns.
• encoding (e.g. 'utf-8') — ensure correct character encoding for non-ASCII text.
• sep — delimiter for CSV (comma, tab '\t' for TSV).
• compression (e.g. 'gzip') — reduce file size for large exports.
• chunksize — write/read large files in pieces to avoid memory issues.
• Excel: use ExcelWriter for multiple sheets and formatting; consider engine (openpyxl, xlsxwriter).
• Database export: use correct dtype mapping and transactions; avoid exporting sensitive data without anonymisation.

Reporting practices:

• Create summary tables with df.describe(), groupby().agg() or pivot_table(), then export these summaries.
• Generate charts (bar, line, histogram, boxplot, heatmap) to visualise main findings and save images with savefig.
• Combine tables and charts in Excel sheets or HTML dashboards; libraries like pandas styling, Jupyter notebooks or profiling tools (ydata‑profiling) can help create richer reports.
• Include metadata (generation date, filtration steps, source) so results are reproducible.

Best practices: set index=False for tidy CSVs, use UTF‑8 encoding, compress large files, export only required columns, anonymise sensitive fields, and include a README or header with context. Test exported files by reloading them to ensure correctness.

Overview: This topic shows how to apply pandas to real datasets: load and inspect data, clean and transform it, compute summary statistics, aggregate/group data, join datasets, and create visualisations to draw insights. Typical workflow: load & inspect → clean → transform → analyse (aggregate/summary/stats) → visualise → save results.

Common steps with short code examples:

1. Load and inspect: Use pd.read_csv to read files, then df.head(), df.info(), df.describe() to inspect.

   import pandas as pd
       df = pd.read_csv('data.csv')
       print(df.head())
       print(df.info())
       print(df.describe())

2. Clean missing or wrong data: identify with df.isna().sum(), remove or impute with df.dropna() or df.fillna(value).

   # drop rows with any missing values
       df_clean = df.dropna()
       # fill numerical missing values with column mean
       df['score'] = df['score'].fillna(df['score'].mean())

3. Filter & transform: use boolean masks, loc, apply or vectorised operations.

   # select students with marks >= 60
       high = df.loc[df['marks'] >= 60]
       # create grade column
       df['grade'] = df['marks'].apply(lambda x: 'A' if x >= 75 else ('B' if x >= 60 else 'C'))

4. Aggregate & group: group by categories and apply aggregations.

   grouped = df.groupby('class').agg({'marks': ['mean', 'median', 'max']})
       pivot = df.pivot_table(index='month', columns='product', values='sales', aggfunc='sum')

5. Merge & reshape: combine datasets with merge or concat.

   df_all = pd.merge(transactions, customers, on='customer_id', how='left')

6. Visualise: use pandas plotting (matplotlib) or seaborn to create histograms, boxplots, scatter plots, bar charts and heatmaps to explore distributions and relationships.
7. Save results: df.to_csv('cleaned.csv', index=False).

Pedagogical tips for exercises: use small real datasets (students marks, sales by month, weather readings). Ask students to (a) state objective; (b) write code to load & inspect; (c) clean; (d) perform at least two aggregations; (e) plot results and interpret; (f) export summary.