Chapter 1 — Data Handling Using Pandas I

What is pandas? pandas is a popular open-source Python library for data manipulation and analysis. It provides fast, flexible, and expressive data structures designed to work with structured (tabular, time series) data easily and intuitively.

Why use pandas? pandas makes tasks like reading/writing data, cleaning, transforming, aggregating, and visualising datasets simple with concise commands. It is widely used in data science, finance, research, and many real-life applications.

Core data structures

• Series — a one-dimensional labeled array (like a column). Each element has an index and a value.
• DataFrame — a two-dimensional table with labeled rows and columns (like a spreadsheet or SQL table). Each column is a Series.

Common workflow

• Input/Output: read data from CSV/Excel/SQL using functions such as pd.read_csv and write using df.to_csv.
• Inspect: use df.head(), df.info(), df.describe() to understand structure and summary statistics.
• Select & filter: access columns with df['col'] or df.col, select rows with loc and iloc, and filter using boolean conditions.
• Clean: handle missing values (df.isnull(), df.dropna(), df.fillna()), change datatypes, remove duplicates.
• Transform: add/remove columns, apply functions, create new features.
• Aggregate & group: use df.groupby(...) with aggregation functions (sum, mean, count) to summarise data.
• Combine & reshape: merge/join tables (pd.merge), concatenate (pd.concat), reshape with pivot_table and melt.
• Visualise: quick plots integrated with matplotlib (df.plot(), df.hist(), etc.) for exploratory analysis.

Short code examples

import pandas as pd
# load data
df = pd.read_csv('students.csv')
# inspect
df.head()
# select
marks = df['marks']
# filter
high = df[df['marks'] >= 75]
# group and aggregate
avg_by_class = df.groupby('class')['marks'].mean()
# fill missing values
df['age'] = df['age'].fillna(df['age'].median())

Tips

• Always inspect df.info() to check datatypes and nulls before analysis.
• Use vectorised operations (column-level) instead of row-by-row loops for speed.
• When working with large datasets, read subsets or use dtype specification to save memory.

Where pandas is used in real life: cleaning sales data for business reports, analyzing student performance, processing sensor logs for IoT, preparing financial time series for modelling, and summarising patient records in healthcare.

What is a Series? A pandas Series is a one-dimensional labeled array capable of holding data of any single data type (integers, floats, strings, Python objects). Each element has an associated label called the index. Think of a Series as a column in a spreadsheet or a dictionary that preserves order and supports vectorized operations.

Creation — common constructors:

• pd.Series([10, 20, 30]) — from list (default integer index)
• pd.Series({'a': 10, 'b': 20}) — from dict (keys become index)
• pd.Series(5, index=['x','y','z']) — scalar repeated for all index labels

Key attributes & properties:

• .index — index labels
• .values — underlying numpy array
• .dtype — data type
• .name — optional name for the Series

Indexing & selection: label-based and integer-location based access are supported. Examples: s['a'], s[0], s[['a','b']], s[1:4]. Boolean masks and loc/iloc are also used: s.loc['a'], s.iloc[0].

Vectorized arithmetic & alignment: arithmetic operations are elementwise and align on index labels. If indices differ, result contains the union of indices and missing values (NaN) where labels are absent. Example: s1 + s2.

Missing values: missing data are represented as NaN. Methods: isnull(), notnull(), dropna(), fillna(value).

Common statistical & utility methods: sum(), mean(), median(), std(), var(), min(), max(), count(), unique(), value_counts(), sort_values(), sort_index(), map(), apply(), astype().

When to use a Series: best for any one-dimensional labeled data: time series, single column of a dataset, sensor readings, prices, counts and categorical frequency vectors. A Series is often the building block for DataFrame columns.

What is a DataFrame?

A DataFrame is a 2-dimensional, tabular data structure provided by the pandas library in Python. It is similar to a spreadsheet or SQL table: data is organized in rows and columns. Each column can have a different data type (integer, float, string, datetime, etc.).

Structure and important concepts

• Rows: represent observations or records (indexed).
• Columns: named series representing variables/attributes.
• Index: labels for rows (can be default 0..n-1 or custom labels).
• dtypes: data type of each column (df.dtypes).

Why use DataFrame? It provides powerful, easy-to-use operations for data cleaning, transformation, filtering, aggregation and export. It integrates with plotting libraries and data sources (CSV, Excel, database).

Common creation methods

• From a Python dictionary of equal-length lists: pd.DataFrame({'A':[1,2], 'B':[3,4]})
• From a list of dictionaries (each dict = row): pd.DataFrame([{'name':'A','age':20}, {'name':'B','age':19}])
• From a list of lists with column names: pd.DataFrame([[1,2],[3,4]], columns=['c1','c2'])
• From pandas Series or NumPy arrays: pd.DataFrame({'col': pd.Series([..])})
• From files: pd.read_csv('file.csv'), pd.read_excel('file.xlsx')

Key attributes and basic methods

• df.shape — (rows, columns)
• df.columns — list of column names
• df.index — row labels
• df.dtypes — column data types
• df.head(n), df.tail(n) — preview top/bottom rows
• df.info() — summary including non-null counts and dtypes
• df.describe() — summary statistics for numeric columns

Selection and filtering

• Column access: df['col'] or df[['c1','c2']]
• Label based: df.loc[row_label, col_label]
• Position based: df.iloc[row_index, col_index]
• Boolean filtering: df[df['age'] > 18]
• Add/modify column: df['new'] = ...
• Drop column/row: df.drop('col', axis=1)

Aggregation, grouping and joins

• df.groupby('group_col')['value_col'].mean() — group and aggregate
• pd.concat([df1, df2]) — stack DataFrames
• pd.merge(df1, df2, on='key') — SQL-like join

Best practices when creating DataFrames

• Prefer structured input (list of dicts or dict of lists) for readable column names.
• Always check df.info() to confirm dtypes and missing data.
• Set a meaningful index where appropriate (e.g., student_id, date).

Small examples (readable Python snippets)

# From dict of lists
data = {'Name': ['Ali','Meera'], 'Marks': [85, 92]}
df = pd.DataFrame(data)

# From list of dicts (each dict = row)
rows = [{'Date':'2025-01-01','Sales':100}, {'Date':'2025-01-02','Sales':120}]
df2 = pd.DataFrame(rows)

# Read from CSV
# df_csv = pd.read_csv('class12_marks.csv')

# Select rows where Marks > 80
high = df[df['Marks'] > 80]

Summary

DataFrame is the central data structure in pandas for tabular data. Knowing how to create DataFrames from various sources and performing basic inspections, selections and aggregations is essential for data handling and analysis in Class 12 Informatics Practices.

What is IO (Input/Output)?

Input/Output (IO) in Pandas means reading data from external sources into DataFrame objects (input) and writing DataFrames back to external formats (output). Common formats: CSV, Excel, JSON, HTML, SQL databases, binary (pickle, parquet), and clipboard.

Why IO matters

Data analysis usually starts with bringing data into memory, cleaning and transforming it, and finally saving results. Efficient, correct IO ensures you read data with correct types, handle missing values, preserve indices, and write in a format that downstream tools expect.

Key read/write functions

• CSV / delimited: pd.read_csv() / DataFrame.to_csv()
• Excel: pd.read_excel() / DataFrame.to_excel()
• JSON: pd.read_json() / DataFrame.to_json()
• HTML tables: pd.read_html() / DataFrame.to_html()
• SQL: pd.read_sql() / DataFrame.to_sql() (requires a DB connection)
• Binary / efficient storage: to_pickle()/read_pickle(), to_parquet()/read_parquet()
• Clipboard (quick copy/paste): pd.read_clipboard()

Important parameters and options

• filepath_or_buffer: filename, URL or file-like object.
• sep / delimiter: column separator (',' for CSV, '\t' for TSV).
• header, names: row to use as column names or a list of names.
• index_col: column(s) to use as row labels.
• usecols: subset of columns to read (improves speed & memory).
• dtype: force column data types.
• parse_dates: columns to parse as datetime.
• na_values: strings to treat as NaN.
• nrows, skiprows: read limited rows or skip headers.
• chunksize: iterate over file in chunks for very large files.
• encoding: file encoding (e.g., 'utf-8', 'latin-1').
• compression: handle compressed files (e.g., 'gzip').

Tips & best practices

• Inspect first rows: pd.read_csv('file.csv', nrows=5) before full load.
• Use usecols and dtype to reduce memory usage.
• For very large files, process with chunksize to avoid memory overflow (aggregate per chunk).
• Always specify encoding if you see UnicodeDecodeError.
• When saving for interoperability, CSV and Excel are broadly supported; for performance use Parquet.
• Preserve date columns with parse_dates to enable time-series operations.

Example workflows

• Read student marks from a CSV, clean missing scores, compute averages, and save results to Excel for teachers.
• Fetch JSON from a REST API (weather data), convert to DataFrame, visualize trends, and store processed data in a database.
• Merge attendance across multiple Excel sheets, export the combined DataFrame to a compressed CSV for archiving.

Common pitfalls

• Wrong delimiter — columns merge into one string. Use sep properly.
• Mixed types in a column cause dtype=object; use dtype or converters to normalize.
• Dates read as strings — use parse_dates to convert.
• Large files out of memory — read in chunks or filter columns while reading.

Small code examples (illustrative)

# Read CSV with specified columns, parse date
import pandas as pd
df = pd.read_csv('students_marks.csv', usecols=['Roll','Name','Date','Marks'], parse_dates=['Date'])

# Process in chunks
reader = pd.read_csv('big_data.csv', chunksize=100000)
for chunk in reader:
    # do processing per chunk
    pass

# Write to Excel and to SQL
df.to_excel('cleaned_marks.xlsx', index=False)
# df.to_sql('marks_table', con=engine, if_exists='replace', index=False)

Using these IO tools correctly helps you reliably move data between files, databases, web APIs, and Pandas for analysis.

Overview: Viewing and summarizing data means inspecting a dataset to understand its contents, structure, types, missing values and basic statistics. In Pandas this is done with a small set of functions that let you quickly get a picture of both numerical and categorical variables before further analysis.

• Viewing rows and columns
  • Use df.head(n) and df.tail(n) to see the first/last n rows.
  • df.shape gives (rows, columns). df.columns lists column names.
  • Use selection (df['col'], df[['c1','c2']], df.loc, df.iloc) to inspect subsets.
• Structure and types
  • df.info() shows index, column dtypes and non-null counts.
  • df.dtypes lists data types (int, float, object, datetime, etc.).
• Summarizing numerical data
  • df.describe() returns count, mean, std, min, 25% (Q1), 50% (median), 75% (Q3), max for numeric columns.
  • Single-column summaries: df['marks'].mean(), .median(), .std(), .min(), .max(), .quantile(0.25).
• Summarizing categorical data
  • df['gender'].value_counts() shows frequencies; add normalize=True for proportions.
  • df.describe(include='object') shows count, unique, top, freq for categorical columns.
• Missing values and uniqueness
  • df.isnull().sum() counts missing values per column.
  • df['id'].nunique() gives number of distinct values; df['col'].unique() lists them.
• Grouped summaries
  • df.groupby('class')['marks'].mean() computes mean marks per class. Use .agg() to get multiple statistics: df.groupby('class')['marks'].agg(['count','mean','std']).
• Practical tips
  • Start with df.head() and df.info(), then df.describe() and df.isnull().sum().
  • Visual checks (histograms, boxplots, bar charts) help spot skew, outliers and category imbalances.

Common Pandas commands (examples):

# view
df.head()
# structure
df.info(); df.dtypes; df.shape
# numeric summary
df.describe()
# categorical summary
df['category'].value_counts()
# missing values
df.isnull().sum()
# group summary
df.groupby('region')['sales'].agg(['count','mean','sum'])

Overview: In Pandas, indexing, selection and slicing are the fundamental operations to access and manipulate subsets of data in Series and DataFrame objects. They let you pick rows, columns, cells, or blocks of data by label, by integer position, or by condition.

Label-based vs Position-based

There are two main styles of selection:

• Label-based (use loc): selects by row and column labels (index names). When slicing with labels, the end bound is inclusive. Example: df.loc['2023-01-01':'2023-01-07'] includes both dates.
• Position-based (use iloc): selects by integer position (like standard Python indexing). When slicing with positions, the end bound is exclusive. Example: df.iloc[0:5] returns rows 0..4.

Common Accessors

• df['col'] or df.col: return a Series for one column. df[['c1','c2']] returns a DataFrame for multiple columns.
• df.loc[row_label] / df.loc[row_label, col_label]: label-based selection.
• df.iloc[row_idx] / df.iloc[row_idx, col_idx]: integer position selection.
• df.at[row_label, col_label] and df.iat[row_idx, col_idx]: fast scalar access (single cell) by label/position.
• Boolean indexing: create a boolean mask and pass it to df[mask] or df.loc[mask] to filter rows.

Slicing rules and behavior

• Label slices with loc include the end label; integer slices with iloc exclude the end index.
• Using df[2:5] is position-based slicing on rows (like iloc[2:5]), returning rows 2..4.
• Chained indexing (e.g., df['col'][mask]) can sometimes return a view or a copy; use loc to reliably set values: df.loc[mask, 'col'] = value.

Setting values safely

To avoid SettingWithCopy warnings and ensure you modify the original DataFrame use loc or iloc for assignment:

df.loc[df['score'] < 40, 'grade'] = 'Fail'
# NOT recommended:
# df['grade'][df['score'] < 40] = 'Fail'  # may produce unpredictable results

Examples of combined selection

You can combine row and column selection in one expression: df.loc[row_labels, ['col1','col2']] or df.iloc[2:6, 0:3]. Boolean masks can be combined with column selection: df.loc[(df['age'] >= 18) & (df['city']=='Delhi'), ['name','email']].

Performance tips

• Use at / iat for single-cell access where speed matters.
• Selecting many rows/columns as a block is faster than looping row-by-row.

Overview
In Pandas, manipulating the structure of a DataFrame—adding, renaming and deleting data—is a common part of data cleaning and preparation. Key concepts are whether operations act on columns or rows, whether they modify the DataFrame in place or return a new DataFrame, and how index alignment works.

Adding data

• Add columns
  • Assignment: df['new'] = values adds a new column. Values can be a scalar, list/Series aligned by index, or expression based on other columns.
  • df.insert(loc, 'col', values) inserts a column at position loc.
  • df.assign(new=expr) returns a new DataFrame with the added column(s) (can chain operations).
• Add rows
  • Create a single-row Series or DataFrame and combine using pd.concat([df, new_df], ignore_index=True). (Note: df.append is deprecated.)
  • Use index assignment for a new row: df.loc[len(df)] = {...} for simple additions (careful with index types).
• Index changes: df.reindex(new_index) to change index (fills with NaN), df.reset_index(drop=True) to reset index to 0..n-1.

Renaming data

• df.rename(columns={'old':'new'}, index={old_idx:new_idx}, inplace=False) renames columns and/or index labels; by default it returns a new DataFrame.
• To replace all column labels at once: df.columns = ['A','B','C'] or use df.set_axis([...], axis=1, inplace=True).
• Renaming is non-destructive (unless inplace=True) and preserves data alignment.

Deleting data

• Drop rows or columns: df.drop(labels, axis=0 or 1, inplace=False). Use axis='columns' or axis='index' instead of numeric axis for clarity.
• del df['col'] or df.pop('col') removes and returns a column. del is in-place.
• Missing data: df.dropna() removes rows (or columns with axis=1) that have NaNs. Use df.fillna(value) to replace NaNs instead.
• Filtering: Often rows are removed by logical conditions: df = df[df['status'] != 'resigned'] (keeps only rows that satisfy the condition).

Practical notes & best practices

• Prefer functions that return new DataFrames (rename, assign, drop) so you can test changes before applying them; use inplace=True only when needed.
• Watch index alignment: when assigning a Series as a new column, Pandas aligns by index. Use ignore_index=True when concatenating new rows if you want a sequential integer index.
• Be careful with chained assignment (e.g., df[col][mask] = val)—it can raise SettingWithCopyWarning. Use df.loc[mask, col] = val instead.
• After many deletions or concatenations, use df.reset_index(drop=True) to clean up the index.

Small code cheatsheet (examples)

# Add/modify column
df['Total'] = df['Math'] + df['Science']
# Insert at column position 1
df.insert(1, 'Percentage', df['Total'] / df['Max_marks'] * 100)
# Add a row
new = pd.DataFrame([{'Name':'Zoe','Math':78,'Science':85}])
df = pd.concat([df, new], ignore_index=True)
# Rename column
df = df.rename(columns={'Math':'Mathematics'})
# Drop a column
df = df.drop(columns=['UnnecessaryColumn'])
# Drop rows with any missing value
df = df.dropna()
# Reset index
df = df.reset_index(drop=True)

What is missing data? Missing data (or missing values) are entries absent from a dataset where a value is expected. They arise from non-response in surveys, sensor failures, data-entry errors, transmission loss, or intentionally omitted fields.

Types of missingness (important for choosing a method):

• MCAR (Missing Completely At Random): Missingness independent of observed and unobserved data. Example: random sensor dropout.
• MAR (Missing At Random): Missingness depends on observed data (e.g., older respondents skip a question more often).
• MNAR (Missing Not At Random): Missingness depends on unobserved data (e.g., very high incomes deliberately not reported).

Why handle missing data? Many algorithms cannot process missing values and missingness can bias statistics (means, correlations) and machine learning models. Handling missing values properly preserves validity of analysis and predictions.

Common workflow / steps:

1. Detect and quantify missingness (e.g., df.isnull().sum(), percent missing per column).
2. Explore pattern of missingness (heatmap, missingness matrix, check relation to other variables).
3. Decide strategy per column or row: remove, impute, or model missingness.
4. Apply method (dropping, simple imputation, interpolation, or advanced imputation).
5. Validate — compare distributions/metrics before and after and, if possible, use holdout data to measure imputation error.

Pandas methods commonly used (useful to mention in Class 12 context):

• df.isnull() / df.notnull() — detect missing values.
• df.isnull().sum() — count missing values per column.
• df.dropna(axis=0 or 1, thresh=...) — drop rows/columns with missing values.
• df.fillna(value) — replace missing values with a constant or computed value.
• df.ffill() / df.bfill() — forward / backward fill.
• df.interpolate(method='linear') — linear interpolation for numeric series (especially time series).

Simple imputation strategies:

• Drop rows/columns when missingness is small or column is useless.
• Constant fill (e.g., 0 or 'Unknown') for categorical data.
• Mean / median / mode imputation — replace numeric missings with column mean/median or categorical missings with mode.
• Forward / backward fill for ordered data (time series).
• Interpolation to estimate intermediate numeric values (linear or time-based).

Advanced methods: K-Nearest Neighbours (KNN) imputation, regression imputation, and multiple imputation (creates several plausible datasets and pools results). These are used when simple methods would bias results.

Cautions:

• Mean imputation reduces variance and can distort correlations.
• Dropping rows may reduce sample size and statistical power.
• Always visualize and, if possible, compare to ground truth or holdout data to estimate imputation error.

Validation ideas: Compare distribution (histogram, boxplot) of original vs imputed data; compute RMSE between imputed and known values when you artificially hide some known values for testing.

Overview: Sorting and reordering are operations to change the sequence of rows or columns in a DataFrame or elements in a Series so that data is easier to read, analyse or visualise. In Pandas the main operations are sort_values(), sort_index() (for sorting) and reindex(), reset_index(), set_index() (for reordering or changing the index).

Sorting by values: Use df.sort_values() (or Series.sort_values()) to order rows by one or more column values. Key parameters: by (column name or list), ascending (True/False or list), inplace (modify in place), and na_position ("first" or "last"). Example:

df_sorted = df.sort_values(by=['score'], ascending=False)

To sort by multiple columns, provide a list: the first column is the primary key, second breaks ties, etc.

df.sort_values(by=['class', 'score'], ascending=[True, False])

Sorting by index (labels): Use sort_index() to sort rows (axis=0) or columns (axis=1) by their labels. Useful when the index is a date or categorical labels.

df.sort_index(axis=0, ascending=True)

Reordering rows and columns: Reordering means assigning a new index or rearranging columns in a specific order. Common methods:

• df.reindex(new_index) — reorders rows to match new_index (can add missing labels with fill_value).
• df.reset_index(drop=False) — move index into a column and set a new integer index.
• df.set_index('col') — make a column the index for label-based operations and sorting.
• You can reorder columns directly: df = df[['col2','col1','col3']].

Important behaviors and tips:

• By default sort_values returns a new object; use inplace=True to modify the original DataFrame.
• Stable sorting means tie order is preserved (Pandas uses quicksort/mergesort depending on kind).
• Missing values (NaN) are placed at end by default (na_position='last'), but you can put them first.
• Use ignore_index=True (Pandas >=1.0) to reset the index in the sorted result to a simple RangeIndex.

Example workflow (students marks): You may set 'student_id' as index, use df.sort_values(['total_marks'], ascending=False) to get a leaderboard, then reset_index() to convert index back to a column for reporting.

Performance note: Sorting is O(n log n) in general. For repeated lookups consider using indices or maintaining sorted structures.

Overview
Basic operations and aggregations in Pandas let you inspect, clean, summarize and derive insights from tabular data (DataFrame/Series). Operations include selection, filtering, sorting, adding/dropping columns and handling missing values. Aggregations compute summary statistics (sum, mean, count, min, max, etc.), often grouped by categories using groupby.

Common inspection & simple operations

• View top/bottom rows: df.head(n), df.tail(n)
• Shape and info: df.shape, df.info(), df.columns
• Summary stats: df.describe() (numeric), df.describe(include='object') (categorical)
• Select columns: df['col'] or df[['col1','col2']]
• Filter rows: df[df['age'] > 18], combine with & and |
• Sort: df.sort_values('sales', ascending=False)
• Add/drop columns: df['total'] = df['mrp'] * df['qty'], df.drop('col', axis=1, inplace=True)
• Missing values: df.isnull().sum(), fill with df.fillna(value) or drop with df.dropna()

Aggregations
Aggregations reduce data to summary statistics. Use built-in functions directly on Series or via DataFrame methods:

• Single-column: df['marks'].mean(), df['marks'].sum(), df['marks'].median(), df['marks'].std()
• Many columns: df.agg(['mean','sum','std']) or df[['math','science']].mean()
• Group-wise: df.groupby('class')['marks'].mean() gives average marks per class
• Multiple aggregations & multiple columns: df.groupby('region').agg({'sales':'sum','profit':'mean'})
• Value counts for categorical frequency: df['category'].value_counts()
• Pivot tables for 2D aggregation: df.pivot_table(index='store', columns='month', values='sales', aggfunc='sum')

GroupBy workflow (most common)
1) Split the data into groups by one or more keys; 2) Apply an aggregation; 3) Combine results into a DataFrame/Series.
Example pattern: df.groupby(['region','product']).agg({'sales':'sum','qty':'sum'})

Apply, agg and transform

• agg (or aggregate): compute one or more summary stats returning reduced outputs (per group or whole DF).
• apply: apply a custom function row-wise or column-wise (flexible but slower).
• transform: return an object indexed like the original (useful to add group-wise normalized columns, e.g., z-scores per group).

Performance & good practices

• Prefer vectorized operations (built-in pandas/numpy) over Python loops for speed.
• Use appropriate dtypes (category for repeated strings) to save memory and speed up groupby/value_counts.
• When chaining modifications, avoid repeated copies or use inplace=True carefully.

Short examples in code

# Basic stats and groupby
df.head()
df.info()
df.describe()

# Aggregations
avg = df['marks'].mean()
s = df.groupby('class')['marks'].sum()

# Multiple aggs per group
summary = df.groupby('class').agg({'marks':['mean','median','max'], 'age':'count'})

# Pivot table
pivot = df.pivot_table(index='store', columns='month', values='sales', aggfunc='sum')

# Value counts
df['grade'].value_counts()

# Create column using transform (percent of group total)
df['pct_of_class'] = df.groupby('class')['marks'].transform(lambda x: x / x.sum() * 100)

Class 12 focus: be comfortable reading DataFrames, filtering data, computing basic statistics, using groupby with one or two aggregations, and interpreting results (means, totals, counts, min/max).

Applying functions to data in pandas means transforming Series or DataFrame values by calling Python functions, NumPy universal functions, or pandas aggregation/transform methods. This is central to cleaning, converting, computing new columns, and aggregating results.

Key pandas tools:

• Series.map(func_or_dict): elementwise mapping on a Series. Can take a function, dict, or Series for value replacement.
• Series.apply(func): apply a function to every element of a Series (often similar to map but more general).
• DataFrame.apply(func, axis): apply a function along an axis. axis=0 applies to each column (Series passed to func), axis=1 applies to each row.
• DataFrame.applymap(func): elementwise function application to every cell in the DataFrame.
• agg/aggregate and transform: aggregate (reduce) multiple values to a single value per group/column (eg. mean, sum) or transform to produce same-shape output (eg. z-score per group).
• Vectorized/NumPy ufuncs: use built-in arithmetic and NumPy functions for fast elementwise operations without Python loops.

When to use which:

• Use map for simple Series replacements or mapping values using dicts.
• Use apply on a Series for custom elementwise logic, or on a DataFrame to operate on rows/columns.
• Use applymap if you need to change every cell in a DataFrame.
• Prefer vectorized operations or NumPy ufuncs where possible for performance.

Performance note: vectorized operations and built-in pandas methods are usually much faster than Python-level apply loops. Use apply/applymap when no direct vectorized alternative exists.

Examples of common tasks: converting units, parsing strings, computing ratios, labeling categories, rounding, handling missing values via custom rules, and group-based transformations using groupby().apply/agg/transform.

Purpose: After cleaning, transforming or analysing data in pandas, results are exported/saved so they can be shared, re-used, loaded by other tools or stored for future use. Exporting means writing a DataFrame or other object to disk (CSV, Excel, JSON, SQL, binary formats) or to a database. Good exporting preserves data types, encoding, and metadata and is efficient for the dataset size.

Common pandas methods

• df.to_csv() — plain-text, widely compatible (good for tabular text exchange).
• df.to_excel() — Excel files (.xlsx, .xls) for business users and reports.
• df.to_json() — JSON for web APIs and hierarchical data interchange.
• df.to_sql() — store DataFrame in SQL database using a connection/SQLAlchemy engine.
• df.to_pickle(), pd.to_hdf(), df.to_parquet(), df.to_feather() — binary formats for fast read/write and preserving types.

Key parameters & best practices

• path: use clear file names and versioning (results_v1.csv).
• index=False when you don’t want the DataFrame index saved as a column.
• encoding='utf-8' to avoid character problems; use encoding='utf-8-sig' for Excel compatibility on Windows.
• sep/delimiter for CSVs (default ','); use sep='\t' for TSV.
• compression: e.g. 'gzip', 'bz2' to save space for large files.
• chunksize & mode='a': write large DataFrames in chunks or append to existing files.
• Choose format by needs: CSV/Excel for human-readability, JSON for web, Parquet/Feather for analytics performance, SQL for transactional storage.
• Store metadata (schema, date generated, processing steps) alongside files or in a README to enable reproducibility.
• Avoid pickling for long-term exchange (Python-version dependent); prefer open formats (Parquet, CSV, JSON).

Examples (short code)

# CSV
df.to_csv('processed_students_v1.csv', index=False, encoding='utf-8')

# Append CSV in chunks
for chunk in pd.read_csv('big_input.csv', chunksize=100000):
    processed = transform(chunk)
    processed.to_csv('big_output.csv', mode='a', header=not os.path.exists('big_output.csv'), index=False)

# Excel
df.to_excel('report.xlsx', sheet_name='Summary', index=False)

# Parquet (fast, columnar)
df.to_parquet('features.parquet', index=False)

# Save to SQL
from sqlalchemy import create_engine
engine = create_engine('sqlite:///results.db')
df.to_sql('sales_summary', con=engine, if_exists='replace', index=False)

# JSON for web APIs
df.to_json('out.json', orient='records', lines=True)

# Pickle (Python-only)
df.to_pickle('df.pkl')

When to use which format (quick guide)

• CSV: interoperability, small-medium tables, human readable.
• Excel: business reports, multiple sheets, formatting for presentation.
• JSON: nested data or web services.
• Parquet/Feather/HDF5: large datasets, analytics pipelines, efficient storage & I/O.
• SQL: transactional or multi-user access and query capability.

Errors & how to avoid them

• Encoding errors — set correct encoding and test special characters.
• Large files — use chunksize or binary columnar formats to avoid memory errors.
• Schema mismatches on append — ensure same columns/order or write code to align columns before appending.