Chapter 7 — Programming in Python

What is Python?
Python is a high-level, interpreted programming language that emphasises readability and a clear syntax. It supports multiple programming paradigms—procedural, object-oriented and limited functional styles—making it suitable for many kinds of tasks from small scripts to larger projects. For students, Python’s simple syntax reduces time spent on language details so they can focus on algorithms and problem solving.

Installing Python and tools
To begin programming install the official Python distribution suited to your operating system. IDLE is the simple editor that comes bundled, but other editors and IDEs (lightweight or full-featured) are also useful. Students should learn how to open a terminal or command prompt, run python or python3 to enter interactive mode, and run scripts with python filename.py. Learn to find the Python version with python --version and how to set PATH if needed.

Interactive mode vs script mode
The interactive shell (REPL) is excellent for quick experiments: evaluating expressions, testing small code snippets, and checking library behaviour. Script mode uses files saved with .py extension and lets you build larger programs that can be re-run and shared. Good practice: prototype in the REPL, then move working code into scripts or modules for reuse.

Project organization and files
A typical project has separate files for different parts of logic. Give files descriptive names, keep related functions in the same module and avoid very long scripts. Use the convention of a main guard if __name__ == '__main__': to run demonstration or test code only when a script is executed directly, not when it is imported.

Comments and documentation
Write clear comments using # for short notes and triple-quoted docstrings for modules, functions and classes. A docstring should state what the item does, its parameters and return value. Good documentation makes code easier to read and maintain and is crucial when working in teams or revising code later.

Style and conventions
Indentation is syntactically significant in Python; use four spaces per level and avoid mixing tabs and spaces. Use meaningful variable and function names and keep functions small and focused. These habits improve readability and reduce bugs.

First programs and testing
Begin with simple programs: printing messages, arithmetic expressions, and reading input. Test each component as you write it. Learn to read error messages and tracebacks; they provide the path to the line where an error happened, which speeds up debugging. These basics form the foundation for all topics that follow.

Primitive data types
Python supports fundamental data types such as integers (int), floating point numbers (float), booleans (bool) and strings (str). Each value has a type; variables are names bound to values and can be rebound to values of different types. Dynamic typing makes code concise but requires discipline: use meaningful names and consistent use so the reader understands the expected type.

Numbers and arithmetic
Integers in Python have arbitrary precision so they do not overflow like in some languages. Floats represent real numbers with finite precision. Common operations include +, -, *, / (floating division), // (floor division), % (modulus) and ** (exponentiation). Functions like abs(), round(), int(), float() and the math module extend numeric capabilities. For example, // returns the integer quotient and % the remainder, useful for dividing tasks into chunks or extracting digits.

Strings and operations
Strings are sequences of characters and are immutable. Create strings with single, double or triple quotes for multi-line text. Important operations include concatenation with +, repetition with *, indexing s[i], slicing s[start:stop:step] and methods such as .lower(), .upper(), .split(), .join(), .strip(), .replace() and .find(). Because strings are immutable, methods return new strings rather than modifying the original.

Booleans and truth value testing
Boolean values True and False are results of comparisons and logical operations. Many objects have truthiness: empty sequences or containers ('' , [], {}, set()) evaluate to False, non-zero numbers evaluate to True. Use and, or, not for logical operations. Be careful comparing floats for equality due to precision; prefer comparison within a small tolerance when needed.

Variables and naming rules
Variable names must start with a letter or underscore and can contain letters, digits and underscores. Avoid reserved keywords. Choose descriptive names (student_count rather than sc) and use consistent style (snake_case). Python allows multiple assignment: a, b = 1, 2 and swapping: a, b = b, a which is concise and clear.

Type conversion and casting
Convert between types explicitly using int(), float(), str(), bool(), list() etc. Implicit conversions occur in numeric expressions but explicit casting avoids surprises. When reading input with input(), always convert the string to the required type and handle exceptions for invalid input using try/except.

Best practices
Document the expected type of variables in comments or docstrings for functions. Use constants with UPPER_CASE names for fixed values. Keep variable scope as small as possible so values are easier to reason about and tests are simpler.

Operator categories
Python has many operators grouped by purpose: arithmetic (+, -, *, /, //, %, **), comparison (==, !=, <, >, <=, >=), logical (and, or, not), bitwise (&, |, ^, ~, <<, >>), membership (in, not in) and identity (is, is not). Understanding each category and its precedence helps avoid logic errors and unexpected results.

Arithmetic rules and behaviour
Remember that / always produces a float, even when dividing integers. Use // for integer division when you want the floor result. The modulus operator % gives the remainder and is useful in cycles and digit extraction. Exponentiation ** is right-associative so 2 ** 3 ** 2 equals 2 ** (3 ** 2). Mixing ints and floats in arithmetic usually yields float results.

Comparison operators
Comparisons produce boolean values. Python supports chaining comparisons such as 0 < x <= 10 which is both concise and readable. Use comparisons to test boundaries in conditions. Be cautious with floats and compare within tolerances when testing equality due to representation errors.

Logical operators and short-circuiting
and and or use short-circuit evaluation: in an expression A and B, B is evaluated only if A is True; in A or B, B is evaluated only if A is False. This property is useful for safe checks, for example testing that a container is non-empty before indexing it. The not operator negates a boolean value.

Bitwise operators
Bitwise operators work on integer bit patterns and include &, |, ^, ~, << and >>. They are handy for low-level tasks and certain algorithmic tricks, but for most high-level applications use arithmetic and logical operators. Understand that negative numbers use two’s complement representation and that shifting and complementing behave accordingly.

Membership and identity
Use in to check membership in sequences or sets and is to compare whether two names reference the same object (useful when comparing to None: x is None). Avoid using is to compare strings or integers for equality since identical-looking values may not always be the same object.

Operator precedence and clarity
Operator precedence affects evaluation order. Parentheses override precedence and should be used to make expressions clearer. When expressions become complex, break them into intermediate variables with descriptive names. This both aids readability and helps debugging by allowing inspection of intermediate values.

Practical advice
Test boundary and corner cases for expressions, especially with division and modulo. Prefer explicit conversions when mixing types. Use the appropriate operator for the intent: use // when you need integer division and % for remainders. Keep expressions simple and clear for maintainability.

Making decisions in programs
Conditional statements let a program choose between paths of execution depending on conditions that evaluate to True or False. The primary constructs in Python are if, if-else and if-elif-else. Carefully written conditions express program logic clearly and reduce bugs.

Basic forms and structure
The simplest form is if condition: statements. For two alternatives use if condition: true_block else: false_block. For multiple mutually exclusive cases use if-elif-else chains. Each condition is evaluated in turn; the first True branch executes and the rest are skipped. Indentation groups the statements belonging to each branch.

Boolean expressions and clarity
Conditions are often combinations of comparisons and logical operators. Write conditions readably: use parentheses to group subconditions when needed and break complex logic into well-named boolean variables or helper functions. For example, valid_age = (age >= 18) makes subsequent code more understandable than repeating the test.

Nested conditionals and refactoring
Nested if statements are useful but can lead to deep indentation. When nesting becomes complex, consider refactoring into functions or using early returns to simplify flow. Early returns exit a function when a particular condition is met, avoiding excessive nesting and improving readability.

Conditional expressions
Python supports a compact ternary form: value_if_true if condition else value_if_false. Use it for simple assignments like status = 'pass' if marks>=40 else 'fail'. Avoid long or complex ternary expressions as they harm readability.

Common pitfalls
A common mistake is using separate if statements when mutually exclusive cases are intended; this can cause multiple branches to execute. Floating-point equality comparisons can be unreliable; compare within a small epsilon. Remember that assignment inside a condition is not allowed in Python, preventing a class of bugs seen in some other languages.

Testing conditionals
Test each branch with representative values including edge cases: minimum and maximum allowed values, empty inputs, zero and negative numbers. Good test coverage for conditionals dramatically reduces runtime surprises and logical errors.

Repeating tasks with loops
Loops allow a program to repeat work until a condition is met or until all items of a collection are processed. Python has two primary loop constructs: for and while. Use for when you want to iterate over a sequence or range, and while when repetition depends on a condition that changes during execution.

For loops and iterables
Python’s for loop iterates directly over items of an iterable such as lists, strings, tuples and generators. This eliminates manual indexing in many cases. Use range(start, stop, step) to generate sequences of integers when index values are needed. The enumerate() built-in is useful when you need both an index and the item: for i, item in enumerate(seq, start=1):.

While loops and termination
A while loop repeats as long as its condition remains True. Ensure loop variables are updated correctly so the loop eventually terminates. Infinite loops are common beginner errors; include a clear exit condition or a break statement to leave the loop when appropriate.

Loop control statements
break exits the nearest loop immediately, commonly used when a search succeeds. continue skips to the next iteration and is useful when you want to ignore certain cases without stopping the loop. Python also supports an optional else block on loops: the else body executes only if the loop completes normally without encountering break, which is handy for indicating 'not found' after a search.

Iterating dictionaries and sets
When iterating dictionaries use .items() to obtain key-value pairs, .keys() for keys and .values() for values. Sets are unordered, so iteration order is not guaranteed. Avoid modifying a container (adding or removing items) while iterating over it; instead build a list of changes and apply them after iteration or iterate over a shallow copy.

Nested loops and complexity
Nested loops handle multi-dimensional data like matrices; however, complexity multiplies with nesting depth. For large inputs prefer algorithms that reduce nesting or use more efficient data structures. Practice tracing nested loops with small examples to understand iteration order and index relations.

Practical patterns
Common loop patterns include accumulation (sum items), filtering (collect items meeting a condition), mapping (transform items), and searching. For many such patterns list comprehensions or generator expressions provide concise alternatives; learn both approaches and use whichever is clearer and more efficient for the task at hand.

Purpose of functions
Functions divide programs into manageable units that perform specific tasks. They promote code reuse, make testing easier, and improve readability. Each function should have a single responsibility and a clear interface: defined parameters and a documented return value.

Defining functions
In Python use def name(parameters): followed by an indented block. Include a docstring immediately after the def line to describe what the function does, its parameters and return value. Keep functions short and focused; long functions are harder to test and understand.

Parameters and calling conventions
Python supports positional and keyword arguments, default values, variable-length positional arguments (*args) and variable-length keyword arguments (**kwargs). Default arguments should be immutable objects to avoid shared-state bugs. Keyword arguments improve readability when calling functions with many parameters.

Return values and multiple returns
Use return to send results back to the caller. A function without return returns None. Python allows returning multiple values as a tuple which can be unpacked by the caller: return a, b. Prefer returning values rather than modifying global state, which keeps functions pure and easier to reason about.

Scope and lifetime
Names defined inside a function are local and not visible outside. Global variables can be read but should be avoided for writing unless necessary; use global declaration only when required. Understanding scope prevents accidental name collisions and hard-to-find bugs.

Higher-order and anonymous functions
Functions are first-class objects: they can be assigned to variables, passed as arguments and returned from other functions. The lambda expression creates a small anonymous function for simple uses. Map, filter and sorted with key functions illustrate these concepts practically.

Testing and documentation
Write unit-like tests for functions covering normal, boundary and invalid inputs. Use assertions in tests to check expected results. Good docstrings and examples in a function's documentation help future users understand correct usage without reading implementation details.

What are modules?
Modules are Python files (.py) that group related code — functions, classes and constants — into a single namespace. Using modules improves organisation and allows code reuse. A module can be imported using import module or from module import name. When imported, the module’s top-level code executes, so keep side-effects minimal.

Creating and structuring packages
A package is a directory containing modules and an __init__.py file which designates the directory as a package. Packages allow hierarchical organisation: package.module provides a namespace that avoids name collisions and clarifies where functionality comes from. Use descriptive names and place related modules together.

Import styles and best practices
Prefer import module and use module.name to make origin clear. from module import name is convenient but can clutter the namespace; avoid from module import * because it makes code harder to read and may overwrite names. Use aliases when appropriate: import math as m for brevity, but keep clarity in mind.

Standard library overview
Python’s standard library contains many ready-to-use modules: math for mathematical functions, random for random numbers, datetime for date and time, os and sys for operating system interactions, json for serialisation, csv for CSV handling and re for regular expressions. Learning to read and use documentation for these modules is a key skill: it avoids reinventing common functions and makes solutions more reliable.

Module execution and __main__ guard
Include if __name__ == '__main__': blocks for demo or test code so the code runs when the module is executed directly but not when it is imported. This pattern supports both reuse and simple testing without extra test frameworks.

Dependencies and portability
Rely on the standard library for classroom and exam work so programs are portable across systems. Third-party packages installed via pip are useful in projects but may not be available in all environments; mention them as advanced topics but do not require them for board exercises.

Practical tips
Design modules with small, testable functions. Keep module responsibilities limited and document public functions with docstrings. This makes modules easier to maintain and reuse in other programs or assignments.

Opening and closing files safely
To work with files use the built-in open function: open(filename, mode). Common modes are 'r' (read), 'w' (write), 'a' (append) and combinations with 'b' for binary. Always close files to free system resources; using with open(...) as f: is the recommended pattern because it ensures the file is closed automatically even if an error occurs inside the block.

Reading strategies
Read the entire file at once with f.read() when the file is small and memory is sufficient. For larger files prefer iterating over the file object line by line: for line in f: which reads one line at a time and is memory efficient. Other methods include f.readline() to read a single line and f.readlines() to return a list of all lines (use carefully for large files).

Writing and appending
Use f.write(string) to write text to files opened in 'w' or 'a' mode. Remember to include newline characters where appropriate because write does not add them automatically. Use mode 'w' to create or overwrite a file and 'a' to add to an existing file without erasing its contents. Use encoding='utf-8' for text files to handle non-ASCII characters reliably.

Parsing structured text
Text files often contain structured records such as comma-separated values. The csv module provides robust handling for reading and writing CSV data, including quoted fields. For simple tasks you can split lines with str.split(',') but handle edge cases like extra whitespace and missing fields carefully. Normalize input by stripping newline characters and unwanted spaces before processing.

Updating files safely
When modifying records, do not edit the file in place. Instead read the original file and write changes to a temporary file; once complete, replace the original with the temporary file. This avoids data loss if the program crashes midway.

Error handling and existence checks
Handle FileNotFoundError when attempting to open a missing file, and use os.path.exists to check existence if needed. Wrap file operations in try/except blocks and provide useful error messages to the user. For binary data use 'rb' and 'wb' modes and handle bytes instead of strings.

Practical examples
Common classroom tasks include counting words or lines in a file, extracting fields from structured records, copying files, and appending logs. Practice reading files line-by-line and processing each record carefully to develop reliable data-processing code.

Importance of handling errors
Programs often face unexpected situations (invalid input, missing files, division by zero). Exceptions signal these runtime errors. Handling exceptions makes programs resilient and helps provide clear, user-friendly messages rather than abrupt crashes. Good exception handling separates normal logic from error-handling logic.

Try, except, else and finally
Use the try block to run code that may raise exceptions. Follow it with one or more except blocks to handle specific exception types. The optional else block runs when no exception occurred and is useful for code that should execute only on success. The finally block executes regardless and is commonly used for cleanup such as closing resources.

Catch specific exceptions
Prefer catching specific exception classes (ValueError, FileNotFoundError, ZeroDivisionError) instead of a bare except which catches everything and can hide programming errors. Use the exception instance: except ValueError as e: to access error details and include them in messages or logs for debugging.

Raising exceptions
Use raise to signal that a function encountered an invalid condition. Raising clear exceptions with informative messages helps callers understand the problem. For domain-specific situations define custom exception classes by inheriting from Exception so calling code can catch and handle them specifically.

Assertions and development checks
Assertions (assert condition, 'message') are useful during development to enforce programmer assumptions. They raise AssertionError when failed. Do not rely on assertions for normal runtime error handling because they can be disabled with optimisation flags.

Resource management and safe patterns
Prefer using context managers (with statements) to manage resources so cleanup happens automatically. When manual cleanup is necessary combine try/finally: try: resource use finally: resource.close() to guarantee release even if an exception occurs.

Debugging and logging
Allow exceptions to propagate during development so tracebacks reveal the cause. In production code catch and log exceptions appropriately, providing the user with friendly messages while preserving stack traces in logs for developers. Thoughtful exception handling improves program reliability and maintainability.

Definition and properties
Lists are ordered, mutable collections that can hold items of different types. They are created using square brackets: [1, 'a', 3.5]. Lists preserve insertion order, allow duplicates and support indexing and slicing. Their mutability means you can change elements after creation, making lists very flexible for many algorithms.

Accessing and modifying
Access elements by index with list[i] where indices start at 0. Negative indices count from the end: list[-1] is the last item. Assign to an index to change a value. Use append(item) to add at the end, insert(index, item) to add at a position, remove(value) to delete the first matching value, pop(index) to remove by index and del to delete slices or single items. Use extend(iterable) to append multiple items efficiently.

Slicing and copying
Slicing list[start:stop:step] returns a new list containing the requested items. Slicing is a convenient way to extract sublists or to copy lists: new_list = old_list[:]. Remember that slicing performs a shallow copy: for nested lists inner objects are still shared. For independent nested copies use the copy module’s deepcopy when needed.

Searching, sorting and reversing
Find an element using in and index methods. Use list.sort() to sort in place or sorted(list) to get a new sorted list. Reverse a list with list.reverse() or reversed(list) to obtain an iterator. Consider data sizes: sorting is O(n log n), and linear searches are O(n).

List comprehensions
Comprehensions are an expressive way to create lists: [expr for item in iterable if condition]. They often replace loops that build lists by appending. Use comprehensions for mapping and filtering in concise, readable code, but avoid overly complex nested comprehensions that reduce clarity.

Nested lists and pitfalls
Nested lists represent matrices and tables. Access elements with list[i][j]. When creating nested lists avoid using multiplication like [[0]*m]*n because inner lists become references to the same object; use nested comprehensions: [[0 for _ in range(m)] for _ in range(n)]. This creates independent inner lists.

Performance considerations
Append is amortised O(1), but inserting near the front or deleting arbitrary positions can be O(n). For algorithms with many front insertions or removals consider other structures (not in core syllabus) like deque. Understanding these performance trade-offs helps design efficient programs.

Tuples: fixed sequences
Tuples are ordered, immutable sequences defined with parentheses or simply comma-separated values: (1, 2, 3) or 1, 2, 3. Because tuples are immutable they can be used as keys in dictionaries when they contain only hashable items. Use tuples to represent fixed records like coordinates or student records where the values should not change.

Sets: unique collections
Sets are unordered collections of unique elements created with curly braces {1, 2, 3} or set(iterable). Sets support fast membership tests and standard set operations: union (|), intersection (&), difference (-) and symmetric difference (^). They are ideal for removing duplicates, checking overlap between groups, and performing membership-based logic. Remember that sets are unindexed and cannot store unhashable types like lists.

Dictionaries: key-value mappings
Dictionaries map keys to values using curly braces: {'a':1, 'b':2}. Keys must be hashable (immutable), while values may be any type. Lookup by key is fast (average O(1)). Use d.get(key, default) to handle missing keys gracefully and d.setdefault(key, default) to create entries if absent. To iterate use d.items() for key-value pairs, d.keys() and d.values().

Common patterns and applications
Dictionaries are commonly used for frequency counts: for item in seq: freq[item] = freq.get(item, 0) + 1. Sets are useful for removing duplicates: unique = set(lst). Tuples are useful when returning multiple values from functions or using composite keys in dictionaries. Choose the right structure based on whether you need ordering, mutability and fast lookup.

Mutability and copying
Lists, sets and dictionaries are mutable. Copying with copy() or slicing creates shallow copies; nested mutable elements are still shared. Use copy.deepcopy when independent nested copies are required. Be careful when sharing mutable objects across variables to avoid unintended side-effects.

Choosing between structures
Pick lists when order and indexing matter, tuples when records are fixed, sets when uniqueness and membership speed matter, and dictionaries when you need key-based access. Understanding these differences helps design clear and efficient solutions to problems.

Core string operations
Strings are sequences of characters. Common operations include indexing, slicing, concatenation, repetition and transformations with methods like .lower(), .upper(), .strip(), .split(), .join() and .replace(). Slicing s[start:stop:step] extracts substrings and is used often in parsing tasks. Since strings are immutable, these methods return new strings rather than changing the original.

Formatting and combining text
Create readable output with f-strings: f'{name} scored {marks}'. The format() method also formats values into templates and supports specification of width, precision and type. For CSV-style output join() can combine fields efficiently: ','.join(fields).

Parsing and tokenising
Split text into words or fields using .split() with a delimiter. Stripping whitespace with .strip() or .rstrip() helps normalise inputs. When parsing structured text, handle edge cases such as extra delimiters or empty fields.

Searching and replacement
Find substrings with .find() (returns -1 if not found) and count occurrences with .count(). Replace substrings with .replace(). For more complex pattern matching and transformation use the re module which supports regular expressions for expressing patterns concisely.

Regular expressions (introductory)
Regular expressions (regex) let you search and extract patterns like digits, words or fixed formats. Basic building blocks include character classes [abc], ranges [a-z], digit shorthand \d, whitespace \s, quantifiers * + ?, exact repetitions {m,n}, anchors ^ and $ for start and end of string, and groups with parentheses. Use re.search() to find a pattern anywhere, re.match() to check from the start, re.findall() to extract all matches, and re.sub() to replace matches. Compile patterns with re.compile() when reusing them for efficiency.

Practical examples and cautions
Use regex to validate simple formats like phone numbers or extract numbers from mixed text. For many simple tasks string methods are faster and clearer; reserve regex for cases where patterns are non-trivial. Always test regex patterns on representative inputs and document complex patterns for future readers.

Unicode and encoding
When handling text from files or networks pay attention to encoding (UTF-8 is common). Python 3 strings are Unicode, so methods work with international text, but ensure correct encoding when reading or writing files. Handle exceptional cases with try/except when decoding may fail.

OOP purpose and benefits
Object-oriented programming organises code by bundling data and behaviour into classes. This models real-world concepts: a class defines the structure (attributes) and behaviour (methods) of objects created from it. OOP supports encapsulation to hide internal details, modular design for reuse, and clearer mapping from problem domain to code.

Defining classes and creating instances
Define a class with class ClassName: and include an __init__(self, ...) method to initialise instance attributes. Attributes attached to self belong to the instance, while attributes defined at class level are shared across all instances. Instantiate objects by calling the class: obj = ClassName(args). Access attributes with obj.attr and call methods with obj.method().

Encapsulation and conventions
Encapsulation controls access to internal state. In Python a single leading underscore _attr signals that an attribute is intended for internal use; name mangling with double leading underscores __attr provides limited protection by changing the attribute name internally. However, Python relies on conventions and documentation more than strict access enforcement.

Special methods and operator behaviour
Implement dunder (double-underscore) methods to integrate with Python features. __str__ returns a human-friendly string representation, __repr__ is for developer-facing representation, __eq__ defines equality, and __len__ supports len(). These make objects behave naturally with printing, comparisons and built-in functions.

Designing classes well
Keep classes focused and small; each class should represent a single responsibility. Group related data and methods together and prefer composition (an object containing another) when relationships are 'has-a' rather than 'is-a'. Document each class and its public methods with docstrings describing expected parameters and returns.

Testing objects
Test classes by creating instances and exercising their methods. Verify that method calls change state in expected ways and that constructors initialise attributes correctly. Small, well-documented classes are easier to test and reuse in larger programs like student record systems, inventory or bank account simulations.

What is inheritance?
Inheritance lets a new class (subclass) reuse, extend or modify behavior from an existing class (base or parent). It promotes code reuse and models natural hierarchies: for example, a Vehicle base class with Car and Bike subclasses. Define a subclass as class Child(Parent): and override or extend methods to change behaviour.

Using super()
When overriding the initializer or other methods, call super() to reuse parent class initialization: super().__init__(...) ensures the base class sets up its part of the object. This avoids duplicating initialization code and keeps subclass focus on additional attributes or behaviour.

Polymorphism concept
Polymorphism allows code to work with objects of different classes through a common interface. If multiple classes implement a method with the same name (for example area()), a function that calls obj.area() can operate on any such object without knowing its concrete class. This leads to flexible and extensible designs.

Multiple inheritance and MRO
Python supports multiple inheritance where a class can inherit from more than one parent. This can be powerful but introduces complexity: Python uses the Method Resolution Order (MRO) to determine which method to call when multiple parents define the same method. Use multiple inheritance only when it models the problem clearly and avoid deep inheritance chains which are difficult to maintain.

Overriding vs overloading
Overriding replaces a base class method in the subclass to change behaviour. Python does not support method overloading by signature; define methods to accept optional parameters or use different method names rather than relying on multiple definitions with different parameter lists.

Design guidance and examples
Prefer composition over inheritance when the relationship is not naturally 'is-a'. Design base classes to define clear interfaces and minimal implementation so subclasses can extend behaviour simply. Demonstrate with animals implementing speak() differently or shapes implementing area(), and write code that uses polymorphism to process collections of objects uniformly.

Understanding recursion
Recursion is a programming technique where a function calls itself to solve a problem by reducing it into smaller subproblems. A correct recursive solution requires a clear base case that stops further recursion and a recursive step that moves toward that base case. Recursion maps naturally to problems with self-similar structure, such as mathematical sequences, tree traversals and divide-and-conquer algorithms.

Designing recursive functions
Identify the smallest input(s) for which the answer is trivial—the base case—and express the solution for larger inputs in terms of smaller ones. Reason about correctness often by induction: assume the recursive call works for smaller inputs and show the current case follows. Trace the call stack for small inputs to ensure the sequence of calls and returns behaves as expected.

Examples and patterns
Factorial is the canonical example: factorial(n) = n * factorial(n-1) with base case factorial(0) = 1. Tree traversal uses recursion to visit a node and then recursively visit its children. Fibonacci numbers have a simple recursive definition but naive recursion repeats work and becomes exponentially slow for larger n; this motivates memoisation or iterative methods.

Performance and space trade-offs
Recursion uses the call stack; each call consumes stack space proportional to the depth of recursion. For large depths Python may hit the recursion limit and raise RecursionError. Tail-call optimisation is not guaranteed in Python, so prefer iterative solutions when depth is large or convert recursion to an explicit stack. Memoisation caches results to avoid repeated computation and can convert exponential-time recursive algorithms into efficient ones.

Debugging and testing recursion
Trace recursive calls with print statements or manually draw the recursion tree to understand flow and intermediate values. Test base cases explicitly and also test boundary values like n=0, n=1 and small positive values. Ensure that each recursive call narrows the problem; otherwise you risk infinite recursion.

When to use recursion
Use recursion when it leads to clearer code or when the problem is naturally recursive (trees, nested structures, divide-and-conquer). For tight performance constraints or deep nesting prefer iterative solutions or techniques that reduce recursion depth.

List comprehensions for concise lists
List comprehensions offer a compact syntax to create lists from iterables: [expression for item in iterable if condition]. They combine mapping (transforming each item) and filtering (selecting items meeting a condition) in a single readable line. For many tasks they are clearer and faster than equivalent for-loops that append to a list.

Examples of comprehensions
Common examples include squares: [x*x for x in range(1,6)], or filtering: [x for x in range(1,21) if x%2==0]. Nested comprehensions can express nested loops: [(i,j) for i in range(3) for j in range(2)] but keep nested comprehensions simple to maintain readability.

Generator expressions and lazy evaluation
Generator expressions use parentheses and produce values lazily: (x*x for x in range(1000000)). Unlike list comprehensions they do not create the entire list in memory, yielding one item at a time which saves memory for large sequences. Use generators when you process data in a pipeline, such as reading and filtering lines from a large file.

Generator functions with yield
Define generator functions using yield inside a def. Each yield produces a value and preserves the function’s state until the next request. Generators are ideal for streaming data, creating infinite sequences, or implementing iterators for custom data structures. Use next(generator, default) to retrieve items safely and handle StopIteration raised when the generator is exhausted.

Performance and readability trade-offs
Comprehensions are usually faster for building lists because they are optimised; however, if memory is an issue, generators are the better choice. Generator code can be slightly more complex to reason about due to lazy evaluation, so document intent when using generators in shared code.

Practical usage patterns
Chain generator expressions to form pipelines that filter, map and aggregate data without building intermediate lists. Use comprehensions for short to medium-size lists that you need to index or reuse. Knowing when to choose between list comprehensions and generators improves both performance and clarity of programs.

Higher-order functions
Higher-order functions either take functions as arguments or return functions. This feature allows flexible code composition: functions like map, filter and sorted accept function parameters to control behaviour. Understanding higher-order functions helps design modular code where behaviour can be injected or customised without changing the caller.

What are decorators?
Decorators are a special case of higher-order functions that modify or extend functions' behaviour in a reusable way. A decorator is a callable that accepts a function and returns a new function (often a wrapper) that adds pre- or post-processing. Use the @decorator syntax above a function definition as shorthand for wrapping the function with the decorator.

Writing a simple decorator
A typical decorator defines an inner wrapper function that accepts arbitrary arguments (*args, **kwargs), performs actions before calling the original function, calls the original function and then performs actions after the call, returning the original result. For example, a logging decorator prints the function name and arguments before calling it, while a timing decorator measures execution time. Return the wrapper from the decorator so the caller uses the enhanced behaviour.

Use-cases and caution
Common uses include logging, caching (memoisation), authorization checks, input validation and retry logic. Decorators can be stacked by applying multiple @ lines; execution order is from the nearest decorator outward. Preserve the original function metadata when writing decorators; functools.wraps can copy metadata but is an advanced detail. Keep decorators simple and predictable to avoid surprising side-effects.

Testing decorated functions
Test both the decorator’s effect and the underlying function’s correctness. Ensure the wrapper passes arguments correctly and returns the same value when appropriate. For caching decorators verify that repeated calls avoid recomputation, and for logging decorators verify messages are produced as expected.

Design practices
Use decorators to keep cross-cutting concerns (like logging) separate from core logic. Document decorator behaviour clearly and avoid modifying function signatures in ways that confuse callers. For board-level study focus on understanding concept and implementing a basic decorator rather than covering every advanced pattern.

Testing fundamentals
Testing ensures code works as intended. Start with manual tests: try typical inputs, boundary values and invalid inputs. For functions create small test scripts that call them with a variety of values and use assert statements to check expected outputs. While formal frameworks like unittest or pytest are available, basic systematic testing with asserts and sample inputs is sufficient for classroom practice.

Debugging techniques
When a program behaves incorrectly read the traceback to find the error location. Use print statements to display variable values at key points, or use an interactive debugger to step through code and inspect state. Rational debugging steps include reproducing the bug consistently, isolating the faulty part, forming a hypothesis about the cause, testing that hypothesis and implementing a fix.

Common sources of bugs
Frequent errors include off-by-one mistakes in loops, incorrect indentation, misuse of mutable default arguments (which share state), wrong assumptions about types, and index errors. Learn to check types and sizes of data during debugging. Defensive coding—validating inputs and using meaningful error messages—reduces the number of runtime surprises.

Code quality and readability
Readable code is easier to test and maintain. Use meaningful names, short functions that do one thing, and consistent indentation. Document functions with docstrings explaining parameters, return types and exceptions. Refactor duplicated code into helper functions to reduce errors and make changes easier.

Performance awareness
Write correct code first, then consider performance. Simple profiling with timing (time.time() or time.perf_counter()) around code sections identifies bottlenecks. Avoid repeated expensive operations in loops, choose appropriate data structures (e.g., dictionary lookups vs list searches) and prefer built-in functions which are often optimised in C.

Version control and collaboration
Saving incremental versions of code and using version control systems helps recover from mistakes and collaborate with others. Even simple habits like keeping backups, incremental commits and clear commit messages improve project organisation and reduce stress when debugging complex problems.

Practical testing workflow
Adopt a cycle: write code, test with unit scenarios, debug failing cases, refactor for clarity, and retest. This systematic approach prevents bugs from accumulating and makes assignments easier to manage under time pressure.