Instructor office hours
Ali Ghodsi
Wednesdays, 1–2 p.m.
M3 4009
Come to the instructor for: questions about course content or the final project.
Fall 2026
An in-depth graduate course on the statistical foundations, computational methods, and modern advances underlying classification and pattern recognition.
Course at a glance
This page is the public home for the course topics, slide releases, and assignments. Enrollment-specific information, submissions, deadlines, and grades are maintained in Waterloo Learn.
Course support
Instructor office hours
Wednesdays, 1–2 p.m.
M3 4009
Come to the instructor for: questions about course content or the final project.
Teaching assistant office hours
Go to the TAs for: questions about assignments or coding.
From the course outline
Textbooks and references
There is no mandatory textbook. The following references cover substantial portions of the course:
| Component | Description | Weight |
|---|---|---|
| Assignments | Four individual assignments at 12.5% each | 50% |
| Paper presentation | Small-group presentation of a selected research paper | 10% |
| Group final project | Research or applied classification project | 40% |
| Total | — | 100% |
| Assessment | Release | Due |
|---|---|---|
| Assignment 1 | Monday, September 28 | Monday, October 19, 11:59 p.m. |
| Assignment 2 | Monday, October 19 | Monday, November 9, 11:59 p.m. |
| Assignment 3 | Monday, November 9 | Monday, November 23, 11:59 p.m. |
| Assignment 4 | Monday, November 23 | Monday, December 7, 11:59 p.m. |
| Paper presentation | Paper-selection process to be announced | Scheduled presentation date |
| Group final project | To be announced | Monday, December 21, 11:59 p.m. |
All dates are tentative and may change. Changes will be announced.
Latest updates
Course site
This page will be updated throughout the term. Slides and assignments will be posted below as they are released.
Communication
Questions about lectures and course content should be posted on Piazza so that answers are available to the class.
Course schedule
Lecture topics and ordering are tentative and may be adjusted during the term.
| Lecture | Coverage | Materials |
|---|---|---|
| Lecture 1 | Course introduction; classification problems; hypothesis class, loss, and search; Bayes classifier and Bayes error | Lecture 1 slides |
| Lecture 2 | Gaussian classifiers; LDA and QDA; covariance assumptions and decision boundaries | Lecture 2 slides |
| Lecture 3 | Principal component analysis (PCA); variance-maximizing projections; Lagrange multipliers; covariance eigendecomposition and SVD; encoding and reconstruction | Lecture 3 slides |
| Lecture 4 | PCA versus supervised class separation; Fisher's linear discriminant analysis for two and multiple classes; within- and between-class scatter; generalized eigenvalue formulation | Lecture 4 slides |
| Lecture 5 | Logistic regression; binomial class probabilities; maximum likelihood; score and Hessian; Newton-Raphson and iteratively reweighted least squares (IRLS) | Lecture 5 slides |
| Lecture 6 | Neural-network history from McCulloch-Pitts through AlexNet; perceptron model and geometric decision boundary; gradient-based perceptron learning; linear separability and convergence | Lecture 6 slides |
| Lecture 7 | Feedforward and deep neural networks; backpropagation via the chain rule; gradient-based weight updates and epochs; stochastic gradient descent, mini-batches, and learning-rate tuning | Lecture 7 slides |
| Lecture 8 | Model selection and cross-validation; regularization and weight decay; data augmentation, noise injection, early stopping, and dropout; Keras neural-network architecture, activation functions, layers, losses, and optimizers | Lecture 8 slides |
| Lecture 9 | Radial basis function (RBF) networks; network architecture, matrix formulation, and closed-form least-squares fitting | Lecture 9 slides |
| Lecture 10 | Model selection; empirical and true error; Stein's unbiased risk estimator (SURE); RBF model complexity, noise-variance estimation, and regularization | Lecture 10 slides |
| Lecture 11 | Support vector machines; maximum-margin geometry and hard-margin optimization; Lagrangian dual formulation, KKT conditions, and support vectors; kernel methods and feature spaces; MATLAB classification of handwritten digits with linear and Gaussian kernels | Lecture 11 slides |
| Lecture 12 | Soft-margin SVMs; slack variables and margin violations; primal and dual optimization, the Lagrangian, and KKT conditions; support-vector cases, kernel formulation, and computing the intercept and classifier | Lecture 12 slides |
| Lecture 13 | Convolutional neural networks: convolution and cross-correlation, sparse connectivity, parameter sharing, and translation equivariance; pooling and downsampling; training convolutional layers and using random filters; ResNet and DenseNet | Lecture 13 slides |
| Lecture 14 | Decision trees for regression and classification; recursive partitioning, splitting criteria, overfitting, and cost-complexity pruning; node impurity, ID3 and CART; random forests, bootstrap sampling, random feature selection, majority voting, and variance reduction | Lecture 14 slides |
| Lecture 15 | Bernoulli and Gaussian Naive Bayes; conditional independence and class scores; k-nearest neighbours, feature scaling, and dimensionality; Mahalanobis metric learning and large-margin nearest neighbours (LMNN); Siamese networks, contrastive and triplet losses, and supervised contrastive learning | Lecture 15 slides |
| Lecture 16 | Supervised PCA; maximum mean discrepancy (MMD) and the Hilbert-Schmidt independence criterion (HSIC); kernel centering and empirical HSIC; label kernels and the eigenvalue solution; comparison with ordinary PCA, prediction, metric-learning connections, and limitations of linear projections | Lecture 16 slides |
| Lecture 17 | Decision trees; recursive partitioning, splitting criteria, overfitting, and pruning | Coming soon |
| Lecture 18 | Boosting; weak learners, adaptive reweighting, and stage-wise additive models | Coming soon |
| Lecture 19 | Bagging, random forests, variance reduction, and ensemble comparison | Coming soon |
| Lecture 20 | Semi-supervised learning, selected emerging methods, and integration of the course's main ideas | Coming soon |
Class examples
Datasets, code, and notebooks used for demonstrations and examples are collected here independently of the lecture schedule.
Classification · Handwritten digits
A 64 × 400 MATLAB matrix named X containing 400 vectorized 8 × 8 handwritten-digit images. Each column is one image: columns 1–200 are examples of the digit 2, and columns 201–400 are examples of the digit 3.
Plots images at specified 2-D coordinates. For example, after PCA or FDA, the handwritten digits can be superimposed on their projected points to reveal patterns in the two-dimensional space.
MATLAB plotting function (.m)LDA and QDA · PCA and FDA · Logistic regression
An interactive MATLAB demo reviewing LDA, QDA, PCA, FDA, and logistic regression using three synthetic classification datasets. It steps through feature extraction, decision boundaries, and model evaluation.
Feature maps · Kernel methods
Visualize a quadratic feature map, a separating plane in feature space, and the corresponding nonlinear boundary in input space. Compare explicit feature inner products with kernel evaluations. Run kernel_demo in MATLAB; no additional toolboxes are required.
Support vectors · Kernel methods
Compare linear, quadratic, and Gaussian kernels on the same data. See when a hard-margin separator exists and how the support vectors determine its boundary. Requires MATLAB with Optimization Toolbox.
Slack variables · Margin violations
Adjust the penalty for margin violations and see how the decision boundary changes on overlapping data. Select a point to inspect its slack and see whether it is a support vector. Requires MATLAB with Optimization Toolbox.
Regression and classification trees · Recursive partitioning
Explore split thresholds, constant predictions within regions, tree growth, overfitting, and impurity criteria. Run decision_tree_demo in MATLAB R2016b or later; no additional toolboxes are required.
Bootstrap sampling · Random feature selection · Majority voting
Compare identical trees, bagging, and random forests. Inspect sampled observations and candidate features, then click a point to see individual tree votes and the majority prediction. Run random_forest_demo in MATLAB R2016b or later; no additional toolboxes are required.
Cost-complexity pruning · Region merging
Merge regions by collapsing tree branches and see how fitted constants, fitting error, and the leaf-count penalty change. Adjust alpha to compare simpler and more detailed trees. Run pruning_demo in MATLAB R2016b or later; no additional toolboxes are required.
Final presentations
Register your group of up to four students now. Choose a presentation paper later, or propose another paper for approval.
Each group has 25 minutes total: approximately 15–20 minutes of prepared material and 5–10 minutes for discussion and questions. The split is flexible; questions are welcome throughout.
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 12 | Classification Under Local Differential Privacy with Model Reversal and Model Averaging |
| 4:25–4:50 p.m. | Group 1 | Probabilistic Multi-Dimensional Classification with Incomplete Data at the Prediction Time |
| 4:50–5:15 p.m. | Group 4 | Classification of high-dimensional data with spiked covariance matrix structure |
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 5 | Rank-Transformed Dissimilarity Profiles for High-Dimensional Classification |
| 4:25–4:50 p.m. | Group 6 | On the Normalization of Confusion Matrices: Methods and Geometric Interpretations |
| 4:50–5:15 p.m. | Group 3 | Instance-Level Costs for Nuanced Classifier Evaluation |
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 16 | Policy-Oriented Binary Classification: Improving (KD-)CART Final Splits for Subpopulation Targeting |
| 4:25–4:50 p.m. | Group 14 | A Statistical Theory of Overfitting for Imbalanced Classification |
| 4:50–5:15 p.m. | Group 11 | Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of View |
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 13 | Class-Conditional Distribution Balancing for Group Robust Classification |
| 4:25–4:50 p.m. | Group 2 | Active Learning with Foundation Model Priors: Efficient Learning under Class Imbalance |
| 4:50–5:15 p.m. | Group 7 | Structured Matrix Scaling for Multi-Class Calibration |
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 9 | Practical estimation of the optimal classification error with soft labels and calibration |
| 4:25–4:50 p.m. | Group 8 | Asymmetric Duos: Sidekicks Improve Uncertainty |
| 4:50–5:15 p.m. | Group 10 | Conformal Prediction for Long-Tailed Classification |
| Time | Group | Paper |
|---|---|---|
| 4:00–4:25 p.m. | Group 15 | Conformal Prediction in Hierarchical Classification with Constrained Representation Complexity |
| 4:25–4:50 p.m. | Unassigned | No group scheduled |
| 4:50–5:15 p.m. | Unassigned | No group scheduled |
A pending paper does not change a group’s scheduled presentation time.
The presentation is worth 10% of the final course grade. The weights below apply to the presentation mark.
| Criterion | What we assess | Weight |
|---|---|---|
| Motivation & problem | Explain why the work matters and clearly define the problem it addresses. | 20% |
| Main idea & technical understanding | Explain how the approach works, with enough technical depth to demonstrate understanding. Establish the main ideas before finer details. | 35% |
| Findings & interpretation | Explain the main results, what we learn from them, and relevant limitations. | 20% |
| Clarity & organization | Teach the essential ideas clearly and selectively. Plan approximately 15–20 minutes of prepared material within the 25-minute session. | 15% |
| Discussion & questions | Allow approximately 5–10 minutes for discussion. Invite questions throughout and respond thoughtfully, demonstrating understanding. | 10% |
You are assessed on understanding and explanation, not on covering every detail. Discussion is assessed by how you facilitate and respond to questions, not by how many questions classmates ask.
Final project
Register your group of up to four students now. Choose a project later, or propose your own for approval. Find project descriptions, starter materials, report templates and the marking scheme on the project page.
Coursework
Individual assignment · 100 marks
Download the assignment PDF and the data ZIP linked above. Extract the data ZIP before starting Question 6 and keep the four CSV filenames unchanged. Read the assignment instructions carefully; the Deliverables box at the end of each question specifies exactly what to submit.
WatIAM_A1.zip to the Assignment 1 Dropbox in Learn, replacing WatIAM with your Waterloo username. Include Q3.py, Q5.py, and Q6.py (or the corresponding .R or .m files), any required helper files, Q6_cv_results.csv, Q6_predictions.csv, and a short README giving run commands, software/dependencies, and random seeds.Use Python, R, or MATLAB and follow the programming requirements in the assignment PDF.
Generative AI use is permitted. After completing the assignment, open Learn and select Submit → Surveys → Assignment 1: Generative AI Use Survey. Everyone must complete the survey, including students who used no generative AI; in that case, answer No to all five questions. The survey is not anonymous so completion can be checked. Your survey responses do not affect your assignment grade. Open Learn.
You may be asked to complete a brief individual quiz or participate in an oral discussion about your assignment. This may include explaining your reasoning and solutions, answering related questions, or explaining and making small modifications to your submitted code. You must demonstrate your own understanding of all submitted work, including work developed with permitted generative AI assistance. These checks must be completed without generative AI or assistance from others. If you cannot demonstrate adequate understanding of your submitted work, your overall assignment mark may be reduced.
Submissions received up to 24 hours after the deadline receive a 10% penalty. Submissions more than 24 hours late receive a grade of zero.
Individual assignment · 100 marks
Download the assignment PDF and the data ZIP linked above. Extract the data ZIP before working on Questions 3, 5, and 6, and keep the supplied filenames and observation order unchanged. Read the assignment instructions carefully; the Deliverables box at the end of each question specifies what to submit.
WatIAM_A2.zip to the Assignment 2 Dropbox in Learn, replacing WatIAM with your Waterloo username. Include Q1.py, Q3.py, Q5.py, and Q6.py (or their .R or .m equivalents), all required helper and input files, Q1_first.csv, Q1_simulation.csv, Q3_results.csv, Q5_results.csv, Q6_results.csv, and a short README listing run commands, dependencies, random seeds, and solver settings. After extracting the ZIP, each main script must run without first running another question.Use Python, R, or MATLAB. Implement the requested RBF fits, risk calculations, kernel evaluations, and prediction rules yourself. Generic constrained-optimization or quadratic-programming solvers are permitted in Questions 3 and 6; construct their objectives and constraints yourself. Follow the detailed programming requirements in the PDF. Questions 2 and 4 require written answers only.
Generative AI use is permitted. After completing the assignment, open Learn and select Submit → Surveys → Assignment 2: Generative AI Use Survey. Everyone must complete the survey, including students who used no generative AI; in that case, answer No to all five questions. The survey is not anonymous so completion can be checked. Your survey responses do not affect your assignment grade. Open Learn.
You may be asked to complete a brief individual quiz or oral discussion about your assignment, including explaining your reasoning, answering related questions, or explaining and making small modifications to your code. You must demonstrate your own understanding of all submitted work, including work developed with permitted generative AI assistance. These checks must be completed without generative AI or assistance from others. If you cannot demonstrate adequate understanding, your overall assignment mark may be reduced.
Submissions received up to 24 hours after the deadline receive a 10% penalty on the earned mark. Submissions more than 24 hours late receive a grade of zero.
Individual assignment · 100 marks
Download the assignment PDF and the data ZIP linked above. Extract the data ZIP before working on Questions 2, 4, and 5, and keep the supplied filenames and observation order unchanged. Read README.md and README_CNN.md for the data conventions. The Deliverables box at the end of each question specifies what to submit.
WatIAM_A3.zip to the Assignment 3 Dropbox in Learn, replacing WatIAM with your Waterloo username. Include Q2.py, Q4.py, and Q5.py (or their .R or .m equivalents), all required helper and input files, Q2_results.csv, Q4_results.csv, Q5_results.csv, and a short README listing run commands, dependencies, seeds, and model settings. Each script must run independently after the ZIP is extracted.Use Python, R, or MATLAB. Neural-network libraries and automatic differentiation are permitted in Question 2. In Question 4, use a library base-tree learner, but implement bootstrap sampling and hard-vote aggregation yourself; a package such as R's ranger may be used for one tree at a time with internal resampling disabled. In Question 5, implement candidate selection, AdaBoost updates, and prediction yourself; do not use a ready-made boosting routine. Follow the detailed programming requirements in the PDF. Questions 1 and 3 require written answers only.
Generative AI use is permitted. After completing the assignment, open Learn and select Submit → Surveys → Assignment 3: Generative AI Use Survey. Everyone must complete the survey, including students who used no generative AI. The survey is not anonymous so completion can be checked. Your survey responses do not affect your assignment grade. Open Learn.
You may be asked to complete a brief individual quiz or oral discussion about your assignment, including explaining your reasoning, answering related questions, or explaining and making small modifications to your code. You must demonstrate your own understanding of all submitted work, including work developed with permitted generative AI assistance. These checks must be completed without generative AI or assistance from others. If you cannot demonstrate adequate understanding, your overall assignment mark may be reduced.
Submissions received up to 24 hours after the deadline receive a 10% penalty on the earned mark. Submissions more than 24 hours late receive a grade of zero.
Course tools
Use Piazza for announcements, lecture questions, clarifications, and class discussion.
Go to PiazzaUse Learn for graded work, submissions, private course documents, deadlines, and grades.
Open Waterloo LearnSign-in may be required. Students should use the course-specific links available in their enrolled Piazza and Learn accounts.
How the course runs
Public slides are posted with the tentative topics. Additional or restricted materials are shared through Learn.
Public assignment instructions are posted on this page. Submission links, feedback, and official grades are provided through Learn.
For assignment and coding questions, attend TA office hours. For course-content or final-project questions, attend instructor office hours. Use a private Learn message or email for personal matters.
Consult Learn and Piazza for authoritative dates, room information, schedule changes, and deadlines.