Interpretable Machine Learning – Christoph Molnar (Free Book) BSCS

BSCS students — and the many BS graduates who go on to MS-level Machine Learning coursework — can read the complete book “Interpretable Machine Learning: A Guide for Making Black Box Models Explainable” by Christoph Molnar free online at its official site. Now in its actively maintained 3rd edition (2025), it is the standard reference for the growing subfield of interpretable machine learning and explainable AI (XAI): once you already know how to train a model, this book teaches you how to open it up, explain its predictions, and check whether it is relying on the real signal or a shortcut.

The book is organized as a reference, not a story to read cover to cover. Five introductory chapters (Introduction, Interpretability, Goals of Interpretability, Methods Overview, Data and Models) set up the vocabulary and the two running examples used throughout. From there, the book’s own five parts move from classical, self-explaining models (linear regression, decision trees, rule lists) through model-agnostic local and global explanation methods (LIME, SHAP, Shapley values, partial dependence plots, accumulated local effects, permutation feature importance) to methods specific to neural networks (feature visualization, saliency maps, adversarial examples). Almost all of the book’s own worked examples use R, though every chapter’s own “Software and alternatives” section points to the equivalent Python package for readers who work in Python instead.

Book Overview

Chapter List

Chapter 1: Introduction

Difficulty: Easy · Key topics: Clever Hans predictors, why interpretability matters, brief history of interpretable ML, how to read the book

The chapter opens with the story of Clever Hans, a horse that appeared to do arithmetic but was actually reading its human questioner’s body language — the book’s running metaphor for a “Clever Hans predictor,” a machine learning model that achieves good test performance by relying on a shortcut rather than the real relationship. It gives three real examples (a whale-audio classifier that picked up recording artifacts, an image classifier that read text in photos, and a wolf-vs-dog classifier that used snow in the background) to argue that good test-set performance alone does not prove a model is trustworthy. It sketches interpretability’s history — interpretable linear regression already in the early 1800s, machine learning’s task-driven approach from the mid-20th century onward, and the post-2015 explosion of model-agnostic interpretability research — and closes by explaining that this is a reference book, best read chapter-by-chapter rather than cover to cover.

Key Points:

• A “Clever Hans predictor” (term from Lapuschkin et al., 2019) is a model that has learned a shortcut instead of the real signal.
• Real examples in the chapter: a whale classifier that used audio-file artifacts, an image classifier that read embedded text, and a wolf/dog classifier that used snow in the background.
• None of these shortcuts lowered test-set performance — that is exactly why they went unnoticed without interpretability tools.
• Linear regression has been used for quantitative modeling since the early 19th century; classical statistics has always depended on keeping models interpretable.
• Interpretable machine learning (also called Explainable AI) “really exploded” as a field around 2015, especially model-agnostic methods.
• The book recommends new readers start with the Interpretability, Goals of Interpretability, and Methods Overview chapters before anything else.

Memory Tip: Remember “Clever Hans” as your mental flag for any model that looks smart but might be exploiting a shortcut — before trusting a high accuracy number, ask “could this be a clever horse?”

Common Mistake: Treating a high test-set accuracy as proof the model learned the right relationship. Clever Hans predictors do not lower test performance — that is precisely why they slip past ordinary evaluation and need interpretability methods to catch.

Important Questions:

• What is a “Clever Hans predictor,” and which example from the chapter illustrates it best? A model that produces correct predictions by relying on a spurious shortcut rather than the real causal signal — for example, an image classifier trained to tell wolves from dogs that actually learned to detect snow in the background, since most training wolf photos had snowy backgrounds.
• Why can’t good test-set performance alone guarantee that a model is trustworthy? Because a shortcut like a background artifact or a recording artifact can produce correct predictions on the test set without the model learning the real, generalizable relationship, so the flaw only shows up once someone looks inside the model.

Chapter 2: Interpretability

Difficulty: Easy · Key topics: definitions of interpretability, interpretability vs explainability, why interpretability matters, human reasons for wanting explanations

This chapter defines interpretability using two working definitions: “the degree to which a human can understand the cause of a decision” (Biran & Cotton, also used by Miller) and “a method is interpretable if a user can correctly and efficiently predict the method’s results” (Kim et al.). It distinguishes interpretability from the related but stronger term “explainability,” following Roscher et al.: interpretability maps an abstract model concept into an understandable form, while explainability requires interpretability plus additional context. It then argues, following Doshi-Velez and Kim, that the demand for interpretability comes from an “incompleteness in problem formalization” — a single metric like accuracy is an incomplete description of most real tasks — and works through human reasons for wanting an explanation: curiosity and learning, finding meaning by resolving contradictions, and the trust that is needed when a decision affects someone’s life.

Key Points:

• Working definition: “Interpretability is the degree to which a human can understand the cause of a decision” (Biran & Cotton / Miller).
• Second definition (Kim et al.): “a method is interpretable if a user can correctly and efficiently predict the method’s results.”
• Per Roscher et al., interpretability maps an abstract model concept into understandable form; explainability is stronger — it requires interpretability plus additional context.
• The need for interpretability comes from an “incompleteness in problem formalization” (Doshi-Velez & Kim) — a single metric such as accuracy is an incomplete description of most real-world tasks.
• Human reasons for wanting explanations include curiosity/learning (the book’s berry-sickness example) and finding meaning by resolving a contradiction (the dog-bite example).
• Low-risk tasks (e.g. a movie recommender) or already extensively studied tasks (e.g. optical character recognition) may not need explanations at all.

Memory Tip: Hold the two definitions together as “cause vs. prediction”: interpretability is understanding the cause of a decision, or being able to correctly predict what the method will output next.

Common Mistake: Assuming “interpretable” and “explainable” always mean the same thing. The chapter is explicit that researchers don’t agree on a shared definition, and the book’s own pragmatic stance treats them as a fuzzy umbrella term rather than two precise synonyms.

Important Questions:

• What does Doshi-Velez and Kim’s phrase “incompleteness in problem formalization” mean, and why does it create a need for interpretability? It means a single evaluation metric, like classification accuracy, is an incomplete description of most real tasks, so knowing only the “what” of a prediction is not enough — you also need the “why” to fully solve the original problem.
• Give two of the human motivations for wanting an explanation described in this chapter, with their examples. Curiosity and learning — the example of feeling sick and learning that red berries caused it, updating one’s mental model; and finding meaning by resolving a contradiction — the example of a dog biting its owner for the first time, resolved by the vet’s explanation that the dog was under stress.

Chapter 3: Goals of Interpretability

Difficulty: Easy · Key topics: three goals (improve, justify, discover), Clever Hans debugging, stakeholder roles, recourse

Following Adadi and Berrada, the chapter organizes interpretability around three goals. Improving the model uses interpretability to debug Clever Hans shortcuts and find feature-engineering ideas — the author’s own example is a water-supply forecasting competition where feature importance revealed that snow in surrounding mountains mattered, leading to better snow-data feature engineering. Justifying the model and its predictions serves different stakeholders (creators, operators, executors, decision subjects, auditors, data subjects, from Tomsett et al.) — a rejected loan applicant needs “recourse,” and a medical-device company must justify its model to regulators. Discovering insights matters when the prediction itself is not the point — a churn model’s marketing team needs to know why a customer might leave, and a scientist studying almond yield needs to know how fertilizer actually affects the outcome, not just what the model predicts.

Key Points:

• Three goals of interpretability (after Adadi & Berrada): improve the model, justify the model and predictions, discover insights.
• Improving the model example: in a water-supply forecasting competition, feature importance revealed that mountain snow mattered, leading to better feature engineering and improved performance.
• Tomsett et al.’s six stakeholder roles: creators, operators, executors, decision subjects, auditors, data subjects.
• “Recourse” is a decision subject’s ability to contest a prediction (e.g. a rejected loan application) — it requires some justification for why the prediction was made.
• A medical device company relying on machine learning must justify the model to regulators (auditors) as part of the approval process.
• Discovering insights matters when the model is a means to understand a relationship, not just to predict — e.g. how fertilizer use affects almond yield.

Memory Tip: Remember “IJD”: Improve, Justify, Discover — and match each to who needs it most: Improve helps creators and operators, Justify helps auditors and decision subjects, Discover helps scientists.

Common Mistake: Assuming interpretability is only about debugging the model (goal 1). Justification (regulatory approval, a rejected applicant’s recourse) and insight discovery (understanding a real-world relationship) are equally real goals with entirely different audiences, and conflating them leads to building the wrong kind of explanation for the situation.

Important Questions:

• What are the three goals of interpretability introduced in this chapter, and give the book’s own example for “improve the model”? Improve the model, justify the model and predictions, and discover insights; the improve-the-model example is a water-supply forecasting competition where feature importance revealed mountain snow was important, leading to better snow-related feature engineering and improved performance.
• What is “recourse,” and why does it require interpretability? Recourse is a decision subject’s ability to contest a prediction that affects them, such as someone whose loan application was rejected; contesting the decision is only possible if there is some justification for why the model made that prediction in the first place.

Chapter 4: Methods Overview

Difficulty: Medium · Key topics: interpretable by design vs post-hoc, model-agnostic vs model-specific, SIPA, Rashomon effect, group-wise methods

This chapter gives the map the rest of the book fills in. It splits interpretability approaches into “interpretable by design” (the learning algorithm itself is constrained to produce interpretable models, like linear regression) and “post-hoc” (methods applied after training). Post-hoc methods split further into model-agnostic (ignore what’s inside the model, only study how output changes with input, following the SIPA principle: sample, intervention, predict, aggregate) and model-specific (e.g. Gini importance for random forests, odds ratios for logistic regression). It introduces the “Rashomon effect,” named after Kurosawa’s 1950 film, for the case where multiple explanations of the same prediction are each valid yet mutually incompatible, and shows that a global method can be turned into a “group-wise” or regional method just by applying it to subsets of the data. It ends by admitting the categories are blurred in practice.

Key Points:

• Two top-level categories: “interpretable by design” (the algorithm itself is constrained) vs “post-hoc” (applied after training).
• Post-hoc splits into model-agnostic (works for any model) and model-specific (e.g. Gini importance for random forests, odds ratios for logistic regression).
• Model-agnostic post-hoc methods follow the SIPA principle: Sample, Intervention, Predict, Aggregate.
• The Rashomon effect, named after the 1950 film, describes multiple equally valid but mutually incompatible explanations for the same prediction.
• A global method becomes a “group-wise” or regional method simply by applying it separately to subsets of the data.
• The chapter admits the categories are blurred — e.g. logistic regression’s coefficients need post-processing (into odds ratios or probabilities) before they are truly interpretable.

Memory Tip: Spell out SIPA in order every time you evaluate a new model-agnostic method: Sample the data, Intervene on it, get Predictions, Aggregate the results — permutation feature importance is the textbook example.

Common Mistake: Treating “by design” vs “post-hoc” as a strict, non-overlapping split. The chapter is explicit that the lines are blurred — logistic regression is technically interpretable by design, but its coefficients still need post-processing to become genuinely understandable at the probability level.

Important Questions:

• What does the SIPA principle stand for, and how does permutation feature importance follow it? Sample, Intervention, Predict, Aggregate; permutation feature importance samples the data, intervenes by permuting a feature’s values, gets model predictions on the permuted data, and aggregates the results into an error increase, which measures the feature’s importance.
• What is the “Rashomon effect,” and where does the name come from? It describes a situation where several explanations of the same prediction are each individually valid but mutually incompatible with one another; the name comes from Kurosawa’s 1950 film Rashomon, in which four witnesses give four different, equally plausible versions of the same murder.

Chapter 5: Data and Models

Difficulty: Easy · Key topics: bike rental dataset (regression), Palmer penguins (classification), feature correlation, mutual information

This chapter introduces the two datasets used throughout the rest of the book. The bike rental dataset (from Capital Bikeshare, Washington D.C.) has daily bike-rental counts as the regression target, with weather and seasonal features — the book takes the simplifying liberty of pretending the weather features are forecasts, so the task becomes predicting tomorrow’s rentals from a weather forecast, the season, and how many bikes were rented recently. The Palmer penguins dataset is used for a classification task, predicting a penguin’s sex from body measurements (with several models trained, including a logistic regression fit separately per species). Both sections study feature dependence — Pearson correlation for numerical features and normalized mutual information for categorical ones — because several interpretation methods introduced later behave differently, or need extra caution, when features are correlated.

Key Points:

• The book uses two running examples throughout: bike rental counts (regression) and Palmer penguin sex (classification).
• For the bike data, weather features are treated as if they were forecasts — a deliberate simplification so the prediction task makes sense.
• Feature dependence is measured with Pearson correlation for numerical features and normalized mutual information for categorical features.
• In the bike data, the strongest correlation is between the rental count two days earlier and temperature.
• In the penguin data, body mass and flipper length are strongly correlated — a fact later chapters must account for when interpreting model-agnostic methods.
• Several models are trained per dataset (e.g. a logistic regression fit separately for each penguin species) so later chapters can compare interpretation methods across different model types.

Memory Tip: Whenever a later chapter mentions “the bike data” or “the penguins” without re-explaining them, it means exactly these two datasets and models from this chapter — come back here if an example’s setup is unclear.

Common Mistake: Skipping this chapter and then being confused later by references to correlated features, “chonkiness,” or which model was trained on which dataset — the feature engineering and correlation analysis done here is assumed knowledge for every later method chapter.

Important Questions:

• Which two datasets does the book use throughout, and what does each one predict? The bike rental dataset (Capital Bikeshare, Washington D.C.), used for a regression task predicting the number of bikes rented; and the Palmer penguins dataset, used for a classification task predicting a penguin’s sex from body measurements.
• Why does the book measure correlation and mutual information between features in this chapter, before any interpretation method is introduced? Because several model-agnostic interpretation methods introduced later (like partial dependence plots and permutation feature importance) can be misleading when features are correlated, so knowing which features are dependent ahead of time is needed to interpret those later methods correctly.

Part 1: Interpretable Models

Chapter 6: Linear Regression

Part 1: Interpretable Models · Difficulty: Easy · Key topics: weighted sum of features, weight interpretation, weight/effect plots, encoding categorical features, Lasso, sparsity

A linear regression model predicts the outcome as a weighted sum of the input features, which is both its central strength and its central limitation. The chapter works through interpreting a weight depending on the feature type — numerical, binary, or categorical — and demonstrates this on the bike rental data, reading estimated weights straight from a fitted table. It introduces the weight plot (visualizing weights and their variance) and the effect plot (multiplying each weight by the actual feature value, which puts features on a comparable, meaningful scale and lets you explain an individual prediction). It closes on sparse linear models: when a dataset has hundreds or thousands of features, Lasso (least absolute shrinkage and selection operator) performs feature selection and shrinkage automatically to keep the model interpretable.

Key Points:

• A linear regression model’s prediction is a weighted sum: y = β₀ + β₁x₁ + … + βₖxₖ.
• For a numerical feature, increasing it by one unit changes the prediction by exactly its weight, holding all other features fixed.
• For a categorical feature, the weight represents the change in prediction from the reference category to the other category.
• The weight plot visualizes weights and their variance; the effect plot multiplies each weight by the instance’s actual feature value, so effects across differently-scaled features become comparable.
• Lasso regularization adds sparsity by shrinking many weights to exactly zero, performing automatic feature selection when there are many features.
• You should never directly compare raw weights of features measured on different scales — multiplying a feature by 1000 (kg to g) shrinks its coefficient by the same factor without changing its real effect.

Practice Tip: Fit a linear regression on the bike data, then build both a weight plot and an effect plot for the same model — watch how a feature with a small raw weight can still dominate the effect plot once its real range of values is accounted for.

Common Mistake: Comparing the raw weights of two features measured on different scales (e.g. temperature in Celsius vs. humidity as a fraction) and concluding the larger weight is the more important feature — the units alone can flip which weight looks bigger.

Important Questions:

• How should you interpret the weight of a numerical feature in a linear regression model, and how does that differ for a categorical feature? For a numerical feature, an increase of one unit changes the predicted outcome by exactly the feature’s weight, with all other features held fixed; for a categorical feature, the weight is the change in predicted outcome when moving from the reference category to the other category, again holding other features fixed.
• What problem does Lasso solve for linear regression models with many features, and how? It restores interpretability when there are hundreds or thousands of features (or more features than instances) by adding an L1 penalty that performs automatic feature selection and shrinks many coefficients to exactly zero, producing a sparser, more interpretable model.

Chapter 7: Logistic Regression

Part 1: Interpretable Models · Difficulty: Easy · Key topics: logistic function, log-odds interpretation, multiplicative effect, odds ratios

The chapter explains why plain linear regression fails for classification (predictions can fall outside [0,1] and residuals are not normally distributed) and introduces logistic regression as the fix: it squeezes a linear equation’s output between 0 and 1 using the logistic function. Weight interpretation changes fundamentally — because the model is not linear in probability, a one-unit increase in a feature does not shift the probability by a fixed amount, it changes the odds multiplicatively by a factor of exp(β). The worked example predicts whether a Chinstrap penguin is female from body measurements, including an engineered “chonkiness” feature. The chapter recommends model-agnostic methods like the partial dependence plot when you need to interpret the outcome directly on the probability scale.

Key Points:

• Linear regression is unsuitable for classification — predictions are not bounded to [0,1] and residuals are not normally distributed.
• Logistic regression squeezes a linear combination of features through the logistic function so predictions land between 0 and 1.
• On the log-odds scale the model is linear, but on the probability scale it is multiplicative — a one-unit increase in a feature multiplies the odds by exp(βj), not a fixed probability shift.
• An odds ratio above 1 means the feature increases the odds of the positive class; below 1 means it decreases them.
• To interpret the outcome directly as a probability (rather than log-odds or odds), the chapter recommends model-agnostic tools such as the partial dependence plot.
• Most of the strengths and limitations of linear regression carry over, plus logistic regression gives you a genuine probability, which is more informative than a bare classification.

Practice Tip: Fit a logistic regression and convert one feature’s weight to an odds ratio with exp(β) — then check your intuition by comparing it against that feature’s partial dependence plot on the probability scale.

Common Mistake: Interpreting a logistic regression weight as if it were a linear regression weight — saying “a one-unit increase in this feature raises the probability by β” is wrong; the true effect is multiplicative on the odds, not additive on the probability.

Important Questions:

• Why can’t linear regression be used directly for a classification problem, according to the chapter? Because its predictions are not constrained to the [0,1] probability range and the residuals of a 0/1 outcome are not normally distributed, both of which violate the assumptions the linear model relies on.
• What does it mean that logistic regression is “multiplicative” rather than linear on the probability scale? A one-unit increase in a feature does not change the predicted probability by a fixed amount; instead it multiplies the odds of the positive outcome by a constant factor, exp(βj), which is why the weight is best read as an odds ratio rather than a probability shift.

Chapter 8: GLM, GAM and More

Part 1: Interpretable Models · Difficulty: Medium · Key topics: non-Gaussian outcomes (GLMs), interactions, nonlinear effects (GAMs), assumption violations

Plain linear regression assumes a Gaussian outcome, no interactions between features, and a strictly linear relationship — assumptions that are routinely violated. This chapter surveys the main extensions built to fix each violation. Generalized Linear Models (GLMs) replace the Gaussian assumption with other distributions (e.g. Poisson for counts, categorical for classes) via a link function. Explicit interaction terms let a linear model capture that one feature’s effect depends on another (illustrated with temperature and workday status affecting bike rentals). Generalized Additive Models (GAMs) replace a feature’s fixed linear effect with a learned smooth function, capturing genuinely nonlinear relationships (like temperature’s effect on rentals) without discarding interpretability. The chapter stresses that each extension must be checked empirically against plain linear regression rather than applied by default.

Key Points:

• Plain linear regression assumes a Gaussian outcome, no feature interactions, and a strictly linear relationship — all three are commonly violated.
• GLMs generalize the outcome distribution (e.g. Poisson for counts, categorical for classes) via a link function, instead of assuming Gaussian errors.
• Adding an interaction term captures that one feature’s effect depends on another’s value — e.g. temperature’s effect on bike rentals differs on workdays vs. non-workdays.
• When you add an interaction term, you must also keep both main effects in the model, or the interaction weight becomes uninterpretable on its own.
• GAMs replace a feature’s fixed linear weight with a learned smooth function, capturing nonlinear effects (e.g. temperature) while staying interpretable.
• Check for these problems empirically — e.g. compare validation performance with and without an interaction term — rather than adding extensions by default.

Memory Tip: Match each broken assumption to its fix: wrong outcome distribution → GLM; missing interactions → interaction terms; wrong-shaped (nonlinear) effect → GAM.

Common Mistake: Interpreting an interaction weight in isolation, ignoring the main effects it’s paired with. The chapter is explicit: you cannot read “the interaction of temperature and workday decreases rentals by X” without accounting for the main effects that are held fixed in that statement.

Important Questions:

• What three assumptions of plain linear regression does this chapter address, and which extension fixes each one? Non-Gaussian outcomes are fixed by GLMs (via a link function to a different distribution); missing feature interactions are fixed by adding explicit interaction terms; and non-linear effects are fixed by GAMs, which replace a fixed weight with a learned smooth function.
• Why must a linear model that includes an interaction term also keep the corresponding main effects? Because the interaction weight alone cannot be interpreted meaningfully without the main effects — leaving them out or interpreting the interaction term in isolation produces a misleading statement about what changes when a feature value changes.

Chapter 9: Decision Tree

Part 1: Interpretable Models · Difficulty: Easy · Key topics: recursive splitting, tree interpretation, feature importance from splits, tree depth and interpretability

Where linear and logistic regression fail on nonlinear relationships or feature interactions, a decision tree splits the data repeatedly on cutoff values, producing subsets that are each simpler to model. Interpretation is direct: starting at the root, each edge in the path to a leaf is an AND-connected condition, and the leaf gives the predicted outcome (the mean of instances in that node). A feature’s overall importance is computed by summing how much each of its splits reduced the variance (or Gini index). The bike-data example uses a depth-2 tree that splits on recent rental count and temperature. The chapter is explicit that CART-style trees are scale-invariant: multiplying a feature by 1000 changes the split threshold’s value but not the tree’s structure or predictions.

Key Points:

• A decision tree predicts by recursively splitting the data on cutoff values in the features, producing distinct subsets (leaves).
• Reading a tree: each path from root to leaf is an AND of conditions, and the leaf’s value is the mean outcome of the instances that land there.
• A feature’s overall importance sums how much every split using that feature reduced the variance (regression) or Gini index (classification).
• Trees are naturally good at capturing feature interactions, since a split’s effect always depends on which branch (i.e. which other feature values) you’re already in.
• Trees are invariant to monotonic transformations and the scale of a feature — converting kilograms to grams changes the split’s numeric threshold but not the tree’s structure.
• Fewer nodes and lower depth make a tree easier to interpret; many hyperparameters control depth and node count, sometimes indirectly, so pick ones that control them directly if interpretability matters.

Practice Tip: Train a depth-2 or depth-3 CART tree on the bike data and trace one full root-to-leaf path by hand, writing out its AND-condition in plain English before checking the leaf’s predicted value.

Common Mistake: Assuming a decision tree naturally handles smooth, linear relationships well — trees can only approximate a linear trend with a staircase of splits, which is inefficient and produces the characteristic lack of smoothness (a tiny input change near a split boundary can flip the prediction).

Important Questions:

• How do you read a prediction out of a decision tree, from root to leaf? Starting at the root, follow the edge indicated by each split’s condition on the instance’s feature values (all conditions along the path are connected by AND) until you reach a leaf node, whose value (the mean outcome of training instances in that leaf) is the tree’s prediction.
• Why are decision trees described as scale-invariant, and what does that mean in practice? Because CART-style split-finding is invariant to monotonic transformations and the scale of a feature, so converting a feature from kilograms to grams changes the numeric value of the split threshold but leaves the tree’s structure and predictions unchanged.

Chapter 10: Decision Rules

Part 1: Interpretable Models · Difficulty: Medium · Key topics: IF-THEN rules, OneR baseline, sequential covering (RIPPER), Bayesian Rule Lists, sparsity

A decision rule is a simple IF-THEN statement: a condition (antecedent) and a prediction. The chapter starts with OneR, the simplest rule-induction algorithm, which picks the single feature carrying the most information about the outcome and creates one rule per unique value of that feature — a genuinely useful baseline to compare more complex models against. It then covers sequential covering (illustrated with RIPPER), a general recipe that repeatedly learns one rule at a time to build a decision list covering the whole dataset, and Bayesian Rule Lists, which first pre-mine frequent patterns as candidate rule conditions and then learn a decision list from that candidate pool. Throughout, the chapter stresses that IF-THEN rules are only genuinely interpretable when the rule set stays short and each rule’s condition stays short (no more than about three clauses).

Key Points:

• A decision rule has the form IF condition THEN prediction; a single rule or a combination forms the model.
• OneR selects the single feature that carries the most information about the outcome and builds one rule per unique value of that feature — it makes a strong, simple baseline.
• Sequential covering (e.g. RIPPER) repeatedly learns one rule at a time, removing the instances it covers, until a decision list explains the whole dataset.
• Bayesian Rule Lists first pre-mine frequent candidate patterns from the data, then learn a decision list from that candidate pool.
• IF-THEN rules are the most interpretable model class in the book — but only when the rule count is small and each rule’s condition has at most around three clauses.
• The research on IF-THEN rules focuses almost entirely on classification and neglects regression, a genuine limitation of the whole model family.

Practice Tip: Run OneR on a small labeled dataset and manually verify its chosen feature by computing, for each feature, the total classification error its per-value rules would produce — OneR should pick the lowest.

Common Mistake: Trusting a “one rule” model without validating it on held-out data, just because it looks trivially simple — the chapter is explicit that even OneR needs a separate validation set to confirm it generalizes, and that ties between features need an explicit tie-breaking rule.

Important Questions:

• How does the OneR algorithm choose which feature to build its rules from? It considers every feature, computes the classification error that a rule-per-unique-value model built from that one feature alone would produce, and selects the feature with the lowest total error (breaking ties by taking the first feature or the lowest chi-squared p-value).
• What is the general recipe of sequential covering, and name one algorithm from that family in the chapter? It repeatedly learns a single rule that covers as much of the remaining data as possible, removes the instances that rule covers, and repeats until a decision list (or set) covers the entire dataset; RIPPER is the algorithm the chapter uses as its worked example.

Chapter 11: RuleFit

Part 1: Interpretable Models · Difficulty: Medium · Key topics: rule generation from trees, sparse linear model with Lasso, interaction detection, interpretation as a linear model

RuleFit combines the interpretability of linear models with automatic interaction detection. It works in two steps: first, it generates a large pool of candidate decision rules by extracting them from the splits of an ensemble of decision trees; second, it fits a sparse linear model (via Lasso) using both the original features and the new rule-features (each rule is a binary feature, 1 if all its conditions hold, 0 otherwise). Because the final model is a linear combination, interpretation is exactly like a normal linear model, except some of the “features” are decision rules that capture interactions the plain linear model could never represent on its own. The chapter recommends keeping each rule to 1–3 conditions for interpretability, since Lasso alone can still leave you with too many non-zero rules.

Key Points:

• RuleFit has two steps: generate candidate rules from the splits of a tree ensemble, then fit a sparse linear model (Lasso) on the original features plus the rules.
• Each rule is a binary feature: 1 if all its conditions are met for an instance, 0 otherwise.
• Because the final model is linear in the original features and the rule-features, interpretation follows the same rules as plain linear regression.
• RuleFit automatically captures feature interactions that a plain linear model would need explicit, hand-crafted interaction terms to represent.
• The chapter recommends keeping rule length to 1–3 conditions — interpretability degrades quickly as the number of non-zero-weighted rules grows.
• Extrapolation is a real risk: “holding all other features equal” is unreliable here because a single original feature can appear inside multiple rules at once, so changing it changes several model terms simultaneously.

Practice Tip: Fit RuleFit on the bike data and inspect the rules with non-zero Lasso weight — count how many conditions each surviving rule has, and flag any with more than three as a red flag for interpretability.

Common Mistake: Interpreting a RuleFit coefficient exactly like an ordinary linear regression coefficient, “holding everything else fixed.” A feature can appear both as a linear term and inside several rules at once, so changing that one feature’s value can move multiple model terms simultaneously — the ceteris-paribus assumption silently breaks.

Important Questions:

• What are the two steps of the RuleFit algorithm? First, generate a large pool of candidate decision rules from the splits of an ensemble of decision trees; second, fit a sparse linear model (using Lasso) on the original features together with the new rule-features, where each rule is a binary indicator that is 1 when all its conditions hold.
• Why does RuleFit recommend keeping each rule short (1–3 conditions), and what happens if you don’t? Because interpretability degrades as rules get longer and as more rules end up with a non-zero weight in the Lasso model — a rule with many conditions becomes hard for a person to read and verify, undermining the whole point of using rules instead of a black box.

Part 2: Local Model-Agnostic Methods

Chapter 12: Ceteris Paribus Plots

Part 2: Local Model-Agnostic Methods · Difficulty: Easy · Key topics: one-feature-at-a-time sensitivity, sparklines, comparing curves across models/instances, correlated features

Ceteris paribus (Latin for “other things equal”) plots show how one instance’s prediction changes as a single feature is varied while every other feature stays fixed. The algorithm is simple: build a grid of values for the feature of interest, predict the outcome at each grid point (with everything else held at the instance’s actual values), and plot the resulting curve. The chapter demonstrates this on a penguin classification example and warns that a value that looks realistic when you vary one feature can become an unrealistic, impossible combination once you consider it alongside the instance’s other feature values — a warning that applies to every ceteris paribus-style method in the book. CP plots can be compacted into minimalist “sparklines” and used creatively: comparing features, comparing models, comparing class probabilities, or comparing different data points side by side.

Key Points:

• A ceteris paribus (CP) plot varies one feature across a grid of values for a single instance, holding all other features fixed, and plots the resulting predictions.
• The algorithm: build an equidistant value grid for the feature, predict at every grid point with other features held at the instance’s real values, plot the curve.
• A CP-plotted combination can look realistic for one pair of features but be physically impossible once you check it against a third correlated feature.
• CP plots compact into minimalist “sparklines,” letting many features be compared side by side on a shared y-axis for one instance.
• You can be creative with comparisons: different features, different models or hyperparameters, different class probabilities, or different data points side by side.
• CP plots only vary one feature at a time, so they cannot show how two features interact with each other.

Practice Tip: Build a CP plot for one penguin instance and one feature, then check whether the range of values you plotted is actually realistic by cross-checking it against a correlated feature in the same instance.

Common Mistake: Trusting a ceteris paribus curve across its whole plotted range without checking feature correlation — the book’s own example shows a value that is realistic for one pair of features but becomes an impossible combination once a third, correlated feature is taken into account.

Important Questions:

• What is the algorithm behind a ceteris paribus plot, in the order the chapter presents it? Build an equidistant grid of values for the feature of interest, then for each grid value predict the outcome for the chosen instance with every other feature held fixed at its real value, and plot the resulting curve of predictions against the grid.
• Why can a ceteris paribus plot show a misleading picture for correlated features? Because changing only the feature of interest while holding correlated features fixed can create data points that look plausible for that one pair of features but are unrealistic or impossible once all the correlated features are considered together.

Chapter 13: Individual Conditional Expectation (ICE)

Part 2: Local Model-Agnostic Methods · Difficulty: Medium · Key topics: one line per instance, centered ICE, derivative ICE, heterogeneity detection

An ICE plot shows one line per instance in the dataset, each tracing how that instance’s prediction changes as a single feature is varied — effectively many ceteris paribus curves drawn on one plot. On the bike-rental example, most ICE lines run roughly parallel, suggesting no strong interaction; on the penguin classification example, coloring lines by another feature’s value can reveal interactions that a single average curve would hide. Because overlapping lines can start at very different predictions, the chapter introduces the centered ICE plot (anchoring every curve to the same starting point so only the change is compared) and the derivative ICE plot (plotting each curve’s slope, which makes heterogeneity easier to spot visually, since parallel original curves become a flat line of derivatives).

Key Points:

• An ICE plot draws one line per instance, showing how that instance’s prediction changes as one feature varies — it’s essentially many CP curves overlaid.
• Parallel ICE lines across the dataset suggest no strong interaction between the plotted feature and the others; diverging or crossing lines suggest one.
• Coloring ICE lines by the value of another feature is a simple way to visually surface interactions.
• The centered ICE plot anchors every curve to the same starting value, so you compare the change in prediction rather than the absolute starting level.
• The derivative ICE (d-ICE) plot shows each curve’s slope instead of its value, making it visually easier to spot heterogeneous (differently-shaped) relationships.
• ICE curves meaningfully display only one feature at a time, and (like CP plots) can include unrealistic combinations when the plotted feature is correlated with others.

Practice Tip: Plot ICE curves for the bike data’s humidity feature colored by another feature’s value, then switch to the centered ICE version and note how the visual picture of heterogeneity changes.

Common Mistake: Reading an uncentered ICE plot and assuming curves that start at different heights therefore behave differently — check the centered version first, since two curves can shift identically but simply start from different baseline predictions.

Important Questions:

• What does an ICE plot show that a single average curve (like a partial dependence plot) cannot? It shows one line per instance rather than one averaged line for the whole dataset, so it reveals heterogeneous relationships — cases where different instances respond very differently to the same feature change — that an averaged curve would hide.
• What problem does the centered ICE plot solve, and how? It solves the problem that ICE curves can be hard to compare when they start at very different predicted values; it anchors every curve to the same starting point (typically the leftmost grid value) and plots only the change from that point, so only the shape of the change is compared, not the absolute starting level.

Chapter 14: LIME

Part 2: Local Model-Agnostic Methods · Difficulty: Medium · Key topics: local surrogate models, tabular/text/image variants, kernel width problem

Local Interpretable Model-agnostic Explanations (LIME) explains one prediction by fitting a simple, interpretable surrogate model (usually sparse linear regression) on data sampled in the local neighborhood of the instance being explained, weighted by proximity. The chapter details three variants: tabular data (samples are drawn from the training data’s feature distribution, which the chapter flags as problematic since it isn’t sampled strictly around the instance itself), text data (variations are created by randomly removing words, represented as binary bag-of-words features), and image data (perturbing whole interpretable “superpixels” rather than individual pixels, since single-pixel changes rarely shift a classifier’s output). Across all three, the biggest strength is that you can swap the underlying black-box model without changing the explanation method, but the choice of neighborhood/kernel width is called out as LIME’s biggest unsolved problem.

Key Points:

• LIME explains a single prediction by fitting a simple, interpretable surrogate model to data sampled around the instance and weighted by proximity to it.
• For tabular data, LIME’s samples actually come from the training data’s mass center rather than tightly around the instance — a limitation the chapter flags directly.
• For text, variations are created by randomly removing words, represented as binary bag-of-words features (word present/absent).
• For images, LIME perturbs groups of similar pixels (“superpixels”) rather than individual pixels, since single-pixel changes rarely move a classifier’s prediction.
• The biggest strength: you keep using the same kind of local surrogate model (e.g. a small decision tree) even if you swap the underlying black-box model entirely.
• The chapter calls the choice of neighborhood/kernel width “the biggest problem with LIME,” recommending you try several kernel settings and sanity-check the results each time.

Practice Tip: Run LIME on one tabular instance with two or three different kernel-width settings and compare the resulting explanations — if they disagree substantially, that instability is exactly the “unsolved problem” the chapter warns about.

Common Mistake: Trusting a single LIME explanation at its default kernel-width setting without testing whether the explanation is stable across other reasonable settings — the chapter is explicit that this is LIME’s biggest unsolved problem and recommends using it “only with great care.”

Important Questions:

• What is the core idea behind LIME, in one sentence? LIME explains an individual prediction by sampling data points in the local neighborhood of that instance, weighting them by proximity, and fitting a simple, interpretable surrogate model (typically sparse linear regression) to approximate the black box’s behavior in that neighborhood.
• Why does the chapter call the choice of neighborhood or kernel width “the biggest problem” with LIME on tabular data? Because there is no principled way to choose how “local” the sampled neighborhood should be, and for tabular data the samples actually come from the training data’s mass center rather than tightly around the instance, so different kernel-width choices can produce noticeably different, unstable explanations for the same prediction.

Chapter 15: Counterfactual Explanations

Part 2: Local Model-Agnostic Methods · Difficulty: Medium · Key topics: the four counterfactual criteria, Wachter et al. loss, Dandl et al. multi-objective search, Rashomon effect for counterfactuals

A counterfactual explanation answers “what is the smallest change to this instance’s features that would flip the prediction to a desired outcome?” The chapter presents two generation methods. Wachter et al.’s method minimizes a two-term loss: the distance between the counterfactual’s prediction and the desired outcome, plus the distance between the counterfactual and the original instance (so the result stays close to reality). Dandl et al. extend this to a four-objective search that also rewards sparsity (few features changed), plausibility (staying close to the data manifold), and being able to find several diverse counterfactuals rather than just one. The worked example uses the German Credit Risk dataset, generating counterfactuals for a support vector machine’s credit-approval predictions. A key practical issue the chapter raises: for any one instance you will usually find many valid counterfactuals (a version of the Rashomon effect), and deciding which ones to actually show the user is a real, unsolved design problem.

Key Points:

• A counterfactual explanation finds the smallest, most realistic feature change that would flip a prediction to a desired outcome — “if X had been different, the prediction would have been Y.”
• Wachter et al.’s method minimizes a loss with two terms: distance to the desired prediction, and distance to the original instance.
• Dandl et al.’s method optimizes four objectives at once: closeness to the desired prediction, similarity to the original instance, sparsity (few features changed), and plausibility (staying realistic).
• For classification tasks, define the counterfactual target in terms of predicted probability rather than the class label alone — it gives a finer, more controllable target.
• For any instance, you will usually find many valid counterfactuals, not one — a version of the Rashomon effect that makes deciding which to show the user a real design problem.
• Counterfactuals are considered very transparent: if you actually change the feature values as suggested, the prediction really does change — no extra assumptions are involved.

Practice Tip: Generate a counterfactual for one instance in the credit-risk example and manually verify it by re-running the model on the suggested feature values — confirm the prediction really does flip as promised.

Common Mistake: Reporting only the single “best” counterfactual found and treating it as the definitive answer — the chapter explicitly warns that many valid, very different counterfactuals usually exist for the same instance (a Rashomon effect), and cherry-picking one hides that ambiguity from the reader.

Important Questions:

• What are the two terms in Wachter et al.’s counterfactual loss function, and what does each one control? One term measures the distance between the counterfactual’s model prediction and the user-specified desired outcome (pulling the search toward the target class or probability); the other measures the distance between the counterfactual and the original instance (keeping the suggested change small and realistic).
• Why does the chapter recommend defining a classification counterfactual in terms of predicted probability rather than just the class label? Because probability gives a finer-grained, more controllable target than a bare class label — you can ask for a specific confidence level (e.g. push the probability to 0.5) rather than only asking for the class to flip, which is closer to how a real decision threshold works.

Chapter 16: Anchors (Scoped Rules)

Part 2: Local Model-Agnostic Methods · Difficulty: Medium · Key topics: rule-based local explanation, precision and coverage, beam search, complexity/runtime

The anchors method explains a single prediction with an IF-THEN rule that “anchors” it: as long as the rule’s condition holds, changing any other feature does not change the prediction. Because checking this exactly is infeasible for continuous or large input spaces, the authors use a beam-search approximation that grows candidate rules while tracking two statistics — precision (how often the anchor’s prediction actually holds when the condition is true) and coverage (what fraction of the input space the anchor applies to). The chapter’s tabular example turns the bike-rental regression into a classification task to demonstrate anchors on it. Compared with LIME, anchors’ output is a rule that is easier to interpret even for non-experts, states its own coverage explicitly, and clearly shows the reader where the explanation stops applying — something a linear surrogate model doesn’t do.

Key Points:

• An anchor is an IF-THEN rule that “anchors” a prediction: whenever the rule’s condition holds, changing other features does not change the prediction.
• Because checking this exactly is infeasible, anchors are found via beam search, tracking a target precision threshold β and a coverage estimate for each candidate rule.
• Precision is how often the anchor’s prediction actually holds when its condition is true; coverage is what fraction of the input space the rule applies to.
• A higher precision threshold gives stronger rules but typically reduces coverage — the chapter recommends experimenting with different thresholds to balance the two.
• Compared to LIME, anchors have an advantage: the rule states its own coverage explicitly, so the reader knows exactly where the explanation stops applying.
• Anchors depend heavily on a well-configured perturbation function, which must be explicitly designed per data domain (tabular, text, image).

Practice Tip: Turn a regression prediction into a binary classification target (as the chapter does with the bike data) and generate an anchor for one instance, then check its reported precision and coverage before trusting the rule.

Common Mistake: Treating an anchor’s precision threshold as a fixed default rather than tuning it — a high precision anchor can end up with almost no coverage, making the rule technically correct but practically useless for explaining anything beyond one narrow slice of the input space.

Important Questions:

• What does it mean for a rule to “anchor” a prediction? It means that whenever the rule’s condition (its set of feature-value constraints) holds for an instance, changes to the other, unconstrained feature values do not change the model’s prediction — the rule is sufficient on its own to fix the outcome.
• What are precision and coverage in the anchors method, and what is the practical trade-off between them? Precision measures how reliably the anchor’s stated prediction actually holds whenever its condition is true, and coverage measures what fraction of the input space the rule applies to; raising the precision threshold typically produces stronger, more reliable rules but reduces how much of the input space they cover.

Chapter 17: Shapley Values

Part 2: Local Model-Agnostic Methods · Difficulty: Hard · Key topics: coalitional game theory, fair payout distribution, value function, Efficiency/Symmetry/Dummy/Additivity, Monte Carlo estimation

Shapley values treat each feature value of an instance as a “player” in a coalition game where the prediction is the payout, and ask how to fairly distribute that payout among the players. The interpretation of the Shapley value for feature j is: how much that feature value contributed to this instance’s prediction, compared to the average prediction across the dataset. The chapter is careful to distinguish the Shapley value from a naive “remove the feature and see what changes” measure — it is not that. The Shapley value is the only attribution method satisfying four properties simultaneously: Efficiency (contributions sum exactly to the difference from the average prediction), Symmetry, Dummy, and Additivity. Because computing it exactly requires evaluating all 2^k feature coalitions, the chapter presents a Monte Carlo sampling approximation for practical use, and notes this problem is revisited with faster, model-specific estimators in the SHAP chapter that follows.

Key Points:

• Shapley values treat each feature value as a “player” in a coalition game where the prediction is the payout, and fairly distribute that payout among the players.
• Interpretation: the Shapley value for feature j is how much that feature value contributed to this prediction, compared to the average prediction across the dataset.
• The Shapley value is not the change in prediction if you simply removed the feature from the model — the chapter explicitly warns against that misreading.
• It is the only method satisfying four properties simultaneously: Efficiency, Symmetry, Dummy, and Additivity.
• Efficiency means the sum of all feature contributions exactly equals the difference between this instance’s prediction and the average prediction.
• Exact computation requires evaluating all 2^k possible feature coalitions, so in practice a Monte Carlo sampling approximation with M samples is used instead, trading variance for speed.

Memory Tip: Anchor the Efficiency property in your head as “the payout always adds up”: every feature’s Shapley value for an instance sums exactly to that instance’s prediction minus the dataset’s average prediction — nothing is left over.

Common Mistake: Confusing a Shapley value with a “leave-one-out” feature effect — the chapter explicitly says the Shapley value is the average contribution across many coalitions, not simply the drop in prediction you’d get from deleting the feature once.

Important Questions:

• What four properties, satisfied together, make the Shapley value unique among attribution methods? Efficiency, Symmetry, Dummy, and Additivity — the chapter notes the Shapley value is the only method in coalitional game theory that satisfies all four simultaneously, which is why it has such a strong theoretical justification.
• Why is exact Shapley value computation usually infeasible, and what does the book recommend instead? Because computing it exactly requires evaluating the prediction for all 2^k possible coalitions of the k features, which grows exponentially; the book recommends a Monte Carlo sampling approximation using M sampled coalitions, accepting some estimation variance for tractable computation time.

Chapter 18: SHAP

Part 2: Local Model-Agnostic Methods · Difficulty: Hard · Key topics: KernelSHAP, TreeSHAP, Permutation Method, SHAP feature importance, summary plot, dependence plot, interaction values

SHAP (SHapley Additive exPlanations) is an estimation framework and toolkit built around Shapley values. The chapter covers three ways to estimate them: KernelSHAP (the original, slow, model-agnostic estimator that links Shapley values to LIME), TreeSHAP (a fast, model-specific estimator for tree-based models — decision trees, random forests, gradient-boosted trees — that reduces the computational complexity dramatically compared to exact KernelSHAP), and the Permutation Method (an efficient model-agnostic estimator that samples cleverly by permuting features). Beyond explaining single predictions, SHAP values aggregate into global explanations: SHAP feature importance (average absolute Shapley value per feature), the summary plot (importance plus effect direction, combined in one chart), the dependence plot (feature value vs. Shapley value across all instances), and SHAP interaction values (splitting a pairwise effect into its true interaction component). The chapter’s worked example uses interventional TreeSHAP to explain a random forest’s penguin-sex predictions.

Key Points:

• SHAP is an estimation framework for Shapley values with three main estimators: KernelSHAP (slow, model-agnostic, the historical link to LIME), TreeSHAP (fast, tree-model-specific), and the Permutation Method (efficient, model-agnostic).
• TreeSHAP reduces computational complexity from KernelSHAP’s exponential-in-features cost to a complexity governed by tree count, leaf count, and tree depth — a large practical speedup for tree ensembles.
• SHAP feature importance averages the absolute Shapley value of a feature across all instances, then ranks features by that average.
• The summary plot combines importance (feature ranking) with effect direction (color = feature value), so you see both how important and how a feature pushes predictions in one chart.
• The dependence plot plots each instance’s feature value against its Shapley value for that feature, revealing the shape of the feature’s effect across the dataset.
• SHAP interaction values split a pairwise effect into the part explained by the two individual features and the part that is genuinely due to their interaction.

Practice Tip: Compute interventional TreeSHAP values for a tree-based model on one instance, then build both a summary plot and a dependence plot from the same values and compare what each chart reveals that the other doesn’t.

Common Mistake: Using slow KernelSHAP by default even when a faster, exact estimator applies — the chapter is explicit that KernelSHAP is impractical to run for many instances, and that TreeSHAP (for tree models) or the Permutation Method should be preferred whenever they apply.

Important Questions:

• Name the three SHAP estimation methods covered in the chapter and what each requires of the model. KernelSHAP is model-agnostic but slow and was the method that originally linked SHAP to LIME; TreeSHAP is fast but requires a tree-based model (decision trees, random forests, gradient-boosted trees); the Permutation Method is model-agnostic and efficient, sampling cleverly via feature permutations.
• What does a SHAP summary plot show that a plain SHAP feature importance ranking does not? The summary plot adds the direction and shape of each feature’s effect (via color, representing the feature’s value) on top of the importance ranking, so you can see not just which features matter most but also whether high or low values of each feature push the prediction up or down.

Part 3: Global Model-Agnostic Methods

Chapter 19: Partial Dependence Plot (PDP)

Part 3: Global Model-Agnostic Methods · Difficulty: Medium · Key topics: marginal effect of one or two features, averaging over other features, correlated-feature risk, PDP-based importance

A partial dependence plot (PDP) shows the marginal effect of one or two features on a model’s predicted outcome, averaged over all other features’ values in the dataset. It can reveal whether a relationship is linear, monotonic, or something more complex. Because the underlying computation forces every instance to take on the plotted feature value(s) regardless of what its correlated features actually are, the chapter’s central warning is that PDPs become misleading when the plotted feature is strongly correlated with others — they can include physically unrealistic combinations. The chapter also covers PDP-based feature importance (a flat PDP curve implies low importance, a highly varying one implies high importance) and recommends reducing a model’s own internal interactions (e.g. shallower trees, monotonicity constraints) to make its PDPs more trustworthy, not just easier to compute.

Key Points:

• A PDP shows the average marginal effect of one or two features on the prediction, averaged over the actual values of all other features.
• A PDP can reveal whether a relationship is linear, monotonic, or more complex — e.g., applied to a linear regression model, the PDP is always a straight line by construction.
• The realistic maximum is two features per PDP, since anything beyond that cannot be meaningfully visualized in 2 or 3 dimensions.
• Correlated features are the central risk: a PDP can average in unrealistic, physically impossible combinations of feature values when the plotted feature correlates with others.
• PDP-based feature importance measures a feature’s importance as how much its PDP curve deviates from its own average — a flat curve means low importance.
• Reducing a model’s internal feature interactions (shallower trees, monotonicity constraints) makes its PDPs both simpler to compute and more trustworthy to interpret.

Practice Tip: Plot a PDP for two correlated features in the penguin data and compare it against the same two features’ ALE plot — the difference between the two curves is a direct, visible demonstration of the correlation problem.

Common Mistake: Interpreting a PDP for a feature that is strongly correlated with another without checking that correlation first — the plot may be averaging over unrealistic, physically impossible feature combinations, which the chapter calls out as the method’s central limitation.

Important Questions:

• What does a partial dependence plot show, and what is the realistic limit on how many features it can display at once? It shows the average marginal effect of one or two chosen features on the model’s prediction, averaged over the actual dataset values of every other feature; realistically at most two features can be shown, since three or more cannot be meaningfully visualized on paper or a screen.
• Why does the chapter warn against trusting a PDP for a feature that is strongly correlated with other features? Because computing a PDP forces every instance in the dataset to take on the plotted feature value regardless of its actual correlated features, which can generate unrealistic, unlikely, or even physically impossible combinations of feature values and bias the resulting curve.

Chapter 20: Accumulated Local Effects (ALE)

Part 3: Global Model-Agnostic Methods · Difficulty: Hard · Key topics: unbiased alternative to PDP, difference-based averaging, estimation via intervals, categorical ALE

Accumulated Local Effects (ALE) plots describe how a feature influences predictions on average, and are presented as a faster, unbiased alternative to PDPs when features are correlated. The key theoretical difference: PDP averages predictions directly, while ALE averages differences in predictions within small local intervals of the feature, then accumulates (sums) those local differences into a global curve. This local-differencing trick avoids the unrealistic combinations that break PDPs, because ALE only ever asks the model about instances within a narrow, realistic interval of the correlated feature space. The chapter constructs a scenario purpose-built to break PDPs and shows ALE handling it correctly, then applies both to the real bike-rental data, noting that in many real applications ALE and PDP end up looking similar despite correlation, in which case interpreting the simpler PDP is fine. ALE also scales better computationally, at roughly O(n) versus PDP’s cost.

Key Points:

• ALE plots describe a feature’s average effect on the prediction, and are recommended as a faster, unbiased alternative to PDPs specifically because of how they handle correlated features.
• The key difference from PDP: PDP averages predictions directly; ALE averages differences in predictions within small local intervals of the feature, then accumulates those local differences into a global curve.
• Because ALE only evaluates the model within narrow, realistic intervals, it avoids the unrealistic feature combinations that make PDPs untrustworthy under correlation.
• ALE scales roughly O(n) with dataset size, since the largest possible number of intervals is bounded by the number of unique feature values, making it faster than PDP in practice.
• For categorical features with a meaningful natural order, use that order when computing ALE — the accumulation only makes sense along an ordered axis.
• In many real applications, ALE and PDP curves look similar even under correlation; when they agree, interpreting the simpler PDP is fine.

Practice Tip: Reproduce the chapter’s synthetic scenario (two strongly correlated features and a model that does something unusual at an unlikely combination), then plot both PDP and ALE for it side by side to see the correlation bias directly.

Common Mistake: Interpreting an ALE curve’s effect across an interval as if it described what happens as you smoothly vary the feature for a typical instance — when features are strongly correlated, the ALE curve at the extreme ends is estimated from very few, possibly unrepresentative data points, and reading too much into that region is a genuine misinterpretation the chapter calls out.

Important Questions:

• What is the key computational difference between how a PDP and an ALE plot are built? A PDP averages the model’s raw predictions directly across the dataset for each value of the plotted feature; an ALE plot instead averages the differences in predictions within small local intervals of the feature and then accumulates (sums) those local differences into a single global curve.
• Why does ALE avoid the correlated-feature problem that affects PDPs? Because ALE only evaluates the model on instances within a narrow local interval of the feature’s actual, realistic range at each step, rather than forcing every instance in the dataset to take on a single fixed feature value regardless of its correlated features, so it never has to average in unrealistic combinations.

Chapter 21: Feature Interaction

Part 3: Global Model-Agnostic Methods · Difficulty: Hard · Key topics: Friedman’s H-statistic, two-way and total interaction strength, partial dependence decomposition

When features interact, a prediction cannot be written as the simple sum of individual feature effects — the effect of one feature depends on the value of another. This chapter presents Friedman’s H-statistic, which measures interaction strength in two forms: a two-way measure (how much two specific features interact with each other) and a total measure (how much one feature interacts with all other features combined). Both are grounded in partial-dependence decomposition and expressed as the share of variance explained by the interaction, which makes the statistic dimensionless and comparable across features and even across different models. The worked example measures interaction strength in a random forest predicting penguin sex, finding body mass has the highest overall interaction strength with the other features.

Key Points:

• When features interact, the prediction cannot be decomposed into a simple sum of each feature’s individual effect — the effect of one depends on another’s value.
• Friedman’s H-statistic comes in two forms: a two-way measure (interaction strength between two specific features) and a total measure (one feature’s interaction with all others combined).
• The H-statistic is grounded in partial-dependence decomposition and defined as the share of variance explained by the interaction — making it dimensionless.
• Being dimensionless means the H-statistic is comparable across different feature pairs and even across entirely different models.
• In the penguin-sex random forest example, body mass had the highest total interaction strength with the other features.
• The H-statistic is computationally expensive, since it involves estimating marginal distributions, and its estimates carry real variance depending on how many data points are sampled.

Memory Tip: Keep the two H-statistic flavors straight by their question: the two-way H-statistic answers “how much do these two specific features interact?”; the total H-statistic answers “how much does this one feature interact with everything else combined?”

Common Mistake: Treating the H-statistic as free to compute at scale — it is genuinely expensive because it requires estimating marginal distributions, and its estimate has real sampling variance, so results can shift noticeably between runs on a subsampled dataset.

Important Questions:

• What is the difference between the two-way and the total H-statistic? The two-way H-statistic measures how strongly two specific features interact with each other, while the total H-statistic measures how strongly one feature interacts with all the other features in the model combined, rather than with any one feature in particular.
• Why is Friedman’s H-statistic described as dimensionless, and why does that matter? It is defined as the share of variance explained by the interaction rather than in the prediction’s original units, which means H-statistic values are comparable across different feature pairs and even across entirely different models, unlike a raw variance number would be.

Chapter 22: Functional Decomposition

Part 3: Global Model-Agnostic Methods · Difficulty: Hard · Key topics: high-dimensional function as sum of components, functional ANOVA, generalized functional ANOVA, decomposing tree ensembles

A supervised machine learning model can be viewed as a high-dimensional function; functional decomposition deconstructs that function into a sum of an intercept, individual feature effects, and interaction terms between features. The chapter frames this as a “key concept” that underlies many other interpretation methods in the book, not a stand-alone technique. It covers functional ANOVA (requiring the prediction function to be square-integrable, and decomposing under the assumption of independent features), a generalized version for dependent (correlated) features, and shows that ALE plots are themselves a form of functional decomposition, with components that are “pseudo-orthogonal” rather than strictly orthogonal like functional ANOVA’s. It also covers a specialized decomposition procedure for tree ensembles (e.g. XGBoost models) involving aggregation and purification steps, and connects the whole topic back to generalized additive models, which build the decomposition into training instead of extracting it afterward.

Key Points:

• Functional decomposition rewrites a high-dimensional prediction function as a sum: an intercept, individual feature effects, and interaction terms between features.
• The chapter treats this as a foundational concept underlying many other methods in the book, not a stand-alone interpretation technique on its own.
• Functional ANOVA requires the prediction function to be square-integrable and typically assumes independent features for its decomposition to be valid.
• A generalized version of functional ANOVA extends the idea to dependent (correlated) features, but naive use of the uniform-integration assumption can create unrealistic extrapolated data.
• ALE plots are themselves a form of functional decomposition — summing the intercept and all 1D/2D ALE components reconstructs the prediction function, though the components are “pseudo-orthogonal” rather than strictly orthogonal.
• Generalized additive models take the opposite approach: instead of decomposing a trained complex function afterward, they build the decomposed structure directly into how the model is trained.

Memory Tip: Treat functional decomposition as the theory underneath the plots you already know: a PDP is one term of the decomposition, the H-statistic measures an interaction term, and a GAM builds the decomposition in from the start instead of extracting it afterward.

Common Mistake: Assuming functional decomposition scales cleanly to high-order interactions — the chapter is explicit that the concept quickly hits practical limits beyond two-feature interactions, both because higher-order components can’t be easily visualized and because the computation becomes intractable.

Important Questions:

• What is functional decomposition, in the terms this chapter uses? It is the technique of deconstructing a high-dimensional machine learning prediction function into a sum of simpler components: an intercept, individual feature effects, and interaction terms between two or more features, so that each component can be studied on its own.
• In what sense are ALE plots themselves a form of functional decomposition? Because summing the intercept together with all the 1D and 2D ALE components reconstructs the original prediction function, exactly as functional decomposition requires, though the resulting components are only “pseudo-orthogonal” rather than strictly orthogonal, unlike functional ANOVA’s components.

Chapter 23: Permutation Feature Importance

Part 3: Global Model-Agnostic Methods · Difficulty: Medium · Key topics: shuffle-and-measure-error-increase, use unseen data, conditional PFI, group-wise PFI

Permutation feature importance (PFI) measures a feature’s importance by shuffling its values (breaking its relationship with the true outcome) and measuring how much the model’s prediction error increases — a feature is “important” if shuffling it makes predictions noticeably worse. The chapter stresses estimating PFI on data the model was not trained on, since computing it on training data can falsely inflate the importance of features the model simply overfit to. It also introduces conditional feature importance, a variant that addresses PFI’s own correlated-feature problem (plain shuffling creates unrealistic data points when features are dependent), and shows a group-wise PFI example on the penguin data, permuting within each species separately so that body-mass measurements aren’t mixed across species that have very different natural size ranges.

Key Points:

• PFI measures a feature’s importance by permuting (shuffling) its values and measuring how much the model’s prediction error increases as a result.
• PFI should be estimated on data the model was not trained on — computing it on training data can falsely highlight overfit, irrelevant features as important.
• PFI works with error metrics where smaller is better (e.g. MAE) and with metrics where larger is better (e.g. accuracy, AUC), as long as you correctly flip the ratio or difference direction.
• Plain (marginal) PFI has a correlation problem: permuting a feature that is correlated with others creates unlikely, unrealistic data points, inflating or distorting the importance estimate.
• Conditional PFI addresses this by permuting within groups defined by correlated features, and typically gives strongly correlated features a much lower importance than marginal PFI does.
• PFI is linked to the model’s error, which is appropriate for some goals but not others — if you want to know how much a feature affects predictions regardless of accuracy, a different method may be more appropriate.

Practice Tip: Compute PFI for the bike-data model twice — once on training data and once on held-out data — and compare the two importance rankings; the training-data version should look artificially inflated for features prone to overfitting.

Common Mistake: Computing permutation feature importance on the model’s own training data. This routinely inflates the apparent importance of features the model has overfit to, since shuffling them destroys a relationship the model only learned by memorizing the training set, not one that generalizes.

Important Questions:

• How does permutation feature importance decide that a feature is “important”? It shuffles (permutes) that feature’s values across the dataset, breaking its real relationship with the outcome, then re-measures the model’s prediction error; if the error increases noticeably compared to the unpermuted data, the feature is considered important, because the model relied on it.
• Why does the chapter recommend computing PFI on unseen (not training) data? Because computing it on the training set can falsely highlight irrelevant features as important due to the model overfitting to spurious patterns in that exact data, giving an overly optimistic and misleading picture of which features actually generalize as important.

Chapter 24: Leave One Feature Out (LOFO) Importance

Part 3: Global Model-Agnostic Methods · Difficulty: Medium · Key topics: retrain without a feature, compare performance drop, LOFO vs. PFI, feature selection use

Leave One Feature Out (LOFO) importance measures a feature’s importance by retraining the model without it and comparing predictive performance to the full model: if dropping a feature makes performance worse, it was important; if performance stays the same or improves, the feature was not helping (or was actively hurting). This is fundamentally different from most other methods in the book, since it requires retraining rather than reusing the already-trained model. On the bike-rental example, temperature and the previous day’s rental count come out as the most important features, and a feature with negative LOFO importance is flagged as a genuine candidate for removal. The chapter positions LOFO as excellent for feature selection but weaker than permutation feature importance for understanding one specific, already-trained model, since retraining changes the model itself.

Key Points:

• LOFO importance retrains the model with one feature removed and compares predictive performance against the full model — a real performance drop means the feature was important.
• Unlike most methods in the book, LOFO requires retraining the model for every feature tested, not just reusing the already-trained model.
• A feature with LOFO importance at or below zero (performance stays the same or improves without it) is a genuine candidate for removal from the model.
• On the bike-rental example, temperature and the previous rental count were the most important features by LOFO.
• LOFO is well suited to feature selection, since it directly measures each feature’s contribution to model performance.
• LOFO is not the best choice for auditing one specific, already-trained model, because retraining without a feature can produce a genuinely different model, not just remove one input from the same model.

Practice Tip: Run LOFO on the bike-data model for three or four features, note which (if any) come out with importance at or below zero, and then check whether removing that feature and retraining actually improves validation performance.

Common Mistake: Using LOFO to explain one specific, already-deployed model’s behavior. Because LOFO retrains a new model for every feature removed, a low or negative LOFO score tells you about feature selection, not necessarily about what your one actual, currently-deployed model relies on.

Important Questions:

• How does LOFO importance measure whether a feature is important, and how does this differ from most other methods in the book? It retrains the model entirely without that feature and compares the resulting predictive performance to the full model — a real drop in performance means the feature mattered; unlike most methods in the book, which reuse the single already-trained model, LOFO requires retraining a new model for every feature tested.
• What does a LOFO importance value at or below zero suggest about a feature? It suggests the feature is not helping model performance (zero) or is actively hurting it (negative), and the chapter notes such a feature is a genuine candidate for removal, since dropping it does not worsen, and may even improve, predictive performance.

Chapter 25: Surrogate Models

Part 3: Global Model-Agnostic Methods · Difficulty: Easy · Key topics: global surrogate, interpretable model approximating a black box, R-squared cutoff problem

A global surrogate model is an interpretable model (e.g. a decision tree) trained to approximate the predictions of an entire black-box model, so you can draw conclusions about the black box by studying the surrogate instead. The chapter frames this the same way engineers use cheap approximations of expensive simulations. Its worked example fits a CART decision tree as a surrogate for a support vector machine predicting bike rentals. The core practical limitation is that any conclusions you draw describe the surrogate’s approximation of the model, not the underlying data or ground truth, and there is no settled rule for how good the surrogate’s fit (e.g. R-squared) needs to be before you can trust its explanations — the chapter poses this openly as an unresolved question (80%? 50%? 99%?).

Key Points:

• A global surrogate model is an interpretable model trained to approximate the predictions of an entire black-box model, so studying the surrogate tells you about the black box.
• The idea mirrors engineering surrogate models: when the real outcome (or a complex model) is expensive or hard to work with directly, a cheap, faster approximation stands in for it.
• The worked example trains a CART decision tree to approximate a support vector machine predicting bike rentals.
• Surrogate models are flexible: any interpretable model can be used as the surrogate, and the underlying black-box model can be swapped without changing the surrogate’s shape.
• Conclusions from a surrogate model are about the model’s behavior, not the ground-truth data, since the surrogate never sees the real outcome, only the black box’s predictions.
• There is no agreed cut-off for how well the surrogate must fit (measured e.g. by R-squared) before its explanations can be trusted — the book explicitly leaves this open.

Practice Tip: Fit a CART surrogate to a black-box model’s predictions, record its R-squared fit to the black box, and only then decide for yourself whether that fit is good enough to trust the surrogate’s explanation of any one prediction.

Common Mistake: Presenting a surrogate model’s explanation as if it describes the real data or ground truth. A surrogate model only ever approximates the black box’s predictions — it never sees the true outcome, so its conclusions are strictly about the model being approximated, not about reality.

Important Questions:

• What is a global surrogate model, and what is it used for? It is an interpretable model (such as a decision tree) trained to predict the outputs of a black-box model rather than the true outcome, used so that studying the simpler, interpretable surrogate lets you draw conclusions about how the black-box model behaves.
• What open question does the chapter raise about trusting a surrogate model’s explanations? It raises the unresolved question of how good the surrogate’s fit to the black box (e.g. its R-squared) needs to be before its explanations can be trusted — the chapter explicitly notes there is no settled cut-off, posing 80%, 50%, and 99% as equally unresolved candidates.

Chapter 26: Prototypes and Criticisms

Part 3: Global Model-Agnostic Methods · Difficulty: Medium · Key topics: MMD-critic, representative vs. poorly-represented instances, greedy search, user study evidence

A prototype is a data instance that is representative of the overall data; a criticism is an instance that the chosen prototypes fail to represent well. Together they give a more complete picture than prototypes alone, since criticisms flag exactly where the prototype set falls short. The chapter’s method, MMD-critic, works in three steps: choose how many prototypes and criticisms to find, greedily search for prototypes whose distribution matches the overall data distribution, then greedily search for criticisms among the points the chosen prototypes explain poorly. The worked example uses a handwritten-digit dataset, where the number of images per digit found as prototypes varies naturally because the search runs across the whole dataset rather than a fixed number per class. A user study cited in the chapter found participants performed image classification tasks best when shown prototypes and criticisms together, rather than just random class examples.

Key Points:

• A prototype is a data instance representative of the overall dataset; a criticism is an instance the chosen prototypes fail to represent well.
• The MMD-critic procedure: (1) choose the number of prototypes and criticisms wanted, (2) greedily search for prototypes matching the overall data distribution, (3) greedily search for criticisms among poorly-represented points.
• In the digit-recognition example, the number of prototype images per digit varies naturally, since the search runs across the entire dataset rather than reserving a fixed number per class.
• A cited user study found people performed best at an image-matching task when shown prototypes and criticisms together, compared to random class examples.
• You are free to choose the number of both prototypes and criticisms — there is no single correct count, only a trade-off between coverage and conciseness.
• The prototype/criticism distinction is based on a chosen cut-off (the prototype count) — too few prototypes pushes points that should be prototypes into the criticism set instead.

Memory Tip: Remember prototypes and criticisms as a matched pair: a prototype says “the data mostly looks like this,” a criticism says “but not this part — look here too.”

Common Mistake: Choosing too few prototypes and treating the resulting criticisms as genuinely unusual or anomalous data points — the chapter warns that with too few prototypes, points that would have been perfectly good prototypes get pushed into the criticism set purely because the cut-off was set too low, not because they’re truly atypical.

Important Questions:

• What is the difference between a prototype and a criticism in the MMD-critic method? A prototype is a data instance that is representative of the overall data distribution, chosen so that the set of prototypes as a whole resembles the real data; a criticism is a data instance that the chosen set of prototypes does not represent well, deliberately included to show where the prototypes fall short.
• What did the user study cited in the chapter find about showing prototypes and criticisms together? Participants who were shown prototypes and criticisms together performed better at visually matching images to their correct class than participants who were shown random images from each class instead.

Part 4: Neural Network Interpretation

Chapter 27: Learned Features

Part 4: Neural Network Interpretation · Difficulty: Medium · Key topics: feature visualization by activation maximization, Network Dissection, human concept alignment

Convolutional neural networks learn abstract features and concepts directly from raw pixels, and this chapter covers two ways to make those learned features explicit. Feature visualization finds, by optimization, the input image that maximally activates a chosen unit (a neuron, a channel, a layer, or a class-probability output) — a technique directly connected to the adversarial examples of a later chapter, since both are optimization problems that maximize a unit’s activation, differing mainly in which class they target and what image they start from. Network Dissection quantifies interpretability more directly: it gathers images with human-labeled visual concepts, measures how strongly each CNN channel activates on those images, and quantifies the alignment between channel activations and the human concepts — tested by the original authors across architectures like AlexNet, VGG, GoogleNet and ResNet, and datasets like ImageNet and Places365.

Key Points:

• Feature visualization finds, via optimization, the input that maximizes the activation of a chosen unit — a neuron, channel, layer, or the final class-probability output.
• Feature visualization and adversarial examples are closely related optimization problems: both maximize a unit’s activation, differing mainly in the target class and the starting image.
• Network Dissection has three steps: gather images with human-labeled visual concepts, measure CNN channel activations on those images, and quantify how well activations align with the labeled concepts.
• The original Network Dissection experiments tested multiple architectures (AlexNet, VGG, GoogleNet, ResNet) trained from scratch on multiple datasets (ImageNet, Places205, Places365).
• Feature visualization has revealed that early layers of image-recognition CNNs learn simple edge and texture detectors, with more abstract concepts emerging in later layers.
• A real limitation: many feature-visualization images are not interpretable at all, showing abstract patterns with no clear human concept or word attached to them.

Memory Tip: Connect feature visualization to adversarial examples with one sentence: both are “find the input that maximizes activation for a target” — feature visualization targets a unit to understand it, adversarial examples target a wrong class to fool the model.

Common Mistake: Assuming every feature-visualization image will be human-interpretable. The chapter is explicit that many of these images show abstract patterns with no clear human concept attached — treating an uninterpretable visualization as a failure of the method, rather than a real and expected outcome, is the common mistake.

Important Questions:

• What is feature visualization, and how is it related to adversarial examples? Feature visualization is an optimization technique that finds the input image maximizing the activation of a chosen neural network unit; it is closely related to adversarial examples because both are activation-maximization optimization problems, differing mainly in which class is targeted (the correct class vs. an incorrect, adversarial one) and which image the optimization starts from.
• What are the three steps of Network Dissection? Gather images labeled with human visual concepts (objects, parts, textures, colors), measure how strongly each CNN channel activates on those images, and quantify how well each channel’s activation aligns with the labeled concepts.

Chapter 28: Saliency Maps

Part 4: Neural Network Interpretation · Difficulty: Medium · Key topics: pixel attribution methods, Vanilla Gradient, Grad-CAM, SmoothGrad, fragility of explanations

Pixel attribution (saliency map) methods highlight which pixels of an image mattered for a neural network’s classification. The chapter surveys several: Vanilla Gradient, the earliest approach, backpropagates the gradient of the target class’s loss all the way to the input pixels; DeconvNet reverses the network’s filtering, pooling and activation operations, producing results close to Vanilla Gradient’s despite a very different-looking derivation; Grad-CAM backpropagates only to the last convolutional layer, producing a coarser but class-discriminative localization map, and Guided Grad-CAM combines it with a pixel-level method for sharper detail; SmoothGrad is not a stand-alone method but an add-on that reduces noise in any gradient-based explanation by averaging over many noise-perturbed copies of the input. The chapter’s most important finding is that pixel attribution explanations can be fragile: small, even imperceptible perturbations to an image can drastically change the explanation while leaving the model’s prediction unchanged, which is why the chapter recommends never relying on a single method’s saliency map alone.

Key Points:

• Pixel attribution (saliency map) methods highlight the pixels of an image that mattered most for a neural network’s prediction, under many different names in the literature.
• Vanilla Gradient backpropagates the target class’s loss gradient all the way to the input pixels — the earliest and simplest approach.
• Grad-CAM backpropagates only to the last convolutional layer, giving a coarser but class-discriminative localization map; Guided Grad-CAM combines it with a pixel-level method for sharper detail.
• SmoothGrad is not a stand-alone method — it’s an extension that reduces noise in any gradient-based explanation by averaging over many noise-perturbed copies of the input image.
• Gradient-based pixel attribution methods are usually much faster to compute than model-agnostic methods like LIME or SHAP.
• A key finding: pixel attribution explanations can be fragile — small, even adversarial perturbations to an image can drastically change the explanation while leaving the model’s actual prediction unchanged.

Memory Tip: Sort the methods by “how far back does the gradient travel”: Vanilla Gradient and DeconvNet go all the way to the input pixels (fine detail, more noise); Grad-CAM stops at the last convolutional layer (coarse but class-specific); SmoothGrad is a noise-reduction add-on you can bolt onto either.

Common Mistake: Trusting a single saliency map as definitive proof of what the model “looked at.” The chapter’s own fragility finding shows that small, sometimes imperceptible input perturbations can produce a completely different-looking explanation while the prediction stays the same — always compare more than one method or perturbation before drawing a conclusion.

Important Questions:

• What is the key difference between how Vanilla Gradient and Grad-CAM produce their saliency maps? Vanilla Gradient backpropagates the gradient of the target class’s loss all the way back to the individual input pixels, producing fine-grained but often noisy maps, while Grad-CAM backpropagates only to the last convolutional layer, producing a coarser localization map that is more class-discriminative but less pixel-precise.
• What did the “fragility” finding discussed in the chapter show about pixel attribution methods? It showed that small, even imperceptible perturbations to an input image can produce drastically different saliency-map explanations while leaving the model’s actual predicted class unchanged, meaning the explanation itself can be manipulated independently of the prediction it is supposed to explain.

Chapter 29: Detecting Concepts

Part 4: Neural Network Interpretation · Difficulty: Hard · Key topics: TCAV, Concept Activation Vectors, directional derivatives, statistical significance testing

Testing with Concept Activation Vectors (TCAV) explains a neural network’s prediction in terms of human-defined concepts rather than raw pixel importance, answering questions like “how much does the concept ‘striped’ influence this model’s zebra classification?” A Concept Activation Vector (CAV) is computed at a chosen network layer by training a linear classifier that separates activations of concept-example images from activations of random images; the CAV is the direction perpendicular to that classifier’s decision boundary. TCAV then computes, for a batch of images of a target class, what fraction have a positive directional derivative in the CAV’s direction — the TCAV score — and tests its statistical significance against multiple randomly-defined CAVs using a two-sided t-test (with a multiple-testing correction, such as Bonferroni, since many concepts and layers are typically tested at once). Its main appeal is that users need no machine learning expertise to define a concept — they just collect example images — but concepts must be well-defined and TCAV tends to perform poorly on shallow networks, where concepts are not yet cleanly separable.

Key Points:

• TCAV measures how much a human-defined concept (e.g. “striped”) influences a model’s prediction for a target class, rather than attributing importance to raw pixels.
• A Concept Activation Vector (CAV) is found by training a linear classifier that separates a network layer’s activations for concept-example images from activations for random images; the CAV points perpendicular to that boundary.
• The TCAV score is the fraction of a target class’s images whose activations have a positive directional derivative in the CAV’s direction at the chosen layer.
• TCAV scores are tested for statistical significance against many randomly-defined concept vectors with a two-sided t-test, using a multiple-testing correction such as Bonferroni.
• Because it only requires collecting example images for a concept, TCAV does not require the user to have machine learning expertise, making it accessible to domain experts.
• TCAV tends to perform poorly on shallow networks, since concepts become more cleanly separable only in deeper layers, and it requires additional concept-image annotation effort.

Memory Tip: Keep the two inputs to a CAV straight: it’s trained on activations from concept-example images versus random images at one chosen layer — get either input set wrong and the resulting direction won’t represent the concept you intended.

Common Mistake: Running a single TCAV test without a multiple-testing correction when checking many concepts or many layers at once — the chapter is explicit that a Bonferroni-style correction (or similar) is needed, since testing enough concepts by chance will eventually produce a “statistically significant” result that means nothing.

Important Questions:

• What is a Concept Activation Vector (CAV), and how is it computed? It is a numerical direction in a neural network layer’s activation space that represents a human-defined concept, computed by training a linear classifier to separate that layer’s activations for images containing the concept from activations for random images, then taking the direction perpendicular to the resulting decision boundary as the CAV.
• Why does TCAV require a statistical significance test using randomly-defined CAVs, rather than just reporting the raw TCAV score? Because a raw TCAV score alone cannot distinguish a genuine concept effect from one that would appear by chance, so the chapter tests the real concept’s TCAV score against scores from many randomly-defined (meaningless) concept vectors using a two-sided t-test, applying a multiple-testing correction since many concepts and layers are typically tested together.

Chapter 30: Adversarial Examples

Part 4: Neural Network Interpretation · Difficulty: Medium · Key topics: intentional perturbations that fool a model, L-BFGS method, fast gradient sign method, cybersecurity framing

An adversarial example is an instance with small, deliberate feature perturbations designed to make a model produce a false prediction — the chapter recommends reading Counterfactual Explanations first, since adversarial examples share the same underlying loss-minimization structure but aim to deceive the model rather than to interpret it. It opens with real-stakes scenarios: a manipulated stop sign that a self-driving car misreads, a disguised spam email, and a knife shaped to evade a weapons scanner. Two concrete methods are covered: Szegedy et al.’s original box-constrained L-BFGS approach, which minimizes a loss balancing distance to the desired wrong class against distance to the original image, and Goodfellow et al.’s fast gradient sign method, which perturbs every pixel by a small fixed amount in the direction (sign) of the loss gradient, exploiting the fact that many neural network architectures (ReLU networks, LSTMs, maxout networks) behave close to linearly. The chapter reframes the whole topic through a cybersecurity lens: ordinary machine learning predicts from a known distribution, while defending against adversarial attacks means robustly handling an unknown distribution of adversarial inputs.

Key Points:

• An adversarial example is an instance with small, intentional perturbations designed to make a model produce an incorrect prediction — closely related to counterfactual explanations, but aimed at deceiving rather than interpreting the model.
• Real-world risk scenarios given in the chapter: a manipulated stop sign fooling a self-driving car’s sign recognition, a disguised spam email, and a knife shaped to evade an airport weapons scanner.
• Szegedy et al.’s original method minimizes a loss balancing the distance to a desired (wrong) prediction against the distance to the original image, solved with box-constrained L-BFGS.
• The fast gradient sign method (Goodfellow et al.) perturbs every input pixel by a small fixed amount in the direction of the sign of the loss gradient.
• This vulnerability occurs especially in architectures that behave close to linearly, such as ReLU networks, LSTMs, and maxout networks.
• The chapter reframes the topic as a cybersecurity problem: standard machine learning predicts from a known data distribution, while defending against adversarial examples means robustly handling an unknown distribution of adversarial inputs.

Memory Tip: Connect the method name to its mechanism: the fast gradient sign method only uses the sign of the gradient (+1 or −1 per pixel), not its magnitude — that’s what makes it fast and why it works especially well on architectures that behave close to linearly.

Common Mistake: Assuming a robust-looking test accuracy means a neural network is safe from adversarial manipulation — the chapter’s whole cybersecurity framing exists precisely because ordinary accuracy measures performance on a known data distribution, while adversarial robustness is a separate property involving an unknown distribution of deliberately crafted inputs.

Important Questions:

• What is the core idea of the fast gradient sign method for generating adversarial images? It perturbs every pixel of the original image by a small fixed amount, adding or subtracting that amount depending on whether the sign of the loss gradient with respect to that pixel is positive or negative, exploiting the fact that many neural network architectures behave close to linearly with respect to their inputs.
• How does the chapter reframe adversarial examples as a cybersecurity problem rather than a purely technical curiosity? It argues that ordinary machine learning deals with “known unknowns” — predicting unseen data points from a known distribution — while defending against adversarial attacks deals with “unknown unknowns”: robustly handling unseen data points drawn from an unknown, adversarially chosen distribution, which is a fundamentally different and harder problem.

Chapter 31: Influential Instances

Part 4: Neural Network Interpretation · Difficulty: Hard · Key topics: deletion diagnostics, DFBETA, Cook’s distance, influence functions, model debugging via training data

A training instance is “influential” when removing it from the training data considerably changes the model’s parameters or predictions — a way of debugging models by examining the training data itself rather than the model’s structure. The chapter covers deletion diagnostics from classical statistics, in particular DFBETA (the effect of deleting one instance on the model’s parameters) and Cook’s distance, both long-established for (generalized) linear regression models. Because true deletion diagnostics require retraining, which is expensive, the chapter also covers influence functions, which approximate the effect of removing (or upweighting) a training instance without actually retraining the model, provided the model has a twice-differentiable loss function. The chapter frames influential-instance analysis as one of the best model-debugging tools available, precisely because it centers the role of training data in the learning process, unlike feature-perturbation methods such as SHAP, which never look at training data directly.

Key Points:

• A training instance is “influential” if deleting it from the training set considerably changes the model’s parameters or predictions — a data-centered debugging tool rather than a feature-centered one.
• DFBETA measures the effect of deleting a single instance on the model’s fitted parameters; Cook’s distance is a related, widely used deletion-diagnostic measure, both long established for (generalized) linear models.
• Influence functions approximate what would happen if you deleted (or upweighted) a training instance and retrained — without actually retraining — provided the model’s loss function is twice differentiable.
• Because deletion diagnostics traditionally require retraining, they are expensive to compute, though the chapter notes growing compute makes previously infeasible calculations increasingly practical.
• Influential-instance analysis is described as one of the best model-debugging tools available, because it emphasizes the role of training data itself in the learning process.
• This is fundamentally different from feature-perturbation approaches like SHAP: influence methods look at which training instances mattered, not which input features mattered for one prediction.

Memory Tip: Keep the contrast sharp: SHAP and PFI ask “which features mattered for this prediction,” while influential-instance methods ask “which training examples mattered for this model” — the two questions debug completely different things.

Common Mistake: Assuming influence functions give the exact same answer as literally deleting the instance and retraining — influence functions are an approximation valid under a twice-differentiable loss, and treating their output as an exact, guaranteed match to real retraining overstates their precision.

Important Questions:

• What does it mean for a training instance to be “influential,” and how do DFBETA and Cook’s distance relate to it? A training instance is influential if deleting it from the training data considerably changes the model’s parameters or predictions; DFBETA and Cook’s distance are classical deletion-diagnostic statistics, established for (generalized) linear regression models, that quantify exactly this kind of change in the model’s fitted parameters when one instance is removed.
• What problem do influence functions solve compared to classical deletion diagnostics, and what condition must the model satisfy? They approximate the effect of deleting or upweighting a training instance without the expense of actually retraining the model for every instance tested, but this approximation requires the model to have a twice-differentiable loss function for the underlying calculation to be valid.

Part 5: Beyond the Methods

Chapter 32: Evaluation of Interpretability Methods

Part 5: Beyond the Methods · Difficulty: Hard · Key topics: Doshi-Velez and Kim’s three evaluation levels, properties of explanations, lack of ground truth

Evaluating interpretability methods is genuinely difficult because there is usually no ground truth for what a “correct” explanation looks like. The chapter presents Doshi-Velez and Kim’s three levels of evaluation: application-level evaluation (put the real explanation into the real product and test it with the actual end user — e.g., testing a fracture-detection tool’s explanations directly with radiologists), function-level evaluation (a proxy evaluation using a formal definition of interpretability, without human subjects, appropriate once a method class has already been validated with humans for similar tasks), and human-level evaluation (simpler human-subject experiments that don’t require the real, high-stakes application). Because ground truth is missing, the chapter also proposes assessing general, qualitative (sometimes quantitative) properties of explanations themselves — such as how well an explanation relates an instance’s feature values to its prediction in a human-understandable way — as a practical stand-in for a true correctness check.

Key Points:

• Evaluating interpretability methods is difficult because there is usually no ground truth for what a “correct” explanation actually is.
• Doshi-Velez and Kim propose three evaluation levels: application-level (real product, real end users, e.g. radiologists testing a fracture-detection explanation), function-level (a formal proxy definition, no human subjects), and human-level (simpler human-subject experiments not tied to the real high-stakes application).
• Function-level evaluation is appropriate once a method class has already been validated with humans for a similar enough task, saving the cost of repeated human studies.
• In the absence of ground truth, the chapter proposes evaluating general properties of explanations directly, rather than a single correctness score.
• One core property discussed is how well an explanation relates an instance’s feature values to its model prediction in a way a human can actually understand.
• This chapter is explicitly marked as targeted at interpretability researchers and advanced practitioners going deeper into the field, not a prerequisite for using the methods in earlier chapters.

Memory Tip: Rank the three evaluation levels by how close they are to the real deployment: application-level is the real product with real users; human-level is simpler human experiments without the real stakes; function-level has no humans at all, relying on a formal proxy definition instead.

Common Mistake: Assuming a method-level or formal proxy evaluation is automatically as trustworthy as a real application-level test. The chapter is explicit that function-level evaluation is only appropriate once a method class has already been validated with humans on a similar enough task — skipping that validation step and relying on a formal proxy alone is the mistake.

Important Questions:

• What are Doshi-Velez and Kim’s three levels of evaluating interpretability, as presented in this chapter? Application-level evaluation (testing the real explanation embedded in the real product with actual end users), function-level evaluation (a formal, proxy definition of interpretability with no human subjects), and human-level evaluation (simpler human-subject experiments that don’t require the real, high-stakes application).
• Why does the chapter recommend evaluating general properties of explanations instead of relying on a single ground-truth correctness score? Because there is usually no ground truth available for what a “correct” explanation of a machine learning prediction actually is, so assessing qualitative (and sometimes quantitative) properties — like how well an explanation relates feature values to the prediction in a human-understandable way — is a more practical, available substitute.

Chapter 33: The Future of Interpretability

Part 5: Beyond the Methods · Difficulty: Easy · Key topics: speculative outlook, digitization and automation premises, model-agnostic tools as the likely future direction

This closing chapter is explicitly framed as a speculative, subjective mental exercise on where interpretable machine learning is headed, built on premises about digitization (more information being captured electronically) and automation. It first speculates about the future of machine learning itself, tempering the field’s usual optimism with the observation that many flashy research results do not translate cleanly into deployed products. It then argues that the future of interpretability will likely keep focusing on model-agnostic tools specifically, because decoupling the explanation method from the underlying model gives it modularity — you can swap in a new, better-performing black-box model without having to build an entirely new interpretation method to match it, which matters a great deal as the pace of new model architectures keeps accelerating.

Key Points:

• The chapter is explicitly speculative — a subjective mental exercise, not a research forecast with hard evidence.
• Its two guiding premises are about digitization (ever more information captured electronically) and increasing automation across products and processes.
• It tempers typical AI optimism by observing that many flashy research results do not translate cleanly into deployed, real-world products.
• Its central prediction: interpretability’s future focus will likely remain on model-agnostic tools rather than model-specific ones.
• The stated reason is modularity — a model-agnostic explanation method keeps working even when you swap in a completely new underlying model, avoiding the need to build a fresh interpretation method for every new architecture.

Memory Tip: Remember the chapter’s own central bet in one line: as models keep changing faster than explanation methods can be rebuilt from scratch, model-agnostic tools win on modularity alone.

Common Mistake: Reading this chapter’s predictions as established fact rather than the author’s own explicitly-labeled speculation — the chapter opens by calling itself a subjective mental exercise, and treating its forecasts as settled conclusions misrepresents what the author himself says about it.

Important Questions:

• What two premises does the author base the chapter’s speculation on? Digitization — the idea that any interesting information will increasingly be captured electronically — and increasing automation of products and processes; both are stated as the assumptions the rest of the chapter’s speculation is built on.
• Why does the chapter predict that interpretability’s future will focus on model-agnostic tools specifically? Because model-agnostic tools are modular — they can be applied to a new underlying model without redesigning the explanation method itself, which matters increasingly as machine learning models and architectures keep changing and improving at a fast pace.

Appendix

Appendix A: What Is Machine Learning?

Appendix · Difficulty: Easy · Key topics: algorithm, model, machine learning definitions, terminology used throughout the book

This appendix defines the core vocabulary the rest of the book assumes, aimed at a reader who wants a plain-language anchor for terms used loosely elsewhere. It defines an algorithm as a set of rules a machine follows to reach a particular goal (comparing it to a cooking recipe: inputs, output, and the steps between them), distinguishes an algorithm from the model it produces once trained on data, and works through what machine learning itself means as the process of a computer program improving its performance on a task through experience (training data) rather than being explicitly programmed with hand-coded rules for every case.

Key Points:

• An algorithm is a set of rules a machine follows to reach a goal — like a recipe that defines inputs, output, and the steps between them.
• A model is what a learning algorithm produces after being run on training data — the algorithm is the procedure, the model is its trained result.
• Machine learning is defined as a computer program improving its performance on a task through experience (data), rather than being explicitly hand-programmed with rules for every case.
• This appendix exists specifically to remove ambiguity from terms used loosely throughout the rest of the book, for readers without a strong prior machine learning background.

Memory Tip: Keep algorithm and model straight with one line: the algorithm is the recipe, the model is the finished dish it produces once it has been trained on your specific ingredients (data).

Common Mistake: Using “algorithm” and “model” interchangeably when discussing a specific trained system — the appendix is explicit that the algorithm is the general learning procedure, while the model is the specific, trained result of running that procedure on data.

Important Questions:

• How does this appendix distinguish an algorithm from a model? An algorithm is the general set of rules or procedure a machine follows to reach a goal, comparable to a recipe; a model is the specific result produced once that algorithm has been trained (run) on a particular dataset, comparable to the finished dish made from the recipe.
• According to this appendix, what makes machine learning different from traditional, explicitly programmed software? Machine learning improves a program’s performance on a task through experience with data (training), rather than requiring a programmer to hand-code explicit rules covering every case the program might encounter.

Read Interpretable Machine Learning Online (Free)

This book is free to read from its official source — author Christoph Molnar publishes the complete, current 3rd edition online under a Creative Commons licence (CC BY-NC-SA 4.0 for the text; the accompanying code is separately MIT-licensed). There is no official free PDF for the complete book — the author sells a paid ebook (Leanpub) and paperback separately, and this page links only to the official free web edition, not to those paid formats or to any unofficial PDF mirror. Click below to open the official site, where every chapter is free to read in full.

How to Study This Book

Start with Chapters 1–5 (Introduction, Interpretability, Goals of Interpretability, Methods Overview, Data and Models) in order, even if you are eager to get to SHAP or LIME. This is exactly what the book’s own Introduction chapter recommends to new readers, and it is not filler — Chapter 5 introduces the bike-rental and Palmer-penguins datasets that every later chapter’s worked example assumes you already know, and Chapter 4’s SIPA principle and by-design/post-hoc split is the vocabulary the rest of the book is written in. If you already have a solid grip on what “interpretability” means from your coursework, you can skim Chapters 2–3 for definitions and go straight to Chapter 4.

Part 1 (Chapters 6–11, Interpretable Models) is best read straight through: Linear Regression → Logistic Regression → GLM/GAM → Decision Tree → Decision Rules → RuleFit. RuleFit (Chapter 11) explicitly assumes you already understand both linear models and decision-tree splitting, since it fuses the two, so do not jump to it out of order.

Part 2 (Chapters 12–18, Local Model-Agnostic Methods) has two built-in prerequisite chains the book states directly: read Ceteris Paribus (12) before ICE (13), since ICE is literally many ceteris paribus curves drawn on one plot, and read Shapley Values (17) before SHAP (18) — the SHAP chapter opens by telling you to read the Shapley Values chapter first, and skipping it will leave the Efficiency/Symmetry/Dummy/Additivity properties in SHAP feeling unmotivated. LIME (14), Counterfactual Explanations (15) and Anchors (16) can be read in any order relative to each other.

Part 3 (Chapters 19–26, Global Model-Agnostic Methods) has one more explicit book-stated dependency: read Partial Dependence Plot (19) before Accumulated Local Effects (20), since the ALE chapter opens by recommending exactly that. Feature Interaction (21) and Functional Decomposition (22) are the two most mathematically demanding chapters in this part — if your course only needs the practical methods (PDP, ALE, Permutation Feature Importance, LOFO, Surrogate Models, Prototypes and Criticisms), these two can be safely deferred to a second pass.

Part 4 (Chapters 27–31, Neural Network Interpretation) only matters if your course actually covers deep learning and CNNs — if it doesn’t, this whole part can be skipped without weakening your grasp of the model-agnostic methods in Parts 2–3, which apply to any model type. If your course does cover it, read Learned Features (27) before Adversarial Examples (30): the book explicitly connects feature visualization and adversarial examples as the same activation-maximization optimization problem aimed at two different targets.

Part 5 is short and the book itself tells you how to treat it: Evaluation of Interpretability Methods (32) opens by saying it is written for interpretability researchers and advanced practitioners, and invites you to “feel free to skip it otherwise” — a student preparing for a standard course exam can safely treat it as optional deeper reading. The Future of Interpretability (33) is a short, low-math closing chapter worth reading once, casually.

Two short parts of the book were left out of the Chapter List above on purpose, and this is exactly what the book itself does with them in its 3rd edition: “Translations” is a maintained list of community translation links, not teaching content, and “Story Time” is a set of three speculative short-fiction vignettes the author himself moved to the end of the book as a change of pace, not additional syllabus — skip both without losing anything technical.

One practical thing to know before you start: almost every worked example and code snippet in this book is written in R, using the author’s own iml package alongside packages like ceterisParibus, DALEX, ALEPlot, rpart, and pre. If your BSCS course teaches in Python, do not let that stop you — every chapter’s “Software and alternatives” section names the Python equivalent (shap, lime, alibi, eli5, PiML, scikit-learn’s built-in PDP support, and more), so you can follow every method’s idea in the text and then implement it in Python on your own.

If you are shaky on basic machine learning vocabulary — the difference between an algorithm and a model, or what “training” really means — read Appendix A (What Is Machine Learning?) first, before Chapter 1. It is short and exists specifically to remove that ambiguity before the real material starts.

Used In These Programs

This book is used for the Machine Learning / Interpretable Machine Learning course in: BS Computer Science. It is also very commonly assigned in MS-level Machine Learning coursework, though this site’s degree tags currently track undergraduate programs only. Browse all Machine Learning books or all Computer Science books.

Who Should Read This

This book is written for a reader who already has a working machine learning model and wants to understand and explain it — it teaches interpretation methods, not how to build or train models in the first place, so it assumes you already know the basics of supervised learning, regression and classification, and are comfortable with university-level math (linear algebra, probability, some calculus). It suits final-year BSCS students in a Machine Learning or Interpretable ML elective, and is a standard reference many carry into an MS in Machine Learning or a research career touching Explainable AI. It is not a good first machine learning book for someone who hasn’t trained a model before — that student should first build a foundation with a general machine learning course or text. It is also, in practice, an R-first book: most of its own worked examples use R packages, so a reader who works exclusively in Python and wants to run the code alongside the text, not just read the “Software and alternatives” pointers, may first want to firm up general programming fluency with a book like Think Python before translating each chapter’s R examples into the Python packages (shap, lime, alibi) that every chapter’s software section names.

Applicable Universities

This book is useful for students at Pakistani universities offering BSCS and related programs including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, and other HEC-recognized institutions.

FAQs

Related Books

• Think Python – Allen B. Downey (3rd Edition) — useful if you plan to run this book’s methods in Python rather than R, since most of the software this book points to (shap, lime, alibi) is Python-based

Interpretable Machine Learning is the standard, actively maintained reference for explaining black-box models — from classical interpretable models through LIME, SHAP and Shapley values to neural-network-specific interpretation methods — free to read at its official source under a Creative Commons licence. Browse more Computer Science books for the rest of your semester.

Interpretable Machine Learning: A Guide for Making Black Box Models Explainable, by Christoph Molnar (3rd Edition, 2025). The book text is licensed under Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0); the accompanying code is separately licensed under the MIT License. Official source: https://christophm.github.io/interpretable-ml-book/