Deep Learning – Goodfellow, Bengio & Courville (Free Book) BSCS

Q: Is Deep Learning by Goodfellow, Bengio and Courville free?

Yes, to read. The complete book is free to read online at its official site, deeplearningbook.org, with MIT Press's permission. It does not carry a Creative Commons or other open licence — MIT Press retains full copyright — so it may be read free online but is not open for reuse or redistribution the way a CC-licensed book is.

Q: Is there an official PDF of the complete book?

No. The only PDF on the official site is the front matter. The authors state their contract with MIT Press restricts easily-copied electronic formats of the complete book, which is why only an HTML edition is offered officially. Unofficial full-book PDFs exist elsewhere online, but they are not the authorized source and this page does not link to them.

Q: Which edition is this, and is it still current in 2026?

This is the original and only edition, published by MIT Press in 2016. The mathematical foundations in Parts I and II (linear algebra, probability, optimization, feedforward and convolutional networks, RNNs and LSTMs) have not dated at all. What the book does not cover, because it predates them, are transformers, attention-only architectures, and large language models — students who need those specifically should treat this book as the prerequisite theory and pair it with newer material for those topics.

Q: How many chapters does the book have?

20 chapters in three parts: Chapter 1 (Introduction) stands alone, Part I (Chapters 2-5) covers applied math and classical machine learning, Part II (Chapters 6-12) covers modern deep network practice, and Part III (Chapters 13-20) covers deep learning research topics including generative models.

Q: Is it good for complete beginners?

Not as a first book. It assumes comfortable linear algebra, probability, and at least an introductory machine learning course, and it has no code at all. A complete beginner should first build programming and basic ML fluency with a book like Think Python, then come to this book once ready to learn the underlying mathematics of deep learning in depth.

Q: Does it cover CNNs, RNNs, LSTMs and generative models like GANs and VAEs?

Yes, in real depth. Convolutional networks are Chapter 9, sequence models including RNNs and LSTM are Chapter 10, and Part III's final chapter (20) covers deep generative models including variational autoencoders and generative adversarial networks, alongside the Boltzmann machine family that came before them.

BSCS students — and the many BS graduates who go on to a Deep Learning or Machine Learning MS course — can read the complete “Deep Learning” textbook by Ian Goodfellow, Yoshua Bengio and Aaron Courville free online at its official site. Published by MIT Press in 2016, it is the book most Pakistani university Deep Learning electives cite as the reference text, and it remains one of the two or three books almost every serious deep learning student eventually works through cover to cover, precisely because it was written to be the field’s shared mathematical foundation rather than a tutorial for one particular framework.

Its structure is worth understanding before you start. Part I (Chapters 2-5) is a self-contained refresher on the applied mathematics and classical machine learning a reader needs — linear algebra, probability, numerical computation, and the capacity/overfitting framework the rest of the book assumes. Part II (Chapters 6-12) is “modern practice”: feedforward networks, regularization, optimization, convolutional networks, recurrent networks and LSTMs, practical advice, and real applications — this is the material a one-semester course typically covers. Part III (Chapters 13-20) is “research”: linear factor models, autoencoders, representation learning, structured probabilistic models, and the generative models — VAEs and GANs — that Chapter 20 ends on. There is no code in the book at all; it teaches the mathematics and the ideas, not an implementation, which is exactly why it pairs so well with a code-first book when you actually want to build something.

Book Overview

Course	Deep Learning / Neural Networks (Computer Science elective)
Degree Programs	BSCS (also very commonly used in MS-level Machine Learning / Deep Learning coursework)
Level	University / Advanced undergraduate and graduate
Edition	2016 (MIT Press) — the only edition; the online text is kept current with an errata list
Author(s)	Ian Goodfellow, Yoshua Bengio, Aaron Courville
Publisher	MIT Press
Structure	20 chapters in three parts (Applied Math and Machine Learning Basics; Deep Networks: Modern Practices; Deep Learning Research), plus bibliography and index
Language	English
Format	Free official HTML edition (read online). No official PDF of the complete book exists.

Chapter List

Chapter 1: Introduction

Difficulty: Easy · Key topics: AI vs machine learning vs representation learning vs deep learning, formal vs intuitive problems, three waves of neural network research, dataset/model size growth

Chapter 1 opens by asking why computers, which excel at formal logical tasks, struggle with things humans do intuitively — recognizing a spoken word or a face. It positions deep learning as a solution built on representation learning: instead of hand-designing features, the system learns increasingly abstract representations of data through a hierarchy of simple transformations. The authors trace three historical waves of neural network research — cybernetics in the 1940s–60s, connectionism in the 1980s–90s, and the current deep learning wave that began in 2006 — explaining why the first two both collapsed into “AI winters” and what changed the third time: much larger labeled datasets, much larger models, and much more powerful computing hardware.

Key Points:

The chapter distinguishes AI, machine learning, representation learning and deep learning as four nested sets, with deep learning the innermost.
Formal problems (chess, logic) were easy for early computers; intuitive problems (recognizing speech or faces) are described as the true challenge to AI.
Representation learning lets a system discover both a mapping from representation to output and the representation itself.
Deep learning solves representation learning by expressing complicated representations as compositions of simpler ones.
Three waves of neural network research are identified: cybernetics (1940s–1960s), connectionism (1980s–1990s), and deep learning (2006–present).
Both earlier waves ended when researchers made unrealistically ambitious claims the technology of the time could not deliver.
Dataset size, model size and accuracy have grown roughly in step since the 1980s, which the book argues largely explains the current wave, more than any single algorithmic breakthrough.

Memory Tip: Remember the four nested circles as AI ⊃ Machine Learning ⊃ Representation Learning ⊃ Deep Learning — each ring is the previous one’s way of avoiding hand-engineering, one layer deeper.

Common Mistake: Treating “deep learning” and “neural network” as if they always mean the same thing. The book’s own definition of deep learning is about the depth of the computational graph used to define a representation, not about using neural networks specifically — several models later in the book (like the probabilistic models in Part III) are not simple feedforward neural networks at all.

Important Questions:

What is the difference between formal and intuitive problems in AI, and why does it matter for deep learning? Formal problems can be solved with a fixed list of logical rules (chess, theorem proving) and were the early triumphs of AI; intuitive problems (recognizing a face, understanding speech) are things people do without being able to describe the rules consciously, and deep learning matters because it lets computers learn those rules from data instead of being told them.
What are the three historical waves of neural network research, and what ended the first two? Cybernetics (1940s–1960s), connectionism (1980s–1990s), and deep learning (2006–present); the first ended after critics exposed the limits of linear models like the perceptron, and the second ended when neural-network ventures promised more than the era’s data and compute could deliver, leading to a second AI winter.

Chapter 2: Linear Algebra

Part 1: Applied Math and Machine Learning Basics · Difficulty: Medium · Key topics: scalars/vectors/matrices/tensors, matrix multiplication, norms, eigendecomposition, SVD, PCA

Chapter 2 is a self-contained refresher for computer-science readers who have not studied continuous mathematics in depth. It builds from scalars, vectors, matrices and tensors to matrix multiplication, the identity and inverse matrix, and the ideas of linear dependence and span that determine whether a system of equations Ax=b has a solution. It then covers norms as ways to measure the size of a vector, works through eigendecomposition and its geometric meaning, and extends decomposition to non-square matrices with the Singular Value Decomposition (SVD). It closes with the Moore-Penrose pseudoinverse, the trace and determinant operators, and a worked derivation of Principal Component Analysis (PCA) as the simplest example of a full machine learning algorithm built from linear algebra alone.

Key Points:

A matrix product is defined element-wise as C_ij = sum_k A_ik B_kj; matrix multiplication is associative and distributive but not commutative.
Ax = b has a unique solution only when the columns of A are linearly independent, i.e. no column can be written as a combination of the others.
The L2 norm is the Euclidean length of a vector; the L1 norm is preferred when the difference between zero and non-zero must be exact.
Eigendecomposition writes a square matrix as A = V diag(λ) V⁻¹, exposing directions (eigenvectors) the matrix only stretches, never rotates.
The Singular Value Decomposition A = UDVᵀ generalizes eigendecomposition to any matrix, including non-square ones.
The Moore-Penrose pseudoinverse gives the minimum-norm solution to underdetermined systems and the least-squares solution to overdetermined ones.
PCA finds the encoding direction that minimizes reconstruction error, and that direction turns out to be the eigenvector of XᵀX with the largest eigenvalue.

Memory Tip: Anchor eigendecomposition and SVD to what each one requires: eigendecomposition needs a square matrix and gives you the matrix’s own “natural axes”; SVD works on any matrix at all and is really eigendecomposition applied twice, once to AAᵀ and once to AᵀA.

Common Mistake: Assuming matrix multiplication commutes the way scalar multiplication does. AB and BA are generally different matrices (and may not even have compatible shapes), while the dot product of two vectors, xᵀy = yᵀx, is commutative — the two rules get mixed up under exam pressure.

Important Questions:

What condition on the columns of A guarantees that Ax = b has exactly one solution for every b? A must be square and its columns must be linearly independent (equivalently, A must be invertible); if the columns are dependent, the system has either no solution or infinitely many.
What makes the Singular Value Decomposition more broadly useful than eigendecomposition? SVD is defined for every matrix, including non-square and non-diagonalizable ones, because it factors A as UDVᵀ using two different orthogonal matrices instead of requiring A to be diagonalizable in a single basis.

Chapter 3: Probability and Information Theory

Part 1: Applied Math and Machine Learning Basics · Difficulty: Medium · Key topics: random variables, marginal/conditional probability, Bayes’ rule, expectation/variance/covariance, Gaussian distribution, Shannon entropy, KL divergence

This chapter gives the probability and information theory needed to describe uncertainty in machine learning, and explains why uncertainty is unavoidable even for a perfectly designed system: it comes from inherent stochasticity, incomplete observability, and models that are necessarily simplified. It works through random variables, probability mass and density functions, marginal and conditional probability, and the chain rule that decomposes a joint distribution into a product of conditionals. A tour of common distributions follows — Bernoulli, multinoulli, and above all the Gaussian, whose central role is justified by the central limit theorem and by having maximum entropy for a given variance. The chapter ends with information theory: self-information, Shannon entropy, and the KL divergence used throughout the book to compare a model’s distribution with the true data distribution.

Key Points:

Frequentist probability treats a probability as the rate an event occurs; Bayesian probability treats it as a degree of belief, and both use the same rules of probability.
Bayes’ rule, P(x|y) = P(x)P(y|x)/P(y), lets a posterior probability be computed from a prior P(x) and a likelihood P(y|x).
Covariance measures how two variables move together, but zero covariance does not imply independence.
The Gaussian distribution N(x; μ, σ²) is the natural default choice when little is known about a real-valued variable, because it has maximum entropy for a fixed variance.
Shannon entropy H(x) = -E[log P(x)] measures the average information in outcomes drawn from a distribution; a near-deterministic distribution has low entropy.
KL divergence D_KL(P||Q) measures how different two distributions are, and is not symmetric: D_KL(P||Q) generally does not equal D_KL(Q||P).
Structured probabilistic models use a graph to show which variables interact directly, avoiding the need to store every possible joint outcome.

Memory Tip: Keep straight which quantity is symmetric and which is not: covariance and correlation are symmetric between two variables, but KL divergence is directional — it always compares one specific distribution to another, and swapping the two changes the answer.

Common Mistake: Assuming that if two variables are uncorrelated (zero covariance) they must be independent. Independence is the stronger condition; two variables can have zero covariance while remaining strongly, just nonlinearly, dependent.

Important Questions:

What is the difference between a probability mass function and a probability density function? A PMF applies to a discrete random variable and gives the probability of each individual outcome directly (values sum to 1); a PDF applies to a continuous variable and its value is not itself a probability — only its integral over an interval gives a probability, and that integral must equal 1 over the whole domain.
Why is KL divergence not a true distance metric? Because it fails symmetry, D_KL(P||Q) ≠ D_KL(Q||P) in general, so it cannot serve as a distance function between two probability distributions even though it is always non-negative and zero only when P equals Q.

Chapter 4: Numerical Computation

Part 1: Applied Math and Machine Learning Basics · Difficulty: Medium · Key topics: overflow/underflow, softmax stabilization, poor conditioning, gradient descent, Jacobian/Hessian, Newton’s method, KKT conditions

Real computers cannot represent real numbers exactly, and this chapter is about the errors finite-precision arithmetic introduces into machine learning algorithms and how to control them. It opens with underflow (numbers rounding to zero) and overflow (numbers rounding to infinity), using the softmax function as the running example: naively exponentiating large numbers overflows, so every practical implementation first subtracts the maximum value before exponentiating. It then defines poor conditioning through the condition number of a matrix, and moves into gradient-based optimization: the gradient, critical points, the Jacobian and Hessian matrices, and how the Hessian’s eigenvalues reveal whether a step of a given size will reduce the cost function. The chapter closes with Newton’s method, which uses the Hessian to jump toward a critical point in one step, and with constrained optimization via the Karush-Kuhn-Tucker (KKT) conditions.

Key Points:

Underflow rounds small numbers to zero; overflow rounds large-magnitude numbers to infinity, and both produce undefined results in later computations.
The numerically stable softmax subtracts max(x_i) from every input before exponentiating, guaranteeing the largest exponent is exactly zero and preventing overflow.
A matrix’s condition number is the ratio of its largest to smallest eigenvalue magnitude; a high condition number means small input errors can be hugely amplified when the matrix is inverted.
A critical point is any point where the gradient is exactly zero — a minimum, a maximum, or a saddle point.
The Hessian matrix of second derivatives determines curvature: its eigenvalues tell you whether a gradient step of a given size will overshoot, undershoot, or land correctly.
Newton’s method uses x* = x₀ - H(f)(x₀)⁻¹ ∇f(x₀) to jump directly toward a critical point, converging faster than gradient descent near a well-behaved minimum but risking attraction to a saddle point.
The KKT approach turns a constrained optimization problem into an unconstrained one using a generalized Lagrangian and multipliers, with complementary slackness ensuring inactive constraints contribute nothing.

Memory Tip: Connect the two overflow-prevention tricks to one idea: subtract the biggest number in sight before you exponentiate. It works for softmax exactly as it works for computing log-sum-exp anywhere else in the book.

Common Mistake: Assuming a small gradient always means training is near a good minimum. A near-zero gradient can just as easily mean a saddle point or a wide flat plateau, and only the Hessian’s eigenvalues (or, empirically, whether the loss keeps decreasing) can distinguish the two.

Important Questions:

Why does the softmax function need to be computed with the “subtract the max” trick rather than the textbook formula directly? Exponentiating the raw logits can overflow to infinity for large values or underflow to zero for very negative ones; subtracting the maximum value first keeps every exponent at or below zero, so the largest term becomes exp(0)=1, and the computation stays numerically stable without changing the mathematical result.
What does a high condition number tell you about a matrix, and why does it matter for machine learning? It means the matrix is close to singular in a practical sense — a small change in the input can produce a very large change in the output when the matrix is inverted or used to solve a linear system, so numerical errors and even ordinary floating-point rounding can be amplified into large, misleading errors during training.

Chapter 5: Machine Learning Basics

Part 1: Applied Math and Machine Learning Basics · Difficulty: Hard · Key topics: task/performance/experience, capacity, overfitting/underfitting, VC dimension, no free lunch theorem, bias-variance, maximum likelihood, SVM, k-NN, curse of dimensionality

This is the last chapter before the book turns to deep networks specifically, and it lays out classical machine learning using Tom Mitchell’s definition: a program learns from experience E on tasks T if performance P improves with E. It works through the standard task types, the difference between supervised and unsupervised experience, and linear regression as a first worked example solved in closed form by the normal equations. The chapter’s central idea is the tradeoff between a model’s capacity and its generalization gap, illustrated by the classic U-shaped curve of training versus test error. It then works through estimators, bias and variance, maximum likelihood and Bayesian estimation, several classical supervised algorithms (logistic regression, SVMs, k-nearest neighbors, decision trees) and unsupervised ones (PCA, k-means), and closes on the curse of dimensionality as the specific limitation that motivates deep learning for the rest of the book.

Key Points:

Mitchell’s definition: a program learns if its performance P at tasks T improves with experience E; supervised experience means the dataset carries a label for each example, unsupervised means it does not.
Generalization (test) error is what actually matters; training error is only useful as a proxy, and the gap between the two grows as capacity increases.
The No Free Lunch theorem says that, averaged over every possible data-generating distribution, no learning algorithm is universally better than any other — a good algorithm is one matched to the kinds of distributions the real world actually produces.
Mean squared error can be decomposed as Bias² + Variance; increasing model capacity typically lowers bias but raises variance, and the best capacity balances the two.
Maximum likelihood estimation is equivalent to minimizing the KL divergence between the empirical data distribution and the model’s distribution.
The kernel trick lets a support vector machine, f(x) = b + Σ α_i k(x, x_i), represent a nonlinear decision boundary using only dot products, without explicitly computing a high-dimensional feature mapping.
The curse of dimensionality means the number of distinct regions a simple model needs to distinguish grows exponentially with the number of input dimensions — exactly what deep, distributed representations are built to avoid.

Memory Tip: Picture generalization error as a U: too little capacity underfits on the left arm, too much capacity overfits on the right arm, and the point you actually want is the bottom of the U, found with a validation set, not the training set.

Common Mistake: Tuning hyperparameters against the test set “just this once.” The test set only estimates generalization error honestly if it is never used to make decisions about the model; hyperparameter choices belong to a separate validation set, and touching the test set for that purpose invalidates it.

Important Questions:

What is the difference between bias and variance, and how do they relate to a model’s capacity? Bias is the error from an estimator’s assumptions being systematically wrong; variance is the error from an estimator’s sensitivity to the specific training sample it saw. As capacity increases, bias tends to fall and variance tends to rise, and mean squared error, their sum, is minimized at some intermediate capacity.
What does maximum likelihood estimation actually optimize, in information-theoretic terms? It is equivalent to minimizing the KL divergence between the empirical distribution defined by the training data and the model’s distribution, which in turn is equivalent to minimizing the cross-entropy between the two.

Chapter 6: Deep Feedforward Networks

Part 2: Deep Networks: Modern Practices · Difficulty: Hard · Key topics: XOR example, ReLU, cross-entropy cost, sigmoid/softmax output units, universal approximation, backpropagation, computational graphs

Chapter 6 opens Part II by introducing the feedforward network (multilayer perceptron) as a chain of layers composing simple functions, and uses the XOR problem to show why: a purely linear model cannot separate XOR’s four points no matter how it is trained, but adding one hidden layer with a ReLU nonlinearity solves it exactly, because the hidden layer can map the two “1” inputs onto the same point in a new representation. From there the chapter builds a full training recipe: cost functions derived from maximum likelihood (mostly cross-entropy, which avoids the vanishing-gradient problem mean squared error causes with saturating output units), a survey of output units (linear for regression, sigmoid for binary classification, softmax for multi-class), and a survey of hidden units with ReLU as the modern default because it does not saturate over its active half. It closes with architecture design and backpropagation, presented as the chain rule applied systematically to a computational graph so every gradient is computed exactly once.

Key Points:

A network with one ReLU hidden layer solves XOR by mapping both (1,0) and (0,1) onto the same point in the learned representation, which a linear output layer can then separate.
Cross-entropy loss is generally preferred over mean squared error for classification because it keeps a useful, non-vanishing gradient even when a sigmoid or softmax output is confidently wrong.
ReLU, g(z) = max(0, z), is the default hidden unit because it is nearly linear over its active region (z > 0), preserving the optimization behaviour of a linear model there while still being nonlinear overall.
The softmax output, softmax(z)_i = exp(z_i) / Σ_j exp(z_j), generalizes the sigmoid to more than two classes and is always paired with the log-likelihood loss so its gradient does not vanish.
A single hidden layer can approximate any continuous function to arbitrary accuracy (the universal approximation theorem), but may need an exponential number of units to do so — depth is what keeps the unit count practical.
Backpropagation is the chain rule applied to a computational graph in a specific order (reverse-mode differentiation) so each intermediate gradient is computed once and reused, instead of exponentially many times.

Memory Tip: Pair the output unit with its natural loss the same way the book always does: sigmoid output goes with binary cross-entropy, softmax output goes with categorical cross-entropy, linear output goes with mean squared error under a Gaussian assumption — mixing and matching is what causes vanishing gradients.

Common Mistake: Believing backpropagation is a different algorithm from the chain rule, or that it is specific to neural networks. It is exactly the multivariable chain rule, applied to any computational graph in an order that reuses each intermediate result — the same procedure can differentiate any differentiable program, not just a layered network.

Important Questions:

Why can’t a purely linear model solve the XOR problem, and how does adding a hidden layer fix it? XOR’s four points are not linearly separable in the original input space, so no single hyperplane can classify them correctly; a hidden layer with a nonlinear activation can transform the inputs into a new representation — in this specific case mapping (1,0) and (0,1) to the same point — in which a linear output layer can then separate the classes.
Why is cross-entropy usually preferred over mean squared error as the cost function for a classification network? Pairing MSE with a saturating output unit like sigmoid or softmax causes the gradient to vanish when the prediction is confidently wrong, stalling learning; cross-entropy’s log term cancels the saturating exponential in the output unit’s gradient, so the model keeps learning even from confidently incorrect predictions.

Chapter 7: Regularization for Deep Learning

Part 2: Deep Networks: Modern Practices · Difficulty: Hard · Key topics: L2/L1 penalties, dataset augmentation, early stopping, parameter sharing, bagging, dropout, adversarial training

Regularization is defined precisely as any change to a learning algorithm intended to reduce test error without necessarily reducing training error — trading a little bias for less variance. The chapter surveys the main techniques in turn: L2 weight decay, which multiplicatively shrinks weights toward zero along the low-curvature directions of the loss; L1 regularization, which drives many weights to exactly zero and performs implicit feature selection; dataset augmentation, which manufactures new training examples through label-preserving transformations; and noise injection, applied to inputs, weights, or labels through label smoothing. It covers early stopping as a regularizer that, under a quadratic approximation, is mathematically equivalent to L2 weight decay, and parameter sharing, the technique that makes convolutional networks (Chapter 9) practical. The chapter ends with two of the most widely used practical techniques in the whole book: bagging/ensembling, and dropout, presented as an efficient approximation to training and averaging an exponential number of sub-networks.

Key Points:

Regularization is formally defined as anything intended to reduce generalization (test) error, even at the cost of training error — it is not the same thing as “making the model simpler.”
L2 regularization (weight decay) adds (λ/2)||w||² to the cost and shrinks weight components most along the low-curvature (flat) directions of the loss surface.
L1 regularization adds λ||w||₁ to the cost and, unlike L2, drives many weights to exactly zero, giving a form of automatic feature selection.
Early stopping, monitored via a validation set, is shown to be approximately equivalent to L2 regularization when the loss surface is quadratic, with training time playing the role L2’s penalty strength normally plays.
Parameter sharing forces two or more parameters to be literally equal; convolutional networks use it to cut the parameter count enormously versus a fully-connected layer.
Dropout trains an exponential number of thinned sub-networks by randomly dropping units during each training step, then approximates their average at test time by scaling weights by the unit’s inclusion probability.
Adversarial training deliberately trains on inputs perturbed in the direction that most increases the loss, improving robustness by discouraging the model from being excessively linear near the training data.

Memory Tip: Contrast the two norm penalties by what they do to a weight near zero: L2 shrinks a weight proportional to its size (it never quite reaches zero on its own), while L1 subtracts a constant amount regardless of size, which is exactly why L1 can push a weight all the way to zero and L2 cannot.

Common Mistake: Treating dropout as noise injected at the input, like dataset augmentation. Dropout randomly zeroes hidden unit activations at every layer it is applied to during training, not the raw input pixels, and its effect is closer to training a huge ensemble of sub-networks that share parameters than to noisy data augmentation.

Important Questions:

Why is early stopping considered a form of regularization? It limits how long the optimizer is allowed to fit the training data, which the book shows is approximately equivalent, under a quadratic approximation of the loss, to adding an L2 penalty — the number of training steps behaves like the inverse of the regularization strength.
How does dropout approximate training an ensemble of networks without actually training separate models? During training, dropout randomly removes each unit with some probability, so every minibatch trains a different, randomly thinned sub-network that shares parameters with all the others; at test time, using all units but scaling their outputs by the inclusion probability approximates averaging the predictions of that whole exponential family of sub-networks in a single forward pass.

Chapter 8: Optimization for Training Deep Models

Part 2: Deep Networks: Modern Practices · Difficulty: Hard · Key topics: empirical risk minimization, ill-conditioning, saddle points, SGD, momentum, weight initialization, AdaGrad/RMSProp/Adam, batch normalization

This chapter separates “learning” from “pure optimization”: training minimizes a surrogate loss (like cross-entropy) on a finite training set hoping to improve a different, intractable quantity (generalization error), and stops early rather than running to convergence. It surveys what actually makes deep network optimization hard — ill-conditioning of the Hessian, the sheer number of saddle points that dominate high-dimensional loss surfaces far more than local minima do, cliffs that cause exploding gradients, and vanishing/exploding gradients from repeated multiplication by the same weight matrix. The chapter builds up from plain stochastic gradient descent to momentum and Nesterov momentum, covers principled weight initialization strategies (including the Glorot/Xavier scheme) needed to break symmetry between units, and surveys the adaptive learning-rate family — AdaGrad, RMSProp, and Adam — along with approximate second-order methods like Newton’s method, conjugate gradients, and L-BFGS.

Key Points:

Learning differs from pure optimization: it minimizes a tractable surrogate loss on a training sample, uses early stopping instead of running to a critical point, and cares about a quantity (generalization error) it cannot directly measure.
In high-dimensional loss surfaces, saddle points vastly outnumber true local minima, and most local minima that do exist have a cost close to the global minimum for large networks.
Momentum accumulates an exponentially decaying moving average of past gradients into a velocity term, letting the optimizer build up speed through consistently-sloped, low-curvature regions.
Glorot/Xavier initialization draws weights from U[-√(6/(m+n)), √(6/(m+n))], chosen to keep the variance of activations and gradients roughly stable across layers at the start of training.
AdaGrad, RMSProp and Adam all adapt the learning rate per parameter using accumulated squared gradients; Adam additionally tracks a momentum-like first moment and corrects both moments for their initialization bias.
Adam’s suggested defaults (learning rate 0.001, β₁ 0.9, β₂ 0.999) are described as fairly robust to the choice of hyperparameters, a major reason for its popularity.
Batch normalization normalizes each layer’s activations using the statistics of the current minibatch, which the chapter credits with making very deep networks dramatically easier to optimize.

Memory Tip: Remember the exploding/vanishing-gradient mechanism as “the same matrix, multiplied t times”: whenever a computation repeatedly multiplies by the same weight matrix W (across layers of a very deep net, or across time steps of a recurrent net), the result scales with W’s eigenvalues raised to the power t, so anything not extremely close to an eigenvalue of 1 either explodes or vanishes.

Common Mistake: Assuming a training run has “gotten stuck” whenever the loss stops decreasing for a while. In high dimensions this is far more often a saddle point or a long, nearly flat plateau than a true local minimum, and the fix is usually momentum, a different learning rate schedule, or patience, not a different model.

Important Questions:

Why do saddle points matter more than local minima for training large neural networks? In high-dimensional loss surfaces the number of saddle points grows exponentially relative to the number of true local minima, so an optimizer is far more likely to encounter and need to escape a saddle point (where the gradient is small but the point is not a minimum) than to get trapped in a bad local minimum.
What problem does Adam solve compared to plain SGD, and how? Adam combines a momentum-style moving average of the gradient (the first moment) with an RMSProp-style adaptively scaled learning rate based on a moving average of the squared gradient (the second moment), and corrects both for their bias toward zero at the start of training, giving each parameter its own effective step size and generally requiring less manual learning-rate tuning than plain SGD.

Chapter 9: Convolutional Networks

Part 2: Deep Networks: Modern Practices · Difficulty: Medium · Key topics: convolution operation, sparse interactions, parameter sharing, equivariance, pooling, strides/padding, Gabor filters and V1 neuroscience

This chapter explains why convolutional networks work for grid-structured data like images through three ideas the book returns to repeatedly: sparse interactions (a small kernel connects to only a small region of the input, cutting parameters and computation), parameter sharing (the same kernel weights are reused at every spatial position instead of learning a separate set per location), and equivariance to translation (shifting the input shifts the output feature map by the same amount). It works through the convolution operation, pooling layers that summarize a local neighborhood (most commonly with a maximum) to build in invariance to small translations, and practical variants — strides, zero padding, “valid” versus “same” output sizes. It closes by tying the architecture to biology: the Gabor-like filters convolutional units learn closely resemble the receptive fields Hubel and Wiesel found in the mammalian visual cortex’s V1 area, and briefly traces the network’s history from 1990s handwriting recognition to the 2012 ImageNet result that made deep learning mainstream.

Key Points:

Sparse interactions connect each output unit to only a small local region of the input (kernel size k versus input size n), reducing parameters and computation from O(m×n) to O(k×n).
Parameter sharing means the same kernel is applied at every spatial position, so a convolutional layer learns one set of weights regardless of the input’s size.
Equivariance to translation means shifting the input shifts the feature map’s output by the same amount; convolution does not, on its own, give invariance to translation, only equivariance.
Pooling (commonly max pooling) reports a summary statistic over a local neighborhood, which is what actually introduces approximate invariance to small translations.
The discrete 2D convolution formula, S(i,j) = Σ_m Σ_n I(m,n) K(i-m, j-n), is what most deep learning libraries call “convolution,” though many actually implement cross-correlation (no kernel flip).
Zero padding controls output size: “valid” convolution (no padding) always shrinks the output, “same” padding keeps the output the same spatial size as the input.

Memory Tip: Keep the three founding ideas in one sentence: fewer connections (sparse interactions), fewer unique weights (parameter sharing), and a well-behaved response to shifting the input (equivariance) — every practical benefit of a convolutional layer traces back to one of those three.

Common Mistake: Saying convolution gives translation invariance. Convolution alone only gives equivariance (the output shifts along with the input); the approximate invariance to small translations a convnet actually exhibits comes from the pooling layers on top of the convolutions, not from the convolution operation itself.

Important Questions:

What is the difference between equivariance and invariance to translation, and which one does the convolution operation itself provide? Equivariance means shifting the input produces a correspondingly shifted output, so the same information is present, just relocated; invariance means the output does not change at all despite the shift. Convolution by itself is equivariant to translation, not invariant — invariance to small translations is added afterward, typically by a pooling layer.
Why does parameter sharing reduce a convolutional layer’s parameter count so much compared to a fully connected layer? A single kernel of size k (independent of the input’s spatial size) is reused at every position it slides over, so the number of learned parameters depends only on the kernel size and number of channels, not on the size of the input or output feature map, unlike a fully connected layer where every input-output pair gets its own weight.

Chapter 10: Sequence Modeling: Recurrent and Recursive Nets

Part 2: Deep Networks: Modern Practices · Difficulty: Hard · Key topics: parameter sharing across time, unfolding, hidden state, bidirectional RNNs, encoder-decoder, vanishing/exploding gradients, LSTM, GRU

Chapter 10 extends the parameter-sharing idea from convolution (shared across space) to sequences (shared across time): a recurrent network reuses the same weight matrices at every time step, updating a hidden state h(t) = tanh(b + W h(t-1) + U x(t)) that in principle lets it summarize an entire variable-length input into a fixed-size vector. Unfolding this recurrence into a computational graph is what makes backpropagation through time possible, and the chapter builds several architectures on that base: bidirectional RNNs that read a sequence in both directions, and encoder-decoder (sequence-to-sequence) models that let input and output sequences have different lengths, which is what makes machine translation possible. The core difficulty is the long-term dependency problem: because the same weight matrix is applied repeatedly, gradients scale with that matrix’s eigenvalues raised to the number of time steps, so they explode or vanish unless those eigenvalues stay very close to 1. LSTM and gated recurrent units (GRUs) solve this with gated self-loops, an additive path for the cell state, and learned gates that decide what to forget, write, and output.

Key Points:

A vanilla RNN shares the same weight matrices across every time step, so its parameter count does not grow with sequence length.
Unfolding the recurrent computational graph turns the recursive definition into an explicit chain, which is what backpropagation through time actually differentiates.
Bidirectional RNNs run one RNN forward through the sequence and a second RNN backward, then combine both hidden states, giving each time step access to both past and future context.
Encoder-decoder architectures use one RNN to compress a variable-length input into a fixed-size context vector, and a second RNN to generate a variable-length output from it, which is why input and output lengths need not match.
Repeated multiplication by the same weight matrix across t time steps scales gradients by that matrix’s eigenvalues raised to the power t, causing vanishing gradients when eigenvalues are below 1 and exploding gradients when they are above 1.
LSTM introduces a cell state with an additive (not purely multiplicative) update path, controlled by input, forget and output gates, so gradients can flow across many time steps without automatically vanishing.
Gated recurrent units (GRUs) simplify the LSTM design into two gates (update and reset) instead of three, generally with fewer parameters and comparable performance on many tasks.

Memory Tip: Picture the vanishing/exploding gradient problem exactly as Chapter 8 described it for deep feedforward nets, just with “layer” replaced by “time step” — the same weight matrix, multiplied by itself t times, is why LSTM’s fix (an additive, gated path) is the sequence-model analogue of a residual connection.

Common Mistake: Assuming LSTM and GRU eliminate the vanishing gradient problem entirely. They make it far less severe by giving gradients an additive path that does not automatically shrink, but very long sequences can still be hard to learn from, which is why gradient clipping and careful initialization from Chapter 8 remain relevant even with gated units.

Important Questions:

Why do vanilla RNNs struggle to learn dependencies that span many time steps? The same weight matrix is applied at every time step, so the gradient contribution from t steps back scales with that matrix’s eigenvalues raised to the power t; unless the eigenvalues are very close to 1, the gradient either vanishes toward zero or explodes as t grows, which makes long-range credit assignment unreliable.
What structural change does the LSTM make to fix this, compared to a vanilla RNN? It adds a cell state updated mostly additively rather than through repeated matrix multiplication, and controls what enters, leaves, and is forgotten from that cell state using learned gates (input, forget and output gates), giving gradients a path through time that does not automatically shrink or grow the way repeated matrix multiplication does.

Chapter 11: Practical Methodology

Part 2: Deep Networks: Modern Practices · Difficulty: Easy · Key topics: performance metrics, default baselines, learning curves, manual vs automatic hyperparameter search, debugging

After ten chapters of theory, Chapter 11 is the book’s advice on actually building a working system, organized as a repeatable procedure. It starts by insisting on choosing the right performance metric before writing any code — precision, recall, F-score or coverage instead of plain accuracy whenever classes are imbalanced or the system is allowed to abstain — because that metric will guide every later decision. It recommends a default baseline for each data type (feedforward network for fixed-size vectors, convolutional network for images, LSTM/GRU-based recurrent network for sequences) with sensible starting choices. Its diagnostic framework is simple: if training error is too high, the problem is capacity or optimization, not data; if the gap between training and test error is too high, more data or more regularization is the answer, and learning curves are the tool for telling which situation you are in. It closes with a real worked example, Google’s Street View house-number transcription system, where the biggest accuracy gains came from fixing how images were cropped, not from a cleverer model.

Key Points:

Choose the performance metric before building anything — precision, recall, F-score, or coverage for imbalanced or refuse-to-decide systems.
The default baseline model depends on data structure: feedforward network for fixed-length vectors, convolutional network for grid-like data, recurrent network for sequences.
If training error itself is unacceptably high, the fix is more capacity or better optimization; only when the gap between training and test error is too large should you reach for more data or regularization.
Grid search tests every combination of hyperparameter values and its cost grows exponentially with the number of hyperparameters; random search is usually more efficient.
Recommended debugging techniques include visualizing model predictions directly, examining the highest-confidence mistakes, and fitting a tiny subset of the data to confirm the model can memorize it at all.
In the Street View case study, correcting a preprocessing bug (the crop region used for input images) produced a bigger accuracy jump than any algorithmic change did.

Memory Tip: Turn the chapter’s diagnostic rule into a two-question checklist before touching any hyperparameter: is training error itself too high (a capacity/optimization problem), or is the training-to-test gap too high (a data/regularization problem)? The fix for one usually makes the other worse, so diagnosing correctly first saves real time.

Common Mistake: Jumping straight to a fancier model or more regularization when training error is already the problem. If a model cannot even fit its own training data well, adding regularization or more data will not help — the fix is more capacity, a better optimizer, or more training time, considered before regularization.

Important Questions:

According to this chapter, what should be decided before any model is built? The performance metric the project will actually be judged on — for example precision/recall or an F-score instead of raw accuracy when classes are imbalanced, or a coverage metric when the system may abstain — because every later modeling and tuning decision is made in service of that metric.
How does the chapter recommend deciding whether to collect more data or add more regularization? By comparing training error to test error: if training error is already too high, more data will not help because the model cannot even fit what it already has; only when the gap between training and test error is large should you weigh gathering more data against strengthening regularization, using learning curves to judge how much more data is likely to help.

Chapter 12: Applications

Part 2: Deep Networks: Modern Practices · Difficulty: Medium · Key topics: GPU/distributed training, model compression, computer vision preprocessing, speech recognition history, n-grams vs neural language models, word embeddings, encoder-decoder translation

This chapter surveys where deep learning was actually deployed as of the book’s writing, across four domains. Large-scale training is covered first: GPU computing, distributed training with asynchronous SGD and parameter servers, and model compression, which trains a small model to mimic a large one’s outputs when the large model’s size was only needed to fight overfitting, not to represent the function itself. Computer vision gets standard preprocessing techniques (global and local contrast normalization, dataset augmentation). Speech recognition is presented historically, from GMM-HMM systems through the 2009-2012 breakthrough of unsupervised pretraining that cut TIMIT phoneme error rates substantially, to end-to-end recurrent systems that eliminate the HMM entirely. The chapter’s longest section is natural language processing: it contrasts n-gram models, which suffer badly from the curse of dimensionality, with neural language models built on distributed word embeddings, and closes on the encoder-decoder architecture for machine translation, including an early treatment of attention as a differentiable way to focus on the relevant part of a long input sequence.

Key Points:

Model compression trains a smaller model to reproduce a larger model’s outputs, useful when the large model’s size was needed to avoid overfitting rather than to represent the required function.
Asynchronous SGD across a distributed system lets multiple machines update a shared set of parameters without waiting for each other, trading a slightly worse update for much higher overall throughput.
Unsupervised pretraining with RBMs was the breakthrough that dropped TIMIT phoneme error rates from around 26% to about 20.7% in 2009-2012, before later end-to-end recurrent approaches removed the need for a separate HMM stage entirely.
N-gram language models suffer from the curse of dimensionality: with vocabulary size |V| and order n, the number of possible n-grams is |V|^n, so most combinations are never seen in training.
Word embeddings solve n-gram sparsity by representing each word as a dense vector, so information learned about one word transfers to distributionally similar words the model may never have seen in that exact context.
Encoder-decoder machine translation compresses a source sentence with one RNN and generates the target sentence with a second RNN; attention mechanisms later let the decoder look back at different parts of the source sequence instead of relying on one fixed-size summary.

Memory Tip: Tie the speech recognition story to the general theme of the whole book: unsupervised pretraining was the technique that got deep networks working at all before better initialization, ReLU, and larger labeled datasets made pretraining largely unnecessary for many tasks — it is a specific, dated example of the “three waves” story told back in Chapter 1.

Common Mistake: Assuming a bigger model is always the answer for large-scale deployment. The model compression story is the opposite lesson: extra capacity is often needed only during training to avoid overfitting, and the trained function itself can frequently be reproduced by a much smaller model at deployment time.

Important Questions:

Why do n-gram language models scale so badly compared to neural language models with word embeddings? An n-gram model must, in principle, see or smooth for every possible sequence of n words drawn from a vocabulary of size |V|, and the number of such sequences, |V|^n, grows exponentially with n; neural language models represent each word as a dense embedding, so what the model learns about one word’s usage generalizes automatically to distributionally similar words it may never have seen in that exact n-word context.
What problem does model compression solve, and why does it work? It addresses the mismatch between the model size needed during training (often large, to avoid overfitting on a big dataset) and the model size actually needed to represent the learned function; a small model can be trained to match the large model’s output distribution, often reaching comparable accuracy with far fewer parameters.

Chapter 13: Linear Factor Models

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: latent variable h, probabilistic PCA, factor analysis, independent component analysis, slow feature analysis, sparse coding, manifold view of PCA

Part III opens with the simplest class of generative models with latent variables: linear factor models, in which observed data x is generated as x = Wh + b + noise from an unobserved latent vector h. Probabilistic PCA and factor analysis both put a standard normal prior on h; they differ only in how much structure the noise term is allowed — factor analysis allows a general diagonal covariance, while probabilistic PCA restricts it to isotropic noise, and taking that noise to zero recovers ordinary PCA exactly. Independent Component Analysis (ICA) is a distinct problem that requires h’s components to be non-Gaussian for the decomposition to be identifiable at all, with electroencephalography source separation as its canonical real application. The chapter also introduces slow feature analysis, which prefers features that change slowly across nearby frames of a sequence, and sparse coding, which learns an overcomplete, sparse code via an L1-penalized optimization rather than a fixed parametric encoder. It closes by reinterpreting PCA geometrically as finding a thin, pancake-shaped region of high probability — the chapter’s first real bridge to the manifold view of data used repeatedly for the rest of the book.

Key Points:

A linear factor model generates data as x = Wh + b + noise from a latent variable h, typically given a simple prior distribution such as a standard normal.
Factor analysis assumes a diagonal noise covariance for each observed variable; probabilistic PCA is the special case with isotropic noise, and as that noise variance goes to zero it becomes ordinary PCA.
ICA requires the latent factors to be non-Gaussian, because a Gaussian latent distribution makes the decomposition into independent components non-identifiable.
Slow feature analysis is trained to prefer learned features that change slowly across consecutive samples of a sequence, on the idea that meaningful high-level factors in the world change slowly compared to raw sensory input.
Sparse coding learns h by solving an optimization problem, arg min_h λ||h||₁ + β||x - Wh||²₂, at inference time rather than computing it with a fixed encoder function.
PCA can be reinterpreted geometrically as fitting a thin, flat, “pancake-shaped” region of high probability to the data, whose orientation is given by the top eigenvectors of the data covariance matrix.

Memory Tip: Keep factor analysis, probabilistic PCA and ordinary PCA on one dial, turning from “general” to “specific”: factor analysis allows any diagonal noise covariance, probabilistic PCA restricts that noise to be isotropic, and ordinary PCA is what is left when that isotropic noise is turned all the way down to zero.

Common Mistake: Applying ICA to Gaussian-distributed latent sources and expecting a unique decomposition. ICA’s identifiability guarantee specifically requires non-Gaussian sources; with Gaussian sources, any orthogonal rotation of a valid solution is an equally valid solution, so the “independent components” found are not actually unique or meaningful.

Important Questions:

What is the mathematical relationship between factor analysis, probabilistic PCA, and ordinary PCA? All three are linear-Gaussian latent variable models of the form x = Wh + b + noise; factor analysis allows a general diagonal noise covariance, probabilistic PCA restricts the noise to be isotropic, and taking that isotropic noise variance to zero in probabilistic PCA recovers ordinary PCA exactly.
Why does Independent Component Analysis require the latent factors to be non-Gaussian? If the latent sources were Gaussian, the joint distribution of Wh would be invariant to any orthogonal rotation of W and h together, so infinitely many different mixing matrices would explain the same observed data equally well, making the true independent components impossible to recover uniquely; non-Gaussianity breaks this rotational symmetry.

Chapter 14: Autoencoders

Part 3: Deep Learning Research · Difficulty: Medium · Key topics: encoder/decoder, undercomplete codes, sparse/denoising/contractive regularization, manifold learning, semantic hashing

An autoencoder learns to reconstruct its own input through a bottleneck, using an encoder h = f(x) and a decoder r = g(h), but the chapter is explicit that copying the input perfectly is not the goal — the constraints placed on the network are what make the learned representation useful. Undercomplete autoencoders enforce this by making the code h smaller than the input, which, with a linear decoder and squared error, is shown to learn the same subspace as PCA. The chapter then covers regularized autoencoders that can safely use a larger code: sparse autoencoders penalize the code’s activations directly, denoising autoencoders are trained to reconstruct a clean input from a deliberately corrupted version (which the chapter connects to estimating the score, the gradient of the log data density), and contractive autoencoders penalize the Jacobian of the encoder so small input perturbations barely change the code. It closes by explaining autoencoders as learning the local structure of the data manifold — sensitive to directions along the manifold, insensitive to directions off of it — with semantic hashing as a practical application to fast retrieval.

Key Points:

An autoencoder is defined by an encoder h = f(x) and a decoder r = g(h); learning is regularized so the network does something useful with the constraint, not so it perfectly copies its input.
An undercomplete autoencoder (code dimension smaller than input) with a linear decoder and squared reconstruction error learns the same subspace PCA would learn.
Sparse autoencoders add a sparsity penalty Ω(h) directly on the hidden code’s activations, interpretable probabilistically as a Laplace prior over the code (equivalent to an L1 penalty).
Denoising autoencoders are trained to reconstruct a clean input x from a corrupted version x-tilde, and the objective is connected to score matching — learning the gradient of the log data density.
Contractive autoencoders add the penalty Ω(h,x) = λ Σ ||∇_x h_i||², encouraging the encoder to be locally insensitive to small perturbations of the input.
Semantic hashing applies an autoencoder with a low-dimensional binary code to enable near-constant-time approximate retrieval, treating learned codes as addresses into a hash table.

Memory Tip: Match each regularized autoencoder to what it directly penalizes: sparse penalizes the code’s activations, contractive penalizes the code’s sensitivity to the input (its Jacobian), and denoising penalizes indirectly, by making reconstruction survive corrupted input rather than adding an explicit penalty term at all.

Common Mistake: Assuming an autoencoder with a large enough hidden layer will still learn something useful without regularization. An unconstrained, high-capacity autoencoder can simply learn the identity function — copy input to output with zero reconstruction error — and learn nothing useful about the data’s structure; the code size alone does not guarantee useful representations, the regularization does.

Important Questions:

Why does an undercomplete linear autoencoder with squared error end up learning the same thing as PCA? Minimizing reconstruction error under a linear encoder and decoder with a code dimension smaller than the input dimension is the same optimization problem PCA solves — both find the lower-dimensional linear subspace that best preserves the variance of the data.
What does a denoising autoencoder learn that a plain undercomplete autoencoder does not? By training the network to reconstruct a clean input from a corrupted version of it, a denoising autoencoder is forced to implicitly learn the structure of the data distribution itself, and this training objective is mathematically connected to estimating the score (the gradient of the log data density), which a plain reconstruction objective does not provide.

Chapter 15: Representation Learning

Part 3: Deep Learning Research · Difficulty: Medium · Key topics: greedy layer-wise pretraining, transfer learning, domain adaptation, disentangling causal factors, distributed representation, exponential gains from depth

This chapter steps back to ask what makes a representation “good,” and argues the answer is one that disentangles the underlying causal factors of the data so downstream tasks become easy. It revisits greedy layer-wise unsupervised pretraining, the technique that first made deep networks trainable before ReLU, dropout and batch normalization made it largely unnecessary outside of NLP, where pretraining word representations on huge unlabeled corpora remains standard. It distinguishes transfer learning, where the task changes but the underlying factors of variation are shared, from domain adaptation, where the task stays fixed but the input distribution shifts, and covers the semi-supervised case where a small amount of labeled data determines which factors of the unlabeled data’s structure are actually worth representing. The chapter’s central technical argument is for distributed representations: a representation built from many partially independent features can distinguish exponentially many regions of input space using only a number of parameters that grows polynomially, which is what gives depth its exponential advantage over shallow models on the right kind of task.

Key Points:

A good representation is defined as one that disentangles the data’s underlying causal factors, making the true structure of the data explicit and easy for a downstream task to use.
Greedy layer-wise unsupervised pretraining trains each layer in sequence as an unsupervised model before fine-tuning the whole network; it acts as a regularizer by choosing initial parameters that sit in a good region of the loss surface.
Transfer learning shares statistical strength when the underlying factors of variation transfer across tasks; domain adaptation is the reverse case, where the task is fixed but the input distribution shifts.
Distributed representations can, with O(n) parameters, distinguish O(2^n) or more distinct configurations, compared to non-distributed methods like k-means, which can only distinguish as many regions as they have parameters.
Depth gives an exponential representational advantage on certain function families: some functions need exponentially many units to represent at insufficient depth but only a polynomial number at sufficient depth.
Zero-shot learning — recognizing a category with no training examples at all — is presented as evidence that a representation has captured meaningful, transferable structure rather than memorizing surface statistics.

Memory Tip: Anchor “distributed” against “one-hot”: a one-hot / symbolic representation needs one dedicated unit per concept, so n units distinguish only n things, while a distributed representation lets each of n units participate in many concepts at once, so n units can in principle distinguish up to 2^n things.

Common Mistake: Confusing transfer learning with domain adaptation. In transfer learning the input domain is often similar but the task or output changes; in domain adaptation the task stays exactly the same but the distribution of inputs shifts — mixing the two up leads to reusing the wrong layers (lower versus upper) for the actual problem at hand.

Important Questions:

What does it mean for a representation to “disentangle” the underlying causal factors of the data? It means the representation’s dimensions correspond, as closely as possible, to the independent real-world factors that actually generated the observed data, rather than to some entangled mixture of raw statistics, which makes the representation directly useful for many different downstream tasks.
Why can distributed representations distinguish exponentially more regions of input space than non-distributed methods with the same number of parameters? A distributed representation lets each of n learned features participate independently in describing many different concepts, so the features can combine in up to roughly 2^n ways, whereas a non-distributed method like k-means effectively dedicates one region to each parameter and so can distinguish only on the order of n regions.

Chapter 16: Structured Probabilistic Models for Deep Learning

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: directed vs undirected graphical models, cliques, partition function, d-separation, energy-based models, restricted Boltzmann machines

A joint distribution over even a modestly sized image has more possible states than there are atoms in the universe, so this chapter introduces graphical models as the tool for representing only the direct interactions between variables. Directed graphical models (Bayesian networks) factor the joint distribution as a product of conditional distributions, one per variable given its parents, with ancestral sampling as the natural way to draw samples from them. Undirected models (Markov random fields) instead define an unnormalized distribution as a product of clique potentials, requiring the often-intractable partition function Z to normalize it — energy-based models, parametrized as exp(-E(x)), are singled out because they guarantee a valid probability without needing E(x) itself to be bounded. The chapter closes on the restricted Boltzmann machine (RBM) as its worked example: its bipartite structure (visible units connected only to hidden units, never to each other) makes both p(h|v) and p(v|h) factorize cleanly, which is what makes RBMs tractable to sample from with block Gibbs sampling even though their partition function remains intractable.

Key Points:

A directed graphical model factors p(x) = Π_i p(x_i | Pa(x_i)), where Pa(x_i) is the set of parent variables of x_i in a directed acyclic graph — the structure used for ancestral sampling.
An undirected graphical model defines an unnormalized distribution as a product of clique potentials, requiring division by the partition function Z to get a valid probability distribution.
An energy-based model writes p-tilde(x) = exp(-E(x)), guaranteeing a non-negative, normalizable quantity for any real-valued energy function E(x).
D-separation (directed graphs) and separation (undirected graphs) are the graph-theoretic tests for reading conditional independence directly from a graph’s structure.
Introducing latent variables h lets a model represent indirect dependencies between visible variables v without needing an explicit, expensive higher-order clique connecting all of them directly.
A restricted Boltzmann machine’s energy function E(v,h) = -b·v - c·h - vᵀWh has no visible-visible or hidden-hidden connections, so p(h|v) and p(v|h) both factorize into independent per-unit conditionals, enabling efficient block Gibbs sampling.

Memory Tip: Tie “restricted” in Restricted Boltzmann Machine directly to what it buys you: removing every visible-visible and hidden-hidden connection is exactly what makes p(h|v) and p(v|h) each factorize into simple, independent per-unit probabilities, which is why RBMs can be sampled efficiently by alternating (block) Gibbs sampling between the two layers.

Common Mistake: Assuming that because a graphical model can be drawn as a graph, sampling from it is always easy. Directed models allow cheap ancestral sampling; undirected models often make sampling itself intractable (needing MCMC) because of the unknown partition function Z, even when the model’s structure looks simple.

Important Questions:

What is the key structural property of a restricted Boltzmann machine, and why does it matter? An RBM has no connections within the visible layer or within the hidden layer — only between the two layers — so given the visible units, all hidden units are conditionally independent of each other (and vice versa), which means both p(h|v) and p(v|h) factor into simple per-unit distributions and the model can be sampled efficiently with block Gibbs sampling even though its partition function stays intractable.
Why can’t a directed graphical model simply be normalized the way an undirected one is? A directed graphical model does not need normalization in the first place, because each conditional factor is already a valid, normalized probability distribution by construction, so their product is automatically a valid joint distribution; undirected models use arbitrary non-negative clique potentials that are not individually normalized, so the whole product must be divided by a partition function Z to become a valid probability distribution.

Chapter 17: Monte Carlo Methods

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: Monte Carlo estimators, importance sampling, Markov chain Monte Carlo, Gibbs sampling, mixing between separated modes

When a sum or integral needed for training (most often the intractable partition function’s gradient) cannot be computed exactly, this chapter’s answer is to approximate it by drawing samples. It starts from the basic Monte Carlo estimator, unbiased and converging to the true value as the sample count grows, then introduces importance sampling for when sampling directly from the target distribution p is not possible: reweighting samples drawn from an easier distribution q by the ratio p-tilde/q corrects for the mismatch. Markov chain Monte Carlo (MCMC) is introduced for the more common case in deep learning, where you cannot even sample from p directly: a Markov chain with the right transition operator will, after enough steps, produce samples whose distribution converges to p regardless of where the chain started. Gibbs sampling, the specific MCMC method used throughout the book for RBMs and related models, resamples each variable (or block of variables) from its conditional given all the others. The chapter closes on the practical failure mode of MCMC: when a distribution has multiple modes separated by a region of very low probability, a chain can become trapped in one mode for a very long time before crossing to another.

Key Points:

The basic Monte Carlo estimator is unbiased and its variance decreases as 1/n with the number of samples n, converging to the true expectation by the law of large numbers.
Importance sampling estimates an expectation under p by sampling from a different, easier distribution q and reweighting each sample by p-tilde(x)/q(x); a poorly chosen q can make the estimator’s variance very high.
MCMC methods construct a Markov chain whose stationary distribution is the target distribution p, so running the chain long enough eventually produces samples approximately drawn from p.
Block Gibbs sampling, which resamples an entire layer of conditionally independent variables at once, is what makes Gibbs sampling practical and fast for restricted Boltzmann machines specifically.
The central practical difficulty with MCMC is mixing: when a distribution has separated modes, the chain can take an extremely long time to cross the low-probability region between them.
Tempering (running chains at a higher “temperature,” β < 1) is mentioned as one strategy for helping a chain move between separated modes more easily.

Memory Tip: Keep the two sampling strategies straight by what they need: importance sampling needs you to be able to evaluate p-tilde(x) and to sample from some other, easier q; MCMC needs neither of those, just the ability to evaluate p-tilde(x) at any point, which is exactly why MCMC is the workhorse for the intractable models in this part of the book.

Common Mistake: Assuming that because an MCMC chain has “converged” to its stationary distribution in theory, a short run in practice gives a representative sample. With multiple, well-separated modes, a chain can look perfectly stable while it is actually still trapped in just one mode and has never visited the others.

Important Questions:

What problem does importance sampling solve, and what is its main risk? It lets you estimate an expectation under a target distribution p using samples drawn from a different, more convenient distribution q, by reweighting each sample by p-tilde(x)/q(x); the main risk is that if q assigns much lower probability than p to regions that matter, the importance weights become huge and rare, giving the estimator very high (sometimes effectively infinite) variance.
Why is “mixing between separated modes” specifically called out as the central practical limitation of MCMC methods? Moving from one probability mode to another requires the chain to pass through a region of much lower probability between them, and the probability of making that transition in any given step falls off exponentially with the height of that low-probability barrier, so in high dimensions a chain can remain stuck sampling from just one mode for a very long time even though it is theoretically guaranteed to converge to the full distribution eventually.

Chapter 18: Confronting the Partition Function

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: positive/negative phase, contrastive divergence, stochastic maximum likelihood, pseudolikelihood, score matching, noise-contrastive estimation, annealed importance sampling

Many of the undirected models in this part of the book have an intractable partition function Z, which splits the maximum likelihood gradient into a positive phase that increases the unnormalized probability of real training data, and a negative phase, involving the gradient of log Z, that decreases the unnormalized probability everywhere else, including at samples the model currently favors. The chapter’s storyline is the family of methods built to approximate that negative phase without ever computing Z exactly. Contrastive divergence starts its Markov chains from real training examples rather than random noise, which is cheap but can miss spurious high-probability modes that lie far from the data; stochastic maximum likelihood (persistent contrastive divergence) instead keeps a small set of chains running continuously across training steps, letting them actually find the model’s real modes at the cost of needing careful tuning. The chapter then covers methods that sidestep Z entirely — pseudolikelihood, score matching, and noise-contrastive estimation, which reframes learning as binary classification between real data and samples from a known noise distribution — and closes with direct estimators of Z itself, principally annealed importance sampling.

Key Points:

The log-likelihood gradient for an energy-based model splits into a positive phase (push down the energy of real data) and a negative phase (push up the energy of samples the model currently generates), with the negative phase coming from the gradient of log Z.
Contrastive divergence initializes its Markov chains at real training examples, which converges quickly but can fail to suppress “spurious modes” — regions of high model probability far from any training data.
Stochastic maximum likelihood (persistent contrastive divergence) keeps its Markov chains running persistently across successive gradient steps, letting the chains wander far enough to discover and suppress the model’s actual modes.
Pseudolikelihood avoids Z entirely by training the model to predict each variable given all of the others, since the partition function cancels out of that conditional probability ratio.
Score matching trains the model to match the derivative of the log density with respect to x, which also makes Z’s contribution disappear, since Z does not depend on x.
Noise-contrastive estimation treats the log partition function as an extra learnable parameter and trains the model to distinguish real data from samples generated by a known, tractable noise distribution.

Memory Tip: Keep “positive phase pushes data down in energy, negative phase pushes everything else up” as the one-line summary of every method in this chapter — contrastive divergence, SML, pseudolikelihood and score matching are all just different, cheaper ways of approximating or entirely avoiding that expensive negative phase.

Common Mistake: Treating contrastive divergence and stochastic maximum likelihood as interchangeable. Contrastive divergence’s chains restart near the data on every step and are cheap but can miss modes far from the training set; SML/PCD’s chains persist across steps specifically to fix that blind spot, at the real cost of needing more careful tuning to avoid the chains straying too far or diverging.

Important Questions:

Why does the maximum likelihood gradient for an intractable energy-based model split into a “positive phase” and a “negative phase”? Because the log-likelihood depends on both the unnormalized probability of the data (which the positive phase pushes up) and on the partition function Z, whose gradient equals an expectation of the same gradient under the model’s own distribution (the negative phase), which pushes down the unnormalized probability everywhere the model currently assigns probability.
What is the key practical difference between contrastive divergence and stochastic maximum likelihood? Contrastive divergence reinitializes its Markov chain at a real training example on every gradient step, which is fast but can fail to find and suppress spurious high-probability modes far from the training data; stochastic maximum likelihood keeps a persistent set of chains running continuously across gradient steps, giving the chains a real chance to wander to and correct those distant modes, at the cost of needing more careful tuning of the learning rate and number of Gibbs steps.

Chapter 19: Approximate Inference

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: inference as optimization, evidence lower bound, EM algorithm, MAP inference, mean field variational inference, learned approximate inference

Computing the exact posterior p(h|v) over a model’s latent variables is generally intractable for the deep, richly structured models in this part of the book, so this chapter reframes inference itself as an optimization problem. It defines the evidence lower bound (ELBO), L(v, θ, q) = E_h~q[log p(h,v)] + H(q), which is always a lower bound on log p(v) for any distribution q, and shows inference is exactly the search for the q that makes that bound as tight as possible. The Expectation-Maximization (EM) algorithm is a specific, coordinate-ascent instance of this: the E-step sets q(h|v) to the exact posterior under the current parameters, and the M-step then maximizes the ELBO over the parameters holding q fixed. MAP inference is the special case where q is restricted to a single point mass (a Dirac distribution), which the chapter connects directly to sparse coding’s own inference procedure from Chapter 13. The chapter’s main technical tool is mean field variational inference, which restricts q to a fully factorized family and derives fixed-point update equations for each factor, closing on the modern alternative of learning an inference network directly, the idea the next chapter’s variational autoencoders build on.

Key Points:

The evidence lower bound, L(v, θ, q) = E_h~q[log p(h,v)] + H(q), is a lower bound on log p(v) for any distribution q over h, and equals log p(v) exactly only when q(h|v) is the true posterior.
Inference is reframed as finding the q that maximizes L; when q can range over all distributions, that maximum is achieved exactly at q(h|v) = p(h|v), so exact inference is a special case of this same optimization.
The EM algorithm alternates an E-step, which sets q to the true posterior under the current parameter values, and an M-step, which maximizes the ELBO with respect to the parameters holding q fixed.
MAP inference computes h* = argmax_h p(h|v) rather than a full posterior, and is a special case of the variational framework where q is restricted to a Dirac distribution at a single point.
Mean field variational inference restricts q to a fully factorized distribution over the latent variables, q(h|v) = Π_i q(h_i|v), trading some accuracy for tractability.
Mean field’s factorized assumption forces it to represent a multimodal true posterior by collapsing onto just one of its modes, since a fully factorized distribution cannot represent correlations between the latent variables.

Memory Tip: Keep MAP inference and full variational inference on the same “how much of q do you let vary” dial: MAP collapses q to a single point (a Dirac distribution) at the most probable h, while mean field variational inference lets q vary over an entire factorized family of distributions — both are optimizing the exact same ELBO objective, just over different-sized search spaces for q.

Common Mistake: Assuming mean field variational inference always gives a reasonable approximate posterior. Because it forces q to fully factorize across the latent variables, it structurally cannot represent correlations between them, so when the true posterior is genuinely multimodal, mean field is forced to pick one mode and ignore the rest.

Important Questions:

What is the evidence lower bound (ELBO), and why is it central to approximate inference? It is the quantity L(v, θ, q) = E_h~q[log p(h,v)] + H(q), a lower bound on the true log-likelihood log p(v) for any choice of q, with equality exactly when q equals the true posterior p(h|v); this reframes inference as choosing the q, within whatever family is tractable, that makes this bound as tight as possible, which is what every method in the chapter actually does.
Why does mean field variational inference struggle with a multimodal true posterior? Because mean field restricts q to a fully factorized distribution over the latent variables, it cannot represent dependencies between them, so when the true posterior has multiple separated modes involving correlated latent variables, mean field’s best factorized approximation is forced to concentrate on a single mode instead of representing the true, multimodal shape of the posterior.

Chapter 20: Deep Generative Models

Part 3: Deep Learning Research · Difficulty: Hard · Key topics: Boltzmann machines, deep belief networks, deep Boltzmann machines, reparameterization trick, variational autoencoders, generative adversarial networks

The book’s final chapter surveys deep generative models, building from the Boltzmann machine (a fully general, fully undirected energy-based model over binary units) down to its restricted, tractable-to-sample cousin the RBM, and then up again into genuinely deep architectures: deep belief networks, which stack RBMs and mix directed and undirected connections, trained one layer at a time by the same greedy layer-wise procedure introduced in Chapter 15, and deep Boltzmann machines, which are fully undirected across all their layers and require mean-field variational inference to approximate their intractable posterior. It then turns to directed generative networks defined by a differentiable function that maps a simple latent noise vector z into a data sample x = g(z; θ), and introduces the reparameterization trick, rewriting a random sample as a deterministic function of parameters plus independent noise, as the key mechanism that lets gradients flow back through a sampling step at all. This sets up the chapter’s two headline architectures: the variational autoencoder (VAE), trained by maximizing the same ELBO objective from Chapter 19 using an encoder network and the reparameterization trick, and the generative adversarial network (GAN), in which a generator network is trained to fool a separately trained discriminator network that is simultaneously learning to tell real data from generated samples.

Key Points:

A general Boltzmann machine is a fully connected, undirected energy-based model over binary units; adding latent units increases its representational power but makes learning and inference expensive, which motivated the restricted (RBM) version.
Deep belief networks stack RBMs and are trained one layer at a time by the same greedy layer-wise procedure from Chapter 15, then combined into a hybrid model with both directed and undirected connections.
Deep Boltzmann machines are undirected across every layer (unlike deep belief networks), and their training relies on mean-field variational inference to approximate their genuinely intractable posterior.
The reparameterization trick rewrites a sample y from p(y|ω) as a deterministic function y = f(z; ω) of parameters ω and an independent noise source z, allowing backpropagation through an otherwise non-differentiable sampling operation.
A variational autoencoder maximizes the ELBO, L(q) = E_z~q(z|x)[log p(x|z)] - D_KL(q(z|x) || p(z)), balancing accurate reconstruction against keeping the learned latent distribution close to a simple prior.
A generative adversarial network trains a generator to produce samples indistinguishable from real data and a discriminator, trained simultaneously, to tell real data and generated samples apart, with each network’s training signal coming from the other’s current performance.

Memory Tip: Trace the reparameterization trick back to why it is needed: you cannot backpropagate through a random sampling operation directly, but if you rewrite “sample z from a Gaussian with mean μ and variance σ²” as z = μ + σ·ε, with ε drawn from a fixed standard normal, the randomness moves entirely into ε and μ, σ become ordinary differentiable parameters — this single rewrite is what makes VAEs trainable by plain backpropagation.

Common Mistake: Treating “deep belief network” and “deep Boltzmann machine” as the same architecture. A deep belief network mixes directed connections in its upper layers with one undirected RBM-like connection at the top and is trained layer by layer; a deep Boltzmann machine is undirected throughout its entire depth and needs mean-field inference across all layers jointly.

Important Questions:

What problem does the reparameterization trick solve, and how does it solve it? Backpropagation needs a differentiable path from parameters to the loss, but a random sampling step like z ~ N(μ, σ²) is not itself differentiable with respect to μ and σ; the trick rewrites the sample as a deterministic function of the parameters and an independent noise variable, for example z = μ + σ·ε with ε ~ N(0,1), so μ and σ become ordinary differentiable inputs and gradients can flow through them normally.
What is the core training difference between a variational autoencoder and a generative adversarial network? A VAE trains a single objective, the ELBO, which explicitly balances a reconstruction term against a term keeping the encoder’s approximate posterior close to a prior distribution, using an encoder and decoder trained jointly by ordinary gradient ascent; a GAN instead trains two separate networks, a generator and a discriminator, against each other in a minimax game, with no explicit reconstruction or likelihood term at all — the generator’s only training signal is how well it currently fools the discriminator.

Read Deep Learning Online (Free)

This book has no Creative Commons licence, and MIT Press retains full copyright, but the authors host the complete text free to read as HTML with the publisher’s permission — the site states plainly that “the online version of the book is now complete and will remain available online for free.” Click below to open the official site, where every chapter is free to read in full. There is no official downloadable PDF of the complete book — the authors explain that their contract with MIT Press restricts easily-copied electronic formats, so the free HTML edition is the only authorized way to read the whole book online. Unofficial PDF copies circulate elsewhere; this page links only to the official source.

→ Read Online Free

How to Study This Book

Read Part I (Chapters 2-5) in order at least once, even though it is math rather than deep learning proper. Chapters 6-8 and Chapter 19 assume the reader is already fluent in the gradient, Jacobian and Hessian language Chapter 4 builds, and Chapter 8’s whole optimization discussion is a direct continuation of it. If your BSCS coursework already covered linear algebra, probability and numerical methods separately, skim Chapters 2-4 for notation and go straight into Chapter 5, which is genuinely new material even for a strong math student.

Part II (Chapters 6-12) is the core of a one-semester Deep Learning or Neural Networks elective at most Pakistani universities, and Chapters 6, 7 and 8 form one tight unit that should be read together — a feedforward network (6) trained without regularization (7) using a poorly chosen optimizer (8) is the single most common reason a student’s own implementation “does not work” as a course project. Chapter 9 (CNNs) and Chapter 10 (RNNs and LSTM) are largely independent of each other and can be read in either order depending on whether your course covers vision or sequence models first.

Do not treat Chapter 11 (Practical Methodology) as filler to skip before an exam. It is short, has almost no new equations, and is the one chapter students most regret skipping before a course project, because it is the only place in the book that tells you what order to actually do things in when a model does not train well.

Part III (Chapters 13-20) is written for a reader planning graduate study or research in deep learning, and it assumes everything in Parts I and II while getting noticeably harder. If your BSCS elective only covers “an introduction to deep learning,” Chapters 13-15 (linear factor models, autoencoders, representation learning) are the most approachable entry points, and Chapter 20 (deep generative models) is worth reading on its own even if you skip Chapters 16-19, since VAEs and GANs come up in almost every modern application; understanding Chapters 16-19 in real depth mostly matters if your course or thesis touches probabilistic graphical models or energy-based models directly.

This is a mathematics-and-theory book, not a code-along book — there is no code in it at all. If you want to implement what a chapter describes, pair it with a deep learning framework’s own documentation, or with a code-first companion book on this site’s Deep Learning shelf, which is built specifically to be read alongside a text like this one, chapter concept next to runnable code.

Do not read Chapter 1 last, treating it as filler. Its account of AI winters and the three waves of neural network research explains why the techniques in Chapters 6-8 exist in the specific form they do, and it is genuinely useful material for the “history and motivation” question that many Pakistani university papers include in their first section.

Used In These Programs

This book is used for the Deep Learning / Neural Networks elective in: BS Computer Science. It is also very commonly assigned in MS-level Machine Learning and Deep Learning coursework, though this site’s degree tags currently track undergraduate programs only. Browse all Deep Learning books or all Computer Science books.

Who Should Read This

This book is written for a reader who already programs comfortably and has completed (or is completing) linear algebra, probability, and an introductory machine learning course, and who wants the actual mathematics behind deep learning rather than a framework tutorial. It suits final-year BSCS students taking a Deep Learning or Neural Networks elective, and it is the standard reference many go on to re-read during an MS in Machine Learning. It is not a good first book for someone who has never written a line of Python and just wants to train a model this weekend — there is no code anywhere in it, and a student in that position should start with a code-first book like Think Python to build programming fluency first, and pair this book’s theory with a hands-on deep learning text once the mathematics in Parts I and II feels comfortable.

Applicable Universities

This book is useful for students at Pakistani universities offering BSCS and related programs including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, and other HEC-recognized institutions.

FAQs

Is Deep Learning by Goodfellow, Bengio and Courville free?

Yes, to read. The complete book is free to read online at its official site, deeplearningbook.org, with MIT Press’s permission. It does not carry a Creative Commons or other open licence — MIT Press retains full copyright — so it may be read free online but is not open for reuse or redistribution the way a CC-licensed book is.

Is there an official PDF of the complete book?

No. The only PDF on the official site is the front matter. The authors state their contract with MIT Press restricts easily-copied electronic formats of the complete book, which is why only an HTML edition is offered officially. Unofficial full-book PDFs exist elsewhere online, but they are not the authorized source and this page does not link to them.

Which edition is this, and is it still current in 2026?

This is the original and only edition, published by MIT Press in 2016. The mathematical foundations in Parts I and II (linear algebra, probability, optimization, feedforward and convolutional networks, RNNs and LSTMs) have not dated at all. What the book does not cover, because it predates them, are transformers, attention-only architectures, and large language models — students who need those specifically should treat this book as the prerequisite theory and pair it with newer material for those topics.

How many chapters does the book have?

20 chapters in three parts: Chapter 1 (Introduction) stands alone, Part I (Chapters 2-5) covers applied math and classical machine learning, Part II (Chapters 6-12) covers modern deep network practice, and Part III (Chapters 13-20) covers deep learning research topics including generative models.

Is it good for complete beginners?

Not as a first book. It assumes comfortable linear algebra, probability, and at least an introductory machine learning course, and it has no code at all. A complete beginner should first build programming and basic ML fluency with a book like Think Python, then come to this book once ready to learn the underlying mathematics of deep learning in depth.

Does it cover CNNs, RNNs, LSTMs and generative models like GANs and VAEs?

Yes, in real depth. Convolutional networks are Chapter 9, sequence models including RNNs and LSTM are Chapter 10, and Part III’s final chapter (20) covers deep generative models including variational autoencoders and generative adversarial networks, alongside the Boltzmann machine family that came before them.

Related Books

Think Python – Allen B. Downey (3rd Edition)

Deep Learning by Goodfellow, Bengio and Courville is the closest thing the field has to a shared textbook — the reference almost every later deep learning course, paper and framework tutorial assumes you have at least skimmed. It has no open licence, but the authors’ MIT Press-approved free HTML edition puts the complete text within reach of any BSCS student willing to work through its mathematics. Browse more Computer Science books for the rest of your semester.

Deep Learning by Ian Goodfellow, Yoshua Bengio and Aaron Courville, MIT Press (2016). No Creative Commons or other open licence — all rights remain with MIT Press. Hosted free to read online by the authors with the publisher’s permission. Official source: https://www.deeplearningbook.org/

Freebooks.pk is an educational resource for students in Pakistan. Textbooks and other books belong to their respective boards, authors and publishers, and all rights remain with them. If you are the copyright holder of any material on this page and want it changed or removed, please contact us or file a request under our DMCA policy. We review every request and remove reported content promptly.

Book Overview

Chapter List

Chapter 1: Introduction

Chapter 2: Linear Algebra

Chapter 3: Probability and Information Theory

Chapter 4: Numerical Computation

Chapter 5: Machine Learning Basics

Chapter 6: Deep Feedforward Networks

Chapter 7: Regularization for Deep Learning

Chapter 8: Optimization for Training Deep Models

Chapter 9: Convolutional Networks

Chapter 10: Sequence Modeling: Recurrent and Recursive Nets

Chapter 11: Practical Methodology

Chapter 12: Applications

Chapter 13: Linear Factor Models

Chapter 14: Autoencoders

Chapter 15: Representation Learning

Chapter 16: Structured Probabilistic Models for Deep Learning

Chapter 17: Monte Carlo Methods

Chapter 18: Confronting the Partition Function

Chapter 19: Approximate Inference

Chapter 20: Deep Generative Models

Read Deep Learning Online (Free)

How to Study This Book

Used In These Programs

Who Should Read This

Applicable Universities

FAQs

Is Deep Learning by Goodfellow, Bengio and Courville free?

Is there an official PDF of the complete book?

Which edition is this, and is it still current in 2026?

How many chapters does the book have?

Is it good for complete beginners?

Does it cover CNNs, RNNs, LSTMs and generative models like GANs and VAEs?

Related Books

Leave a Comment Cancel reply