Think Bayes: Bayesian Statistics in Python

BSCS, BS Statistics, and BS Data Science students taking a Bayesian Statistics or advanced Statistics course can read the complete textbook “Think Bayes: Bayesian Statistics in Python” by Allen B. Downey, free from its official source, Green Tea Press. It’s a 20-chapter book that teaches Bayesian reasoning the way a programmer would learn it — by writing and running Python code on real, worked problems, rather than memorizing formulas from a traditional textbook.

This current 2nd edition works through a wide range of real Bayesian problems — from the classic cookie and locomotive problems through classification, regression, and modern sampling-based methods like MCMC — making it a practical companion for anyone who already knows some Python and statistics and wants to build genuine Bayesian intuition through code.

Book Overview

Chapter List

Chapter 1: Probability

Difficulty: Beginner · Bayesian Statistics — Week 1 · Key topics: Probability rules, conjunction, the sum and product rules, working with real datasets in Python

This opening chapter reviews the basic rules of probability — how to compute the probability of a conjunction (both A and B), and the sum and product rules — using a real dataset (survey data from the General Social Survey) rather than abstract coin-flip and dice examples. It sets the book’s practical, code-first tone: every probability rule introduced is immediately applied to real, messy data using Python, not just proven abstractly.

Key Points:

• The probability of a conjunction (A and B both happening) equals P(A) times P(B), but only when A and B are independent — a condition that must be checked, not assumed.
• This book computes probabilities directly from real survey data (the General Social Survey) rather than idealized dice or coin examples, establishing its hands-on, data-first approach from Chapter 1.
• The sum rule (for mutually exclusive events) and product rule (for independent events) are the two basic building blocks every later Bayesian calculation in this book relies on.
• Working with a real dataset means handling practical realities — missing values, coded categories — that a toy probability example would never surface.
• This chapter’s approach — compute probabilities as fractions of real data rather than symbolic formulas — mirrors Downey’s other books (like Think Stats) and previews the entire book’s method.

Practice Tip: Load the actual GSS dataset used in this chapter and recompute the example probabilities yourself in Python — this book’s method only really sinks in once you’ve written the small queries yourself rather than just reading the printed numbers.

Common Mistake: Applying the product rule (multiplying two probabilities to get their conjunction) without first checking that the two events are actually independent — for dependent events this significantly overstates or understates the true joint probability.

Important Questions:

• Q1: When is it valid to compute P(A and B) as P(A) × P(B)? Only when events A and B are independent — meaning knowing whether A happened tells you nothing about whether B happened. For dependent events, this simple multiplication gives an incorrect result; you need conditional probability instead.
• Q2: Why does this chapter use real survey data instead of coins or dice to introduce probability rules? Real data forces you to handle the practical complications — missing values, coded categories, sampling quirks — that every genuine data analysis involves, and previews this book’s overall computational, code-first approach to statistics rather than treating probability purely as abstract symbol manipulation.

Chapter 2: Bayes’s Theorem

Difficulty: Beginner · Bayesian Statistics — Week 1–2 · Key topics: Bayes’s theorem, prior and posterior probability, the diachronic interpretation, the cookie problem

This chapter derives Bayes’s theorem directly from the product rule and introduces the book’s central recurring example, the cookie problem, to build intuition. It introduces the key vocabulary — prior probability, likelihood, and posterior probability — and the “diachronic” interpretation of Bayes’s theorem: updating a belief over time as new evidence arrives, which becomes the book’s dominant mental model for every later chapter.

Key Points:

• Bayes’s theorem follows directly from the product rule of probability — it’s not a separate, mysterious formula but a straightforward algebraic rearrangement.
• The prior represents what you believe before seeing new data; the posterior is your updated belief after incorporating that data via the likelihood of the data under each hypothesis.
• The “diachronic” interpretation treats Bayes’s theorem as updating a belief over time, one piece of evidence at a time — this becomes the book’s central mental model.
• The cookie problem (inferring which of two bowls a cookie came from, given its flavor) is a small, concrete example that makes the abstract prior/likelihood/posterior vocabulary tangible.
• A posterior from one update can become the prior for the next update — Bayesian updating is naturally sequential, incorporating one piece of evidence at a time.

Memory Tip: Keep the cookie problem’s simple prior/likelihood/posterior table in mind as a template — nearly every harder problem in this book, no matter how complex, is solved by setting up exactly this same three-column structure.

Common Mistake: Confusing the likelihood of the data given a hypothesis with the posterior probability of the hypothesis given the data — these are different quantities related by Bayes’s theorem, not interchangeable, and conflating them (the base-rate fallacy) is one of the most common errors in applied probability.

Important Questions:

• Q1: In Bayes’s theorem, what is the difference between the prior and the posterior? The prior is your belief about a hypothesis’s probability before seeing a particular piece of evidence. The posterior is your updated belief about that same hypothesis after incorporating that evidence via Bayes’s theorem — the posterior from one update can serve as the prior for the next.
• Q2: What does the “diachronic” interpretation of Bayes’s theorem mean? It treats Bayesian inference as a process of updating a belief over time, incorporating one new piece of evidence at a time rather than all at once. Each update’s posterior becomes the starting prior for the next update, making Bayesian reasoning naturally sequential.

Chapter 3: Distributions

Difficulty: Beginner · Bayesian Statistics — Week 2 · Key topics: The Pmf class, representing a distribution of hypotheses, the dice problem, the locomotive problem

This chapter introduces the book’s core programming tool, a Pmf (probability mass function) object used throughout to represent a distribution over hypotheses, and shows how to perform a full Bayesian update in code by looping over hypotheses, multiplying by likelihoods, and normalizing. It works through the dice problem and the classic locomotive/German tank problem as worked examples of updating a Pmf with new evidence.

Key Points:

• A Pmf represents a full probability distribution over a set of possible hypotheses, not just a single point estimate — this is the book’s central data structure for Bayesian inference.
• A Bayesian update in code follows one repeated pattern: multiply each hypothesis’s prior probability by the likelihood of the observed data under that hypothesis, then normalize so the probabilities sum to 1.
• The locomotive problem (estimating a total population size, like the number of trains a railroad operates, from observing one randomly numbered train) is a classic example of Bayesian estimation from a single observation.
• Normalizing a Pmf after each update — dividing by the total so probabilities sum to 1 — is a mechanical but essential final step of every Bayesian update.
• Representing an entire distribution (rather than a single best-guess number) is what lets Bayesian methods naturally express uncertainty about an estimate.

Memory Tip: Implement the Pmf update loop (multiply by likelihood, then normalize) yourself in Python for the locomotive problem before moving on — this exact update pattern is reused, almost unchanged, in every subsequent chapter of the book.

Common Mistake: Forgetting to normalize a Pmf after multiplying by likelihoods — the un-normalized values are proportional to the true posterior probabilities but don’t actually sum to 1, so any probability read directly off them before normalizing is wrong.

Important Questions:

• Q1: What does a Pmf represent in this book, and why is it central to the Bayesian approach used here? A Pmf represents a full probability distribution over a set of possible hypotheses, assigning each one a probability. It’s central because Bayesian inference in this book means updating this whole distribution given new data, rather than computing a single point estimate — the full distribution captures uncertainty that a single number cannot.
• Q2: What are the two steps of a Bayesian update on a Pmf, in order? First, multiply each hypothesis’s current (prior) probability by the likelihood of the observed data under that hypothesis. Second, normalize the resulting values by dividing each by their total sum, so the updated (posterior) probabilities once again sum to 1.

Chapter 4: Estimating Proportions

Difficulty: Beginner · Bayesian Statistics — Week 3 · Key topics: The Euro problem, estimating a proportion (like a coin’s bias) from observed data, choice of prior, credible intervals

This chapter works through the Euro problem — estimating the true bias of a coin (or any proportion) from a sequence of observed flips — as a running example for estimating a continuous-valued proportion. It covers how different choices of prior (uniform versus other starting beliefs) affect the posterior, and introduces credible intervals as the Bayesian analogue of a confidence interval, summarizing the posterior distribution’s spread.

Key Points:

• The Euro problem estimates an unknown proportion (like a coin’s true probability of heads) as a full posterior distribution over all possible values between 0 and 1, not a single point estimate.
• A uniform prior represents genuine initial uncertainty (all proportions between 0 and 1 equally likely) but is just one possible choice — other priors encode different starting assumptions and can lead to different posteriors, especially with little data.
• A credible interval is the Bayesian analogue of a confidence interval — a range that contains a stated percentage (e.g. 90%) of the posterior probability, but with a more directly intuitive interpretation than a classical confidence interval.
• As more data accumulates, the posterior distribution’s dependence on the choice of prior shrinks — with enough evidence, different reasonable priors converge toward similar posteriors.
• Unlike a classical confidence interval, a Bayesian credible interval directly states the probability that the true value lies in the given range, given the data and prior — a more natural interpretation for many practical purposes.

Memory Tip: Try the Euro problem with two different priors (a uniform prior and a prior peaked near a fair coin) on the same small dataset, and watch how much the posteriors differ — then repeat with a much larger dataset and watch them converge. Seeing this in code makes the prior’s role, and its fading influence with more data, concrete.

Common Mistake: Interpreting a Bayesian credible interval and a classical confidence interval as meaning exactly the same thing — while numerically similar in simple cases, a credible interval directly states the probability the true value lies within it given the data and prior, while a classical confidence interval’s correct interpretation is about the reliability of the interval-construction procedure across repeated sampling, not about this one interval.

Important Questions:

• Q1: In the Euro problem, what does the posterior distribution represent? It represents the full range of belief about the coin’s true probability of heads, assigning a probability (density) to every possible value between 0 and 1, given the observed flip data and the chosen prior — rather than collapsing to a single best-guess estimate.
• Q2: Why does the choice of prior matter less as more data is collected? With little data, the likelihood from the observed evidence is relatively weak, so the prior’s assumptions strongly shape the posterior. As more data accumulates, the likelihood from the evidence becomes increasingly dominant, and posteriors computed from different reasonable priors tend to converge toward similar distributions.

Chapter 5: Estimating Counts

Difficulty: Intermediate · Bayesian Statistics — Week 4 · Key topics: Estimating an unknown total count, the train problem revisited, sensitivity to the choice of prior for unbounded quantities

This chapter revisits and extends the locomotive/train problem from Chapter 3 to more carefully examine estimating an unbounded count (like a total population size) rather than a bounded proportion. It shows that, unlike the Euro problem, estimating an unbounded quantity can remain quite sensitive to the choice of prior even with a fair amount of data, and discusses strategies (like a power-law prior) for choosing a reasonable prior for this harder class of problem.

Key Points:

• Estimating an unbounded count (like an unknown total population size) is a harder Bayesian problem than estimating a bounded proportion, because there’s no natural upper limit to constrain the prior.
• Unlike the Euro problem’s proportion (bounded between 0 and 1), an unbounded count’s posterior can remain meaningfully sensitive to the prior’s assumptions even with a reasonable amount of data.
• A power-law prior (assigning higher prior probability to smaller values, decreasing according to a power law) is often more realistic than a uniform prior for many real-world count-estimation problems, like population or production totals.
• Choosing a sensible upper bound for a Pmf representing an unbounded quantity is a practical necessity in code, even though the true mathematical distribution has no upper bound — this bound should be generous enough not to distort the posterior.
• Sensitivity analysis — deliberately trying several different reasonable priors and comparing the resulting posteriors — is a practical way to check how much a conclusion actually depends on subjective prior assumptions.

Memory Tip: Whenever you estimate an unbounded quantity (a count, not a proportion), always run the same analysis with at least two meaningfully different reasonable priors and compare the posteriors — if they differ substantially, that sensitivity itself is important information to report alongside the estimate.

Common Mistake: Using a uniform prior by default for estimating an unbounded count without considering whether it’s actually a reasonable assumption — a uniform prior over a very wide range effectively encodes a strong (and often unrealistic) belief that very large values are just as likely as small ones, which can distort the resulting posterior.

Important Questions:

• Q1: Why is estimating an unbounded count generally harder, from a Bayesian perspective, than estimating a bounded proportion? A bounded proportion (like a coin’s bias) has a fixed range (0 to 1), so different reasonable priors tend to converge with enough data. An unbounded count has no natural upper limit, so the choice of prior (especially how quickly it assigns lower probability to larger values) can keep meaningfully affecting the posterior even with a fair amount of data.
• Q2: What is a power-law prior, and why might it be more appropriate than a uniform prior for estimating a count? A power-law prior assigns progressively lower prior probability to larger values according to a power-law relationship, reflecting the common real-world pattern that very large values of a count (like a population or production total) are genuinely less likely than smaller ones. This is often more realistic than a uniform prior, which would treat all values up to some cutoff as equally likely.

Chapter 6: Odds and Addends

Difficulty: Intermediate · Bayesian Statistics — Week 5 · Key topics: Odds versus probability, Bayes’s rule in odds form, the addition of independent random quantities, the Bayesian bandit problem

This chapter introduces odds as an alternative to probability for expressing belief, and shows how Bayes’s theorem takes a particularly clean multiplicative form when expressed in odds (the “Bayes factor” form). It also covers how to compute the distribution of a sum of two independent random quantities by convolving their distributions, and introduces the multi-armed bandit problem as an application combining both ideas.

Key Points:

• Odds express belief as a ratio (e.g. 3-to-1) rather than a probability between 0 and 1; the two are mathematically interconvertible but odds are often more natural for updating with new evidence.
• In odds form, Bayes’s theorem becomes a simple multiplication: posterior odds equal prior odds times the likelihood ratio (the Bayes factor) — a computationally and conceptually clean way to combine evidence.
• The distribution of the sum of two independent random quantities is found by convolving their two individual distributions — a standard but non-obvious operation this chapter introduces concretely in code.
• The multi-armed bandit problem (deciding which of several uncertain options to keep trying, based on evidence gathered so far) is a classic application combining Bayesian updating with sequential decision-making.
• Expressing updates as Bayes factors makes it easy to see exactly how much a given piece of evidence shifts belief, independent of what the prior odds happened to be.

Memory Tip: Practice converting between probability and odds in both directions until it’s automatic (odds = p/(1-p); probability = odds/(1+odds)) — the odds form of Bayes’s theorem is used repeatedly in later chapters, and fumbling the conversion slows down otherwise straightforward calculations.

Common Mistake: Confusing odds with probability when they’re numerically similar for values near 0.5, and using them interchangeably in a calculation — odds and probability diverge substantially away from 0.5 (e.g. a probability of 0.9 corresponds to odds of 9-to-1, not 0.9-to-1), so mixing the two representations in one calculation produces wrong results.

Important Questions:

• Q1: What is the “Bayes factor” form of Bayes’s theorem, expressed in odds? Posterior odds equal prior odds multiplied by the likelihood ratio (the Bayes factor) of the observed data under the two competing hypotheses. This multiplicative form is often more convenient than the probability form, especially for combining multiple independent pieces of evidence one at a time.
• Q2: How do you find the probability distribution of the sum of two independent random quantities? By convolving their two individual probability distributions — for each possible sum value, summing the products of probabilities of every pair of individual values that add up to it. This is a standard operation for combining independent random quantities, implemented directly in code in this chapter.

Chapter 7: Minimum, Maximum, and Mixture

Difficulty: Intermediate · Bayesian Statistics — Week 5–6 · Key topics: Distributions of the minimum/maximum of several random quantities, mixture distributions, combining several Pmfs into one

This chapter covers how to derive the probability distribution of the minimum or maximum of several independent random quantities (for example, the fastest of several race times), building on the CDF concepts. It also introduces mixture distributions — combining several different distributions into one overall distribution, weighted by how likely each component distribution is — a technique used when a quantity could plausibly have come from one of several different underlying processes.

Key Points:

• The CDF of the maximum of several independent random quantities is the product of their individual CDFs; the CDF of the minimum uses a related but distinct formula involving each variable’s survival function.
• Deriving the minimum or maximum’s distribution builds directly on the CDF concepts, showing a concrete, practical use for representing a distribution via its cumulative form rather than its PMF/PDF form.
• A mixture distribution combines several component distributions into one overall distribution, each component weighted by its own probability — useful when a quantity could plausibly arise from one of several distinct underlying processes.
• Computing with a mixture distribution in code means summing each component Pmf, scaled by its mixture weight, into one combined Pmf.
• Both the min/max and mixture techniques in this chapter are building blocks reused in later, more applied chapters (like the classification and comparison chapters).

Memory Tip: Work through a small concrete example (like the minimum of three dice rolls) by hand and in code side by side — verifying the two match builds confidence in the CDF-based min/max formulas before applying them to a less intuitive real problem.

Common Mistake: Treating a mixture distribution as if it were the average of its component distributions’ parameters, rather than a properly weighted combination of the full distributions themselves — a mixture of two very different distributions can have a shape (e.g. bimodal) that no single parameter average would ever produce.

Important Questions:

• Q1: How is the CDF of the maximum of several independent random quantities related to their individual CDFs? The CDF of the maximum equals the product of the individual CDFs of each of the random quantities, evaluated at the same point — because the maximum is less than or equal to some value x only if every individual quantity is less than or equal to x.
• Q2: What is a mixture distribution, and when is it an appropriate model? A mixture distribution combines several different component distributions into one overall distribution, with each component weighted by its own probability of applying. It’s appropriate when a quantity of interest could plausibly have arisen from one of several genuinely distinct underlying processes or subpopulations, rather than a single uniform process.

Chapter 8: Poisson Processes

Difficulty: Intermediate · Bayesian Statistics — Week 6 · Key topics: Poisson processes, the exponential distribution, estimating a rate parameter, the goal-scoring problem

This chapter covers Poisson processes — a model for events (like goals scored in a match, or customer arrivals) that occur independently and at a constant average rate over time. It covers the Poisson and exponential distributions that arise from this model, and works through the goal-scoring problem: using a Bayesian approach to estimate an unknown scoring rate from observed match data.

Key Points:

• A Poisson process models events occurring independently at a constant average rate over time, such as goals in a match or customer arrivals at a shop.
• The number of events in a fixed time interval under a Poisson process follows the Poisson distribution; the time between consecutive events follows the exponential distribution.
• The goal-scoring problem estimates an unknown, underlying scoring rate as a full posterior distribution from observed match data, rather than just reporting the observed goal count as if it were the true rate.
• A Poisson process’s rate parameter itself can be uncertain and estimated with Bayesian methods, layering one level of uncertainty (the rate) beneath another (the count of events given that rate).
• This chapter’s goal-scoring example previews the “hierarchical” style of reasoning — uncertainty about a parameter, and then uncertainty about outcomes given that parameter — used more explicitly in later chapters.

Memory Tip: Keep straight which distribution answers which question: use the Poisson distribution when asking “how many events in this time interval,” and the exponential distribution when asking “how long until the next event” — mixing these up is a very common source of confusion with Poisson-process problems.

Common Mistake: Treating a single observed count (like goals scored in one match) as if it were the true, exact underlying scoring rate, rather than one noisy sample generated from that rate — a Bayesian approach instead treats the rate itself as uncertain and estimates a full posterior distribution over its plausible values.

Important Questions:

• Q1: What are the two key assumptions of a Poisson process? That events occur independently of each other (one event’s occurrence doesn’t change the probability of another), and that they occur at a constant average rate over the time period being modeled — no systematic speeding up or slowing down.
• Q2: In the goal-scoring problem, what does the posterior distribution represent, and why isn’t the observed goal count used directly as the rate? The posterior distribution represents the full range of belief about the team’s true, underlying scoring rate, given the observed match data. A single observed goal count is just one noisy sample from that underlying rate (subject to randomness), so using it directly as “the rate” would ignore the genuine uncertainty a Bayesian approach is designed to capture.

Chapter 9: Decision Analysis

Difficulty: Intermediate · Bayesian Statistics — Week 7 · Key topics: Using a posterior distribution to make an optimal decision, expected value, the price-is-right problem

This chapter shows how a Bayesian posterior distribution feeds directly into a decision, not just a belief — using the famous “Price is Right” showcase bidding problem as its worked example. It covers computing expected value (or expected loss) for each possible decision by combining the posterior distribution with a defined loss or gain function, and choosing the decision that optimizes that expectation.

Key Points:

• A Bayesian posterior distribution isn’t just an end in itself — it can be directly combined with a decision’s payoff or loss structure to compute the expected value of each possible choice.
• The showcase-bidding problem (bidding as close as possible to an unknown prize value without going over) is a classic example where the optimal bid is not simply your best point-estimate guess, but depends on the shape of the whole posterior and the specific rules governing wins and losses.
• Computing expected value for a decision means summing, over every possible true value in the posterior, that value’s probability multiplied by the payoff or loss of the decision under that particular outcome.
• Bayesian decision analysis often recommends a decision that differs from simply choosing the posterior’s most likely value or its mean — the loss function’s specific shape (e.g. asymmetric penalties) can shift the optimal choice.
• This chapter is a direct, practical bridge from statistical inference (estimating what’s true) to decision-making (choosing what to do), which is often the actual point of a real-world Bayesian analysis.

Memory Tip: When solving a decision problem, explicitly write out the loss or payoff function for every combination of possible decision and possible true value before computing anything — skipping this step is the most common reason a decision-analysis calculation goes wrong.

Common Mistake: Assuming the best decision is always to act on the posterior’s single most likely value (the mode) or its mean, without considering the loss function’s shape — when losses or gains are asymmetric (like the showcase-bidding rule that any bid over the actual price wins nothing), the optimal decision can differ meaningfully from either of those simple summary statistics.

Important Questions:

• Q1: How does Bayesian decision analysis combine a posterior distribution with a decision? It computes, for each possible decision, the expected value (or expected loss) by summing, over every possible true value the posterior assigns positive probability to, that value’s probability multiplied by the payoff or loss that decision would produce under that particular true value. The decision with the best expected value is chosen.
• Q2: Why might the optimal decision in a problem like showcase bidding not simply be the posterior’s mean or most likely value? Because the loss or payoff function can be asymmetric or have unusual shape — for instance, bidding even slightly over the true value might mean winning nothing at all, while bidding under still wins something proportional to how close the guess was. This asymmetry can make a bid somewhat below the posterior’s mean the genuinely optimal choice, not merely a cautious compromise.

Chapter 10: Testing

Difficulty: Advanced · Bayesian Statistics — Week 8 · Key topics: Bayesian hypothesis testing, the Bayes factor as evidence, comparing this book’s approach to classical significance testing

This chapter covers Bayesian hypothesis testing — using the Bayes factor (the ratio of likelihoods of the data under two competing hypotheses) to quantify how strongly observed evidence favors one hypothesis over another. It contrasts this approach with classical null-hypothesis significance testing and its p-values, arguing that the Bayesian approach gives a more directly interpretable measure of evidential strength.

Key Points:

• A Bayes factor is the ratio of the likelihood of the observed data under one hypothesis to its likelihood under a competing hypothesis — a direct, interpretable measure of how much the evidence favors one over the other.
• Unlike a classical p-value, a Bayes factor directly compares two specific hypotheses against each other, rather than only assessing evidence against a single null hypothesis in isolation.
• A Bayes factor combines with prior odds (via the odds form of Bayes’s theorem from Chapter 6) to produce posterior odds — making explicit how strong the evidence needs to be to overturn a strongly-held prior belief.
• This book generally argues that Bayes factors give a more directly useful and interpretable answer to “how much should this evidence change my mind” than a classical p-value does.
• Bayesian hypothesis testing requires explicitly specifying both competing hypotheses being compared, which forces more clarity about what’s actually being tested than a generic null-hypothesis significance test.

Memory Tip: When computing a Bayes factor, always be explicit about exactly which two hypotheses are being compared — unlike a classical significance test’s single null hypothesis, a Bayes factor is meaningless without a clearly specified alternative to compare against.

Common Mistake: Treating a Bayes factor as directly equivalent to a classical p-value, or assuming the two approaches will always lead to the same conclusion — they answer related but genuinely different questions, and can sometimes point toward different practical conclusions, especially with borderline evidence.

Important Questions:

• Q1: What does a Bayes factor of 8 (in favor of hypothesis A over hypothesis B) actually mean? It means the observed data is 8 times more likely under hypothesis A than under hypothesis B. Combined with the prior odds between A and B (via the odds form of Bayes’s theorem), it produces updated posterior odds reflecting how much this evidence should shift belief toward A.
• Q2: How does Bayesian hypothesis testing (via Bayes factors) differ conceptually from classical null-hypothesis significance testing? Bayesian testing directly compares the likelihood of the observed data under two explicitly specified competing hypotheses, producing an interpretable measure of relative evidential strength. Classical significance testing instead asks how surprising the observed data would be under a single null hypothesis alone, producing a p-value that doesn’t directly quantify evidence for any specific alternative.

Chapter 11: Comparison

Difficulty: Advanced · Bayesian Statistics — Week 8–9 · Key topics: Comparing two unknown quantities, joint distributions, the Kim Rossmo profiling problem, computing P(A > B)

This chapter covers how to compare two uncertain quantities — for example, determining the probability that one unknown proportion or rate is greater than another — by working with their joint posterior distribution. It shows how to compute quantities like P(A > B) directly from two Pmfs, and applies these comparison techniques to a real, previously-published geographic profiling application.

Key Points:

• Comparing two independent uncertain quantities (like P(A > B), where A and B each have their own posterior distribution) requires considering their joint distribution, not just comparing their individual point estimates like means.
• P(A > B) is computed by summing, over every combination of a value for A and a value for B where A actually exceeds B, the product of their individual probabilities (assuming independence).
• Comparing full posterior distributions rather than single point estimates can reveal that a comparison’s answer is far less certain than comparing means alone would suggest.
• This chapter connects Bayesian comparison techniques to a real, previously-published application (geographic offender profiling), showing these methods applied outside a purely illustrative textbook example.
• Working with a joint distribution over two uncertain quantities is a stepping stone toward the more explicitly multi-parameter models used in even later chapters (like regression).

Memory Tip: Before computing P(A > B) in code, sanity-check with a simple case where you already know the rough answer intuitively (e.g. two nearly identical distributions should give a result close to 0.5) — this catches indexing or independence-assumption mistakes early.

Common Mistake: Comparing two uncertain quantities using only their posterior means (concluding A > B simply because mean(A) > mean(B)) without considering the full distributions — when both posteriors have meaningful spread and overlap, the actual probability that A exceeds B on any given comparison can be much closer to 50% than comparing means alone would suggest.

Important Questions:

• Q1: How do you compute P(A > B) when A and B are two independent quantities with their own posterior distributions? Sum, over every pair of possible values (a from A’s distribution, b from B’s distribution) where a is greater than b, the product of A’s probability of a and B’s probability of b. This properly accounts for the full uncertainty in both quantities, not just their point estimates.
• Q2: Why can comparing two distributions’ means give a misleading sense of certainty compared to computing P(A > B) directly? Two distributions can have different means while still overlapping substantially, meaning there’s real uncertainty about which quantity is actually larger in any given case. Comparing means alone hides this overlap and can create false confidence; computing P(A > B) directly from the full distributions reveals the actual, potentially much less certain, probability.

Chapter 12: Classification

Difficulty: Advanced · Bayesian Statistics — Week 9 · Key topics: Naive Bayes classification, the penguin classification problem, combining multiple features

This chapter applies Bayesian updating to classification — deciding which of several categories an observation most likely belongs to, given several measured features — using a real dataset of penguin measurements as its worked example. It covers the naive Bayes assumption (treating features as conditionally independent given the class) and shows how multiple features combine into a single classification decision through repeated Bayesian updates.

Key Points:

• Classification, from a Bayesian perspective, is repeated Bayesian updating: starting from a prior over possible classes, and updating with each observed feature’s likelihood under each class in turn.
• The naive Bayes assumption treats each feature as conditionally independent of the others, given the class — a simplifying assumption that’s often not exactly true, but makes the calculation tractable and frequently still works well in practice.
• Combining multiple features (like several physical measurements of a penguin) means multiplying together each feature’s individual likelihood contribution, exactly as odds-form Bayes’s theorem combines multiple pieces of evidence.
• This chapter’s real dataset (penguin species classification from physical measurements) grounds the naive Bayes technique in a genuine, verifiable classification task rather than a purely synthetic example.
• Naive Bayes classification directly reuses this book’s core update-and-normalize Pmf pattern from Chapter 3, now applied with multiple sequential pieces of evidence (features) instead of just one.

Memory Tip: Before trusting a naive Bayes classifier’s output, explicitly check whether the conditional-independence assumption is at least roughly reasonable for your features — two strongly correlated features treated as independent can cause the classifier to effectively double-count the same piece of evidence.

Common Mistake: Assuming naive Bayes classification requires all features to be genuinely, perfectly independent given the class before it can be trusted at all — in practice, naive Bayes classifiers often perform reasonably well even when the independence assumption is only approximately true, though strongly correlated features can still meaningfully distort the result.

Important Questions:

• Q1: What is the “naive” assumption in naive Bayes classification? That the different features used for classification (e.g. several physical measurements) are conditionally independent of each other, given the class. This assumption is often not exactly true in real data, but it greatly simplifies the calculation and the classifier frequently still performs reasonably well in practice.
• Q2: How does classifying based on multiple features connect back to the book’s earlier Bayesian update pattern? It applies the same repeated update-and-normalize pattern introduced in Chapter 3, but now sequentially incorporating each feature’s likelihood as one more piece of evidence, one class-Pmf update at a time, until all features have been incorporated into the final posterior over classes.

Chapter 13: Inference

Difficulty: Advanced · Bayesian Statistics — Week 10 · Key topics: Multi-parameter Bayesian models, joint posteriors over several unknowns, marginal distributions

This chapter extends Bayesian inference to problems with more than one unknown parameter at once, working with a joint posterior distribution over several parameters simultaneously rather than a single Pmf. It covers extracting a marginal distribution for a single parameter of interest by summing out the others, when the parameters of a model are related and must be estimated together.

Key Points:

• A multi-parameter Bayesian model produces a joint posterior distribution over all its unknown parameters simultaneously, not a separate independent posterior for each one.
• A marginal distribution for one parameter of interest is obtained by summing (marginalizing) the joint posterior over all the other parameters’ possible values.
• When two or more parameters in a model are genuinely related, estimating them jointly (rather than one at a time, holding others fixed) correctly captures their interdependence.
• Working with a joint posterior over multiple parameters is computationally heavier than a single-parameter Pmf, since the space of joint hypotheses grows multiplicatively with each added parameter.
• This chapter’s move to multi-parameter joint posteriors is a necessary stepping stone toward the more explicitly multi-parameter models (regression, MCMC) covered in the book’s later chapters.

Memory Tip: When working with a joint posterior over two parameters, visualize it as a two-dimensional grid or heatmap before trying to extract any marginal distribution — seeing the joint shape directly makes clear why simply summing out one dimension gives the correct marginal for the other.

Common Mistake: Estimating two related parameters of a model completely separately (one at a time, each with the other treated as fixed or ignored) instead of jointly — when parameters genuinely interact, estimating them independently can miss real correlations between them and understate the true combined uncertainty.

Important Questions:

• Q1: What is a marginal distribution, in the context of a joint posterior over multiple parameters? It’s the distribution for a single parameter of interest, obtained by summing the joint posterior over all possible values of every other parameter. It effectively collapses a multi-dimensional joint distribution down to the one dimension you care about, while still correctly accounting for the full uncertainty from the other parameters.
• Q2: Why is jointly estimating two related parameters different from estimating each one separately? Jointly estimating related parameters correctly captures any correlation or interdependence between them in the resulting joint posterior. Estimating each one separately, holding the other fixed or ignoring it, can miss this interdependence and produce a distorted, overly narrow, or simply incorrect picture of the true combined uncertainty.

Chapter 14: Survival Analysis

Difficulty: Advanced · Bayesian Statistics — Week 11 · Key topics: Survival analysis from a Bayesian perspective, the light bulb problem, censored data

This chapter revisits survival analysis (the time until an event occurs, like a light bulb failing) from a Bayesian perspective, complementing the classical Kaplan-Meier approach covered in Downey’s other book, Think Stats. It works through the light bulb problem, showing how to handle censored observations (bulbs that haven’t failed yet by the time of observation) within a Bayesian updating framework rather than a classical estimator.

Key Points:

• Survival analysis studies the time until an event occurs (like equipment failure) — here approached through direct Bayesian updating of a posterior over the underlying survival-time distribution’s parameters.
• Censored data (observations where the event of interest hasn’t yet occurred by the time of observation) provides real, usable evidence in a Bayesian framework — it updates the posterior by ruling out short survival times, without needing a specialized separate estimator.
• The light bulb problem estimates the parameters of an underlying lifetime distribution from a mix of failure times and (censored) still-working bulbs, all within a single Bayesian update process.
• This Bayesian approach to survival analysis complements, rather than replaces, the classical Kaplan-Meier estimator approach (covered in the author’s companion book, Think Stats) — both are legitimate techniques for the same underlying problem.
• Handling censored data naturally within ordinary Bayesian updating (as one more kind of likelihood) shows the flexibility of the book’s general update-and-normalize framework for genuinely tricky real-world data.

Memory Tip: When setting up the likelihood for a censored observation, remember it isn’t zero and isn’t the same as an observed failure time — it’s the probability that the true failure time is greater than the observed censoring time, a subtly different quantity that’s easy to set up incorrectly at first.

Common Mistake: Excluding censored observations (bulbs still working at the end of observation) from a Bayesian survival analysis entirely, treating only the observed failures as data — this discards real evidence (that these particular bulbs survived at least this long) and biases the resulting posterior toward shorter estimated lifetimes.

Important Questions:

• Q1: How does a Bayesian approach to survival analysis handle a censored observation? It incorporates the censored observation as a specific kind of evidence in the Bayesian update: the likelihood of that observation is the probability that the true failure time exceeds the observed censoring time, given each candidate set of distribution parameters — not zero, and not the same as an actual observed failure time.
• Q2: How does this chapter’s Bayesian light bulb problem relate to the classical Kaplan-Meier approach to survival analysis? Both address the same underlying problem — estimating a survival-time distribution from data that includes censored observations — but through different mechanics. The Kaplan-Meier estimator (a classical, non-Bayesian method) builds a survival curve directly from the data; this chapter’s Bayesian approach instead updates a full posterior distribution over the underlying distribution’s parameters, treating censored data as one more kind of evidence in that update.

Chapter 15: Mark and Recapture

Difficulty: Advanced · Bayesian Statistics — Week 11–12 · Key topics: Mark-and-recapture population estimation, the grizzly bear problem, estimating population size from capture-recapture data

This chapter covers mark-and-recapture, a classic technique (originally from ecology) for estimating an unknown total population size by capturing, marking, releasing, and later recapturing individuals, and observing how many recaptured individuals were previously marked. It works through the grizzly bear problem as a worked Bayesian estimation example, building on the count-estimation ideas from Chapter 5.

Key Points:

• Mark-and-recapture estimates an unknown total population size from a two-stage sampling process: marking a sample of individuals, then later observing what fraction of a second sample was already marked.
• The fraction of marked individuals found in the second sample provides evidence, via Bayesian updating, about the total population size — a smaller recaptured-marked fraction suggests a larger total population.
• This technique directly builds on the unbounded-count estimation ideas from Chapter 5, applying that same general approach to a specific, practically important ecological and epidemiological estimation problem.
• The grizzly bear problem models the specific mechanics of the capture-recapture process’s likelihood function, showing that a slightly more involved likelihood doesn’t require abandoning the same basic Bayesian update-and-normalize approach.
• Mark-and-recapture techniques generalize well beyond ecology — the same statistical logic applies to estimating totals in software bug-counting, epidemiology, and other “count what you can’t fully observe” problems.

Memory Tip: Before coding the grizzly bear problem’s full likelihood, work out by hand what evidence a very high versus very low recaptured-marked fraction would imply about the total population — having that intuition first makes it much easier to verify the code’s Bayesian update is behaving sensibly.

Common Mistake: Assuming the fraction of marked individuals recaptured in the second sample directly equals the fraction of the total population that was marked, and using that ratio alone as a point estimate — without a full Bayesian treatment, this simple ratio approach ignores the real sampling uncertainty, especially with small sample sizes, that a proper posterior distribution captures.

Important Questions:

• Q1: What is the basic logic behind mark-and-recapture population estimation? A sample of individuals is captured, marked, and released. Later, a second sample is captured, and the fraction of that second sample which was already marked provides evidence about the total population size — a low recaptured-marked fraction suggests the total population is large (the marked individuals are a small fraction of it), while a high fraction suggests a smaller total population.
• Q2: How does the mark-and-recapture problem in this chapter connect to the unbounded-count estimation techniques from Chapter 5? Both problems involve estimating an unknown, unbounded total count from indirect evidence, using a Bayesian posterior over a wide range of plausible population sizes rather than a single point estimate. The grizzly bear problem applies that same general framework, but with a more specific likelihood function tailored to the mechanics of the two-stage capture-recapture sampling process.

Chapter 16: Logistic Regression

Difficulty: Advanced · Bayesian Statistics — Week 12 · Key topics: Logistic regression from a Bayesian perspective, modeling a binary outcome, comparing to the classical MLE approach

This chapter introduces logistic regression — modeling a binary outcome’s probability as a function of one or more explanatory variables — from a Bayesian perspective, computing a full posterior over the model’s coefficients rather than a single maximum-likelihood point estimate. It connects to the classical logistic regression covered in Think Stats, showing the same underlying model handled with full Bayesian uncertainty instead of point estimates alone.

Key Points:

• Logistic regression models a binary outcome’s probability as a function of explanatory variables via the logistic (sigmoid) function applied to a linear combination of those variables.
• A Bayesian approach to logistic regression produces a full joint posterior distribution over the model’s coefficients, rather than a single best-fit (maximum likelihood) point estimate for each coefficient.
• Having a posterior over coefficients (rather than a single point estimate) directly supports quantifying uncertainty in predictions, not just uncertainty in the coefficients themselves.
• This Bayesian treatment of logistic regression connects directly to the classical (maximum-likelihood) logistic regression covered in the author’s companion book, Think Stats, as two different ways of fitting the same underlying model.
• As the amount of data grows, a Bayesian logistic regression’s posterior typically concentrates increasingly tightly around the same coefficient values a classical maximum-likelihood fit would find.

Memory Tip: When interpreting a Bayesian logistic regression’s results, look at the posterior distribution over a coefficient, not just its mean — a wide posterior for a coefficient (even with a reasonable-looking mean) signals real uncertainty about whether that variable actually matters, information a single point estimate would hide.

Common Mistake: Reporting only the posterior mean of each logistic regression coefficient and discarding its spread, effectively treating the Bayesian result the same as a classical point-estimate fit — this throws away the main practical advantage of the Bayesian approach, which is a full, usable measure of coefficient (and prediction) uncertainty.

Important Questions:

• Q1: What does logistic regression model, and what function connects the linear combination of explanatory variables to that outcome? It models the probability of a binary outcome (like yes/no, survived/didn’t). The logistic (sigmoid) function is applied to a linear combination of the explanatory variables, converting an unbounded linear score into a properly bounded probability between 0 and 1.
• Q2: What is the key practical difference between a Bayesian and a classical (maximum-likelihood) approach to fitting a logistic regression? A Bayesian approach produces a full joint posterior distribution over the model’s coefficients, directly capturing uncertainty about each coefficient’s true value (and, by extension, uncertainty in predictions). A classical maximum-likelihood approach instead produces a single best-fit point estimate for each coefficient, typically supplemented separately with standard errors rather than a full joint distribution.

Chapter 17: Regression

Difficulty: Advanced · Bayesian Statistics — Week 12–13 · Key topics: Linear regression from a Bayesian perspective, posterior over slope and intercept, prediction intervals

This chapter covers ordinary linear regression — fitting a line to continuous data — from a Bayesian perspective, computing a joint posterior over the line’s slope and intercept (and the residual noise level) rather than a single least-squares best-fit line. It shows how this full posterior directly supports prediction intervals that properly reflect both parameter uncertainty and the model’s residual noise.

Key Points:

• Bayesian linear regression computes a joint posterior distribution over the line’s slope, intercept, and residual noise level, rather than a single best-fit line from ordinary least squares.
• A prediction interval built from this full posterior properly reflects two separate sources of uncertainty: uncertainty about the fitted line’s parameters themselves, and the inherent residual noise around any fitted line.
• With a large amount of data, a Bayesian regression’s posterior for the slope and intercept typically concentrates near the same values ordinary least-squares regression would find, converging toward the same practical answer.
• Unlike classical least-squares regression’s point estimates and separately-computed confidence intervals, the Bayesian posterior directly and naturally answers questions like “what’s the probability the true slope is positive?”
• This chapter connects directly back to the ordinary least-squares regression covered in Think Stats, presenting the same underlying linear model through a fully Bayesian lens.

Memory Tip: Distinguish carefully between a confidence interval for the regression line itself (capturing parameter uncertainty alone) and a prediction interval for a new individual data point (which must also include residual noise) — conflating the two is one of the most common regression mistakes, Bayesian or classical.

Common Mistake: Using only the posterior means of the slope and intercept to build a prediction interval, ignoring the residual noise term entirely — this produces an interval that’s far too narrow, since it only reflects uncertainty about the line’s exact position and completely omits the genuine scatter of individual data points around any given line.

Important Questions:

• Q1: What three things does a Bayesian linear regression’s joint posterior distribution typically cover? The line’s slope, its intercept, and the level of residual noise (how much individual data points scatter around the fitted line) — all estimated jointly as one combined posterior distribution, rather than as three separately-estimated quantities.
• Q2: Why does a proper prediction interval need to account for both parameter uncertainty and residual noise, not just one or the other? Parameter uncertainty alone (uncertainty about exactly where the true line lies) doesn’t account for the fact that individual data points always scatter around any given line due to inherent noise. A prediction interval for a genuinely new data point needs both sources combined — otherwise it will be systematically too narrow and understate the true range of plausible new observations.

Chapter 18: Conjugate Priors

Difficulty: Advanced · Bayesian Statistics — Week 13 · Key topics: Conjugate priors, the beta-binomial and gamma-Poisson pairs, closed-form Bayesian updates versus grid approximation

This chapter introduces conjugate priors — specific pairings of prior distribution and likelihood function (like the beta distribution paired with a binomial likelihood) for which the posterior has a closed-form solution, computable instantly without the grid-based numerical approximation used throughout most of the book. It covers when conjugate priors are available and worth using, and when the book’s general grid-approximation approach is still the more practical choice.

Key Points:

• A conjugate prior is a specific choice of prior distribution that, paired with a particular likelihood function, produces a posterior in the same distributional family as the prior — allowing the update to be computed instantly via a closed-form formula.
• The beta distribution is the conjugate prior for a binomial likelihood (as in the Euro coin-bias problem); the gamma distribution is the conjugate prior for a Poisson likelihood (as in the goal-scoring problem).
• Conjugate priors are computationally much faster than the grid-approximation approach used throughout most of this book, since they avoid explicitly representing and updating a full discretized Pmf.
• Conjugate priors exist only for specific prior-likelihood pairings — many realistic Bayesian models used elsewhere in the book have no known conjugate prior and must rely on grid approximation or other numerical methods instead.
• This chapter reframes several of the book’s earlier worked examples (like the Euro problem) using their conjugate closed-form solutions, showing the same answers reached both numerically and analytically.

Memory Tip: Recognize the two conjugate pairs covered here by their use case: beta-binomial for estimating a proportion from success/failure counts, and gamma-Poisson for estimating a rate from an observed count of events — spotting which pair applies lets you skip straight to a closed-form update instead of building a numerical grid.

Common Mistake: Assuming every Bayesian estimation problem has a convenient conjugate prior and searching for one before resorting to grid approximation — most realistic models covered later in this book (multi-parameter regression, mixtures, MCMC-based models) have no known simple conjugate solution, and grid approximation or sampling methods are the practical default, not a fallback of last resort.

Important Questions:

• Q1: What makes a prior distribution “conjugate” to a particular likelihood function? The posterior distribution, after updating that conjugate prior with the likelihood, comes out in the exact same distributional family as the prior itself — so the update can be computed instantly with a closed-form formula, without needing to numerically represent and update a full discretized probability distribution.
• Q2: What is the main practical advantage of using a conjugate prior when one is available, and what is its main limitation? The main advantage is computational speed and simplicity — the posterior is computed instantly via a known formula rather than grid approximation. The main limitation is that conjugate priors only exist for specific, limited prior-likelihood pairings, so many realistic, more complex Bayesian models have no available conjugate solution and must rely on other numerical methods.

Chapter 19: MCMC

Difficulty: Advanced · Bayesian Statistics — Week 14 · Key topics: Markov Chain Monte Carlo, sampling-based Bayesian inference, using PyMC, when grid approximation stops being practical

This chapter introduces Markov Chain Monte Carlo (MCMC) methods, using the PyMC library, as a sampling-based alternative to this book’s grid-approximation approach for Bayesian inference — necessary once a model has too many parameters for grid approximation to remain computationally practical. It covers the basic idea of drawing samples from a posterior distribution rather than computing it explicitly on a grid, and using those samples to answer the same kinds of questions.

Key Points:

• MCMC methods draw a large number of representative samples from a posterior distribution, rather than computing the distribution explicitly on a discretized grid, as most of the book’s earlier chapters do.
• Grid approximation becomes computationally impractical once a model has more than a few parameters, since the number of grid points needed grows multiplicatively with each additional parameter — MCMC scales much better to higher-dimensional models.
• PyMC (a Python probabilistic programming library) is used in this chapter to define a Bayesian model declaratively and let the library’s MCMC sampler handle drawing posterior samples automatically.
• Once you have posterior samples from MCMC (rather than a full explicit distribution), most of the same quantities computed throughout the book — means, credible intervals, P(A > B) comparisons — can be estimated directly from those samples.
• This chapter is a deliberate bridge from the book’s hand-built grid-approximation methods to the sampling-based tools (like PyMC and Stan) commonly used for large-scale, real-world Bayesian modeling.

Memory Tip: Re-solve one of the book’s earlier grid-approximation problems (like the Euro problem) using PyMC and compare the results — seeing that MCMC sampling and explicit grid approximation converge on the same answer for a problem simple enough for both builds real confidence before relying on MCMC for a model too complex for grid approximation.

Common Mistake: Reaching for grid approximation on a model with many parameters simply because it was this book’s primary method throughout, without recognizing that grid approximation’s cost grows multiplicatively with each added parameter — for models with more than a handful of parameters, this makes grid approximation computationally impractical, and a sampling-based method like MCMC becomes necessary instead.

Important Questions:

• Q1: Why does grid approximation, this book’s primary method through most chapters, become impractical for models with many parameters? The number of grid points needed to adequately cover the space of possible parameter combinations grows multiplicatively with each additional parameter — a model with even a modest number of parameters can require a computationally infeasible number of grid points to evaluate explicitly.
• Q2: What is the basic idea behind MCMC, as an alternative to grid approximation? Instead of explicitly computing a posterior distribution’s probability at every point on a grid, MCMC methods draw a large number of representative samples directly from the posterior distribution. Standard quantities of interest (means, credible intervals, comparisons between quantities) can then be estimated directly from this collection of samples, rather than from an explicit distribution.

Chapter 20: Approximate Bayesian Computation

Difficulty: Advanced · Bayesian Statistics — Week 14–15 · Key topics: Approximate Bayesian Computation (ABC), simulation-based inference when the likelihood is hard to compute directly

This closing chapter covers Approximate Bayesian Computation (ABC), a simulation-based technique for Bayesian inference used when a model’s likelihood function is difficult or impossible to write down explicitly, but the model can still be simulated. It covers the core ABC idea — simulating data under many candidate parameter values and keeping only those whose simulated output closely resembles the actual observed data — as a way to approximate a posterior without ever computing a likelihood directly.

Key Points:

• Approximate Bayesian Computation (ABC) is used when a model’s likelihood function is intractable or impossible to write down explicitly, but the underlying process can still be simulated given a set of parameters.
• The core ABC procedure: repeatedly simulate data under many different candidate parameter values, and keep only the candidate parameter values whose simulated output closely matches the actually observed data.
• The retained (accepted) candidate parameter values, taken together, form an approximation to the true posterior distribution — without ever needing to compute an explicit likelihood function.
• ABC trades exactness for applicability — it’s an approximation, and its quality depends heavily on how “closely matching” simulated versus observed data is defined and how much simulation is run.
• As this book’s final chapter, ABC represents the most flexible but also most computationally intensive technique covered, appropriate specifically for problems where every earlier, more exact chapter’s methods don’t apply.

Memory Tip: Before reaching for ABC on a new problem, first ask honestly whether the model’s likelihood really can’t be written down explicitly — ABC is a powerful last resort for genuinely intractable likelihoods, not a shortcut to avoid the more precise grid-approximation or MCMC methods covered earlier when those remain applicable.

Common Mistake: Reaching for Approximate Bayesian Computation as a default technique for any hard Bayesian problem, rather than only when the model’s likelihood genuinely cannot be computed directly — ABC’s approximation introduces its own error and computational cost, so grid approximation or MCMC (which use the likelihood directly and more precisely) should be preferred whenever they’re actually applicable.

Important Questions:

• Q1: When is Approximate Bayesian Computation (ABC) the appropriate technique to use, as opposed to grid approximation or MCMC? When the model’s likelihood function is intractable or effectively impossible to compute directly, but the underlying data-generating process can still be simulated given a set of candidate parameter values. Grid approximation and MCMC both require being able to evaluate a likelihood; ABC is specifically for cases where that’s not possible.
• Q2: Describe the basic ABC procedure in one or two sentences. Simulate data repeatedly under many different candidate parameter values, then keep only the candidate values whose simulated output closely resembles the actually observed data. These retained candidate values, taken together, approximate the true posterior distribution over the model’s parameters.

Read Think Bayes Online (Free)

This book is free from its official source — author Allen B. Downey publishes the current, actively maintained 2nd edition under a Creative Commons licence. There is no official free PDF for this edition — the legitimate free way to read it is the complete, official web edition below, with every chapter fully readable online at no cost, alongside the accompanying Jupyter notebooks.

How to Study This Book

This book is organized to build progressively: Chapters 1–9 (Probability through Decision Analysis) establish the core Bayesian updating pattern — prior, likelihood, posterior — using increasingly involved worked examples, and should be read in order.

Chapters 10 onward (Testing through Approximate Bayesian Computation) apply that core pattern to more specialized problems — hypothesis testing, comparison, classification, regression, and advanced sampling methods — and can be read more selectively depending on your course’s focus.

This book teaches Bayesian statistics by building small Python programs around a shared Pmf (probability mass function) data structure introduced in Chapter 3 — actually running the example code for each chapter, not just reading the text, is essential to how this book is meant to be used.

Chapters 18–20 (Conjugate Priors, MCMC, and Approximate Bayesian Computation) are more advanced and computationally intensive — some courses may treat them as optional or cover them only briefly; check your own syllabus.

This book assumes working knowledge of Python and basic probability — readers who need a gentler introduction to statistics with Python first may want to work through the author’s companion book, Think Stats, before this one.

Used In These Programs

This book is used as an applied, programming-based Bayesian statistics text in: BSCS, Programming, and Python-based Bayesian Statistics or advanced Statistics coursework. Browse all Python books or all Computer Science category books.

Who Should Read This

Think Bayes is written for students and self-learners who already know some Python and basic probability, and want to learn Bayesian statistics by writing code on real, worked problems rather than through formula-heavy derivations. It suits BSCS, BS Statistics, and BS Data Science students taking a Bayesian Statistics or advanced Statistics course, and is especially useful for anyone who has already worked through a first statistics course (such as the author’s companion book, Think Stats) and wants a practical, computational path into Bayesian methods.

Applicable Universities

This book is useful for students at Pakistani universities offering BSCS, BS Statistics, or BS Data Science programs with a Bayesian Statistics, Probabilistic Modeling, or advanced Statistics course, including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, GIKI, and other HEC-recognized institutions, and for self-learners who already know Python and want a free, code-first Bayesian statistics reference.

FAQs

Related Books

• Think Stats – Allen B. Downey (a gentler, foundational statistics-with-Python companion to this book)
• Python for Everybody – Charles R. Severance (a foundational Python course for readers newer to programming)

Think Bayes is a free, code-first Bayesian statistics textbook that teaches Bayesian updating, estimation, hypothesis testing, and modeling by writing Python programs on real, worked problems. Browse more Computer Science books for the rest of your semester.

Think Bayes: Bayesian Statistics in Python, by Allen B. Downey. Free under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 licence. Access for free at https://greenteapress.com/wp/think-bayes/