Bayesian Methods for Hackers – Free Book (BSCS/BS Stats)

BSCS and BS Statistics students can read “Bayesian Methods for Hackers” (full title: “Probabilistic Programming and Bayesian Methods for Hackers”) by Cameron Davidson-Pilon, a free, computation-first introduction to Bayesian inference built entirely around PyMC. Instead of the usual two or three chapters of probability theory before Bayesian inference even starts, the book gets you building working models and drawing posterior samples in Chapter 1, and argues that computing power lets you skip the intractable mathematics most textbooks get stuck on.

The book is six chapters plus a short Prologue, free to read on its official GitHub repository as Jupyter notebooks, rendered live through nbviewer or run locally after cloning. There is no official PDF — the author’s own README calls a generated PDF “the least-preferred method to read the book” because it is static, so this page links to the free notebooks rather than to a download. Each chapter folder also carries the original 2013–2016 notebooks for historical reference and a maintained current-PyMC version (pinned to PyMC 5 in the repo’s own requirements file) — the chapter list below tells you exactly which file to open.

Book Overview

Chapter List

Chapter 1: Introduction to Bayesian Methods

Difficulty: Easy · Key topics: Bayesian vs frequentist probability, Bayes’ theorem, prior and posterior, Poisson distribution, change-point detection, PyMC models, MCMC sampling

The chapter opens with a debugging analogy — trusting code more after it passes many tests without ever being fully certain — to introduce Bayesian inference as belief updated by evidence rather than certainty derived from randomness. It contrasts the frequentist view of probability as long-run frequency with the Bayesian view of probability as a measure of belief, then states Bayes’ theorem, P(A|X) ∝ P(X|A)P(A), connecting a prior belief P(A) to an updated posterior belief P(A|X). The chapter’s running example models a person’s daily text-message counts as two different Poisson rates, λ₁ and λ₂, separated by an unknown switch-point day τ, builds this as a PyMC model, and uses MCMC sampling to recover posterior distributions for all three unknowns. It closes by using those posterior samples to estimate the expected number of texts on any given day.

Key Points:

• Bayesian probability is a measure of belief, updated by evidence; frequentist probability is the long-run frequency of an event.
• Bayes’ theorem, P(A|X) ∝ P(X|A)P(A), is how a prior belief P(A) becomes a posterior belief P(A|X) once evidence X is observed.
• As more evidence accumulates, the posterior is increasingly shaped by the data and less by the prior.
• The chapter’s worked example models daily text counts with two Poisson rates, λ₁ and λ₂, and an unknown switch-point day τ.
• MCMC returns thousands of samples from the posterior distribution rather than a single number — that is what a Bayesian answer looks like.

Practice Tip: Run the text-message change-point model on the book’s own data, then re-run it after synthetically shifting where the rate change happens in the data, and confirm the posterior for τ moves to match.

Common Mistake: Treating the posterior mean as “the answer” and discarding the spread of the distribution, instead of reading that spread as how confident (or not) the inference actually is.

Important Questions:

• What is the practical difference between a frequentist estimate and a Bayesian estimate of an unknown quantity? A frequentist method returns a single number; a Bayesian method returns a full distribution of plausible values, so you can see how confident you should be, not just what the estimate is.
• In the text-message example, what do λ₁, λ₂ and τ represent? λ₁ and λ₂ are the Poisson rate parameters before and after a behaviour change; τ is the unknown day the change happened. All three are inferred from the observed daily counts.

Chapter 2: A Little More on PyMC

Difficulty: Medium · Key topics: stochastic and deterministic variables, binomial distribution, randomized-response privacy trick, A/B testing, logistic regression, Challenger O-ring failure, goodness of fit

This chapter separates PyMC’s two kinds of variable: stochastic (genuinely random, like pm.Poisson) and deterministic (computed from other variables, via pm.Deterministic or ordinary arithmetic on PyMC variables). It works a Bayesian A/B testing example for comparing two website or treatment designs, then introduces the binomial distribution through a clever privacy-preserving survey algorithm: each student secretly flips two coins so an interviewer can never tell whether a “yes” is a genuine confession or a coin result, yet the true cheating rate is still recoverable in aggregate. The chapter’s centrepiece models the 1986 Challenger disaster, fitting a logistic function of outside temperature to O-ring defects and asking, honestly, whether that model is actually a good fit to the data rather than assuming it.

Key Points:

• A stochastic PyMC variable represents a random quantity; a deterministic variable is computed from other variables via pm.Deterministic or ordinary Python arithmetic on them.
• The binomial distribution has two parameters — N trials and p, the per-trial probability — and is discrete on the integers 0 to N.
• The randomized-response trick lets an interviewer never know whether an individual “yes” is a real admission or a coin flip, while the true rate is still recoverable across the whole group.
• The Challenger O-ring model uses a logistic function of temperature and predicts an almost certain defect at 31°F, the actual launch-day temperature.
• Checking goodness of fit means simulating artificial datasets from your fitted model and comparing them to the real observed data, not just assuming the model is right because it compiled and ran.

Practice Tip: Build the cheating-survey model with a made-up N and observed “yes” count, and confirm the recovered posterior for the true cheating rate lands systematically below the raw “yes” proportion, since half of every “yes” answer is just a coin flip.

Common Mistake: Reading a deterministic PyMC variable as if it carries independent uncertainty of its own, when its whole posterior is entirely a function of the stochastic variables that define it.

Important Questions:

• Why does the coin-flip (randomized-response) algorithm protect a student’s privacy while still letting you estimate the true cheating rate? No observer can tell whether a given “yes” is a genuine admission or the result of a second, hidden coin flip, but because the coin’s own probability is known exactly, the true underlying rate can still be solved for algebraically from the aggregate count.
• What does “checking goodness of fit” mean in this chapter, and how is it done for the Challenger model? It means testing whether your fitted model could plausibly have generated the observed data, done here by simulating new artificial datasets from the fitted logistic model and comparing their shape to the real O-ring data.

Chapter 3: Opening the Black Box of MCMC

Difficulty: Medium · Key topics: the posterior as a landscape, MCMC as intelligent search, mixture models, clustering, autocorrelation, thinning, trace diagnostics, starting values

The chapter pictures every Bayesian inference problem as an N-dimensional space of possible unknowns, with the prior as a surface sitting over it and the true posterior as a deformed, data-shaped version of that surface. MCMC is framed as an intelligent search for the “mountains” of high probability in that space, returning a sequence of samples (called traces) rather than a formula. It works through an unsupervised clustering example using a mixture model, then gives a sharp warning: elements from different posterior samples must never be mixed, because within one MCMC draw the unknowns are jointly dependent on each other. The chapter closes with diagnosing convergence through autocorrelation, and reducing it by thinning the trace or giving the sampler a sensible starting value.

Key Points:

• MCMC does not return a formula for the posterior; it returns a sequence of samples, called traces, drawn from it.
• In an N-unknown model, the space of possible parameter combinations grows exponentially with N — naive brute-force search over that space is infeasible.
• Elements from different posterior samples must never be mixed, because the unknowns within a single MCMC draw are jointly dependent (e.g. two cluster standard deviations constrain each other).
• Autocorrelation measures how related a chain’s value at time t is to its value at time t-k; high autocorrelation means consecutive samples are not really independent.
• Thinning — keeping only every nth sample — is a standard way to reduce autocorrelation in a trace.
• Giving the sampler a sensible starting value (e.g. the data’s own mean, via testval) speeds convergence, though MCMC will still converge eventually from a poor start.

Memory Tip: Picture the posterior as a mountain range hidden in fog, and MCMC as a hiker repeatedly asking “is this pebble more likely to be from the mountain I’m looking for than the last one?” — the pebbles collected over the whole walk are your samples.

Common Mistake: Pairing element i of one unknown’s trace with element j (where i ≠ j) of a dependent unknown’s trace, instead of always indexing every unknown from the same MCMC iteration.

Important Questions:

• Why can’t you just search an N-dimensional parameter space directly to find where the posterior peaks? Because the volume of the space grows exponentially with the number of unknowns (the curse of dimensionality), so MCMC performs an intelligent, sample-based search instead of an exhaustive one.
• What does thinning a trace accomplish, and when would you need it? Keeping only every nth sample reduces autocorrelation between the kept samples, which matters because many post-processing steps assume the samples they use are independent of each other.

Chapter 4: The Greatest Theorem Never Told

Difficulty: Medium · Key topics: the Law of Large Numbers, convergence at large N, small-sample bias, Reddit/ratings sorting, star-rating correction, expected value from MCMC samples

The chapter states the Law of Large Numbers formally — the average of N independent samples of a random variable converges to its expected value as N grows — and proves the intuition by splitting a two-valued variable’s average into its weighted frequencies. It ties this directly back to Bayesian computation: sampling-based inference turns analytically intractable expected-value integrals into simple sample averages. It then shows where the law fails at small N, using a geographic-averages example and the well-known problem of sorting items (Reddit comments, star ratings) by their average score, where a handful of enthusiastic votes can push a thin-evidence item to the top. A corrected ranking formula is derived that discounts scores with too little supporting data.

Key Points:

• The Law of Large Numbers: the average of N i.i.d. samples of a random variable Z converges to E[Z] as N grows, provided E[Z] is finite.
• This is why sampling-based Bayesian inference works at all — instead of solving an intractable integral for an expected value, you average your MCMC samples.
• The law only holds for large N; averages from small samples (a county with few residents, a product with one review) are unreliable and should not be trusted at face value.
• Sorting items purely by their raw average rating is exploitable by items with very few, coincidentally high, votes.
• The corrected ranking formula (using upvotes/downvotes as parameters a and b) shrinks small-sample scores toward uncertainty instead of trusting the raw average.
• The same shrinkage idea extends from binary upvote/downvote data to N-star rating systems by rescaling stars to a 0–1 proportion first.

Memory Tip: “Small sample, big claim, don’t believe it” — any striking statistic built on a tiny N (one county, two reviews, one quarter of data) deserves exactly the suspicion this chapter teaches for a single 5-star review.

Common Mistake: Ranking or sorting items by a raw average (rating, click-through rate, conversion rate) without adjusting for how many observations that average is actually based on.

Important Questions:

• Why does the Law of Large Numbers matter for Bayesian inference specifically, beyond being a general probability result? It justifies estimating a posterior expected value by simply averaging your MCMC samples instead of solving an analytically intractable integral by hand.
• What goes wrong when items are ranked purely by average rating, and how does the chapter fix it? Items with very few ratings can show an extreme, unreliable average purely by chance; the fix uses a formula that discounts a score based on how few votes support it, so a rating needs more supporting evidence to rank highly.

Chapter 5: Would You Rather Lose an Arm or a Leg?

Difficulty: Hard · Key topics: loss functions, squared-error loss, asymmetric loss, Bayesian point estimates, the Bayes action, The Price Is Right example, Kaggle Dark Worlds competition

The chapter defines a loss function L(θ, θ̂) as a way to score how costly an estimate θ̂ is given the true, unknown parameter θ, starting from squared-error loss and then showing an asymmetric loss that penalizes over- and under-estimates differently, which matches real business costs like server-capacity planning better than a symmetric penalty. Because the true parameter is itself a random variable with a posterior distribution in Bayesian inference, the chapter argues for minimizing the expected loss over that whole posterior rather than comparing a loss against a single guess. Two worked examples anchor the idea: optimizing a bid on The Price Is Right’s Showcase, and reconstructing the winning approach to Kaggle’s “Observing Dark Worlds” competition, where dark-matter halo positions were inferred from galaxy shape data using uniform and log-uniform priors.

Key Points:

• A loss function L(θ, θ̂) scores how costly an estimate θ̂ is given the true parameter θ; squared-error loss, (θ - θ̂)², is the common default.
• An asymmetric loss function penalizes overestimates and underestimates differently, which matters whenever the real cost of the two error directions is not equal.
• Because the true parameter is uncertain in Bayesian inference, you minimize the expected loss averaged over the whole posterior, not the loss against one sampled guess.
• The estimate that minimizes expected loss is called the Bayes action.
• The Dark Worlds example used uniform priors for each halo’s x/y position and a log-uniform prior for the dominant halo’s mass, reflecting the domain knowledge that larger halos dominate a sky’s gravitational-lensing signal.

Practice Tip: Plot expected-loss curves for both a symmetric and an asymmetric loss function on the same toy posterior, and watch the Bayes action shift as you change the penalty ratio — that shift is what makes the abstraction concrete.

Common Mistake: Defaulting to squared-error loss even when the real-world cost of over- and under-estimating is clearly unequal (under-provisioning servers usually costs more than over-provisioning them), instead of picking a loss function that reflects the actual decision.

Important Questions:

• What is a “Bayes action,” and how does it differ from simply reporting the posterior mean? The Bayes action is the estimate that minimizes expected loss under a chosen loss function; it equals the posterior mean only under squared-error loss, and differs from it whenever the loss function is asymmetric or otherwise non-quadratic.
• Why does the chapter insist on expected loss rather than the loss for one sampled value? Because the true parameter is unknown and represented by a whole posterior distribution, so a sound decision has to account for how costly an estimate is on average across every plausible true value, not just one draw.

Chapter 6: Getting Our Priorities Straight

Difficulty: Hard · Key topics: objective vs subjective priors, empirical Bayes, Gamma/Wishart/Beta distributions, multi-armed bandits, eliciting expert priors, conjugate priors, effect of sample size on the prior

The chapter separates objective priors, meant to let the data dominate (a flat/uniform prior is the classic example), from subjective priors, which deliberately encode a practitioner’s domain knowledge — and argues that neither is automatically safe, since a flat prior still embeds a choice about what range is plausible. Empirical Bayes is introduced as a hybrid trick that estimates a prior’s own hyperparameters from the observed data. The chapter then catalogs useful priors — Gamma, the Wishart distribution for covariance matrices, and the Beta distribution for probabilities and proportions — and uses a Beta prior to work the multi-armed bandit problem, where each pull updates belief about which option pays best. It closes by showing that a prior’s influence on the posterior shrinks mathematically as the sample size N grows.

Key Points:

• An objective prior (e.g. flat/uniform) tries to let the data dominate the inference; a subjective prior deliberately encodes the practitioner’s domain knowledge.
• Empirical Bayes estimates a prior’s hyperparameters from the observed data itself, blending frequentist and Bayesian technique — the chapter notes this makes it “not a true Bayesian method,” since the data is effectively used twice.
• The Beta distribution, defined on [0,1], is the standard prior for probabilities and proportions, and is the engine behind the chapter’s Bayesian multi-armed-bandit algorithm.
• The Wishart distribution is a prior over positive semi-definite matrices, used when the unknown itself is a covariance matrix.
• A conjugate prior (e.g. Beta with binomial data) produces a posterior in the same distribution family, which can avoid needing MCMC at all.
• As N grows, the likelihood increasingly dominates the posterior, so the exact choice of prior matters less — enough data corrects a badly chosen prior.

Memory Tip: “Objective ≠ assumption-free” — a flat prior still encodes a choice (that every value in its range is equally plausible), so treat the word “objective” as a claim to check, not a synonym for safe.

Common Mistake: Assuming a uniform prior over a value’s whole possible range is automatically a neutral, assumption-free choice, when restricting that range at all (e.g. Uniform(0.5, 1) instead of Uniform(0, 1)) already encodes prior knowledge.

Important Questions:

• What is the practical difference between an objective and a subjective prior, and when would you prefer one over the other? An objective prior aims to let the data speak with minimal bias, which suits scientific research where impartiality matters; a subjective prior deliberately encodes trustworthy domain knowledge, which speeds convergence and sharpens inference when that knowledge is reliable.
• Why does a Beta prior fit naturally with the multi-armed bandit problem? Each bandit’s win probability is a value between 0 and 1, exactly the interval the Beta distribution is defined on, and it updates in closed form after each win/loss observation.

Read Bayesian Methods for Hackers Online (Free)

This book has no official PDF — it is free Jupyter notebooks on its official GitHub repository. Click below to open the official source, where every chapter can be read online through nbviewer or downloaded by cloning the repository.

How to Study This Book

Read the Prologue first even though it gets no chapter block of its own here — it is two short paragraphs explaining why the author built a computation-first, math-second introduction to Bayesian inference, and it sets the tone for everything after it.

Chapters 1 to 3 are the spine of the book and should be read in order: Chapter 1 builds the Bayesian mindset and your first PyMC model, Chapter 2 teaches you to actually construct PyMC models with stochastic and deterministic variables, and Chapter 3 explains what the sampler is doing so the earlier “black box” stops feeling like magic. Do not skip Chapter 3 — students who treat MCMC as a button to press instead of an algorithm to understand are the ones who later mix up posterior samples or misread convergence problems.

Chapters 4 to 6 are more independent of each other and can be read in the order your course needs them. Chapter 4 (the Law of Large Numbers) is worth reading before Chapter 5 even if your course skips it, because Chapter 5’s whole idea of “expected loss” is really the Law of Large Numbers applied to a decision. Chapter 6 on priors is the most mathematically dense chapter and pairs naturally with a probability or statistics course covering conjugate distributions.

Open the right notebook file. Each chapter folder in the GitHub repository contains several versions: *_PyMC2.ipynb and *_PyMC3.ipynb are the original 2013–2016 notebooks and use APIs that modern PyMC has since removed — useful only as historical reading. *_PyMC_current.ipynb is the maintained version, written for current PyMC (pinned to pymc>=5.0.1 in the repo’s own requirements.txt), and is the one to actually run. *_Pyro.ipynb and *_TFP.ipynb are community ports of the same six chapters to Pyro and TensorFlow Probability, useful if your course uses one of those frameworks instead of PyMC.

Run the code, do not just read it. This book’s whole argument is that probabilistic programming makes Bayesian inference tractable by computation rather than by hand-derived integrals — that argument only lands if you actually run the PyMC models and watch the posterior samples come back.

The book assumes you can already write Python and are comfortable with NumPy, Matplotlib and basic probability (random variables, distributions, expected value). It is not a first course in either Python or probability; pair it with Think Python or a statistics/probability text if either is shaky before starting Chapter 1.

Used In These Programs

This book is used for Probabilistic Programming and Bayesian Statistics electives in: BS Computer Science. Browse all Python books, all Statistics books, or all Computer Science books.

Who Should Read This

Bayesian Methods for Hackers is written for a student or working programmer who already codes in Python and wants a computation-first route into Bayesian inference, rather than the two or three chapters of measure-theoretic probability that a traditional statistics textbook opens with. It suits a BSCS or BS Statistics student taking a Probabilistic Programming or applied Bayesian statistics elective, and it suits anyone preparing for a Kaggle-style competition where a Bayesian approach might help, since two of its six chapters are built directly from real competition data. It is not the right starting point if you have never programmed in Python or never met a probability distribution before — the author says plainly that the book assumes both, and a shaky foundation in either will make Chapter 1’s PyMC model harder than it needs to be. Pair it with a Python fundamentals text such as Think Python if your Python is still new, since none of the six chapters slow down to re-teach the language.

Applicable Universities

This book is useful for students at Pakistani universities offering BSCS and BS Statistics including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, and other HEC-recognized institutions.

FAQs

Related Books

• Think Python – Allen B. Downey (3rd Edition) — build the Python fluency this book assumes, if it is still shaky
• Problem Solving with Algorithms and Data Structures using Python
• Dive Into Python 3 – Mark Pilgrim

Bayesian Methods for Hackers earns its title: it gets a working PyMC model into your hands in Chapter 1 instead of making you wait through chapters of measure theory first, and its six chapters cover enough ground — Bayesian modelling, MCMC, the Law of Large Numbers, loss functions and priors — to carry a full Probabilistic Programming or applied Bayesian statistics elective. Open the current-PyMC notebook for each chapter, run the code as you read, and pair it with a Python fundamentals book first if your Python is still new. Browse more Computer Science books for the rest of your semester.

Bayesian Methods for Hackers (Probabilistic Programming and Bayesian Methods for Hackers) by Cameron Davidson-Pilon. Copyright 2013 Cameron Davidson-Pilon. Free under the MIT License. Official source: https://github.com/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers