Speech and Language Processing PDF Download – Jurafsky & Martin (3rd Edition Draft)

BSCS students can download “Speech and Language Processing” (3rd edition, draft) by Daniel Jurafsky and James H. Martin — the most widely used free Natural Language Processing textbook in the world, hosted directly by Stanford University and updated continuously by its authors as the field itself changes. The complete book is organized into three volumes the authors themselves label — Large Language Models, Advanced LLM Topics and Tools, and Annotating Linguistic Structure — and is free to download chapter by chapter or as a single combined PDF from the official site.

Read it knowing exactly what it is: an actively-revised draft, not a finished, published book. The authors post individual chapter PDFs as they rewrite them — the version referenced on this page is the release dated August 19, 2026, in which Chapter 1 was substantially rewritten and a new Chapter 10 on interpretability was added as an explicit work-in-progress stub. There is no Creative Commons licence attached to it; the authors retain ordinary copyright and simply say, in their own words, “feel free to use the draft chapters and slides in your classes, print it out, whatever.” That is the same honest, freely-hosted-but-not-openly-licensed pattern already used on this site for OSTEP and the Algorithms DPV notes. This page groups the book’s 26 numbered chapters (25 written, 1 not yet written) and 11 appendices into 15 topic blocks that follow the authors’ own three-volume structure, since the book itself is too large for one block per chapter to stay readable.

Book Overview

Chapter List

Introduction to Language Models (Chapter 1)

Difficulty: Easy · Key topics: language models, next-token prediction, distributional hypothesis, embeddings, pretraining, instruction tuning, scaling laws

Chapter 1 defines a large language model in the book’s own plain terms — a system that predicts the next word from context — and grounds this in the distributional hypothesis that meaning is learned from the statistical company words keep. It sketches four historical periods in NLP that led to today’s LLM era, then lays out the four-stage training pipeline: pretraining on raw text, instruction tuning to follow tasks, preference alignment for safety, and reinforcement learning for reasoning. It weighs real benefits (translation, accessibility, education) against real harms (bias, environmental cost, overreliance), and closes by showing that inference is the same conditional-prediction machinery running forward, which is what makes prompting and agents possible.

Key Points:

• A language model is, in the book’s own words, “a computational system that can predict the next word from previous words.”
• The distributional hypothesis: meaning is learned from co-occurrence patterns, not explicit rules.
• Text is processed as tokens, not words — some languages need 2–3 tokens per word.
• Model quality scales with three factors: parameter count, training data size, and compute (scaling laws).
• Training has four stages: pretraining, instruction tuning, preference alignment, and RL for reasoning.
• In-context learning happens purely inside a prompt and vanishes once the conversation ends — no weights are updated.
• The book explicitly warns against treating “learn” or “think” as literal claims, calling them “convenient shorthand.”

Memory Tip: Read every later chapter as an answer to one question — what exactly is being predicted, and from what context? Pretraining predicts the next token from raw text; instruction tuning predicts a response from an instruction; alignment predicts which of two responses a human would prefer. Same prediction machinery, different training data each time.

Common Mistake: Treating “the model understood my question” as a factual claim rather than shorthand for a statistical process — confusing the metaphor for the mechanism is the single most common misreading of the whole book.

Important Questions:

• How does the distributional hypothesis connect to what a language model learns during pretraining? By repeatedly predicting the next word across billions of tokens, the model implicitly acquires knowledge of language, concepts, and the world — ontological facts, semantic relationships, and more — purely from statistical patterns in word co-occurrence, without any explicit symbolic instruction.
• Why is instruction tuning necessary if pretraining already produces powerful language models? Pretraining alone teaches token prediction, not instruction-following — early models just generated text loosely related to a prompt instead of responding correctly to it. Instruction tuning trains on tens or hundreds of thousands of instruction-response pairs so the model generalises to following instructions it has never seen before.

Tokenization and N-gram Language Models (Chapters 2–3)

Difficulty: Medium · Key topics: byte-pair encoding, Unicode/UTF-8, regular expressions, edit distance, Markov assumption, maximum likelihood estimation, perplexity, smoothing

Chapter 2 asks what actually counts as a “word,” and shows the question is too slippery for tokenization to rely on: languages disagree on word boundaries, vocabularies grow without bound, and morphology varies enormously. Byte-Pair Encoding is presented as the data-driven fix, merging frequent character pairs into a fixed subword vocabulary, alongside Unicode/UTF-8 encoding, regular expressions, and edit distance for measuring string similarity. Chapter 3 then builds n-gram language models on those tokens: the Markov assumption limits context to the previous n−1 tokens, maximum likelihood estimation counts n-grams to get probabilities, and perplexity measures how well a model predicts held-out text. Because most n-grams are never seen in training, the chapter works through Laplace smoothing, interpolation, and backoff so unseen n-grams still get some probability.

Key Points:

• BPE trains by repeatedly merging the most frequent adjacent token pair, run only inside pre-segmented words.
• Subword tokenization solves the “unknown word problem” — any input can be built from known pieces, down to single characters if needed.
• The Markov assumption: an n-gram model only looks at the previous n−1 tokens, not the whole history.
• Maximum likelihood estimate: P(wn|wn-1) = C(wn-1 wn) / C(wn-1).
• Perplexity is the inverse probability of the test set, normalized by its length — lower is better.
• Laplace smoothing adds 1 to every count so unseen n-grams still get nonzero probability, at the cost of shifting a lot of mass away from frequent ones.
• Interpolation blends unigram, bigram, and trigram estimates with learned weights instead of committing to one n-gram order.

Practice Tip: Build a bigram model on a small corpus by hand once — count every pair, divide by the first word’s count, then generate five words by sampling from those probabilities. Watching it produce almost-plausible nonsense is the fastest way to understand what “the model has no idea what a sentence means” actually looks like.

Common Mistake: Assuming a higher n always gives a better model. With too little training data, 4-grams start reproducing training sentences verbatim (the chapter’s own Shakespeare example) instead of generalizing — more context is not free.

Important Questions:

• Why do modern systems use subword tokenization instead of whole-word tokenization? Word-based tokenization fails because the vocabulary grows without bound, many languages lack clear word boundaries (Chinese, Thai), and morphology makes “word” hard to define across languages. Subword tokenization represents any unseen word as a sequence of known pieces, falling back to individual characters if necessary.
• Why is perplexity usually computed and compared in log space? Multiplying many small probabilities together underflows to zero on real hardware. Storing and summing log probabilities avoids that numerical problem while preserving the same ranking between models — lower perplexity still means a better model.

Classification and Word Embeddings (Chapters 4–5)

Difficulty: Medium · Key topics: logistic regression, sigmoid, cross-entropy loss, gradient descent, precision/recall, word2vec, cosine similarity, bias in embeddings

Chapter 4 introduces logistic regression as the workhorse supervised classifier for tasks like sentiment analysis and spam detection: a weight vector and bias are learned from labeled data, the sigmoid function turns their weighted sum into a probability, and gradient descent minimizes cross-entropy loss to find good weights. It extends this to multi-class problems via softmax and insists that precision, recall and F1, not raw accuracy, are the right metrics for unbalanced classes like spam. Chapter 5 moves from labels to meaning, using the distributional hypothesis to build word embeddings — dense vectors that place words with similar contexts near each other in vector space, measured with cosine similarity. It contrasts simple count-based vectors with the skip-gram method behind word2vec, and closes by showing embeddings inherit, and can even amplify, the biases already present in their training text.

Key Points:

• σ(z) = 1/(1+e^-z) squashes the weighted sum w·x+b into a probability between 0 and 1.
• Cross-entropy loss penalizes a confident wrong prediction heavily, which is what gradient descent’s updates are working against.
• Accuracy is the wrong metric for spam-style unbalanced classes — a classifier that always says “not spam” can hit 99.99% accuracy and catch nothing.
• Word2vec’s skip-gram method treats co-occurring words in ordinary text as free supervision — no manual labeling required.
• Cosine similarity normalizes the dot product of two vectors, so it measures direction (meaning) rather than raw magnitude.
• Static embeddings assign one fixed vector per word type; contextual embeddings (Chapter 9) vary the vector by context instead.
• Embeddings can amplify stereotypes already in the training text — a widely used embedding set resolves “computer programmer” − “man” + “woman” to “homemaker.”

Practice Tip: Compute cosine similarity between a handful of downloaded word2vec or GloVe vectors yourself. Pairs like king/queen or Paris/France sit closer together than unrelated words — seeing that geometry work is more convincing than reading the formula.

Common Mistake: Treating logistic regression’s output as a hard yes/no decision by definition. It is a probability; 0.5 is just a conventional threshold, and shifting it is a legitimate design choice depending on whether false positives or false negatives cost more.

Important Questions:

• Why is accuracy a poor metric for spam detection, and what should be used instead? Accuracy doesn’t work when classes are unbalanced — labeling everything “not spam” scores 99.99% accuracy on a dataset that is 0.01% spam while catching none of it. Precision and recall measure, respectively, how many flagged emails are actually spam and how much real spam was caught.
• Why do word embeddings pick up gender and other social biases? Embeddings learn purely from the statistical distribution of words in their training corpus, and that corpus already contains real-world stereotypes. The chapter’s own example shows embeddings can amplify bias beyond what appears in the raw input statistics, not just reproduce it.

Neural Networks and Transformers (Chapters 6–7)

Difficulty: Hard · Key topics: perceptron, XOR problem, activation functions, backpropagation, self-attention, multi-head attention, positional embeddings, decoding strategies

Chapter 6 explains why single-layer perceptrons cannot solve problems like XOR, and how stacking layers with nonlinear activation functions (sigmoid, tanh, ReLU) lets a network learn representations no single layer could. It walks through feedforward classification, cross-entropy loss for multiple classes, and backpropagation — computing gradients by the chain rule across a computation graph. Chapter 7 builds the transformer on top of this: self-attention lets every token look at every other token, computing relevance with query, key and value projections; multi-head attention runs several of these in parallel to capture different relationships; and positional embeddings let the otherwise order-blind attention mechanism know where each token sits. It finishes with how transformers are pretrained by next-token prediction and how decoding methods — greedy, sampling, temperature, top-k and top-p — turn a probability distribution into generated text.

Key Points:

• A single perceptron is a linear classifier and provably cannot solve XOR; a two-layer network with a nonlinear hidden layer can.
• Without a nonlinear activation function, stacking layers collapses mathematically into one linear layer — depth needs nonlinearity to matter.
• ReLU(z) = max(z,0) avoids the vanishing-gradient problem that saturates sigmoid and tanh in deep networks.
• Self-attention computes a weighted sum of context representations using scaled dot-product similarity between query and key vectors, scaled by √dk for numerical stability.
• An attention mask sets future positions to −∞ before softmax so a causal (left-to-right) language model cannot see ahead.
• Multi-head attention runs several attention computations in parallel with separate learned projections, then concatenates the results.
• Temperature sampling divides logits by τ before softmax, trading off greedy decoding’s repetitiveness against pure random sampling’s incoherence.

Memory Tip: Keep the residual stream picture in mind — every layer’s attention and feedforward block only adds something to a token’s running representation, it never replaces it. That is why a transformer can be very deep without losing the original token information.

Common Mistake: Assuming attention “looks ahead” at future tokens during generation. The causal mask explicitly zeroes out any attention to later positions before the softmax — without it, next-token prediction would be trivial to cheat and the model could never generate coherently left to right.

Important Questions:

• Why can’t a single perceptron compute XOR, but a two-layer network can? XOR is not linearly separable — no single line separates its positive and negative cases. A hidden layer can learn an intermediate representation that transforms the input into a space where the classes become linearly separable, which the output layer can then split with a line.
• Why does the transformer divide attention scores by √dk before applying softmax? Raw dot products between query and key vectors can grow arbitrarily large as dimensionality increases, which pushes softmax into a region with vanishing gradients and numerical instability. Dividing by √dk keeps scores in a stable range for training.

Post-training: Aligning Large Language Models (Chapter 8)

Difficulty: Hard · Key topics: instruction tuning, LoRA, reward models, Bradley-Terry model, DPO, PPO, KL penalty, RLVR

This chapter covers the three stages that turn a raw pretrained model into a helpful, safer assistant. Instruction tuning (supervised fine-tuning) retrains the model on curated instruction-response pairs using the exact same next-word-prediction objective as pretraining — only the data changes — often through parameter-efficient methods like LoRA that freeze the original weights and learn small low-rank update matrices instead. Preference-based alignment then goes further: a reward model is trained on human or machine judgments between two candidate responses using the Bradley-Terry framework, and the language model is optimized against that reward, either through reinforcement learning (PPO) or the more direct DPO method, with a KL-divergence penalty stopping it from drifting too far from its pretrained knowledge. A newer third stage, reinforcement learning with verifiable rewards, trains directly on tasks like math and code where correctness can be checked automatically.

Key Points:

• Instruction tuning (SFT) uses the identical next-word-prediction loss as pretraining — only the training data (instructions plus correct responses) changes.
• LoRA freezes the base weight matrix W and learns two small low-rank matrices instead, so h = xW + xAB.
• A reward model scores outputs using the Bradley-Terry model: P(oi ≻ oj|x) = σ(zi − zj).
• Direct Preference Optimization (DPO) optimizes the policy straight from preference data, without training a separate reward model or sampling online.
• A KL-divergence penalty keeps the aligned model from “forgetting everything it learned during pretraining.”
• Reinforcement Learning with Verifiable Rewards (RLVR) trains on tasks with an automatically checkable right answer, such as programming and math.

Memory Tip: Line the three stages up as three different training sets on the same loss function: pretraining = raw web text, instruction tuning = instruction-and-answer pairs, preference alignment = pairs of answers ranked against each other. The objective barely changes; what changes each time is what counts as “correct.”

Common Mistake: Thinking instruction tuning changes the model’s architecture or objective. It doesn’t — it is ordinary supervised fine-tuning with the same cross-entropy loss, just on a different, smaller, curated dataset.

Important Questions:

• Why is preference-based alignment still needed after instruction tuning, and what does the KL-divergence term protect against? Instruction tuning alone leaves room for hallucinations and unsafe outputs. The KL term penalizes the model for drifting too far from its pretrained starting point during preference optimization, preventing it from forgetting what it learned during pretraining while it is being pushed toward preferred outputs.
• How does LoRA reduce the computational cost of fine-tuning compared to updating the full weight matrix? LoRA freezes the large weight matrix W and instead learns two much smaller low-rank matrices A and B, so gradients only need to be computed for those small matrices, not the whole model, and the learned update can simply be added to the pretrained weights afterward without slowing down inference.

Masked Language Models and Interpretability (Chapters 9–10)

Difficulty: Hard · Key topics: BERT, bidirectional attention, fine-tuning, contextual embeddings, probing, logit lens, induction heads

Chapter 9 introduces bidirectional transformer encoders like BERT, trained with masked language modeling: a portion of input tokens are hidden, replaced, or left alone, and the model predicts the original token using context from both directions at once, something a causal decoder-only model cannot do. It shows how a small classifier head added on top of the encoder adapts BERT to classification and to sequence labeling tasks like named entity recognition, using the special [CLS] token for sentence-level output. Chapter 10, explicitly marked in the source as a work-in-progress stub, opens the question of interpretability: how contextual embeddings represent different senses of the same word, how probing classifiers reveal what a layer encodes, and how mechanisms like induction heads have been identified as the circuits behind a model’s ability to learn from examples inside a single prompt.

Key Points:

• BERT masks tokens three ways during training — mostly [MASK], sometimes a random token, sometimes left unchanged — so the model doesn’t overfit to a mask token that never appears at inference time.
• “Encoder models are not generative models; they aren’t used to generate text,” in the book’s own words — MLMs produce context vectors, not autoregressive samples.
• The [CLS] token’s final-layer vector is used as the whole sequence’s representation for classification tasks.
• Contextual embeddings give the same word a different vector in every sentence, unlike the one-vector-per-word static embeddings of Chapter 5.
• Probing trains a small classifier on frozen embeddings to see what a layer encodes — but a successful probe shows correlation, not proof the model actually uses that information.
• Induction heads perform pattern completion (spotting “A…B…A” and predicting “B” again), and disabling them measurably hurts in-context learning.
• Chapter 10 is explicitly incomplete in this draft — its sections on sparse auto-encoders and causal interpretability techniques are headers only, with no content written yet.

Memory Tip: Whenever a chapter asks “does the model know X,” separate two different claims — a probe finding X encoded in the activations, versus an ablation experiment showing removing that circuit changes the output. Only the second one is real evidence the model uses it.

Common Mistake: Expecting a masked/bidirectional model like BERT to generate fluent continuations the way a decoder-only LLM does. It was never trained to predict left-to-right, only to fill in masked positions using context from both sides.

Important Questions:

• Why does BERT use three different masking strategies instead of always inserting [MASK]? Replacing tokens with [MASK] would create a mismatch between pretraining and later use, since [MASK] tokens never appear during fine-tuning or real inference. Mixing in random tokens and unchanged tokens trains the model to predict correctly regardless of whether a mask is actually present.
• What is anisotropy in contextual embeddings, and why does standardizing them help? Anisotropy is the tendency of all embedding vectors to point in roughly the same direction, which makes cosine similarity between unrelated words unrealistically high. Standardizing (z-scoring) each dimension makes the space more isotropic, so cosine similarity better reflects real semantic closeness.

Information Retrieval and Retrieval-Augmented Generation (Chapter 11)

Difficulty: Medium · Key topics: hallucination, TF-IDF, inverted index, dense retrieval, RAG, bi-encoders

This chapter tackles a core weakness of large language models: they hallucinate, generating fluent but factually wrong answers instead of admitting they don’t know. It presents information retrieval as the fix, starting from classical sparse methods — TF-IDF weighting and the inverted index — that rank documents by exact word overlap with a query, then moves to dense retrieval, which represents queries and documents as neural embeddings so retrieval can match meaning rather than exact wording. Retrieval-Augmented Generation (RAG) puts these together: retrieve documents likely to answer the question, then generate the answer conditioned on them, rather than relying purely on what the model memorized during pretraining. The chapter closes by covering evaluation metrics for both retrieval quality and the question-answering systems built on top of it.

Key Points:

• Hallucination is defined plainly as “a response that is not faithful to the facts of the world.”
• Classical retrieval (tf-idf, BM25) only works with exact word overlap between query and document — the “vocabulary mismatch problem.”
• Dense retrieval embeds both query and document with a neural model and scores similarity by dot product, catching synonyms that word overlap would miss.
• RAG, in the book’s own words: “use information retrieval techniques to retrieve documents likely to have useful information, then use an LLM to generate an answer conditioned on these documents.”
• A full encoder that jointly processes query and document is more accurate but must re-encode every document per query — too slow for real use; bi-encoders pre-compute document embeddings instead, trading some accuracy for speed.
• Irrelevant or wrong retrieved passages can actively hurt an LLM’s answer instead of helping it.

Practice Tip: Try the failure mode yourself — ask an LLM a very specific, obscure factual question with no retrieval, then ask again after pasting in one correct paragraph from a source. The jump in answer quality is the whole argument for RAG in one demo.

Common Mistake: Assuming more retrieved documents is always better. Retrieval noise — irrelevant or wrong passages mixed in with the useful ones — can drag answer quality down, which is why reranking and better retrieval, not just retrieving more, is the real lever.

Important Questions:

• Why do classical tf-idf and BM25 retrievers struggle with vocabulary mismatch, and how do dense retrievers fix it? Classical retrieval only works when the query and document share exact words, so users must guess the author’s terminology. Dense retrievers embed both into the same vector space, so they can match synonyms and paraphrases based on meaning rather than exact overlap.
• What is the accuracy/speed trade-off between a full joint encoder and a bi-encoder in dense retrieval? A full encoder can process the query and document together with cross-attention for more accurate relevance judgments, but it must be re-run for every document on every query, which is impractical at scale. A bi-encoder pre-computes document embeddings once and compares them to a query embedding with a fast dot product, at some cost to accuracy.

Machine Translation and Recurrent Networks (Chapters 13–14)

Difficulty: Hard · Key topics: encoder-decoder, cross-attention, beam search, backtranslation, RNN, vanishing gradients, LSTM gates

Chapter 13 examines machine translation both as an application and as evidence of how languages differ systematically — in word order, vocabulary, and how much they leave pronouns unspoken (referential density) — before presenting the encoder-decoder transformer with cross-attention as the standard architecture: the decoder attends over every encoder state while generating the target sentence. It covers building training data from parallel corpora, beam search decoding with length normalization, handling low-resource languages via backtranslation and multilingual models, and evaluation with the chrF metric and human raters, plus the real problem of gender bias in pronoun translation. Chapter 14 backs up to explain recurrent neural networks, which process language step by step through a hidden state that in principle carries information across an entire sequence, and shows why simple RNNs struggle with vanishing gradients over long sequences — motivating the LSTM’s gated architecture, which updates memory by addition rather than repeated multiplication, and the attention mechanism that fixes the single-vector bottleneck in RNN encoder-decoder translation.

Key Points:

• Referential density: pro-drop languages like Chinese and Japanese permit or require dropping pronouns that English needs made explicit — a genuine inference problem at translation time, not just vocabulary lookup.
• Cross-attention lets the decoder attend to every encoder hidden state, not just a single final summary vector.
• Beam search needs length normalization, because raw sequence probability keeps shrinking with every extra token and would otherwise bias the search toward shorter, truncated translations.
• Backtranslation creates synthetic parallel data for low-resource languages by translating monolingual target-language text back into the source language with a reverse-direction system.
• An RNN’s hidden state at time t depends on its hidden state at t-1, letting information flow forward — but repeated multiplication during backpropagation through time causes gradients to vanish over long sequences.
• The LSTM’s forget gate updates its context via addition rather than multiplication, which is exactly what lets gradients flow back further without vanishing.

Memory Tip: Gate names describe themselves if read as verbs — the forget gate decides what to drop from memory, the input gate decides what new information to add, the output gate decides what to expose next. All three are separate small networks watching the same input.

Common Mistake: Assuming greedy decoding — always picking the single best next word — gives the best full translation. A locally optimal word choice can lead to a globally worse sentence, which is exactly why beam search keeps multiple candidate sequences alive instead of committing early.

Important Questions:

• Why is length normalization necessary in beam search decoding for machine translation? Language models assign lower total probability to longer sequences by construction, so an un-normalized beam search would systematically prefer shorter, truncated translations over longer, more complete and more accurate ones. Dividing by sequence length corrects this bias.
• How do LSTM gates solve the vanishing-gradient problem that limits simple RNNs? Simple RNNs repeatedly multiply gradients by the same weight matrix during backpropagation through time, causing them to vanish over long sequences. The LSTM’s forget, input and output gates instead update the context via addition, letting gradients flow backward across many more time steps without shrinking to zero.

Speech Processing: Phonetics, Recognition, and Synthesis (Chapters 15–17)

Difficulty: Hard · Key topics: phones, ARPAbet, formants, MFCCs, CTC, Word Error Rate, audio codecs, residual vector quantization

Chapter 15 lays the acoustic groundwork: articulatory phonetics describes how the vocal tract produces distinct phones, ARPAbet gives them computable symbols, and acoustic phonetics explains formants and spectrograms — visualizations of frequency over time — before showing the signal-processing pipeline (sampling, windowing, Fourier analysis, mel filtering) that turns a raw waveform into MFCC feature vectors. Chapter 16 builds automatic speech recognition on those features, comparing two dominant paradigms: encoder-decoder models with cross-attention (like Whisper) and CTC-based self-supervised models (like wav2vec2.0 and HuBERT) that output one symbol per audio frame and collapse repeats, evaluated with Word Error Rate. Chapter 17 closes the loop with text-to-speech, where modern systems like VALL-E first compress speech into discrete audio tokens with a neural codec using residual vector quantization, then use a two-stage language model to generate new speech, including zero-shot voice cloning from just seconds of sample audio.

Key Points:

• English orthography is a poor guide to pronunciation — the letter ‘c’ alone maps to different phones in “cougar” and “cell” — which is why ARPAbet/IPA symbols, not letters, are used for speech processing.
• Formants (F1, F2) are the frequency bands the vocal tract amplifies for a given vowel shape, and they are what a spectrogram makes visible.
• The mel scale spaces frequencies the way human hearing does — finer resolution below 1000 Hz — which is why MFCCs use mel filtering rather than raw linear frequency.
• Word Error Rate = (Insertions + Substitutions + Deletions) / Reference Words × 100 is the standard ASR evaluation metric.
• CTC outputs one label per audio frame and collapses repeats and blanks into the final transcript, enabling true streaming recognition, unlike attention-based encoder-decoder models.
• Modern TTS systems like VALL-E treat speech synthesis as language modeling over discrete audio tokens produced by a neural codec, not direct waveform generation.
• Residual vector quantization encodes each audio frame as several hierarchical tokens instead of one, capturing both coarse and fine acoustic detail.

Memory Tip: Trace the audio pipeline as one chain: waveform → windowed frames → Fourier spectrum → mel filter bank → MFCCs (recognition) or codec tokens (synthesis). All three chapters are variations on turning sound into discrete, model-friendly units, or turning those units back into sound.

Common Mistake: Thinking CTC and encoder-decoder ASR models are interchangeable. CTC’s frame-independence assumption makes it fast and streamable but weaker at modeling output history, while encoder-decoder attention models implicitly learn a language model and are usually more accurate but not naturally streaming — the right choice depends on the application.

Important Questions:

• Why are convolutional layers with stride greater than 1 essential in modern speech-recognition encoders? Acoustic frames occur roughly every 10ms while words last around 250ms, so the raw acoustic sequence is far longer than the text it corresponds to. Strided convolutions compress the sequence length before it reaches the attention or CTC layers, making training and inference tractable.
• Why does VALL-E use residual vector quantization instead of a single quantizer to represent speech? A single quantizer would lose acoustic detail. Residual vector quantization produces several hierarchical tokens per frame — the first capturing the primary content and later ones encoding successive residual detail — which mirrors VALL-E’s two-stage generation, where the most important codes are generated first and refined afterward.

Sequence Labeling: POS Tagging and Named Entity Recognition (Chapter 18)

Difficulty: Medium · Key topics: Hidden Markov Model, Viterbi algorithm, Conditional Random Field, BIO tagging, gazetteer

This chapter treats part-of-speech tagging and named entity recognition as instances of the same problem: assigning one label to every word in a sequence. It shows that although only 14–15% of word types are genuinely ambiguous, those words are common enough to make up more than half of running text, so real disambiguation is unavoidable. Two classical algorithms are worked through in full: the generative Hidden Markov Model, decoded with the Viterbi dynamic-programming algorithm to find the single most probable tag sequence, and the discriminative Conditional Random Field, which scores sequences directly using arbitrary local features instead of separate transition and emission probabilities. For named entities, the chapter shows how BIO tagging converts a span-recognition problem into the same word-by-word labeling framework used for POS tags.

Key Points:

• Most word types (85–86%) are unambiguous; the small ambiguous minority still accounts for 55–67% of tokens in running text.
• An HMM is defined by transition probabilities A, emission probabilities B, and an initial distribution — Viterbi finds the single best tag sequence via dynamic programming over this model.
• HMM emission probability P(word|tag) asks a backwards question — if generating this tag, how likely is this word — not “which tag fits this word,” which is easy to get turned around.
• BIO tagging (Begin/Inside/Outside) turns named-entity span detection into ordinary per-word sequence labeling.
• A CRF is discriminative: it models P(tags|words) directly with arbitrary feature functions (prefixes, suffixes, word shape), rather than needing separate probability tables like an HMM.
• A gazetteer — a reference list such as known place names — is a simple but effective binary feature for NER.
• Human tagging agreement caps out around 97%; the book attributes much of the remaining gap to inconsistent training data, not missing algorithms.

Memory Tip: Say the HMM emission question out loud exactly as the book frames it — “if I were going to generate this tag, how likely is this particular word?” That backwards framing is what actually makes the Viterbi math work, and it is the part most students get turned around on first.

Common Mistake: Reading HMM emission probability P(word|tag) as “the probability this word has this tag” — it is the reverse conditional, and mixing the two up breaks the reasoning the whole tagger depends on.

Important Questions:

• Why is the most-frequent-tag baseline (around 92% accuracy) so close to human agreement (around 97%)? Most word types are unambiguous, so strong statistical patterns already solve most cases. The book attributes a large share of the remaining gap to errors and inconsistencies in the training data itself, not to missing algorithmic sophistication.
• How does a CRF incorporate features like word shape that an HMM struggles to use directly? A CRF is a log-linear model that can append arbitrary local feature functions — prefixes, suffixes, capitalization patterns — directly into its scoring, while an HMM would need to awkwardly fold such features into one of its two fixed probability tables.

Syntactic Parsing: Constituency and Dependency (Chapters 19–20)

Difficulty: Hard · Key topics: context-free grammar, Chomsky Normal Form, CKY algorithm, PARSEVAL, dependency relations, transition-based parsing, graph-based parsing

Chapter 19 formalizes phrase structure with context-free grammars, where every non-terminal expands via production rules, and shows that once a grammar is converted to Chomsky Normal Form, the CKY dynamic-programming algorithm can find every possible parse for a sentence in polynomial time — though CKY by itself only recognizes and represents all parses, it does not decide which one is correct. Disambiguating structurally ambiguous sentences is handled by scoring individual spans with a neural classifier and combining scores recursively to select the single best full parse. Chapter 20 presents dependency grammar as an alternative that directly links each word to its syntactic head rather than building phrase brackets, which handles free word order more gracefully, and covers two parsing strategies: fast, greedy transition-based parsing using a stack and buffer, and slower but more accurate graph-based parsing that scores every possible edge and finds the maximum spanning tree.

Key Points:

• A context-free grammar in Chomsky Normal Form only allows rules of the shape A → BC or A → a, which is exactly what lets CKY split every span at one binary point.
• CKY fills an upper-triangular table bottom-up and, by itself, “returns an efficient representation of the set of parse trees for a sentence, but doesn’t tell us which parse tree is the right one” — disambiguation is a separate step.
• PARSEVAL scores a parser against a gold treebank using labeled precision, recall and F1 over bracketed spans.
• A dependency tree gives every word exactly one head (except the root), and typed relations like NSUBJ or OBJ encode grammatical function directly on the arc.
• Transition-based dependency parsing is a greedy, linear-time algorithm using shift/left-arc/right-arc operations on a stack and buffer.
• Graph-based parsing scores every possible edge and finds the maximum spanning tree, and is more accurate than transition-based parsing on long sentences precisely because it isn’t greedy.
• Evaluation for dependency parsers uses UAS (unlabeled attachment score) and LAS (labeled attachment score).

Memory Tip: Constituency answers “what phrase is this part of”; dependency answers “which word is this word’s boss.” A language with flexible word order keeps its dependency links intact far better than its phrase brackets — that is the whole reason both formalisms still coexist in the field.

Common Mistake: Believing CKY alone picks the correct parse. It only proves a parse exists and compactly stores every possible one — choosing the single best parse needs an extra scoring step, whether a probabilistic grammar or a neural span classifier.

Important Questions:

• Why must a context-free grammar be converted to Chomsky Normal Form before running CKY? CKY relies on splitting every span at exactly one point, which requires each non-terminal above the part-of-speech level to have exactly two daughters. CNF guarantees this binary-branching property, which the original grammar may not have.
• Why are graph-based dependency parsers generally more accurate than transition-based parsers on long sentences? Transition-based parsers make greedy, local decisions one step at a time and can propagate an early mistake through the rest of the parse, especially when a head is far from its dependent. Graph-based parsers score the whole set of possible edges at once and find the globally best tree, avoiding that greedy error propagation.

Information Extraction and Semantic Role Labeling (Chapters 21–22)

Difficulty: Medium · Key topics: relation extraction, distant supervision, temporal normalization, template filling, thematic roles, PropBank, FrameNet, selectional restrictions

Chapter 21 covers pulling structured facts out of unstructured text through three related tasks: relation extraction, handled with hand-written patterns, supervised classifiers, bootstrapping from seed patterns, or distant supervision from large databases; event extraction and temporal normalization, which map expressions like “next Tuesday” onto standard calendar dates and order events relative to each other; and template filling, which populates predefined slots from a document. Chapter 22 turns to semantic roles — who did what to whom — showing that the same event can surface in many different syntactic forms (diathesis alternations), which is why a meaning-level representation is needed above syntax. It presents two competing frameworks, PropBank’s verb-specific numbered arguments and FrameNet’s frame-based roles shared across related words, plus selectional restrictions as soft rather than hard constraints on what fills a given role.

Key Points:

• Distant supervision generates large but noisy training data automatically by aligning sentences with facts already in a database, trading label quality for scale.
• Temporal normalization maps expressions like “next Tuesday” to a standard calendar date using the document’s own timestamp as a reference point.
• Not every event is expressed by a verb — nominalizations like “the increase” and light verbs like “took effect” complicate the assumption that “verb equals event.”
• Diathesis alternations (e.g. “The rock broke the window” vs. “The window broke”) show that semantic roles like Agent and Theme do not map consistently onto grammatical subject and object.
• PropBank labels arguments per verb sense as Arg0 (proto-agent), Arg1 (proto-patient), and so on; FrameNet instead groups semantically related words into shared frames with frame-specific roles.
• Selectional restrictions (“eat” wants an edible theme) are represented as soft preferences, not hard rules, because real sentences violate them constantly through negation or figurative use.

Memory Tip: Whenever role-labeling feels arbitrary, replace the sentence with its diathesis alternation and check whether the semantic role moved with the noun phrase or stayed with the grammatical slot. If the role follows the noun phrase across “X broke Y” and “Y broke,” that is the whole argument for why syntax alone can’t do this job.

Common Mistake: Assuming grammatical subject always equals Agent. The break example shows the subject position can hold an Instrument, a Theme, or an Agent depending on which alternation is used — semantic role is independent of syntactic position.

Important Questions:

• What is the trade-off between distant supervision and fully supervised learning for relation extraction? Distant supervision uses existing databases to generate large amounts of training data cheaply, avoiding expensive hand labeling, but tends to produce lower-precision results and only works for relations that already exist in a database. Supervised learning needs expensive labeled data but produces more reliable results for the relations it is trained on.
• Why are selectional restrictions better modeled as soft preferences than hard constraints? Real sentences violate selectional restrictions constantly, through negation (“you can’t eat gold for lunch”) or figurative language (“ate glass”). Treating them as hard constraints would incorrectly reject perfectly grammatical sentences, so modern systems score the fit between predicate and argument as a soft preference instead.

Lexical Semantics, Coreference, and Discourse (Chapters 23–25)

Difficulty: Medium · Key topics: valence/arousal/dominance, connotation frames, coreference chains, mention-ranking, entity linking, Rhetorical Structure Theory, Centering Theory

Chapter 23 moves from single-word sentiment features to full affective lexicons, covering theoretical models of emotion, how lexicons are built by crowdsourcing or bootstrapped from seed words, and connotation frames, which capture how a predicate frames its arguments rather than just a word’s polarity. Chapter 24 tackles coreference resolution — deciding which mentions in a text refer to the same real-world entity — surveying mention-pair, mention-ranking and entity-based architectures, entity linking to knowledge bases like Wikipedia, and documented failure cases including Winograd schema problems and gender bias in pronoun resolution. Chapter 25 closes with discourse coherence: how sentences connect into a coherent whole through explicit coherence relations (Rhetorical Structure Theory, the Penn Discourse TreeBank), entity-based coherence tracked by Centering Theory and the entity grid, and global structure in genres like persuasive essays and scientific papers.

Key Points:

• The dimensional model of emotion scores a word on three axes — valence (pleasantness), arousal (energy), and dominance (control) — rather than assigning it to one of a fixed set of basic emotions.
• Connotation frames encode how a predicate portrays its arguments (e.g. “violate” frames its object as a sympathetic victim), which is richer than a single sentiment score for a whole sentence.
• 60–70% of noun phrases in the OntoNotes coreference corpus are singletons — mentioned once and never linked again — which is part of why coreference is harder than it first looks.
• A mention-ranking model scores every earlier mention (plus a null option) for a candidate anaphor, fixing the mention-pair model’s inability to compare candidate antecedents against each other directly.
• Entity linking (“wikification”) maps a resolved mention to an actual knowledge-base entry, one step beyond just clustering mentions together.
• Rhetorical Structure Theory builds a full hierarchical tree of nucleus-satellite relations across a document; the Penn Discourse TreeBank instead only annotates pairwise relations, with no larger tree.
• Centering Theory tracks a “backward-looking center” (the current discourse focus) and “forward-looking centers” (candidates for the next sentence’s focus) to explain why some sentence sequences read as coherent and others don’t.

Memory Tip: For coreference, don’t think “find matching pronouns” — think “build the discourse model a human reader is silently maintaining.” Every architecture in this chapter, from mention-pair to full neural end-to-end systems, is approximating that same mental model.

Common Mistake: Assuming coreference resolution is basically solved by exact string matching or simple pronoun rules. The book states plainly that correctly detecting which expressions are even referential “seems to still be an unsolved problem,” and systems still fail noticeably on non-referential pronouns and Winograd-schema-style cases that need real-world reasoning.

Important Questions:

• Why does mention-ranking outperform the simpler mention-pair architecture for coreference resolution? The mention-pair classifier judges each candidate antecedent independently and is never trained to compare candidates against each other, and it ignores the wider discourse model. Mention-ranking scores all candidate antecedents together for a given mention, letting the model directly choose the best one relative to the alternatives.
• What is the practical difference between Rhetorical Structure Theory and Penn Discourse TreeBank annotation? RST builds a complete hierarchical tree spanning an entire document, organizing every span into nucleus-satellite relations. PDTB only annotates pairwise relations between adjacent or nearby text spans, without building any larger tree structure over the whole document.

Conversation and Dialogue Systems (Chapter 26)

Difficulty: Easy · Key topics: turns, speech acts, grounding, adjacency pairs, dialogue acts, implicature, initiative

This chapter identifies the structural properties that make human conversation work, as a foundation for building systems that can hold one: turns (who is speaking and when), speech acts (what an utterance is doing — asserting, directing, promising, acknowledging), grounding (actively confirming mutual understanding, not just passively hearing), adjacency pairs (a question sets up an expected answer), initiative (who is steering the conversation), and implicature (inferences a listener is expected to draw beyond the literal words). It then shows how these properties are annotated computationally as dialogue acts, using tagsets like DAMSL, to analyze both human-human conversation and, increasingly, human-LLM conversation.

Key Points:

• A speech act falls into one of four classes: constatives (asserting), directives (requesting), commissives (promising), and acknowledgments.
• Grounding is defined as the hearer actively confirming understanding to the speaker, not just silently comprehending — which is why backchannels like “uh-huh” matter.
• Adjacency pairs are two-part structures (question/answer, offer/acceptance) where the first part creates a strong expectation for the second.
• Implicature lets a listener infer more than what was literally said, guided by conversational maxims such as relevance.
• Initiative ranges from fully system-controlled dialogue to mixed initiative, which is how natural human conversation actually behaves.
• Dialogue acts combine speech-act type and grounding function into a single tag used to annotate real conversation corpora.

Memory Tip: Whenever a dialogue example seems not to directly answer the question asked, look for the implicature instead of assuming the system failed — a working system has to reason it out via the maxim of relevance, the same way a human listener would.

Common Mistake: Treating grounding as just “the listener understood.” The book’s definition is stronger — grounding requires the hearer to actively signal that understanding back to the speaker, which is why acknowledgments and continuers exist as a distinct category from silent comprehension.

Important Questions:

• In the book’s travel-agent example, why doesn’t the client’s line about a meeting directly answer the question about a travel date, and how does the system still infer the answer? The client states a meeting window rather than a travel date. The system uses conversational implicature — the maxim of relevance — to infer that mentioning the meeting must be relevant to the travel plan, and concludes the client likely wants to arrive before the meeting starts.
• What is the difference between “continued attention” and “acknowledgment” as grounding strategies? Continued attention is the weakest form of grounding, where the hearer simply shows they are listening without an explicit response. Acknowledgment is stronger and involves an explicit signal, such as saying “OK” or a continuer like “uh-huh,” actively confirming understanding rather than passively attending.

Appendices A–K: Reference Algorithms and Resources

Appendix · Difficulty: Hard · Key topics: Hidden Markov Models, Naive Bayes, Kneser-Ney smoothing, spelling correction, statistical parsing, Combinatory Categorial Grammar, WordNet, PPMI

Beyond the 26 numbered chapters, the book carries eleven lettered appendices (A through K) that hold the mathematical and algorithmic detail the main chapters assume or compress: Hidden Markov Models with the forward, Viterbi and Baum-Welch algorithms worked through in full; Naive Bayes classification; Kneser-Ney smoothing; spelling correction; statistical and grammar-based constituency parsing; Combinatory Categorial Grammar; logical meaning representations; word senses and WordNet; pointwise mutual information; and frame-based dialogue systems. These are reference material rather than a linear reading path — students go to the specific appendix a chapter cites, not through all eleven in order.

Key Points:

• Appendix A works through all three classic HMM algorithms: the forward algorithm for sequence probability, Viterbi for the best hidden-state path, and Baum-Welch (forward-backward) for unsupervised parameter learning.
• The Markov assumption is stated formally in Appendix A as P(qi = a | q1...qi-1) = P(qi = a | qi-1) — only the immediately preceding state matters.
• Appendix C’s Kneser-Ney smoothing is the specific technique Chapter 3 gestures at as the strongest classical n-gram smoothing method, without deriving it in full there.
• Appendix I extends Chapter 5’s embeddings with WordNet’s hand-built sense hierarchy — a useful contrast between symbolic and learned meaning representations.
• These appendices are cited from specific chapters rather than meant to be read start to finish; use them the way the main text uses them, as lookup references.

Memory Tip: Don’t read the appendices as a 27th chapter. Jump to Appendix A only once Chapter 18’s HMM tagging needs the Viterbi derivation spelled out, to Appendix I once Chapter 5 or 10’s word-sense discussion needs WordNet detail, and so on — that is how the book itself points to them.

Common Mistake: Skipping the appendices entirely because they aren’t numbered chapters. Several — especially A (HMMs) and C (smoothing) — contain the actual algorithm derivations that the numbered chapters assume the reader already knows.

Important Questions:

• What are the three classic problems an HMM solves, and which algorithm solves each? Computing the probability of an observation sequence is solved by the forward algorithm; finding the single most likely hidden-state sequence is solved by the Viterbi algorithm; and learning the model’s own parameters from unlabeled data is solved by the Baum-Welch (forward-backward) algorithm.
• Why is the Markov assumption useful for sequence models even though it is obviously false about real language? Limiting the model to only the immediately previous state (or previous n−1 tokens) makes probability estimation tractable with a manageable number of parameters, and in practice captures enough local structure to work well, even though real language clearly depends on much longer-range context than that.

Download Speech and Language Processing PDF (Free)

This book is free from its official source and is not hosted on this site, since it is revised too frequently for a mirrored copy to stay current. Click below to open Stanford’s official chapter index, where every chapter PDF, the combined full-book PDF, and the lecture slides can be downloaded directly.

How to Study This Book

Read Volume I (Chapters 1–8) in order — it is a genuine sequence, not a set of independent topics. Chapter 1 sets vocabulary you need for the rest of the book (tokens, embeddings, pretraining), and Chapter 2’s tokenization plus Chapter 3’s n-gram models are assumed knowledge for every later probability-based algorithm in the book, including the HMM in Chapter 18 and Appendix A.

Chapters 4–5 (classification, embeddings) and Chapters 6–7 (neural networks, transformers) build directly on each other. Do not skip Chapter 6’s backpropagation explanation to jump straight to transformers in Chapter 7 — Chapter 7 assumes you already understand gradient-based training.

Chapter 8 (post-training) only makes sense once Chapter 7’s pretraining is clear — it is literally what happens after Chapter 7 in a real training pipeline, and the chapter is written that way.

Volume II (Chapters 9–17) is much more modular. Chapter 9 (masked language models) and Chapter 10 (interpretability, still an explicit stub in this draft — two of its sections are headers with no content yet) can be read any time after Chapter 7. Chapter 11 (retrieval and RAG) stands on its own and is a good chapter to read early if your course is centered on applications rather than architecture. Chapter 12, “Agents,” is listed on the book’s own site but has not been written yet as of this update — it is not missing from this page’s reading plan, it simply does not exist yet.

Chapters 13–14 (machine translation, RNNs) and Chapters 15–17 (phonetics, ASR, TTS) are two separate tracks inside Volume II. A course that skips speech entirely can stop after Chapter 14 without losing continuity, since Chapters 15–17 do not feed back into anything in Volume III.

Volume III (Chapters 18–26) assumes Chapters 1–9 but not the speech chapters. Chapter 18 (sequence labeling) is the natural bridge — it reuses the HMM math from Appendix A directly, so read that appendix alongside it rather than after. Chapters 19–20 (parsing) and 21–22 (information extraction, semantic roles) build on each other in that order; Chapters 23–25 (lexical semantics, coreference, discourse) can be read in almost any order relative to each other.

Treat the appendices (A–K) purely as reference. Go to Appendix A when Chapter 18’s HMM math gets dense, to Appendix C when Chapter 3’s smoothing section points to it, and so on — reading all eleven start to finish before the main chapters is not how the book is designed to be used.

Because this is a living draft that the authors revise chapter by chapter (the most recent full update on the book’s own site is dated August 19, 2026, and Chapter 1 itself was substantially rewritten in that update), always check the book’s own site for the current version of a chapter before an exam, rather than relying on a printed or cached copy from earlier in the semester — chapter content and numbering have both moved between updates.

Used In These Programs

This book is used for the Natural Language Processing / Computational Linguistics course in: BS Computer Science. Browse all NLP books or all Computer Science books.

Who Should Read This

This book is written for students who already have a solid grounding in programming, probability, and linear algebra — it is not an introduction to any of those. It fits naturally in a junior- or senior-year BSCS Natural Language Processing or Computational Linguistics elective, and its Volume I chapters double as a fast, rigorous primer on how large language models actually work for students heading into an ML- or AI-focused final-year project. Its worked examples and exercises expect comfort with Python, so students who need a Python refresher first should start with Think Python before working through the neural network and transformer chapters here. It is not a good fit for a first exposure to programming, and it is not a good fit for a student who wants only the speech-processing chapters without the LLM background, since Chapters 15–17 assume familiarity with the neural network material in Chapters 6–9.

Applicable Universities

This book is useful for students at Pakistani universities offering BS Computer Science and related NLP/AI electives, including Punjab University, Virtual University, COMSATS, FAST, UET, NUST, and other HEC-recognized institutions.

FAQs

Related Books

• Think Python – Allen B. Downey (3rd Edition) — the Python fundamentals this book assumes you already have
• Problem Solving with Algorithms and Data Structures using Python — the algorithms and data-structure background behind this book’s Python-based exercises
• Discrete Mathematics: An Open Introduction – Oscar Levin (4th Edition) — the probability and combinatorics foundation behind the n-gram and HMM chapters
• Open Data Structures – Pat Morin — general algorithmic foundations for BSCS students

Speech and Language Processing (3rd edition draft) is the most current, most thorough free NLP textbook available to a Pakistani BSCS student today, and it keeps growing — its chapters on transformers, retrieval-augmented generation and post-training are more up to date than almost any printed textbook can be, because they are revised in place. Read Volume I in order, use the appendices as reference rather than a 27th chapter, and check the authors’ own site before an exam in case a chapter has moved since you last downloaded it. Browse more Computer Science books for the rest of your semester.

Speech and Language Processing: An Introduction to Natural Language Processing, Computational Linguistics, and Speech Recognition with Language Models (3rd edition, draft) by Daniel Jurafsky and James H. Martin. Freely hosted by the authors at Stanford University; copyright is retained by the authors and no open licence is attached. Official source: https://web.stanford.edu/~jurafsky/slp3/