Statistics 2nd Year Chapter 14 Simple Linear Regression and Correlation Notes (ICS Part-II)

Chapters 12 and 13 dealt with a single variable at a time — estimating or testing claims about one population mean or proportion. This chapter opens a new area of statistics devoted to relationships BETWEEN two variables. Two very different but closely related questions are asked about any pair of variables: is there a mathematical rule that lets us predict one variable from the other (regression), and how strong or close is the linear association between them (correlation)? Regression analysis fits the 'best' straight line through a scatter of paired observations and uses it to estimate or predict the value of one variable (the dependent, or regressand) from a given value of the other (the independent, or regressor). Correlation analysis, by contrast, treats both variables symmetrically and measures the degree to which they move together, without singling out either one as dependent.

The chapter builds the whole machinery from first principles: the distinction between a functional and a statistical relation, the population regression model and its sample counterpart, the method of least squares used to fit the best line, the sample correlation coefficient r and its properties, and finally the tight algebraic relationship that ties regression coefficients and the correlation coefficient together. A recurring and important theme is that regression and correlation are mathematically linked but conceptually distinct — and that neither one, however strong, proves that one variable causes changes in the other.

Learning Objectives

• Distinguish a functional relation from a statistical relation between two variables
• Define regressor, regressand, regression function, and regression curve
• State and apply the least squares principle to fit a straight line to paired data
• Derive and use the least squares estimates a and b of the population regression line
• Use coding and scaling to simplify regression computations for equally spaced x-values
• State the limitations of linear regression
• Define positive, negative, and no correlation between two variables
• Compute and interpret the sample correlation coefficient r and list its key properties
• Relate the correlation coefficient r to the two regression coefficients b_yx and b_xy
• Explain why a high correlation does not, by itself, establish causation

Key Concepts

14.1 Relations Between Variables

A functional relation between two variables x and y is a perfect relation: the value of the dependent variable y is uniquely and exactly determined by the value of the independent variable x through a mathematical formula, y=f(x). Observations plotted on a graph all fall exactly on the line or curve of the relationship — this exactness is the defining feature of a functional relation. A statistical relation, by contrast, is not exact: the value of y is not uniquely determined merely from knowledge of x, and plotted observations scatter around, rather than fall exactly on, a line or curve. Most real relationships in business, economics, and the social sciences — income and expenditure, fertilizer and crop yield, advertising cost and sales, a father's height and his son's height — are statistical, not functional, relations.

A separate question from the mathematical form of a relationship is whether it is causal: does a genuine cause-and-effect mechanism link the two variables, or do they merely tend to move together? Regression analysis and correlation analysis, on their own, make no assertions whatsoever about causality — establishing cause requires evidence and reasoning that goes well beyond fitting a line or computing a correlation coefficient.

14.2 Regression Analysis

Regression analysis provides a method for estimating an average (often linear) relationship between two or more variables, allowing the investigator to explain and predict the value of one variable from known values of the others. In the simplest case, only two variables are involved: the regressor (also called the predictor, independent, controlled, or explanatory variable, usually denoted x) is the variable that forms the basis for estimation, its values fixed in advance by the experimenter and free of random error; the regressand (also called the response, predictand, dependent, or explained variable, usually denoted Y) is the variable whose value is being estimated or predicted, and which is subject to random variation even for a fixed value of x.

If mu_Y|x = E(Y|x) denotes the expected value of the distribution of Y for a given, fixed value of x, then the simple regression is defined as mu_Y|x = f(x), where f(x) may be linear, quadratic, exponential, or any other functional form describing how the regressor relates to the response. Because the points only are given and the function f(x) must be 'worked backwards' or regressed from them, this function is called the regression function. The regression curve is the locus of these expected values mu_Y|x traced out as x varies — the curve joining the expected value of Y's distribution at each possible value of x.

14.3 Curve Fitting

Curve fitting is the process of estimating, from an observed sample, the parameters of the population regression function relating a response variable to a regressor variable. The least squares principle, due to the French mathematician Adrien Legendre, states that among all possible curves that could be fitted to a set of observed data, the sum of squares of the residuals (the observed values minus their corresponding fitted/estimated values) should be made as small as possible; the curve achieving this minimum is called the least squares fit. Using this principle avoids the personal bias that would otherwise creep in if a curve were fitted to data purely 'by eye'.

Before selecting the form of curve to fit, a scatter diagram is drawn: a set of points in a rectangular coordinate system, with x plotted horizontally and y plotted vertically, where each point represents one observed pair (x_i, y_i). A visual inspection of the scatter diagram reveals the apparent nature of the relationship — whether the points tend to run from lower-left to upper-right (a direct relationship, Y tends to increase as x increases), from upper-left to lower-right (an inverse relationship, Y tends to decrease as x increases), whether the pattern looks straight (linear) or bent (curvilinear), and how tightly or loosely the points cluster around the apparent pattern.

14.4 Simple Linear Regression

When the simple regression describes the expected value of the dependent variable Y as a LINEAR function of the independent variable x, the regression is called simple linear regression, written mu_Y|x = alpha + beta.x. Here alpha is the intercept of the line along the y-axis (the value of mu_Y|x when x=0), and beta — the simple linear regression coefficient — is the change in the mean of Y's probability distribution per unit increase in x, i.e. the slope of the line, which remains constant at every value of x. Geometrically, beta is measured by tan(theta), where theta is the angle the line makes with the positive x-axis: if beta is positive the line slopes upward, and if beta is negative the line slopes downward.

14.5 The Simple Linear Regression Model

For a fixed value x_i of the regressor, the associated response is a random variable Y_i with mean mu_Y|xi = alpha + beta.x_i and constant variance sigma^2 (assumed the same for every value of x — the variance does not depend on x, even though the mean does). The deviation of Y_i from its mean is the random error, epsilon_i = Y_i – (alpha + beta.x_i), and the resulting population regression model is Y_i = alpha + beta.x_i + epsilon_i, where alpha and beta are the (unknown) population intercept and slope. This model is 'simple' because there is only one regressor, 'linear in parameters' because no parameter appears as an exponent or is multiplied by another parameter, and 'linear in the independent variable' because x appears only to the first power.

Since alpha and beta are unknown population parameters, they must be estimated from sample data (x1,y1), (x2,y2), …, (xn,yn). Let a be the best estimate of alpha and b be the best estimate of beta; then the sample simple linear regression line is y-hat = a + b.x, and for any specific x_i, y-hat_i = a + b.x_i is the best point estimate of mu_Y|xi. The residual (or deviation, or prediction error) for observation i is e_i = y_i – y-hat_i = y_i – a – b.x_i, giving the sample regression model y_i = a + b.x_i + e_i.

The covariance of two variables X and Y, denoted s_xy, measures their linear mutual variability: s_xy = [sum(x_i – xbar)(y_i – ybar)]/n = [sum(x_i.y_i)]/n – xbar.ybar. Its sign shows the direction of mutual variability — positive when the variables tend to move together, negative when they move in opposite directions. If z_i = x_i +/- y_i, then s_z^2 = s_x^2 + s_y^2 +/- 2.s_xy in general, reducing to s_z^2 = s_x^2 + s_y^2 when X and Y are independent (s_xy=0).

Theorem 14.1 (Least Squares Point Estimation): given n observed pairs (x_i, y_i), the least squares line y-hat = a + b.x has slope b = [n.sum(x_i.y_i) – (sum x_i)(sum y_i)] / [n.sum(x_i^2) – (sum x_i)^2] = [sum(x_i-xbar)(y_i-ybar)] / [sum(x_i-xbar)^2] = s_xy/s_x^2, and intercept a = [sum y_i – b.sum x_i]/n = ybar – b.xbar. The least squares line always passes through the point of means (xbar, ybar), and the least squares estimate of mu_Y|xi is y-hat_i = ybar + b(x_i – xbar).

Coding and Scaling (Theorem 14.2): the sample regression coefficient b is independent of a change of origin but NOT independent of a change of scale; if u_i=(x_i-p)/h and v_i=(y_i-q)/k, then b_yx = (k/h).b_vu. When the x-values are equally spaced at interval h, a special coding takes the origin at the middle value (odd n) or the average of the two middle values (even n), and rescales by h (or h/2), making sum(u_i)=0 and greatly simplifying the arithmetic; the resulting line of Y on u, y-hat=a+bu with a=ybar and b=sum(u_i.y_i)/sum(u_i^2), is then transformed back to the original x-scale.

Properties of the least squares line: the sum of residuals is zero (sum e_i = 0); the sum of the observed y-values equals the sum of the fitted values (sum y_i = sum y-hat_i), so ybar equals the mean of the fitted values too; the sum of squared residuals, sum(e_i^2) = sum(y_i^2) – a.sum(y_i) – b.sum(x_i.y_i), is the smallest possible value achievable by ANY straight line through the data; and the line always passes through the point of means (xbar, ybar) — the 'centre of gravity' of the data.

Limitations of Linear Regression: (i) it applies only to relationships that can genuinely be described by a straight line — a scatter diagram should be examined first to check this; (ii) the least-squares procedure will always produce a 'best-fit' line, even when no real linear relationship exists at all, so a formal test of significance is needed to confirm the slope b is 'real' and not just sampling noise; (iii) the regression equation is asymmetrical — the equation predicting Y from x cannot simply be algebraically rearranged to predict x from Y, a separate regression of x on Y must be fitted; (iv) the regression equation is reliable only over the range of x actually observed in the sample, and should not be extrapolated far beyond that range.

14.6 Simple Linear Correlation

Simple linear correlation measures the strength or closeness of a linear relationship between two variables, addressing two related questions: are the two variables related at all, and how closely does one variable's regression line fit the scatter of observed points? Positive (direct) correlation occurs when the two variables tend to move in the same direction (both increase or both decrease together), corresponding to regression lines with positive slopes; it is called perfect positive correlation if every point lies exactly on a straight line with positive slope. Negative (inverse) correlation occurs when the variables move in opposite directions, corresponding to negative-sloped regression lines; perfect negative correlation means every point lies exactly on a line with negative slope. No correlation exists when one least squares regression line is horizontal and the other is vertical — equivalently, if X and Y are independent, Cov(X,Y)=0, which implies the correlation coefficient rho=0.

14.7 Correlation Analysis

In pure correlation problems, both variables X and Y are random, and the relationship is considered simultaneously and symmetrically — unlike regression, where one variable is singled out as dependent. Typical correlation examples include heights and weights of persons, ages of husbands and wives at marriage, and marks in two different subjects. Given a random sample of n pairs (x_i,y_i) from a bivariate population, the sample correlation coefficient r (or r_xy) is defined as r = s_xy/(s_x.s_y), which expands into several algebraically equivalent computing formulas, most commonly r = [n.sum(x_i.y_i) – (sum x_i)(sum y_i)] / sqrt{[n.sum(x_i^2)-(sum x_i)^2][n.sum(y_i^2)-(sum y_i)^2]}.

Properties of r: (1) r is symmetrical in X and Y, i.e. r_xy = r_yx; (2) r equals the covariance of the two variables measured in standard units, r=Cov(z_x,z_y); (3) the MAGNITUDE |r| is unaffected by a change of origin or scale in either variable, though r changes sign if the variables are multiplied/divided by constants of opposite sign; (4) r always lies between -1 and +1 inclusive; (5) |r| is the geometric mean of the two regression coefficients, r = +/-sqrt(b_yx . b_xy), with the sign matching the common sign of b_yx and b_xy; (6) r is zero whenever one of the two variables is constant. Theorem 14.3 formalises property (3): the correlation coefficient is independent of the origin and the scale of measurement of the variables (up to a possible sign flip if the scale change reverses direction).

Goodness of Fit and the Regression-Correlation Link: for a bivariate sample there are two distinct least squares lines — Y on X, y-hat=a_yx+b_yx.x, and X on Y, x-hat=a_xy+b_xy.y — with b_yx=s_xy/s_x^2=r.s_y/s_x and b_xy=s_xy/s_y^2=r.s_x/s_y. Because s_x and s_y are always positive, s_xy, b_yx, b_xy, and r always share the same sign, and if any one of these four quantities is zero, all the others must be zero too. Theorem 14.4 states the two regression lines always intersect at the point of means (xbar, ybar). Theorem 14.5 states that r is the slope of the regression lines when both variables are expressed in standard (z-score) units. Theorem 14.6 states that the two regression lines coincide (become identical) exactly when every sample point lies exactly on a single straight line, i.e. when |r|=1.

Correlation and Causation: a high correlation coefficient shows only that the observed data are consistent with a linear association — it does NOT, by itself, establish that one variable causes changes in the other. A high correlation between two variables may arise because X causes Y, because Y causes X, because some third factor Z affects both X and Y simultaneously (a common cause, sometimes called a spurious or nonsense correlation), or purely by chance. Only further, careful investigation beyond the correlation coefficient itself — controlled experiments, theoretical reasoning, checking for confounding variables — can establish whether a causal relationship genuinely exists.

Important Definitions

Key Facts and Relations

Topic	Key Fact / Relation
Population simple linear regression	mu_Y|x = alpha + beta.x; Y_i = alpha + beta.x_i + epsilon_i
Sample simple linear regression	y-hat = a + bx; y_i = a + b.x_i + e_i, e_i = y_i – y-hat_i
Least squares slope, b	b = [n.sum(x_i y_i) – (sum x_i)(sum y_i)] / [n.sum(x_i^2) – (sum x_i)^2] = s_xy/s_x^2
Least squares intercept, a	a = [sum y_i – b.sum x_i]/n = ybar – b.xbar
Covariance, s_xy	s_xy = [sum(x_i-xbar)(y_i-ybar)]/n = [sum x_i y_i]/n – xbar.ybar
Sample correlation coefficient, r	r = s_xy/(s_x.s_y) = [n.sum(x_i y_i)-(sum x_i)(sum y_i)] / sqrt{[n.sum x_i^2-(sum x_i)^2][n.sum y_i^2-(sum y_i)^2]}
Regression coefficients via r	b_yx = s_xy/s_x^2 = r.s_y/s_x; b_xy = s_xy/s_y^2 = r.s_x/s_y
r as geometric mean of regression coefficients	r = (+/-) sqrt(b_yx . b_xy), sign = common sign of b_yx and b_xy
Regression equation of Y on X (via r)	y-hat = ybar + (r.s_y/s_x)(x – xbar)
Regression equation of X on Y (via r)	x-hat = xbar + (r.s_x/s_y)(y – ybar)
Coding and scaling relation	u_i=(x_i-p)/h, v_i=(y_i-q)/k => b_yx = (k/h).b_vu
Sum of squared residuals identity	sum(e_i^2) = sum(y_i^2) – a.sum(y_i) – b.sum(x_i y_i) [minimum value over all straight lines]

Diagrams

Scatter Diagram Patterns: A 2×2 panel of scatter diagrams illustrating the main relationship patterns covered in the chapter: direct linear relationship (positive slope, points rising left to right), inverse linear relationship (negative slope, points falling left to right), a curvilinear relationship (points following a bending curve rather than a line), and no apparent relationship (points scattered with no discernible pattern), matching Fig 14.1 of the textbook

Least Squares Fitted Line with Residuals: A scatter diagram of Example 14.2's data (x=0,1,2,3,4; y=1.0,1.8,3.3,4.5,6.3) with the least squares regression line y-hat=0.72+1.33x drawn through the points, and a vertical dashed segment from each observed point down (or up) to the fitted line representing its residual e_i, illustrating the least squares principle of minimizing the sum of squared residuals

Two Regression Lines Intersecting at the Point of Means: A scatter diagram with both least squares lines drawn — the regression of Y on X and the regression of X on Y — shown crossing exactly at the point (xbar, ybar), illustrating Theorem 14.4 and how the angle between the two lines narrows as the correlation |r| approaches 1 (lines nearly coincide) and widens as |r| approaches 0 (lines nearly perpendicular)

Short Questions & Answers

Long Questions & Answers

Trace the full development of the simple linear regression model from the population regression line through to the sample least squares line, explaining the role of each symbol and using Example 14.1 to illustrate the computations.

What is the population regression line, and what do alpha and beta represent?

For a given x, the dependent variable Y is random, with mean mu_Y|x = E(Y|x) and constant variance sigma_Y|x^2 = sigma^2 (an assumption called homoscedasticity). If these conditional means lie on a straight line, the population regression line is mu_Y|x = alpha + beta*x, where alpha is the population y-intercept (the mean of Y when x=0) and beta is the population slope, the change in mean Y per unit increase in x. For an individual x_i, Y_i = alpha + beta*x_i + epsilon_i, where epsilon_i is the random error.

What is the sample regression line, and how are a and b found?

Since alpha, beta, mu_Y|x, and sigma^2 are unknown population parameters, they are estimated from n observed pairs (x_i, y_i): a estimates alpha, b estimates beta, giving the fitted line y-hat = a + bx, with the residual e_i = y_i – y-hat_i estimating the unobservable error. The least squares formulas are b = [n*sum(x_iy_i) – (sum x_i)(sum y_i)] / [n*sum(x_i^2) – (sum x_i)^2], equivalently s_xy/s_x^2, and a = [sum y_i – b*sum x_i]/n = y-bar – b*x-bar.

Walk through Example 14.1: how are a and b computed from the eight students' data?

Eight students' matriculation scores (480 to 640) paired with their GPAs (2.7 to 3.7) give sum x_i=4320, sum y_i=24.8, sum(x_i^2)=2355800, sum(x_iy_i)=13492, with n=8. Substituting into the least squares formulas gives b = [8(13492)-(4320)(24.8)] / [8(2355800)-4320^2] = 0.00435, and a = [24.8-0.00435(4320)]/8 = 0.751, so the fitted line is y-hat = 0.751 + 0.00435x. For a student scoring x=600, the predicted GPA is 0.751+0.00435(600) = 3.361.

What properties are guaranteed to hold for any least squares line?

The sum of the residuals, sum(e_i), is always exactly zero. The sum of the observed y-values equals the sum of the fitted y-hat values, so the mean of the fitted values equals y-bar. The sum of squared residuals is the smallest possible value achievable by any straight line through the data — that is the entire point of the least squares principle. And the fitted line always passes through the point of means (x-bar, y-bar); in Example 14.1, x-bar=540, y-bar=3.1, and indeed y-hat=0.751+0.00435(540)=3.10.

Define the sample correlation coefficient r, list its key properties, explain how it connects algebraically to the two regression coefficients b_yx and b_xy, and explain why correlation does not establish causation.

What is the sample correlation coefficient r, and how does correlation analysis differ from regression analysis?

Unlike regression, correlation treats X and Y symmetrically as jointly random, with no distinguished independent or dependent variable — just a bivariate population sampled as pairs (x_i, y_i). The sample correlation coefficient, r or r_xy, is defined as r = s_xy/(s_x*s_y), the covariance of X and Y divided by the product of their standard deviations, computable as r = [n*sum(x_iy_i)-(sum x_i)(sum y_i)] / sqrt{[n*sum(x_i^2)-(sum x_i)^2][n*sum(y_i^2)-(sum y_i)^2]}.

What are the key properties of r regarding symmetry, standardization, and invariance to origin and scale?

r is symmetrical: r_xy = r_yx, unlike the two regression coefficients, which generally differ. r can also be seen as the covariance between the two variables after converting each to standardized z-score units. Usefully, the magnitude |r| is unaffected by a change of origin or by rescaling both variables by constants of the same sign — Example 14.5 confirms r=0.956 whether computed from raw height/weight data or from deviations about assumed means; r only flips sign if the rescaling constants have opposite signs.

What range does r take, and what does a value near +1, -1, or 0 mean?

r always lies within the interval [-1, +1]. The extreme values +1 and -1 occur only when every sample point lies exactly on a straight line with positive or negative slope (perfect correlation). A value near zero indicates little or no linear association — though a strong nonlinear relationship could still exist even when r is near zero, since r only measures linear association.

How does r connect algebraically to the two regression coefficients b_yx and b_xy?

The magnitude |r| is the geometric mean of the two regression slopes: r = +/- sqrt(b_yx * b_xy), with the sign of r matching the shared sign of b_yx and b_xy. This gives b_yx = r*s_y/s_x and b_xy = r*s_x/s_y. The two regression lines always intersect at the point of means (x-bar, y-bar); r equals their common slope when the data are standardized; and the two lines coincide exactly when |r|=1, while being at right angles to each other when r=0.

Why does a high correlation not prove causation?

A high |r| is consistent with several different explanations: X may genuinely cause Y, Y may cause X, an unmeasured third factor may independently drive both (a 'spurious' correlation, like the textbook's example of city temperature and birth rate), or the correlation may simply be a product of chance in a small sample. Establishing genuine causation requires evidence well beyond the correlation coefficient itself — controlled experiments, a plausible mechanism, ruling out confounders, and replication in independent samples.

Multiple Choice Questions (MCQs)

Quick Revision Summary

• Functional relation = exact, y=f(x) | Statistical relation = not exact, points scatter around the curve
• Regressor (x) = independent/predictor, fixed in advance | Regressand (Y) = dependent/response, random
• Least squares principle: minimize sum(e_i^2), the sum of SQUARED residuals
• Population model: mu_Y|x = alpha + beta.x, Y_i = alpha + beta.x_i + epsilon_i
• Sample model: y-hat = a + bx; b = s_xy/s_x^2; a = ybar – b.xbar
• Least squares line ALWAYS passes through (xbar, ybar); sum(e_i)=0; sum(y_i)=sum(y-hat_i)
• Covariance s_xy: positive = variables move together, negative = move oppositely
• Correlation r = s_xy/(s_x s_y); |r| unaffected by change of origin/scale; -1<=r<=1
• r = +/-sqrt(b_yx . b_xy); b_yx = r.s_y/s_x; b_xy = r.s_x/s_y
• Two regression lines meet at (xbar, ybar); coincide when |r|=1; right angles when r=0
• Regression/correlation never proves causation — could be X causes Y, Y causes X, a third factor, or chance

Exam Tips

• Always draw or imagine the scatter diagram FIRST — it tells you whether a straight-line fit even makes sense before you compute anything
• Remember b (slope) uses s_xy/s_x^2, NOT s_xy/(s_x.s_y) — that second formula is for r, a very common mix-up
• When x-values are equally spaced, use the coding trick (origin at the middle, unit = the common interval) to avoid large, error-prone numbers
• Check your answer: does the fitted line pass through (xbar, ybar)? If not, recheck your arithmetic
• |r| is unaffected by adding/subtracting or multiplying/dividing by a POSITIVE constant — use this to simplify awkward raw data before computing r
• Never claim causation from r alone in an exam answer — always note that regression/correlation analysis makes no assertion about cause and effect