Linear Least Squares Regression

I. Why linear regression?

A. Its simple.

B. It fits many functions pretty well.

C. Many nonlinear functions can be transformed to linear.

II. Why least squares regression?

A. Because it works better than the alternatives in many cases.

B. Because it is easy to work with mathematically.

III. Derivation of the least-squares parameters of a line

A. Equation of a line:

(a is the y intercept, b is the slope

B. We wish to minimize the sum of squared deviations of estimated y values () from the actual y values (y):

C. To simplify the mathematics, we can transform the x_i's to be difference scores by subtracting their mean:

x_i=x_i- Hence, the Sx_i= 0.

D. Substitute equation for line into summation above.

=

1. To find the a (intercept) and b (slope) that will make this expression the smallest, take the partial derivatives of the expression, set them equal to zero, and solve the equations.

= S 2(-1)(y_i-a-bx_i)= 0

-2 S(y_i-a-bx_i) = 0 distributive rule: -2(w+u)=-2w+(-2u)

S (y_i-a-bx_i) = 0 divide both sides by -2

Sy_i - Na - bSx_i = 0 carry out the summation

Sy_i - Na = 0 recall that Sx_i= 0

S y_i = Na add Na to both sides

S y_i/N = a divide by N

S y_i/N =

= S 2(-x_i)(y_i-a-bx_i) = 0

-2 S( x_iy_i -ax_i -bx_i²⁾= 0 distributive rule: -2(w+u)=-2w+(-2u)

S (x_iy_i -ax_i -bx_i²⁾ = 0 ivide both sides by -2

Sx_iy_i -Sax_i -Sbx_i² = 0 carry out the summation

Sx_iy_i -aSx_i -bSx_i² = 0 distributive rule again

S x_iy_i - bS x_i² = 0 recall that Sx_i= 0

S x_iy_i = bS x_i² add bSx_i² to both sides

S x_iy_i/S x_i² = b divide by Sx_i²

IV. Goodness of fit of regression model

A. Decomposition of variation

Total SS = SS regression + SS residual

Total variation around mean of Y= variation “explained” by line + unexplained variation around line.

= +

B. Proportion of variance explained by line

1. = =

C. Estimate of variance s²

s²=

where k=number of predictors (in linear regression=1). Note: 2 df are lost, 1 for each parameter estimated (no variance with only 2 points)

D. F-ratio testing variance explained by line

=

df_reg = k = # of predictors; df_res = N-k-1

V. Testing parameters of regression model

A. Purpose

1. The estimated a and b are sample estimates of the true population parameters. Want to determine how close the sample estimates are to the true population parameters.

2. Confidence intervals give a range of values within which the population parameters will lie with a stated degree of certainty (a).

3. Tests of significance ask whether for some given degree of certainty, the population value of a parameter may be different from some given value (usually 0).

B. Standard Assumptions

1. Given a regression model: y_i=a + bx_i + e_i

2. y_i are independently and identically distributed with variance s².

3. e_i are independently and identically distributed with mean 0 and variance s².

4. x_i are fixed (not random variables).

C. Statistics for b

1. Derivation of variance of b

b = Sx_iy_i/Sx_i² see above

= S(x_i/Sx_i²)y_i distributive rule

= Sw_iy_i rewrite equation; let w_i=x_i/Sx_i²

so, b is a linear combination of random variables

var(b) = Sw_i²var(y_i) variance of linear combination of random variables

= Sw_i²s² by assumption 2 above

= Sx_i² s² replacing wi by its equivalent

(Sx_i²)²

= s² simplifying by canceling one Sxi2

Sx_i²

2. Standard error of b

s_b = =

3. T-test for b

t = df=N-k-1

4. Confidence interval for b

ß = b ± t_a/2

5. Note: large variation in x will yield smaller s_b and larger t. With small variation in x, estimates are unstable.

D. The constant is rarely of interest. When it is, similar tests can be performed. Note that the constant is simply the regression coefficient for a predictor that does not vary in the data. The variance of a = s²/n.

VI. Problems

A. Outliers

1. There may be observations for which the relation(s) between the criterion and the predictor(s) are not summarized well by the regression equation.

B. Heteroscedasticity

1. The variance around the regression line may not be constant. Hence, the equation predicts better in some ranges than in others.

C. Curvilinearity

1. The regression line may systematically underestimate in some ranges and overestimate in others because the relation between the criterion and the predictor(s) is not linear.

D. Autocollinearity

1. The observations (and the residuals) may be correlated (frequently a problem with time series data) yielding inaccurate parameter estimates that may appear to be more precise than they really are.

E. Nonlinearity

1. The relation(s) between the predictor(s) and the criterion may be nonlinear. For example:

a. y= a + bx +bx² + e

b. y= a + bln(x) + e

2. Note that in some cases, relations that are theoretically nonlinear may be transformed into linear relations.

a. Example - learning theory transformed to linear

t_i = ab^xi a > 0 [positive] 0<b<1 [fraction]

t_i = time taken to perform task on ith occasion

x_i = # of trials

b. Converted to linear

ln(t_i) = ln(ab^xi) taking natural log of both sides

= ln(a) + ln(b^xi) ln(w*u)=ln(w)+ln(u)

= ln(a) + xln(b) ln(w^u)=u*ln(w)

3. However, some relations cannot be transformed to linear (intractable nonlinearity).

VII. Detection of Problems

A. Think about the data

1. What relations between the variables might be expected?

2. What problems may occur?

B. Plot the data

1. Plot criterion by predictors and examine form and spread

C. Plot the residuals after performing the regression.

1. Plot residuals or standardized residuals by predicted values of the criterion.

D. Examine diagnostic statistics for the presence of outliers.

1. Large residuals

a. Some authors suggest the rule of thumb that any standardized residual > 2 requires inspection.

b. Some authors suggest examining studentized residuals because the residuals may not be homoscedastic. The studentized residual is calculated by dividing each residual by its estimated standard deviation. The estimated standard deviation of a residual is defined as:

where s=standard error of the estimate (see above).

The studentized residuals can then be tested against a t distribution with N-k-1 degrees of freedom. Note that in practice this is not an appropriate test because the t tests are not independent. However, it is a way of locating unusually large residuals.

2. Leverage

a. Some authors suggest examining the “leverage” of an observation, defined as the quantity in brackets above:

h_i =

A large leverage is an indication that the value of the predictor is far from the mean of that predictor.

3. Cook’s D (Cook, 1977)

a. Leverage does not measure the influence of an observation on the criterion. Cook’s D does:

where e_si² = studentized residual for observation i; h_i=leverage for observation i. Approximate tests of the significance of D are available, but usually one simply looks for D’s that are large relative to the others in the dataset.

4. Delete suspect observations

a. One should be able to delete points without substantial effect on parameters. If one cannot, then either the number of observations is too small or the observation in question is an outlier.

b. Some computer programs routinely report DFBETA_i which is the regression coefficient that would result were observation i to be deleted.