These measures assess the impact of each observation by considering the model fit when that observation is omitted.
Let \(\hat{Y}_{i(i)}\) be the predicted value of \(Y_i\) when the \(i\)-th observation is left out of the regression model.
\[ \text{PRESS}_p = \sum_{i=1}^n \left(Y_i - \hat{Y}_{i(i)}\right)^2 = \sum e_{i(i)}^2 \]
\[ d = \frac{\sum_{i=1}^n (e_i - e_{i-1})^2}{\sum_{i=1}^n e_i^2} \]
| Name | Expression | Use |
|---|---|---|
| Residual | \(e_i = y_i - \hat{y}_i\) | Residual plots |
| Standardized residual | \(r_i = \frac{e_i}{s \sqrt{1 - h_{ii}}}\) | Identify outliers |
| Studentized residual | \(t_i = \frac{e_i}{s_{(i)} \sqrt{1 - h_{ii}}}\) | Test outlying \(Y\) values |
| Deleted residual | \(e_{i(i)} = y_i - \hat{y}_{i(i)} = \frac{e_i}{1 - h_{ii}}\) | Calculate PRESS |
For studentized residuals, at \(\alpha = 0.05\) we expect about 5% to be greater than 2 or less than –2.
| Pattern | Potential Issue | Solution |
|---|---|---|
| Linear means, constant SD | Model fits well | No action needed |
| Curved means, equal SD | Nonlinearity | Transform \(X\) |
| Curved means, increasing SD | Nonlinearity + heteroscedasticity | Transform \(Y\) |
| Skewed residuals | Non-normality | Can still model the mean, but CIs/PIs may be unreliable. Consider transformations. |
| Linear means, increasing SD | Heteroscedasticity | Use weighted regression |
Residual patterns to watch for. Each panel shows a scatter with bin-wise means and standard deviations (dark line and bars). Patterns suggest remedies: (a) compare the distribution of \(Y\) across \(X\); (b) curvature may call for transforming \(X\); (c) symmetric curvature suggests adding a quadratic term; (d) fan-shaped spread indicates transforming \(Y\); (e) right-skewed residuals should be reported; (f) variance that changes with \(X\) motivates weighted regression.
When the spread of residuals changes with fitted values, transforming the response variable \(Y\) can often stabilize the variance.
Examples of variance-stabilizing transformations. When the spread of residuals changes with fitted values, transforming the response variable \(Y\) can help stabilize variance. The panels illustrate three common recommendations: (a) a log transformation for decreasing variance, (b) a square root transformation for increasing variance, and (c) a reciprocal transformation for variance proportional to the mean. These transformations change the interpretation of model coefficients, and results may need to be back-transformed for reporting.
Log transformations can be highly effective and may yield more interpretable regression results.
The three most common forms are log-linear, linear-log, and log-log models.
\[ \log(\hat{Y}_i) = \beta_0 + \beta_1 X_i \]
Derivation: From log response to median interpretation
Interpretation: A one unit increase in \(X\) is associated with an \(e^{\beta_1}\) multiplicative change in the median of \(Y\).
\[ \hat{Y}_i = \beta_0 + \beta_1 \log (X_i) \]
Derivation: Mean change after doubling \(X\)
Interpretation:
A doubling of \(X\) is associated with a \(\beta_1 \log(2)\) unit change in the mean of \(Y\).
\[ \log (\hat{Y}_i) = \beta_0 + \beta_1 \log (X_i) \]
Derivation: Effect of doubling \(X\)
Interpretation:
A doubling of \(X\) is associated with a \(2^{\beta_1}\) multiplicative change in the median of \(Y\).
| Model Type | Equation | Interpretation |
|---|---|---|
| Log-linear | \(\log Y = \beta_0 + \beta_1 X\) | 1 unit increase in \(X\) \(\rightarrow\) \(e^{\beta_1}\) change in median \(Y\) |
| Linear-log | \(Y = \beta_0 + \beta_1 \log X\) | Doubling \(X\) \(\rightarrow\) \(\beta_1 \log(2)\) change in mean \(Y\) |
| Log-log | \(\log Y = \beta_0 + \beta_1 \log X\) | Doubling \(X\) \(\rightarrow\) ×\(2^{\beta_1}\) change in median \(Y\) |
Note: Logging \(Y\) generally shifts interpretation from the mean to the median.
The following SAS code fits both a simple linear regression model and a separate means model (one mean per iron content level). The ANOVA tables from each model are then used to compute the extra-sum-of-squares (lack-of-fit) test manually.
/* linear regression model */
proc glm data = IronCor;
model Corrosion = IronContent / solution;
run;
/* separate means model (7 groups) */
proc glm data = IronCor;
class IronContent;
model Corrosion = IronContent;
run;
data critval;
criticalValue = finv(0.95, 5, 6);
run;
proc print data = critval;
run;
data pval;
pValue = 1-probf(9.28, 5, 6);
run;
proc print data = pval;
run;
ANOVA output from SAS for the linear regression model (left) and separate means model (right) used in the lack-of-fit test.
Extra-sum-of-squares table showing the manual calculation of the lack-of-fit test from SAS output.
| Source | DF | SS | MS | F | Pr > F |
|---|---|---|---|---|---|
| Model | 5 | 91.07 | 18.21 | 9.28 | 0.0009 |
| Error / Full (SMM) | 6 | 11.78 | 1.96 | ||
| Total / Reduced (LRM) | 11 | 102.85 |
Conclusion: The lack-of-fit test compares the linear regression model (reduced) to the separate means model (full). There is strong evidence, to suggest the linear regression model lacks fit with respect to the separate means model, p-value = 0.0009.