Simple Linear Regression Analysis

Regression analysis is a statistical technique that attempts to explore and model the relationship between two or more variables. For example, an analyst may want to know if there is a relationship between road accidents and the age of the driver. Regression analysis forms an important part of the statistical analysis of the data obtained from designed experiments and is discussed briefly in this chapter. Every experiment analyzed in a Weibull++ DOE foilo includes regression results for each of the responses. These results, along with the results from the analysis of variance (explained in the One Factor Designs and General Full Factorial Designs chapters), provide information that is useful to identify significant factors in an experiment and explore the nature of the relationship between these factors and the response. Regression analysis forms the basis for all Weibull++ DOE folio calculations related to the sum of squares used in the analysis of variance. The reason for this is explained in Appendix B. Additionally, DOE folios also include a regression tool to see if two or more variables are related, and to explore the nature of the relationship between them.

This chapter discusses simple linear regression analysis while a subsequent chapter focuses on multiple linear regression analysis.

Simple Linear Regression Analysis

A linear regression model attempts to explain the relationship between two or more variables using a straight line. Consider the data obtained from a chemical process where the yield of the process is thought to be related to the reaction temperature (see the table below).

This data can be entered in the DOE folio as shown in the following figure:

And a scatter plot can be obtained as shown in the following figure. In the scatter plot yield, $y_i$ is plotted for different temperature values, $x_i$.

It is clear that no line can be found to pass through all points of the plot. Thus no functional relation exists between the two variables $x$ and $Y$. However, the scatter plot does give an indication that a straight line may exist such that all the points on the plot are scattered randomly around this line. A statistical relation is said to exist in this case. The statistical relation between $x$ and $Y$ may be expressed as follows:

$Y={\beta }_0+{\beta }_{1}x+ϵ$

The above equation is the linear regression model that can be used to explain the relation between $x$ and $Y$ that is seen on the scatter plot above. In this model, the mean value of $Y$ (abbreviated as $E(Y)$) is assumed to follow the linear relation:

$E(Y)={\beta }_0+{\beta }_{1}x$

The actual values of $Y$ (which are observed as yield from the chemical process from time to time and are random in nature) are assumed to be the sum of the mean value, $E(Y)$, and a random error term, $ϵ$:

$\begin{array}{rl}Y=& E(Y)+ϵ\\ =& {\beta }_0+{\beta }_{1}x+ϵ\end{array}$

The regression model here is called a simple linear regression model because there is just one independent variable, $x$, in the model. In regression models, the independent variables are also referred to as regressors or predictor variables. The dependent variable, $Y$ , is also referred to as the response. The slope, ${\beta }_1$, and the intercept, ${\beta }_0$ , of the line $E(Y)={\beta }_0+{\beta }_{1}x$ are called regression coefficients. The slope, ${\beta }_1$, can be interpreted as the change in the mean value of $Y$ for a unit change in $x$.

The random error term, $ϵ$, is assumed to follow the normal distribution with a mean of 0 and variance of ${\sigma }^2$. Since $Y$ is the sum of this random term and the mean value, $E(Y)$, which is a constant, the variance of $Y$ at any given value of $x$ is also ${\sigma }^2$. Therefore, at any given value of $x$, say $x_i$, the dependent variable $Y$ follows a normal distribution with a mean of ${\beta }_0+{\beta }_1{x}_i$ and a standard deviation of $\sigma$. This is illustrated in the following figure.

Fitted Regression Line

The true regression line is usually not known. However, the regression line can be estimated by estimating the coefficients ${\beta }_1$ and ${\beta }_0$ for an observed data set. The estimates, ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$, are calculated using least squares. (For details on least square estimates, refer to Hahn & Shapiro (1967).) The estimated regression line, obtained using the values of ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$, is called the fitted line. The least square estimates, ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$, are obtained using the following equations:

${\hat{\beta }}_1=\frac{\sum _{i=1}^n{y}_i{x}_i-\frac{(\sum _{i=1}^n{y}_i)(\sum _{i=1}^n{x}_i)}{n}}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}$

${\hat{\beta }}_0=\overline{y}-{\hat{\beta }}_1\overline{x}$

where $\overline{y}$ is the mean of all the observed values and $\overline{x}$ is the mean of all values of the predictor variable at which the observations were taken. $\overline{y}$ is calculated using $\overline{y}=(1/n)\sum {)}_{i=1}^n{y}_i$ and $\overline{x}$ is calculated using $\overline{x}=(1/n)\sum {)}_{i=1}^n{x}_i$.

Once ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$ are known, the fitted regression line can be written as:

$\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_{1}x$

where $\hat{y}$ is the fitted or estimated value based on the fitted regression model. It is an estimate of the mean value, $E(Y)$. The fitted value,${\hat{y}}_i$, for a given value of the predictor variable, $x_i$, may be different from the corresponding observed value, $y_i$. The difference between the two values is called the residual, $e_i$:

$e_i=y_i-{\hat{y}}_i$

Calculation of the Fitted Line Using Least Square Estimates

The least square estimates of the regression coefficients can be obtained for the data in the preceding table as follows:

$\begin{array}{rl}{\hat{\beta }}_1=& \frac{\sum _{i=1}^n{y}_i{x}_i-\frac{(\sum _{i=1}^n{y}_i)(\sum _{i=1}^n{x}_i)}{n}}{\sum _{i=1}^n(x_i-\overline{x}{)}^2}\\ =& \frac{322516-\frac{4158x1871}{25}}{5697.36}\\ =& 1.9952\approx 2.00\end{array}$

$\begin{array}{rl}{\hat{\beta }}_0=& \overline{y}-{\hat{\beta }}_1\overline{x}\\ =& 166.32-2x74.84\\ =& 17.0016\approx 17.00\end{array}$

Knowing ${\hat{\beta }}_0$ and ${\hat{\beta }}_1$, the fitted regression line is:

$\begin{array}{rl}\hat{y}=& {\hat{\beta }}_0+{\hat{\beta }}_{1}x\\ =& 17.0016+1.9952\times x\\ \approx & 17+2x\end{array}$

This line is shown in the figure below.

Once the fitted regression line is known, the fitted value of $Y$ corresponding to any observed data point can be calculated. For example, the fitted value corresponding to the 21st observation in the preceding table is:

$\begin{array}{rl}{\hat{y}}_{21}=& {\hat{\beta }}_0+{\hat{\beta }}_1{x}_{21}\\ =& (17.0016)+(1.9952)\times 93\\ =& 202.6\end{array}$

The observed response at this point is $y_{21}=194$. Therefore, the residual at this point is:

$\begin{array}{rl}{e}_{21}=& y_{21}-{\hat{y}}_{21}\\ =& 194-202.6\\ =& -8.6\end{array}$

In DOE folios, fitted values and residuals can be calculated. The values are shown in the figure below.

Hypothesis Tests in Simple Linear Regression

The following sections discuss hypothesis tests on the regression coefficients in simple linear regression. These tests can be carried out if it can be assumed that the random error term, $ϵ$, is normally and independently distributed with a mean of zero and variance of ${\sigma }^2$.

t Tests

The $t$ tests are used to conduct hypothesis tests on the regression coefficients obtained in simple linear regression. A statistic based on the $t$ distribution is used to test the two-sided hypothesis that the true slope, ${\beta }_1$, equals some constant value, ${\beta }_{1,0}$. The statements for the hypothesis test are expressed as:

$\begin{array}{rlr}{H}_0& :& {\beta }_1={\beta }_{1,0}\\ H_1& :& {\beta }_1\ne {\beta }_{1,0}\end{array}$

The test statistic used for this test is:

$T_0=\frac{{\hat{\beta }}_1-{\beta }_{1,0}}{se({\hat{\beta }}_1)}$

where ${\hat{\beta }}_1$ is the least square estimate of ${\beta }_1$, and $se({\hat{\beta }}_1)$ is its standard error. The value of $se({\hat{\beta }}_1)$ can be calculated as follows:

$se({\hat{\beta }}_1)=\sqrt{\frac{\frac{\sum _{i=1}^n}{e_i^2}}{\sum _{i=1}^n}}$

The test statistic, $T_0$ , follows a $t$ distribution with $(n-2)$ degrees of freedom, where $n$ is the total number of observations. The null hypothesis, $H_0$, is accepted if the calculated value of the test statistic is such that:

$-t_{\alpha /2,n-2}<T_0<t_{\alpha /2,n-2}$

where $t_{\alpha /2,n-2}$ and $-t_{\alpha /2,n-2}$ are the critical values for the two-sided hypothesis. $t_{\alpha /2,n-2}$ is the percentile of the $t$ distribution corresponding to a cumulative probability of $(1-\alpha /2)$ and $\alpha$ is the significance level.

If the value of ${\beta }_{1,0}$ used is zero, then the hypothesis tests for the significance of regression. In other words, the test indicates if the fitted regression model is of value in explaining variations in the observations or if you are trying to impose a regression model when no true relationship exists between $x$ and $Y$. Failure to reject $H_0:{\beta }_1=0$ implies that no linear relationship exists between $x$ and $Y$. This result may be obtained when the scatter plots of against are as shown in (a) of the following figure and (b) of the following figure. (a) represents the case where no model exits for the observed data. In this case you would be trying to fit a regression model to noise or random variation. (b) represents the case where the true relationship between $x$ and $Y$ is not linear. (c) and (d) represent the case when $H_0:{\beta }_1=0$ is rejected, implying that a model does exist between $x$ and $Y$. (c) represents the case where the linear model is sufficient. In the following figure, (d) represents the case where a higher order model may be needed.

A similar procedure can be used to test the hypothesis on the intercept. The test statistic used in this case is:

$T_0=\frac{{\hat{\beta }}_0-{\beta }_{0,0}}{se({\hat{\beta }}_0)}$

where ${\hat{\beta }}_0$ is the least square estimate of ${\beta }_0$, and $se({\hat{\beta }}_0)$ is its standard error which is calculated using:

$se({\hat{\beta }}_0)=\sqrt{\frac{\sum _{i=1}^n}{e_i^2}[\frac{1}{n}+\frac{{\overline{x}}^2}{\sum _{i=1}^n}]}$

Example

The test for the significance of regression for the data in the preceding table is illustrated in this example. The test is carried out using the $t$ test on the coefficient ${\beta }_1$. The hypothesis to be tested is $H_0:{\beta }_1=0$. To calculate the statistic to test $H_0$, the estimate, ${\hat{\beta }}_1$, and the standard error, $se({\hat{\beta }}_1)$, are needed. The value of ${\hat{\beta }}_1$ was obtained in this section. The standard error can be calculated as follows:

$\begin{array}{rl}se({\hat{\beta }}_1)& =\sqrt{\frac{\frac{\sum _{i=1}^n}{e_i^2}}{\sum _{i=1}^n}}\\ =& \sqrt{\frac{(371.627/23)}{5679.36}}\\ =& 0.0533\end{array}$

Then, the test statistic can be calculated using the following equation:

$\begin{array}{rlrlr}{t}_0& =& \frac{{\hat{\beta }}_1-{\beta }_{1,0}}{se({\hat{\beta }}_0)}=& \frac{1.9952-0}{0.0533}=& 37.4058\end{array}$

The $p$ value corresponding to this statistic based on the $t$ distribution with 23 (n-2 = 25-2 = 23) degrees of freedom can be obtained as follows:

$\begin{array}{rl}pvalue=& 2\times (1-P(T\le t_0)\\ =& 2\times (1-0.999999)\\ =& 0\end{array}$

Assuming that the desired significance level is 0.1, since $p$ value < 0.1, $H_0:{\beta }_1=0$ is rejected indicating that a relation exists between temperature and yield for the data in the preceding table. Using this result along with the scatter plot, it can be concluded that the relationship between temperature and yield is linear.

In Weibull++ DOE folios, information related to the $t$ test is displayed in the Regression Information table as shown in the following figure. In this table the $t$ test for ${\beta }_1$ is displayed in the row for the term Temperature because ${\beta }_1$ is the coefficient that represents the variable temperature in the regression model. The columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the test and the $p$ value for the $t$ test, respectively. These values have been calculated for ${\beta }_1$ in this example. The Coefficient column represents the estimate of regression coefficients. The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Two Level Factorial Experiments. Columns Low Confidence and High Confidence represent the limits of the confidence intervals for the regression coefficients and are explained in Confidence Interval on Regression Coefficients.

Analysis of Variance Approach to Test the Significance of Regression

The analysis of variance (ANOVA) is another method to test for the significance of regression. As the name implies, this approach uses the variance of the observed data to determine if a regression model can be applied to the observed data. The observed variance is partitioned into components that are then used in the test for significance of regression.

Sum of Squares

The total variance (i.e., the variance of all of the observed data) is estimated using the observed data. As mentioned in Statistical Background, the variance of a population can be estimated using the sample variance, which is calculated using the following relationship:

$s^2=\frac{\underset{i=1}{\sum ^n}{(y_i-\overline{y})}^2}{n-1}$

The quantity in the numerator of the previous equation is called the sum of squares. It is the sum of the square of deviations of all the observations, $y_i$, from their mean, $\overline{y}$. In the context of ANOVA this quantity is called the total sum of squares (abbreviated $S{S}_T$) because it relates to the total variance of the observations. Thus:

$S{S}_T=\underset{i=1}{\sum ^n}{(y_i-\overline{y})}^2$

The denominator in the relationship of the sample variance is the number of degrees of freedom associated with the sample variance. Therefore, the number of degrees of freedom associated with $S{S}_T$, $dof(S{S}_T)$, is $n-1$. The sample variance is also referred to as a mean square because it is obtained by dividing the sum of squares by the respective degrees of freedom. Therefore, the total mean square (abbreviated $M{S}_T$) is:

$M{S}_T=\frac{S{S}_T}{dof(S{S}_T)}=\frac{S{S}_T}{n-1}$

When you attempt to fit a regression model to the observations, you are trying to explain some of the variation of the observations using this model. If the regression model is such that the resulting fitted regression line passes through all of the observations, then you would have a "perfect" model (see (a) of the figure below). In this case the model would explain all of the variability of the observations. Therefore, the model sum of squares (also referred to as the regression sum of squares and abbreviated $S{S}_R$) equals the total sum of squares; i.e., the model explains all of the observed variance:

$S{S}_R=S{S}_T$

For the perfect model, the regression sum of squares, $S{S}_R$, equals the total sum of squares, $S{S}_T$, because all estimated values, ${\hat{y}}_i$, will equal the corresponding observations, $y_i$. $S{S}_R$ can be calculated using a relationship similar to the one for obtaining $S{S}_T$ by replacing $y_i$ by ${\hat{y}}_i$ in the relationship of $S{S}_T$. Therefore:

$S{S}_R=\underset{i=1}{\sum ^n}{({\hat{y}}_i-\overline{y})}^2$

The number of degrees of freedom associated with $S{S}_R$ is 1.

Based on the preceding discussion of ANOVA, a perfect regression model exists when the fitted regression line passes through all observed points. However, this is not usually the case, as seen in (b) of the following figure.

In both of these plots, a number of points do not follow the fitted regression line. This indicates that a part of the total variability of the observed data still remains unexplained. This portion of the total variability or the total sum of squares, that is not explained by the model, is called the residual sum of squares or the error sum of squares (abbreviated $S{S}_E$). The deviation for this sum of squares is obtained at each observation in the form of the residuals, $e_i$. The error sum of squares can be obtained as the sum of squares of these deviations:

$S{S}_E=\underset{i=1}{\sum ^n}{e}_i^2=\underset{i=1}{\sum ^n}{(y_i-{\hat{y}}_i)}^2$

The number of degrees of freedom associated with $S{S}_E$, $dof(S{S}_E)$, is $(n-2)$. The total variability of the observed data (i.e., total sum of squares, $S{S}_T$) can be written using the portion of the variability explained by the model, $S{S}_R$, and the portion unexplained by the model, $S{S}_E$, as:

$S{S}_T=S{S}_R+S{S}_E$

The above equation is also referred to as the analysis of variance identity and can be expanded as follows:

$\underset{i=1}{\sum ^n}{(y_i-\overline{y})}^2=\underset{i=1}{\sum ^n}{({\hat{y}}_i-\overline{y})}^2+\underset{i=1}{\sum ^n}{(y_i-{\hat{y}}_i)}^2$

Mean Squares

As mentioned previously, mean squares are obtained by dividing the sum of squares by the respective degrees of freedom. For example, the error mean square, $M{S}_E$, can be obtained as:

$M{S}_E=\frac{S{S}_E}{dof(S{S}_E)}=\frac{S{S}_E}{n-2}$

The error mean square is an estimate of the variance, ${\sigma }^2$, of the random error term, $ϵ$, and can be written as:

${\hat{\sigma }}^2=\frac{S{S}_E}{n-2}$

Similarly, the regression mean square, $M{S}_R$, can be obtained by dividing the regression sum of squares by the respective degrees of freedom as follows:

$M{S}_R=\frac{S{S}_R}{dof(S{S}_R)}=\frac{S{S}_R}{1}$

F Test

To test the hypothesis $H_0:{\beta }_1=0$, the statistic used is based on the $F$ distribution. It can be shown that if the null hypothesis $H_0$ is true, then the statistic:

$F_0=\frac{M{S}_R}{M{S}_E}=\frac{S{S}_R/1}{S{S}_E/(n-2)}$

follows the $F$ distribution with $1$ degree of freedom in the numerator and $(n-2)$ degrees of freedom in the denominator. $H_0$ is rejected if the calculated statistic, $F_0$, is such that:

$F_0>f_{\alpha ,1,n-2}$

where $f_{\alpha ,1,n-2}$ is the percentile of the $F$ distribution corresponding to a cumulative probability of ($1-\alpha$) and $\alpha$ is the significance level.

Example

The analysis of variance approach to test the significance of regression can be applied to the yield data in the preceding table. To calculate the statistic, $F_0$, for the test, the sum of squares have to be obtained. The sum of squares can be calculated as shown next. The total sum of squares can be calculated as:

$\begin{array}{rl}S{S}_T=& \underset{i=1}{\sum ^n}{(y_i-\overline{y})}^2\\ =& \underset{i=1}{\sum ^{25}}{(y_i-166.32)}^2\\ =& 22979.44\end{array}$

The regression sum of squares can be calculated as:

$\begin{array}{rl}S{S}_R=& \underset{i=1}{\sum ^n}{({\hat{y}}_i-\overline{y})}^2\\ =& \underset{i=1}{\sum ^{25}}{({\hat{y}}_i-166.32)}^2\\ =& 22607.81\end{array}$

The error sum of squares can be calculated as:

$\begin{array}{rl}S{S}_E=& \underset{i=1}{\sum ^n}{(y_i-{\hat{y}}_i)}^2\\ =& \underset{i=1}{\sum ^{25}}{(y_i-{\hat{y}}_i)}^2\\ =& 371.63\end{array}$

Knowing the sum of squares, the statistic to test $H_0:{\beta }_1=0$ can be calculated as follows:

$\begin{array}{rl}{f}_0=& \frac{M{S}_R}{M{S}_E}\\ =& \frac{S{S}_R/1}{S{S}_E/(n-2)}\\ =& \frac{22607.81/1}{371.63/(25-2)}\\ =& 1399.20\end{array}$

The critical value at a significance level of 0.1 is $f_{0.05,1,23}=2.937$. Since $f_0>f_{\alpha ,1,n-2}$, $H_0:{\beta }_1=0$ is rejected and it is concluded that ${\beta }_1$ is not zero. Alternatively, the $p$ value can also be used. The $p$ value corresponding to the test statistic, $f_0$, based on the $F$ distribution with one degree of freedom in the numerator and 23 degrees of freedom in the denominator is:

$\begin{array}{rl}pvalue=& 1-P(F\le f_0)\\ =& 1-0.999999\\ =& 4.17E-22\end{array}$

Assuming that the desired significance is 0.1, since the $p$ value < 0.1, then $H_0:{\beta }_1=0$ is rejected, implying that a relation does exist between temperature and yield for the data in the preceding table. Using this result along with the scatter plot of the above figure, it can be concluded that the relationship that exists between temperature and yield is linear. This result is displayed in the ANOVA table as shown in the following figure. Note that this is the same result that was obtained from the $t$ test in the section t Tests. The ANOVA and Regression Information tables in Weibull++ DOE folios represent two different ways to test for the significance of the regression model. In the case of multiple linear regression models these tables are expanded to allow tests on individual variables used in the model. This is done using extra sum of squares. Multiple linear regression models and the application of extra sum of squares in the analysis of these models are discussed in Multiple Linear Regression Analysis.

Confidence Intervals in Simple Linear Regression

A confidence interval represents a closed interval where a certain percentage of the population is likely to lie. For example, a 90% confidence interval with a lower limit of $A$ and an upper limit of $B$ implies that 90% of the population lies between the values of $A$ and $B$. Out of the remaining 10% of the population, 5% is less than $A$ and 5% is greater than $B$. (For details refer to the Life data analysis reference.) This section discusses confidence intervals used in simple linear regression analysis.

Confidence Interval on Regression Coefficients

A 100 ($1-\alpha$) percent confidence interval on ${\beta }_1$ is obtained as follows:

${\hat{\beta }}_1\pm t_{\alpha /2,n-2}\cdot se({\hat{\beta }}_1)$

Similarly, a 100 ($1-\alpha$) percent confidence interval on ${\beta }_0$ is obtained as:

${\hat{\beta }}_0\pm t_{\alpha /2,n-2}\cdot se({\hat{\beta }}_0)$

Confidence Interval on Fitted Values

A 100 ($1-\alpha$) percent confidence interval on any fitted value, ${\hat{y}}_i$, is obtained as follows:

${\hat{y}}_i\pm t_{\alpha /2,n-2}\sqrt{{\hat{\sigma }}^2[\frac{1}{n}+\frac{{(x_i-\overline{x})}^2}{\underset{i=1}{\sum ^n}{(x_i-\overline{x})}^2}]}$

It can be seen that the width of the confidence interval depends on the value of $x_i$ and will be a minimum at $x_i=\overline{x}$ and will widen as $|x_i-\overline{x}|$ increases.

Confidence Interval on New Observations

For the data in the preceding table, assume that a new value of the yield is observed after the regression model is fit to the data. This new observation is independent of the observations used to obtain the regression model. If $x_p$ is the level of the temperature at which the new observation was taken, then the estimate for this new value based on the fitted regression model is:

$\begin{array}{rl}{\hat{y}}_p=& {\hat{\beta }}_0+{\hat{\beta }}_1{x}_p\\ =& 17.0016+1.9952\times x_p\end{array}$

If a confidence interval needs to be obtained on ${\hat{y}}_p$, then this interval should include both the error from the fitted model and the error associated with future observations. This is because ${\hat{y}}_p$ represents the estimate for a value of $Y$ that was not used to obtain the regression model. The confidence interval on ${\hat{y}}_p$ is referred to as the prediction interval. A 100 ($1-\alpha$) percent prediction interval on a new observation is obtained as follows:

${\hat{y}}_p\pm t_{\alpha /2,n-2}\sqrt{{\hat{\sigma }}^2[1+\frac{1}{n}+\frac{{(x_p-\overline{x})}^2}{\underset{i=1}{\sum ^n}{(x_p-\overline{x})}^2}]}$

Example

To illustrate the calculation of confidence intervals, the 95% confidence intervals on the response at $x=93$ for the data in the preceding table is obtained in this example. A 95% prediction interval is also obtained assuming that a new observation for the yield was made at $x=91$.

The fitted value, ${\hat{y}}_i$, corresponding to $x=93$ is:

$\begin{array}{rl}{\hat{y}}_{21}=& {\hat{\beta }}_0+{\hat{\beta }}_1{x}_{21}\\ =& 17.0016+1.9952\times 93\\ =& 202.6\end{array}$

The 95% confidence interval $(\alpha =0.05)$ on the fitted value, ${\hat{y}}_{21}=202.6$, is:

$\begin{array}{rl}=& {\hat{y}}_i\pm t_{\alpha /2,n-2}\sqrt{{\hat{\sigma }}^2[\frac{1}{n}+\frac{{(x_i-\overline{x})}^2}{\underset{i=1}{\sum ^n}{(x_i-\overline{x})}^2}]}\\ =& 202.6\pm t_{0.025,23}\sqrt{M{S}_E[\frac{1}{25}+\frac{{(93-74.84)}^2}{5679.36}]}\\ =& 202.6\pm 2.069\sqrt{16.16[\frac{1}{25}+\frac{{(93-74.84)}^2}{5679.36}]}\\ =& 202.6\pm 2.602\end{array}$

The 95% limits on ${\hat{y}}_{21}$ are 199.95 and 205.2, respectively. The estimated value based on the fitted regression model for the new observation at $x=91$ is:

$\begin{array}{rl}{\hat{y}}_p=& {\hat{\beta }}_0+{\hat{\beta }}_1{x}_p\\ =& 17.0016+1.9952\times 91\\ =& 198.6\end{array}$

The 95% prediction interval on ${\hat{y}}_p=198.6$ is:

$\begin{array}{rl}=& {\hat{y}}_p\pm t_{\alpha /2,n-2}\sqrt{{\hat{\sigma }}^2[1+\frac{1}{n}+\frac{{(x_p-\overline{x})}^2}{\underset{i=1}{\sum ^n}{(x_p-\overline{x})}^2}]}\\ =& 198.6\pm t_{0.025,23}\sqrt{M{S}_E[1+\frac{1}{25}+\frac{{(93-74.84)}^2}{5679.36}]}\\ =& 198.6\pm 2.069\sqrt{16.16[1+\frac{1}{25}+\frac{{(93-74.84)}^2}{5679.36}]}\\ =& 198.6\pm 2.069\times 4.1889\\ =& 198.6\pm 8.67\end{array}$

The 95% limits on ${\hat{y}}_p$ are 189.9 and 207.2, respectively. In Weibull++ DOE folios, confidence and prediction intervals can be calculated from the control panel. The prediction interval values calculated in this example are shown in the figure below as Low Prediction Interval and High Prediction Interval, respectively. The columns labeled Mean Predicted and Standard Error represent the values of ${\hat{y}}_p$ and the standard error used in the calculations.

Measures of Model Adequacy

It is important to analyze the regression model before inferences based on the model are undertaken. The following sections present some techniques that can be used to check the appropriateness of the model for the given data. These techniques help to determine if any of the model assumptions have been violated.

Coefficient of Determination ($R^2$)

The coefficient of determination is a measure of the amount of variability in the data accounted for by the regression model. As mentioned previously, the total variability of the data is measured by the total sum of squares, $S{S}_T$. The amount of this variability explained by the regression model is the regression sum of squares, $S{S}_R$. The coefficient of determination is the ratio of the regression sum of squares to the total sum of squares.

$R^2=\frac{S{S}_R}{S{S}_T}$

$R^2$ can take on values between 0 and 1 since $R^2=\frac{S{S}_R}{S{S}_T}$ . For the yield data example, $R^2$ can be calculated as:

$\begin{array}{rl}{R}^2=& \frac{S{S}_R}{S{S}_T}\\ =& \frac{22607.81}{22979.44}\\ =& 0.98\end{array}$

Therefore, 98% of the variability in the yield data is explained by the regression model, indicating a very good fit of the model. It may appear that larger values of $R^2$ indicate a better fitting regression model. However, $R^2$ should be used cautiously as this is not always the case. The value of $R^2$ increases as more terms are added to the model, even if the new term does not contribute significantly to the model. Therefore, an increase in the value of $R^2$ cannot be taken as a sign to conclude that the new model is superior to the older model. Adding a new term may make the regression model worse if the error mean square, $M{S}_E$, for the new model is larger than the $M{S}_E$ of the older model, even though the new model will show an increased value of $R^2$. In the results obtained from the DOE folio, $R^2$ is displayed as R-sq under the ANOVA table (as shown in the figure below), which displays the complete analysis sheet for the data in the preceding table.

The other values displayed with are S, R-sq(adj), PRESS and R-sq(pred). These values measure different aspects of the adequacy of the regression model. For example, the value of S is the square root of the error mean square, $M{S}_E$, and represents the "standard error of the model." A lower value of S indicates a better fitting model. The values of S, R-sq and R-sq(adj) indicate how well the model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. R-sq(adj), PRESS and R-sq(pred) are explained in Multiple Linear Regression Analysis.

Residual Analysis

In the simple linear regression model the true error terms, ${ϵ}_i$, are never known. The residuals, $e_i$, may be thought of as the observed error terms that are similar to the true error terms. Since the true error terms, ${ϵ}_i$, are assumed to be normally distributed with a mean of zero and a variance of ${\sigma }^2$, in a good model the observed error terms (i.e., the residuals, $e_i$) should also follow these assumptions. Thus the residuals in the simple linear regression should be normally distributed with a mean of zero and a constant variance of ${\sigma }^2$. Residuals are usually plotted against the fitted values, ${\hat{y}}_i$, against the predictor variable values, $x_i$, and against time or run-order sequence, in addition to the normal probability plot. Plots of residuals are used to check for the following:

1. Residuals follow the normal distribution.

2. Residuals have a constant variance.

3. Regression function is linear.

4. A pattern does not exist when residuals are plotted in a time or run-order sequence.

5. There are no outliers.

Examples of residual plots are shown in the following figure. (a) is a satisfactory plot with the residuals falling in a horizontal band with no systematic pattern. Such a plot indicates an appropriate regression model. (b) shows residuals falling in a funnel shape. Such a plot indicates increase in variance of residuals and the assumption of constant variance is violated here. Transformation on $Y$ may be helpful in this case (see Transformations). If the residuals follow the pattern of (c) or (d), then this is an indication that the linear regression model is not adequate. Addition of higher order terms to the regression model or transformation on $x$ or $Y$ may be required in such cases. A plot of residuals may also show a pattern as seen in (e), indicating that the residuals increase (or decrease) as the run order sequence or time progresses. This may be due to factors such as operator-learning or instrument-creep and should be investigated further.

Example

Residual plots for the data of the preceding table are shown in the following figures. One of the following figures is the normal probability plot. It can be observed that the residuals follow the normal distribution and the assumption of normality is valid here. In one of the following figures the residuals are plotted against the fitted values, ${\hat{y}}_i$, and in one of the following figures the residuals are plotted against the run order. Both of these plots show that the 21st observation seems to be an outlier. Further investigations are needed to study the cause of this outlier.

Lack-of-Fit Test

As mentioned in Analysis of Variance Approach, ANOVA, a perfect regression model results in a fitted line that passes exactly through all observed data points. This perfect model will give us a zero error sum of squares ($S{S}_E=0$). Thus, no error exists for the perfect model. However, if you record the response values for the same values of $x_i$ for a second time, in conditions maintained as strictly identical as possible to the first time, observations from the second time will not all fall along the perfect model. The deviations in observations recorded for the second time constitute the "purely" random variation or noise. The sum of squares due to pure error (abbreviated $S{S}_{PE}$) quantifies these variations. $S{S}_{PE}$ is calculated by taking repeated observations at some or all values of $x_i$ and adding up the square of deviations at each level of $x$ using the respective repeated observations at that $x$ value.

Assume that there are $n$ levels of $x$ and $m_i$ repeated observations are taken at each $i$ the level. The data is collected as shown next:

$\begin{array}{rl}& y_{11},y_{12},....,y_{1m_1}\text{ repeated observations at }{x}_1\\ & y_{21},y_{22},....,y_{2m_2}\text{ repeated observations at }{x}_2\\ & ...\\ & y_{i1},y_{i2},....,y_{i{m}_i}\text{ repeated observations at }{x}_i\\ & ...\\ & y_{n1},y_{n2},....,y_{n{m}_n}\text{ repeated observations at }{x}_n\end{array}$

The sum of squares of the deviations from the mean of the observations at $i$ the level of $x$, $x_i$, can be calculated as:

$\underset{j=1}{\sum ^{m_i}}{(y_{ij}-{\overline{y}}_i)}^2$

where ${\overline{y}}_i$ is the mean of the $m_i$ repeated observations corresponding to $x_i$ (${\overline{y}}_i=(1/m_i)\underset{j=1}{\overset{m_i}{}}{y}_{ij}$). The number of degrees of freedom for these deviations is ($m_i-1$ ) as there are $m_i$ observations at $i$ the level of $x$ but one degree of freedom is lost in calculating the mean, ${\overline{y}}_i$.

The total sum of square deviations (or $S{S}_{PE}$) for all levels of $x$ can be obtained by summing the deviations for all $x_i$ as shown next:

$S{S}_{PE}=\underset{i=1}{\sum ^n}\underset{j=1}{\sum ^{m_i}}{(y_{ij}-{\overline{y}}_i)}^2$

The total number of degrees of freedom associated with $S{S}_{PE}$ is:

$\begin{array}{rl}=& \underset{i=1}{\sum ^n}(m_i-1)\\ =& \underset{i=1}{\sum ^n}{m}_i-n\end{array}$

If all $m_i=m$, (i.e., $m$ repeated observations are taken at all levels of $x$), then $\underset{i=1}{\overset{n}{}}{m}_i=nm$ and the degrees of freedom associated with $S{S}_{PE}$ are:

$=nm-n$

The corresponding mean square in this case will be:

$M{S}_{PE}=\frac{S{S}_{PE}}{nm-n}$

When repeated observations are used for a perfect regression model, the sum of squares due to pure error, $S{S}_{PE}$, is also considered as the error sum of squares, $S{S}_E$. For the case when repeated observations are used with imperfect regression models, there are two components of the error sum of squares, $S{S}_E$. One portion is the pure error due to the repeated observations. The other portion is the error that represents variation not captured because of the imperfect model. The second portion is termed as the sum of squares due to lack-of-fit (abbreviated $S{S}_{LOF}$) to point to the deficiency in fit due to departure from the perfect-fit model. Thus, for an imperfect regression model:

$S{S}_E=S{S}_{PE}+S{S}_{LOF}$

Knowing $S{S}_E$ and $S{S}_{PE}$, the previous equation can be used to obtain $S{S}_{LOF}$:

$S{S}_{LOF}=S{S}_E-S{S}_{PE}$

The degrees of freedom associated with $S{S}_{LOF}$ can be obtained in a similar manner using subtraction. For the case when $m$ repeated observations are taken at all levels of $x$, the number of degrees of freedom associated with $S{S}_{PE}$ is:

$dof(S{S}_{PE})=nm-n$

Since there are $nm$ total observations, the number of degrees of freedom associated with $S{S}_E$ is:

$dof(S{S}_E)=nm-2$

Therefore, the number of degrees of freedom associated with $S{S}_{LOF}$ is:

$\begin{array}{rl}=& dof(S{S}_E)-dof(S{S}_{PE})\\ =& (nm-2)-(nm-n)\\ =& n-2\end{array}$

The corresponding mean square, $M{S}_{LOF}$, can now be obtained as:

$M{S}_{LOF}=\frac{S{S}_{LOF}}{n-2}$

The magnitude of $S{S}_{LOF}$ or $M{S}_{LOF}$ will provide an indication of how far the regression model is from the perfect model. An $F$ test exists to examine the lack-of-fit at a particular significance level. The quantity $M{S}_{LOF}/M{S}_{PE}$ follows an $F$ distribution with $(n-2)$ degrees of freedom in the numerator and $(nm-n)$ degrees of freedom in the denominator when all $m_i$ equal $m$. The test statistic for the lack-of-fit test is:

$F_0=\frac{M{S}_{LOF}}{M{S}_{PE}}$

If the critical value $f_{\alpha ,n-2,mn-n}$ is such that:

$F_0>f_{\alpha ,n-2,nm-n}$

it will lead to the rejection of the hypothesis that the model adequately fits the data.

Example

Assume that a second set of observations are taken for the yield data of the preceding table. The resulting observations are recorded in the following table. To conduct a lack-of-fit test on this data, the statistic $F_0=M{S}_{LOF}/M{S}_{PE}$, can be calculated as shown next.

Calculation of Least Square Estimates

The parameters of the fitted regression model can be obtained as:

$\begin{array}{rl}{\hat{\beta }}_1=& \frac{\underset{i=1}{\sum ^{50}}{y}_i{x}_i-\frac{\left(\underset{i=1}{\sum ^{50}}{y}_i\right)\left(\underset{i=1}{\sum ^{50}}{x}_i\right)}{50}}{\underset{i=1}{\sum ^{50}}{(x_i-\overline{x})}^2}\\ =& \frac{648532-\frac{8356\times 3742}{50}}{11358.72}\\ =& 2.04\end{array}$

$\begin{array}{rl}{\hat{\beta }}_0=& \overline{y}-{\hat{\beta }}_1\overline{x}\\ =& 167.12-2.04\times 74.84\\ =& 14.47\end{array}$

Knowing ${\hat{\beta }}_1$ and ${\hat{\beta }}_0$, the fitted values, ${\hat{y}}_i$, can be calculated.

Calculation of the Sum of Squares

Using the fitted values, the sum of squares can be obtained as follows:

$\begin{array}{rl}S{S}_T=& \underset{i=1}{\sum ^{50}}{(y_i-\overline{y})}^2\\ =& 47907.28\end{array}$

$\begin{array}{rl}S{S}_R=& \underset{i=1}{\sum ^{50}}{({\hat{y}}_i-\overline{y})}^2\\ =& 47258.91\end{array}$

$\begin{array}{rl}S{S}_E=& \underset{i=1}{\sum ^{50}}{(y_i-{\hat{y}}_i)}^2\\ =& 648.37\end{array}$

Calculation of $M{S}_{LOF}$

The error sum of squares, $S{S}_E$, can now be split into the sum of squares due to pure error, $S{S}_{PE}$, and the sum of squares due to lack-of-fit, $S{S}_{LOF}$. $S{S}_{PE}$ can be calculated as follows considering that in this example $n=25$ and $m=2$:

$\begin{array}{rl}S{S}_{PE}& =\underset{i=1}{\sum ^n}\underset{j=1}{\sum ^{m_i}}{(y_{ij}-{\overline{y}}_i)}^2\\ & =\underset{i=1}{\sum ^{25}}\underset{j=1}{\sum ^2}{(y_{ij}-{\overline{y}}_i)}^2\\ & =350\end{array}$

The number of degrees of freedom associated with $S{S}_{PE}$ is:

$\begin{array}{rl}dof(S{S}_{PE})& =nm-n\\ & =25\times 2-25\\ & =25\end{array}$

The corresponding mean square, $M{S}_{PE}$, can now be obtained as:

$\begin{array}{rl}M{S}_{PE}& =\frac{S{S}_{PE}}{dof(S{S}_{PE})}\\ & =\frac{350}{25}\\ & =14\end{array}$

$S{S}_{LOF}$ can be obtained by subtraction from $S{S}_E$ as:

$\begin{array}{rl}S{S}_{LOF}& =S{S}_E-S{S}_{PE}\\ & =648.37-350\\ & =298.37\end{array}$

Similarly, the number of degrees of freedom associated with $S{S}_{LOF}$ is:

$\begin{array}{rl}dof(S{S}_{LOF})& =dof(S{S}_E)-dof(S{S}_{PE})\\ & =(nm-2)-(nm-n)\\ & =23\end{array}$

The lack-of-fit mean square is:

$\begin{array}{rl}M{S}_{LOF}& =\frac{M{S}_{LOF}}{dof(M{S}_{LOF})}\\ & =\frac{298.37}{23}\\ & =12.97\end{array}$

Calculation of the Test Statistic

The test statistic for the lack-of-fit test can now be calculated as:

$\begin{array}{rl}{f}_0& =\frac{M{S}_{LOF}}{M{S}_{PE}}\\ & =\frac{12.97}{14}\\ & =0.93\end{array}$

The critical value for this test is:

$f_{0.05,23,25}=1.97$

Since $f_0<f_{0.05,23,25}$, we fail to reject the hypothesis that the model adequately fits the data. The $p$ value for this case is:

$\begin{array}{rl}pvalue& =1-P(F\le f_0)\\ & =1-0.43\\ & =0.57\end{array}$

Therefore, at a significance level of 0.05 we conclude that the simple linear regression model, $y=14.47+2.04x$, is adequate for the observed data. The following table presents a summary of the ANOVA calculations for the lack-of-fit test.

Transformations

The linear regression model may not be directly applicable to certain data. Non-linearity may be detected from scatter plots or may be known through the underlying theory of the product or process or from past experience. Transformations on either the predictor variable, $x$, or the response variable, $Y$, may often be sufficient to make the linear regression model appropriate for the transformed data. If it is known that the data follows the logarithmic distribution, then a logarithmic transformation on $Y$ (i.e., $Y^{\ast }=\log (Y)$) might be useful. For data following the Poisson distribution, a square root transformation ($Y^{\ast }=\sqrt{Y}$) is generally applicable.

Transformations on $Y$ may also be applied based on the type of scatter plot obtained from the data. The following figure shows a few such examples.

For the scatter plot labeled (a), a square root transformation ($Y^{\ast }=\sqrt{Y}$) is applicable. While for the plot labeled (b), a logarithmic transformation (i.e., $Y^{\ast }=\log (Y)$) may be applied. For the plot labeled (c), the reciprocal transformation ($Y^{\ast }=1/Y$) is applicable. At times it may be helpful to introduce a constant into the transformation of $Y$. For example, if $Y$ is negative and the logarithmic transformation on $Y$ seems applicable, a suitable constant, $k$, may be chosen to make all observed $Y$ positive. Thus the transformation in this case would be $Y^{\ast }=\log (k+Y)$ .

The Box-Cox method may also be used to automatically identify a suitable power transformation for the data based on the relation:

$Y^{\ast }=Y^{\lambda }$

Here the parameter $\lambda$ is determined using the given data such that $S{S}_E$ is minimized (details on this method are presented in One Factor Designs).