Multiple Linear Regression Analysis

This chapter expands on the analysis of simple linear regression models and discusses the analysis of multiple linear regression models. A major portion of the results displayed in Weibull++ DOE folios are explained in this chapter because these results are associated with multiple linear regression. One of the applications of multiple linear regression models is Response Surface Methodology (RSM). RSM is a method used to locate the optimum value of the response and is one of the final stages of experimentation. It is discussed in Response Surface Methods. Towards the end of this chapter, the concept of using indicator variables in regression models is explained. Indicator variables are used to represent qualitative factors in regression models. The concept of using indicator variables is important to gain an understanding of ANOVA models, which are the models used to analyze data obtained from experiments. These models can be thought of as first order multiple linear regression models where all the factors are treated as qualitative factors. ANOVA models are discussed in the One Factor Designs and General Full Factorial Designs chapters.

Multiple Linear Regression Model

A linear regression model that contains more than one predictor variable is called a multiple linear regression model. The following model is a multiple linear regression model with two predictor variables, $x_1$ and $x_2$.

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$

The model is linear because it is linear in the parameters ${\beta }_0$, ${\beta }_1$ and ${\beta }_2$. The model describes a plane in the three-dimensional space of $Y$, $x_1$ and $x_2$. The parameter ${\beta }_0$ is the intercept of this plane. Parameters ${\beta }_1$ and ${\beta }_2$ are referred to as partial regression coefficients. Parameter ${\beta }_1$ represents the change in the mean response corresponding to a unit change in $x_1$ when $x_2$ is held constant. Parameter ${\beta }_2$ represents the change in the mean response corresponding to a unit change in $x_2$ when $x_1$ is held constant. Consider the following example of a multiple linear regression model with two predictor variables, $x_1$ and $x_2$ :

$Y=30+5x_1+7x_2+ϵ$

This regression model is a first order multiple linear regression model. This is because the maximum power of the variables in the model is 1. (The regression plane corresponding to this model is shown in the figure below.) Also shown is an observed data point and the corresponding random error, $ϵ$. The true regression model is usually never known (and therefore the values of the random error terms corresponding to observed data points remain unknown). However, the regression model can be estimated by calculating the parameters of the model for an observed data set. This is explained in Estimating Regression Models Using Least Squares.

One of the following figures shows the contour plot for the regression model the above equation. The contour plot shows lines of constant mean response values as a function of $x_1$ and $x_2$. The contour lines for the given regression model are straight lines as seen on the plot. Straight contour lines result for first order regression models with no interaction terms.

A linear regression model may also take the following form:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_{12}{x}_1{x}_2+ϵ$

A cross-product term, $x_1{x}_2$, is included in the model. This term represents an interaction effect between the two variables $x_1$ and $x_2$. Interaction means that the effect produced by a change in the predictor variable on the response depends on the level of the other predictor variable(s). As an example of a linear regression model with interaction, consider the model given by the equation $Y=30+5x_1+7x_2+3x_1{x}_2+ϵ$. The regression plane and contour plot for this model are shown in the following two figures, respectively.

Now consider the regression model shown next:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_1^2+{\beta }_3{x}_1^3+ϵ$

This model is also a linear regression model and is referred to as a polynomial regression model. Polynomial regression models contain squared and higher order terms of the predictor variables making the response surface curvilinear. As an example of a polynomial regression model with an interaction term consider the following equation:

$Y=500+5x_1+7x_2-3x_1^2-5x_2^2+3x_1{x}_2+ϵ$

This model is a second order model because the maximum power of the terms in the model is two. The regression surface for this model is shown in the following figure. Such regression models are used in RSM to find the optimum value of the response, $Y$ (for details see Response Surface Methods for Optimization). Notice that, although the shape of the regression surface is curvilinear, the regression model is still linear because the model is linear in the parameters. The contour plot for this model is shown in the second of the following two figures.

All multiple linear regression models can be expressed in the following general form:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+...+{\beta }_k{x}_k+ϵ$

where $k$ denotes the number of terms in the model. For example, the model can be written in the general form using $x_3=x_1^2$, $x_4=x_2^3$ and $x_5=x_1{x}_2$ as follows:

$Y=500+5x_1+7x_2-3x_3-5x_4+3x_5+ϵ$

Estimating Regression Models Using Least Squares

Consider a multiple linear regression model with $k$ predictor variables:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+...+{\beta }_k{x}_k+ϵ$

Let each of the $k$ predictor variables, $x_1$, $x_2$... $x_k$, have $n$ levels. Then $x_{ij}$ represents the $i$ th level of the $j$ th predictor variable $x_j$. For example, $x_{51}$ represents the fifth level of the first predictor variable $x_1$, while $x_{19}$ represents the first level of the ninth predictor variable, $x_9$. Observations, $y_1$, $y_2$... $y_n$, recorded for each of these $n$ levels can be expressed in the following way:

$\begin{array}{rl}{y}_1=& {\beta }_0+{\beta }_1{x}_{11}+{\beta }_2{x}_{12}+...+{\beta }_k{x}_{1k}+{ϵ}_1\\ y_2=& {\beta }_0+{\beta }_1{x}_{21}+{\beta }_2{x}_{22}+...+{\beta }_k{x}_{2k}+{ϵ}_2\\ & ..\\ y_i=& {\beta }_0+{\beta }_1{x}_{i1}+{\beta }_2{x}_{i2}+...+{\beta }_k{x}_{ik}+{ϵ}_i\\ & ..\\ y_n=& {\beta }_0+{\beta }_1{x}_{n1}+{\beta }_2{x}_{n2}+...+{\beta }_k{x}_{nk}+{ϵ}_n\end{array}$

The system of $n$ equations shown previously can be represented in matrix notation as follows:

$y=X\beta +ϵ$

where

$y=\left[\begin{array}{c}{y}_1\\ y_2\\ .\\ .\\ .\\ y_n\end{array}\right]X=\left[\begin{array}{ccccccc}1& x_{11}& x_{12}& .& .& .& x_{1n}\\ 1& x_{21}& x_{22}& .& .& .& x_{2n}\\ .& .& .& & & & .\\ .& .& .& & & & .\\ .& .& .& & & & .\\ 1& x_{n1}& x_{n2}& .& .& .& x_{nn}\end{array}\right]$

$\beta =\left[\begin{array}{c}{\beta }_0\\ {\beta }_1\\ .\\ .\\ .\\ {\beta }_n\end{array}\right]\text{ and }ϵ=\left[\begin{array}{c}{ϵ}_1\\ {ϵ}_2\\ .\\ .\\ .\\ {ϵ}_n\end{array}\right]$

The matrix $X$ is referred to as the design matrix. It contains information about the levels of the predictor variables at which the observations are obtained. The vector $\beta$ contains all the regression coefficients. To obtain the regression model, $\beta$ should be known. $\beta$ is estimated using least square estimates. The following equation is used:

$\hat{\beta }={(X^{\prime }X)}^{-1}{X}^{\prime }y$

where ${}^{\prime }$ represents the transpose of the matrix while ${}^{-1}$ represents the matrix inverse. Knowing the estimates, $\hat{\beta }$, the multiple linear regression model can now be estimated as:

$\hat{y}=X\hat{\beta }$

The estimated regression model is also referred to as the fitted model. The observations, $y_i$, may be different from the fitted values ${\hat{y}}_i$ obtained from this model. The difference between these two values is the residual, $e_i$. The vector of residuals, $e$, is obtained as:

$e=y-\hat{y}$

The fitted model can also be written as follows, using $\hat{\beta }={(X^{\prime }X)}^{-1}{X}^{\prime }y$:

$\begin{array}{rlr}\hat{y}& =& X\hat{\beta }\\ & =& X{(X^{\prime }X)}^{-1}{X}^{\prime }y\\ & =& Hy\end{array}$

where $H=X{(X^{\prime }X)}^{-1}{X}^{\prime }$. The matrix, $H$, is referred to as the hat matrix. It transforms the vector of the observed response values, $y$, to the vector of fitted values, $\hat{y}$.

Example

An analyst studying a chemical process expects the yield to be affected by the levels of two factors, $x_1$ and $x_2$. Observations recorded for various levels of the two factors are shown in the following table. The analyst wants to fit a first order regression model to the data. Interaction between $x_1$ and $x_2$ is not expected based on knowledge of similar processes. Units of the factor levels and the yield are ignored for the analysis.

The data of the above table can be entered into the Weibull++ DOE folio using the multiple linear regression folio tool as shown in the following figure.

A scatter plot for the data is shown next.

The first order regression model applicable to this data set having two predictor variables is:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$

where the dependent variable, $Y$, represents the yield and the predictor variables, $x_1$ and $x_2$, represent the two factors respectively. The $X$ and $y$ matrices for the data can be obtained as:

$X=\left[\begin{array}{ccc}1& 41.9& 29.1\\ 1& 43.4& 29.3\\ .& .& .\\ .& .& .\\ .& .& .\\ 1& 77.8& 32.9\end{array}\right]y=\left[\begin{array}{c}251.3\\ 251.3\\ .\\ .\\ .\\ 349.0\end{array}\right]$

The least square estimates, $\hat{\beta }$, can now be obtained:

$\begin{array}{rlr}\hat{\beta }& =& {(X^{\prime }X)}^{-1}{X}^{\prime }y\\ & =& {\left[\begin{array}{ccc}17& 941& 525.3\\ 941& 54270& 29286\\ 525.3& 29286& 16254\end{array}\right]}^{-1}\left[\begin{array}{c}4902.8\\ 276610\\ 152020\end{array}\right]\\ & =& \left[\begin{array}{c}-153.51\\ 1.24\\ 12.08\end{array}\right]\end{array}$

Thus:

$\hat{\beta }=\left[\begin{array}{c}{\hat{\beta }}_0\\ {\hat{\beta }}_1\\ {\hat{\beta }}_2\end{array}\right]=\left[\begin{array}{c}-153.51\\ 1.24\\ 12.08\end{array}\right]$

and the estimated regression coefficients are ${\hat{\beta }}_0=-153.51$, ${\hat{\beta }}_1=1.24$ and ${\hat{\beta }}_2=12.08$. The fitted regression model is:

$\begin{array}{rlr}\hat{y}& =& {\hat{\beta }}_0+{\hat{\beta }}_1{x}_1+{\hat{\beta }}_2{x}_2\\ & =& -153.5+1.24x_1+12.08x_2\end{array}$

The fitted regression model can be viewed in the Weibull++ DOE folio, as shown next.

A plot of the fitted regression plane is shown in the following figure.

The fitted regression model can be used to obtain fitted values, ${\hat{y}}_i$, corresponding to an observed response value, $y_i$. For example, the fitted value corresponding to the fifth observation is:

$\begin{array}{rlr}{\hat{y}}_i& =& -153.5+1.24x_{i1}+12.08x_{i2}\\ {\hat{y}}_5& =& -153.5+1.24x_{51}+12.08x_{52}\\ & =& -153.5+1.24(47.3)+12.08(29.9)\\ & =& 266.3\end{array}$

The observed fifth response value is $y_5=273.0$. The residual corresponding to this value is:

$\begin{array}{rlr}{e}_i& =& y_i-{\hat{y}}_i\\ e_5& =& y_5-{\hat{y}}_5\\ & =& 273.0-266.3\\ & =& 6.7\end{array}$

In Weibull++ DOE folios, fitted values and residuals are shown in the Diagnostic Information table of the detailed summary of results. The values are shown in the following figure.

The fitted regression model can also be used to predict response values. For example, to obtain the response value for a new observation corresponding to 47 units of $x_1$ and 31 units of $x_2$, the value is calculated using:

$\begin{array}{rlr}\hat{y}(47,31)& =& -153.5+1.24(47)+12.08(31)\\ & =& 279.26\end{array}$

Properties of the Least Square Estimators for Beta

The least square estimates, ${\hat{\beta }}_0$, ${\hat{\beta }}_1$, ${\hat{\beta }}_2$... ${\hat{\beta }}_k$, are unbiased estimators of ${\beta }_0$, ${\beta }_1$, ${\beta }_2$... ${\beta }_k$, provided that the random error terms, ${ϵ}_i$, are normally and independently distributed. The variances of the $\hat{\beta }$ s are obtained using the ${(X^{\prime }X)}^{-1}$ matrix. The variance-covariance matrix of the estimated regression coefficients is obtained as follows:

$C={\hat{\sigma }}^2{(X^{\prime }X)}^{-1}$

$C$ is a symmetric matrix whose diagonal elements, $C_{jj}$, represent the variance of the estimated $j$ th regression coefficient, ${\hat{\beta }}_j$. The off-diagonal elements, $C_{ij}$, represent the covariance between the $i$ th and $j$ th estimated regression coefficients, ${\hat{\beta }}_i$ and ${\hat{\beta }}_j$. The value of ${\hat{\sigma }}^2$ is obtained using the error mean square, $M{S}_E$. The variance-covariance matrix for the data in the table (see Estimating Regression Models Using Least Squares) can be viewed in the DOE folio, as shown next.

Calculations to obtain the matrix are given in this example. The positive square root of $C_{jj}$ represents the estimated standard deviation of the $j$ th regression coefficient, ${\hat{\beta }}_j$, and is called the estimated standard error of ${\hat{\beta }}_j$ (abbreviated $se({\hat{\beta }}_j)$ ).

$se({\hat{\beta }}_j)=\sqrt{C_{jj}}$

Hypothesis Tests in Multiple Linear Regression

This section discusses hypothesis tests on the regression coefficients in multiple linear regression. As in the case of simple linear regression, these tests can only be carried out if it can be assumed that the random error terms, ${ϵ}_i$, are normally and independently distributed with a mean of zero and variance of ${\sigma }^2$. Three types of hypothesis tests can be carried out for multiple linear regression models:

1. Test for significance of regression: This test checks the significance of the whole regression model.
2. $t$ test: This test checks the significance of individual regression coefficients.
3. $F$ test: This test can be used to simultaneously check the significance of a number of regression coefficients. It can also be used to test individual coefficients.

Test for Significance of Regression

The test for significance of regression in the case of multiple linear regression analysis is carried out using the analysis of variance. The test is used to check if a linear statistical relationship exists between the response variable and at least one of the predictor variables. The statements for the hypotheses are:

$\begin{array}{rlr}& H_0:& {\beta }_1={\beta }_2=...={\beta }_k=0\\ & H_1:& {\beta }_j\ne 0\text{ for at least one }j\end{array}$

The test for $H_0$ is carried out using the following statistic:

$F_0=\frac{M{S}_R}{M{S}_E}$

where $M{S}_R$ is the regression mean square and $M{S}_E$ is the error mean square. If the null hypothesis, $H_0$, is true then the statistic $F_0$ follows the $F$ distribution with $k$ degrees of freedom in the numerator and $n-$ ( $k+1$ ) degrees of freedom in the denominator. The null hypothesis, $H_0$, is rejected if the calculated statistic, $F_0$, is such that:

$F_0>f_{\alpha ,k,n-(k+1)}$

Calculation of the Statistic $F_0$

To calculate the statistic $F_0$, the mean squares $M{S}_R$ and $M{S}_E$ must be known. As explained in Simple Linear Regression Analysis, the mean squares are obtained by dividing the sum of squares by their degrees of freedom. For example, the total mean square, $M{S}_T$, is obtained as follows:

$M{S}_T=\frac{S{S}_T}{dof(S{S}_T)}$

where $S{S}_T$ is the total sum of squares and $dof(S{S}_T)$ is the number of degrees of freedom associated with $S{S}_T$. In multiple linear regression, the following equation is used to calculate $S{S}_T$ :

$S{S}_T=y^{\prime }[I-(\frac{1}{n})J]y$

where $n$ is the total number of observations, $y$ is the vector of observations (that was defined in Estimating Regression Models Using Least Squares), $I$ is the identity matrix of order $n$ and $J$ represents an $n\times n$ square matrix of ones. The number of degrees of freedom associated with $S{S}_T$, $dof(S{S}_T)$, is ( $n-1$ ). Knowing $S{S}_T$ and $dof(S{S}_T)$ the total mean square, $M{S}_T$, can be calculated.

The regression mean square, $M{S}_R$, is obtained by dividing the regression sum of squares, $S{S}_R$, by the respective degrees of freedom, $dof(S{S}_R)$, as follows:

$M{S}_R=\frac{S{S}_R}{dof(S{S}_R)}$

The regression sum of squares, $S{S}_R$, is calculated using the following equation:

$S{S}_R=y^{\prime }[H-(\frac{1}{n})J]y$

where $n$ is the total number of observations, $y$ is the vector of observations, $H$ is the hat matrix and $J$ represents an $n\times n$ square matrix of ones. The number of degrees of freedom associated with $S{S}_R$, $dof(S{S}_E)$, is $k$, where $k$ is the number of predictor variables in the model. Knowing $S{S}_R$ and $dof(S{S}_R)$ the regression mean square, $M{S}_R$, can be calculated. The error mean square, $M{S}_E$, is obtained by dividing the error sum of squares, $S{S}_E$, by the respective degrees of freedom, $dof(S{S}_E)$, as follows:

$M{S}_E=\frac{S{S}_E}{dof(S{S}_E)}$

The error sum of squares, $S{S}_E$, is calculated using the following equation:

$S{S}_E=y^{\prime }(I-H)y$

where $y$ is the vector of observations, $I$ is the identity matrix of order $n$ and $H$ is the hat matrix. The number of degrees of freedom associated with $S{S}_E$, $dof(S{S}_E)$, is $n-(k+1)$, where $n$ is the total number of observations and $k$ is the number of predictor variables in the model. Knowing $S{S}_E$ and $dof(S{S}_E)$, the error mean square, $M{S}_E$, can be calculated. The error mean square is an estimate of the variance, ${\sigma }^2$, of the random error terms, ${ϵ}_i$.

${\hat{\sigma }}^2=M{S}_E$

Example

The test for the significance of regression, for the regression model obtained for the data in the table (see Estimating Regression Models Using Least Squares), is illustrated in this example. The null hypothesis for the model is:

$H_0:{\beta }_1={\beta }_2=0$

The statistic to test $H_0$ is:

$F_0=\frac{M{S}_R}{M{S}_E}$

To calculate $F_0$, first the sum of squares are calculated so that the mean squares can be obtained. Then the mean squares are used to calculate the statistic $F_0$ to carry out the significance test. The regression sum of squares, $S{S}_R$, can be obtained as:

$S{S}_R=y^{\prime }[H-(\frac{1}{n})J]y$

The hat matrix, $H$ is calculated as follows using the design matrix $X$ from the previous example:

$\begin{array}{rlr}H& =& X{(X^{\prime }X)}^{-1}{X}^{\prime }\\ & =& \left[\begin{array}{ccccc}0.27552& 0.25154& .& .& -0.04030\\ 0.25154& 0.23021& .& .& -0.029120\\ .& .& .& .& .\\ .& .& .& .& .\\ -0.04030& -0.02920& .& .& 0.30115\end{array}\right]\end{array}$

Knowing $y$, $H$ and $J$, the regression sum of squares, $S{S}_R$, can be calculated:

$\begin{array}{rlr}S{S}_R& =& y^{\prime }[H-(\frac{1}{n})J]y\\ & =& 12816.35\end{array}$

The degrees of freedom associated with $S{S}_R$ is $k$, which equals to a value of two since there are two predictor variables in the data in the table (see Multiple Linear Regression Analysis). Therefore, the regression mean square is:

$\begin{array}{rlr}M{S}_R& =& \frac{S{S}_R}{dof(S{S}_R)}\\ & =& \frac{12816.35}{2}\\ & =& 6408.17\end{array}$

Similarly to calculate the error mean square, $M{S}_E$, the error sum of squares, $S{S}_E$, can be obtained as:

$\begin{array}{rlr}S{S}_E& =& y^{\prime }[I-H]y\\ & =& 423.37\end{array}$

The degrees of freedom associated with $S{S}_E$ is $n-(k+1)$. Therefore, the error mean square, $M{S}_E$, is:

$\begin{array}{rlr}M{S}_E& =& \frac{S{S}_E}{dof(S{S}_E)}\\ & =& \frac{S{S}_E}{(n-(k+1))}\\ & =& \frac{423.37}{(17-(2+1))}\\ & =& 30.24\end{array}$

The statistic to test the significance of regression can now be calculated as:

$\begin{array}{rlr}{f}_0& =& \frac{M{S}_R}{M{S}_E}\\ & =& \frac{6408.17}{423.37/(17-3)}\\ & =& 211.9\end{array}$

The critical value for this test, corresponding to a significance level of 0.1, is:

$\begin{array}{rlr}{f}_{\alpha ,k,n-(k+1)}& =& f_{0.1,2,14}\\ & =& 2.726\end{array}$

Since $f_0>f_{0.1,2,14}$, $H_0:$ ${\beta }_1={\beta }_2=0$ is rejected and it is concluded that at least one coefficient out of ${\beta }_1$ and ${\beta }_2$ is significant. In other words, it is concluded that a regression model exists between yield and either one or both of the factors in the table. The analysis of variance is summarized in the following table.

Test on Individual Regression Coefficients (t Test)

The $t$ test is used to check the significance of individual regression coefficients in the multiple linear regression model. Adding a significant variable to a regression model makes the model more effective, while adding an unimportant variable may make the model worse. The hypothesis statements to test the significance of a particular regression coefficient, ${\beta }_j$, are:

$\begin{array}{rlr}& H_0:& {\beta }_j=0\\ & H_1:& {\beta }_j\ne 0\end{array}$

The test statistic for this test is based on the $t$ distribution (and is similar to the one used in the case of simple linear regression models in Simple Linear Regression Anaysis):

$T_0=\frac{{\hat{\beta }}_j}{se({\hat{\beta }}_j)}$

where the standard error, $se({\hat{\beta }}_j)$, is obtained. The analyst would fail to reject the null hypothesis if the test statistic lies in the acceptance region:

$-t_{\alpha /2,n-2}<T_0<t_{\alpha /2,n-2}$

This test measures the contribution of a variable while the remaining variables are included in the model. For the model $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_1{x}_1+{\hat{\beta }}_2{x}_2+{\hat{\beta }}_3{x}_3$, if the test is carried out for ${\beta }_1$, then the test will check the significance of including the variable $x_1$ in the model that contains $x_2$ and $x_3$ (i.e., the model $\hat{y}={\hat{\beta }}_0+{\hat{\beta }}_2{x}_2+{\hat{\beta }}_3{x}_3$ ). Hence the test is also referred to as partial or marginal test. In DOE folios, this test is displayed in the Regression Information table.

Example

The test to check the significance of the estimated regression coefficients for the data is illustrated in this example. The null hypothesis to test the coefficient ${\beta }_2$ is:

$H_0:{\beta }_2=0$

The null hypothesis to test ${\beta }_1$ can be obtained in a similar manner. To calculate the test statistic, $T_0$, we need to calculate the standard error. In the example, the value of the error mean square, $M{S}_E$, was obtained as 30.24. The error mean square is an estimate of the variance, ${\sigma }^2$.

Therefore:

$\begin{array}{rlr}{\hat{\sigma }}^2& =& M{S}_E\\ & =& 30.24\end{array}$

The variance-covariance matrix of the estimated regression coefficients is:

$\begin{array}{rlr}C& =& {\hat{\sigma }}^2{(X^{\prime }X)}^{-1}\\ & =& 30.24\left[\begin{array}{ccc}336.5& 1.2& -13.1\\ 1.2& 0.005& -0.049\\ -13.1& -0.049& 0.5\end{array}\right]\\ & =& \left[\begin{array}{ccc}10176.75& 37.145& -395.83\\ 37.145& 0.1557& -1.481\\ -395.83& -1.481& 15.463\end{array}\right]\end{array}$

From the diagonal elements of $C$, the estimated standard error for ${\hat{\beta }}_1$ and ${\hat{\beta }}_2$ is:

$\begin{array}{rlr}se({\hat{\beta }}_1)& =& \sqrt{0.1557}=0.3946\\ se({\hat{\beta }}_2)& =& \sqrt{15.463}=3.93\end{array}$

The corresponding test statistics for these coefficients are:

$\begin{array}{rlr}{(t_0)}_{{\hat{\beta }}_1}& =& \frac{{\hat{\beta }}_1}{se({\hat{\beta }}_1)}=\frac{1.24}{0.3946}=3.1393\\ {(t_0)}_{{\hat{\beta }}_2}& =& \frac{{\hat{\beta }}_2}{se({\hat{\beta }}_2)}=\frac{12.08}{3.93}=3.0726\end{array}$

The critical values for the present $t$ test at a significance of 0.1 are:

$\begin{array}{rlr}{t}_{\alpha /2,n-(k+1)}& =& t_{0.05,14}=1.761\\ -t_{\alpha /2,n-(k+1)}& =& -t_{0.05,14}=-1.761\end{array}$

Considering ${\hat{\beta }}_2$, it can be seen that ${(t_0)}_{{\hat{\beta }}_2}$ does not lie in the acceptance region of $-t_{0.05,14}<t_0<t_{0.05,14}$. The null hypothesis, $H_0:{\beta }_2=0$, is rejected and it is concluded that ${\beta }_2$ is significant at $\alpha =0.1$. This conclusion can also be arrived at using the $p$ value noting that the hypothesis is two-sided. The $p$ value corresponding to the test statistic, ${(t_0)}_{{\hat{\beta }}_2}=3.0726$, based on the $t$ distribution with 14 degrees of freedom is:

$\begin{array}{rlr}pvalue& =& 2\times (1-P(T\le |t_0|)\\ & =& 2\times (1-0.9959)\\ & =& 0.0083\end{array}$

Since the $p$ value is less than the significance, $\alpha =0.1$, it is concluded that ${\beta }_2$ is significant. The hypothesis test on ${\beta }_1$ can be carried out in a similar manner.

As explained in Simple Linear Regression Analysis, in DOE folios, the information related to the $t$ test is displayed in the Regression Information table as shown in the figure below.

In this table, the $t$ test for ${\beta }_2$ is displayed in the row for the term Factor 2 because ${\beta }_2$ is the coefficient that represents this factor in the regression model. Columns labeled Standard Error, T Value and P Value represent the standard error, the test statistic for the $t$ test and the $p$ value for the $t$ test, respectively. These values have been calculated for ${\beta }_2$ in this example. The Coefficient column represents the estimate of regression coefficients. These values are calculated as shown in this example. The Effect column represents values obtained by multiplying the coefficients by a factor of 2. This value is useful in the case of two factor experiments and is explained in Two-Level Factorial Experiments. Columns labeled Low Confidence and High Confidence represent the limits of the confidence intervals for the regression coefficients and are explained in Confidence Intervals in Multiple Linear Regression. The Variance Inflation Factor column displays values that give a measure of multicollinearity. This is explained in Multicollinearity.

Test on Subsets of Regression Coefficients (Partial F Test)

This test can be considered to be the general form of the $t$ test mentioned in the previous section. This is because the test simultaneously checks the significance of including many (or even one) regression coefficients in the multiple linear regression model. Adding a variable to a model increases the regression sum of squares, $S{S}_R$. The test is based on this increase in the regression sum of squares. The increase in the regression sum of squares is called the extra sum of squares. Assume that the vector of the regression coefficients, $\beta$, for the multiple linear regression model, $y=X\beta +ϵ$, is partitioned into two vectors with the second vector, ${\theta }_2$, containing the last $r$ regression coefficients, and the first vector, ${\theta }_1$, containing the first ( $k+1-r$ ) coefficients as follows:

$\beta =\left[\begin{array}{c}{\theta }_1\\ {\theta }_2\end{array}\right]$

with:

${\theta }_1=[{\beta }_0,{\beta }_1...{\beta }_{k-r}{]}^{\prime }\text{ and }{\theta }_2=[{\beta }_{k-r+1},{\beta }_{k-r+2}...{\beta }_k{]}^{\prime }$

The hypothesis statements to test the significance of adding the regression coefficients in ${\theta }_2$ to a model containing the regression coefficients in ${\theta }_1$ may be written as:

$\begin{array}{rlr}& H_0:& {\theta }_2=0\\ & H_1:& {\theta }_2\ne 0\end{array}$

The test statistic for this test follows the $F$ distribution and can be calculated as follows:

$F_0=\frac{S{S}_R({\theta }_2|{\theta }_1)/r}{M{S}_E}$

where $S{S}_R({\theta }_2|{\theta }_1)$ is the the increase in the regression sum of squares when the variables corresponding to the coefficients in ${\theta }_2$ are added to a model already containing ${\theta }_1$, and $M{S}_E$ is obtained from the equation given in Simple Linear Regression Analysis. The value of the extra sum of squares is obtained as explained in the next section.

The null hypothesis, $H_0$, is rejected if $F_0>f_{\alpha ,r,n-(k+1)}$. Rejection of $H_0$ leads to the conclusion that at least one of the variables in $x_{k-r+1}$, $x_{k-r+2}$... $x_k$ contributes significantly to the regression model. In a DOE folio, the results from the partial $F$ test are displayed in the ANOVA table.

Types of Extra Sum of Squares

The extra sum of squares can be calculated using either the partial (or adjusted) sum of squares or the sequential sum of squares. The type of extra sum of squares used affects the calculation of the test statistic for the partial $F$ test described above. In DOE folios, selection for the type of extra sum of squares is available as shown in the figure below. The partial sum of squares is used as the default setting. The reason for this is explained in the following section on the partial sum of squares.

Partial Sum of Squares

The partial sum of squares for a term is the extra sum of squares when all terms, except the term under consideration, are included in the model. For example, consider the model:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_{12}{x}_1{x}_2+ϵ$

The sum of squares of regression of this model is denoted by $S{S}_R({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12})$. Assume that we need to know the partial sum of squares for ${\beta }_2$. The partial sum of squares for ${\beta }_2$ is the increase in the regression sum of squares when ${\beta }_2$ is added to the model. This increase is the difference in the regression sum of squares for the full model of the equation given above and the model that includes all terms except ${\beta }_2$. These terms are ${\beta }_0$, ${\beta }_1$ and ${\beta }_{12}$. The model that contains these terms is:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_{12}{x}_1{x}_2+ϵ$

The sum of squares of regression of this model is denoted by $S{S}_R({\beta }_0,{\beta }_1,{\beta }_{12})$. The partial sum of squares for ${\beta }_2$can be represented as $S{S}_R({\beta }_2|{\beta }_0,{\beta }_1,{\beta }_{12})$ and is calculated as follows:

$\begin{array}{rlr}S{S}_R({\beta }_2|{\beta }_0,{\beta }_1,{\beta }_{12})& =& S{S}_R({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12})-S{S}_R({\beta }_0,{\beta }_1,{\beta }_{12})\end{array}$

For the present case, ${\theta }_2=[{\beta }_2{]}^{\prime }$ and ${\theta }_1=[{\beta }_0,{\beta }_1,{\beta }_{12}{]}^{\prime }$. It can be noted that for the partial sum of squares ${\beta }_1$ contains all coefficients other than the coefficient being tested.

A Weibull++ DOE folio has the partial sum of squares as the default selection. This is because the $t$ test is a partial test, i.e., the $t$ test on an individual coefficient is carried by assuming that all the remaining coefficients are included in the model (similar to the way the partial sum of squares is calculated). The results from the $t$ test are displayed in the Regression Information table. The results from the partial $F$ test are displayed in the ANOVA table. To keep the results in the two tables consistent with each other, the partial sum of squares is used as the default selection for the results displayed in the ANOVA table. The partial sum of squares for all terms of a model may not add up to the regression sum of squares for the full model when the regression coefficients are correlated. If it is preferred that the extra sum of squares for all terms in the model always add up to the regression sum of squares for the full model then the sequential sum of squares should be used.

Example

This example illustrates the $F$ test using the partial sum of squares. The test is conducted for the coefficient ${\beta }_1$ corresponding to the predictor variable $x_1$ for the data. The regression model used for this data set in the example is:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$

The null hypothesis to test the significance of ${\beta }_1$ is:

$H_0:{\beta }_1=0$

The statistic to test this hypothesis is:

$F_0=\frac{S{S}_R({\beta }_1|{\beta }_2)/r}{M{S}_E}$

where $S{S}_R({\beta }_1|{\beta }_2)$ represents the partial sum of squares for ${\beta }_1$, $r$ represents the number of degrees of freedom for $S{S}_R({\beta }_1|{\beta }_2)$ (which is one because there is just one coefficient, ${\beta }_1$, being tested) and $M{S}_E$ is the error mean square and has been calculated in the second example as 30.24.

The partial sum of squares for ${\beta }_1$ is the difference between the regression sum of squares for the full model, $Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$, and the regression sum of squares for the model excluding ${\beta }_1$, $Y={\beta }_0+{\beta }_2{x}_2+ϵ$. The regression sum of squares for the full model has been calculated in the second example as 12816.35. Therefore:

$S{S}_R({\beta }_0,{\beta }_1,{\beta }_2)=12816.35$

The regression sum of squares for the model $Y={\beta }_0+{\beta }_2{x}_2+ϵ$ is obtained as shown next. First the design matrix for this model, $X_{{\beta }_0,{\beta }_2}$, is obtained by dropping the second column in the design matrix of the full model, $X$ (the full design matrix, $X$, was obtained in the example). The second column of $X$ corresponds to the coefficient ${\beta }_1$ which is no longer in the model. Therefore, the design matrix for the model, $Y={\beta }_0+{\beta }_2{x}_2+ϵ$, is:

$X_{{\beta }_0,{\beta }_2}=\left[\begin{array}{cc}1& 29.1\\ 1& 29.3\\ .& .\\ .& .\\ 1& 32.9\end{array}\right]$

The hat matrix corresponding to this design matrix is $H_{{\beta }_0,{\beta }_2}$. It can be calculated using $H_{{\beta }_0,{\beta }_2}=X_{{\beta }_0,{\beta }_2}{(X_{{\beta }_0,{\beta }_2}^{\prime }{X}_{{\beta }_0,{\beta }_2})}^{-1}{X}_{{\beta }_0,{\beta }_2}^{\prime }$. Once $H_{{\beta }_0,{\beta }_2}$ is known, the regression sum of squares for the model $Y={\beta }_0+{\beta }_2{x}_2+ϵ$, can be calculated as:

$\begin{array}{rlr}S{S}_R({\beta }_0,{\beta }_2)& =& y^{\prime }[H_{{\beta }_0,{\beta }_2}-(\frac{1}{n})J]y\\ & =& 12518.32\end{array}$

Therefore, the partial sum of squares for ${\beta }_1$ is:

$\begin{array}{rlr}S{S}_R({\beta }_1|{\beta }_2)& =& S{S}_R({\beta }_0,{\beta }_1,{\beta }_2)-S{S}_R({\beta }_0,{\beta }_2)\\ & =& 12816.35-12518.32\\ & =& 298.03\end{array}$

Knowing the partial sum of squares, the statistic to test the significance of ${\beta }_1$ is:

$\begin{array}{rlr}{f}_0& =& \frac{S{S}_R({\beta }_1|{\beta }_2)/r}{M{S}_E}\\ & =& \frac{298.03/1}{30.24}\\ & =& 9.855\end{array}$

The $p$ value corresponding to this statistic based on the $F$ distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

$\begin{array}{rlr}pvalue& =& 1-P(F\le f_0)\\ & =& 1-0.9928\\ & =& 0.0072\end{array}$

Assuming that the desired significance is 0.1, since $p$ value < 0.1, $H_0:{\beta }_1=0$ is rejected and it can be concluded that ${\beta }_1$ is significant. The test for ${\beta }_2$ can be carried out in a similar manner. In the results obtained from the DOE folio, the calculations for this test are displayed in the ANOVA table as shown in the following figure. Note that the conclusion obtained in this example can also be obtained using the $t$ test as explained in the example in Test on Individual Regression Coefficients (t Test). The ANOVA and Regression Information tables in the DOE folio represent two different ways to test for the significance of the variables included in the multiple linear regression model.

Sequential Sum of Squares

The sequential sum of squares for a coefficient is the extra sum of squares when coefficients are added to the model in a sequence. For example, consider the model:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_{12}{x}_1{x}_2+{\beta }_3{x}_3+{\beta }_{13}{x}_1{x}_3+{\beta }_{23}{x}_2{x}_3+{\beta }_{123}{x}_1{x}_2{x}_3+ϵ$

The sequential sum of squares for ${\beta }_{13}$ is the increase in the sum of squares when ${\beta }_{13}$ is added to the model observing the sequence of the equation given above. Therefore this extra sum of squares can be obtained by taking the difference between the regression sum of squares for the model after ${\beta }_{13}$ was added and the regression sum of squares for the model before ${\beta }_{13}$ was added to the model. The model after ${\beta }_{13}$ is added is as follows:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_{12}{x}_1{x}_2+{\beta }_3{x}_3+{\beta }_{13}{x}_1{x}_3+ϵ$

This is because to maintain the sequence all coefficients preceding ${\beta }_{13}$ must be included in the model. These are the coefficients ${\beta }_0$, ${\beta }_1$, ${\beta }_2$, ${\beta }_{12}$ and ${\beta }_3$. Similarly the model before ${\beta }_{13}$ is added must contain all coefficients of the equation given above except ${\beta }_{13}$. This model can be obtained as follows:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+{\beta }_{12}{x}_1{x}_2+{\beta }_3{x}_3+ϵ$

The sequential sum of squares for ${\beta }_{13}$ can be calculated as follows:

$\begin{array}{rlr}S{S}_R({\beta }_{13}|{\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12},{\beta }_3)& =& S{S}_R({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12},{\beta }_3,{\beta }_{13})-S{S}_R({\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12},{\beta }_3)\end{array}$

For the present case, ${\theta }_2=[{\beta }_{13}{]}^{\prime }$ and ${\theta }_1=[{\beta }_0,{\beta }_1,{\beta }_2,{\beta }_{12},{\beta }_3{]}^{\prime }$. It can be noted that for the sequential sum of squares ${\beta }_1$ contains all coefficients proceeding the coefficient being tested.

The sequential sum of squares for all terms will add up to the regression sum of squares for the full model, but the sequential sum of squares are order dependent.

Example

This example illustrates the partial $F$ test using the sequential sum of squares. The test is conducted for the coefficient ${\beta }_1$ corresponding to the predictor variable $x_1$ for the data. The regression model used for this data set in the example is:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$

The null hypothesis to test the significance of ${\beta }_1$ is:

$H_0:{\beta }_1=0$

The statistic to test this hypothesis is:

$F_0=\frac{S{S}_R({\beta }_0,{\beta }_1)/r}{M{S}_E}$

where $S{S}_R({\beta }_0,{\beta }_1)$ represents the sequential sum of squares for ${\beta }_1$, $r$ represents the number of degrees of freedom for $S{S}_R({\beta }_0,{\beta }_1)$ (which is one because there is just one coefficient, ${\beta }_1$, being tested) and $M{S}_E$ is the error mean square and has been calculated in the second example as 30.24.

The sequential sum of squares for ${\beta }_1$ is the difference between the regression sum of squares for the model after adding ${\beta }_1$, $Y={\beta }_0+{\beta }_1{x}_1+ϵ$, and the regression sum of squares for the model before adding ${\beta }_1$, $Y={\beta }_0+ϵ$. The regression sum of squares for the model $Y={\beta }_0+{\beta }_1{x}_1+ϵ$ is obtained as shown next. First the design matrix for this model, $X_{{\beta }_0,{\beta }_1}$, is obtained by dropping the third column in the design matrix for the full model, $X$ (the full design matrix, $X$, was obtained in the example). The third column of $X$ corresponds to coefficient ${\beta }_2$ which is no longer used in the present model. Therefore, the design matrix for the model, $Y={\beta }_0+{\beta }_1{x}_1+ϵ$, is:

$X_{{\beta }_0,{\beta }_1}=\left[\begin{array}{cc}1& 41.9\\ 1& 43.4\\ .& .\\ .& .\\ 1& 77.8\end{array}\right]$

The hat matrix corresponding to this design matrix is $H_{{\beta }_0,{\beta }_1}$. It can be calculated using $H_{{\beta }_0,{\beta }_1}=X_{{\beta }_0,{\beta }_1}{(X_{{\beta }_0,{\beta }_1}^{\prime }{X}_{{\beta }_0,{\beta }_1})}^{-1}{X}_{{\beta }_0,{\beta }_1}^{\prime }$. Once $H_{{\beta }_0,{\beta }_1}$ is known, the regression sum of squares for the model $Y={\beta }_0+{\beta }_1{x}_1+ϵ$ can be calculated as:

$\begin{array}{rlr}S{S}_R({\beta }_0,{\beta }_1)& =& y^{\prime }[H_{{\beta }_0,{\beta }_1}-(\frac{1}{n})J]y\\ & =& 12530.85\end{array}$

The regression sum of squares for the model $Y={\beta }_0+ϵ$ is equal to zero since this model does not contain any variables. Therefore:

$S{S}_R({\beta }_0)=0$

The sequential sum of squares for ${\beta }_1$ is:

$\begin{array}{rlr}S{S}_R({\beta }_1|{\beta }_0)& =& S{S}_R({\beta }_0,{\beta }_1)-S{S}_R({\beta }_0)\\ & =& 12530.85-0\\ & =& 12530.85\end{array}$

Knowing the sequential sum of squares, the statistic to test the significance of ${\beta }_1$ is:

$\begin{array}{rlr}{f}_0& =& \frac{S{S}_R({\beta }_0,{\beta }_1)/r}{M{S}_E}\\ & =& \frac{12530.85/1}{30.24}\\ & =& 414.366\end{array}$

The $p$ value corresponding to this statistic based on the $F$ distribution with 1 degree of freedom in the numerator and 14 degrees of freedom in the denominator is:

$\begin{array}{rlr}pvalue& =& 1-P(F\le f_0)\\ & =& 1-0.999999\\ & =& 8.46\times {10}^{-12}\end{array}$

Assuming that the desired significance is 0.1, since $p$ value < 0.1, $H_0:{\beta }_1=0$ is rejected and it can be concluded that ${\beta }_1$ is significant. The test for ${\beta }_2$ can be carried out in a similar manner. This result is shown in the following figure.

Confidence Intervals in Multiple Linear Regression

Calculation of confidence intervals for multiple linear regression models are similar to those for simple linear regression models explained in Simple Linear Regression Analysis.

Confidence Interval on Regression Coefficients

A 100 ($1-\alpha$) percent confidence interval on the regression coefficient, ${\beta }_j$, is obtained as follows:

${\hat{\beta }}_j\pm t_{\alpha /2,n-(k+1)}\sqrt{C_{jj}}$

The confidence interval on the regression coefficients are displayed in the Regression Information table under the Low Confidence and High Confidence columns as shown in the following figure.

Confidence Interval on Fitted Values, ${\hat{y}}_i$ A 100 ($1-\alpha$) percent confidence interval on any fitted value, ${\hat{y}}_i$, is given by:

${\hat{y}}_i\pm t_{\alpha /2,n-(k+1)}\sqrt{{\hat{\sigma }}^2{x}_i^{\prime }{(X^{\prime }X)}^{-1}{x}_i}$

where:

$x_i=\left[\begin{array}{c}1\\ x_{i1}\\ .\\ .\\ .\\ x_{ik}\end{array}\right]$

In the above example, the fitted value corresponding to the fifth observation was calculated as ${\hat{y}}_5=266.3$. The 90% confidence interval on this value can be obtained as shown in the figure below. The values of 47.3 and 29.9 used in the figure are the values of the predictor variables corresponding to the fifth observation the table.

Confidence Interval on New Observations

As explained in Simple Linear Regression Analysis, the confidence interval on a new observation is also referred to as the prediction interval. The prediction interval takes into account both the error from the fitted model and the error associated with future observations. A 100 ($1-\alpha$) percent confidence interval on a new observation, ${\hat{y}}_p$, is obtained as follows:

${\hat{y}}_p\pm t_{\alpha /2,n-(k+1)}\sqrt{{\hat{\sigma }}^2(1+x_p^{\prime }{(X^{\prime }X)}^{-1}{x}_p)}$

where:

$x_p=\left[\begin{array}{c}1\\ x_{p1}\\ .\\ .\\ .\\ x_{pk}\end{array}\right]$

$x_{p1}$,..., $x_{pk}$ are the levels of the predictor variables at which the new observation, ${\hat{y}}_p$, needs to be obtained.

In multiple linear regression, prediction intervals should only be obtained at the levels of the predictor variables where the regression model applies. In the case of multiple linear regression it is easy to miss this. Having values lying within the range of the predictor variables does not necessarily mean that the new observation lies in the region to which the model is applicable. For example, consider the next figure where the shaded area shows the region to which a two variable regression model is applicable. The point corresponding to $p$ th level of first predictor variable, $x_1$, and $p$ th level of the second predictor variable, $x_2$, does not lie in the shaded area, although both of these levels are within the range of the first and second predictor variables respectively. In this case, the regression model is not applicable at this point.

Measures of Model Adequacy

As in the case of simple linear regression, analysis of a fitted multiple linear regression model is important before inferences based on the model are undertaken. This section presents some techniques that can be used to check the appropriateness of the multiple linear regression model.

Coefficient of Multiple Determination, R²

The coefficient of multiple determination is similar to the coefficient of determination used in the case of simple linear regression. It is defined as:

$\begin{array}{rlr}{R}^2& =& \frac{S{S}_R}{S{S}_T}\\ & =& 1-\frac{S{S}_E}{S{S}_T}\end{array}$

$R^2$ indicates the amount of total variability explained by the regression model. The positive square root of $R^2$ is called the multiple correlation coefficient and measures the linear association between $Y$ and the predictor variables, $x_1$, $x_2$... $x_k$.

The value of $R^2$ increases as more terms are added to the model, even if the new term does not contribute significantly to the model. An increase in the value of $R^2$ cannot be taken as a sign to conclude that the new model is superior to the older model. A better statistic to use is the adjusted $R^2$ statistic defined as follows:

$\begin{array}{rlr}{R}_{adj}^2& =& 1-\frac{M{S}_E}{M{S}_T}\\ & =& 1-\frac{S{S}_E/(n-(k+1))}{S{S}_T/(n-1)}\\ & =& 1-(\frac{n-1}{n-(k+1)})(1-R^2)\end{array}$

The adjusted $R^2$ only increases when significant terms are added to the model. Addition of unimportant terms may lead to a decrease in the value of $R_{adj}^2$.

In a DOE folio, $R^2$ and $R_{adj}^2$ values are displayed as R-sq and R-sq(adj), respectively. Other values displayed along with these values are S, PRESS and R-sq(pred). As explained in Simple Linear Regression Analysis, the value of S is the square root of the error mean square, $M{S}_E$, and represents the "standard error of the model."

PRESS is an abbreviation for prediction error sum of squares. It is the error sum of squares calculated using the PRESS residuals in place of the residuals, $e_i$, in the equation for the error sum of squares. The PRESS residual, $e_{(i)}$, for a particular observation, $y_i$, is obtained by fitting the regression model to the remaining observations. Then the value for a new observation, ${\hat{y}}_p$, corresponding to the observation in question, $y_i$, is obtained based on the new regression model. The difference between $y_i$ and ${\hat{y}}_p$ gives $e_{(i)}$. The PRESS residual, $e_{(i)}$, can also be obtained using $h_{ii}$, the diagonal element of the hat matrix, $H$, as follows:

$e_{(i)}=\frac{e_i}{1-h_{ii}}$

R-sq(pred), also referred to as prediction $R^2$, is obtained using PRESS as shown next:

$R_{pred}^2=1-\frac{PRESS}{S{S}_T}$

The values of R-sq, R-sq(adj) and S are indicators of how well the regression model fits the observed data. The values of PRESS and R-sq(pred) are indicators of how well the regression model predicts new observations. For example, higher values of PRESS or lower values of R-sq(pred) indicate a model that predicts poorly. The figure below shows these values for the data. The values indicate that the regression model fits the data well and also predicts well.

Residual Analysis

Plots of residuals, $e_i$, similar to the ones discussed in Simple Linear Regression Analysis for simple linear regression, are used to check the adequacy of a fitted multiple linear regression model. The residuals are expected to be normally distributed with a mean of zero and a constant variance of ${\sigma }^2$. In addition, they should not show any patterns or trends when plotted against any variable or in a time or run-order sequence. Residual plots may also be obtained using standardized and studentized residuals. Standardized residuals, $d_i$, are obtained using the following equation:

$\begin{array}{rlr}{d}_i& =& \frac{e_i}{\sqrt{{\hat{\sigma }}^2}}\\ & =& \frac{e_i}{\sqrt{M{S}_E}}\end{array}$

Standardized residuals are scaled so that the standard deviation of the residuals is approximately equal to one. This helps to identify possible outliers or unusual observations. However, standardized residuals may understate the true residual magnitude, hence studentized residuals, $r_i$, are used in their place. Studentized residuals are calculated as follows:

$\begin{array}{rlr}{r}_i& =& \frac{e_i}{\sqrt{{\hat{\sigma }}^2(1-h_{ii})}}\\ & =& \frac{e_i}{\sqrt{M{S}_E(1-h_{ii})}}\end{array}$

where $h_{ii}$ is the $i$ th diagonal element of the hat matrix, $H$. External studentized (or the studentized deleted) residuals may also be used. These residuals are based on the PRESS residuals mentioned in Coefficient of Multiple Determination, R². The reason for using the external studentized residuals is that if the $i$ th observation is an outlier, it may influence the fitted model. In this case, the residual $e_i$ will be small and may not disclose that $i$ th observation is an outlier. The external studentized residual for the $i$ th observation, $t_i$, is obtained as follows:

$t_i=e_i{\left[\frac{n-k}{S{S}_E(1-h_{ii})-e_i^2}\right]}^{0.5}$

Residual values for the data are shown in the figure below. Standardized residual plots for the data are shown in next two figures. The Weibull++ DOE folio compares the residual values to the critical values on the $t$ distribution for studentized and external studentized residuals.

For other residuals the normal distribution is used. For example, for the data, the critical values on the $t$ distribution at a significance of 0.1 are $t_{0.05,14}=1.761$ and $-t_{0.05,14}=-1.761$ (as calculated in the example, Test on Individual Regression Coefficients (t Test)). The studentized residual values corresponding to the 3rd and 17th observations lie outside the critical values. Therefore, the 3rd and 17th observations are outliers. This can also be seen on the residual plots in the next two figures.

Outlying x Observations

Residuals help to identify outlying $y$ observations. Outlying $x$ observations can be detected using leverage. Leverage values are the diagonal elements of the hat matrix, $h_{ii}$. The $h_{ii}$ values always lie between 0 and 1. Values of $h_{ii}$ greater than $2(k+1)/n$ are considered to be indicators of outlying $x$ observations.

Influential Observations Detection

Once an outlier is identified, it is important to determine if the outlier has a significant effect on the regression model. One measure to detect influential observations is Cook's distance measure which is computed as follows:

$D_i=\frac{r_i^2}{(k+1)}\left[\frac{h_{ii}}{(1-h_{ii})}\right]$

To use Cook's distance measure, the $D_i$ values are compared to percentile values on the $F$ distribution with $(k+1,n-(k+1))$ degrees of freedom. If the percentile value is less than 10 or 20 percent, then the $i$ th case has little influence on the fitted values. However, if the percentile value is close to 50 percent or greater, the $i$ th case is influential, and fitted values with and without the $i$ th case will differ substantially.

Example

Cook's distance measure can be calculated as shown next. The distance measure is calculated for the first observation of the data. The remaining values along with the leverage values are shown in the figure below (displaying Leverage and Cook's distance measure for the data).

The standardized residual corresponding to the first observation is:

$\begin{array}{rlr}{r}_1& =& \frac{e_1}{\sqrt{M{S}_E(1-h_{11})}}\\ & =& \frac{1.3127}{\sqrt{30.3(1-0.2755)}}\\ & =& 0.2804\end{array}$

Cook's distance measure for the first observation can now be calculated as:

$\begin{array}{rlr}{D}_1& =& \frac{r_1^2}{(k+1)}\left[\frac{h_{11}}{(1-h_{11})}\right]\\ & =& \frac{{0.2804}^2}{(2+1)}\left[\frac{0.2755}{(1-0.2755)}\right]\\ & =& 0.01\end{array}$

The 50th percentile value for $F_{3,14}$ is 0.83. Since all $D_i$ values are less than this value there are no influential observations.

Lack-of-Fit Test

The lack-of-fit test for simple linear regression discussed in Simple Linear Regression Analysis may also be applied to multiple linear regression to check the appropriateness of the fitted response surface and see if a higher order model is required. Data for $m$ replicates may be collected as follows for all $n$ levels of the predictor variables:

$\begin{array}{rlr}& & y_{11},y_{12},....,y_{1m}m\text{ repeated observations at the first level }\\ & & y_{21},y_{22},....,y_{2m}m\text{ repeated observations at the second level}\\ & & ...\\ & & y_{i1},y_{i2},....,y_{im}m\text{ repeated observations at the }i\text{th level}\\ & & ...\\ & & y_{n1},y_{n2},....,y_{nm}m\text{ repeated observations at the }n\text{th level }\end{array}$

The sum of squares due to pure error, $S{S}_{PE}$, can be obtained as discussed in the Simple Linear Regression Analysis as:

$S{S}_{PE}=\underset{i=1}{\sum ^n}\underset{j=1}{\sum ^m}{(y_{ij}-{\overline{y}}_i)}^2$

The number of degrees of freedom associated with $S{S}_{PE}$ are:

$dof(S{S}_{PE})=nm-n$

Knowing $S{S}_{PE}$, sum of squares due to lack-of-fit, $S{S}_{LOF}$, can be obtained as:

$S{S}_{LOF}=S{S}_E-S{S}_{PE}$

The number of degrees of freedom associated with $S{S}_{LOF}$ are:

$\begin{array}{rlr}dof(S{S}_{LOF})& =& dof(S{S}_E)-dof(S{S}_{PE})\\ & =& n-(k+1)-(nm-n)\end{array}$

The test statistic for the lack-of-fit test is:

$\begin{array}{rlr}{F}_0& =& \frac{S{S}_{LOF}/dof(S{S}_{LOF})}{S{S}_{PE}/dof(S{S}_{PE})}\\ & =& \frac{M{S}_{LOF}}{M{S}_{PE}}\end{array}$

Other Topics in Multiple Linear Regression

Polynomial Regression Models

Polynomial regression models are used when the response is curvilinear. The equation shown next presents a second order polynomial regression model with one predictor variable:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_{11}{x}_1^2+ϵ$

Usually, coded values are used in these models. Values of the variables are coded by centering or expressing the levels of the variable as deviations from the mean value of the variable and then scaling or dividing the deviations obtained by half of the range of the variable.

$codedvalue=\frac{actualvalue-mean}{halfofrange}$

The reason for using coded predictor variables is that many times $x$ and $x^2$ are highly correlated and, if uncoded values are used, there may be computational difficulties while calculating the ${(X^{\prime }X)}^{-1}$ matrix to obtain the estimates, $\hat{\beta }$, of the regression coefficients using the equation for the $F$ distribution given in Statistics Background on DOE.

Qualitative Factors

The multiple linear regression model also supports the use of qualitative factors. For example, gender may need to be included as a factor in a regression model. One of the ways to include qualitative factors in a regression model is to employ indicator variables. Indicator variables take on values of 0 or 1. For example, an indicator variable may be used with a value of 1 to indicate female and a value of 0 to indicate male.

$x_1=\{\begin{array}{l}1\text{ Female}\\ 0\text{ Male}\end{array}$

In general ( $n-1$ ) indicator variables are required to represent a qualitative factor with $n$ levels. As an example, a qualitative factor representing three types of machines may be represented as follows using two indicator variables:

$\begin{array}{rlrlr}{x}_1& =& 1,x_2& =& 0\text{ Machine Type I}\\ x_1& =& 0,x_2& =& 1\text{ Machine Type II}\\ x_1& =& 0,x_2& =& 0\text{ Machine Type III}\end{array}$

An alternative coding scheme for this example is to use a value of -1 for all indicator variables when representing the last level of the factor:

$\begin{array}{rlrlr}{x}_1& =& 1,x_2& =& 0\text{ Machine Type I}\\ x_1& =& 0,x_2& =& 1\text{ Machine Type II}\\ x_1& =& -1,x_2& =& -1\text{ Machine Type III}\end{array}$

Indicator variables are also referred to as dummy variables or binary variables.

Example

Consider data from two types of reactors of a chemical process shown where the yield values are recorded for various levels of factor $x_1$. Assuming there are no interactions between the reactor type and $x_1$, a regression model can be fitted to this data as shown next.

Since the reactor type is a qualitative factor with two levels, it can be represented by using one indicator variable. Let $x_2$ be the indicator variable representing the reactor type, with 0 representing the first type of reactor and 1 representing the second type of reactor.

$x_2=\{\begin{array}{l}0\text{ Reactor Type I}\\ 1\text{ Reactor Type II}\end{array}$

Data entry in the DOE folio for this example is shown in the figure after the table below. The regression model for this data is:

$Y={\beta }_0+{\beta }_1{x}_1+{\beta }_2{x}_2+ϵ$

The $X$ and $y$ matrices for the given data are:

The estimated regression coefficients for the model can be obtained as:

$\begin{array}{rlr}\hat{\beta }& =& {(X^{\prime }X)}^{-1}{X}^{\prime }y\\ & =& \left[\begin{array}{c}153.7\\ 2.4\\ -27.5\end{array}\right]\end{array}$

Therefore, the fitted regression model is:

$\hat{y}=153.7+2.4x_1-27.5x_2$

Note that since $x_2$ represents a qualitative predictor variable, the fitted regression model cannot be plotted simultaneously against $x_1$ and $x_2$ in a two-dimensional space (because the resulting surface plot will be meaningless for the dimension in $x_2$ ). To illustrate this, a scatter plot of the data against $x_2$ is shown in the following figure.

It can be noted that, in the case of qualitative factors, the nature of the relationship between the response (yield) and the qualitative factor (reactor type) cannot be categorized as linear, or quadratic, or cubic, etc. The only conclusion that can be arrived at for these factors is to see if these factors contribute significantly to the regression model. This can be done by employing the partial $F$ test discussed in Multiple Linear Regression Analysis (using the extra sum of squares of the indicator variables representing these factors). The results of the test for the present example are shown in the ANOVA table. The results show that $x_2$ (reactor type) contributes significantly to the fitted regression model.

Multicollinearity

At times the predictor variables included in a multiple linear regression model may be found to be dependent on each other. Multicollinearity is said to exist in a multiple regression model with strong dependencies between the predictor variables. Multicollinearity affects the regression coefficients and the extra sum of squares of the predictor variables. In a model with multicollinearity the estimate of the regression coefficient of a predictor variable depends on what other predictor variables are included the model. The dependence may even lead to change in the sign of the regression coefficient. In a such models, an estimated regression coefficient may not be found to be significant individually (when using the $t$ test on the individual coefficient or looking at the $p$ value) even though a statistical relation is found to exist between the response variable and the set of the predictor variables (when using the $F$ test for the set of predictor variables). Therefore, you should be careful while looking at individual predictor variables in models that have multicollinearity. Care should also be taken while looking at the extra sum of squares for a predictor variable that is correlated with other variables. This is because in models with multicollinearity the extra sum of squares is not unique and depends on the other predictor variables included in the model.

Multicollinearity can be detected using the variance inflation factor (abbreviated $VIF$ ). $VIF$ for a coefficient ${\beta }_j$ is defined as:

$VIF=\frac{1}{(1-R_j^2)}$

where $R_j^2$ is the coefficient of multiple determination resulting from regressing the $j$ th predictor variable, $x_j$, on the remaining $k$ -1 predictor variables. Mean values of $VIF$ considerably greater than 1 indicate multicollinearity problems. A few methods of dealing with multicollinearity include increasing the number of observations in a way designed to break up dependencies among predictor variables, combining the linearly dependent predictor variables into one variable, eliminating variables from the model that are unimportant or using coded variables.

Example

Variance inflation factors can be obtained for the data below.

To calculate the variance inflation factor for $x_1$, $R_1^2$ has to be calculated.$R_1^2$ is the coefficient of determination for the model when $x_1$ is regressed on the remaining variables. In the case of this example there is just one remaining variable which is $x_2$. If a regression model is fit to the data, taking $x_1$ as the response variable and $x_2$ as the predictor variable, then the design matrix and the vector of observations are:

$X_{R_1}=\left[\begin{array}{cc}1& 29.1\\ 1& 29.3\\ .& .\\ .& .\\ .& .\\ 1& 32.9\end{array}\right]y_{R_1}=\left[\begin{array}{c}41.9\\ 43.4\\ .\\ .\\ .\\ 77.8\end{array}\right]$

The regression sum of squares for this model can be obtained as:

$\begin{array}{rl}S{S}_R=& y_{R_1}^{\prime }[H_{R_1}-(\frac{1}{n})J]y_{R_1}\\ =& 1988.6\end{array}$

where $H_{R_1}$ is the hat matrix (and is calculated using $H_{R_1}=X_{R_1}{(X_{R_1}^{\prime }{X}_{R_1})}^{-1}{X}_{R_1}^{\prime }$ ) and $J$ is the matrix of ones. The total sum of squares for the model can be calculated as:

$\begin{array}{rl}S{S}_T=& y^{\prime }[I-(\frac{1}{n})J]y\\ =& 2182.9\end{array}$

where $I$ is the identity matrix. Therefore:

$\begin{array}{rl}{R}_1^2=& \frac{S{S}_R}{S{S}_T}\\ =& \frac{1988.6}{2182.9}\\ =& 0.911\end{array}$

Then the variance inflation factor for $x_1$ is:

$\begin{array}{rl}VI{F}_1=& \frac{1}{(1-R_1^2)}\\ =& \frac{1}{1-0.911}\\ =& 11.2\end{array}$

The variance inflation factor for $x_2$, $VI{F}_2$, can be obtained in a similar manner. In the DOE folios, the variance inflation factors are displayed in the VIF column of the Regression Information table as shown in the following figure. Since the values of the variance inflation factors obtained are considerably greater than 1, multicollinearity is an issue for the data.