Template:Linear regression (least squares) gompz

Linear Regression (Least Squares)

The method of least squares requires that a straight line be fitted to a set of data points. If the regression is on [math]\displaystyle{ Y }[/math] , then the sum of the squares of the vertical deviations from the points to the line is minimized. If the regression is on [math]\displaystyle{ X }[/math] , the line is fitted to a set of data points such that the sum of the squares of the horizontal deviations from the points to the line is minimized. To illustrate the method, this section presents a regression on [math]\displaystyle{ Y }[/math] . Consider the linear model [2]:

[math]\displaystyle{ {{Y}_{i}}={{\widehat{\beta }}_{0}}+{{\widehat{\beta }}_{1}}{{X}_{i1}}+{{\widehat{\beta }}_{2}}{{X}_{i2}}+...+{{\widehat{\beta }}_{p}}{{X}_{ip}} }[/math]

or in matrix form where bold letters indicate matrices:

[math]\displaystyle{ Y=X\beta }[/math]

where:

[math]\displaystyle{ Y=\left[ \begin{matrix} {{Y}_{1}} \\ {{Y}_{2}} \\ \vdots \\ {{Y}_{N}} \\ \end{matrix} \right] }[/math]

[math]\displaystyle{ X=\left[ \begin{matrix} 1 & {{X}_{1,1}} & \cdots & {{X}_{1,p}} \\ 1 & {{X}_{2,1}} & \cdots & {{X}_{2,p}} \\ \vdots & \vdots & \ddots & \vdots \\ 1 & {{X}_{N,1}} & \cdots & {{X}_{N,p}} \\ \end{matrix} \right] }[/math]

and:

[math]\displaystyle{ \beta =\left[ \begin{matrix} {{\beta }_{0}} \\ {{\beta }_{1}} \\ \vdots \\ {{\beta }_{p}} \\ \end{matrix} \right] }[/math]

The vector [math]\displaystyle{ \beta }[/math] holds the values of the parameters. Now let [math]\displaystyle{ \widehat{\beta } }[/math] be the estimates of these parameters, or the regression coefficients. The vector of estimated regression coefficients is denoted by:

[math]\displaystyle{ \widehat{\beta }=\left[ \begin{matrix} {{\widehat{\beta }}_{0}} \\ {{\widehat{\beta }}_{1}} \\ \vdots \\ {{\widehat{\beta }}_{p}} \\ \end{matrix} \right] }[/math]

Solving for [math]\displaystyle{ \beta }[/math] in Eqn. (linear) requires the analyst to left multiply both sides by the transpose of [math]\displaystyle{ X }[/math] , [math]\displaystyle{ {{X}^{T}} }[/math] :

[math]\displaystyle{ ({{X}^{T}}X)\widehat{\beta }={{X}^{T}}Y }[/math]

Now the term [math]\displaystyle{ ({{X}^{T}}X) }[/math] becomes a square and invertible matrix. Then taking it to the other side of the equation gives:

[math]\displaystyle{ \widehat{\beta }={{(}^{T}}^{-1}{{X}^{T}}Y }[/math]