Linear Regression

Introduction

Suppose you conducted an experiment which resulted in a set of measured or otherwise observed values. For whatever good reasons you assumed to find a specific type of relationship between the measured variables. However, due to errors or inaccuracies in the measurements or in your physical model, the observed values do not perfectly fit a relationship of the expected kind.

Regression analysis, or in case of an expected linear relation, linear regression provides a method to determine the best fit relation to the observed data. An often used form of regression is the method of least squares.

In this article, data fitting using the linear least squares method is explained, needed formulas are derived and an interactive application is provided that finds the linear equation that best fits the data. The goal of linear least squares is to find the parameters a and b of the best fit line $y=a<!--\; INVISIBLE\; TIMES\; -->x+b$.

JavaScript application

The least squares line of best fit

The deviation, "error" or residual r_i (fig.1.), is the difference between an observed value and the value provided by the best fit relationship. In case of linear regression we want to find a best fit line $y=a<!--\; INVISIBLE\; TIMES\; -->x+b$ so:

$$r_i=y_i-<!--\; -\; -->\left(a<!--\; INVISIBLE\; TIMES\; -->x_i+b\right)$$ [1]

Variance is a useful concept to quantify how much a set of values fluctuates about its mean. Variance is defined as:

$$\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->{\left(t_i-<!--\; -\; -->\bar{t}\right)}^2$$

Where $\bar{t}$ represents the arithmetic mean of the set of values and n the number of elements in the set.

If we consider the set of residuals:

$$y_1-\left(a<!--\; INVISIBLE\; TIMES\; -->x_1+b\right),y_2-\left(a<!--\; INVISIBLE\; TIMES\; -->x_2+b\right),\dots ,y_n-\left(a<!--\; INVISIBLE\; TIMES\; -->x_n+b\right)$$

The mean of the set of residuals must be zero (or at least close to zero, if the line is a good fit), and therefore the variance of this set is:

$$\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->{\left(y_i-\left(a<!--\; INVISIBLE\; TIMES\; -->x_i+b\right)\right)}^2=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->{r_i}^2$$

The best fit line is the line in which case the variance is minimal and so where $E_{\left(a,b\right)}=\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->{r_i}^2$ is minimal.

Finding the minimum requires that the gradient is zero, and hence that both partial derivatives with respect to parameters a and b are zero:

$$\begin{array}{lll}\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->a}=0& \text{and}& \frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->b}=0\end{array}$$

Applying the chain rule on each term of the sum:

$$\begin{array}{l}\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_1}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_1}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->a}+\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_2}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_2}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->a}+\dots +\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_n}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_n}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->a}=\\ 2<!--\; INVISIBLE\; TIMES\; -->r_1<!--\; INVISIBLE\; TIMES\; -->x_1+2<!--\; INVISIBLE\; TIMES\; -->r_2<!--\; INVISIBLE\; TIMES\; -->x_2+\dots +2<!--\; INVISIBLE\; TIMES\; -->r_n<!--\; INVISIBLE\; TIMES\; -->x_n=\\ 2<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->r_i<!--\; INVISIBLE\; TIMES\; -->x_i=0\iff \sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->r_i<!--\; INVISIBLE\; TIMES\; -->x_i=0\end{array}$$ and $$\begin{array}{l}\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_1}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_1}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->b}+\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_2}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_2}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->b}+\dots +\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->E}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_n}<!--\; INVISIBLE\; TIMES\; -->\frac{\partial <!--\; INVISIBLE\; SEPARATOR\; -->r_n}{\partial <!--\; INVISIBLE\; SEPARATOR\; -->b}=\\ 2<!--\; INVISIBLE\; TIMES\; -->r_1\times 1+2<!--\; INVISIBLE\; TIMES\; -->r_2\times 1+\dots +2<!--\; INVISIBLE\; TIMES\; -->r_n\times 1=\\ 2<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->r_i=0\iff \sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->r_i=0\end{array}$$

Substitute [1] in both equations:

$$\begin{array}{l}\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->\left(y_i-a<!--\; INVISIBLE\; TIMES\; -->x_i-b\right)<!--\; INVISIBLE\; TIMES\; -->x_i=\\ \sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->x_i<!--\; INVISIBLE\; TIMES\; -->y_i-a<!--\; INVISIBLE\; TIMES\; -->{x_i}^2-b<!--\; INVISIBLE\; TIMES\; -->x_i=0\end{array}$$ and $$\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->y_i-a<!--\; INVISIBLE\; TIMES\; -->x_i-b=0$$

Which results in the so called normal equations (in shortened notation):

$$\sum <!--\; INVISIBLE\; SEPARATOR\; -->x<!--\; INVISIBLE\; TIMES\; -->y-a<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x^2-b<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x=0$$ [2]

$$\sum <!--\; INVISIBLE\; SEPARATOR\; -->y-a<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x-n<!--\; INVISIBLE\; TIMES\; -->b=0$$ [3]

Solving these normal equations for parameters a and b provides us the equation for the best fit line:

From [3]:

$$b=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->y-a<!--\; INVISIBLE\; TIMES\; -->\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x$$

With $\bar{x}=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x$ and $\bar{y}=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->y$ being arithmetic means:

$$b=\bar{y}-a<!--\; INVISIBLE\; TIMES\; -->\bar{x}$$

Substitute in [2]:

$$a=\frac{\bar{y}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x-\sum <!--\; INVISIBLE\; SEPARATOR\; -->x<!--\; INVISIBLE\; TIMES\; -->y}{\bar{x}<!--\; INVISIBLE\; TIMES\; -->\sum <!--\; INVISIBLE\; SEPARATOR\; -->x-\sum <!--\; INVISIBLE\; SEPARATOR\; -->x^2}$$

Summary

Let $\left(x_1,y_1\right),\left(x_2,y_2\right),\dots ,\left(x_i,y_i\right),\dots ,\left(x_n,y_n\right)$ be the n observed data points. Then the best fit line trough these data points is the line $y=a<!--\; INVISIBLE\; TIMES\; -->x+b$ with:

$$a=\frac{\bar{y}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->x_i-\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->x_i<!--\; INVISIBLE\; TIMES\; -->y_i}{\bar{x}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->x_i-\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->{x_i}^2}$$ $$b=\bar{y}-a<!--\; INVISIBLE\; TIMES\; -->\bar{x}$$

With $\bar{x}=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->x_i$ and $\bar{y}=\frac{1}{n}<!--\; INVISIBLE\; TIMES\; -->\sum _{i=1}^n<!--\; INVISIBLE\; SEPARATOR\; -->y_i$ being arithmetic means.