Least Squares:Simple Fits

From CheLabWiki

…from CheLabWiki, an online resource for chemical-engineering laboratories located at www.chelabwiki.org; Site Revision #755; 6 January 2009.

Community Portal → Data Analysis → Least Squares:Simple Fits

1 The Normal Equations
2 Sample Results
3 See Also
4 References

Once we have found functions of $x$ and $y$ that are related (more or less) linearly, we need to quantify the relation; that is, we need to find the slope and intercept of a representative line. But which line shall we use? Although we may have found a linear representation of our data, some scatter remains because of experimental errors. Further, the linearization is probably only approximate, not exact. So any of an essentially infinite number of straight lines might be chosen to represent the data. We need criteria that can be imposed systematically and that produce a unique straight-line representation of our data.^[1]

The Normal Equations

Figure 1. Example of five points

y (x)

that might be correlated by a straight line. One possible straight line is shown.

Consider Figure 1, which shows five measured points and an arbitrary straight line that might represent the measurements. We assume the usual experimental situation in which we have controlled the values of $x$ and measured corresponding values for $y$ . Hence, the uncertainties all reside in the measured $y$ -values. We have already estimated the quality of the measurements, so we have an uncertainty $u i$ assigned to each measured $y i$ ; we use those uncertainties to weight the data.

Define the deviation $δ y i$ to be the vertical distance between a measured point $y i$ and the corresponding $y$ value on the straight line (One such deviation is labeled on Figure 1.),

(1)

$\delta y_i \ = \ y_i - (mx_i + b)$

Here $m$ is the slope and $b$ is the intercept of the line. Of the many lines we might use, we choose the one that minimizes the sum of the squares of the weighted deviations. Therefore, over $N$ measured points, we have a minimization problem with respect to the slope and intercept:

(2)

$\underset{ \{m,b\} }{\mbox{Min}}\sum_i^N \left ( \frac{\delta y_i}{u_i} \right )^2$

By placing the uncertainties in the denominator, measurements with small uncertainties are weighted more than those with large uncertainties.

The minimization problem (2) is solved by forming the derivatives with respect to $m$ and $b$ , then setting each to zero:

(3)

$\frac{\partial}{\partial m} \left [ \sum_i^N \left ( \frac{\delta y_i}{u_i} \right )^2 \right ] \ = \ 0$

(4)

$\frac{\partial}{\partial b} \left [ \sum_i^N \left ( \frac{\delta y_i}{u_i} \right )^2 \right ] \ = \ 0$

These are two equations that can be solved for two unknowns: $m$ and $b$ . In fact, the two equations (3) and (4) are linear in $m$ and $b$ , so they can be solved using linear algebra. The results are called the normal least-squares equations:^[2]

(5)

$m \ = \ \frac{S\,S_{xy} - S_x\,S_y}{\Delta}$

(6)

$b \ = \ \frac{S_{xx}\,S_y - S_x\,S_{xy}}{\Delta}$

where

(7)

$\Delta \ = \ S\,S_{xx} - (S_x)^2$

In these equations $N$ is the number of measured points being fitted and the $S x x$ , etc. represent the following sums:^[2]

(8)

$S \ = \ \sum_i^N \frac{1}{u_i^2} \quad \quad S_x \ = \ \sum_i^N \frac{x_i}{u_i^2} \quad \quad S_y \ = \ \sum_i^N \frac{y_i}{u_i^2} \quad \quad S_{xx} \ = \ \sum_i^N \frac{x_i^2}{u_i^2} \quad \quad S_{xy} \ = \ \sum_i^N \frac{x_i y_i}{u_i^2}$

Sample Results

For the five points shown in Figure 1, all weighted equally (all $u i$ = 1), the normal equations give the least-squares line as

(9)

$y \ = \ 1.99x + 5.13$

which is shown in Figure 2.

Figure 3. For the same data as in Figure 2, but with uncertainties that increase with

x

, the normal equations (3)-(8) give a least-squares line (10) that differs from the one in Figure 2. Error bars are expanded uncertainties at the 68% confidence level.

Figure 2. Same five points as in Figure 1, with least-squares line (9) computed from normal equations (3)-(8) using equal weights (all

u

= 1). Error bars are expanded uncertainties at the 68% confidence level.

If the data in Figure 1 are not all equally reliable, then the weights will not all be the same and the normal equations will yield some least-squares line other than that in Figure 2. For example, if the uncertainties increase with $x$ and $y$ , as in Figure 3, then the resulting least-squares line is

Figure 4. For the same data as in Figures 2 and 3, but with uncertainties that decrease with increasing

x

, the normal equations (3)-(8) give a least-squares line (11) that differs from both the one in Figure 2 and the one in Figure 3. Error bars are expanded uncertainties at the 68% confidence level.

(10)

$y \ = \ 1.82x + 5.56$

Figure 3 shows that this line lies near the most reliable point (at $x$ = 1.0) and not so near the least reliable point (at $x$ = 5.0).

In contrast, if the uncertainties decrease with increasing $x$ and $y$ , as in Figure 4, then the normal equations give

(11)

$y \ = \ 2.09x + 4.79$

The line in Figure 4 lies close to the reliable point at $x$ = 5.0, and not so close to the less reliable point at $x$ = 1.0. Figures 2-4 show how we can use uncertainties in measured data to bias least-squares fits. In this way we can try to force fits to reproduce those portions of the data that are most reliable.

Comments on interpreting these normal least-squares equations are given on another page.

References

↑ J. M. Haile, Analysis of Data, Macatea Productions, Central, SC, 2003. ISBN 0-9728602-0-7.
↑ ^2.0 ^2.1 W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling, Numerical Recipes, Cambridge University Press, Cambridge, 1986. ISBN 0521308119

Retrieved from "http://www.chelabwiki.org/wiki/Least_Squares:Simple_Fits"

Category: Data Analysis