Summary: Linear regression and principal components analysis are examples of plane fitting methods. Plane fitting is a very important activity in multivariate statistical analysis. The geometry of plane fitting is surprisingly complex, but general insights into it can be gained by considering the problem of fitting a line (“one-dimensional plane”) to only three or four bivariate, quantitative data points. Graphical analysis reveals that any line fitting method must have “singularities”, i.e., data sets near which the line fitting method is unstable. (For example, collinear data sets are the singularities of least squares linear regression.)
Singularities can be classified according to the effects they have on the behavior of the line fitting method and those effects can be quantified as well. The dimension of (“degrees of freedom” in) the set of all singularities of a line fitting method is related to the probability of getting a data set near a singularity. These ideas are illustrated in principal component analysis and least squares and least absolute deviation linear regression.