skip to main content
research-article

Extra-Precise Iterative Refinement for Overdetermined Least Squares Problems

Published: 01 February 2009 Publication History

Abstract

We present the algorithm, error bounds, and numerical results for extra-precise iterative refinement applied to overdetermined linear least squares (LLS) problems. We apply our linear system refinement algorithm to Björck’s augmented linear system formulation of an LLS problem. Our algorithm reduces the forward normwise and componentwise errors to O(εw), where εw is the working precision, unless the system is too ill conditioned. In contrast to linear systems, we provide two separate error bounds for the solution x and the residual r. The refinement algorithm requires only limited use of extra precision and adds only O(mn) work to the O(mn2) cost of QR factorization for problems of size m-by-n. The extra precision calculation is facilitated by the new extended-precision BLAS standard in a portable way, and the refinement algorithm will be included in a future release of LAPACK and can be extended to the other types of least squares problems.

References

[1]
Arioli, M., Baboulin, M., and Gratton, S. 2007. A partial condition number for linear least squares problems. SIAM J. Matrix Anal. Appl. 29, 2, 413--433.
[2]
Bailey, D. 2009. A Fortran-90 double-double precision library. http://crd.lbl.gov/~dhbailey/mpdist.
[3]
Björck, Å. 1967. Iterative refinement of linear least squares solutions I. BIT 7, 257--278.
[4]
Björck, Å. 1991. Component-Wise perturbation analysis and error bounds for linear least squares solutions. BIT 31, 238--244.
[5]
Björck, Å. 1996. Numerical Methods for Least Squares Problems. SIAM, Philadelphia, PA.
[6]
Björck, Å. and Golub, G. 1967. Iterative refinement of linear least squares solution by Householder transformation (Algol programming). BIT 7, 322--337.
[7]
Cucker, F., Diao, H., and Wei, Y. 2007. On mixed and componentwise condition numbers for Moore-Penrose inverse and linear least squares problems. Math. Comput. 76, 258, 947--963.
[8]
Demmel, J., Hida, Y., Kahan, W., Li, X., Mukherjee, S., and Riedy, E. 2006. Error bounds from extra-precise iterative refinement. ACM Trans. Math. Softw. 32, 2, 325--351.
[9]
Forsythe, G. E. and Moler, C. B. 1967. Computer Solution of Linear Algebraic Systems. Prentice-Hall, Englewood Cliffs, NJ.
[10]
Gulliksson, M. 1994. Iterative refinement for constrained and weighted linear least squares. BIT 34, 2, 239--253.
[11]
Higham, N. J. 1988. FORTRAN codes for estimating the one-norm of a real or complex matrix, with applications to condition estimation. ACM Trans. Math. Softw. 14, 4, 381--396.
[12]
Higham, N. J. 1996. Accuracy and Stability of Numerical Algorithms. SIAM, Philadelphia, PA.
[13]
Kielbasinski, A. and Schwetlick, H. 1988. Numerische Lineare Algebra: Eine Computerorientierte Einführung. VEB Deutscher, Berlin.
[14]
Langou, J., Langou, J., Luszczek, P., Kurzak, J., Buttari, A., and Dongarra, J. 2006. Exploiting the performance of 32 bit floating point arithmetic in obtaining 64 bit accuracy (revisiting iterative refinement for linear systems). Tech. rep., University of Tennessee, Knoxville, TN. June.
[15]
Li, X. S., Demmel, J. W., Bailey, D. H., Henry, G., Hida, Y., Iskandar, J., Kahan, W., Kang, S. Y., Kapur, A., Martin, M. C., Thompson, B. J., Tung, T., and Yoo, D. J. 2002. Design, implementation and testing of extended and mixed precision BLAS. ACM Trans. Math. Softw. 28, 2, 152--205.
[16]
Oettli, W. and Prager, W. 1964. Compatibility of approximate solution of linear equations with given error bounds for coefficients and right-hand sides. Numer. Math. 6, 405--409.
[17]
Reid, J. 2000. Implicit scaling of linear least squares problems. BIT 40, 1, 146--157.
[18]
Rigal, J. and Gaches, J. 1967. On the compatibility of a given solution with the data of a linear system. J. ACM 14, 3, 543--548.
[19]
Stewart, G. W. 1977. On the perturbation of pseudo-inverses, projections and linear least squares problems. SIAM Rev. 19, 4, 634--662.
[20]
Walden, B., Karlson, R., and J., S. 1995. Optimal backward perturbation bounds for the linear least squares problem. Numer. Lin. Alg. Appl. 2, 3, 271--286.
[21]
Wampler, R. H. 1970. A report on the accuracy of some widely used least squares computer programs (in applications). J. Amer. Statist. Assoc. 65, 330, 549--565.

Cited By

View all
  • (2024)Mixed precision Rayleigh quotient iteration for total least squares problemsNumerical Algorithms10.1007/s11075-023-01665-z96:2(777-798)Online publication date: 1-Jun-2024
  • (2023)Newly Released Capabilities in the Distributed-Memory SuperLU Sparse Direct SolverACM Transactions on Mathematical Software10.1145/357719749:1(1-20)Online publication date: 21-Mar-2023
  • (2023)New regularization techniques for ill-conditioning problems and their applications: Choices of regularization parametersEngineering Analysis with Boundary Elements10.1016/j.enganabound.2023.04.013152(347-361)Online publication date: Jul-2023
  • Show More Cited By

Recommendations

Reviews

Jesse Louis Barlow

The algorithm for iterative refinement of least squares given here is an update of the 1967 work of Bj?rck [1], with more recent ideas about componentwise errors and condition estimation. These ideas influenced stopping criteria, the use of the Hager-Higham condition estimator [2,3] to estimate relevant conditioning parameters, and the discussion of the role of extra-precision. Iterative refinement cannot be expected to improve the solutions of problems that are ill-conditioned, but can provide significant improvement for non-ill-conditioned problems, which are called "acceptably conditioned" here. The authors state two goals for the refinement algorithm: "for acceptably conditioned problems [to] obtain small errors," and that "a small error bound returned from the code [be] trustworthy." One new theorem is proven, but it is similar to bounds that have appeared in previous work by some of the authors [4]; the new results of the paper are about the numerical tests of the software. The authors do one million test problems, of which 577,412 are acceptably conditioned by standard normwise measures. All but 35 of these problems converged according to their stopping criteria, and for those the condition number in the infinity norm is larger than or close to the condition threshold. Thus, they could arguably be put in the ill-conditioned category. For the convergent problems, the algorithm delivered tiny errors for the solution and least squares residuals, both normwise and componentwise-so the first goal was achieved. The second goal was also achieved for the acceptably conditioned problems-they were close to actual error and did not underestimate true errors. The software described is useful for providing the best solution that can be reasonably expected to acceptably conditioned least squares problems. Online Computing Reviews Service

Access critical reviews of Computing literature here

Become a reviewer for Computing Reviews.

Comments

Information & Contributors

Information

Published In

cover image ACM Transactions on Mathematical Software
ACM Transactions on Mathematical Software  Volume 35, Issue 4
February 2009
108 pages
ISSN:0098-3500
EISSN:1557-7295
DOI:10.1145/1462173
Issue’s Table of Contents
Permission to make digital or hard copies of all or part of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from [email protected]

Publisher

Association for Computing Machinery

New York, NY, United States

Publication History

Published: 01 February 2009
Accepted: 01 June 2008
Revised: 01 May 2008
Received: 01 May 2007
Published in TOMS Volume 35, Issue 4

Permissions

Request permissions for this article.

Check for updates

Author Tags

  1. BLAS
  2. LAPACK
  3. Linear algebra
  4. floating-point arithmetic

Qualifiers

  • Research-article
  • Research
  • Refereed

Funding Sources

Contributors

Other Metrics

Bibliometrics & Citations

Bibliometrics

Article Metrics

  • Downloads (Last 12 months)19
  • Downloads (Last 6 weeks)3
Reflects downloads up to 21 Oct 2024

Other Metrics

Citations

Cited By

View all
  • (2024)Mixed precision Rayleigh quotient iteration for total least squares problemsNumerical Algorithms10.1007/s11075-023-01665-z96:2(777-798)Online publication date: 1-Jun-2024
  • (2023)Newly Released Capabilities in the Distributed-Memory SuperLU Sparse Direct SolverACM Transactions on Mathematical Software10.1145/357719749:1(1-20)Online publication date: 21-Mar-2023
  • (2023)New regularization techniques for ill-conditioning problems and their applications: Choices of regularization parametersEngineering Analysis with Boundary Elements10.1016/j.enganabound.2023.04.013152(347-361)Online publication date: Jul-2023
  • (2022)A Computational Study of Using Black-box QR Solvers for Large-scale Sparse-dense Linear Least Squares ProblemsACM Transactions on Mathematical Software10.1145/349452748:1(1-24)Online publication date: 16-Feb-2022
  • (2022)Mixed precision algorithms in numerical linear algebraActa Numerica10.1017/S096249292200002231(347-414)Online publication date: 9-Jun-2022
  • (2022)Multistage mixed precision iterative refinementNumerical Linear Algebra with Applications10.1002/nla.243429:4Online publication date: 22-Feb-2022
  • (2021)The Verification of the Least Square Solution of an Overdetermined Linear SystemAdvances in Applied Mathematics10.12677/AAM.2021.10516910:05(1598-1606)Online publication date: 2021
  • (2020)Three-Precision GMRES-Based Iterative Refinement for Least Squares ProblemsSIAM Journal on Scientific Computing10.1137/20M131682242:6(A4063-A4083)Online publication date: 17-Dec-2020
  • (2018)Augmented Arithmetic Operations Proposed for IEEE-754 20182018 IEEE 25th Symposium on Computer Arithmetic (ARITH)10.1109/ARITH.2018.8464813(45-52)Online publication date: Jun-2018
  • (2018)Matrix MethodsComputational Methods in Physics10.1007/978-3-319-78619-3_3(121-186)Online publication date: 22-Jun-2018
  • Show More Cited By

View Options

Get Access

Login options

Full Access

View options

PDF

View or Download as a PDF file.

PDF

eReader

View online with eReader.

eReader

Media

Figures

Other

Tables

Share

Share

Share this Publication link

Share on social media