{"id":1275,"date":"2023-05-31T08:54:35","date_gmt":"2023-05-31T08:54:35","guid":{"rendered":"https:\/\/nag.com\/?page_id=1275"},"modified":"2023-07-14T14:19:05","modified_gmt":"2023-07-14T14:19:05","slug":"limited-memory-nonlinear-conjugate-gradient-solver","status":"publish","type":"page","link":"https:\/\/nag.com\/limited-memory-nonlinear-conjugate-gradient-solver\/","title":{"rendered":"Limited-Memory Nonlinear Conjugate Gradient Solver"},"content":{"rendered":"\n
\n
\n
\n
\n
\n
\n \n

Limited-Memory Nonlinear Conjugate Gradient Solver<\/h1>\n \n
<\/div>\n\n \n <\/div>\n <\/div>\n <\/div>\n <\/div>\n <\/div>\n<\/div>\n\n\n\n
<\/div>\n\n
\n
\n
\n

At Mark 27 n<\/i><\/span>AG introduced a novel solver\u00a0handle_solve_bounds_foas (e04kf)<\/a>\u00a0<\/span>for large-scale unconstrained or bound-constrained nonlinear optimization problems. It offers a significant performance improvement over other available alternatives. It is based on the nonlinear conjugate gradient method. Thanks to its low memory requirements it is suitable even for problems with millions of variables and beyond. It is part of n<\/i><\/span>AG’s ongoing effort to expand and improve its offering in mathematical optimization.<\/p>\n

Large-scale nonlinear optimization problems of the form<\/p>\n

\\[ \\begin{align} \\min f(x)& \\label{p}\\\\ \\text{subject to}&\\quad\\ell\\leq x\\leq u,\\quad x\\in\\mathbb{R}^n, \\nonumber \\end{align} \\]<\/span>(1)<\/p>\n

where\u00a0\\(f(x)\\)<\/span>\u00a0is smooth with first-order derivatives, are quite ubiquitous in real-life applications. This kind of problem becomes quite challenging to solve when the search space is substantially large (\\(n=10^5\\)<\/span>\u00a0or more). It is often the case that evaluating the second-order derivatives (Hessian) is prohibitively expensive (in time or resources) or outright impossible. First-order methods are designed to rely only on first-order derivatives (gradient) and make clever use of gradient information to build a model of\u00a0\\(f(x)\\)<\/span>\u00a0to find good search directions. Methods such as conjugate gradient (CG) or quasi-Newton (QN) have been well studied and are commonly used to solve\u00a0(1).<\/p>\n

Improving on Well-established Methods<\/h2>\n

CG methods have their origin in linear optimization and have been successfully extended to nonlinear optimization. The main aspect that characterize these methods are their low memory requirements and convergence speed. They have gained recent popularity with the introduction of new ways on how to deal with numerical ill-conditioning and the coupling with very fast linesearch algorithms. CG methods represent an attractive compromise between the simple steepest descent method and the more memory hungry QN methods.<\/p>\n

Implementations of first-order methods are not only ubiquitous and have widespread use, but have also demonstrated that they endure the challenges of ever-growing problem size imposed by industry. Most notable are applications in statistics and Machine Learning, e.g. parameter calibration for log-linear models and conditional random fields (\\(\\ell_2\\)<\/span>-regularization).<\/p>\n

One of the most used first-order method is\u00a0L-BFGS-B<\/a><\/span>\u00a0which is a limited-memory bound constrained version of the\u00a0BFGS<\/a>\u00a0<\/span>QN algorithm. The main idea is to approximate the inverse Hessian matrix using a limited amount of direction and gradient vectors to represent implicitly the approximation. Recent CG methods incorporate QN ideas\u00a0[5]\u00a0in which they retain convergence speed and low memory requirement while exploiting curvature information provided by the QN approximation to the Hessian. This combination has made the limited-memory CG a competitive alternative to L-BFGS-B.<\/p>\n

n<\/i><\/span>AG’s\u00a0e04kf<\/a>\u00a0is a modern implementation of a bound-constrained nonlinear CG method that incorporates a limited-memory QN approximation similar in idea to L-BFGS-B. The methods implemented into this new solver have a very low memory footprint and are based on groundbreaking research in the field of first-order methods.<\/p>\n

Features of the New Solver<\/h2>\n