{"id":1430,"date":"2021-03-16T14:56:00","date_gmt":"2021-03-16T14:56:00","guid":{"rendered":"https:\/\/nag.com\/?post_type=insights&p=867"},"modified":"2023-07-05T12:00:45","modified_gmt":"2023-07-05T12:00:45","slug":"the-right-tool-for-the-job-dense-v-sparse","status":"publish","type":"insights","link":"https:\/\/nag.com\/insights\/the-right-tool-for-the-job-dense-v-sparse\/","title":{"rendered":"The Right Tool for the Job: Dense v Sparse"},"content":{"rendered":"
\n
\n
\n

Finding the most suitable solver for your problem can be challenging, but don\u2019t worry! It\u2019s not like finding a needle in a haystack as the decision trees in\u00a0Section 5 of n<\/i><\/span>AG Optimization E04 Chapter Introduction<\/a>\u00a0offer invaluable help. The decision tree will take you through a series of questions that will guide you to the correct solver, starting with<\/p>\n

Is the problem sparse\/large-scale?<\/b><\/p>\n

In this blog, we will demonstrate two types of optimization problem to show the importance of this question and provide guidance on how to answer it.<\/p>\n

What is sparsity?<\/h3>\n

We call a matrix sparse if its entries are mostly zero. The data representation for a sparse problem has the form of a sparse matrix which usually defines the linear constraint matrix or Jacobian matrix of the nonlinear constraints. A large-scale problem usually has a fairly large number of variables and constraints, yet only a small number of the constraints involve all variables, and most constraints depend on only small subsets of the variables. Thus, the problem is sparse. Figure 1 depicts the sparsity pattern of a sparse matrix of dimension \\(n=3948\\)<\/b> we can see it is fairly sparse as the \\(density = nnz \/ n2 = 0.756\\%\\) <\/strong>is below \\(1\\%\\).<\/strong><\/p>\n

Capturing and passing the sparse structure of a problem, if it exists, to the right solver is essential as<\/span>\u00a0the specialized solver takes advantage of the sparse structure of the matrix and avoids wasting<\/span>\u00a0computation and memory on the zeros<\/span><\/strong>.<\/p>\n

\"\"<\/p>\n

Impact of sparsity on Cholesky factorization<\/h3>\n

To highlight the importance of exploiting the sparsity of a problem, let\u2019s see an example of\u00a0Cholesky factorization<\/a>\u00a0which is widely used in numerical algorithms such as Interior Point Method (IPM) in convex optimization. All large-scale optimization solvers in the world exploit the sparsity pattern very carefully during matrix factorization to achieve both numerical stability and efficiency.<\/p>\n

The Cholesky factorization of a real symmetric positive definite matrix \\(M\\)<\/strong> is to find a factor \\(L\\) such that \\(M = LL^T\\)<\/span><\/span><\/strong>, where \\(L\\)<\/strong> is a lower triangular matrix. We have selected 9 positive definite matrices in the Matrix Market [1] that arise from disciplines such as structural engineering and structural mechanics. As shown in Table 1, all of the matrices are fairly sparse with density of less than \\(1.5\\%\\).<\/strong><\/p>\n\n\n\n\n\n\n\n\n\n\n\n\n
Prob. ID<\/td>\nMatrix<\/td>\n\\(n\\)<\/strong><\/td>\n\\(nnz\\)<\/strong><\/td>\nDensity(%)<\/td>\n<\/tr>\n
1<\/td>\nbcsstk15<\/td>\n3948<\/td>\n117816<\/td>\n0.756<\/td>\n<\/tr>\n
2<\/td>\nbcsstk16<\/td>\n4884<\/td>\n290378<\/td>\n1.217<\/td>\n<\/tr>\n
3<\/td>\ns1rmt3m1<\/td>\n5489<\/td>\n217651<\/td>\n0.722<\/td>\n<\/tr>\n
4<\/td>\ns2rmt3m1<\/td>\n5489<\/td>\n217681<\/td>\n0.722<\/td>\n<\/tr>\n
5<\/td>\ns1rmq4m1<\/td>\n5489<\/td>\n262411<\/td>\n0.871<\/td>\n<\/tr>\n
6<\/td>\ns3rmq4m1<\/td>\n5489<\/td>\n262943<\/td>\n0.873<\/td>\n<\/tr>\n
7<\/td>\ns2rmq4m1<\/td>\n5489<\/td>\n263351<\/td>\n0.874<\/td>\n<\/tr>\n
8<\/td>\nbcsstk17<\/td>\n10974<\/td>\n428650<\/td>\n0.356<\/td>\n<\/tr>\n
9<\/td>\nbcsstk18<\/td>\n11948<\/td>\n149090<\/td>\n0.104<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table\u00a01<\/strong>:\u00a0Statistics on 9 sparse matrices used to test dense and sparse Cholesky factorization. \\(n\\)<\/strong> number of variables, \\(nnz\\) <\/strong><\/span>number of nonzeros, density:= <\/span>\\(nnz \/ n^2)\\)<\/strong>\u00a0<\/span><\/p>\n

As discussed before, storing only nonzero entries will reduce memory requirements and exploiting the sparsity pattern will reduce the computational cost greatly. To illustrate this point, we have tested both sparse and dense Cholesky routines on the selected dataset and the result is shown in Figure 2. As we can see, overall the sparse Cholesky is faster than the dense one for all test cases. The reason is not hard to imagine as\u00a0the dense algorithm is<\/span>\u00a0linked to the dimension of the matrix, whereas the sparse algorithm is mainly connected to<\/span>\u00a0the number of nonzeros<\/span><\/strong>. A huge loss in performance from the dense algorithm appears for problem 8 and 9 <\/span><\/span><\/span><\/span>due to the big jump in problem size, yet from its sparse counterpart, there is uniformly good performance as the density remains pretty low across all test cases.<\/p>\n

\"sparse<\/p>\n

Figure\u00a02<\/strong>:\u00a0Computational time comparison between sparse and dense Cholesky<\/span><\/p>\n

To further illustrate how sparsity affects the performance of an algorithm, we randomly generated symmetric positive definite matrices of size 1000, 3000 and 5000. For matrices of a certain size, the density varies from 10% to 100%. Then both sparse and dense Cholesky were called to factorize the matrices and the computational time comparison can be found in Figure\u00a03. Unsurprisingly, for matrices of the same size, the time for dense Cholesky doesn\u2019t vary too much even as the density increases. However, sparse Cholesky uses more and more time as the matrix gets denser, and at some point, it will require more computational time than the dense algorithm.\u00a0For matrices with relatively high density (such as 20% and above), the tradeoff is no longer viable and the dense counterpart becomes a better choice<\/strong>.<\/p>\n

\"Comparison<\/p>\n

Figure\u00a03:\u00a0<\/strong>Comparison of factorization methods on matrices of size 1000, 3000, 5000: various density<\/span><\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n

<\/div>\n\n\n
\n
\n
\n

Example in Linear Programming (LP)<\/h3>\n

A general LP problem has a standard form:<\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n\n \n \n \n \n \n \n minimize<\/mtext>\n <\/mstyle>\n <\/mrow>\n \n x<\/mi>\n ∈<\/mo>\n \n \n ℜ<\/mi>\n <\/mrow>\n \n n<\/mi>\n <\/mrow>\n <\/msup>\n <\/mrow>\n <\/munder>\n <\/mtd>\n \n \n \n c<\/mi>\n <\/mrow>\n \n T<\/mi>\n <\/mrow>\n <\/msup>\n x<\/mi>\n <\/mtd>\n <\/mtr>\n \n \n \n <\/mtr>\n \n \n \n subject to<\/mtext>\n <\/mstyle>\n <\/mtd>\n \n A<\/mi>\n x<\/mi>\n =<\/mo>\n b<\/mi>\n ,<\/mo>\n <\/mtd>\n <\/mtr>\n \n \n \n <\/mtr>\n \n \n \n x<\/mi>\n ≥<\/mo>\n 0<\/mn>\n .<\/mo>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n<\/math>\n\n\n
\n
\n
\n

Here we used two active-set methods\u00a0lp<\/span>_solve<\/span><\/a>\u00a0(e04mf<\/span><\/a>) (dense solver) and\u00a0qpconvex2<\/span>_sparse<\/span>_solve<\/span><\/a>\u00a0(e04nq<\/span><\/a>) (sparse solver) from the\u00a0n<\/i><\/span>AG Library<\/a> to solve the above model and the results are shown in Table 2.\u00a0Both test cases have a density less than\u00a01<\/span>.<\/span>0<\/span>%<\/span><\/span><\/span>\u00a0<\/mn>
<\/mi><\/math><\/span><\/span>and e04nq<\/span><\/a>\u00a0spent less than 1 second for each case while the dense solver was significantly slower. Therefore,\u00a0passing a<\/span>\u00a0sparse problem to a dense solver will completely kill your performance<\/span><\/strong>.<\/p>\n\n\n\n\n\n\n
Problem<\/td>\nn\u00a0 \u00a0 m<\/td>\nnnz\u00a0 Density<\/td>\ne04mf<\/td>\ne04nq<\/td>\n<\/tr>\n
\u00a0<\/td>\n\u00a0<\/td>\n\u00a0<\/td>\nIter\u00a0 \u00a0Time<\/td>\nIter\u00a0 \u00a0 Time<\/td>\n<\/tr>\n
25fv47<\/td>\n1571\u00a0 821<\/td>\n11127\u00a0 \u00a00.86%<\/td>\n8030\u00a0 36.9s<\/td>\n6923\u00a0 0.46s<\/td>\n<\/tr>\n
bnl2<\/td>\n3489\u00a0 \u00a02324<\/td>\n16124\u00a0 \u00a00.2%<\/td>\n7568. 229.1s<\/td>\n4560\u00a0 0.58s<\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n <\/div>\n <\/div>\n<\/div>\n\n\n
<\/div>\n\n\n
\n
\n
\n

Be prepared for the surprise from dedicated solvers<\/h3>\n

Recently, a\u00a0n<\/i><\/span>AG Library<\/a>\u00a0user contacted our\u00a0Technical Support team<\/a> with the following \\(1_{1}\\)<\/strong>\u00a0
<\/mn><\/mrow><\/msub><\/math><\/span><\/span>linear fitting problem<\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n\n \n \n \n \n \n \n minimize<\/mtext>\n <\/mstyle>\n <\/mrow>\n \n x<\/mi>\n ∈<\/mo>\n \n \n ℜ<\/mi>\n <\/mrow>\n \n n<\/mi>\n <\/mrow>\n <\/msup>\n <\/mrow>\n <\/munder>\n <\/mtd>\n \n ∥<\/mo>\n A<\/mi>\n x<\/mi>\n –<\/mo>\n b<\/mi>\n \n \n ∥<\/mo>\n <\/mrow>\n \n 1<\/mn>\n <\/mrow>\n <\/msub>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n<\/math>\n\n\n
\n
\n
\n

where matrix \\(A\\)<\/strong> is tall and slim, e.g., of dimension 1000 x 21 and full dense matrix. At first glance, it might appear to be a dense problem as there are no zeros in the problem data \\(A\\) <\/strong>and \\(b\\). <\/strong>The \\(1_{1}\\) <\/strong>norm minimization can be reformulated to Linear Programming, so the user performed the transformation and used active-set methods lp_solve<\/a> (e04mf<\/a>) (dense solver). However, the solver was not as fast as he expected and the user sought our advice.<\/p>\n

This type of misunderstanding and confusion is not unusual and care needs to be taken. In order to decide whether the problem is sparse or dense, we need to consider the data of the problem model that the solver solves, not the original data. For instance, problem 2 can be reformulated into LP problem<\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n\n \n \n \n \n \n \n minimize<\/mtext>\n <\/mstyle>\n <\/mrow>\n \n x<\/mi>\n ∈<\/mo>\n \n \n ℜ<\/mi>\n <\/mrow>\n \n n<\/mi>\n <\/mrow>\n <\/msup>\n ,<\/mo>\n y<\/mi>\n ∈<\/mo>\n \n \n ℜ<\/mi>\n <\/mrow>\n \n m<\/mi>\n <\/mrow>\n <\/msup>\n <\/mrow>\n <\/munder>\n <\/mtd>\n \n \n \n ∑<\/mo>\n <\/mrow>\n \n i<\/mi>\n <\/mrow>\n <\/munder>\n \n \n y<\/mi>\n <\/mrow>\n \n i<\/mi>\n <\/mrow>\n <\/msub>\n <\/mtd>\n <\/mtr>\n \n \n \n <\/mtr>\n \n \n \n subject to<\/mtext>\n <\/mstyle>\n <\/mtd>\n \n –<\/mo>\n y<\/mi>\n ≤<\/mo>\n A<\/mi>\n x<\/mi>\n –<\/mo>\n b<\/mi>\n ≤<\/mo>\n y<\/mi>\n ,<\/mo>\n <\/mtd>\n <\/mtr>\n \n \n \n <\/mtr>\n \n \n \n y<\/mi>\n ≥<\/mo>\n 0<\/mn>\n .<\/mo>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n<\/math>\n\n\n
\n
\n
\n

Taking a closer look at the inequality constraints<\/p>\n <\/div>\n <\/div>\n<\/div>\n\n\n\n –<\/mo>\n y<\/mi>\n ≤<\/mo>\n A<\/mi>\n x<\/mi>\n –<\/mo>\n b<\/mi>\n ≤<\/mo>\n y<\/mi>\n \n ⇔<\/mo>\n \n \n \n \n \n \n A<\/mi>\n <\/mtd>\n \n –<\/mo>\n I<\/mi>\n <\/mtd>\n <\/mtr>\n \n \n –<\/mo>\n A<\/mi>\n <\/mtd>\n \n –<\/mo>\n I<\/mi>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n <\/mrow>\n <\/mfenced>\n \n \n \n \n \n x<\/mi>\n <\/mtd>\n <\/mtr>\n \n \n y<\/mi>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n <\/mrow>\n <\/mfenced>\n ≤<\/mo>\n \n \n \n \n \n b<\/mi>\n <\/mtd>\n <\/mtr>\n \n \n –<\/mo>\n b<\/mi>\n <\/mtd>\n <\/mtr>\n <\/mtable>\n <\/mrow>\n <\/mfenced>\n ,<\/mo>\n<\/math>\n\n\n
\n
\n
\n

where \\(I\\)\u00a0<\/strong>is the identity matrix of dimension \\(m\\)<\/strong>, we can see the number of variables has increased by \\(m\\) <\/strong>and many zero entries have been introduced to the model. Thus, a sparse LP solver should be chosen for this particular model. We tested it with a matrix \\(A\\)\u00a0<\/strong>of dimension 1000 x 21 using the active-set method solver lp_solve<\/a> (e04mf<\/a>) and sparse Interior Point Method solver handle_solve_lp_ipm<\/a> (e04mt<\/a>) from the n<\/i><\/span>AG Library<\/a>. As we can see in Table 3, the sparse LP solver again shows great performance compared to the dense solver.<\/p>\n\n\n\n\n
Solver<\/td>\ne04mf<\/td>\ne04mt<\/td>\ne02ga<\/td>\n<\/tr>\n
Time<\/td>\n49.7s<\/td>\n0.18s<\/span><\/strong><\/td>\n0.007s<\/span><\/strong><\/td>\n<\/tr>\n<\/tbody>\n<\/table>\n

Table\u00a03<\/strong>: Three solvers for \\(1_{1}\\)<\/strong> linear fitting problem<\/p>\n

Clearly, the sparse solver is the reason for the improved computation time. However, the dedicated solver glin_l1sol<\/a> (e02ga<\/a>) for problem (2) did an even better job than the sparse LP solver, since it exploits thoroughly the known problem structure.\u00a0Overall the dedicated solver delivered a 7100x speedup over the original attempt. <\/span><\/strong>Therefore, it is always recommended to try a dedicated solver first for a particular problem. Dedicated solvers are slightly out of the scope of this blog, please see\u00a0<\/span>n<\/i><\/span>AG Library documentation<\/a>\u00a0for the full content of n<\/i><\/span>AG offerings in dedicated solvers.<\/span><\/p>\n

Things to remember<\/h3>\n

Sparsity is a great feature and should be considered carefully.<\/p>\n