{"id":1447,"date":"2020-06-05T15:34:00","date_gmt":"2020-06-05T15:34:00","guid":{"rendered":"https:\/\/nag.com\/?post_type=insights&p=1249"},"modified":"2023-07-04T17:11:27","modified_gmt":"2023-07-04T17:11:27","slug":"iterative-methods-for-toeplitz-systems","status":"publish","type":"insights","link":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/","title":{"rendered":"Iterative Methods for Toeplitz Systems"},"content":{"rendered":"
\n
\n
\n

Blog Author:\u00a0Dr Jennifer Pestana<\/a>\u00a0– Mathematics and Statistics Lecturer, University of Strathclyde<\/h4>\n

Linear systems involving Toeplitz matrices arise in many applications, including differential and integral equations and signal and image processing (see, e.g.,\u00a0this article<\/a>\u00a0and the books by\u00a0Ng<\/a>, and\u00a0Chan and Jin<\/a>). More recently, Toeplitz systems have appeared in\u00a0discretisations of fractional diffusion problems<\/a>. This is because fractional diffusion operators are non-local, and lead to dense matrices; if these dense matrices are Toeplitz, it’s possible to develop\u00a0fast solvers<\/a>.<\/p>\n

Here, we show that in addition to these tailored direct approaches, preconditioned iterative methods can be competitive for these problems. Perhaps surprisingly, this is true even when the Toeplitz matrix is dense.<\/p>\n

Toeplitz Linear Systems<\/h3>\n

We want to solve a linear system\u00a0\\(Ax=b\\)<\/span>, where\u00a0\\(A \\in \\mathbb{R}^{n\\times n}\\)<\/span>\u00a0is a\u00a0Toeplitz matrix<\/a>, i.e., a matrix with constant diagonals. Since\u00a0\\(A\\)<\/span>\u00a0has (at most)\u00a0\\(2n-1\\)<\/span>\u00a0degrees of freedom, it’s perhaps not surprising that we can solve this linear system with\u00a0\\(O(n^2)\\)<\/span>\u00a0or fewer flops, using fast or\u00a0superfast\u00a0solvers. In fact, if\u00a0\\(A\\)<\/span>\u00a0is symmetric positive definite, we can apply the n<\/i><\/span>AG routine f04fff<\/a>.<\/p>\n

But what if our Toeplitz matrix is nonsymmetric? We would still like a fast solver, but in this case codes for fast and superfast direct solvers aren’t so easy to come by. As we shall see, an alternative is to instead apply preconditioned iterative methods. (Note that these methods can also be applied if\u00a0\\(A\\)<\/span>\u00a0is symmetric.)<\/p>\n

Let’s first set up our nonsymmetric problem. This one happens to be a\u00a0lower Hessenberg matrix<\/a>\u00a0from a fractional diffusion problem (Example 4.2 from\u00a0this paper<\/a>), but there’s nothing particularly special about it.<\/p>\n

In [1]:<\/div>\n <\/div>\n <\/div>\n<\/div>\n\n
\n
\n
\n 
from naginterfaces.library.lapacklin import dgesv    # A general solver\nfrom naginterfaces.library.sparse import real_gen_basic_setup, real_gen_basic_solver, real_gen_basic_diag # Nonsymmetric iterative method codes\nfrom naginterfaces.library.sparse import real_symm_basic_setup, real_symm_basic_solver, real_symm_basic_diag # Symmetric iterative method codes\nfrom pytictoc import TicToc # For timing\nimport numpy as np\nimport scipy.linalg\nfrom scipy.fftpack import fft, ifft # For fast matrix-vector products\n\ntimer = TicToc()\n\n# Construct a real, nonsymmetric Toeplitz matrix \nmatrix_size = 10000\nalpha=1.3;\ng=np.zeros(matrix_size+1)\ng[0]=1\nfor i in range(matrix_size):\n    g[i+1]=(1-(alpha+1)\/(i+1))*g[i]\ncol = g[1:matrix_size+1]\nrow = np.concatenate([g[1::-1]*np.ones(2),np.zeros(matrix_size-2)])\n\nA = scipy.linalg.toeplitz(col,row)\n\n# Construct a right-hand side \nx = np.random.rand((matrix_size))\nx = x\/np.sqrt(matrix_size)\nb = np.matmul(A,x)\nb_nag_gen = np.reshape(b,[matrix_size,1]) # Reshape b for the generic n<\/i><\/span>AG solver<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Let’s start by applying a\u00a0general-purpose solver<\/a>:<\/p>\n
\n
In [2]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
\n
timer.tic()\n[asol,ipiv,x_direct] = dgesv(A, b_nag_gen)\ntimer.toc()\nt_direct = timer.tocvalue()\n\n# This next line will enable us to compare with the other solution vectors we obtain\nx_direct = np.reshape(x_direct,matrix_size) <\/code><\/pre>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Elapsed time is 15.417057 seconds.<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Krylov Subspace Methods<\/h3>\n
We want to see if we can do better by exploiting the Toeplitz structure. Here, we’ll use\u00a0Krylov subspace methods<\/a>. Since these are iterative methods, the time they take to run depends on both the cost per iteration and the number of iterations.<\/p>\n
One of the main contributors to the cost per iteration is the computation of one or two matrix-vector products with\u00a0\\(A\\)<\/span>\u00a0at each iteration. This is why Krylov methods are usually used for sparse matrices. However, if\u00a0\\(A\\)<\/span>\u00a0is Toeplitz, then regardless of whether it is sparse or dense, matrix-vector products can be performed quickly via a\u00a0circulant embedding and the fast Fourier transform<\/a>.<\/p>\n
Let’s see this in action. To do so, we need to set up a couple of vectors that store the eigenvalues of the circulant embedding matrices. (We have one for\u00a0\\(A\\)<\/span>\u00a0and one for\u00a0\\(A^T\\)<\/span>\u00a0since both will be needed by the iterative solver.)<\/p>\n
\n
In [3]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Avec = fft(np.concatenate([col,np.zeros(1),row[matrix_size:0:-1]]))\nATvec = fft(np.concatenate([row,np.zeros(1),col[matrix_size:0:-1]]))<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
We’ll first test the speed of the usual matrix-vector product with\u00a0\\(A\\)<\/span>:<\/p>\n
\n
In [4]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
# Set up a random vector\nnp.random.seed(2)\ny = np.random.rand(matrix_size)\n\n# Perform the usual matrix-vector product\ntimer.tic()\nz_slow = A.dot(y)\ntimer.toc()\nt_slow = timer.tocvalue()<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Elapsed time is 0.054436 seconds.<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
And now the fast matrix-vector product:<\/p>\n
\n
In [5]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
timer.tic()\nz = ifft(np.multiply(Avec,fft(np.concatenate([y,np.zeros(matrix_size)]))))\nz_fast = np.asarray(z.real[0:matrix_size], order='C')\ntimer.toc()\nt_fast = timer.tocvalue()<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Elapsed time is 0.009971 seconds.<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
which is quite a bit faster:<\/p>\n
\n
In [6]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
t_slow\/t_fast<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[6]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
5.1808748916460114<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
The difference in speed becomes more pronounced as the dimension\u00a0\\(n\\)<\/span>\u00a0increases since a generic matrix-vector product is\u00a0\\(O(n^2)\\)<\/span>\u00a0while the fast Toeplitz product is\u00a0\\(O(n\\log n)\\)<\/span>. To check that we’re not compromising on accuracy, we can look at\u00a0\\(\\|z_{\\text{fast}} – z_{\\text{slow}}\\|_2\\)<\/span>, from which we see that the fast Toeplitz matrix-vector product is pretty accurate.<\/p>\n
In [7]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
np.linalg.norm(z_fast - z_slow)<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[7]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
3.1262523228454646e-14<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Preconditioning<\/h3>\n
The number of iterations of our Krylov method depends on\u00a0\\(A\\)<\/span>,\u00a0\\(b\\)<\/span>\u00a0and the initial guess of\u00a0\\(x\\)<\/span>\u00a0in a highly nonlinear manner. However, for our problem (and many other Toeplitz problems) the number of iterations needed for an acceptably accurate solution is high.<\/p>\n
To reduce the number of iterations needed it is typical to\u00a0precondition<\/a>, i.e., to change the linear system to an equivalent one with “better” properties. Left preconditioning replaces the original system by\u00a0\\(P^{-1}Ax = P^{-1}b\\)<\/span>, for some invertible matrix\u00a0\\(P \\in \\mathbb{R}^{n\\times n}\\)<\/span>, but right preconditioning, and preconditioning that preserves symmetry, are possible.<\/p>\n
There are many preconditioners for Toeplitz systems, a number of which are based on properties of a scalar-valued function known as the generating function or symbol. More information on suitable preconditioners can be found in the books by Ng, and Chan and Jin, mentioned at the start of this post.<\/p>\n
We’re using the\u00a0Strang preconditioner<\/a>\u00a0here. To apply this preconditioner quickly, we’ll make use of scipy’s\u00a0fast circulant solve<\/a>\u00a0(which is again based on the fast Fourier transform). This means we only need to store the first column,\u00a0\\(c\\)<\/span>, of our preconditioner.<\/p>\n
In [8]:<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
midpoint = np.uintc(np.floor(matrix_size\/2))\nc = np.concatenate([col[0:midpoint+1],row[matrix_size-midpoint-1:0:-1]])<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Now we’re ready to apply our Krylov subspace method. Here we’ll use n<\/i><\/span>AG’s left-preconditioned restarted GMRES (GMRES(50)). However, since we will need fewer than 50 iterations, RGMRES is equivalent to standard left-preconditioned GMRES.<\/p>\n
The termination criterion is the default one in the n<\/i><\/span>AG routine\u00a0f11bdf<\/a>\u00a0with a tolerance of\u00a0\\(10^{-8}\\).<\/span> The n<\/i><\/span>AG iterative solvers use reverse communication; for more information see the\u00a0documentation<\/a>.<\/p>\n
In [9]:<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
timer.tic()\n\n# Settings for RGMRES\nmethod = 'RGMRES'\nprecon = 'P'\nm = 50\ntol = 1e-8\nmaxitn = 10\nanorm = 0\nsigmax = 0\nmonit = -1\n\n# Initialisation routine\ncomm = real_gen_basic_setup(method, precon, matrix_size, m, tol, maxitn, anorm, sigmax, monit, norm=None, weight='N', iterm=1)\n\nirevcm = 0;\nu = np.zeros(matrix_size)\nv = b\nwgt = np.zeros(matrix_size)\n\n# GMRES solver\nwhile (irevcm != 4):\n    irevcm = real_gen_basic_solver(irevcm,u,v,wgt,comm)\n    if irevcm == -1: # v = A^T*u     \n        y = ifft(np.multiply(ATvec,fft(np.concatenate([u,np.zeros(matrix_size)]))))\n        v = np.asarray(y.real[0:matrix_size], order='C')\n    elif irevcm == 1: # v = A*y\n        y = ifft(np.multiply(Avec,fft(np.concatenate([u,np.zeros(matrix_size)]))))\n        v = np.asarray(y.real[0:matrix_size], order='C')\n    elif irevcm == 2:\n        v = np.asarray(scipy.linalg.solve_circulant(c,u), order='C')\n    elif irevcm == 3:\n        [itn,stplhs,stprhs,anorm,sigmax] = real_gen_basic_diag(comm)\n\n[itn,stplhs,stprhs,anorm,sigmax] = real_gen_basic_diag(comm)\ntimer.toc()\nt_gmres = timer.tocvalue()   \n\nx_gmres = u # store the approximate solution<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Elapsed time is 0.074340 seconds.<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
This is quite a lot faster than the general solver:<\/p>\n
In [10]:<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
t_direct\/t_gmres<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[10]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
206.8624024704361<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
We can also see how many GMRES iterations were performed:<\/p>\n
In [11]:<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
itn<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[11]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
10<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
and the relative residual norm at termination (\\(\\|b-Ax_{\\text{gmres}}\\|_2\/\\|b\\|_2\\)<\/span>)<\/p>\n
\n
In [12]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
np.linalg.norm(b-np.matmul(A,x_gmres))\/np.linalg.norm(b)<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out<\/span>\u00a0[12]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
2.8162012822505773e-14<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
which looks pretty good! Now let us compare our solution with the one obtained by the general purpose solver. We’ll measure the relative error,\u00a0\\(\\|x_{\\text{direct}} – x_{\\text{gmres}}\\|_2\/\\|x_{\\text{direct}}\\|_2\\)<\/span>:<\/p>\n
\n
In [13]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
np.linalg.norm(x_direct - x_gmres)\/np.linalg.norm(x_direct)<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[13]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
1.1140269961814175e-11<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
so the two solutions are very close! This shows that, for Toeplitz systems, Krylov subspace methods are certainly worth considering.<\/p>\n
Symmetrisation of Toeplitz matrices<\/h3>\n
For Toeplitz problems we have another trick up our sleeve: we can transform our nonsymmetric Toeplitz matrix to a symmetric\u00a0Hankel<\/a>\u00a0matrix by\u00a0flipping the rows (or columns)<\/a>\u00a0of\u00a0\\(A\\)<\/span>. Mathematically, we solve\u00a0\\(YAx = Yb\\)<\/span>\u00a0(or\u00a0\\(AYz = b\\)<\/span>,\u00a0\\(z = Yx\\)<\/span>), where\u00a0\\(Y\\)<\/span>\u00a0is the\u00a0reverse identity matrix<\/a>.<\/p>\n
The advantage of symmetrising is that we can use a Krylov subspace method with some nice properties. We’ll use MINRES, which is quite similar to GMRES but typically has a lower cost per iteration (and some other nice features).<\/p>\n
To set up, we’ll first flip the right-hand side vector.<\/p>\n
\n
In [14]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Yb = np.asarray(np.flipud(b),order='C')<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
The next thing to sort out is the preconditioner, which for MINRES should be symmetric positive definite (to preserve symmetry). We’ll simply take\u00a0\\((C^TC)^{1\/2}\\)<\/span>, where\u00a0\\(C\\)<\/span>\u00a0is the Strang preconditioner we used above. This is sometimes called the absolute value circulant preconditioner. Below is a cheap way of computing the first column of this circulant, but the details aren’t important.<\/p>\n
In [15]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
c_abs = ifft(np.abs(fft(c)))\nc_abs = c_abs.real<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
We’re now ready to apply n<\/i><\/span>AG’s\u00a0preconditioned MINRES<\/a>\u00a0with the absolute value circulant preconditioner. We’re again using a tolerance of\u00a0\\(10^{-8}\\)<\/span>\u00a0for termination, but note that the stopping criterion is different from the RGMRES one (for details, see the\u00a0documentation<\/a>\u00a0for f11gdf).<\/p>\n
\n
In [16]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
timer.tic()\n# Settings for MINRES\nmethod = 'MINRES'\nprecon = 'P'\ntol = 1e-8\nmaxitn = 500\nanorm = 0\nsigmax = 0\nmaxits = 7\nmonit = -1\n\n# Initialisation routine\n[lwreq,comm] = real_symm_basic_setup(method, precon, matrix_size, tol, maxitn, anorm, sigmax, maxits, monit)\n\nirevcm = 0;\nu = np.zeros(matrix_size)\nv = Yb\nwgt = np.zeros(matrix_size)\n\n# MINRES solver\nwhile (irevcm != 4):\n    irevcm = real_symm_basic_solver(irevcm,u,v,wgt,comm)\n    if irevcm == 1: # v = A*u\n        y =  ifft(np.multiply(Avec,fft(np.concatenate([u,np.zeros(matrix_size)]))))\n        v = np.asarray(np.flipud(y.real[0:matrix_size]),order='C');\n    elif irevcm == 2:\n        v = np.asarray(scipy.linalg.solve_circulant(c_abs,u), order='C')\n    elif irevcm == 3:\n        [itn,stplhs,stprhs,anorm,sigmax,its,sigerr] = real_symm_basic_diag(comm)\n        \n[itn,stplhs,stprhs,anorm,sigmax,its,sigerr] = real_symm_basic_diag(comm)\ntimer.toc()\nt_minres = timer.tocvalue()  \n\nx_minres = u # store the approximate solution<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Elapsed time is 0.049735 seconds.<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
This is also much faster than the direct solver:<\/p>\n
\n
In [17]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
t_direct\/t_minres<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[17]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
307.79616807696027<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
It’s also faster than GMRES because MINRES has a low cost per iteration:<\/p>\n
\n
In [18]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
t_gmres\/t_minres<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out<\/span>\u00a0[18]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
1.4879270684335653<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
It’s also faster than GMRES because MINRES has a low cost per iteration:<\/p>\n
\n
In [19]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
13<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
However, MINRES does require more iterations than GMRES, which highlights that both the cost per iteration and the number of iterations determine the total time of the Krylov subspace method.<\/p>\n
The relative residual norm at termination (\\(\\|b-Ax_{\\text{minres}}\\|_2\/\\|b\\|_2\\)<\/span>) is also quite small, although it’s not as small as the GMRES relative residual norm we obtained:<\/p>\n
\n
In [20]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
np.linalg.norm(b-np.matmul(A,x_minres))\/np.linalg.norm(b)<\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out\u00a0<\/span>[20]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
9.670593217978096e-09<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Looking now at the relative error,\u00a0\\(\\|x_{\\text{direct}} – x_{\\text{minres}}\\|_2\/\\|x_{\\text{direct}}\\|_2\\)<\/span>, we see that this is also small, although again not as small as for GMRES. (Decreasing the tolerance would improve the accuracy of the MINRES solution, at a cost of more iterations.)<\/p>\n
\n
In [21]:<\/div>\n<\/div>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
np.linalg.norm(x_direct - x_minres)\/np.linalg.norm(x_direct)<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Out<\/span>\u00a0[21]:<\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
1.6661088967643173e-06<\/code><\/pre>\n        <\/div>\n    <\/div>\n<\/div>\n\n
\n    
\n        
\n            
Conclusions<\/h3>\n
Direct solvers for Toeplitz matrices can be extremely fast. However, for certain problems, particularly those for which a fast direct method isn’t readily available, iterative solvers can be a great alternative. In the example above, both GMRES and MINRES were orders of magnitude faster than the general-purpose solver.<\/p>\n
Iterative solvers are often daunting for practitioners because they are not “black box” methods. Hopefully, the example above shows that this need not be a barrier; fast matrix-vector products and preconditioners may only require a couple of lines of code. Our experience is that the MINRES method proposed above is typically faster than GMRES when a good preconditioner is available, but other nonsymmetric solvers, such as CGS, BiCGStab or TFQMR may also be good options. The nice thing about the n<\/i><\/span>AG Library is that switching between these iterative methods requires only a few small changes to the code.<\/p>\n
It’s important to note that for some problems the simple preconditioners above may not be so suitable. However, there are now many options available for different types of systems, with the books by\u00a0Ng<\/a>, and\u00a0Chan and Jin<\/a>\u00a0a great place to start.<\/p>\n
NOTE FROM n<\/i><\/span>AG: Toeplitz solver provision is a current area of development research at n<\/i><\/span>AG. If you’re looking for a solver today, see\u00a0‘Real Toeplitz Solve’<\/a>\u00a0for the n<\/i><\/span>AG Library for Python version, and the n<\/i><\/span>AG Library equivalent\u00a0see here<\/a>.<\/strong><\/p>\n        <\/div>\n    <\/div>\n<\/div>\n\n\n
\n    
\n        
\n            
\n                
\n                                
\n    \n                    \n                    
<\/div>\n\n                    \n                     <\/i><\/a>                <\/div>\n                <\/div>\n            <\/div>\n        <\/div>\n    <\/div>\n<\/div>\n","protected":false},"excerpt":{"rendered":"
Linear systems involving Toeplitz matrices arise in many applications, including differential and integral equations and signal and image processing (see, e.g., this article and the books by Ng, and Chan and Jin). More recently, Toeplitz systems have appeared in discretisations of fractional diffusion problems.<\/p>\n","protected":false},"author":16,"featured_media":1250,"parent":0,"menu_order":0,"template":"","meta":{"content-type":"","footnotes":""},"post-tag":[39,18,40],"class_list":["post-1447","insights","type-insights","status-publish","has-post-thumbnail","hentry"],"yoast_head":"\nIterative Methods for Toeplitz Systems - nAG<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Iterative Methods for Toeplitz Systems - nAG\" \/>\n<meta property=\"og:description\" content=\"Linear systems involving Toeplitz matrices arise in many applications, including differential and integral equations and signal and image processing (see, e.g., this article and the books by Ng, and Chan and Jin). More recently, Toeplitz systems have appeared in discretisations of fractional diffusion problems.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/\" \/>\n<meta property=\"og:site_name\" content=\"nAG\" \/>\n<meta property=\"article:modified_time\" content=\"2023-07-04T17:11:27+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"2000\" \/>\n\t<meta property=\"og:image:height\" content=\"1000\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:site\" content=\"@NAGTalk\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/\",\"url\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/\",\"name\":\"Iterative Methods for Toeplitz Systems - nAG\",\"isPartOf\":{\"@id\":\"https:\/\/nag.com\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage\"},\"image\":{\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage\"},\"thumbnailUrl\":\"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg\",\"datePublished\":\"2020-06-05T15:34:00+00:00\",\"dateModified\":\"2023-07-04T17:11:27+00:00\",\"breadcrumb\":{\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage\",\"url\":\"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg\",\"contentUrl\":\"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg\",\"width\":2000,\"height\":1000,\"caption\":\"coding screen\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/nag.com\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Insights\",\"item\":\"https:\/\/nag.com\/insights\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Iterative Methods for Toeplitz Systems\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/nag.com\/#website\",\"url\":\"https:\/\/nag.com\/\",\"name\":\"NAG\",\"description\":\"Robust, trusted numerical software and computational expertise.\",\"publisher\":{\"@id\":\"https:\/\/nag.com\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/nag.com\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\/\/nag.com\/#organization\",\"name\":\"Numerical Algorithms Group\",\"alternateName\":\"NAG\",\"url\":\"https:\/\/nag.com\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\/\/nag.com\/#\/schema\/logo\/image\/\",\"url\":\"https:\/\/nag.com\/wp-content\/uploads\/2023\/11\/NAG-Logo.png\",\"contentUrl\":\"https:\/\/nag.com\/wp-content\/uploads\/2023\/11\/NAG-Logo.png\",\"width\":1244,\"height\":397,\"caption\":\"Numerical Algorithms Group\"},\"image\":{\"@id\":\"https:\/\/nag.com\/#\/schema\/logo\/image\/\"},\"sameAs\":[\"https:\/\/x.com\/NAGTalk\",\"https:\/\/www.linkedin.com\/company\/nag\/\",\"https:\/\/www.youtube.com\/user\/NumericalAlgorithms\",\"https:\/\/github.com\/numericalalgorithmsgroup\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Iterative Methods for Toeplitz Systems - nAG","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/","og_locale":"en_US","og_type":"article","og_title":"Iterative Methods for Toeplitz Systems - nAG","og_description":"Linear systems involving Toeplitz matrices arise in many applications, including differential and integral equations and signal and image processing (see, e.g., this article and the books by Ng, and Chan and Jin). More recently, Toeplitz systems have appeared in discretisations of fractional diffusion problems.","og_url":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/","og_site_name":"nAG","article_modified_time":"2023-07-04T17:11:27+00:00","og_image":[{"width":2000,"height":1000,"url":"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg","type":"image\/jpeg"}],"twitter_card":"summary_large_image","twitter_site":"@NAGTalk","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/","url":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/","name":"Iterative Methods for Toeplitz Systems - nAG","isPartOf":{"@id":"https:\/\/nag.com\/#website"},"primaryImageOfPage":{"@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage"},"image":{"@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage"},"thumbnailUrl":"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg","datePublished":"2020-06-05T15:34:00+00:00","dateModified":"2023-07-04T17:11:27+00:00","breadcrumb":{"@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#primaryimage","url":"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg","contentUrl":"https:\/\/nag.com\/wp-content\/uploads\/2023\/05\/code-night.jpg","width":2000,"height":1000,"caption":"coding screen"},{"@type":"BreadcrumbList","@id":"https:\/\/nag.com\/insights\/iterative-methods-for-toeplitz-systems\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/nag.com\/"},{"@type":"ListItem","position":2,"name":"Insights","item":"https:\/\/nag.com\/insights\/"},{"@type":"ListItem","position":3,"name":"Iterative Methods for Toeplitz Systems"}]},{"@type":"WebSite","@id":"https:\/\/nag.com\/#website","url":"https:\/\/nag.com\/","name":"NAG","description":"Robust, trusted numerical software and computational expertise.","publisher":{"@id":"https:\/\/nag.com\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/nag.com\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/nag.com\/#organization","name":"Numerical Algorithms Group","alternateName":"NAG","url":"https:\/\/nag.com\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/nag.com\/#\/schema\/logo\/image\/","url":"https:\/\/nag.com\/wp-content\/uploads\/2023\/11\/NAG-Logo.png","contentUrl":"https:\/\/nag.com\/wp-content\/uploads\/2023\/11\/NAG-Logo.png","width":1244,"height":397,"caption":"Numerical Algorithms Group"},"image":{"@id":"https:\/\/nag.com\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/x.com\/NAGTalk","https:\/\/www.linkedin.com\/company\/nag\/","https:\/\/www.youtube.com\/user\/NumericalAlgorithms","https:\/\/github.com\/numericalalgorithmsgroup"]}]}},"_links":{"self":[{"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/insights\/1447","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/insights"}],"about":[{"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/types\/insights"}],"author":[{"embeddable":true,"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/users\/16"}],"version-history":[{"count":15,"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/insights\/1447\/revisions"}],"predecessor-version":[{"id":2950,"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/insights\/1447\/revisions\/2950"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/media\/1250"}],"wp:attachment":[{"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/media?parent=1447"}],"wp:term":[{"taxonomy":"post-tag","embeddable":true,"href":"https:\/\/nag.com\/wp-json\/wp\/v2\/post-tag?post=1447"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}