1. Introduction
Consider the system of nonlinear equations
where F(x) : D ⊂ Rn → Rn is Fréchet differentiable. Let F′(x) denote the Fréchet derivative of F at x.
Such equations (1.1) often arise in many important practical fields (e.g., physics and engineering, etc.). For example, input-output systems, least squares problems, finite difference or finite element problems, integral or differential equations, constrained function minimization, complementarity problems, variational inequalities, calculation of the load flows for power systems and solving initial or boundary value problems in ordinary or partial differential equations, etc.
Among all kinds of numerical methods for solving the nonlinear equations (1.1), Newton method [18,24,33] is the most classical one. In general, suppose that xk is the current approximate solution; a new approximate solution xk+1 can be computed through the following general form:
Algorithm 1.1 : Newton method
1. Let x0 ∈ Rn be a given initial guess.
2. For k = 0 until convergence do.
If n is not too large, the Newton method is attractive because it converges rapidly from any sufficiently good initial data. However, Newton method has two disadvantages from the point of view of practical computation: one is that it requires computing Jacobian matrices, and the other is that it requires solving linear equations (1.2) exactly. Computing the exact solution using a direct method such as Gaussian elimination may be expensive if the Jacobian matrix is large and may not be justified when xk is far from the exact solution x∗. In order to overcome the disadvantage of Newton method, using an iterative method and solving (1.2) approximately are reasonable. This approach was first considered by Dembo, Eisenstat and Steihaug in [11] (such a variant is the so-called inexact Newton method).
Algorithm 1.2 : Inexact Newton method
1. Let x0 ∈ Rn be a given initial guess.
2. For k = 0 until convergence do.
Remark 1.1. In the above algorithm, sk is the inexact Newton step and (1.4) is the inexact Newton condition. ηk is the forcing term for the k-th iteration step which may depend on xk; taking ηk ≡ 0 gives the famous Newton method.
In typical applications, the choice of the forcing terms is critical to the efficiency of the method and can affect robustness as well. Usually, it is hard to choose a good sequence of forcing terms. In computational practice, several authors considered some concrete strategies. We list here the following:
where 0 < p1 < p2 < p3 < 1 are prescribed at first, and In addition, assume that η0 is given and
There are three types of convergence issues about inexact Newton method: global, local and semi-local convergence analysis. The first is the convergence analysis based on the whole domain, the second is the convergence analysis based on a neighborhood of the solution x∗, and the last is the convergence analysis based on a neighborhood of the initial guess x0. Recently, several authors have studied the global convergence (see [14,31]), local convergence (see [10,20,23,34]) and semi-local convergence (see [2,3,4,19,22,29,30,32,36]) of inexact Newton method and proposed application in different fields [5,21,28].
After this method is established, some iteration methods are considered by many authors based on it. AIN (Accelerated Inexact Newton) method is presented by Fokkema, Sleijpen and Voest in [17]. They have shown how the classical Newton iteration scheme for nonlinear problems can be accelerated in a similar way as standard Richardson-type iteration schemes for linear equation. REGINN (Regularization based on Inexact Newton iteration) method is presented by Rieder in [25,26,27]. INM (Inexact Newton Multigrid) is considered by Brown, Vassilevski and Woodward in [6]. They have proved optimal-order and mesh-independent convergence of an inexact Newton method where the linear Jacobian systems are solved with multigrid techniques. Also, PIN (Preconditioned Inexact Newton) method is considered by Cai and Keyes in [8].
In this paper we consider a modified inexact Newton method. Our modification uses to replace F′ (xk) sk = −F (xk) + rk, where The use of this method is particularly appropriate when an exact solution of Newton equation is difficult to obtain and/or when evaluating and preparing the Jacobian for the computation is costly, and this method has fast convergence as well.
The rest of this paper is organized as follows. In section 2, a modified inexact Newton algorithm is established. In section 3, we will present the semi-local convergence result for the given algorithm. Moreover, another theorem shows the affine invariance of the convergence of our proposed method. In section 4, we give three test problems using the algorithm presented in this paper to show its convergence properties and robustness. In the last section, some concluding remarks are made.
2. A modified inexact Newton method
In this section, we introduce a modified inexact Newton method.
From the Algorithm 1.2, we know that the inexact Newton iteration scheme is
Note that (2.1) can be rewritten as
if F′ (x) is invertible for every x ∈ D.
Consider one-dimensional nonlinear equation
where f (x) : R → R and f′ (x) is invertible for every x ∈ R. It easily follows from (2.2) that the inexact Newton iteration scheme of (2.3) is
If ηk ≡ 0, then rk in (2.4) is 0. Moreover, (2.4) yields the famous Newton iteration scheme as follows
In fact, from the point of view of geometry, Newton iteration scheme (2.5) uses tangent line of point (xk, f (xk)) to approximate curved line. We claim that, if the tangent line of the point (xk − f′ (xk)−1 f (xk), f (xk − f′ (xk)−1 f (xk))) is used to replace the former one, then a new scheme can be obtained as follows
Furthermore by generalizing this method to n-dimensional case, we have
For simplicity, let Hence, (2.7) reduces to
However, we have to solve two inverse matrices at each iteration, which will cost a lot of computational time. Meanwhile, it is hard to get because it is difficult to calculate F′ (xk)−1 even if the scale of problems is medium. Here, we give a new scheme, which can use the information of the former iteration adequately and save CUP-time. Here, we take Moreover, a modified inexact Newton algorithm is obtained.
Algorithm 2.1 : Modified inexact Newton method
1. Let x0 ∈ Rn be a given initial guess.
2. For k = 0 until convergence do.
Remark 2.1. In the above algorithm, sk is the inexact Newton step and (2.10) is the modified inexact Newton condition. ηk is the forcing term for the k-th iteration step which may depend on xk. Taking ηk ≡ 0 and give the famous Newton method. Because has already known in the previous iteration, it is easy to get
3. Semi-local convergence analysis of modified inexact Newton method
In this section, we will present the semi-local convergence result for Algorithm 2.1 and an affine invariant version is also presented.
The following well-known results are useful for our theorems, one can find them in [24].
Definition 3.1. F′ is Lipschitz continuous in D, if there exists L ≥ 0 such that
for every x, y ∈ D.
Lemma 3.1. Let F : D ⊂ Rn → Rn be Fréchet differentiable and F′ be Lipschitz continuous satisfying (3.1). Then,
for every x, y ∈ D.
Lemma 3.2. Let F : D ⊂ Rn → Rn be Fréchet differentiable and F′ be invertible for every x ∈ D such that
Then,
for every x, y, z ∈ D.
Theorem 3.3. Suppose F : D ⊂ Rn → Rn is Fréchet differentiable and F′ is Lipschitz continuous in D. Let x0 ∈ D and || F (x0 ) || ≤ η. Assume
where Then the sequence of modified inexact Newton method defined by (2.9) stays in S and converges to x∗ which satisfies F (x∗) = 0.
Proof. Firstly, we will prove
by induction.
For k = 0, (3.5) holds evidently.
Assume that (3.5) is true for k ≤ m.
Now, we prove the assertion for k = m + 1. Since
By Lemma 3.1, it follows that
where L1 and L2 are Lipschitz constants.
Using (3.6) and (3.7), we have
Hence, it follows from the assumption of induction, (2.10) and (3.3) that
This gives (3.5) and the induction is complete.
Indeed, by (2.9) and (3.5),
By the definition of δ, we have
Therefore, for m ≥ 0,
Thus, by (3.3), {xk} is a Cauchy sequence and converges to x∗ as k → +∞. Moreover,
for any m ≥ 0. In view of (3.4), we obtain xm ∈ S ⊂ D, which implies that x∗ ∈ S ⊂ D as well.
Finally, we assert that
Hence, F (x∗) = 0. The proof is completed. □
It is well-known [13,35] that many Newton-like methods are affine invariant in the sense that when they are used to solve the affinely transformed problem
where A is any nonsingular n × n matrix.
Example. Consider the Newton iteration
For F, we use affine transform F = AF. Then
Thus, we assert that the Newton iteration sequence {xk} is affine invariant.
Convergence conditions for affine invariant methods should themselves be invariant under the transformations of this form [13]. Now it is clear that even if the method (2.9) is affine invariant, the condition (2.10) is not affine invariant. In order to give semi-local convergence theorem for the modified inexact Newton method with affine invariant condition, we improve the method (2.9) with the above condition (2.10) as follows
These improvements for inexact Newton method were proposed by Bai and Tong [4]. Here, we take them for our method.
The following theorem concerns the affine invariance of our algorithm.
Theorem 3.4. Suppose F : D ⊂ Rn → Rn is Fréchet differentiable and the modified inexact Newton method is defined by (3.10) and (3.11). Assume F′ is invertible for every x ∈ D such that
Let x0 ∈ D and || F′ (x0)−1 F (x0) || ≤ η. Suppose further that
where Then the sequence of modified inexact Newton method stays in S and converges to x∗ which satisfies F (x∗) = 0.
Proof. The proof is similar to the proof of Theorem 3.1. Firstly, we will prove
by induction.
For k = 0, (3.14) holds evidently.
Assume that (3.14) is true for k ≤ m.
Now, we prove the assertion for k = m+1. Let Tm+1 = || F′ (xm+1)−1 F (xm+1) || for short. Since
By Lemma 3.2, it follows that
where L1 and L2 are constant.
Using (3.15) and (3.16),
Hence, it follows from the assumption of induction, (3.11) and (3.12) that
This gives (3.14) and the induction is complete.
Indeed, by (3.10) and (3.14)
By the definition of δ, we have
Therefore, for m ≥ 0,
Thus, by (3.12), {xk} is a Cauchy sequence and converges to x∗ as k → +∞. Moreover,
for any m ≥ 0. In view of (3.13), we obtain xm ∈ S ⊂ D, which implies that x∗ ∈ S ⊂ D as well.
Finally, we assert that
Hence, F(x∗) = 0. The proof is completed. □
Next, we consider the rate of convergence of the modified inexact Newton method. First, we give the following definition given by R.S. Dembo et al. [11].
Definition 3.2. Let {xk} be a sequence which converges to x∗. Then xk → x∗ with order at least q (q > 1) if
Theorem 3.5. Under the assumptions of Theorem 3.1, if the modified inexact Newton iterates {xk} converge to x∗, then xk → x∗ with order at least 2 if and only if
Proof. Assume that xk → x∗ with order at least 2. Note that
Taking norms, we arrive at
Therefore, by Lemma 3.1, the continuity of F′ at x∗ and the assumption that xk → x∗ with order at least 2, we have
Conversely, assume that ||rk|| = O(||xk − x∗||2). Note that
Taking norms, we arrive at
Therefore, by Lemma 3.1 and the assumption that ||rk|| = O(||xk − x∗||2), we have
4. Numerical experiments
In this section, we give three test problems using the algorithm presented in this paper to show its convergence property and robustness. The purpose of the first two problems are to show that Algorithm 2.1 is useful in the nonlinear case. By useful we mean that the algorithm presented has faster convergence. In the third problem, a nonlinear PDE is solved.
Example 1. Consider the following nonlinear equations:
with x∗ = (1, 1)T.
Take η = 10−4 given by Cai, Gropp, Keyes and Tidriri [9]. Using the modified inexact Newton method, we can obtain the iterative solutions listed in Table 1 and 2 with initial guess (−1,−1)T and (510, 1021)T, respectively.
Table 1.Comparison of iterative solutions of two algorithms with the initial guess (−1,−1)T
Table 2.Comparison of iterative solutions of two algorithms with the initial guess (510, 1021)T
In Table 1, a comparison of iterative solutions of two algorithms of Example 1 with the initial guess (−1,−1)T is shown. Here, denote the iterative solutions of inexact Newton method while represent the iterative solutions of modified inexact Newton method. We have seen from Table 1 that the modified inexact Newton method has faster convergence.
In Table 2, a comparison of iterative solutions of two algorithms of Example 1 with the initial guess (510, 1021)T is shown. We have seen from Table 2 that two methods have a wide convergence domain and the modified inexact Newton method has faster convergence.
In Fig. 1, we present profiles for the history of absolute error for two algo-rithms of Example 1 with the initial guess (−1,−1)T. We have seen from Fig. 1 that the absolute error using modified inexact Newton method becomes small faster than that of inexact Newton method. It is observed that the proposed method performs better than the inexact Newton method.
Fig. 1.History of absolute error for two algorithms with the initial guess (−1, −1)T.
In Fig. 2, we present profiles for history of absolute error for two algorithms of Example 1 with the initial guess (510, 1021)T. We have seen from Fig. 2 that the modified inexact Newton method may perform several times faster than the inexact Newton method.
Fig. 2.History of absolute error for two algorithms with the initial guess (510, 1021)T .
Example 2. Consider the following nonlinear equations:
Take η = 10−4 given by Cai, Gropp, Keyes and Tidriri [9]. Using the modified inexact Newton method, we can obtain the iterative solutions. Here, we use ||F(xk)|| ≤ 1.0e − 9 as the stopping rule for this example.
Results for Example 2 are listed in Table 3, where No.Eq1 denotes the number of equation in the first step, No.Eqk represents the number of equation in the k-th step (k > 1), No.Fd1 denotes the number of F′ in the first step, No.Fdk represents the number of F′ in the k-th step, No.it denotes the number of iter-ation and Res stands for the value of ||F(xk)|| when our stop rule is satisfied. As shown in Table 3, the number of iteration by the modified inexact Newton method is less than that of the classical inexact Newton method. Meanwhile, we have seen from Table 3 that the number of F′ by the modified inexact Newton method in the k-th step is 2, but has already known in the previous iteration. Hence, the number of F′ is indeed 1. Although the number of equation of the modified inexact Newton method is nearly twice of that of the classical one, the CPU-time of the former method is still less. All in all, our method indeed saves on computation.
Table 3.Results for Example 2
Example 3. Consider one-dimensional Burgers’ equation:
where the positive number is the coefficient of viscosity, and Re denotes the Reynolds number. This equation has an exact solution in the form of the infinite series
where Ij(x) is the first type of the j-th modified Bessel function. When j = 35, it is used as an approximation to the infinite sum (4.2).
We solve (4.1) with finite difference method and the modified inexact Newton method. First, we discretize in space with centered difference to obtain a system of ordinary differential equations, which we write as
Then the nonlinear equations that should be solved for the implicit Euler method with a time step τ is
Moreover, one solves, at each time step, the nonlinear equations
Then, we use the modified inexact Newton method to solve nonlinear equations (4.5).
Let h = 1/M be the mesh width in space and set xi = ih for i = 1, 2, . . . ,M−1. Let τ = T/N be the mesh width in time and set tn = nτ for n = 1, 2, . . . ,N. u(xi, tn) is the approximate solution of u(x, t). Discretization is on a 100 × 100 grid, so that N = 100 and M = 100. Hence, the spatial mesh width h = 0.01 and the time step τ = 0.01. Take ηmax = 0.9 used by Eisenstat and Walker in [15]. Here, GMRES(m) algorithm is used for linear systems and m = 40.
In Table 4, a comparison of the numerical solution with the exact solution at different space points of (0, 1) for Example 3 at T = 0.1 and v = 0.1 is shown. It is observed that the proposed method gives sharp results. In order to show the physical behaviour of the given problem, we give surface plots of the computed solutions for different values of the coefficient of viscosity, v in Figs. 3 and 4.
Table 4.Comparison of the numerical solution with the exact solution at different space points of Example 3 at T = 0.1 for ν = 0.1
Fig. 3.Numerical solutions profiles of Example 3 for ν = 0.1, h = 0.01 and τ = 0.01.
Fig. 4.Numerical solutions profiles of Example 3 for ν = 0.01, h = 0.01 and τ = 0.01.
5. Concluding remarks
The modified inexact Newton method for nonlinear equations has been presented. It is shown that the method given performs several times faster than the inexact Newton method. A semi-local convergence theorem for the modified inexact Newton method is studied and an affine invariant version is also presented. We then give three numerical examples which show that the modified inexact Newton scheme is more efficient than traditional inexact Newton strategy. Therefore, it is suggested to use the modified inexact Newton to get the numerical solution of the nonlinear equations effectively.
References
- H.B. An, Z.Y. Mo and X.P. Liu, A choice of forcing terms in inexact Newton method, J. Comput. Appl. Math., 200 (2007), 47-60. https://doi.org/10.1016/j.cam.2005.12.030
- I.K. Argyros, On the semilocal convergence of inexact Newton methods in Banach spaces, J. Comput. Appl. Math., 228 (2009), 434-443. https://doi.org/10.1016/j.cam.2008.10.005
- I.K. Argyros, A new convergence theorem for the inexact Newton methods based on assumptions involving the second Frechet derivative, Comput. Math. Appl., 37 (1999), 109-115. https://doi.org/10.1016/S0898-1221(99)00091-7
- Z.Z. Bai and P.L. Tong, On the affine invariant convergence theorems of inexact Newton method and Broyden's method, J. UEST China, 23 (1994), 535-539 (in Chinese).
- Z.J. Bai, Inexact Newton methods for inverse eigenvalue problems, Appl. Math. Comput., 172 (2006), 682-689. https://doi.org/10.1016/j.amc.2004.11.023
- P.N. Brown, P.S. Vassilevski and C.S. Woodward, On mesh-independent convergence of an inexact Newton-Multigrid algorithm, SIAM J. Sci. Comput., 25 (2003), 570-590. https://doi.org/10.1137/S1064827502407822
- P.N. Brown and Y. Saad, Hybird Krylov methods for nonlinear systems of equations, SIAM J. Sci. Stat. Comput., 11 (1990), 450-481. https://doi.org/10.1137/0911026
- X.C. Cai and D.E. Keyes, Nonlinearly preconditioned inexact Newton algorithms, SIAM J. Sci. Comput., 24 (2002), 183-200. https://doi.org/10.1137/S106482750037620X
- X.C. Cai, W.D. Gropp, D.E. Keyes and M.D. Tidriti, Newton-Krylov-Schwarz methods in CFD, in: Proceedings of the International Workshop on Numerical Methods for the Navier-Stokes Equations, Vieweg, Braunschwieg, 1995.
- J.H. Chen and W.G. Li, Convergence behaviour of inexact Newton methods under weak Lipschitz condition, J. Comput. Appl. Math., 191 (2006), 143-164. https://doi.org/10.1016/j.cam.2005.03.076
- R.S. Dembo, S.C. Eisenstat and T. Steihaug, Inexact Newton methods, SIAM J. Numer. Anal., 19 (1982), 400-408. https://doi.org/10.1137/0719025
- R.S. Dembo and T. Steihaug, Truncated Newton algorithms for large-scale optimization, Math. Program, 26 (1983), 190-212. https://doi.org/10.1007/BF02592055
- P. Deuflhard and G. Heindl, Affine invariant convergence theorems for Newton's method and extensions to related methods, SIAM J. Numer. Anal., 16 (1979), 1-10. https://doi.org/10.1137/0716001
- S.C. Eisenstat and H.F. Walker, Globally convergent inexact Newton methods, SIAM J. Optim., 4 (1994), 393-422. https://doi.org/10.1137/0804022
- S.C. Eisenstat and H.F. Walker, Choosing the forcing term in an inexact Newton method, SIAM J. Sci. Comput., 17 (1996), 16-32. https://doi.org/10.1137/0917003
- R.N. Elias, A.L.G.A. Coutinho and M.A.D. Martins, Inexact Newton-type methods for the solution of steady incompressible viscoplastic flows with the SUPG/PSPG finite element formulation, Comput. Methods Appl. Mech. Engrg., 195 (2006), 3145-3167. https://doi.org/10.1016/j.cma.2004.08.015
- D.R. Fokkema, G.L.G. Sleijpen and H.A.V.D. Vorst, Accelerated inexact Newton schemes for large system of nonlinear equations, SIAM J. Sci. Comput., 19 (1998), 657-674. https://doi.org/10.1137/S1064827595296148
- A. Galantai, The theory of Newton's method, J. Comput. Appl. Math., 124 (2000), 25-44. https://doi.org/10.1016/S0377-0427(00)00435-0
- X.P. Guo, On semilocal convergence of inexact Newton methods, J. Comput. Math., 25 (2007), 231-242.
- C. Li and W.P. Shen, Local convergence of inexact methods under the Holder condition, J. Comput. Appl. Math., 222 (2008), 544-560. https://doi.org/10.1016/j.cam.2007.12.001
- J.M. Martinez and L. Qi, Inexact Newton methods for solving nonsmooth equations, J. Comput. Appl. Math., 60 (1995), 127-145. https://doi.org/10.1016/0377-0427(94)00088-I
- I. Moret, A Kantorovich type theorem for inexact Newton methods, Numer. Funct. Anal. Optim., 10 (1989), 351-365. https://doi.org/10.1080/01630568908816307
- B. Morini, Convergence behaviour of inexact Newton method, Math. Comput., 68 (1999), 1605-1613. https://doi.org/10.1090/S0025-5718-99-01135-7
- J.M. Ortega and W.C. Rheinboldt, Iterative Solution of Nonlinear Equations in Several Variables, SIAM, Philadelphia, PA, 2000.
- A. Rieder, Inexact Newton regularization using conjugate gradients as inner iteration, SIAM J. Numer. Anal., 43 (2005), 604-622. https://doi.org/10.1137/040604029
- A. Rieder, On the regularization of nonliear ill-posed problems via inexact Newton iterations, Inverse Probl., 15 (1999), 309-327. https://doi.org/10.1088/0266-5611/15/1/028
- A. Rieder, On convergence rates of inexact Newton regularization, Numer. Math., 88 (2001), 347-365. https://doi.org/10.1007/PL00005448
- J.N. Shadid, R.S. Tuminaro and H.F. Walker, An inexact Newton method for fully coupled solution of the Navier-Stokes equations with heat and mass transport, J. Comput. Phys., 137 (1997), 155-185. https://doi.org/10.1006/jcph.1997.5798
- W.P. Shen and C. Li, Convergence criterion of inexact methods for operators with Holder continuous derivatives, Taiwan. J. Math., 12 (2008), 1865-1882. https://doi.org/10.11650/twjm/1500405093
- W.P. Shen and C. Li, Kantorovich-type convergence criterion for inexact Newton methods, Appl. Numer. Math., 59 (2009), 1599-1611. https://doi.org/10.1016/j.apnum.2008.11.002
- M.V. Solodov and B.F. Svaiter, A globally convergent inexact Newton method for systems of monotone equations, Reformulation: Nonsmooth, Piecewise Smooth, Semismooth and Smoothing Methods, Kluwer Academic Publishers, 1998.
- M. Wu, A new semi-local convergence theorem for the inexact Newton methods, Appl. Numer. Math., 200 (2008), 80-86.
- T. Yamamoto, Historical developments in convergence analysis for Newton's and Newton-like method, J. Comput. Appl. Math., 124 (2000), 1-23. https://doi.org/10.1016/S0377-0427(00)00417-9
- T.J. Ypma, Local convergence of inexact Newton methods, SIAM J. Numer. Anal., 21 (1984), 583-590. https://doi.org/10.1137/0721040
- T.J. Ypma, Affine invariant convergence for Newton's methods, BIT, 22 (1982), 108-118. https://doi.org/10.1007/BF01934400
- J.L. Zhou, S.Y. Zhang, G.P. Yang and J.R. Tan, A convergence theorem for the inexact Newton methods based on Holder continuous Frechet derivative, Appl. Numer. Math., 197 (2008), 206-211.