### Table 2: Results for Gradient method with single penalty.

"... In PAGE 4: ... (a) To take a very small value of K after the limiting value of the constraint is crossed or (b) to progressively decrease the value of K for each itera- tion. In case of penalty method, initially the values of penalty for all the constraints are taken as 1and few results are tab- ulated in Table2 . Table 3 shows the results considering the SI.... ..."

### Table 1. Final objective value and nal square norm of the gradient obtained by two optimization methods on the PENALTY3 problem.

"... In PAGE 3: ... Several examples of this behavior are described in [16]. In Table1 we present a striking example that was obtained when the inexact Newton method described in [6] and the limited memory code L-BFGS-B [16] (using m = 5 correction pairs) were applied to problem PENALTY3 from the CUTE collection [4]. Both methods were run until no further progress could be made in reducing the function values; we report the nal function values and gradient square norms obtained by each method.... In PAGE 4: ...Table1 may be unreliable for quasi-Newton and limited memory methods. We will also make some general observations relating the rate of convergence of an optimization iteration and the nal accuracy achieved in the objective function.... ..."

### Table 1. Flowchart of the Parallel BP To the best of our knowledge there are no published deterministic convergence proofs for either the parallel or serial BP algorithm. In [18] it is proven that the sequence of weights generated by the serial BP either converges to a point that is almost surely stationary or it diverges. In contrast, our approach is deterministic. Our proof which is based on the results of Section 2, covers both serial and parallel cases as well as the computationally important methods with a momentum term. We note the equivalence of BP to a deterministic perturbed gradient algorithm. We are now ready to apply the analysis of Section 2 to backpropagation. Theorem 3.1. Let S lt;n be any bounded set. If the learning and momentum

"... In PAGE 9: ... For simplicity we shall assume that the learning and momentum rates re- main xed within each major iteration. Table1 gives a owchart of the parallel BP algorithm.We are now ready to state and prove convergence of the parallel BP algorithm.... ..."

### Table 1. Flowchart of the Parallel BP To the best of our knowledge there are no published deterministic convergence proofs for either the parallel or serial BP algorithm. In [18] it is proven that the sequence of weights generated by the serial BP either converges to a point that is almost surely stationary or it diverges. In contrast, our approach is deterministic. Our proof which is based on the results of Section 2, covers both serial and parallel cases as well as the computationally important methods with a momentum term. We note the equivalence of BP to a deterministic perturbed gradient algorithm. We are now ready to apply the analysis of Section 2 to backpropagation. Theorem 3.1. Let S lt;n be any bounded set. If the learning and momentum

"... In PAGE 9: ... For simplicity we shall assume that the learning and momentum rates re- main xed within each major iteration. Table1 gives a owchart of the parallel BP algorithm.We are now ready to state and prove convergence of the parallel BP algorithm.... ..."

### Table 1: Error on the control curve : behavior of the L2-error versus the penalty parameter quot;, and number of iterates for the conjugate gradient algorithm for the resolution of the obstacle problem and the Newton method.

2004

"... In PAGE 8: ...o satisfy the obstacle condition on . Let N = 40 and M = 500 be xed. Let the nal time be T = 0:1 and the time step = 1=60. To illustrate the in uence of the penalty parameter, Table1 shows the value of jj(u quot; ) jjL2( ) at time T = 0:1 for di erent values of the parameter quot;, as well as the mean number of... In PAGE 9: ...Table 1: Error on the control curve : behavior of the L2-error versus the penalty parameter quot;, and number of iterates for the conjugate gradient algorithm for the resolution of the obstacle problem and the Newton method. Table1 illustrates the good convergence properties of the conjugate gradi- ent/Newton method. The resolution of the di usion problem is not discussed here.... ..."

### Table 1: Convergence results for the gradient of the solution

2005

### Table 3. Computational results for test problems.

"... In PAGE 11: ... Better programs required fewer generations and less computation time. Table3 compares the new encoding method, the penalty encoding method and in- teger programming for the test problems. The left half of Table 3 (a) shows the computa- tional results for the new encoding method and the right half presents for the penalty en- coding method.... In PAGE 11: ... Table 3 compares the new encoding method, the penalty encoding method and in- teger programming for the test problems. The left half of Table3 (a) shows the computa- tional results for the new encoding method and the right half presents for the penalty en- coding method. #8 means that the number of plants is eight, and so on.... In PAGE 11: ... The last three rows show the goal value of the prob- lem, the generation and the time cost in which the goal is attained. Table3 (b) displays the results from using integer programming. According to the schema theorem and the ... In PAGE 12: ... The goal solutions can be thought of as sufficiently good solutions. Table3 shows that the new method requires fewer generations and computation time to produce goal solutions. Table 3 enables the trend line for the number of genera- tions required by the genetic algorithm to be drawn, versus the number of plants.... In PAGE 12: ... Table 3 shows that the new method requires fewer generations and computation time to produce goal solutions. Table3 enables the trend line for the number of genera- tions required by the genetic algorithm to be drawn, versus the number of plants. The gradient of Fig.... In PAGE 13: ... The LINGO package was used to solve these test problems by integer programming. For these cases, LINGO can get feasible solutions only ( Table3 (b)) because the test problems are all highly complex. The com- putational results of this section demonstrate that the new encoding method improves the performance of genetic algorithms by reducing their search space.... ..."

### Table 1: Iteration counts for iterative solutions of FIT2P. Algo- rithm switched phase at step 17.

"... In PAGE 22: ... The results for FIT2P are tabulated in Table 1. Table1 : Iteration counts for iterative solutions of FIT2P. Algo- rithm switched phase at step 17.... In PAGE 23: ...Indeed, as shown in Table1 , the number of PCG iterations taken to solve the normal equations generally increases as the IPM converges to a solution. On the other hand, when the two-phase algorithm switches to the RAE system (which occurs at the 17th IPM step), the number of SQMR iterations taken to solve the preconditioned RAE system generally decreases as the IPM solution converges.... ..."

### Table 5.1 Rates of convergence for the L1 penalty algorithm

1999

Cited by 11

### Table 3: Final penalty.

"... In PAGE 19: ... Finally, it is worth noting that the total solution time was far below the costs of solving these problems by earlier methods, especially for large problems. In Table3 the nal values of the penalty coe cient 1= are shown. We see that our rules for updating the penalty successfully adapt it to the shape of the function considered.... ..."