## On the Power of Democratic Networks (1996)

### Cached

### Download Links

- [ftp.idiap.ch]
- [ftp.idiap.ch]
- [www.idiap.ch]
- [publications.idiap.ch]
- DBLP

### Other Repositories/Bibliography

Venue: | SWISS FEDERAL INSTITUTE OF TECHNOLOGY, DEPARTMENT OF MATHEMATICS |

Citations: | 2 - 1 self |

### BibTeX

@TECHREPORT{Mayoraz96onthe,

author = {Eddy Mayoraz},

title = {On the Power of Democratic Networks},

institution = {SWISS FEDERAL INSTITUTE OF TECHNOLOGY, DEPARTMENT OF MATHEMATICS},

year = {1996}

}

### OpenURL

### Abstract

Linear Threshold Boolean units (LTUs) are the basic processing components of artificial neural networks of Boolean activations. Quantization of their parameters is a central question in hardware implementation, when numerical technologies are used to store the configuration of the circuit. In the previous studies on the circuit complexity of feedforward neural networks, no differences had been made between a network with "small" integer weights and one composed of majority units (LTUs with weights in f\Gamma1; 0; +1g), since any connection of weight w (w integer) can be simulated by jwj connections of value sgn(w). This paper will focus on the circuit complexity of democratic networks, i.e. circuits of majority units with at most one connection between each pair of units. The main results presented are the following: any Boolean function can be computed by a depth-3 non-degenerate democratic network and can be expressed as a linear threshold function of majorities; AT-LEAST-k and AT-...

### Citations

829 |
Estimation of Dependences Based on Empirical Data
- Vapnik
- 1982
(Show Context)
Citation Context ...majorities (i.e. without 0 weights) over the n inputs. 4 5 The VC-dimension of MAJ 1 A characterization of the computational power of a class of functions is given by the VapnikChervonenkis dimension =-=[20]-=-. In order to formalize this notion, we first introduce some preliminary definitions. A dichotomy of p points ! 1 ; : : : ; ! p in some space\Omega is a partition of these points into two disjoint cla... |

239 |
Geometrical and statistical properties of systems of linear inequalities with application to pattern recognition
- Cover
- 1965
(Show Context)
Citation Context ... subset \Gamma 2 \Omega\Gamma When\Omega = IR n , all dichotomies of n + 1 points are linearly separable if and only if the n+1 points are not contained in a n \Gamma 1 dimensional hyperplane of IR n =-=[3]-=-. This result proves that the VC-dimension of the set of all linearly separable Boolean functions of n arguments is n + 1. The following proposition shows that the VC-dimension does not change when th... |

221 |
Parity, circuits and the polynomial-time hierarchy
- Furst, Saxe, et al.
- 1984
(Show Context)
Citation Context ...e, constant depth circuits composed of LTUs became popular when it was first shown that there is no polynomial size, constant depth circuit computing PARITY with only AND, OR and NOT processing units =-=[4]-=-. A few years later it was proved that, even if we add the PARITY function to this previous set of basic units, there is no polynomial size, constant depth circuit able to compute MAJORITY [16]. In co... |

171 |
Threshold Logic and its Applications
- Muroga
- 1971
(Show Context)
Citation Context ...its with size bounded polynomially in n are considered, many questions remain. On the one hand, since the number of linear threshold Boolean functions of at most n arguments is in 2 \Theta(n 2 ) (see =-=[11, 21]-=-), B n 6ae LT d for any constant depth d. On the other hand, LT 1 ` 0 LT 2 is the only inclusion known to be proper in the whole hierarchy LT 1 ae LT 2 ae LT 3 ae : : :. The quantization of the parame... |

128 |
Threshold circuits of bounded depth
- Hajnal, Maass, et al.
- 1993
(Show Context)
Citation Context ...require weights of exponential size. Thus, the most important restriction of LTUs which as been considered in the literature has small weights, i.e. integer weights bounded polynomially in the fan-in =-=[6, 15, 18]-=-. Let d LT d denote the set of Boolean functions computable by a depth-d polynomial size circuit composed of LTUs with small weights. The strongest relationship between LT d and d LT d has been obtain... |

86 | Harmonic analysis of polynomial threshold functions
- Bruck
- 1990
(Show Context)
Citation Context ...h denotes the length of the longest oriented path in N ; and by its size s(N ), which will be defined, in the present study, as the number of processing units in N . According to the notation used in =-=[2]-=-, LT 1 denotes the set of all linear threshold Boolean functions, while LT d (resp. LT d ) represents the set of all Boolean functions that can be computed by a feedforward network composed of LTUs, w... |

54 |
Circuit Complexity and Neural Networks
- Parberry
- 1994
(Show Context)
Citation Context ...rameters w i (i = 0; : : : ; n) of the LTUs is essential for any hardware implementation using numerical technologies to store the w i s. A famous result due to Muroga, Toda and Takasu [12] (see also =-=[14]-=- for a concise proof) shows that the weights of any linear threshold function of n inputs can be integers bounded from above by (n + 1) n+1 2 2 n : It is easy to see that some Boolean functions such a... |

37 | Simulating threshold circuits by majority circuits
- Goldmann, Karpinski
- 1998
(Show Context)
Citation Context ... functions computable by a depth-d polynomial size circuit composed of LTUs with small weights. The strongest relationship between LT d and d LT d has been obtained recently by Goldmann and Karpinski =-=[5]-=-, who proved that LT d ae d LT d+1 8ds1. The class of linear threshold Boolean functions with integer parameters w i bounded by a constant, constitutes naturally the next stage in this simplification ... |

36 |
Parallel computation with threshold functions
- Parberry, Schnitger
- 1988
(Show Context)
Citation Context ...require weights of exponential size. Thus, the most important restriction of LTUs which as been considered in the literature has small weights, i.e. integer weights bounded polynomially in the fan-in =-=[6, 15, 18]-=-. Let d LT d denote the set of Boolean functions computable by a depth-d polynomial size circuit composed of LTUs with small weights. The strongest relationship between LT d and d LT d has been obtain... |

33 |
Neural Computation of Arithmetic Functions
- Siu, Bruck
- 1990
(Show Context)
Citation Context ... known characterization of symmetric functions is the following: f is symmetric if there exists k integers t 1 ; : : : ; t k such that f(b 1 ; : : : ; b n ) = 1 iff P n i=1 b i 2 ft 1 ; : : : ; t k g =-=[6, 17]-=-. Corollary 3.5 Any symmetric Boolean function is in MAJ 3 . Proof: This result is a direct consequence of proposition 3.4 and of construction presented in [6, 17]: f(b) = maj \Gamma (AT-LEAST-t 1 (b)... |

29 |
T.Kailath, Depth-Size Tradeoffs for Neural Computation
- Siu, Roychowdhury
- 1991
(Show Context)
Citation Context ...en no jumping connections over layers are allowed, otherwise the size s of the depth-2 circuit can be divided by 2 [10] and if depth-3 networks are considered, the size can even be reduced to O( p n) =-=[19]-=-. Although the above construction for PARITY reduces by a factor 2 the size s of the depth-3 democratic network compared to the general construction for a symmetric function (corollary 3.5), it can no... |

22 |
Linear-Input Logic
- Minnick
- 1961
(Show Context)
Citation Context ...weight signs w = (+1; \Gamma1; +1; : : :). This construction is the smallest known when no jumping connections over layers are allowed, otherwise the size s of the depth-2 circuit can be divided by 2 =-=[10]-=- and if depth-3 networks are considered, the size can even be reduced to O( p n) [19]. Although the above construction for PARITY reduces by a factor 2 the size s of the depth-3 democratic network com... |

21 |
Theory of majority decision elements
- Muroga, Toda, et al.
- 1961
(Show Context)
Citation Context ...ation of the parameters w i (i = 0; : : : ; n) of the LTUs is essential for any hardware implementation using numerical technologies to store the w i s. A famous result due to Muroga, Toda and Takasu =-=[12]-=- (see also [14] for a concise proof) shows that the weights of any linear threshold function of n inputs can be integers bounded from above by (n + 1) n+1 2 2 n : It is easy to see that some Boolean f... |

12 |
Asymptotics of the logarithm of the number of threshold functions of the algebra of logic
- Zuev
- 1989
(Show Context)
Citation Context ...its with size bounded polynomially in n are considered, many questions remain. On the one hand, since the number of linear threshold Boolean functions of at most n arguments is in 2 \Theta(n 2 ) (see =-=[11, 21]-=-), B n 6ae LT d for any constant depth d. On the other hand, LT 1 ` 0 LT 2 is the only inclusion known to be proper in the whole hierarchy LT 1 ae LT 2 ae LT 3 ae : : :. The quantization of the parame... |

11 |
bounds on the size of bounded depth circuits over a complete basis with logical addition
- Razborov, Lower
- 1987
(Show Context)
Citation Context ...g units [4]. A few years later it was proved that, even if we add the PARITY function to this previous set of basic units, there is no polynomial size, constant depth circuit able to compute MAJORITY =-=[16]-=-. In contrast, it is interesting to determine the complexity of democratic networks computing these basic functions. Since the binary conjunction 2-AND is a majority function, the conjunction of n arg... |

6 | Depth-size tradeo s for neural computation - Siu, Roychowdhury, et al. - 1991 |

3 |
Learning by chir without storing internal representations
- Nabutovsky, Grossman, et al.
- 1990
(Show Context)
Citation Context ...m with no more than n units on the single hidden layer, and without jumping connections. Note that this construction has been discovered independently by T. Grossman and is mentioned without proof in =-=[13]-=-. Proposition 3.6 n-PARITY can be computed by a depth-2 non-degenerate democratic network composed of n hidden units. Proof: On the hypercube IB n , let us call the point \Gammab the antipodal of b; e... |

2 |
A constructive training algorithm for feedforward neural networks with ternary weights
- Aviolat, Mayoraz
- 1994
(Show Context)
Citation Context ...dies, we are designing training algorithms for democratic networks [7, 8, 9]. Among other approaches, we attempted to develop methods constructing the network layer by layer during the training phase =-=[1]-=-. In this context, it is highly important to know, for example, whether there exists a depth-2 network for any task that has to be loaded. The present paper investigates the computational power of fee... |

2 | Maximizing the robustness of a linear threshold classifier with discrete weights
- Mayoraz, Robert
- 1994
(Show Context)
Citation Context ...to know whether the loading is possible or not for a given quantization level of the parameters. More specifically, in some other studies, we are designing training algorithms for democratic networks =-=[7, 8, 9]-=-. Among other approaches, we attempted to develop methods constructing the network layer by layer during the training phase [1]. In this context, it is highly important to know, for example, whether t... |

1 |
Maximizing the stability of a majority perceptron using tabu search
- Mayoraz
- 1992
(Show Context)
Citation Context ...to know whether the loading is possible or not for a given quantization level of the parameters. More specifically, in some other studies, we are designing training algorithms for democratic networks =-=[7, 8, 9]-=-. Among other approaches, we attempted to develop methods constructing the network layer by layer during the training phase [1]. In this context, it is highly important to know, for example, whether t... |

1 |
Boolean Networks with Discrete Weights: Computational Power and Training
- Feedforward
- 1993
(Show Context)
Citation Context ...to know whether the loading is possible or not for a given quantization level of the parameters. More specifically, in some other studies, we are designing training algorithms for democratic networks =-=[7, 8, 9]-=-. Among other approaches, we attempted to develop methods constructing the network layer by layer during the training phase [1]. In this context, it is highly important to know, for example, whether t... |

1 | Linear-input logic, IEEETrans - Minnick - 1961 |