## Generalized binary search (2008)

### Cached

### Download Links

Venue: | In Proceedings of the 46th Allerton Conference on Communications, Control, and Computing |

Citations: | 30 - 0 self |

### BibTeX

@INPROCEEDINGS{Nowak08generalizedbinary,

author = {Robert Nowak},

title = {Generalized binary search},

booktitle = {In Proceedings of the 46th Allerton Conference on Communications, Control, and Computing},

year = {2008},

pages = {568--574}

}

### OpenURL

### Abstract

This paper addresses the problem of noisy Generalized Binary Search (GBS). GBS is a well-known greedy algorithm for determining a binary-valued hypothesis through a sequence of strategically selected queries. At each step, a query is selected that most evenly splits the hypotheses under consideration into two disjoint subsets, a natural generalization of the idea underlying classic binary search. GBS is used in many applications, including fault testing, machine diagnostics, disease diagnosis, job scheduling, image processing, computer vision, and active learning. In most of these cases, the responses to queries can be noisy. Past work has provided a partial characterization of GBS, but existing noise-tolerant versions of GBS are suboptimal in terms of query complexity. This paper presents an optimal algorithm for noisy GBS and demonstrates its application to learning multidimensional threshold functions. 1

### Citations

8983 | N.: The nature of statistical learning theory
- Vapnik
- 1995
(Show Context)
Citation Context ... hypotheses. The number of unique labeling assignments generated by these hypotheses can be bounded in terms of the VC dimension of the class; see the book by Vapnik for more information on VC theory =-=[19]-=-. As a result, it suffices to consider a finite subset of the hypotheses consisting of just one representative of each unique labeling assignment. Furthermore, the computational complexity of GBS is p... |

1718 | Compressed sensing
- Donoho
(Show Context)
Citation Context ... the structure of H is related to the notion of arbitrary queries examined in the work of Kulkarni et al [8], and somewhat to the theory of compressed sensing developed by Candes et al [9] and Donoho =-=[10]-=-. B. Multidimensional Problems Let H = {hi} N i=1 be a collection of multidimensional threshold functions of the following form. The threshold of each hi determined by (possibly nonlinear) decision su... |

1300 | Robust uncertainty principles: exact signal reconstruction from highly incomplete frequency information
- Candès, Romberg, et al.
(Show Context)
Citation Context ... query space to the structure of H is related to the notion of arbitrary queries examined in the work of Kulkarni et al [8], and somewhat to the theory of compressed sensing developed by Candes et al =-=[9]-=- and Donoho [10]. B. Multidimensional Problems Let H = {hi} N i=1 be a collection of multidimensional threshold functions of the following form. The threshold of each hi determined by (possibly nonlin... |

647 |
Queries and concept learning
- Angluin
- 1988
(Show Context)
Citation Context ...ashev and Zigangirov [4] and more recently by Pelc et al [30], is equivalent to a noisy binary search problem. There is a large literature on learning from queries; see the review articles by Angluin =-=[31]-=-, [32]. This paper focuses exclusively on membership queries (i.e., an x ∈ X is the query and the response is h∗ (x)), although18 other types of queries (equivalence, subset, superset, disjointness, ... |

334 | Selective sampling using the query by committee algorithm
- Freund, Seung, et al.
- 1997
(Show Context)
Citation Context ...d applications by Geman and Jedynak [12] and Arkin et al [13]), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et al =-=[16]-=-, Dasgupta [17], Balcan et al [7], and Castro and Nowak [18].15 First we show that the pair (R d , H) is 1-neighborly. Each A ∈ A is a polytope in R d . These polytopes are generated by intersections... |

223 | Information-Based Complexity
- Traub, Wo´zniakowski, et al.
- 1988
(Show Context)
Citation Context ..., then each point evaluation provides one bit in the m-bit binary expansion of t. Thus, classic binary search is information-theoretically optimal; see the book by Traub, Wasilkowski and Wozniakowski =-=[35]-=- for a nice treatment of classic bisection and binary search. The main results of this paper generalize the salient aspects of classic binary search to a much broader class of problems. In many (if no... |

184 |
Constructing optimal binary decision trees is NPcomplete
- Hyafil, Rivest
- 1976
(Show Context)
Citation Context ... with the minimum average or worst case depth. In general the determination of the optimal tree (in either sense above) is a combinatorial problem and was shown by Hyafil and Rivest to be NP-complete =-=[1]-=-. Therefore, this paper investigates the performance of a greedy procedure called generalized binary search (GBS), depicted below in Fig. 1. At each step GBS selects a query that results in the most e... |

158 | An active testing model for tracking roads in satellite images
- Geman, Jedynak
- 1996
(Show Context)
Citation Context ...e induced query space. The problem of learning multidimensional threshold functions arises commonly in computer vision (see the review of Swain and Stricker [11] and applications by Geman and Jedynak =-=[12]-=- and Arkin et al [13]), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et al [16], Dasgupta [17], Balcan et al [7], a... |

89 | Coarse sample complexity bounds for active learning
- Dasgupta
- 2005
(Show Context)
Citation Context ...e other half. Therefore, GBS determines the optimal hypothesis in no more than O(log |H|) steps, independent of the dimension. Related results for this special case have been previously reported; see =-=[9]-=- and the references therein. Note that even if the hyperplanes do not pass through the origin (b = 0), O(log |H|) convergence is still attained so long as |b| is not too large. This generalizes earli... |

83 |
Promising directions in active vision
- Swain, Stricker
- 1993
(Show Context)
Citation Context ...then forming linear decision surfaces in the induced query space. The problem of learning multidimensional threshold functions arises commonly in computer vision (see the review of Swain and Stricker =-=[11]-=- and applications by Geman and Jedynak [12] and Arkin et al [13]), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et ... |

75 | Analysis of a greedy active learning strategy
- Dasgupta
- 2004
(Show Context)
Citation Context ...by Geman and Jedynak [12] and Arkin et al [13]), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et al [16], Dasgupta =-=[17]-=-, Balcan et al [7], and Castro and Nowak [18].15 First we show that the pair (R d , H) is 1-neighborly. Each A ∈ A is a polytope in R d . These polytopes are generated by intersections of the halfspa... |

63 | How many queries are needed to learn
- Hellerstein, Pillaipakkamnatt, et al.
- 1996
(Show Context)
Citation Context ... a binary-valued function in a finite collection of hypotheses can be bounded (above and below) in terms of a combinatorial parameter of (X , H) due to Hegedüs [33] (see the work of Hellerstein et al =-=[34]-=- for related work). Due to its combinatorial nature, computing such bounds are generally NP-hard. In contrast, the geometric relationship between X and H developed in this paper leads to an upper boun... |

56 | Minimax bounds for active learning
- Castro, Novak
- 2008
(Show Context)
Citation Context ...]), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et al [16], Dasgupta [17], Balcan et al [7], and Castro and Nowak =-=[18]-=-.15 First we show that the pair (R d , H) is 1-neighborly. Each A ∈ A is a polytope in R d . These polytopes are generated by intersections of the halfspaces corresponding to the hypotheses. Any two ... |

51 |
On generalized teaching dimensions and the query complexity of learning
- Hegedüs
- 1995
(Show Context)
Citation Context ... number of queries required to determine a binary-valued function in a finite collection of hypotheses can be bounded (above and below) in terms of a combinatorial parameter of (X , H) due to Hegedüs =-=[33]-=- (see the work of Hellerstein et al [34] for related work). Due to its combinatorial nature, computing such bounds are generally NP-hard. In contrast, the geometric relationship between X and H develo... |

49 |
Computing with noisy information
- Feige, Raghavan, et al.
- 1994
(Show Context)
Citation Context ...[21] and Zigangirov [22]), experimental design (e.g., as studied by Rényi [23], [24]), disease diagnosis (e.g., see the work of Loveland [25]), fault-tolerant computing (e.g., the work of Feige et al =-=[26]-=-), the scheduling problem considered by Kosaraju et al [27], computer vision problems investigated by Geman and Jedynak [12] and Arkin et al [13]), image processing problems studied by Korostelev and ... |

41 | Margin based active learning
- Balcan, Broder, et al.
- 2007
(Show Context)
Citation Context ...hin ɛ of it). The size of such an H satisfies log N ≤ C d log ɛ −1 , for a constant C > 0, which is the proportional to the minimum query complexity possible in this setting, as shown by Balcan et al =-=[7]-=-. Those authors also present an algorithm with roughly the same query complexity for this problem. However, their algorithm is specifically designed for ( ) 2 β(1−α) α(1−β) Note that the factor 1 − − ... |

41 | Active learning in the non-realizable case - Kääriäinen - 2006 |

40 |
Coping with errors in binary search procedures
- Kleitman, Meyer, et al.
- 1980
(Show Context)
Citation Context ...daptive sampling procedures proposed by Korostelov and Kim [14], [15] for image recovery and binary classification problems studied by Castro and Nowak [18]. We also mention the works of Rivest et al =-=[37]-=-, Spencer [38] and Aslam and Dhagat [39], and Dhagat et al [40], which consider adversarial situations in which the total number of erroneous oracle responses is fixed in advance. One straightforward ... |

35 |
Introduction to Algorithms, Third Edition
- Cormen, Leiserson, et al.
- 2009
(Show Context)
Citation Context ... of GBS can be much better, requiring as few as log2(N) queries. In classic binary search, for example, half of the hypotheses are eliminated at each step (e.g., refer to the textbook by Cormen et al =-=[29]-=-). Rényi first considered a form of binary search with noise [23] and explored its connections with information theory [24]. In particular, the problem of sequential transmission over a binary symmetr... |

32 | Decision trees for geometric models
- Arkin, Mitchell, et al.
- 1998
(Show Context)
Citation Context .... The problem of learning multidimensional threshold functions arises commonly in computer vision (see the review of Swain and Stricker [11] and applications by Geman and Jedynak [12] and Arkin et al =-=[13]-=-), image processing studied by Korostelev and Kim [14], [15], and active learning research; for example the investigations by Freund et al [16], Dasgupta [17], Balcan et al [7], and Castro and Nowak [... |

30 | On a problem in information theory - Rényi - 1962 |

30 | personal communication - Scott |

27 |
Partition of space
- Buck
- 1943
(Show Context)
Citation Context ...H|. The error bound follows immediately from Theorem 4 since c ∗ = 0 and (R d , H) is 1-neighborly, as shown in Section V-B. The polynomial-time computational complexity follows from the work of Buck =-=[6]-=-, as discussed in Section V-B. Suppose that H is an ɛ-dense set with respect to a uniform probability measure on a ball in R d (i.e., for any hyperplane of the form (4) H contains a hypothesis whose p... |

27 |
Zigangirov, “An interval estimation problem for controlled observations
- Burnashev, Sh
- 1974
(Show Context)
Citation Context ...ilistic bisection procedure. The key elements of the procedure and the analysis of its convergence are fundamentally different from those in the classic binary search work of Burnashev and Zigangirov =-=[4]-=- and Karp and Kleinberg [3]. VII. CONCLUSIONS AND POSSIBLE EXTENSIONS This paper investigated a generalization of classic binary search, called GBS, that extends it to arbitrary query and hypothesis s... |

26 | Tsitsiklis, “Active learning using arbitrary binary valued queries
- Kulkarni, Mitter, et al.
- 1993
(Show Context)
Citation Context ...e successively halved using arbitrary queries, and so it is possible to determine h ∗ with log 2 N arbitrary queries, the informationtheoretically optimal query complexity discussed by Kulkarni et al =-=[8]-=-. Membership queries are the most natural in function learning problems, and because this paper deals only with this type we will simply refer to them as queries throughout the rest of the paper. The ... |

26 |
Searching in the presence of linearly bounded errors
- Aslam, Dhagat
- 1991
(Show Context)
Citation Context ...Korostelov and Kim [14], [15] for image recovery and binary classification problems studied by Castro and Nowak [18]. We also mention the works of Rivest et al [37], Spencer [38] and Aslam and Dhagat =-=[39]-=-, and Dhagat et al [40], which consider adversarial situations in which the total number of erroneous oracle responses is fixed in advance. One straightforward approach to noisy GBS is to follow the G... |

23 |
Decision tree design from a communication theory standpoint
- Goodman, Smyth
- 1988
(Show Context)
Citation Context ...of GBS is proportional to N |X | in this case. VI. RELATED WORK Generalized binary search can be viewed as a generalization of classic binary search, Shannon-Fano coding as noted by Goodman and Smyth =-=[20]-=-, and channel coding with noiseless feedback as studied by Horstein [21]. Problems of this nature arise in many applications, including channel coding (e.g., the work of Horstein [21] and Zigangirov [... |

22 | Performance bounds on the splitting algorithm for binary testing
- Garey, Graham
- 1974
(Show Context)
Citation Context ...es that are within a factor of log N of the optimal search tree depth; this result has been discovered independently by several researchers including Loveland [25], Garey and Graham [25], Arkin et al =-=[28]-=-, and Dasgupta [17]. The greedy procedure is referred to here as Generalized Binary Search (GBS) or the splitting algorithm, and it reduces to classic binary search, as discussed in Section V-A. The n... |

21 |
On an Optimal Split Tree Problem
- Kosaraju, Przytycka, et al.
- 1999
(Show Context)
Citation Context ...udied by Rényi [23], [24]), disease diagnosis (e.g., see the work of Loveland [25]), fault-tolerant computing (e.g., the work of Feige et al [26]), the scheduling problem considered by Kosaraju et al =-=[27]-=-, computer vision problems investigated by Geman and Jedynak [12] and Arkin et al [13]), image processing problems studied by Korostelev and Kim [14], [15], and active learning research; for example t... |

19 |
Noisy binary search and its applications
- Karp, Kleinberg
(Show Context)
Citation Context ...e idea of repeating each query of the GBS several times, in order to overcome the uncertainty introduced by the6 noise. Similar approaches are proposed in the work Kääriäinen [2]. Karp and Kleinberg =-=[3]-=- analyze of this strategy for noise-tolerant classic binary search. This is essentially like using a simple repetition code to communicate over a noisy channel. This procedure is termed noise-tolerant... |

17 | On playing twenty questions with a liar
- Dhagat, Gacs, et al.
- 2004
(Show Context)
Citation Context ..., [15] for image recovery and binary classification problems studied by Castro and Nowak [18]. We also mention the works of Rivest et al [37], Spencer [38] and Aslam and Dhagat [39], and Dhagat et al =-=[40]-=-, which consider adversarial situations in which the total number of erroneous oracle responses is fixed in advance. One straightforward approach to noisy GBS is to follow the GBS algorithm, but to re... |

16 |
Ulam’s searching game with a fixed number of lies
- Spencer
- 1992
(Show Context)
Citation Context ...ng procedures proposed by Korostelov and Kim [14], [15] for image recovery and binary classification problems studied by Castro and Nowak [18]. We also mention the works of Rivest et al [37], Spencer =-=[38]-=- and Aslam and Dhagat [39], and Dhagat et al [40], which consider adversarial situations in which the total number of erroneous oracle responses is fixed in advance. One straightforward approach to no... |

14 |
Sequential Decoding Using Noiseless Feedback
- Horstein
(Show Context)
Citation Context ...zed binary search can be viewed as a generalization of classic binary search, Shannon-Fano coding as noted by Goodman and Smyth [20], and channel coding with noiseless feedback as studied by Horstein =-=[21]-=-. Problems of this nature arise in many applications, including channel coding (e.g., the work of Horstein [21] and Zigangirov [22]), experimental design (e.g., as studied by Rényi [23], [24]), diseas... |

14 |
Performance bounds for binary testing with arbitrary weights
- Loveland
- 1985
(Show Context)
Citation Context ...ications, including channel coding (e.g., the work of Horstein [21] and Zigangirov [22]), experimental design (e.g., as studied by Rényi [23], [24]), disease diagnosis (e.g., see the work of Loveland =-=[25]-=-), fault-tolerant computing (e.g., the work of Feige et al [26]), the scheduling problem considered by Kosaraju et al [27], computer vision problems investigated by Geman and Jedynak [12] and Arkin et... |

11 |
On minimax rates of convergence in image models under sequential design
- Korostelev
- 1999
(Show Context)
Citation Context ...unctions arises commonly in computer vision (see the review of Swain and Stricker [11] and applications by Geman and Jedynak [12] and Arkin et al [13]), image processing studied by Korostelev and Kim =-=[14]-=-, [15], and active learning research; for example the investigations by Freund et al [16], Dasgupta [17], Balcan et al [7], and Castro and Nowak [18].15 First we show that the pair (R d , H) is 1-nei... |

10 | Upper and lower bounds for active learning - Castro, Nowak - 2006 |

5 |
Concurrency Theory
- on
- 2004
(Show Context)
Citation Context ...Horstein [21]. Problems of this nature arise in many applications, including channel coding (e.g., the work of Horstein [21] and Zigangirov [22]), experimental design (e.g., as studied by Rényi [23], =-=[24]-=-), disease diagnosis (e.g., see the work of Loveland [25]), fault-tolerant computing (e.g., the work of Feige et al [26]), the scheduling problem considered by Kosaraju et al [27], computer vision pro... |

4 |
Active learning in the non-realizable case,” in Algorithmic Learning Theory
- Kääriäinen
- 2006
(Show Context)
Citation Context ...thor in [41], and is based on a straightforward, iterated application of Chernoff’s bound. Similar strategies have been suggested as a general approach for devising noise-tolerant learning algorithms =-=[2]-=-. Using an adaptive procedure for adjusting the number of repetitions of each query yields an NGBS algorithm with20 query complexity bound proportional to log N log log N δ , the same order as that o... |

4 |
Coping with errors in binary search procedure
- Rivest, Meyer, et al.
- 1980
(Show Context)
Citation Context ...daptive sampling procedures proposed by Korostelov and Kim [10], [11] for image recovery and binary classification problems studied by Castro and Nowak [31]. We also mention the works of Rivest et al =-=[32]-=-, Spencer [33] and Aslam and Dhagat [35], and Dhagat et al [34], which consider adversarial situations in which the total number of erroneous oracle responses is fixed in advance. One straightforward ... |

2 |
On a problem in information theory,” MTA
- Rényi
- 1961
(Show Context)
Citation Context ...ed by Horstein [21]. Problems of this nature arise in many applications, including channel coding (e.g., the work of Horstein [21] and Zigangirov [22]), experimental design (e.g., as studied by Rényi =-=[23]-=-, [24]), disease diagnosis (e.g., see the work of Loveland [25]), fault-tolerant computing (e.g., the work of Feige et al [26]), the scheduling problem considered by Kosaraju et al [27], computer visi... |

1 |
Rates of convergence fo the sup-norm risk in image models under sequential designs
- Korostelev, Kim
- 2000
(Show Context)
Citation Context ...ns arises commonly in computer vision (see the review of Swain and Stricker [11] and applications by Geman and Jedynak [12] and Arkin et al [13]), image processing studied by Korostelev and Kim [14], =-=[15]-=-, and active learning research; for example the investigations by Freund et al [16], Dasgupta [17], Balcan et al [7], and Castro and Nowak [18].15 First we show that the pair (R d , H) is 1-neighborl... |

1 | The geometry of generalized binary search. 2009. Preprint available at http://arxiv.org/abs/0910.4397 - Nowak |

1 |
Upper bounds for the error probability of feedback channels
- Zigangirov
- 1970
(Show Context)
Citation Context ...], and channel coding with noiseless feedback as studied by Horstein [21]. Problems of this nature arise in many applications, including channel coding (e.g., the work of Horstein [21] and Zigangirov =-=[22]-=-), experimental design (e.g., as studied by Rényi [23], [24]), disease diagnosis (e.g., see the work of Loveland [25]), fault-tolerant computing (e.g., the work of Feige et al [26]), the scheduling pr... |

1 |
Searching games with erro r– fifty years of coping with liars
- Pelc
- 2002
(Show Context)
Citation Context ... problem of sequential transmission over a binary symmetric channel with noiseless feedback, as formulated by Horstein [21] and studied by Burnashev and Zigangirov [4] and more recently by Pelc et al =-=[30]-=-, is equivalent to a noisy binary search problem. There is a large literature on learning from queries; see the review articles by Angluin [31], [32]. This paper focuses exclusively on membership quer... |

1 |
revisited,” Springer Lecture Notes
- “Queries
- 2001
(Show Context)
Citation Context ...and Zigangirov [4] and more recently by Pelc et al [30], is equivalent to a noisy binary search problem. There is a large literature on learning from queries; see the review articles by Angluin [31], =-=[32]-=-. This paper focuses exclusively on membership queries (i.e., an x ∈ X is the query and the response is h∗ (x)), although18 other types of queries (equivalence, subset, superset, disjointness, and ex... |

1 |
Message transmission in a binary symmetric channel with noiseless feedback 9random transmission time),” Probl
- Zigangirov
- 1968
(Show Context)
Citation Context ...been first posed by Rényi [23]. The noisy binary search problem arises in sequential transmission over a binary symmetric channel with noiseless feedback studied by Horstein [21] and Zigangirov [22], =-=[36]-=-. The survey paper by Pelc et al [30] discusses the connections between search and coding problems. In channel coding with feedback, each threshold corresponds to a unique binary codeword (the binary ... |

1 | bounds for active learning - “Minimax - 2008 |