## Collaborative Expert Portfolio Management

Citations: | 6 - 3 self |

### BibTeX

@MISC{Stern_collaborativeexpert,

author = {David Stern and Ralf Herbrich and Thore Graepel and Horst Samulowitz and Luca Pulina and Armando Tacchella},

title = {Collaborative Expert Portfolio Management},

year = {}

}

### OpenURL

### Abstract

We consider the task of assigning experts from a portfolio of specialists in order to solve a set of tasks. We apply a Bayesian model which combines collaborative filtering with a feature-based description of tasks and experts to yield a general framework for managing a portfolio of experts. The model learns an embedding of tasks and problems into a latent space in which affinity is measured by the inner product. The model can be trained incrementally and can track non-stationary data, tracking potentially changing expert and task characteristics. The approach allows us to use a principled decision theoretic framework for expert selection, allowing the user to choose a utility function that best suits their objectives. The model component for taking into account the performance feedback data is pluggable, allowing flexibility. We apply the model to manage a portfolio of algorithms to solve hard combinatorial problems. This is a well studied area and we demonstrate a large improvement on the state of the art in one domain (constraint solving) and in a second domain (combinatorial auctions) created a portfolio that performed significantly better than any single algorithm.

### Citations

1189 | Factor graphs and the sum-product algorithm
- Kschischang, Frey, et al.
- 2001
(Show Context)
Citation Context ... latent variables: p(U, V, u, v|r, x, y) ∝ ∫ ∫ ∫ ∫ p(s, t, U, V, u, v, z, ˜r, r|x, y) ds dt dz d˜r. s t z ˜r This inference is performed by approximate message passing on the equivalent factor graph [=-=Kschischang, Frey, and Loeliger, 2001-=-]. We assume a full factorization of the joint distribution and minimize an α-divergence (a generalization of the Kullback-Leibler divergence) between the true and approximate marginals [Minka, 2005].... |

1036 | Empirical analysis of predictive algorithms for collaborative filtering
- Breese, Heckerman, et al.
- 1998
(Show Context)
Citation Context ...hey have previously rated and the users who have rated items in common with them, using implicit descriptions of users and items obtained from a (sparse) matrix of previous ratings of items by users [=-=Breese, Heckerman, and Kadie, 1998-=-; Varian and Resnick, 1997]. Matchbox [Stern, Herbrich, and Graepel, 2009] is a recommendation system model which combines these two sources of information and here we apply it to recommending experts... |

1017 |
Recommender systems
- Resnick, Varian
- 1997
(Show Context)
Citation Context ...ers who have rated items in common with them, using implicit descriptions of users and items obtained from a (sparse) matrix of previous ratings of items by users [Breese, Heckerman, and Kadie, 1998; =-=Varian and Resnick, 1997-=-]. Matchbox [Stern, Herbrich, and Graepel, 2009] is a recommendation system model which combines these two sources of information and here we apply it to recommending experts to tasks. Unlike other co... |

149 | Heavy-tailed phenomena in satisfiability and constraint satisfaction problems
- Gomes, Selman, et al.
- 2000
(Show Context)
Citation Context ...out, T , after which the algorithm is terminated and the problem is not solved. Typically, the distribution over the time it takes for algorithms to solve hard combinatorial problems has a long tail [=-=Gomes et al., 2000-=-]. With a modest time-out this means many problems are solved quite quickly but also many reach time-out with the remaining solution times being roughly evenly distributed between the time-out and the... |

86 |
The algorithm selection problem
- Rice
- 1976
(Show Context)
Citation Context ... divide and conquer approach using a portfolio of specialist experts. An example of this is the case where the expert is an algorithm [Smith-Miles, 2008; Gomes and Selman, 1997; Horvitz et al., 2001; =-=Rice, 1976-=-; Streeter and Smith, 2008]. A portfolio approach means that several parallel and independent efforts to design an algorithm for a task can be combined so as to leverage the best aspects of each of th... |

78 | Integrated Methods for Optimization
- Hooker
- 2007
(Show Context)
Citation Context ...to the scheduling problem [Streeter and Smith, 2008]. Using this method we can solve 2195 problems in the test set. Finally, we performed a similar analysis for Linear Program (LP) solvers (see e.g. [=-=Hooker, 2006-=-]). The portfolio Algorithm K #Solved Time Utility GL - 2531 1025 1249 CASS - 861 207 -1768 CPLEX - 2648 445 1672 Matchbox 1 2648 445 1672 Matchbox 2 2677 232 1804 Matchbox 3 2680 240 1807 Oracle - 27... |

72 | Gaussian Processes for Ordinal Regression
- Chu, Ghahramani
- 2005
(Show Context)
Citation Context ...ickly. The mapping from rank to latent performance may not be linear and this mapping can change from solver to solver. We relate the latent performance r to ranks l via a cumulative threshold model [=-=Chu and Ghahramani, 2005-=-; Stern, Herbrich, and Graepel, 2009]. For each solver, v, we maintain solver-specific thresholds hv ∈ R 2 which divide the latent rating axis into three consecutive intervals (h v(i−1), h v(i)) each ... |

63 | A Bayesian approach to tackling hard computational problems
- Horvitz, Ruan, et al.
- 2001
(Show Context)
Citation Context ...tasks. This suggests a divide and conquer approach using a portfolio of specialist experts. An example of this is the case where the expert is an algorithm [Smith-Miles, 2008; Gomes and Selman, 1997; =-=Horvitz et al., 2001-=-; Rice, 1976; Streeter and Smith, 2008]. A portfolio approach means that several parallel and independent efforts to design an algorithm for a task can be combined so as to leverage the best aspects o... |

48 | Divergence measures and message passing
- Minka
- 2005
(Show Context)
Citation Context ...oeliger, 2001]. We assume a full factorization of the joint distribution and minimize an α-divergence (a generalization of the Kullback-Leibler divergence) between the true and approximate marginals [=-=Minka, 2005-=-]. We approximate all factors by Gaussian distributions. The factor graph for this model and details of all the message calculations and update schedule are given in Stern, Herbrich, and Graepel [2009... |

46 |
Algorithm portfolio design : theory vs. practice
- P, Selman
- 1997
(Show Context)
Citation Context ...more effective at those tasks. This suggests a divide and conquer approach using a portfolio of specialist experts. An example of this is the case where the expert is an algorithm [Smith-Miles, 2008; =-=Gomes and Selman, 1997-=-; Horvitz et al., 2001; Rice, 1976; Streeter and Smith, 2008]. A portfolio approach means that several parallel and independent efforts to design an algorithm for a task can be combined so as to lever... |

31 | Matchbox: large scale online bayesian recommendations - Stern, Herbrich, et al. - 2009 |

26 |
Cross-disciplinary perspectives on metalearning for algorithm selection
- Smith-Miles
- 2009
(Show Context)
Citation Context ... often simpler and more effective at those tasks. This suggests a divide and conquer approach using a portfolio of specialist experts. An example of this is the case where the expert is an algorithm [=-=Smith-Miles, 2008-=-; Gomes and Selman, 1997; Horvitz et al., 2001; Rice, 1976; Streeter and Smith, 2008]. A portfolio approach means that several parallel and independent efforts to design an algorithm for a task can be... |

18 | Empirical hardness models: Methodology and a case study on combinatorial auctions
- Brown, Nudelman, et al.
(Show Context)
Citation Context ... portfolio can be beneficial is hard combinatorial problems where typically there does not exist one single approach that outperforms any other across a range of real-world problems [Xu et al., 2008; =-=Leyton-Brown, Nudelman, and Shoham, 2009-=-]. BeCopyright c○ 2010, Association for the Advancement of Artificial Intelligence (www.aaai.org). All rights reserved. cause of their exponentially large search spaces, these problems are in general ... |

14 | A.: A self-adaptive multi-engine solver for quantified boolean formulas
- Pulina, Tacchella
- 2009
(Show Context)
Citation Context ...e in algorithm performance can be exploited by machine learning methods to produce a portfolio of algorithms that greatly outperforms any individual algorithm in domains such as constraint reasoning [=-=Pulina and Tacchella, 2009-=-; Xu et al., 2008; Gomes and Selman, 1997] and combinatorial auctions [Leyton-Brown, Nudelman, and Shoham, 2009]. The goal of this paper is to address some of the limitations of previous work and prov... |

9 | New techniques for algorithm portfolio design
- Streeter, Smith
- 2008
(Show Context)
Citation Context ...conquer approach using a portfolio of specialist experts. An example of this is the case where the expert is an algorithm [Smith-Miles, 2008; Gomes and Selman, 1997; Horvitz et al., 2001; Rice, 1976; =-=Streeter and Smith, 2008-=-]. A portfolio approach means that several parallel and independent efforts to design an algorithm for a task can be combined so as to leverage the best aspects of each of them. An example class of ta... |

3 | Time to learn or time to forget? strengths and weaknesses of a self-adaptive approach to reasoning in quantified boolean formulas - Pulina, Tacchella - 2008 |

1 | Solving Quantified Boolean Formulas - Samulowitz |