� We survey the literature on prediction mechanisms, including prediction markets and peer prediction systems. We pay particular attention to the design process, highlighting the objectives and properties that are important in the design of good prediction mechanisms. Mechanism design has been described as “inverse game theory. ” Whereas game theorists ask what outcome results from a game, mechanism designers ask what game produces a desired outcome. In this sense, game theorists act like scientists and mechanism designers like engineers. In this article, we survey a number of mechanisms created to elicit predictions, many newly proposed within the last decade. We focus on the engineering questions: How do they work and why? What factors and goals are most important in their

Abstract. We consider the problem of truthfully sampling opinions of a population for statistical analysis purposes, such as estimating the population distribution of opinions. To obtain accurate results, the surveyor must incentivize individuals to report unbiased opinions. We present a rewarding scheme to elicit opinions that are representative of the population. In contrast with the related literature, we do not assume a specific information structure. In particular, our method does not rely on a common prior assumption. 1

We study a problem where a group of agents has to decide how a joint reward should be shared among them. We focus on settings where the share that each agent receives depends on the subjective opinions of its peers concerning that agent’s contribution to the group. To this end, we introduce a mechanism to elicit and aggregate subjective opinions as well as for determining agents ’ shares. The intuition behind the proposed mechanism is that each agent who believes that the others are telling the truth has its expected share maximized to the extent that it is well-evaluated by its peers and that it is truthfully reporting its opinions. Under the assumptions that agents are Bayesian decision-makers and that the underlying population is sufficiently large, we show that our mechanism is incentive-compatible, budgetbalanced, and tractable. We also present strategies to make this mechanism individually rational and fair.

Peer prediction methods allow the truthful elicitation of private signals (e.g., experiences, or opinions) in regard to a true world state when this ground truth is unobservable. The original peer prediction method is incentive compatible for any finite number of agents n ≥ 2 but critically relies on a common prior, shared by all agents and the center. The Bayesian Truth Serum (BTS) relaxes this assumption. While it still assumes that the agents share a common prior, this prior need not be known by the center. However, BTS is proven to be incentive compatible only for a large enough number of agents, and this number depends on the prior and is thus unknown to the mechanism. In this paper, we present a robust BTS for the elicitation of binary information which is incentive compatible for any n ≥ 3, taking advantage of a particularity of the quadratic scoring rule. Our mechanism is the first peer prediction method that does not rely on knowledge of the common prior to provide strict incentive compatibility for any n ≥ 3. Moreover, and in contrast to the original BTS, our mechanism is numerically robust and ex post individually rational.

Arguably, it all started with Mike Dertouzos ’ vision on the Information Marketplace [2]. Then, an explosion occurred. Social networks. Social computing. Social software. Groupware. Shareware. Open-source software. Personalized query answering and personalized information systems. Tagging. Folksonomies. Log and clickstream mining. Recommender systems. Crowdsourcing. Human-in-the loop and humancentered systems. Provably, these buzzwords have dominated the academic landscape within the data systems (and not only) community. There is a fundamental paradigm shift going on here. The old world, where the human was simply a passive user, has given way to a new world where humans contribute data, (storage, communication, and compute) resources, and software. Further, recently, humans take on tasks that actually alleviate and improve the jobs performed by machines and recent research from different domains have started looking into this realm where humans and computers share tasks, collaborating to achieve goals [4,