-
Adaptive Crowdsourcing Via Self-Supervised Learning
Authors:
Anmol Kagrecha,
Henrik Marklund,
Benjamin Van Roy,
Hong Jun Jeon,
Richard Zeckhauser
Abstract:
Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate. We develop a new approach -- predict-each-worker -- that leverages self-supervised learning and a novel aggregation scheme. This approach adapts weights assigned to crowdworkers based on estimates they provided for previous quantities. When skills vary across c…
▽ More
Common crowdsourcing systems average estimates of a latent quantity of interest provided by many crowdworkers to produce a group estimate. We develop a new approach -- predict-each-worker -- that leverages self-supervised learning and a novel aggregation scheme. This approach adapts weights assigned to crowdworkers based on estimates they provided for previous quantities. When skills vary across crowdworkers or their estimates correlate, the weighted sum offers a more accurate group estimate than the average. Existing algorithms such as expectation maximization can, at least in principle, produce similarly accurate group estimates. However, their computational requirements become onerous when complex models, such as neural networks, are required to express relationships among crowdworkers. Predict-each-worker accommodates such complexity as well as many other practical challenges. We analyze the efficacy of predict-each-worker through theoretical and computational studies. Among other things, we establish asymptotic optimality as the number of engagements per crowdworker grows.
△ Less
Submitted 1 February, 2024; v1 submitted 24 January, 2024;
originally announced January 2024.
-
Playing Divide-and-Choose Given Uncertain Preferences
Authors:
Jamie Tucker-Foltz,
Richard Zeckhauser
Abstract:
We study the classic divide-and-choose method for equitably allocating divisible goods between two players who are rational, self-interested Bayesian agents. The players have additive values for the goods. The prior distributions on those values are common knowledge. We consider both the cases of independent values and values that are correlated across players (as occurs when there is a common-val…
▽ More
We study the classic divide-and-choose method for equitably allocating divisible goods between two players who are rational, self-interested Bayesian agents. The players have additive values for the goods. The prior distributions on those values are common knowledge. We consider both the cases of independent values and values that are correlated across players (as occurs when there is a common-value component).
We describe the structure of optimal divisions in the divide-and-choose game and identify several cases where it is possible to efficiently compute equilibria. An approximation algorithm is presented for the case when the distribution over the chooser's value for each good follows a normal distribution, along with a randomized approximation algorithm for the case of uniform distributions over intervals.
A mixture of analytic results and computational simulations illuminates several striking differences between optimal strategies in the cases of known versus unknown preferences. Most notably, given unknown preferences, the divider has a compelling "diversification" incentive in creating the chooser's two options. This incentive leads to multiple goods being divided at equilibrium, quite contrary to the divider's optimal strategy when preferences are known.
In many contexts, such as buy-and-sell provisions between partners, or in judging fairness, it is important to assess the relative expected utilities of the divider and chooser. Those utilities, we show, depend on the players' levels of knowledge about each other's values, the correlations between the players' values, and the number of goods being divided. Under fairly mild assumptions, we show that the chooser is strictly better off for a small number of goods, while the divider is strictly better off for a large number of goods.
△ Less
Submitted 18 October, 2024; v1 submitted 7 July, 2022;
originally announced July 2022.
-
The Online Laboratory: Conducting Experiments in a Real Labor Market
Authors:
John J. Horton,
David G. Rand,
Richard J. Zeckhauser
Abstract:
Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid---both internally and externally---as laboratory and field experiments, while requiring far less money and time to design and to…
▽ More
Online labor markets have great potential as platforms for conducting experiments, as they provide immediate access to a large and diverse subject pool and allow researchers to conduct randomized controlled trials. We argue that online experiments can be just as valid---both internally and externally---as laboratory and field experiments, while requiring far less money and time to design and to conduct. In this paper, we first describe the benefits of conducting experiments in online labor markets; we then use one such market to replicate three classic experiments and confirm their results. We confirm that subjects (1) reverse decisions in response to how a decision-problem is framed, (2) have pro-social preferences (value payoffs to others positively), and (3) respond to priming by altering their choices. We also conduct a labor supply field experiment in which we confirm that workers have upward sloping labor supply curves. In addition to reporting these results, we discuss the unique threats to validity in an online setting and propose methods for coping with these threats. We also discuss the external validity of results from online domains and explain why online results can have external validity equal to or even better than that of traditional methods, depending on the research question. We conclude with our views on the potential role that online experiments can play within the social sciences, and then recommend software development priorities and best practices.
△ Less
Submitted 16 April, 2010;
originally announced April 2010.
