Limits...
Can Simple Transmission Chains Foster Collective Intelligence in Binary-Choice Tasks?

View Article: PubMed Central - PubMed

ABSTRACT

In many social systems, groups of individuals can find remarkably efficient solutions to complex cognitive problems, sometimes even outperforming a single expert. The success of the group, however, crucially depends on how the judgments of the group members are aggregated to produce the collective answer. A large variety of such aggregation methods have been described in the literature, such as averaging the independent judgments, relying on the majority or setting up a group discussion. In the present work, we introduce a novel approach for aggregating judgments—the transmission chain—which has not yet been consistently evaluated in the context of collective intelligence. In a transmission chain, all group members have access to a unique collective solution and can improve it sequentially. Over repeated improvements, the collective solution that emerges reflects the judgments of every group members. We address the question of whether such a transmission chain can foster collective intelligence for binary-choice problems. In a series of numerical simulations, we explore the impact of various factors on the performance of the transmission chain, such as the group size, the model parameters, and the structure of the population. The performance of this method is compared to those of the majority rule and the confidence-weighted majority. Finally, we rely on two existing datasets of individuals performing a series of binary decisions to evaluate the expected performances of the three methods empirically. We find that the parameter space where the transmission chain has the best performance rarely appears in real datasets. We conclude that the transmission chain is best suited for other types of problems, such as those that have cumulative properties.

No MeSH data available.


Description of the environment.Assumed distributions of confidence among the individuals who provide the correct answer to the problem (in blue), and among those who provide a wrong answer to the problem (in red). The interval of confidence values ranges from c = 0 (very uncertain) to c = 1 (very certain). The blue and red distributions are beta distributions with shape parameters α1 = 8 and β1 = 2 (mean value: 0.8), and α0 = 3 and β0 = 3 (mean value: 0.5), respectively. In the simulations, a proportion q1 of the sample population gives the correct answer and have confidence levels drawn from the blue distribution, and a proportion q0 = 1 − q1 of the sample population gives a wrong answer and have confidence levels drawn from the red distribution. In empirical data, the shape parameters of the blue and red distributions depend on the nature of the task.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC5120860&req=5

pone.0167223.g001: Description of the environment.Assumed distributions of confidence among the individuals who provide the correct answer to the problem (in blue), and among those who provide a wrong answer to the problem (in red). The interval of confidence values ranges from c = 0 (very uncertain) to c = 1 (very certain). The blue and red distributions are beta distributions with shape parameters α1 = 8 and β1 = 2 (mean value: 0.8), and α0 = 3 and β0 = 3 (mean value: 0.5), respectively. In the simulations, a proportion q1 of the sample population gives the correct answer and have confidence levels drawn from the blue distribution, and a proportion q0 = 1 − q1 of the sample population gives a wrong answer and have confidence levels drawn from the red distribution. In empirical data, the shape parameters of the blue and red distributions depend on the nature of the task.

Mentions: As a starting point, we assume a large sample population in which a proportion q1 of individuals would independently choose the correct solution to the problem in the absence of any interaction or social influence. Reversely, a proportion q0 = 1 − q1 of individuals in the sample population would independently choose the wrong solution. In addition, every individual’s answer is associated with a confidence level c describing how confident the individual is about his or her answer. We define the confidence level as a continuous value ranging from 0 to 1, where c = 0 refers to individuals who are very uncertain about their answer, and c = 1 refers to individuals who are very certain about their answer. In the simulations, we describe the confidence levels of the individuals who give the correct answer with a beta distribution Ω1 that has shape parameters α1 and β1, and the confidence levels of the individuals who give a wrong answer with a beta distribution Ω0 that has shape parameters α0 and β0. That is, the confidence levels follow different distributions depending on whether the associated answer is correct or wrong. For many problems, confidence can be a good proxy for accuracy, but this tendency is not systematic and often non-linear [8,10]. In fact, the shape of these two distributions depends on the nature and the statement of the problem. To begin with, we assume the two distributions Ω1 and Ω0 represented in Fig 1, for which correct answers are on average associated to higher confidence levels than wrong answers, but a considerable overlap exists between the two distributions (i.e. an individual with a wrong answer can possibly be more confident than an individual with the correct answer).


Can Simple Transmission Chains Foster Collective Intelligence in Binary-Choice Tasks?
Description of the environment.Assumed distributions of confidence among the individuals who provide the correct answer to the problem (in blue), and among those who provide a wrong answer to the problem (in red). The interval of confidence values ranges from c = 0 (very uncertain) to c = 1 (very certain). The blue and red distributions are beta distributions with shape parameters α1 = 8 and β1 = 2 (mean value: 0.8), and α0 = 3 and β0 = 3 (mean value: 0.5), respectively. In the simulations, a proportion q1 of the sample population gives the correct answer and have confidence levels drawn from the blue distribution, and a proportion q0 = 1 − q1 of the sample population gives a wrong answer and have confidence levels drawn from the red distribution. In empirical data, the shape parameters of the blue and red distributions depend on the nature of the task.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC5120860&req=5

pone.0167223.g001: Description of the environment.Assumed distributions of confidence among the individuals who provide the correct answer to the problem (in blue), and among those who provide a wrong answer to the problem (in red). The interval of confidence values ranges from c = 0 (very uncertain) to c = 1 (very certain). The blue and red distributions are beta distributions with shape parameters α1 = 8 and β1 = 2 (mean value: 0.8), and α0 = 3 and β0 = 3 (mean value: 0.5), respectively. In the simulations, a proportion q1 of the sample population gives the correct answer and have confidence levels drawn from the blue distribution, and a proportion q0 = 1 − q1 of the sample population gives a wrong answer and have confidence levels drawn from the red distribution. In empirical data, the shape parameters of the blue and red distributions depend on the nature of the task.
Mentions: As a starting point, we assume a large sample population in which a proportion q1 of individuals would independently choose the correct solution to the problem in the absence of any interaction or social influence. Reversely, a proportion q0 = 1 − q1 of individuals in the sample population would independently choose the wrong solution. In addition, every individual’s answer is associated with a confidence level c describing how confident the individual is about his or her answer. We define the confidence level as a continuous value ranging from 0 to 1, where c = 0 refers to individuals who are very uncertain about their answer, and c = 1 refers to individuals who are very certain about their answer. In the simulations, we describe the confidence levels of the individuals who give the correct answer with a beta distribution Ω1 that has shape parameters α1 and β1, and the confidence levels of the individuals who give a wrong answer with a beta distribution Ω0 that has shape parameters α0 and β0. That is, the confidence levels follow different distributions depending on whether the associated answer is correct or wrong. For many problems, confidence can be a good proxy for accuracy, but this tendency is not systematic and often non-linear [8,10]. In fact, the shape of these two distributions depends on the nature and the statement of the problem. To begin with, we assume the two distributions Ω1 and Ω0 represented in Fig 1, for which correct answers are on average associated to higher confidence levels than wrong answers, but a considerable overlap exists between the two distributions (i.e. an individual with a wrong answer can possibly be more confident than an individual with the correct answer).

View Article: PubMed Central - PubMed

ABSTRACT

In many social systems, groups of individuals can find remarkably efficient solutions to complex cognitive problems, sometimes even outperforming a single expert. The success of the group, however, crucially depends on how the judgments of the group members are aggregated to produce the collective answer. A large variety of such aggregation methods have been described in the literature, such as averaging the independent judgments, relying on the majority or setting up a group discussion. In the present work, we introduce a novel approach for aggregating judgments—the transmission chain—which has not yet been consistently evaluated in the context of collective intelligence. In a transmission chain, all group members have access to a unique collective solution and can improve it sequentially. Over repeated improvements, the collective solution that emerges reflects the judgments of every group members. We address the question of whether such a transmission chain can foster collective intelligence for binary-choice problems. In a series of numerical simulations, we explore the impact of various factors on the performance of the transmission chain, such as the group size, the model parameters, and the structure of the population. The performance of this method is compared to those of the majority rule and the confidence-weighted majority. Finally, we rely on two existing datasets of individuals performing a series of binary decisions to evaluate the expected performances of the three methods empirically. We find that the parameter space where the transmission chain has the best performance rarely appears in real datasets. We conclude that the transmission chain is best suited for other types of problems, such as those that have cumulative properties.

No MeSH data available.