Limits...
Can Simple Transmission Chains Foster Collective Intelligence in Binary-Choice Tasks?

View Article: PubMed Central - PubMed

ABSTRACT

In many social systems, groups of individuals can find remarkably efficient solutions to complex cognitive problems, sometimes even outperforming a single expert. The success of the group, however, crucially depends on how the judgments of the group members are aggregated to produce the collective answer. A large variety of such aggregation methods have been described in the literature, such as averaging the independent judgments, relying on the majority or setting up a group discussion. In the present work, we introduce a novel approach for aggregating judgments—the transmission chain—which has not yet been consistently evaluated in the context of collective intelligence. In a transmission chain, all group members have access to a unique collective solution and can improve it sequentially. Over repeated improvements, the collective solution that emerges reflects the judgments of every group members. We address the question of whether such a transmission chain can foster collective intelligence for binary-choice problems. In a series of numerical simulations, we explore the impact of various factors on the performance of the transmission chain, such as the group size, the model parameters, and the structure of the population. The performance of this method is compared to those of the majority rule and the confidence-weighted majority. Finally, we rely on two existing datasets of individuals performing a series of binary decisions to evaluate the expected performances of the three methods empirically. We find that the parameter space where the transmission chain has the best performance rarely appears in real datasets. We conclude that the transmission chain is best suited for other types of problems, such as those that have cumulative properties.

No MeSH data available.


Impact of the group size N.(A) The color-coding indicates the probability of success of the chain method as a function of the group size N and the contribution threshold τ. The column of values at N = 10 corresponds to the red curve shown in Fig 3. (B) Comparison of the performance of the chain method with the contribution threshold set to τ = 0.85 (in red), the majority rule (in grey), and the weighted-majority rule (in blue). These results are computed assuming a proportion of correct answer q1 = 0.6 and the confidence distributions shown in Fig 1.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC5120860&req=5

pone.0167223.g004: Impact of the group size N.(A) The color-coding indicates the probability of success of the chain method as a function of the group size N and the contribution threshold τ. The column of values at N = 10 corresponds to the red curve shown in Fig 3. (B) Comparison of the performance of the chain method with the contribution threshold set to τ = 0.85 (in red), the majority rule (in grey), and the weighted-majority rule (in blue). These results are computed assuming a proportion of correct answer q1 = 0.6 and the confidence distributions shown in Fig 1.

Mentions: The weakness of the majority rule in the previous case study was the relatively small group size (N = 10). In fact, the majority of the individuals in the entire sample population do actually provide the correct answer. Yet, the majority of the N = 10 group members has only 73% chance to produce a correct answer because the majority within the group often points toward the wrong answer [25]. Hence, group size matters for the majority rule. How does it impact the outcome of the transmission chain? To address this question, we run an additional series of simulations, this time varying the contribution threshold τ as well as the group size N. We generate again 1000 groups of size N with q1 = 0.6 and the confidence distributions shown in Fig 1, and measure the frequency of correct answers produced by the chain for different values of τ and N. As Fig 4A shows, group size has relatively little influence on the chain performances, which increase rapidly until N ≈ 10 and plateaus for larger group sizes. This result is consistent with the previous result showing that the ratio between positive and negative contributors is more important than the total number of contributors. In addition, the optimal contribution threshold only marginally varies between τ = 0.8 and τ = 0.9 with increasing group size. Fig 4B compares the evolution of the majority, the weighted-majority and the chain with τ = 0.85 for increasing values of N. While the performance of the majority increases slowly with N, the weighted majority and the transmission chain reach higher performances for smaller group size. This weak dependency on N for these two methods results from the fact that they also rely on the individuals’ confidence and can thus extract the correct answer from a smaller number of individuals. The weighted-majority and the transmission chain have relatively similar performances in this environment, with the chain converging slightly faster to its best performance, and the weighted majority converging slower but reaching a slightly higher performance level.


Can Simple Transmission Chains Foster Collective Intelligence in Binary-Choice Tasks?
Impact of the group size N.(A) The color-coding indicates the probability of success of the chain method as a function of the group size N and the contribution threshold τ. The column of values at N = 10 corresponds to the red curve shown in Fig 3. (B) Comparison of the performance of the chain method with the contribution threshold set to τ = 0.85 (in red), the majority rule (in grey), and the weighted-majority rule (in blue). These results are computed assuming a proportion of correct answer q1 = 0.6 and the confidence distributions shown in Fig 1.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC5120860&req=5

pone.0167223.g004: Impact of the group size N.(A) The color-coding indicates the probability of success of the chain method as a function of the group size N and the contribution threshold τ. The column of values at N = 10 corresponds to the red curve shown in Fig 3. (B) Comparison of the performance of the chain method with the contribution threshold set to τ = 0.85 (in red), the majority rule (in grey), and the weighted-majority rule (in blue). These results are computed assuming a proportion of correct answer q1 = 0.6 and the confidence distributions shown in Fig 1.
Mentions: The weakness of the majority rule in the previous case study was the relatively small group size (N = 10). In fact, the majority of the individuals in the entire sample population do actually provide the correct answer. Yet, the majority of the N = 10 group members has only 73% chance to produce a correct answer because the majority within the group often points toward the wrong answer [25]. Hence, group size matters for the majority rule. How does it impact the outcome of the transmission chain? To address this question, we run an additional series of simulations, this time varying the contribution threshold τ as well as the group size N. We generate again 1000 groups of size N with q1 = 0.6 and the confidence distributions shown in Fig 1, and measure the frequency of correct answers produced by the chain for different values of τ and N. As Fig 4A shows, group size has relatively little influence on the chain performances, which increase rapidly until N ≈ 10 and plateaus for larger group sizes. This result is consistent with the previous result showing that the ratio between positive and negative contributors is more important than the total number of contributors. In addition, the optimal contribution threshold only marginally varies between τ = 0.8 and τ = 0.9 with increasing group size. Fig 4B compares the evolution of the majority, the weighted-majority and the chain with τ = 0.85 for increasing values of N. While the performance of the majority increases slowly with N, the weighted majority and the transmission chain reach higher performances for smaller group size. This weak dependency on N for these two methods results from the fact that they also rely on the individuals’ confidence and can thus extract the correct answer from a smaller number of individuals. The weighted-majority and the transmission chain have relatively similar performances in this environment, with the chain converging slightly faster to its best performance, and the weighted majority converging slower but reaching a slightly higher performance level.

View Article: PubMed Central - PubMed

ABSTRACT

In many social systems, groups of individuals can find remarkably efficient solutions to complex cognitive problems, sometimes even outperforming a single expert. The success of the group, however, crucially depends on how the judgments of the group members are aggregated to produce the collective answer. A large variety of such aggregation methods have been described in the literature, such as averaging the independent judgments, relying on the majority or setting up a group discussion. In the present work, we introduce a novel approach for aggregating judgments—the transmission chain—which has not yet been consistently evaluated in the context of collective intelligence. In a transmission chain, all group members have access to a unique collective solution and can improve it sequentially. Over repeated improvements, the collective solution that emerges reflects the judgments of every group members. We address the question of whether such a transmission chain can foster collective intelligence for binary-choice problems. In a series of numerical simulations, we explore the impact of various factors on the performance of the transmission chain, such as the group size, the model parameters, and the structure of the population. The performance of this method is compared to those of the majority rule and the confidence-weighted majority. Finally, we rely on two existing datasets of individuals performing a series of binary decisions to evaluate the expected performances of the three methods empirically. We find that the parameter space where the transmission chain has the best performance rarely appears in real datasets. We conclude that the transmission chain is best suited for other types of problems, such as those that have cumulative properties.

No MeSH data available.