Limits...
Participation and contribution in crowdsourced surveys.

Swain R, Berger A, Bongard J, Hines P - PLoS ONE (2015)

Bottom Line: In particular, we found that: the rate at which participants submitted new survey questions followed a heavy-tailed distribution; the distribution in the types of questions posed was similar; and many users posed non-obvious yet predictive questions.While we did not find a significant relationship between the quantity of participation and the quality of contribution for both response submissions and question submissions, we did find several other more nuanced participant behavior patterns, which did correlate with contribution in one of the three surveys.We conclude that there exists an optimal time for users to pose questions early on in their participation, but only after they have submitted a few responses to other questions.

View Article: PubMed Central - PubMed

Affiliation: Computer Science Department (graduated), University of Vermont, Burlington, United States of America.

ABSTRACT
This paper identifies trends within and relationships between the amount of participation and the quality of contributions in three crowdsourced surveys. Participants were asked to perform a collective problem solving task that lacked any explicit incentive: they were instructed not only to respond to survey questions but also to pose new questions that they thought might-if responded to by others-predict an outcome variable of interest to them. While the three surveys had very different outcome variables, target audiences, methods of advertisement, and lengths of deployment, we found very similar patterns of collective behavior. In particular, we found that: the rate at which participants submitted new survey questions followed a heavy-tailed distribution; the distribution in the types of questions posed was similar; and many users posed non-obvious yet predictive questions. By analyzing responses to questions that contained a built-in range of valid response we found that less than 0.2% of responses lay outside of those ranges, indicating that most participants tend to respond honestly to surveys of this form, even without explicit incentives for honesty. While we did not find a significant relationship between the quantity of participation and the quality of contribution for both response submissions and question submissions, we did find several other more nuanced participant behavior patterns, which did correlate with contribution in one of the three surveys. We conclude that there exists an optimal time for users to pose questions early on in their participation, but only after they have submitted a few responses to other questions. This suggests that future crowdsourced surveys may attract more predictive questions by prompting users to pose new questions at specific times during their participation and limiting question submission at non-optimal times.

No MeSH data available.


Question contribution after submitting responses in the EnergyMinder survey.The model of size 14 from Table 10 is used to investigate the relationship between question contribution and the number of responses submitted before the user submitted their question. The variable ‘qs_sub’ is kept constant at the mean value of 7.98.
© Copyright Policy
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC4383627&req=5

pone.0120521.g006: Question contribution after submitting responses in the EnergyMinder survey.The model of size 14 from Table 10 is used to investigate the relationship between question contribution and the number of responses submitted before the user submitted their question. The variable ‘qs_sub’ is kept constant at the mean value of 7.98.

Mentions: Table 10 reports the models found by Eureqa using this reduced EnergyMinder dataset. The variables ‘rbq’ and ‘qpq’ appear in many of the models, and always in the denominator. Along with ‘tos’ and ‘tbq’, these four variables are proxies for the amount of activity that occurs before the question is submitted. While variable ‘tbq’ does not appear in any of the models, the other three variables all appear in models in an inverse context. This suggests that questions submitted later in the study were not as predictive of the outcome variable. However this is just a generalization that does not always hold true as we see when looking at the placement of variable ‘rbq’ in the five largest models. In these five models, ‘rbq’ appears twice in the denominator: once as a squared term and once with a negative coefficient. This pattern of usage of the variable ‘rbq’ begins in the model of size 14 and, along with the usage of ‘qs_sub’, results in a substantial increase in R2 over the model of size 10. A roughly linear positive relationship exists between the variable ‘qs_sub’ and ‘c’. For ‘rbq’, the net effect of these two appearences in the model of size 14 impacts ‘c’ in a positive way when the user had submitted fewer than eight responses before posing their question and negatively when the user had submitted more than eight responses before posing their question (Fig. 6). We found that the average number of responses submitted before question submission was 174, so more often than not, ‘rbq’ is used in these models to reduce the value of ‘c’, yet it is useful to know that at low values of ‘rbq’, it actually increases the value of ‘c’ in these models. This suggests that questions submitted late in the study, after participants had already contributed a great deal, were of relatively low quality. However, taking the time to submit a few responses before submitting a question proved beneficial to question quality. We conclude from this that there appears to be a “sweet spot” when participating in a crowdsourced survey: it is best to submit a few responses before submitting a question, but not too many.


Participation and contribution in crowdsourced surveys.

Swain R, Berger A, Bongard J, Hines P - PLoS ONE (2015)

Question contribution after submitting responses in the EnergyMinder survey.The model of size 14 from Table 10 is used to investigate the relationship between question contribution and the number of responses submitted before the user submitted their question. The variable ‘qs_sub’ is kept constant at the mean value of 7.98.
© Copyright Policy
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC4383627&req=5

pone.0120521.g006: Question contribution after submitting responses in the EnergyMinder survey.The model of size 14 from Table 10 is used to investigate the relationship between question contribution and the number of responses submitted before the user submitted their question. The variable ‘qs_sub’ is kept constant at the mean value of 7.98.
Mentions: Table 10 reports the models found by Eureqa using this reduced EnergyMinder dataset. The variables ‘rbq’ and ‘qpq’ appear in many of the models, and always in the denominator. Along with ‘tos’ and ‘tbq’, these four variables are proxies for the amount of activity that occurs before the question is submitted. While variable ‘tbq’ does not appear in any of the models, the other three variables all appear in models in an inverse context. This suggests that questions submitted later in the study were not as predictive of the outcome variable. However this is just a generalization that does not always hold true as we see when looking at the placement of variable ‘rbq’ in the five largest models. In these five models, ‘rbq’ appears twice in the denominator: once as a squared term and once with a negative coefficient. This pattern of usage of the variable ‘rbq’ begins in the model of size 14 and, along with the usage of ‘qs_sub’, results in a substantial increase in R2 over the model of size 10. A roughly linear positive relationship exists between the variable ‘qs_sub’ and ‘c’. For ‘rbq’, the net effect of these two appearences in the model of size 14 impacts ‘c’ in a positive way when the user had submitted fewer than eight responses before posing their question and negatively when the user had submitted more than eight responses before posing their question (Fig. 6). We found that the average number of responses submitted before question submission was 174, so more often than not, ‘rbq’ is used in these models to reduce the value of ‘c’, yet it is useful to know that at low values of ‘rbq’, it actually increases the value of ‘c’ in these models. This suggests that questions submitted late in the study, after participants had already contributed a great deal, were of relatively low quality. However, taking the time to submit a few responses before submitting a question proved beneficial to question quality. We conclude from this that there appears to be a “sweet spot” when participating in a crowdsourced survey: it is best to submit a few responses before submitting a question, but not too many.

Bottom Line: In particular, we found that: the rate at which participants submitted new survey questions followed a heavy-tailed distribution; the distribution in the types of questions posed was similar; and many users posed non-obvious yet predictive questions.While we did not find a significant relationship between the quantity of participation and the quality of contribution for both response submissions and question submissions, we did find several other more nuanced participant behavior patterns, which did correlate with contribution in one of the three surveys.We conclude that there exists an optimal time for users to pose questions early on in their participation, but only after they have submitted a few responses to other questions.

View Article: PubMed Central - PubMed

Affiliation: Computer Science Department (graduated), University of Vermont, Burlington, United States of America.

ABSTRACT
This paper identifies trends within and relationships between the amount of participation and the quality of contributions in three crowdsourced surveys. Participants were asked to perform a collective problem solving task that lacked any explicit incentive: they were instructed not only to respond to survey questions but also to pose new questions that they thought might-if responded to by others-predict an outcome variable of interest to them. While the three surveys had very different outcome variables, target audiences, methods of advertisement, and lengths of deployment, we found very similar patterns of collective behavior. In particular, we found that: the rate at which participants submitted new survey questions followed a heavy-tailed distribution; the distribution in the types of questions posed was similar; and many users posed non-obvious yet predictive questions. By analyzing responses to questions that contained a built-in range of valid response we found that less than 0.2% of responses lay outside of those ranges, indicating that most participants tend to respond honestly to surveys of this form, even without explicit incentives for honesty. While we did not find a significant relationship between the quantity of participation and the quality of contribution for both response submissions and question submissions, we did find several other more nuanced participant behavior patterns, which did correlate with contribution in one of the three surveys. We conclude that there exists an optimal time for users to pose questions early on in their participation, but only after they have submitted a few responses to other questions. This suggests that future crowdsourced surveys may attract more predictive questions by prompting users to pose new questions at specific times during their participation and limiting question submission at non-optimal times.

No MeSH data available.