Limits...
Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development.

Makkar SR, Williamson A, Turner T, Redman S, Louviere J - Health Res Policy Syst (2015)

Bottom Line: The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document.The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

View Article: PubMed Central - PubMed

Affiliation: The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, NSW, 2007, Australia. steve.makkar@saxinstitute.org.au.

ABSTRACT

Background: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use.

Results: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.

Conclusions: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

No MeSH data available.


Scoring tool for conceptual research use.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4523001&req=5

Fig4: Scoring tool for conceptual research use.

Mentions: Using conceptual research use as an example, if it was evident that research increased the policymakers’ core understanding (utility = 0.88) of background aspects of the health issue (utility = 1.19) and the policy context (utility = 1.41), and he/she cited specific examples of research (utility = 2.81), he/she would be assigned a score of 0.88 + 1.19 + 1.41 + 2.81 = 6.29 (out of 9), which would represent moderate conceptual research use (Figure 4). The full scoring tool is provided in Additional file 3.Figure 4


Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development.

Makkar SR, Williamson A, Turner T, Redman S, Louviere J - Health Res Policy Syst (2015)

Scoring tool for conceptual research use.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4523001&req=5

Fig4: Scoring tool for conceptual research use.
Mentions: Using conceptual research use as an example, if it was evident that research increased the policymakers’ core understanding (utility = 0.88) of background aspects of the health issue (utility = 1.19) and the policy context (utility = 1.41), and he/she cited specific examples of research (utility = 2.81), he/she would be assigned a score of 0.88 + 1.19 + 1.41 + 2.81 = 6.29 (out of 9), which would represent moderate conceptual research use (Figure 4). The full scoring tool is provided in Additional file 3.Figure 4

Bottom Line: The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document.The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

View Article: PubMed Central - PubMed

Affiliation: The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, NSW, 2007, Australia. steve.makkar@saxinstitute.org.au.

ABSTRACT

Background: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use.

Results: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.

Conclusions: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

No MeSH data available.