Limits...
Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development.

Makkar SR, Williamson A, Turner T, Redman S, Louviere J - Health Res Policy Syst (2015)

Bottom Line: The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document.The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

View Article: PubMed Central - PubMed

Affiliation: The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, NSW, 2007, Australia. steve.makkar@saxinstitute.org.au.

ABSTRACT

Background: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use.

Results: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.

Conclusions: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

No MeSH data available.


Subactions and levels for each research use domain.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4523001&req=5

Fig2: Subactions and levels for each research use domain.

Mentions: A vast number of examples of each research use domain were identified from the literature (over 100) and interviews (approximately 40). Similar examples were then categorised into groups. Each group was given an action label that encompassed all the examples within that group. These action labels became the subactions for a particular research use domain. For example, using research to understand the current prevalence rate of a disease and using research to understand risk factors for a particular health condition, were both examples of Conceptual Research Use identified in the literature. These two examples were grouped together to form a specific subaction of Conceptual Research Use: using research to inform one’s general background understanding of the health issue (subaction 1a; Figure 2).Figure 2


Using conjoint analysis to develop a system of scoring policymakers' use of research in policy and program development.

Makkar SR, Williamson A, Turner T, Redman S, Louviere J - Health Res Policy Syst (2015)

Subactions and levels for each research use domain.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4523001&req=5

Fig2: Subactions and levels for each research use domain.
Mentions: A vast number of examples of each research use domain were identified from the literature (over 100) and interviews (approximately 40). Similar examples were then categorised into groups. Each group was given an action label that encompassed all the examples within that group. These action labels became the subactions for a particular research use domain. For example, using research to understand the current prevalence rate of a disease and using research to understand risk factors for a particular health condition, were both examples of Conceptual Research Use identified in the literature. These two examples were grouped together to form a specific subaction of Conceptual Research Use: using research to inform one’s general background understanding of the health issue (subaction 1a; Figure 2).Figure 2

Bottom Line: The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document.The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

View Article: PubMed Central - PubMed

Affiliation: The Sax Institute, Level 13, Building 10, 235 Jones Street, Ultimo, NSW, 2007, Australia. steve.makkar@saxinstitute.org.au.

ABSTRACT

Background: The importance of utilising the best available research evidence in the development of health policies, services, and programs is increasingly recognised, yet few standardised systems for quantifying policymakers' research use are available. We developed a comprehensive measurement and scoring tool that assesses four domains of research use (i.e. instrumental, conceptual, tactical, and imposed). The scoring tool breaks down each domain into its key subactions like a checklist. Our aim was to develop a tool that assigned appropriate scores to each subaction based on its relative importance to undertaking evidence-informed health policymaking. In order to establish the relative importance of each research use subaction and generate this scoring system, we conducted conjoint analysis with a sample of knowledge translation experts.

Methods: Fifty-four experts were recruited to undertake four choice surveys. Respondents were shown combinations of research use subactions called profiles, and rated on a 1 to 9 scale whether each profile represented a limited (1-3), moderate (4-6), or extensive (7-9) example of research use. Generalised Estimating Equations were used to analyse respondents' choice data, which calculated a utility coefficient for each subaction. A large utility coefficient indicated that a subaction was particularly influential in guiding experts' ratings of extensive research use.

Results: Utility coefficients were calculated for each subaction, which became the points assigned to the subactions in the scoring system. The following subactions yielded the largest utilities and were regarded as the most important components of each research use domain: using research to directly influence the core of the policy decision; using research to inform alternative perspectives to deal with the policy issue; using research to persuade targeted stakeholders to support a predetermined decision; and using research because it was a mandated requirement by the policymaker's organisation.

Conclusions: We have generated an empirically derived and context-sensitive means of measuring and scoring the extent to which policymakers used research to inform the development of a policy document. The scoring system can be used by organisations to not only quantify the extent of their research use, but also to provide them with insights into potential strategies to improve subsequent research use.

No MeSH data available.