Limits...
Piloting an approach to rapid and automated assessment of a new research initiative: Application to the National Cancer Institute's Provocative Questions initiative.

Hsu ER, Williams DE, Dijoseph LG, Schnell JD, Finstad SL, Lee JS, Greenspan EJ, Corrigan JG - Res Eval (2013)

Bottom Line: Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing.Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements.We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.

View Article: PubMed Central - PubMed

Affiliation: Office of Science Planning and Assessment, National Cancer Institute, Bethesda, MD 20892, USA Thomson Reuters, Rockville, MD 20850, USA and Center for Strategic Scientific Initiatives, National Cancer Institute, Bethesda, MD 20892, USA.

ABSTRACT
Funders of biomedical research are often challenged to understand how a new funding initiative fits within the agency's portfolio and the larger research community. While traditional assessment relies on retrospective review by subject matter experts, it is now feasible to design portfolio assessment and gap analysis tools leveraging administrative and grant application data that can be used for early and continued analysis. We piloted such methods on the National Cancer Institute's Provocative Questions (PQ) initiative to address key questions regarding diversity of applicants; whether applicants were proposing new avenues of research; and whether grant applications were filling portfolio gaps. For the latter two questions, we defined measurements called focus shift and relevance, respectively, based on text similarity scoring. We demonstrate that two types of applicants were attracted by the PQs at rates greater than or on par with the general National Cancer Institute applicant pool: those with clinical degrees and new investigators. Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing. Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements. We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.

No MeSH data available.


Related in: MedlinePlus

Gini index for each question. Higher values indicate higher diversity in applicant discipline categories. The maximum possible Gini index for a perfectly uniform distribution over six categories is 0.833.
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC3814301&req=5

rvt024-F1: Gini index for each question. Higher values indicate higher diversity in applicant discipline categories. The maximum possible Gini index for a perfectly uniform distribution over six categories is 0.833.

Mentions: Figure 1 shows the computed Gini index for each question. Question 2, which had applicants from all six discipline categories, had the highest Gini index. A number of questions had a Gini index of 0.691 or higher (questions 10, 19, 24, 20, 14, and 17). Question 11 had the lowest Gini index; although it did have applicants from four discipline categories, it was dominated by those from the basic/life sciences. Question 16, which only had applicants from the MD and MD/PhD disciplines, had the second lowest Gini index.Figure 1.


Piloting an approach to rapid and automated assessment of a new research initiative: Application to the National Cancer Institute's Provocative Questions initiative.

Hsu ER, Williams DE, Dijoseph LG, Schnell JD, Finstad SL, Lee JS, Greenspan EJ, Corrigan JG - Res Eval (2013)

Gini index for each question. Higher values indicate higher diversity in applicant discipline categories. The maximum possible Gini index for a perfectly uniform distribution over six categories is 0.833.
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC3814301&req=5

rvt024-F1: Gini index for each question. Higher values indicate higher diversity in applicant discipline categories. The maximum possible Gini index for a perfectly uniform distribution over six categories is 0.833.
Mentions: Figure 1 shows the computed Gini index for each question. Question 2, which had applicants from all six discipline categories, had the highest Gini index. A number of questions had a Gini index of 0.691 or higher (questions 10, 19, 24, 20, 14, and 17). Question 11 had the lowest Gini index; although it did have applicants from four discipline categories, it was dominated by those from the basic/life sciences. Question 16, which only had applicants from the MD and MD/PhD disciplines, had the second lowest Gini index.Figure 1.

Bottom Line: Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing.Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements.We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.

View Article: PubMed Central - PubMed

Affiliation: Office of Science Planning and Assessment, National Cancer Institute, Bethesda, MD 20892, USA Thomson Reuters, Rockville, MD 20850, USA and Center for Strategic Scientific Initiatives, National Cancer Institute, Bethesda, MD 20892, USA.

ABSTRACT
Funders of biomedical research are often challenged to understand how a new funding initiative fits within the agency's portfolio and the larger research community. While traditional assessment relies on retrospective review by subject matter experts, it is now feasible to design portfolio assessment and gap analysis tools leveraging administrative and grant application data that can be used for early and continued analysis. We piloted such methods on the National Cancer Institute's Provocative Questions (PQ) initiative to address key questions regarding diversity of applicants; whether applicants were proposing new avenues of research; and whether grant applications were filling portfolio gaps. For the latter two questions, we defined measurements called focus shift and relevance, respectively, based on text similarity scoring. We demonstrate that two types of applicants were attracted by the PQs at rates greater than or on par with the general National Cancer Institute applicant pool: those with clinical degrees and new investigators. Focus shift scores tended to be relatively low, with applicants not straying far from previous research, but the majority of applications were found to be relevant to the PQ the application was addressing. Sensitivity to comparison text and inability to distinguish subtle scientific nuances are the primary limitations of our automated approaches based on text similarity, potentially biasing relevance and focus shift measurements. We also discuss potential uses of the relevance and focus shift measures including the design of outcome evaluations, though further experimentation and refinement are needed for a fuller understanding of these measures before broad application.

No MeSH data available.


Related in: MedlinePlus