Limits...
Evaluation considerations for EHR-based phenotyping algorithms: A case study for drug-induced liver injury.

Overby CL, Weng C, Haerian K, Perotte A, Friedman C, Hripcsak G - AMIA Jt Summits Transl Sci Proc (2013)

Bottom Line: We conduct a measurement study and report qualitative (i.e., perceptions of evaluation approach effectiveness) and quantitative (i.e., inter-rater reliability) measures.We also conduct a demonstration study and report qualitative (i.e., appropriateness of results) and quantitative (i.e., positive predictive value) measures.Results from the demonstration study informed changes to our algorithm.

View Article: PubMed Central - PubMed

Affiliation: Department of Biomedical Informatics, Columbia University, New York, NY.

ABSTRACT
Developing electronic health record (EHR) phenotyping algorithms involves generating queries that run across the EHR data repository. Algorithms are commonly assessed within demonstration studies. There remains, however, little emphasis on assessing the precision and accuracy of measurement methods during the evaluation process. Depending on the complexity of an algorithm, interim refinements may be required to improve measurement methods. Therefore, we develop an evaluation framework that incorporates both measurement and demonstration studies. We evaluate a baseline EHR phenotyping algorithm for drug induced liver injury (DILI) developed in collaboration with electronic Medical Records Genomics (eMERGE) network participants. We conduct a measurement study and report qualitative (i.e., perceptions of evaluation approach effectiveness) and quantitative (i.e., inter-rater reliability) measures. We also conduct a demonstration study and report qualitative (i.e., appropriateness of results) and quantitative (i.e., positive predictive value) measures. Given results from the measurement study, our evaluation approach underwent multiple changes including the addition of laboratory value visualization and an expanded review of clinical notes. Results from the demonstration study informed changes to our algorithm. For example, given the goal of eMERGE to identify patients who may have a genetic susceptibility to DILI, we excluded overdose patients.

No MeSH data available.


Related in: MedlinePlus

Baseline algorithm derived from iSAEC case definition.[1]
© Copyright Policy
Related In: Results  -  Collection


getmorefigures.php?uid=PMC3814479&req=5

f1-amia_tbi_2013_130: Baseline algorithm derived from iSAEC case definition.[1]

Mentions: We initially adapt a DILI case definition and algorithm informed by the International Serious Adverse Events Consortium (iSAEC).[1] Each portion of the case definition (Box 1, 1a–2b) is translated into a baseline executable EHR algorithm (Figure 1, A1-D). The algorithm leveraged primarily structured data from the NewYork-Presbyterian Hospital (NYP) clinical data warehouse (CDW). In the first step of the algorithm (Figure 1, A1; Box 1, 1), liver injury ICD-9 diagnosis and procedure codes were specified as inclusion criteria according to Observational Medical Outcomes Partnership (OMOP).[21] Exposure to a drug was specified as any medication ordered within 90 days prior to liver injury diagnosis (Figure 1, A2; Box 1, 1a). We excluded patients with a history of the same diagnosis from Step A2 within 5 years previous (Figure 1, B; Box 1, 1b). We also leveraged the NYP Medical Entities Dictionary (MED) that contains over 60,000 concepts organized into a sematic network of terms.[22, 23] We queried the MED hierarchy to compile laboratory codes including intestinal alkaline phosphatase (ALP), alanine aminotransferase (ALT), and serum bilirubin indirect (Bilirubin). We assess whether peak laboratory values within 180 days following a medication order were above thresholds specified by iSAEC (Figure 1, C1–C4; Box 1, 1a). We also used the MED to compile ICD-9 codes for other diagnoses (Figure 1, D; Box 1, 2b) for exclusion from algorithm results. We report counts at each step of the algorithm.


Evaluation considerations for EHR-based phenotyping algorithms: A case study for drug-induced liver injury.

Overby CL, Weng C, Haerian K, Perotte A, Friedman C, Hripcsak G - AMIA Jt Summits Transl Sci Proc (2013)

Baseline algorithm derived from iSAEC case definition.[1]
© Copyright Policy
Related In: Results  -  Collection

Show All Figures
getmorefigures.php?uid=PMC3814479&req=5

f1-amia_tbi_2013_130: Baseline algorithm derived from iSAEC case definition.[1]
Mentions: We initially adapt a DILI case definition and algorithm informed by the International Serious Adverse Events Consortium (iSAEC).[1] Each portion of the case definition (Box 1, 1a–2b) is translated into a baseline executable EHR algorithm (Figure 1, A1-D). The algorithm leveraged primarily structured data from the NewYork-Presbyterian Hospital (NYP) clinical data warehouse (CDW). In the first step of the algorithm (Figure 1, A1; Box 1, 1), liver injury ICD-9 diagnosis and procedure codes were specified as inclusion criteria according to Observational Medical Outcomes Partnership (OMOP).[21] Exposure to a drug was specified as any medication ordered within 90 days prior to liver injury diagnosis (Figure 1, A2; Box 1, 1a). We excluded patients with a history of the same diagnosis from Step A2 within 5 years previous (Figure 1, B; Box 1, 1b). We also leveraged the NYP Medical Entities Dictionary (MED) that contains over 60,000 concepts organized into a sematic network of terms.[22, 23] We queried the MED hierarchy to compile laboratory codes including intestinal alkaline phosphatase (ALP), alanine aminotransferase (ALT), and serum bilirubin indirect (Bilirubin). We assess whether peak laboratory values within 180 days following a medication order were above thresholds specified by iSAEC (Figure 1, C1–C4; Box 1, 1a). We also used the MED to compile ICD-9 codes for other diagnoses (Figure 1, D; Box 1, 2b) for exclusion from algorithm results. We report counts at each step of the algorithm.

Bottom Line: We conduct a measurement study and report qualitative (i.e., perceptions of evaluation approach effectiveness) and quantitative (i.e., inter-rater reliability) measures.We also conduct a demonstration study and report qualitative (i.e., appropriateness of results) and quantitative (i.e., positive predictive value) measures.Results from the demonstration study informed changes to our algorithm.

View Article: PubMed Central - PubMed

Affiliation: Department of Biomedical Informatics, Columbia University, New York, NY.

ABSTRACT
Developing electronic health record (EHR) phenotyping algorithms involves generating queries that run across the EHR data repository. Algorithms are commonly assessed within demonstration studies. There remains, however, little emphasis on assessing the precision and accuracy of measurement methods during the evaluation process. Depending on the complexity of an algorithm, interim refinements may be required to improve measurement methods. Therefore, we develop an evaluation framework that incorporates both measurement and demonstration studies. We evaluate a baseline EHR phenotyping algorithm for drug induced liver injury (DILI) developed in collaboration with electronic Medical Records Genomics (eMERGE) network participants. We conduct a measurement study and report qualitative (i.e., perceptions of evaluation approach effectiveness) and quantitative (i.e., inter-rater reliability) measures. We also conduct a demonstration study and report qualitative (i.e., appropriateness of results) and quantitative (i.e., positive predictive value) measures. Given results from the measurement study, our evaluation approach underwent multiple changes including the addition of laboratory value visualization and an expanded review of clinical notes. Results from the demonstration study informed changes to our algorithm. For example, given the goal of eMERGE to identify patients who may have a genetic susceptibility to DILI, we excluded overdose patients.

No MeSH data available.


Related in: MedlinePlus