Limits...
Concordance and predictive value of two adverse drug event data sets.

Cami A, Reis BY - BMC Med Inform Decis Mak (2014)

Bottom Line: A minor difference of 1.1% was found in the AUROC of PPN when SIDER 2012 was used for validation instead of Lexi-comp 2010.In conclusion, the ADE and drug counts in Lexi-comp and SIDER data sets were highly correlated and the choice of validation set did not greatly affect the overall prediction performance of PPN.Our results also suggest that it is important to be aware of the differences that exist among ADE data sets, especially in modeling applications focused on specific drug and ADE categories.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Emergency Medicine, Boston Children's Hospital, 1 Autumn Street, 5th Floor, Boston, MA 02215, USA. aurel.cami@childrens.harvard.edu.

ABSTRACT

Background: Accurate prediction of adverse drug events (ADEs) is an important means of controlling and reducing drug-related morbidity and mortality. Since no single "gold standard" ADE data set exists, a range of different drug safety data sets are currently used for developing ADE prediction models. There is a critical need to assess the degree of concordance between these various ADE data sets and to validate ADE prediction models against multiple reference standards.

Methods: We systematically evaluated the concordance of two widely used ADE data sets - Lexi-comp from 2010 and SIDER from 2012. The strength of the association between ADE (drug) counts in Lexi-comp and SIDER was assessed using Spearman rank correlation, while the differences between the two data sets were characterized in terms of drug categories, ADE categories and ADE frequencies. We also performed a comparative validation of the Predictive Pharmacosafety Networks (PPN) model using both ADE data sets. The predictive power of PPN using each of the two validation sets was assessed using the area under Receiver Operating Characteristic curve (AUROC).

Results: The correlations between the counts of ADEs and drugs in the two data sets were 0.84 (95% CI: 0.82-0.86) and 0.92 (95% CI: 0.91-0.93), respectively. Relative to an earlier snapshot of Lexi-comp from 2005, Lexi-comp 2010 and SIDER 2012 introduced a mean of 1,973 and 4,810 new drug-ADE associations per year, respectively. The difference between these two data sets was most pronounced for Nervous System and Anti-infective drugs, Gastrointestinal and Nervous System ADEs, and postmarketing ADEs. A minor difference of 1.1% was found in the AUROC of PPN when SIDER 2012 was used for validation instead of Lexi-comp 2010.

Conclusions: In conclusion, the ADE and drug counts in Lexi-comp and SIDER data sets were highly correlated and the choice of validation set did not greatly affect the overall prediction performance of PPN. Our results also suggest that it is important to be aware of the differences that exist among ADE data sets, especially in modeling applications focused on specific drug and ADE categories.

Show MeSH

Related in: MedlinePlus

ROC curves of PPN model corresponding to the Lexi-comp 2010 and SIDER 2012 reference standards.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4150549&req=5

Figure 3: ROC curves of PPN model corresponding to the Lexi-comp 2010 and SIDER 2012 reference standards.

Mentions: We compared the predictive scores generated by the PPN model trained on data from Lexi-comp 2005 with the newly introduced drug-ADE associations in Lexi-comp 2010 and SIDER 2012 data sets. We found a very minor overall change in the Area Under the Receiver Operating Characteristic curve (AUROC) of the PPN model when SIDER 2012 data were used for validation instead of Lexi-comp 2010 data: 0.84 vs. 0.85, respectively (Figure 3). Stratification of AUROC by top-level MedDRA category – i.e. System Organ Class (SOC) –showed that there is no clear relationship between the number of SIDER-only associations in a category and the change in AUROC when SIDER is used for validation instead of Lexi-comp (Table 3). In fact, the relative change in AUROC was less than 5% for all but six SOCs which are rather general and not specifically related to a body organ or system: 1) Congenital, familial and genetic disorders (relative change 34%), 2) Surgical and medical procedures (relative change 20%), 3) General disorders (relative change 12%), 4) Injury, poisoning and procedural disorders (relative change 7%), 5) Investigations (relative change 7%), 6) Pregnancy, puerperium and perinatal conditions (relative change 5.1%).


Concordance and predictive value of two adverse drug event data sets.

Cami A, Reis BY - BMC Med Inform Decis Mak (2014)

ROC curves of PPN model corresponding to the Lexi-comp 2010 and SIDER 2012 reference standards.
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4150549&req=5

Figure 3: ROC curves of PPN model corresponding to the Lexi-comp 2010 and SIDER 2012 reference standards.
Mentions: We compared the predictive scores generated by the PPN model trained on data from Lexi-comp 2005 with the newly introduced drug-ADE associations in Lexi-comp 2010 and SIDER 2012 data sets. We found a very minor overall change in the Area Under the Receiver Operating Characteristic curve (AUROC) of the PPN model when SIDER 2012 data were used for validation instead of Lexi-comp 2010 data: 0.84 vs. 0.85, respectively (Figure 3). Stratification of AUROC by top-level MedDRA category – i.e. System Organ Class (SOC) –showed that there is no clear relationship between the number of SIDER-only associations in a category and the change in AUROC when SIDER is used for validation instead of Lexi-comp (Table 3). In fact, the relative change in AUROC was less than 5% for all but six SOCs which are rather general and not specifically related to a body organ or system: 1) Congenital, familial and genetic disorders (relative change 34%), 2) Surgical and medical procedures (relative change 20%), 3) General disorders (relative change 12%), 4) Injury, poisoning and procedural disorders (relative change 7%), 5) Investigations (relative change 7%), 6) Pregnancy, puerperium and perinatal conditions (relative change 5.1%).

Bottom Line: A minor difference of 1.1% was found in the AUROC of PPN when SIDER 2012 was used for validation instead of Lexi-comp 2010.In conclusion, the ADE and drug counts in Lexi-comp and SIDER data sets were highly correlated and the choice of validation set did not greatly affect the overall prediction performance of PPN.Our results also suggest that it is important to be aware of the differences that exist among ADE data sets, especially in modeling applications focused on specific drug and ADE categories.

View Article: PubMed Central - HTML - PubMed

Affiliation: Division of Emergency Medicine, Boston Children's Hospital, 1 Autumn Street, 5th Floor, Boston, MA 02215, USA. aurel.cami@childrens.harvard.edu.

ABSTRACT

Background: Accurate prediction of adverse drug events (ADEs) is an important means of controlling and reducing drug-related morbidity and mortality. Since no single "gold standard" ADE data set exists, a range of different drug safety data sets are currently used for developing ADE prediction models. There is a critical need to assess the degree of concordance between these various ADE data sets and to validate ADE prediction models against multiple reference standards.

Methods: We systematically evaluated the concordance of two widely used ADE data sets - Lexi-comp from 2010 and SIDER from 2012. The strength of the association between ADE (drug) counts in Lexi-comp and SIDER was assessed using Spearman rank correlation, while the differences between the two data sets were characterized in terms of drug categories, ADE categories and ADE frequencies. We also performed a comparative validation of the Predictive Pharmacosafety Networks (PPN) model using both ADE data sets. The predictive power of PPN using each of the two validation sets was assessed using the area under Receiver Operating Characteristic curve (AUROC).

Results: The correlations between the counts of ADEs and drugs in the two data sets were 0.84 (95% CI: 0.82-0.86) and 0.92 (95% CI: 0.91-0.93), respectively. Relative to an earlier snapshot of Lexi-comp from 2005, Lexi-comp 2010 and SIDER 2012 introduced a mean of 1,973 and 4,810 new drug-ADE associations per year, respectively. The difference between these two data sets was most pronounced for Nervous System and Anti-infective drugs, Gastrointestinal and Nervous System ADEs, and postmarketing ADEs. A minor difference of 1.1% was found in the AUROC of PPN when SIDER 2012 was used for validation instead of Lexi-comp 2010.

Conclusions: In conclusion, the ADE and drug counts in Lexi-comp and SIDER data sets were highly correlated and the choice of validation set did not greatly affect the overall prediction performance of PPN. Our results also suggest that it is important to be aware of the differences that exist among ADE data sets, especially in modeling applications focused on specific drug and ADE categories.

Show MeSH
Related in: MedlinePlus