Limits...
Accuracy of models for the 2001 foot-and-mouth epidemic.

Tildesley MJ, Deardon R, Savill NJ, Bessell PR, Brooks SP, Woolhouse ME, Grenfell BT, Keeling MJ - Proc. Biol. Sci. (2008)

Bottom Line: These claims are generally based on a comparison between model results and epidemic data at fairly coarse spatio-temporal resolution.By contrast, while the accuracy of predicting culls is higher (20-30%), this is lower than expected from the comparison between model epidemics.These results generally support the contention that the type of the model used in 2001 was a reliable representation of the epidemic process, but highlight the difficulties of predicting the complex human response, in terms of control strategies to the perceived epidemic risk.

View Article: PubMed Central - PubMed

Affiliation: Department of Biological Sciences and Mathematics Institute, University of Warwick, Gibbet Hill Road, Coventry CV4 7AL, UK. m.j.tildesley@warwick.ac.uk

ABSTRACT
Since 2001 models of the spread of foot-and-mouth disease, supported by the data from the UK epidemic, have been expounded as some of the best examples of problem-driven epidemic models. These claims are generally based on a comparison between model results and epidemic data at fairly coarse spatio-temporal resolution. Here, we focus on a comparison between model and data at the individual farm level, assessing the potential of the model to predict the infectious status of farms in both the short and long terms. Although the accuracy with which the model predicts farms reporting infection is between 5 and 15%, these low levels are attributable to the expected level of variation between epidemics, and are comparable to the agreement between two independent model simulations. By contrast, while the accuracy of predicting culls is higher (20-30%), this is lower than expected from the comparison between model epidemics. These results generally support the contention that the type of the model used in 2001 was a reliable representation of the epidemic process, but highlight the difficulties of predicting the complex human response, in terms of control strategies to the perceived epidemic risk.

Show MeSH

Related in: MedlinePlus

Model and data comparison for the entire country. (a) The daily number of farms that report infection (black) and farms that were culled (grey), together with the timings of national control measures for the 2001 epidemic. (b) Similar results from a single replicate model simulation, starting with the conditions on 23 February 2001. (c–h) Accuracy (solid lines) and associated repeatability (dashed lines) results (together with 95% CIs) for various time intervals and various farm types. If t is the time on the x-axis, the accuracy results are (c) accuracyall(23 February, t), (d) accuracyall(t, t+14), (e) accuracyreported(23 February, t), (f) accuracyreported(t, t+14), (g) accuracyculls(23 February, t), (h) accuracyculls(t, t+14). At least 2500 simulations were used to determine each data point. Regional plots, for Cumbria, Devon, the rest of England, Wales and Scotland, are shown in the electronic supplementary material.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
getmorefigures.php?uid=PMC2376304&req=5

fig2: Model and data comparison for the entire country. (a) The daily number of farms that report infection (black) and farms that were culled (grey), together with the timings of national control measures for the 2001 epidemic. (b) Similar results from a single replicate model simulation, starting with the conditions on 23 February 2001. (c–h) Accuracy (solid lines) and associated repeatability (dashed lines) results (together with 95% CIs) for various time intervals and various farm types. If t is the time on the x-axis, the accuracy results are (c) accuracyall(23 February, t), (d) accuracyall(t, t+14), (e) accuracyreported(23 February, t), (f) accuracyreported(t, t+14), (g) accuracyculls(23 February, t), (h) accuracyculls(t, t+14). At least 2500 simulations were used to determine each data point. Regional plots, for Cumbria, Devon, the rest of England, Wales and Scotland, are shown in the electronic supplementary material.

Mentions: Here, it is worth stressing the three important points about these measures of model fit. The first is that accuracy measures that focus solely on reported cases or culls (e.g. accuracyreported) are only informative if the number of cases and culls in the model closely matches the data. For example, very high levels of reported case accuracy could be obtained if the model simply overestimated the number of cases. We note however that our model closely matches the temporal pattern of observed cases and culls (figure 2b). Second, comparable levels of accuracy and repeatability can be achieved when the model captures little of the observed spatial structure—when the model matches the temporal dynamics but not the spatial. However, we again note that our model has been shown to be a good match for the general spatial pattern of cases (Keeling et al. 2001). Finally, accuracyreported can be thought of as the sensitivity of the epidemiological prediction; however, as this quantity varies both spatially and temporally and depends on the prediction of culls, we retain the term accuracy. Similarly, we can calculate the specificity of the epidemiological prediction (assuming the model on average predicts the observed number of reported cases)specificityreported=numberof farms−numberof reported×accuracyreportednumberof farms−number ofreported.Hence, our measure of accuracy naturally encompasses some of the standard measures of agreement between models and data.


Accuracy of models for the 2001 foot-and-mouth epidemic.

Tildesley MJ, Deardon R, Savill NJ, Bessell PR, Brooks SP, Woolhouse ME, Grenfell BT, Keeling MJ - Proc. Biol. Sci. (2008)

Model and data comparison for the entire country. (a) The daily number of farms that report infection (black) and farms that were culled (grey), together with the timings of national control measures for the 2001 epidemic. (b) Similar results from a single replicate model simulation, starting with the conditions on 23 February 2001. (c–h) Accuracy (solid lines) and associated repeatability (dashed lines) results (together with 95% CIs) for various time intervals and various farm types. If t is the time on the x-axis, the accuracy results are (c) accuracyall(23 February, t), (d) accuracyall(t, t+14), (e) accuracyreported(23 February, t), (f) accuracyreported(t, t+14), (g) accuracyculls(23 February, t), (h) accuracyculls(t, t+14). At least 2500 simulations were used to determine each data point. Regional plots, for Cumbria, Devon, the rest of England, Wales and Scotland, are shown in the electronic supplementary material.
© Copyright Policy - open-access
Related In: Results  -  Collection

License
Show All Figures
getmorefigures.php?uid=PMC2376304&req=5

fig2: Model and data comparison for the entire country. (a) The daily number of farms that report infection (black) and farms that were culled (grey), together with the timings of national control measures for the 2001 epidemic. (b) Similar results from a single replicate model simulation, starting with the conditions on 23 February 2001. (c–h) Accuracy (solid lines) and associated repeatability (dashed lines) results (together with 95% CIs) for various time intervals and various farm types. If t is the time on the x-axis, the accuracy results are (c) accuracyall(23 February, t), (d) accuracyall(t, t+14), (e) accuracyreported(23 February, t), (f) accuracyreported(t, t+14), (g) accuracyculls(23 February, t), (h) accuracyculls(t, t+14). At least 2500 simulations were used to determine each data point. Regional plots, for Cumbria, Devon, the rest of England, Wales and Scotland, are shown in the electronic supplementary material.
Mentions: Here, it is worth stressing the three important points about these measures of model fit. The first is that accuracy measures that focus solely on reported cases or culls (e.g. accuracyreported) are only informative if the number of cases and culls in the model closely matches the data. For example, very high levels of reported case accuracy could be obtained if the model simply overestimated the number of cases. We note however that our model closely matches the temporal pattern of observed cases and culls (figure 2b). Second, comparable levels of accuracy and repeatability can be achieved when the model captures little of the observed spatial structure—when the model matches the temporal dynamics but not the spatial. However, we again note that our model has been shown to be a good match for the general spatial pattern of cases (Keeling et al. 2001). Finally, accuracyreported can be thought of as the sensitivity of the epidemiological prediction; however, as this quantity varies both spatially and temporally and depends on the prediction of culls, we retain the term accuracy. Similarly, we can calculate the specificity of the epidemiological prediction (assuming the model on average predicts the observed number of reported cases)specificityreported=numberof farms−numberof reported×accuracyreportednumberof farms−number ofreported.Hence, our measure of accuracy naturally encompasses some of the standard measures of agreement between models and data.

Bottom Line: These claims are generally based on a comparison between model results and epidemic data at fairly coarse spatio-temporal resolution.By contrast, while the accuracy of predicting culls is higher (20-30%), this is lower than expected from the comparison between model epidemics.These results generally support the contention that the type of the model used in 2001 was a reliable representation of the epidemic process, but highlight the difficulties of predicting the complex human response, in terms of control strategies to the perceived epidemic risk.

View Article: PubMed Central - PubMed

Affiliation: Department of Biological Sciences and Mathematics Institute, University of Warwick, Gibbet Hill Road, Coventry CV4 7AL, UK. m.j.tildesley@warwick.ac.uk

ABSTRACT
Since 2001 models of the spread of foot-and-mouth disease, supported by the data from the UK epidemic, have been expounded as some of the best examples of problem-driven epidemic models. These claims are generally based on a comparison between model results and epidemic data at fairly coarse spatio-temporal resolution. Here, we focus on a comparison between model and data at the individual farm level, assessing the potential of the model to predict the infectious status of farms in both the short and long terms. Although the accuracy with which the model predicts farms reporting infection is between 5 and 15%, these low levels are attributable to the expected level of variation between epidemics, and are comparable to the agreement between two independent model simulations. By contrast, while the accuracy of predicting culls is higher (20-30%), this is lower than expected from the comparison between model epidemics. These results generally support the contention that the type of the model used in 2001 was a reliable representation of the epidemic process, but highlight the difficulties of predicting the complex human response, in terms of control strategies to the perceived epidemic risk.

Show MeSH
Related in: MedlinePlus