Limits...
Comparing a paper based monitoring and evaluation system to a mHealth system to support the national community health worker programme, South Africa: an evaluation.

Neupane S, Odendaal W, Friedman I, Jassat W, Schneider H, Doherty T - BMC Med Inform Decis Mak (2014)

Bottom Line: Clinical referrals, data accuracy and supervised visits were compared and analysed for the paper and phone systems.The most common error that occurred was summing the total number of visits and/or activities across the five household activity indicators.Few supervised home visits were recorded in either system and there was no evidence of the team leader following up on the automatic notifications received on their cell phones.

View Article: PubMed Central - HTML - PubMed

Affiliation: School of Public Health, University of the Western Cape, Cape Town, South Africa. sunisha.neupane@gmail.com.

ABSTRACT

Background: In an attempt to address a complex disease burden, including improving progress towards MDGs 4 and 5, South Africa recently introduced a re-engineered Primary Health Care (PHC) strategy, which has led to the development of a national community health worker (CHW) programme. The present study explored the development of a cell phone-based and paper-based monitoring and evaluation (M&E) system to support the work of the CHWs.

Methods: One sub-district in the North West province was identified for the evaluation. One outreach team comprising ten CHWs maintained both the paper forms and mHealth system to record household data on community-based services. A comparative analysis was done to calculate the correspondence between the paper and phone records. A focus group discussion was conducted with the CHWs. Clinical referrals, data accuracy and supervised visits were compared and analysed for the paper and phone systems.

Results: Compared to the mHealth system where data accuracy was assured, 40% of the CHWs showed a consistently high level (>90% correspondence) of data transfer accuracy on paper. Overall, there was an improvement over time, and by the fifth month, all CHWs achieved a correspondence of 90% or above between phone and paper data. The most common error that occurred was summing the total number of visits and/or activities across the five household activity indicators. Few supervised home visits were recorded in either system and there was no evidence of the team leader following up on the automatic notifications received on their cell phones.

Conclusions: The evaluation emphasizes the need for regular supervision for both systems and rigorous and ongoing assessments of data quality for the paper system. Formalization of a mHealth M&E system for PHC outreach teams delivering community based services could offer greater accuracy of M&E and enhance supervision systems for CHWs.

Show MeSH
Absolute discrepancies on paper forms with standard error of mean. Discrepancies (average of ten CHWs) between weekly and monthly paper forms on five types of visits- Pregnancy (P), Post natal (PN), Treatment Adherence (TA), Under 5 and Home based care (U5).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
getmorefigures.php?uid=PMC4150556&req=5

Figure 2: Absolute discrepancies on paper forms with standard error of mean. Discrepancies (average of ten CHWs) between weekly and monthly paper forms on five types of visits- Pregnancy (P), Post natal (PN), Treatment Adherence (TA), Under 5 and Home based care (U5).

Mentions: A discrepancy was defined as a difference between the accumulated weekly data versus the monthly data. A further analysis of data transfer accuracy (Figure 2) showed that it was in particular for the “under 5” and “treatment adherence support” indicators, that discrepancies between the weekly and monthly totals were observed. For example, CHW1 had 68 visits on her weekly forms for “treatment adherence support”, whereas she reported 46 on the monthly form. This shows an absolute discrepancy of 22. When the CHWs were asked about the discrepancies in the paper forms, they reported that it was due to the lack of understanding the indicator definitions: “We never got [proper] training for the paper system. We don’t know about the new paper forms. We were given the paper forms and its up to us to understand those with our knowledge. We are confused on household visits and other elements. When new tools are added, we never get additional training” (FGD November 2012).


Comparing a paper based monitoring and evaluation system to a mHealth system to support the national community health worker programme, South Africa: an evaluation.

Neupane S, Odendaal W, Friedman I, Jassat W, Schneider H, Doherty T - BMC Med Inform Decis Mak (2014)

Absolute discrepancies on paper forms with standard error of mean. Discrepancies (average of ten CHWs) between weekly and monthly paper forms on five types of visits- Pregnancy (P), Post natal (PN), Treatment Adherence (TA), Under 5 and Home based care (U5).
© Copyright Policy - open-access
Related In: Results  -  Collection

License 1 - License 2
Show All Figures
getmorefigures.php?uid=PMC4150556&req=5

Figure 2: Absolute discrepancies on paper forms with standard error of mean. Discrepancies (average of ten CHWs) between weekly and monthly paper forms on five types of visits- Pregnancy (P), Post natal (PN), Treatment Adherence (TA), Under 5 and Home based care (U5).
Mentions: A discrepancy was defined as a difference between the accumulated weekly data versus the monthly data. A further analysis of data transfer accuracy (Figure 2) showed that it was in particular for the “under 5” and “treatment adherence support” indicators, that discrepancies between the weekly and monthly totals were observed. For example, CHW1 had 68 visits on her weekly forms for “treatment adherence support”, whereas she reported 46 on the monthly form. This shows an absolute discrepancy of 22. When the CHWs were asked about the discrepancies in the paper forms, they reported that it was due to the lack of understanding the indicator definitions: “We never got [proper] training for the paper system. We don’t know about the new paper forms. We were given the paper forms and its up to us to understand those with our knowledge. We are confused on household visits and other elements. When new tools are added, we never get additional training” (FGD November 2012).

Bottom Line: Clinical referrals, data accuracy and supervised visits were compared and analysed for the paper and phone systems.The most common error that occurred was summing the total number of visits and/or activities across the five household activity indicators.Few supervised home visits were recorded in either system and there was no evidence of the team leader following up on the automatic notifications received on their cell phones.

View Article: PubMed Central - HTML - PubMed

Affiliation: School of Public Health, University of the Western Cape, Cape Town, South Africa. sunisha.neupane@gmail.com.

ABSTRACT

Background: In an attempt to address a complex disease burden, including improving progress towards MDGs 4 and 5, South Africa recently introduced a re-engineered Primary Health Care (PHC) strategy, which has led to the development of a national community health worker (CHW) programme. The present study explored the development of a cell phone-based and paper-based monitoring and evaluation (M&E) system to support the work of the CHWs.

Methods: One sub-district in the North West province was identified for the evaluation. One outreach team comprising ten CHWs maintained both the paper forms and mHealth system to record household data on community-based services. A comparative analysis was done to calculate the correspondence between the paper and phone records. A focus group discussion was conducted with the CHWs. Clinical referrals, data accuracy and supervised visits were compared and analysed for the paper and phone systems.

Results: Compared to the mHealth system where data accuracy was assured, 40% of the CHWs showed a consistently high level (>90% correspondence) of data transfer accuracy on paper. Overall, there was an improvement over time, and by the fifth month, all CHWs achieved a correspondence of 90% or above between phone and paper data. The most common error that occurred was summing the total number of visits and/or activities across the five household activity indicators. Few supervised home visits were recorded in either system and there was no evidence of the team leader following up on the automatic notifications received on their cell phones.

Conclusions: The evaluation emphasizes the need for regular supervision for both systems and rigorous and ongoing assessments of data quality for the paper system. Formalization of a mHealth M&E system for PHC outreach teams delivering community based services could offer greater accuracy of M&E and enhance supervision systems for CHWs.

Show MeSH