Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda

The project consists of two interventions implemented simultaneously. Using a factorial design, each intervention will be measured independently in addition to the interaction effect between interventions.

The first intervention are facility level PIRs sent to health facilities monthly via mobile phone. The reports use Ministry of Health (MoH) Health Management Information System (HMIS) data and compare performance over the past 3 months to performance at the same time 12 months before. A PIR comprises 3 SMS reports. The three different SMS reports sent each month are described below:

1. AVERAGE THREE MONTH REPORT: The average of each indicator for the previous three months. The indicators are first antenatal care visit, fourth antenatal care visit, deliveries and third doses of PCV vaccine administered. In addition, the reporting rate for the past three months will be revealed. If a facility fails to report an indicator in a given month, the score for that indicator in that month is zero.

3. CHANGE FROM LAST YEAR: The average three month report from this year compared to the same period last year. For example, the August report from this year is composed of an average of May, June and July 2015, compared to May, June and July 2014. The total score represents overall change from last year and is an average of the individual changes observed among the four indicators.

3. RANK: A facility’s improvement compared to other facilities in their district. The biggest improvements from the previous year rank highest. HCIIs and HCIIIs are ranked separately, so facilities are told their rank out of the number of HCII or HCIII facilities in their district. The report also names the facility performing just above them and the facility performing just below.

The second intervention is district level PIRs sent to the District Health Office every month. One report is sent by SMS, and a second report by email. The components of these message are as follows:

1. EMAIL- The district health office receives the same monthly reports as the facility. The three reports are aggregated into a single line on a spreadsheet into an email that will be sent from the MoH. The email will contain the same data as the three reports shown above so that the District Health Office (DHO) can follow-up with the facilities that are under-performing and congratulate the leaders.

2. SMS - At the same time, an SMS will be sent to the DHO identifying the top-3 and bottom-3 performing facilities in a district as well as a reminder to the DHO to look at the full facility level reports, which have been sent by email.

2015-10-21

2016-10-31

The key outcome variables of interest are increases in the utilization of reproductive, maternal, neonatal, and child (RMNCH) health services (RMNCH) as measured by data in the HMIS system. Specifically, we will look for changes in the indicators included in the reports sent to facilities and districts; number of first antenatal care visits, the number of fourth antenatal care visits, number of deliveries performed at health facilities and the number of pneumococcal conjugate vaccine third doses delivered. In addition, we will also use actual reporting rates as an outcome variable.

In addition to the outcomes in the HMIS system, we will conduct an endline facility survey to get a more comprehensive measure of overall health facility quality in a random sample of 30% of the health facilities in 50% of the districts. The endline survey will capture many dimensions of facility quality: facility operating hours, availability of key equipment or facilities, health financing, training, availability and stock out of commodities and medicines, data that can be corroborated in HMIS reports, etc. The facility visits will be unannounced.

The primary outcomes for this study will be the main indicators outlined above which will be extracted directly from the HMIS system. In addition, we plan to compare these data with data collected at the health facility level in our health facility audits, and we also plan to construct a measure of health facility quality, which will take into consideration numerous dimensions of heath facility quality, including the appearance and cleanliness of health facilities, availability of equipment and supplies, health financing, and hours of operation.

Our randomization strategy will take place at the facility and district levels using a factorial design. First, the 112 districts in Uganda were analyzed for existing transparency and accountability initiatives. All districts without a major initiative underway were included in the study, which left 95 districts. This provides us with a sample of 1417 HCII and HCIII facilities.

The process of creating, disseminating and studying the impact of performance indicator reports is detailed below.

a. Facilities submit a monthly outpatient report on paper called HMIS 105. This report is sent to a technical officer who enters the data into DHIS2.
b. We downloaded data for every Health Center Level II and III in 95 districts, at the 1,417 health facilities. From this data, we will generate performance indicator reports for each facility selected to participate. These reports will be based on four key indicators:
i. Number of 1st antenatal care visits, number of 4th antenatal care visits, number of deliveries and number of PCV3 vaccines administered.
c. The 1,417 facilities will first be divided into two randomly selected groups, 708 treatment and 709 control facilities. Then the 95 districts will be divided into treatment and control districts:
i. In Treatment districts, District Health Teams will receive summary reports by mTrac SMS messages monthly and in email form monthly as well. The report sent to the district will be supplemented with a ranking of facilities, highlighting top-performers. Note that all facilities in treatment districts are included in reports.
ii. In control districts, no report will be sent to the District Health Teams. Note that these control districts will contain facilities that have been sent facility indicator reports directly.

d. The facility level treatment consists of facility level indicator reports sent directly to directors or “in-charge” and data officers at selected facilities by SMS through the mTrac system.
i.The report consists of a mean score of each chosen indicator as well as a measure of the frequency of reporting. The period of the report will cover a three month span to account for expected month to month fluctuations in service provision and the failure to report in a given month. Note that the report will be distributed monthly but is composed of the previous three months of data reported. In addition, the report will contain a comparison to performance during the same three month period 12-months before. Each indicator will be compared as well as an aggregate measure of change from the previous year. Last the change from last year will be used to rank facilities of the same type within a given district. The facility receiving the report will learn where they stand in the district in addition to receiving the name of the facility immediately above and below their position.
e.Note that treatment facilities in this arm will be contained in districts in which district level health officials receive information about facilities (treatment districts) and also in districts where level health officials do not receive information about facilities (control districts). Similarly, control facilities that themselves do not receive performance indicator reports will be contained within some districts that receive reports about that facility. Purely control facilities will be facilities that do not receive performance indicator reports and are located within control districts.
f. For our analysis, we plan to study additional outcomes not contained in the performance indicator reports such as reporting rates, number of outpatient attendees, malaria treatment rates and stock of tracer medications, all of which will also be sourced from the HMIS system.
i. To ensure reliability of the HMIS data, we will conduct facility level audits on a randomly selected subsample of facilities towards the end of the project. We estimate that 30% of facilities, roughly 100 facilities per treatment arm, will be audited. Our enumerators will ask the in-charge at the facility to view the facility’s records and record the data in addition to observing the condition of the facility at the time of the visit.

Facilities and districts were randomized using the "randomize" command in Stata 13.0. Districts were balanced on their population and proportion of the population defined as "rural" by the 2014 Uganda Census. Districts were then blocked by region. Facilities were balanced on the average number of reported antenatal care first and fourth visits as well as number of deliveries. Historical data for the entire 12 month period used for the randomization did not exist for pneumococcal conjugate vaccine and this variable was omitted for the purpose of balance. Facilities were then blocked by their type as classified by the MoH, II or III.

The randomization unit is the health facility. The study is randomized at two levels. First, the 95 eligible districts in Uganda are randomly divided into treatment and control. Then the facilities within the districts are randomly divided into two groups.

No

Not clustered

1417 health facilities.

Treatment Facilities in Control Districts: 374
Treatment Facilities in Treatment Districts: 334
Control Facilites in Control Districts: 374
Control Facilites in Treatment Districts: 335