x

We are happy to announce that all trial registrations will now be issued DOIs (digital object identifiers). For more information, see here.
Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda
Last registered on October 21, 2015

Pre-Trial

Trial Information
General Information
Title
Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda
RCT ID
AEARCTR-0000798
Initial registration date
October 21, 2015
Last updated
October 21, 2015 3:10 AM EDT
Location(s)
Region
Primary Investigator
Affiliation
New York University
Other Primary Investigator(s)
PI Affiliation
Georgetown University
PI Affiliation
Georgetown University
PI Affiliation
Makerere University School of Public Health
Additional Trial Information
Status
In development
Start date
2015-10-21
End date
2016-12-31
Secondary IDs
Abstract
Increasing transparency and accountability in the heath sector is thought to improve health service delivery outcomes; however, there is little empirical evidence on the ways in which this might occur in a developing country setting. In Uganda, UNICEF in partnership with the Ministry of Health of Uganda has developed facility-level performance indicator reports (PIRs) that are based on administrative data and can be distributed to health facilities via mTrac - an innovative mobile phone platform that is operating nationally at scale. The PIRs provide health facilities with detailed information on their performance on four key reproductive, maternal, newborn and child health indicators relative to other facilities and relative to their past performance. Using a randomized control experimental design and targeting approximately 1,400 facilities, this project aims to test the impact of increased transparency through the sharing of performance indicator reports with front line health workers on facility-level performance after 12 months. Simultaneously, we plan to evaluate the impact of also sharing similar performance reports with district health officials in a random sample of about half of 95 districts in the country. The goal is to shed light on the ways in which strengthening accountability chains in Uganda might help improve health system performance.
External Link(s)
Registration Citation
Citation
Atuyambe, Lynn et al. 2015. "Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda." AEA RCT Registry. October 21. https://doi.org/10.1257/rct.798-1.0.
Former Citation
Atuyambe, Lynn et al. 2015. "Strengthening Accountability Chains: Measuring Impact in Health Service Delivery in Uganda." AEA RCT Registry. October 21. https://www.socialscienceregistry.org/trials/798/history/5683.
Sponsors & Partners

There are documents in this trial unavailable to the public. Use the button below to request access to this information.

Request Information
Experimental Details
Interventions
Intervention(s)
The project consists of two interventions implemented simultaneously. Using a factorial design, each intervention will be measured independently in addition to the interaction effect between interventions.

The first intervention consists of facility-level performance indicator reports (PIRs) sent to health facilities monthly via mobile phone. The reports use Ministry of Health (MoH) Health Management Information System (HMIS) data and compare performance over the past 3 months to performance at the same time 12 months before. A PIR comprises 3 SMS reports. The three different SMS reports sent each month are described below:

1. AVERAGE THREE MONTH REPORT: The average of each indicator for the previous three months. The indicators are first antenatal care visit, fourth antenatal care visit, deliveries and third doses of pneumococcal conjugate vaccine administered. In addition, the reporting rate for the past three months will be revealed. If a facility fails to report an indicator in a given month, the score for that indicator in that month is zero.
2. CHANGE FROM LAST YEAR: The average three month report from this year compared to the same period last year. For example, the August report from this year is composed of an average of May, June and July 2015, compared to May, June and July 2014. The total score represents overall change from last year and is an average of the individual changes observed among the four indicators.
3. RANK: A facility’s improvement compared to other facilities in their district. The biggest improvements from the previous year rank highest. HCIIs and HCIIIs are ranked separately, so facilities are told their rank out of the number of HCII or HCIII facilities in their district. The report also names the facility performing just above them and the facility performing just below.

The second intervention is district-level PIRs sent to the District Health Office every month. One report is sent by SMS, and a second report by email. The components of these message are as follows:

1. EMAIL: The district health office receives the same monthly reports as the facility. The three reports are aggregated into a single line per facility on a spreadsheet in an email that will be sent from the MoH. The email will contain the same data as the three reports shown above so that the District Health Office (DHO) can follow-up with the facilities that are under-performing and congratulate the leaders.
2. SMS: At the same time, an SMS will be sent to DHO officials identifying the top-3 and bottom-3 performing HCII and HCIII facilities in a district as well as a reminder to the DHO to look at the full facility level reports, which have been sent by email.
Intervention Start Date
2015-10-21
Intervention End Date
2016-10-31
Primary Outcomes
Primary Outcomes (end points)
The key outcome variables of interest are increases in the utilization of reproductive, maternal, neonatal, and child health services (RMNCH) as measured by data in the HMIS system. Specifically, we will look for changes in the indicators included in the reports sent to facilities and districts: number of first antenatal care visits, number of fourth antenatal care visits, number of deliveries performed at health facilities and the number of pneumococcal conjugate vaccine third doses delivered. In addition, we will also use actual reporting rates as an outcome variable.

In addition to the outcomes in the HMIS system, we will conduct an endline facility survey to get a more comprehensive measure of overall health facility quality in a random sample of 30% of the health facilities in 50% of the districts. The endline survey will capture many dimensions of facility quality: facility operating hours, availability of key equipment or facilities, health financing, training, availability and stock out of commodities and medicines, data that can be corroborated in HMIS reports, etc. The facility visits will be unannounced.
Primary Outcomes (explanation)
The primary outcomes for this study will be the main indicators outlined above which will be extracted directly from the HMIS system. In addition, we plan to compare these data with data collected at the health facility level in our health facility audits, and we also plan to construct a measure of health facility quality, which will take into consideration numerous dimensions of heath facility quality, including the appearance and cleanliness of health facilities, availability of equipment and supplies, health financing, and hours of operation.
Secondary Outcomes
Secondary Outcomes (end points)
Secondary Outcomes (explanation)
Experimental Design
Experimental Design
Our randomization strategy will take place at the facility and district levels using a factorial design. First, the 112 districts in Uganda were analyzed for existing transparency and accountability initiatives. All districts without a major initiative underway were included in the study, which left 95 districts. This provided us with a sample of 1,417 HCII and HCIII facilities.

The process of creating, disseminating and studying the impact of performance indicator reports is detailed below.

a. Facilities submit a monthly outpatient report on paper called HMIS 105. This report is sent to a technical officer who enters the data into DHIS2.
b. We downloaded data for every Health Center Level II and III in 95 districts, at the 1,417 health facilities. From this data, we will generate performance indicator reports for each facility selected to participate. These reports will be based on four key indicators:
i. Number of 1st antenatal care visits, number of 4th antenatal care visits, number of deliveries and number of PCV3 vaccines administered.
c. The 95 districts were first divided into two randomly selected groups, 47 treatment districts and 48 control districts. Then the 1,417 facilities were divided into 708 treatment and 709 control facilities.
i. In Treatment districts, District Health Teams will receive summary reports by mTrac SMS messages monthly and in email form monthly as well. The report sent to the district will be supplemented with a ranking of facilities, highlighting top-performers. Note that all facilities in treatment districts are included in reports.
ii. In control districts, no report will be sent to the District Health Teams. Note that these control districts will contain facilities that have been sent facility indicator reports directly.
d. The facility-level treatment consists of facility-level indicator reports sent directly to directors or “in-charge” and data officers at selected facilities by SMS through the mTrac system.
i.The report consists of a mean score of each chosen indicator as well as a measure of the frequency of reporting. The period of the report will cover a three month span to account for expected month to month fluctuations in service provision and the failure to report in a given month. Note that the report will be distributed monthly but is composed of the previous three months of data reported. In addition, the report will contain a comparison to performance during the same three month period 12-months before. Each indicator will be compared as well as an aggregate measure of change from the previous year. Last, the change from last year will be used to rank facilities of the same type within a given district. The facility receiving the report will learn where they stand in the district in addition to receiving the name of the facility immediately above and below their position.
e.Note that treatment facilities in this arm will be contained in districts in which district level health officials receive information about facilities (treatment districts) and also in districts where level health officials do not receive information about facilities (control districts). Similarly, control facilities that themselves do not receive performance indicator reports will be contained within some districts that receive reports about that facility. Purely control facilities will be facilities that do not receive performance indicator reports and are located within control districts.
f. For our analysis, we plan to study additional outcomes not contained in the performance indicator reports such as reporting rates, number of outpatient attendees, malaria treatment rates and stock of tracer medications, all of which will also be sourced from the HMIS system.
i. To ensure reliability of the HMIS data, we will conduct facility level audits on a randomly selected subsample of facilities towards the end of the project. We estimate that 30% of facilities, roughly 100 facilities per treatment arm, will be audited. Our enumerators will ask the in-charge at the facility to view the facility’s records and record the data in addition to observing the condition of the facility at the time of the visit.
Experimental Design Details
Randomization Method
Facilities and districts were randomized using the "randomize" command in Stata 13.0, with 500 minimum randomizations run.

Districts were balanced on their population and proportion of the population defined as "rural" by the 2014 Uganda Census. Districts were then blocked by region.

Facilities were balanced on the average number of reported antenatal care first and fourth visits as well as number of deliveries. Historical data for the entire 12 month period used for the randomization did not exist for pneumococcal conjugate vaccine and this variable was omitted for the purpose of balance. Facilities were then blocked by their type as classified by the MoH, II or III.
Randomization Unit
The study is randomized at two levels with the unit of randomization as the health facility. The selection process and criteria for each level are explained below.

First, districts containing the ACT Health project, administered by GOAL, were dropped from the sample due to close overlap between interventions. The remaining 95 eligible districts were randomly divided into treatment and control.
a. Districts were balanced on their total population and proportion of the population defined as "rural" by the 2014 Uganda Census.
b. Districts were then blocked by region.

The facilities within the districts were randomly divided into treatment and control groups. Data to conduct the randomization was selected from the Uganda DHIS2 database using the following method:
a. Monthly submitted data for all government owned, functional health facilities level II and III was downloaded and analyzed.
b. The period of analysis extended from May 2014 to April 2015 with one observation per month per facility on each indicator.
c. Four variables (ANC1, ANC4, deliveries and PCV3) were found to meet the selection criteria:
i. In line with MoH priorities for improvement
ii. Exists on current and future versions of HMIS reports
iii. Show low variance, as measured by a standard deviation close to the mean

This process provided 1,417 eligible facilities within the 95 sample districts. Facilities were balanced on the average number of reported antenatal care first and fourth visits as well as number of deliveries. Historical data for the entire 12 month period used for the randomization did not exist for pneumococcal conjugate vaccine and this variable was omitted for the purpose of balance. The facilities were then blocked by district assignment and health facility type, i.e. the treatment and control groups will contain approximately the same number of HCIIs and HCIIIs respectively.
Was the treatment clustered?
No
Experiment Characteristics
Sample size: planned number of clusters
Not clustered
Sample size: planned number of observations
1417 health facilities.
Sample size (or number of clusters) by treatment arms
Treatment Facilities in Control Districts: 374
Treatment Facilities in Treatment Districts: 334
Control Facilites in Control Districts: 374
Control Facilites in Treatment Districts: 335

Treatment districts: 47
Control districts: 48
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The study has been powered (80%) to detect an effect size of 0.2 standard deviations (σ) from the mean or larger. The power calculations were conducted with a sample of DHIS2 data and used the mean values and standard deviations of the primary outcome variables. An effect identified by the standard regression above that is larger than 0.2 σ from the mean will be reliably detected. To power the study we conducted a simulation in Stata 13 using the command “factorial sim”. This allowed us to estimate the effect of the facility and district level interventions as well as the interaction effect between them. The study is sufficiently powered to detect an effect at the facility and district levels, however only a very small interaction effect can be detected. Below are the probabilities that an effect of a given size will be detected. The effect sizes shown are our estimate of the potential effect of this intervention with the following key variables. ANC1: 0.999 at the facility level (0.1 sd), 0.887 at the district level (0.2 sd), and 0.341 at the interaction level (0.5 sd) ANC4: 0.997 at the facility level (0.1 sd), 0.878 at the district level (0.2 sd), and 0.358 at the interaction level (0.5 sd) Deliveries: 0.992 at the facility level (0.1 sd), 0.846 at the district level (0.2 sd), and 0.319 at the interaction level (0.5 sd) Note that historical data for the entire 12 month period used for the randomization did not exist for pneumococcal conjugate vaccine so this variable was omitted.
IRB
INSTITUTIONAL REVIEW BOARDS (IRBs)
IRB Name
Makerere University School of Public Health: Higher Degrees, Research and Ethics Committee
IRB Approval Date
2015-05-07
IRB Approval Number
293
IRB Name
New York University - University Committee on Activities Involving Human Subjects
IRB Approval Date
2015-01-06
IRB Approval Number
15-10501
IRB Name
Uganda National Council for Science and Technology
IRB Approval Date
2015-08-19
IRB Approval Number
HS 1875
Post-Trial
Post Trial Information
Study Withdrawal
Intervention
Is the intervention completed?
No
Is data collection complete?
Data Publication
Data Publication
Is public data available?
No
Program Files
Program Files
Reports and Papers
Preliminary Reports
Relevant Papers