Our randomization strategy will take place at the facility and district levels using a factorial design. First, the 112 districts in Uganda were analyzed for existing transparency and accountability initiatives. All districts without a major initiative underway were included in the study, which left 95 districts. This provided us with a sample of 1,417 HCII and HCIII facilities.
The process of creating, disseminating and studying the impact of performance indicator reports is detailed below.
a. Facilities submit a monthly outpatient report on paper called HMIS 105. This report is sent to a technical officer who enters the data into DHIS2.
b. We downloaded data for every Health Center Level II and III in 95 districts, at the 1,417 health facilities. From this data, we will generate performance indicator reports for each facility selected to participate. These reports will be based on four key indicators:
i. Number of 1st antenatal care visits, number of 4th antenatal care visits, number of deliveries and number of PCV3 vaccines administered.
c. The 95 districts were first divided into two randomly selected groups, 47 treatment districts and 48 control districts. Then the 1,417 facilities were divided into 708 treatment and 709 control facilities.
i. In Treatment districts, District Health Teams will receive summary reports by mTrac SMS messages monthly and in email form monthly as well. The report sent to the district will be supplemented with a ranking of facilities, highlighting top-performers. Note that all facilities in treatment districts are included in reports.
ii. In control districts, no report will be sent to the District Health Teams. Note that these control districts will contain facilities that have been sent facility indicator reports directly.
d. The facility-level treatment consists of facility-level indicator reports sent directly to directors or “in-charge” and data officers at selected facilities by SMS through the mTrac system.
i.The report consists of a mean score of each chosen indicator as well as a measure of the frequency of reporting. The period of the report will cover a three month span to account for expected month to month fluctuations in service provision and the failure to report in a given month. Note that the report will be distributed monthly but is composed of the previous three months of data reported. In addition, the report will contain a comparison to performance during the same three month period 12-months before. Each indicator will be compared as well as an aggregate measure of change from the previous year. Last, the change from last year will be used to rank facilities of the same type within a given district. The facility receiving the report will learn where they stand in the district in addition to receiving the name of the facility immediately above and below their position.
e.Note that treatment facilities in this arm will be contained in districts in which district level health officials receive information about facilities (treatment districts) and also in districts where level health officials do not receive information about facilities (control districts). Similarly, control facilities that themselves do not receive performance indicator reports will be contained within some districts that receive reports about that facility. Purely control facilities will be facilities that do not receive performance indicator reports and are located within control districts.
f. For our analysis, we plan to study additional outcomes not contained in the performance indicator reports such as reporting rates, number of outpatient attendees, malaria treatment rates and stock of tracer medications, all of which will also be sourced from the HMIS system.
i. To ensure reliability of the HMIS data, we will conduct facility level audits on a randomly selected subsample of facilities towards the end of the project. We estimate that 30% of facilities, roughly 100 facilities per treatment arm, will be audited. Our enumerators will ask the in-charge at the facility to view the facility’s records and record the data in addition to observing the condition of the facility at the time of the visit.