Impact evaluation of a performance-based financing project in the health sector in the Democratic Republic of Congo

Performance payments will be made to health service providers and health administration units contracted in the target areas. Public, quasi-public and private health facilities, including health centers and first level referral hospitals, will be targeted in rural and urban areas. The purchasing, verification, community mobilization and coaching will be organized through provincial purchasing agencies (EUPs: Etablissement d’utilité publique). PBF payments will be paid to these health facilities in proportion to, and in payment for, achieved quantity and quality results. Facility payments will be based on (i) the quantity of services defined through a package of basic and complementary activities delivered to the targeted population, and (ii) the technical quality of these services as measured through comprehensive quantified quality checklists. Facility payments will be made quarterly after service volumes have been verified and quality of technical support and care has been assessed and certified by the EUPs, and validated through special governing boards at the provincial level. After the quantity and quality of services provided are certified, payment will be released to contracted health facilities via fundholding arrangements.

Modified quality checklists (quality incentives): An innovative aspect of the PBF approach in DRC will involve the content of the quality checklist that will be used for assessing the quality of care in health centers and hospitals. The amount of money paid for the quantity of services can be increased by up to 25% according to the quarterly quality score, if the facility exceeds the threshold score of 50%. In addition to determining the payment amount to facilities, the quality checklist also provides guidelines for good practice and structures the supervision of the facilities by the staff of the health zones. The study will evaluate the use of different types of quality checklists at different sets of health facilities. All checklists will include measure of structural quality, process and content of care. Examples of measures of structural quality are: proper storage of drugs, cleanliness of the facility and existence of required registries and protocols. Measures of process and content of care include indicators such as correct entries into registries and medical records and checking these records for proper application of the protocols. In selected health facilities, ‘Vignettes Quality Checklist’ will also include healthcare providers’ responses to clinical vignettes as indicators of quality of care. The vignettes will cover cases that represent the most common causes of morbidity and mortality and will be repeated each quarter. It is theorized that the quarterly repetition of these cases will reinforce the protocol related to these common cases. However, these measures are also a limited in representing quality of care as there might be a gap between what providers know and do when they are actually treating the patients.

2016-09-01

2022-05-15

1. Utilization of incentivized health services measured through household surveys
2. Measures of quality of care: facility infrastructure and availability of supplies and equipment; providers' adherence to clinical protocols; patients' satisfaction

Data on service utilization, health behaviors and health outcomes will be collected through household surveys conducted in the catchment areas served by the health facilities. Quality of health services will be collected through health facility assessment as well as reports by patients on the care they received. Both household and facility surveys will be conducted by a survey firm that is not involved in any aspect of the implementation of the project.

1. Utilization of non-incentivized services
2. Prices paid for services
3. Staff motivation and satisfaction
4. Providers' clinical knowledge
5. Health outcomes (anthropometric measures)
6. Household health behaviors

Although the ultimate goal of the project is to improve maternal and child health outcomes, it is important to keep in mind that detecting such outcomes might require a long duration as well as a very large sample. The study is not powered to detect statistically significant changes in health outcomes over the proposed time frame. Therefore, as reflected in the research questions above, the impact evaluation will focus also on the intermediate outputs of the project: utilization and quality.

The research questions will be answered employing randomized assignment of the different project components. The PBF intervention will be randomly assigned at the health zone level while the quality checklist intervention will be assigned at the catchment area level (Health zones have an average of about 10 catchment area with at least one health center in each catchment area).

The randomization of the health zones into the PBF intervention has been conducted through public randomization ceremonies in the different districts. Representatives of the different health zones picked folded pieces of paper from a transparent bucket.
The randomization of the vignette quality checklist will be done in office by a computer.

The PBF intervention was randomized at the health zone level.
Variations in the PBF approach (pay-for-vignettes and community engagement approaches) will be randomized at the health center level.

Yes

There are 100 health zones and 500 catchment areas (5 in each health zone)
The baseline and endline surveys will cover the full sample. A midline survey will be conducted in a subset of 42 health zones (and 210 health centers) in order to measure short-term changes in quality of care.

500 health centers 5000 women with recent pregnancies 2000 health providers

Half of the sample (50 health zones and 250 catchment areas) is in either the PBF or control treatment arm
For the midline assessment, 21 health zones in ech of treatment and control arms will be surveyed.

Background
Health systems’ weakness remains one of the primary obstacles towards achieving universal access to quality healthcare in low-income settings. Performance-based financing (PBF) programs have been increasingly used to increase access to quality care in LMICs. However, evidence on the impacts of these programs remains fragmented and inconclusive. We analyze the health system impacts of the PBF program in the Democratic Republic of the Congo (DRC), one of the largest such programs introduced in LMICs to date.

Methods
We used a health systems perspective to analyze the benefits of PBF relative to unconditional financing of health facilities. Fifty-eight health zones in six provinces were randomly assigned to either a control group (28 zones) in which facilities received unconditional transfers or to a PBF program (30 zones) that started at the end of 2016. Follow-up data collection took place in 2021–2022 and included health facility assessments, health worker interviews, direct observations of consultations and deliveries, patient exit interviews, and household surveys. Using multivariate regression models, we estimated the impact of the program on 55 outcomes in seven health system domains: structural quality, technical process quality, non-technical process quality, service fees, facility management, providers’ satisfaction, and service coverage. We used random-effects meta-analysis to generate pooled average estimates within each domain.

Results
The PBF program improved the structural quality of health facilities by 4 percentage points (ppts) (95% CI 0.01–0.08), technical process quality by 5 ppts (0.03–0.07), and non-technical process by 2 ppts (0–0.04). PBF also increased coverage of priority health services by 3 ppts (0.02–0.04). Improvements were also observed for facility management (9 ppts, 0.04–0.15), service fee policies, and users’ satisfaction with service affordability (14 ppts, 0.07–0.20). Service fees and health workers’ satisfaction were not affected by the program.

Conclusions
The results suggest that well-designed PBF programs can lead to improvements in most health systems domains relative to comparable unconditional financing. However, the large persisting gaps suggest that additional changes, such as allocating more resources to the health system and reforming the human resources for health management, will be necessary in DRC to achieve the ambitious global universal health coverage and mortality goals.

Shapira, G., Clarke-Deelder, E., Booto, B.M. et al. Impacts of performance-based financing on health system performance: evidence from the Democratic Republic of Congo. BMC Med 21, 381 (2023).

https://link.springer.com/article/10.1186/s12916-023-03062-8#article-info