Kenya Patient Safety Impact Evaluation

1272 health facilities in three Kenyan counties–Kakamega, Kilifi and Meru–are experimentally allocated to three arms: (1) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities; (2) high-intensity inspections with enforcement of warnings and sanctions for non-compliant facilities and public disclosure of inspection results; and (3) “business-as-usual” low-probability inspections (the control group).

The new regulatory framework used for this experiment was developed previous to the implementation of these pilots including: (A) a refined Joint Health Inspections Checklist (henceforth Checklist) that is easy to deploy and is focused on the fundamentals of patient safety; (B) a scoring system that allows facilities to be categorized according to the level of risk presented to patients; (C) scores that trigger warnings and sanctions to be enforced according to a facility’s level of risk.

Treatment 1 - High-intensity inspections: Every facility in this treatment group will be inspected using a checklist of safety standards that was developed in consensus and refined over 3 years. The health facilities will receive a copy of the Checklist and will be warned or sanctioned according to their risk performance.

Treatment 2 - High-intensity inspections and public disclosure of patient safety performance: Every facility in this treatment group will be inspected as in treatment 1. Following the inspection, facilities will be assigned a letter-grade to be prominently posted outside the health facility. The health facilities will receive a copy of the Checklist results with details on their assessment and grade. A campaign introducing patients to the grading system will be included as part of the accountability system.

Control - “Business-as-usual” health inspections: Health facilities in this group will operate under the current regime for health inspections, which results in inspections less than 4% of the time. These facilities did not receive routine inspections during the implementation period.

2016-11-14

2017-12-15

We examine the impact of the intervention on a triad of outcome groups that are important in settings with public and private providers: Quality of care and Patient Safety, Quantity and Prices.

Quality and Patient Safety will be measured through three main indicators that gauge diverse dimensions of quality and patient safety at different points in the continuum of the health care chain: (1) Adherence to a Checklist developed by the regulators and that includes indicators of quality and patient safety related to protocols, infrastructure, and equipment at the health facility level; (2) Adherence to patient safety practices related to infection prevention and control; (3) Adherence to case-specific checklists of essential and recommended care for four medical cases (subject to budget availability). Quantity (demand) of care services will be measured through patient flow records from health facilities and administrative data, and prices will be measured through patient exit surveys.

Below is a list of the main outcomes of interest for this analysis, as well as the source and the method of data collection used in each case.

• Adherence to a Checklist of quality of care and patient safety standards. This is a Checklist for Singular/Joint Inspections for Public and Private Medical Institutions, which has been developed by a technical working group (TWG) including the regulatory bodies under the Ministry of Health, the private sector, and other stakeholders. This Checklist includes indicators of infrastructure, equipment, staff characteristics, and protocols to measure quality and patient safety across all units of a health facility, and hence it allows measuring how facilities stand with respect to the standards set by the regulation. The results are mapped into a score and the measure of adherence we will use is the score as a percentage of the maximum score possible for each facility. We developed an electronic version of this Checklist. Data will be collected through surveys to the facility in-charge and staff, as well as through observation and verification during the surveys following closely the protocols that inspectors follow when they conduct inspections.

• Adherence to patient safety practices related to infection prevention, and control (IPC) in primary care, which are measured by direct observation of select practices in 3 outpatient sites. These outcomes will measure adherence to patient safety practices for five groups: (i) hand hygiene, (ii) injection and blood draw practices, (iii) use of gloves, (iv) disinfection of reusable equipment, and (v) waste segregation. This selection was based on their high-value (i.e. strong link to health-care infections), high frequency of opportunities (i.e., points in time during the care chain when a patient safety practice should be performed to avoid likely adverse events), and feasibility to develop valid, reliable, and generalizable indicators in the planned time and conditions. Adherence indicators are based on indications and actions and adapted to the specific violation.

• Adherence to case-specific checklists of essential and recommended care for 4 medical cases (subject to budget availability). This dimension of quality of care and patient safety is measured through Standardized Patients, which allows us to gauge the extent of correct treatment (both under and over-treatment) in these facilities. The method of Standardized Patients (SPs) was used in the first large-scale population study in India in 2008-2009 (See, e.g., Das et al., 2012). SPs are people from the local community who are extensively trained to present the same case to multiple providers. To assess the quality of care, SPs are trained to recall all history questions, examinations, and diagnoses. They are debriefed with a structured questionnaire within one hour of the interaction. The quality of medical advice is assessed by the time spent with patients, by providers’ adherence to case-specific checklists of essential and recommended care, the likelihood of correct diagnosis, and the appropriateness of treatment. SPs will undergo an exit survey where the data on their interaction with the doctor is recorded.

• Price of services at health facilities. The data on prices will come from a short survey that will be collected from patients from various demographic characteristics exiting the health facilities, as well as from SPs. In large health facilities, sampling of patients will be done randomly using a skip routine.

• Quantity (demand) of health care services: The data on demand corresponds to the number of patients in the last complete month at the time of the survey, collected at the health facility level through a survey from the forms reported to the MOH, or from the facility's alternative record system.

We use an stratified cluster randomized experimental design to estimate the causal impact of the interventions at the facility level, where the clusters are markets of health facilities (where facilities are within 4 km from the market center) and stratification happens at the market size and county level. We experimentally allocate around 273 markets and all facilities (around 1,272) in each market in the three evaluation counties to one of the three accountability systems (treatment 1, 2 and control). We will also present results at the market level. For this analysis, this is an stratified random experimental design where the unit of analysis is the same as the unit of randomization.

For the analysis at the facility level, this evaluation uses a cluster randomized experimental design to estimate the causal impact of the interventions, where the clusters are markets of health facilities, and the cluster size is the market size, or number of health facilities per cluster. All local markets of health facilities in the three counties–Kakamega, Kilifi, and Meru–will be randomized into treatment and control groups, taking into consideration the size of the market. At the market level, the design is an stratified random design, with stratification at the market size.

The three counties were chosen after an extensive consultation with representatives of the Kenyan county health executives, based on prior analysis of health markets in all Kenyan counties. Specifically, we constructed health markets in each county and highlighted the counties that contained markets with diverse sizes. In some counties like Turkana, there were few markets with more than one health facility (typically in the arid Northern counties) and evaluations from these counties would not be predictive of performance in denser health markets. In other counties, such as Mombasa, most health facilities are in markets with a large number of other health facilities. In such counties, the predictive power of the evaluation for smaller markets is poor. Ten counties satisfied our technical criteria of health markets with diverse sizes. The analysis and counties were discussed with the county health executives, who decided the ultimate study areas.

The choice of a cluster-randomized experimental design stems from the nature of the intervention and the relevance of capturing the effect of the intervention at the market level. Particularly, the treatment applied to one health facility is thought to have potential spillovers on geographically proximate health facilities through consumer demand. Therefore, the cluster-randomized design was selected because of the potential spillover at the market-level, and the policy-relevance of impacts at this level in terms of quality of care and patient safety, quantity and prices.

The markets are defined at this stage by geographic delimitations. Based on preliminary data from the baseline from around 8,500 patients surveyed we find that around 75% of the respondents live 4 km or closer from the health facility they visited. Using this distance, we apply a clustering algorithm called the K-means algorithm that identifies markets of health facilities by the nearest center, conditional on the K-distance to the center. Our algorithm stops creating new clusters when all facilities are within the predefined distance (4 km) from their respective market center. Based on this exercise, we estimate there are 273 such markets in the three counties selected, with diverse market size ranging from singletons to markets with 2, 3, 4, 5 or more health facilities.

In total, we have 16 strata. For each county, we have 5 strata by market size for markets with 1, 2, 3, 4-10, and 11+ health facilities. In addition, we worry that extreme-value markets may not be balanced for analysis at the health facility level therefore a stratification of extreme values was added for Meru. Therefore, we have an additional stratum for market size 34 or more health facilities (3 markets) in Meru for a total of 16 strata. This is important because, for our analysis at the health facility level, imbalances in the market size (e.g., that all big markets end up only in one treatment arm or in the control group) will affect that level of analysis. We have 87 markets in the control group (411 facilities), 90 markets in the treatment group 1 (388 facilities), and 96 markets in the treatment group 2 with scorecards (473 facilities).

Randomization done in office by a computer.

Market of health facilities

Yes

273 markets of health facilities

273 markets of health facilities, 1,272 health facilities

Treatment 1: 90 markets, 388 health facilities
Treatment 2: 96 markets, 473 health facilities
Control: 87 markets, 411 health facilities

Adherence to Checklist Standards (score/max score, %) / MDE= (0.27 SD; 25%) Adherence to IPC practices (safe action/indication) (%) / MDE= (0.22 SD, 27%) Adherence to Checklist of essential recommended care (%) / MDE=(0.22 SD, 27%) - (subject to budget availability) Price per Visit (KES) / MDE = (0.27 SD, 20%) Quantity (Monthly patient flow) (Outpatients) / MDE (0.27 SD, 26%)