Making Information Actionable: Experimental Evidence from Kenyan Courts

Last registered on January 06, 2022

Pre-Trial

Trial Information

General Information

Title
Making Information Actionable: Experimental Evidence from Kenyan Courts
RCT ID
AEARCTR-0006228
Initial registration date
July 24, 2020
Last updated
January 06, 2022, 12:58 PM EST

Locations

Region

Primary Investigator

Affiliation
McGill University

Other Primary Investigator(s)

Additional Trial Information

Status
In development
Start date
2018-12-01
End date
2023-01-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
Courts in developing countries face numerous constraints that prevent them from providing efficient and fair justice to citizens, or a strong institutional environment conducive to investment and growth. We ask: can low-cost, information-based interventions, using data regularly captured by administrative systems, help judicial officers overcome common incentive and behavioral constraints, and in so doing improve court performance? This note describes the pre-analysis plan of an intervention that will be implemented together with the World Bank and the Kenyan judiciary.
External Link(s)

Registration Citation

Citation
Chemin, Matthieu. 2022. "Making Information Actionable: Experimental Evidence from Kenyan Courts." AEA RCT Registry. January 06. https://doi.org/10.1257/rct.6228-1.1
Experimental Details

Interventions

Intervention(s)
This study will test two interventions:

1. Actionable information: Providing judicial officers a one-page, simplified feedback form every month, that shows them 1) the case clearance rate (CCR; number of cases resolved divided by number of cases filed), 2) the number of adjournments and their top three reasons, 3) the predicted impact of reducing adjournments on the CCR

2. Actionable information with deliberation: As above, but also sharing the information with Court User Committees (CUCs) that include representatives from public security agencies and local communities, to focus discussion around performance, create bottom-up accountability, and collectively arrive at ways to remove the bottlenecks.

The feedback forms were sent once in January 2019. The goal will be to send additional feedback forms in 2020 and 2021 every month for a duration of one year.
Intervention Start Date
2019-01-01
Intervention End Date
2022-12-01

Primary Outcomes

Primary Outcomes (end points)
The primary outcome is the proportion of hearings ending in an adjournment.

We will use a difference-in-differences analysis, with court and month fixed effects.

The research plan includes subgroup analyses by:

• Baseline time to disposition

• Baseline number of adjournments, disaggregated by their main reasons

• Strength of feedback (i.e., 1) spread between CCR and recommended target, 2) number of adjournments reported, and 3) size of predicted impact)

• Nature of top reasons of adjournments

• Type of court (high court, magistrate court, emloyment and labour relations court, environment and land court)

• Type of case (civil (e.g., succession, commercial), criminal (e.g., property, violent))

The randomization was stratified on 1) regions (8), and 2) time to disposition. These stratification dummies will be added to the model.
Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)
• Adjournments disaggregated by reason for adjournment. In particular, adjournments can be “internal”, i.e., under the control of the judge (e.g., court not sitting) or “external”, i.e., caused by other parties (e.g., lawyer not ready).

• Case clearance rate (cases disposed of divided by cases filed) as a standard measure of productivity of courts, as well as cases disposed, cases filed

• Time to disposition: Considering the feedback forms were only sent once in January 2019, it may seem unreasonable to expect an effect on the time to disposition of cases. If additional feedback forms can be sent in 2020 and 2021 every month for a duration of one year, there may be an effect on time to disposition. Since the study period is only a year and given that it takes an average of seven months to resolve a criminal case, it is reasonable to expect an effect on the time to resolve criminal cases. In contrast, it takes on average four years to resolve a civil case, so it may be harder to detect an effect on the time to disposition of civil cases--however, some categories of civil cases are resolved faster than others, so we will narrow down on these categories by considering all case types that are resolved (on average) within the course of a year.

• Backlog (percentage of cases that have been pending for more than 1 year)

• Age of case resolved (to see whether judges decide to resolve new versus older cases)

• Time taken for case hearing

• Time taken from filing to first hearing of cases

• Legal representation

• Quality: appeal rates, conviction, dismissal

• Access: total caseload

From digitization of cases extracted from Kenya Law database at the High Court level:

• assignment of judge to litigant [both a check-of and distortion of random assignment]

• decision direction

• whether the decision is appealed

• citation count of the decision, length of decision, number of laws/acts cited in the judgement, how many times the judgement in question has been cited

• time to disposition [if scrapeable from the text of the data]

• number of published cases

• gender slant, gender access to courts

• If possible to identify economic disputes (contracts, security of property rights, credit), number of such cases; time to disposition and direction of decision in favor of certain parties (creditors versus lenders, or big firms versus individuals)

For the above data at the High Court level, the research plan includes further subgroup analyses by:

• Proportion of lower courts treated (since cases for lower-level courts will percolate the High Courts)

Additionally, a daily event study when the lower court decisions percolate into the higher court will be implemented. The dates of the lower court filing or resolution might be used as separate event study dates along side the treatment dates of the lower courts, to check whether the quality of decisions at the lower-level courts impacts the process of cases at the High Court level.

Other analysis if data is available:

• If court user satisfaction surveys are available: satisfaction with court services

• If employee surveys are available: use of case management, motivation

• If CUC meeting minutes are available: procedures put in place to deal with adjournments

• If PMMU evaluations are available: evaluation score, court use of funds (compliance of courts with budget utilization of allocated funds), attendance of judges to the yearly judicial training

If there is an effect on adjournment and time to disposition (especially if the additional feedback forms are sent in 2020 and 2021 every month for a duration of one year), then we may expect downstream economic effects:

• If Kenya Integrated Household Budget Survey: investment, business creation, access to credit, consumption

• If World Bank Enterprise Surveys is available: contracting behavior, trust in courts, courts as obstacles to business

Multiple hypothesis testing will be done with the Sharpened False Discovery Rate (FDR) adjusted q-values (Anderson, 2008).
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
The unit of randomization is the court station level. A court station is a physical compound that may contain several courts. One station has one Court User Committee (CUC). There are 124 court stations in Kenya. There are three groups of equal size:

• Court stations receiving the feedback report every month

• Court stations receiving the feedback report every month, shared with local court user committees (CUC) and discussed at quarterly CUC meetings

• A control group of stations

To randomize, we stratified the sample on geographical variables (the Kenyan judiciary established a list of 8 regions that closely match the eight existing regions of Kenya) and baseline time to disposition in the station.
Experimental Design Details
Not available
Randomization Method
The randomization was done in office by a computer.
Randomization Unit
The unit of randomization is the court station level.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
The planned number of clusters is the 123 court stations in Kenya.
Sample size: planned number of observations
The planned number of observations is the universe of cases going through the courts in Kenya.
Sample size (or number of clusters) by treatment arms
41 court stations receiving the feedback report, 41 receiving the feedback report shared with local court user committees (CUC), 41 court stations control.
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
IRB

Institutional Review Boards (IRBs)

IRB Name
McGill University Research Ethics Board
IRB Approval Date
2020-07-08
IRB Approval Number
20-06-027