Improving the Quality of Legal Aid: Impact Evaluation of Tech-Enabled Mediation in Peru

Last registered on December 14, 2021

Pre-Trial

Trial Information

General Information

Title
Improving the Quality of Legal Aid: Impact Evaluation of Tech-Enabled Mediation in Peru
RCT ID
AEARCTR-0008607
Initial registration date
December 14, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
December 14, 2021, 4:35 PM EST

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Locations

Region

Primary Investigator

Affiliation
The World Bank

Other Primary Investigator(s)

PI Affiliation
The World Bank
PI Affiliation
University of California, Los Angeles
PI Affiliation
The World Bank

Additional Trial Information

Status
In development
Start date
2021-11-12
End date
2023-02-01
Secondary IDs
Prior work
This trial does not extend or rely on any prior RCTs.
Abstract
How to raise the quality of publicly provided services in in the face of tight resource constraints? In this project, we evaluate the introduction of a tech-based, low-cost platform allowing extra-judicial mediators in Peruvian free mediation centers to monitor their own performance. We assess the impact of the platform on both mediators' performance and on socio-economic characteristics of the users of these mediation services.
External Link(s)

Registration Citation

Citation
Chen, Daniel Li et al. 2021. "Improving the Quality of Legal Aid: Impact Evaluation of Tech-Enabled Mediation in Peru." AEA RCT Registry. December 14. https://doi.org/10.1257/rct.8607-1.0
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Partnering with the Peruvian Ministry of Justice, we developed a platform named "App del Conciliador" (Mediator App), which leverages rich administrative data on extra-judicial mediation cases from Peruvian free public mediation centers. The platform provides mediators with rolling performance reviews that we expect to increase the salience of outlier characteristics, such as unusually lengthy average resolution times compared with colleagues. We aim to make mediators aware of their own abilities and the key areas for improvement. Thus, mediators may become better self-managers and can proactively address shortcomings, while prioritizing casework that optimizes the marginal value of their limited time. In addition, by comparing their performance to that of other peer mediators on similar case types, the Mediator App will provide feedback on what areas they have greater room to improve on.
Intervention (Hidden)
In the first RCT, the mediators whose centers have been randomly selected (50%) will receive access to the App and weekly reminders to log in. In the App, they will be able to monitor their individual performance on three key indicators: the number of mediation cases they work on, the rate of settlements with partial or full agreement, and the average number of days it takes them to close a mediation file. Their performance will be compared to both their past performance and the average performance of all mediators. This will provide a clear indication to each mediator of the level of his/her own performance on each of these metrics, and demonstrate any progress or setbacks.

All metrics are broken down according to the different types of cases each mediator is handling. We expect that mediators will thereby get to know better their performance both individually and in comparison with their peers and feel motivated to improve their performance. In the longer run, we expect this improved performance to increase the number of citizens served at the mediation centers and to improve the efficiency and rate of agreement achieved through mediation. We will measure what is the effect on the wellbeing of the parties involved in mediation, measured by economic and health indicators as well as by indicators measuring their children’s performance at school (if applicable).
Intervention Start Date
2021-12-06
Intervention End Date
2022-04-06

Primary Outcomes

Primary Outcomes (end points)
(A) performance-related self-awareness
(B) performance
(B.i) number of mediation cases
(B.ii) percentage of successful mediation cases
(B.iii) duration of mediation cases
Primary Outcomes (explanation)
(A) performance-related self-awareness: we assume that the improvement of performance requires improved self-management which, in turn, relies on accurate estimates of performance indicators. These estimates are our primary outcome (A).
(B) performance: We measure mediators' weekly performance along the following three indicators:
(B.i) monthly number of mediation cases worked on
(B.ii) percentage of successful mediation cases
(B.iii) average duration of mediation cases in natural days

Secondary Outcomes

Secondary Outcomes (end points)
(I) economic status
(II) health status
(III) educational performance
(IV) workload of the courts in the respective districts of free mediation centers
Secondary Outcomes (explanation)
Secondary outcomes refer to users of free mediation services:
(I) economic status: the economic status of the parties involved in mediation as reported in an endline survey
(II) health status: the health status of parties involved in mediation cases as reported in an endline survey
(III) educational performance: the educational performance of students involved in mediation cases (mostly children of the mediating parties) as captured in Peru's Information System to Support the Management of the Educational Institution (SIAGIE)
(IV) workload of the courts in the respective districts of free mediation centers: the workload of the courts in the respective districts of free mediation centers in terms of case types that are eligible for mediation

Experimental Design

Experimental Design
In our experiment we use simple lottery to assign mediation centers either to the control group or to our single treatment consisting in access to the Mediator App. The mediators belonging to mediation centers selected for treatment (50%) will receive access to the App and weekly reminders to log in. In the App, they will be able to monitor their performance on three key indicators: the number of mediation cases they work on, the rate of settlements with partial or full agreement, and the average number of days it takes them to close a mediation file. Their current performance will be compared to both their past performance and the average performance of all mediators. This will provide a clear indication to each mediator of the level of his/her own performance on each of these metrics, and demonstrate any progress or setbacks.
Experimental Design Details
Randomization Method
First, we retrieved from the administrative data all the mediators who were at any point active in the dataset. We combined this data with a short-list of all on duty mediators provided to us by the Ministry of Justice to have the universe of all on-duty and off-duty (but potentially returning) mediators.
We proceed with stratified randomization for treatment assignment for mediators belonging to a single-mediator center. The strata are defined as: 1) the number of cases mediated by the mediator, 2) district population and 3) on/off-duty status. We convert the first two variables into binary variables (high/low) to increase the size of each group and reduce imbalance between treatment and control. The probability weight for treatment assignment was 0.5.
Additionally, 8.9 percent of our sample of mediators belong to centers with multiple mediators - the remainder are centers with a single mediator. For those centers with multiple mediators, we randomly assigned them a treatment status with a 0.5 probability weight. In our randomization procedure, all centers with multiple mediators were assigned a treatment status. As a result, all mediators belonging to these centers were assigned to the treatment group.
Randomization of treatment assignment leaves us with with 39 mediators in control and 40 mediators in the treatment group. To check for covariate balance, we estimate a simple OLS regression, using as outcome the treatment assignment, and as predictors a set of covariates that we consider as potentially correlated with potential outcomes: the department in which the mediator is located, population size of district, work experience, case duration and success rate for agreements.
We find that almost all of the coefficients are not statistically significant and the covariates across treated and control groups are indeed balanced, with the exception of two departments out of 25 – Huanuco and Piura. To account for this minor imbalance, a full set of controls, including department fixed effects, will be included in our main model specificationss.
Randomization Unit
mediation center level (where most centers have 1 mediator per center)
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
75 mediation centers
Sample size: planned number of observations
For 14 weeks, we will observe the performance of 79 mediators every week.
Sample size (or number of clusters) by treatment arms
treatment (i.e., access to app): 40 mediators from 39 mediation centers
control: 39 mediators from 36 mediation centers
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
For each of our primary outcome variables the following applies: unit: mediation center MDE: 19% SD: 1 Please see the appendix for a detailed description of our power calculation.
Supporting Documents and Materials

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
IRB

Institutional Review Boards (IRBs)

IRB Name
University of California Los Angeles Institutional Review Board
IRB Approval Date
2021-10-21
IRB Approval Number
IRB#21-001678

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials