Evaluation of Conecta Ideas in Chile

Last registered on January 26, 2023

View Trial History

Pre-Trial

Trial Information

General Information

Title

Evaluation of Conecta Ideas in Chile

RCT ID

AEARCTR-0009545

Initial registration date

June 03, 2022

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published

June 06, 2022, 5:59 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated

January 26, 2023, 10:03 AM EST

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Country

Chile

Region

Region Metropolitana

Primary Investigator

Name

Mauricio Romero

Affiliation

ITAM

Contact Primary Investigator

Other Primary Investigator(s)

PI Name

Mauricio Romero

PI Affiliation

ITAM

Contact Investigator

PI Name

Paul Gertler

PI Affiliation

University of California, Berkeley

Contact Investigator

Additional Trial Information

Status

Completed

Start date

2022-04-28

End date

2022-12-31

Keywords

Education

Additional Keywords

learning platform, symbolic incentives, tangible incentives

JEL code(s)

Secondary IDs

Prior work

This trial does not extend or rely on any prior RCTs.

Abstract

We propose a large-scale evaluation of a learning platform designed in 2002 by AutoMind, a multidisciplinary team where the leadership is now at the University of Chile. The evaluation design will allow us to test the effect of Conecta Ideas vis-a-vis a control group. Moreover, the experimental design will feature three treatment groups (in addition to the control group). First, a group where students will earn purely symbolic incentives at the individual level. These incentives will be given for using the platform and for getting correct answers to exercises. Second, a group where these symbolic incentives translate to tangible prizes (e.g., books) for students. Third, a group where prizes are given as a function of the performance of the classroom. In addition to testing the effect of Conecta Ideas, this experimental design will allow us to test: a) individual vs group incentives, and b) tangible, but non-monetary incentives vs. purely symbolic incentives.

External Link(s)

Registration Citation

Citation

Gertler, Paul, Mauricio Romero and Mauricio Romero. 2023. "Evaluation of Conecta Ideas in Chile." AEA RCT Registry. January 26. https://doi.org/10.1257/rct.9545-1.1

Sponsors & Partners

Interventions

Intervention(s)

Conecta Ideas (CI) sessions will be carried out in the computer labs of the schools. In a typical CI session, students work to solve the same set of 20 to 30 exercises assigned to them, which are aligned to the topics covered in regular math instruction that week. Students receive automatic feedback regarding whether their answers are correct. Lab coordinators, hired and supervised by the Conecta Ideas team, are
responsible for conducting the learning sessions at the computer lab in collaboration with regular classroom teachers. Lab coordinators will receive a day of training and ongoing supervision from the implementation team. While teachers will not receive formal training, the program promotes learning-bydoing. The software provides virtual stickers and prizes to students who do well in each CI session. It also keeps track of student’s progress (in the current session and across sessions) and offers each student personalized reports in the form of graphs to motive study effort.

Intervention Start Date

2022-06-06

Intervention End Date

2022-11-30

Primary Outcomes

Primary Outcomes (end points)

Mathematics' test scores of student

Primary Outcomes (explanation)

Secondary Outcomes

Secondary Outcomes (end points)

Secondary Outcomes (explanation)

Experimental Design

The evaluation design will allow us to test the effect of Conecta Ideas vis-a-vis a control group, and at the same time, it will allow us to measure the effect of varying the implementation details of Conecta Ideas (in particular, by testing different incentive designs). We are able to study multiple questions by relying on standardized tests taken by student.

Experimental Design Details

Randomization Method

We conducted the randomization in Stata in a two-stage process. First, we randomized at the school level, stratifying by whether schools have one or more sections in Grade 4 and the quartile of an index that summarized the average IRT score and the standard deviation of scores within a school. The probability of being assigned to control was 0.4, while the probability of being assigned to each of the three treatments (plain vanilla Conecta Ideas, CI with individual incentives, and CI with group incentives) was 0.2. After schools were assigned to one of the four experimental groups, from each treatment school, we randomly selected a single section (in case there were multiple sections in Grade 4) for treatment.

Randomization Unit

We randomized at the school level, stratifying by whether schools have one or more sections in Grade 4 and the quartile of an index that summarized the average IRT score and the standard deviation of scores within a school

Was the treatment clustered?

Yes

Experiment Characteristics

Sample size: planned number of clusters

235 Schools

Sample size: planned number of observations

10,000 students

Sample size (or number of clusters) by treatment arms

We have three treatment groups: plain vanilla Conecta Ideas (CI), Conecta Ideas with individual incentives (II), and Conecta Ideas with group incentives (GI). We assigned 50 schools to each of these groups, and we have 85 schools left that serve as controls.

Minimum detectable effect size for main outcomes (accounting for sample design and clustering)

Based on the baseline test, we assume that ρ =.145. We assume (conservatively) that the baseline covariates (e.g., the baseline test score, age, and gender) explain 30% the variance, and thus that σ = 0.84 ≈√1 − 0.3. Finally, taking into account attrition at the student level, we assume n =24. Therefore, we can detect an effect size of at least (MDE) 0.21σ with power 90% (β) using a 5% test of level (α) when comparing any of the treatments to the control. When comparing any two treatment groups, we have an MDE of 0.23σ

Supporting Documents and Materials

IRB