|
Field
Last Published
|
Before
October 17, 2023 10:55 AM
|
After
October 26, 2023 07:19 PM
|
|
Field
Intervention (Public)
|
Before
We administer a math test online, where subjects attempt to solve 30 randomly-sample questions from 3 standardized tests of various difficulty levels (4th grade, 8th grade, and High school). These tests belong to the TIMSS family of tests, which are administered every four years in several countries.
|
After
We administer a math test online, where subjects attempt to solve 30 randomly-sample questions from 3 standardized tests of various difficulty levels (4th grade, 8th grade, and High school). These tests belong to the TIMSS family of tests, which are administered every four years in several countries. We are restricting our attention to multiple-choice questions, which represent the large majority of available questions and are more directly comparable between them. The size of the question pool will be around 420.
|
|
Field
Experimental Design (Public)
|
Before
Subjects see a total of 30 questions, split into 3 blocks of 10. Questions are randomly sampled from the total pool of questions, and block order is randomized. For this stage there are no "treatments".
We expect subjects' performance to be decreasing on average across test levels. At the extremes of the ability distribution, a flatter slope can be expected given that performance (defined as success rate across questions seen) is bounded between 0 and 1.
We also expect that within-test performance will decrease with question difficulty, proxied by the % of students who answered the question correctly.
|
After
Subjects see a total of 30 questions. Questions are randomly sampled from the total pool of questions. For this stage there are no "treatments".
The goal of this exercise is to create an index of difficulty across all questions, using average success rates. We expect (tautologically) that a given subject' performance will be decreasing with question difficulty (computed with average performance). At the extremes of the ability distribution, a flatter slope can be expected given that performance is bounded between 0 and 1.
|
|
Field
Randomization Method
|
Before
Randomization done through qualtrics survey flow (for blocks) and block randomization (for questions within a block).
|
After
Randomization done through qualtrics survey flow.
|
|
Field
Planned Number of Clusters
|
Before
Between 200 and 350 subjects.
|
After
We want to obtain a large enough number of responses for each item, to create a reliable difficulty index. We would need between 20 and 40 responses per question: for 420 questions this means between 280 and 560 subjects approximately.
|
|
Field
Planned Number of Observations
|
Before
Between 200 and 350 subjects.
|
After
Between 280 and 560 subjects
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
Between 200 and 350 subjects.
|
After
Between 280 and 560 subjects
|