Culturally and context specific student assessment

We estimate the effect of adapting and deploying a universal test similar to the National Assessment Program – Literacy and Numeracy (NAPLAN) test to children from different cultures and circumstances.

The sample will include a minimum of (i) 240 Aboriginal and Torres Strait Islander (ATSI) students currently in year 6 (11-12 years old) and their class peers, and (ii) 240 ATSI students currently in year 8 (14-15 year old) and their class peers. In each group, half of the student will be randomly assigned into the group receiving the standard test, whilst the other will receive the contextualized test.

The goal of this project is to explore whether students are more responsive and engaged in assessment tasks if they are locally contextualized and, importantly, whether students perform better on standardized assessments if these are made relevant to their own backgrounds and cultural competencies. We are additionally investigating the mechanism behind these effects.

2021-03-22

2021-03-26

1. Differences in test performance between students who undertook the standard tests compared to those who undertook the contextualized tests.
2. Differences in perceived test effort between students who undertook the standard tests compared to those who undertook the contextualized tests.
3. Differences in perceived test relevance between students who undertook the standard tests compared to those who undertook the contextualized tests.

Important details:
• There are 2 experimental groups: one control and one treatment.
• In control ("Standard Test"), students will receive a standard numeracy and reading test (in the same style as NAPLAN).
• In treatment ("Culturally Appropriate Test"), students will receive a revised version of the standard test that will take into account their context and cultural background.
• There are no differences in the educational content of the two tests (just merely in their form) and each of the numeracy and reading tasks will take 50 minutes regardless of their (standard or contextualised) form.

Randomization will be carried out by a computer.

Randomization will be assigned at the individual (student) level.

No

NA

The analysis involving main outcomes (1) is based on (i) 240 ATSI students currently in year 6 and 500 of their class peers, and (ii) 240 ATSI students currently in year 8 and 500 of their class peers. Depending on the effort indicator mentioned in main outcomes (2-3), we will have the same number of observations as above.

We have one treatment ("Culturally Appropriate Test") and one control (“Standard Test”), across 4 groups:
1. ATSI students in year 6: 120 treatment and 120 control
2. ATSI students in year 8: 120 treatment and 120 control
3. Peers of ATSI students in year 6: 250 treatment and 250 control
4. Peers of ATSI students in year 8: 250 treatment and 250 control

We conduct power calculations using the 2019 NAPLAN scores for the relevant groups. We assume power of 0.80 and significance level of 0.05. For test scores, we are powered for the following minimum detectable effect sizes: • ATSI year 6 reading: 25 points; 0.36 standard deviations; 5.5% • ATSI year 6 numeracy: 21 points; 0.36 standard deviations; 4.8% • ATSI year 8 numeracy: 25 points; 0.36 standard deviations; 5.0% • Peers of ATSI year 6 reading: 17 points; 0.25 standard deviations; 3.4% • Peers of ATSI year 6 numeracy: 17 points; 0.25 standard deviations; 3.4% • Peers of ATSI year 8 numeracy: 20 points; 0.25 standard deviations; 3.5%