|
Field
Last Published
|
Before
April 10, 2025 07:33 AM
|
After
May 22, 2025 12:33 AM
|
|
Field
Intervention (Public)
|
Before
Each RCT (one for mortgage, one for savings account) will be a three-arm survey experiment. The effects of prompts (attention control + 2 treatments) will be tested via vignettes. Participants are asked to provide advice to their friend in the vignette. A mock-up banking app showing their friend’s financial position is shown and the app screen includes one of the three prompts.
The text in the prompts are as below.
Mortgage stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): A lower interest rate will save you money on your home loan. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Your home loan interest rate is higher than average. 7.8% your current rate, 6.5% average rate for similar home loans. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Savings stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): Grow your savings with a higher interest rate Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Grow your savings with a higher interest rate A competitive savings account will offer an interest rate of 4% or higher. Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
|
After
This Pre-analyis Plan (PAP) contains analyses for the Randomised Controlled Trial (RCT) components of the survey.
This revised PAP details two main changes:
1. Additional data collection (wave 2) to address a display error discovered in the Savings Treatment 2 condition along with our updated analytical approach. Note that wave 2 data collection commenced on 01 May 2025 prior to the submission of this PAP revision. However, no data analysis has been or will be conducted before finalisation of this revised analysis plan.
2. Changes to the way we code the primary outcome variables. While we originally planned to use a dictionary-matching approach, this was unable to deliver adequate accuracy in testing. Therefore the data will be fully hand-coded.
Each RCT (one for mortgage, one for savings account) will be a three-arm survey experiment. The effects of prompts (attention control + 2 treatments) will be tested via vignettes. Participants are asked to provide advice to their friend in the vignette. A mock-up banking app showing their friend’s financial position is shown and the app screen includes one of the three prompts.
The text in the prompts are as below.
Mortgage stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): A lower interest rate will save you money on your home loan. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Your home loan interest rate is higher than average. 7.8% your current rate, 6.5% average rate for similar home loans. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Savings stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): Grow your savings with a higher interest rate Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Grow your savings with a higher interest rate A competitive savings account will offer an interest rate of 4% or higher. Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
|
|
Field
Intervention End Date
|
Before
March 28, 2025
|
After
March 22, 2025
|
|
Field
Experimental Design (Public)
|
Before
This project includes 2 RCTs (one for mortgage stream, one for savings stream) embedded in surveys. Each RCT will be a 3-arm randomised controlled field trial. All participants will be shown a vignette asking them to recommend actions to help their friend improve their financial position. Part of the vignette includes a banking app screen which contains prompts (interventions).
|
After
This project includes 2 RCTs (one for mortgage stream, one for savings stream) embedded in surveys. Each RCT will be a 3-arm randomised controlled field trial. All participants will be shown a vignette asking them to recommend actions to help their friend improve their financial position. Part of the vignette includes a banking app screen which contains prompts (interventions).
The trial design for the Savings stream mirrors that of the Mortgage stream except for the differences in the vignette presented.
During the implementation of the original study, we discovered a display error in the Savings T2 condition where a negative sign was missing from the personal loan component display. To address this issue, we are conducting additional data collection (wave 2) with the corrected display. This wave 2 data collection uses an unequal allocation approach to efficiently address this issue:
• Savings T2 condition: 1,000 participants
• Savings Control condition: 40 participants
• Savings T1 condition: 40 participants
The home loan stream conditions are not included in wave 2 data collection as they did not contain any errors in wave 1. All procedures, measures, and randomisation in wave 2 are identical to wave 1, except for the correction of the display error in Savings T2.
Meta-analytic approach for savings stream
To maximise statistical power while addressing the display error in the savings T2 condition, we will use the following analytical approach:
For savings Control and Treatment 1 conditions:
We will calculate the proportion differences (e.g., percentage point difference between treatment and control and cohen’s h) separately for each wave
We will calculate the standard error for each difference and then meta-analytically combine these estimates using a random-effects model
We will report both the separate wave estimates and the combined estimate with 95% CIs
Heterogeneity between waves will be assessed using I^2and Q statistics
For the Savings T2 condition:
Only wave 2 data (with the corrected display) will be used in analyses
Comparisons with other conditions will use the meta-analytically combined estimates for those conditions
For all home loan conditions:
Analyses will use only wave 1 data as no additional data is being collected.
This approach allows us to maintain causal inference while addressing the specific error that occurred, as it preserves the randomised nature of the design in both waves.
|
|
Field
Planned Number of Observations
|
Before
approximately 4200-4500 participants for each RCT
|
After
approximately 4200-4500 participants for the mortgage RCT stream and about 3800-4100 for the savings RCT stream.
|
|
Field
Sample size (or number of clusters) by treatment arms
|
Before
We expect approximately 4200-4500 participants for each RCT (one for mortgage and one for savings) with approximately 1400-1500 participants in each arm of the RCT
|
After
We expect approximately 4000-4500 participants for each RCT (one for mortgage and one for savings) with approximately 1400-1500 participants in each arm of the RCT (except of the savings stream Treatment group 2).
As we will not be analysing wave 1 Treatment group 2 data, we will have approximately 1000 participants for this treatment group.
|
|
Field
Power calculation: Minimum Detectable Effect Size for Main Outcomes
|
Before
Based on previous work and a pilot study, we estimate the effect size for this study will be around 10-15 percentage points.
We will set the alpha at 5% and power at 90% as this is a survey experiment and we do not want to miss a true effect. With a sample size of around 4200 we will have 90% power to detect an effect size of 5pp or approximately 0.1 Cohen’s h.
Power calculations were conducted in R, version 4.4.1 using ‘pwr’ package version 1.3-0.
|
After
Based on previous work and a pilot study, we estimate the effect size for this study will be around 10-15 percentage points.
We will set the alpha at 5% and power at 90% as this is a survey experiment and we do not want to miss a true effect. With a sample size of around 4200 we will have 90% power to detect an effect size of 5pp or approximately 0.1 Cohen’s h.
Wave 2
We conducted power calculations to determine the minimum sample sizes needed for our wave 2 data collection. With our wave 1 sample, of approximately 1,500 participants per condition, and using a meta-analytic approach that combines data across waves for Savings Control and T1, we can achieve 90% power (alpha = 0.05) to detect a 5% difference with the following wave 2 sample sizes:
• Savings T2: 150 participants would be sufficient, but we have allocated 1,000 to ensure consistent precision.
• Savings Control and T1: 40 participants each
The larger sample for Savings T2 provides substantially narrower CIs and greater precision, while the smaller samples for Control and T1 are sufficient when combined meta-analytically with the wave 1 data.
Power calculations were conducted in R, version 4.4.1 using ‘pwr’ package version 1.3-0.
|