Banking survey experiments

This Pre-analyis Plan (PAP) contains analyses for the Randomised Controlled Trial (RCT) components of the survey.

This revised PAP details two main changes:
1. Additional data collection (wave 2) to address a display error discovered in the Savings Treatment 2 condition along with our updated analytical approach. Note that wave 2 data collection commenced on 01 May 2025 prior to the submission of this PAP revision. However, no data analysis has been or will be conducted before finalisation of this revised analysis plan.
2. Changes to the way we code the primary outcome variables. While we originally planned to use a dictionary-matching approach, this was unable to deliver adequate accuracy in testing. Therefore the data will be fully hand-coded.

Each RCT (one for mortgage, one for savings account) will be a three-arm survey experiment. The effects of prompts (attention control + 2 treatments) will be tested via vignettes. Participants are asked to provide advice to their friend in the vignette. A mock-up banking app showing their friend’s financial position is shown and the app screen includes one of the three prompts.
The text in the prompts are as below.
Mortgage stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): A lower interest rate will save you money on your home loan. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Your home loan interest rate is higher than average. 7.8% your current rate, 6.5% average rate for similar home loans. Visit moneysmart.gov.au/mortgagecalculator to see how much you could save with a lower interest rate. This information is required by the Australian Government.
Savings stream
Attention control: Check out our great range of products! We have a wide range of products to suit your financial needs. Visit banking.com.au/all-products for more information. Eligibility requirements may apply.
Treatment 1 (simple prompt): Grow your savings with a higher interest rate Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.
Treatment 2 (detailed prompt): Grow your savings with a higher interest rate A competitive savings account will offer an interest rate of 4% or higher. Visit moneysmart.gov.au to check if you are getting the best interest rate for your savings. This information is required by the Australian Government.

2025-03-17

2025-03-22

Savings stream primary outcome: A binary variable based on recommendations provided by survey respondent about engaging with the market where 1 is engaging with the market (e.g. switching to a higher interest savings account or looking for a better interest rate savings account) and the rest is coded 0.
Mortgage stream primary outcome: A binary variable based on recommendations provided by survey respondents about engaging with the market where 1 is engaging with the market (e.g. looking for a better mortgage product, asking for a reprice, switching banks to get better interest rate on home loan) and the rest is coded 0.
The question asked to participants for the primary outcome is ‘After looking at your friend's banking app screen below, what are some of the actions you suggest that he takes to improve his financial position?’

See Pre-analysis Plan for further details and a copy of the script.

This project includes 2 RCTs (one for mortgage stream, one for savings stream) embedded in surveys. Each RCT will be a 3-arm randomised controlled field trial. All participants will be shown a vignette asking them to recommend actions to help their friend improve their financial position. Part of the vignette includes a banking app screen which contains prompts (interventions).
The trial design for the Savings stream mirrors that of the Mortgage stream except for the differences in the vignette presented.
During the implementation of the original study, we discovered a display error in the Savings T2 condition where a negative sign was missing from the personal loan component display. To address this issue, we are conducting additional data collection (wave 2) with the corrected display. This wave 2 data collection uses an unequal allocation approach to efficiently address this issue:
• Savings T2 condition: 1,000 participants
• Savings Control condition: 40 participants
• Savings T1 condition: 40 participants
The home loan stream conditions are not included in wave 2 data collection as they did not contain any errors in wave 1. All procedures, measures, and randomisation in wave 2 are identical to wave 1, except for the correction of the display error in Savings T2.
Meta-analytic approach for savings stream
To maximise statistical power while addressing the display error in the savings T2 condition, we will use the following analytical approach:
For savings Control and Treatment 1 conditions:
We will calculate the proportion differences (e.g., percentage point difference between treatment and control and cohen’s h) separately for each wave
We will calculate the standard error for each difference and then meta-analytically combine these estimates using a random-effects model
We will report both the separate wave estimates and the combined estimate with 95% CIs
Heterogeneity between waves will be assessed using I^2and Q statistics
For the Savings T2 condition:
Only wave 2 data (with the corrected display) will be used in analyses
Comparisons with other conditions will use the meta-analytically combined estimates for those conditions
For all home loan conditions:
Analyses will use only wave 1 data as no additional data is being collected.
This approach allows us to maintain causal inference while addressing the specific error that occurred, as it preserves the randomised nature of the design in both waves.

Participants will be randomised into one of the three cells in each RCT within the survey platform, Qualtrics, using the inbuilt randomisation functionality, with roughly equal probability of assignment across the groups (using the ‘evenly present elements’ option in Qualtrics).

Individual level

No

No clusters

approximately 4200-4500 participants for the mortgage RCT stream and about 3800-4100 for the savings RCT stream.

We expect approximately 4000-4500 participants for each RCT (one for mortgage and one for savings) with approximately 1400-1500 participants in each arm of the RCT (except of the savings stream Treatment group 2).
As we will not be analysing wave 1 Treatment group 2 data, we will have approximately 1000 participants for this treatment group.

Based on previous work and a pilot study, we estimate the effect size for this study will be around 10-15 percentage points. We will set the alpha at 5% and power at 90% as this is a survey experiment and we do not want to miss a true effect. With a sample size of around 4200 we will have 90% power to detect an effect size of 5pp or approximately 0.1 Cohen’s h. Wave 2 We conducted power calculations to determine the minimum sample sizes needed for our wave 2 data collection. With our wave 1 sample, of approximately 1,500 participants per condition, and using a meta-analytic approach that combines data across waves for Savings Control and T1, we can achieve 90% power (alpha = 0.05) to detect a 5% difference with the following wave 2 sample sizes: • Savings T2: 150 participants would be sufficient, but we have allocated 1,000 to ensure consistent precision. • Savings Control and T1: 40 participants each The larger sample for Savings T2 provides substantially narrower CIs and greater precision, while the smaller samples for Control and T1 are sufficient when combined meta-analytically with the wave 1 data. Power calculations were conducted in R, version 4.4.1 using ‘pwr’ package version 1.3-0.