|
Field
Last Published
|
Before
October 28, 2024 12:52 PM
|
After
October 31, 2024 12:54 PM
|
|
Field
Primary Outcomes (End Points)
|
Before
The primary outcomes are as follows:
1) indicator variable for whether the football coach responds to the email,
2) indicator variable for whether the landlord responds to the email
|
After
The primary outcomes are as follows:
1) indicator variable for whether the football coach responds to the email,
2) indicator variable for whether the realtor or landlord (listing poster) responds to the email
|
|
Field
Primary Outcomes (Explanation)
|
Before
The outcome variable is coded as 1 if there is any type of response and is 0 otherwise.
|
After
The outcome variable is coded as 1 if there is any type of response and is 0 otherwise. We wait at least 3 business days to allow sufficient time for the realtor/landlord or coach before recording the response.
|
|
Field
Experimental Design (Public)
|
Before
The study takes place across 4 countries in central Europe: Austria, Hungary, Czech Republic, and Slovakia. We send emails to amateur football clubs asking them to join for trial practice and send requests to view a rental property. To assess whether taking efforts to learn the local language as a foreigner increases opportunities to socially integrate across these two dimensions, we compare response rates across various text variations.
We will have 6 different message manipulations: (1) local name and proficient local language, (2) foreign name and proficient local language, (3) foreign name, broken local language, and taking local language class, (4) foreign name and broken local language, (5) foreign name, English, and taking a local language class, (5) foreign name, English, and taking a local language class, and (6) foreign name and English. The control group is the text version where a native-sounding name and the native language are used, remaining are 5 different treatment conditions that use a foreign-sounding name.
The messages that employ the broken local language were constructed after consulting online language teachers, native speakers of the language, and chatGPT. We first wrote both the broken and proficient messages in English and then had online language teachers translate the messages into German, Czech, Slovak, and Hungarian.
|
After
The study takes place across 4 countries in central Europe: Austria, Hungary, Czech Republic, and Slovakia. We send emails to amateur football clubs asking them to join for trial practice and send requests to view a rental property. To assess whether taking efforts to learn the local language as a foreigner increases opportunities to socially integrate across these two dimensions, we compare response rates across various text variations.
We will have 6 different message manipulations: (1) local name and proficient local language, (2) foreign name and proficient local language, (3) foreign name, broken local language, and taking local language class, (4) foreign name and broken local language, (5) foreign name, English, and taking a local language class, (5) foreign name, English, and taking a local language class, and (6) foreign name and English. The control group is the text version where a native-sounding name and the native language are used, remaining are 5 different treatment conditions that use a foreign-sounding name.
The messages that employ the broken local language were constructed after consulting online language teachers, native speakers of the language, and chatGPT. We first wrote both the broken and proficient messages in English and then had online language teachers translate the messages into German, Czech, Slovak, and Hungarian. Even for writing the messages in broken English, we consulted an online English language teacher.
|
|
Field
Randomization Method
|
Before
We carry out randomization in Stata to assign football clubs and rental properties to the 6 text variations in a way so that we have balance in the number of observations across each manipulation within each strata. Randomization is stratified at the country-region level (e.g., county level in Hungary). Additionally we will send the emails over Mon - Friday day of the week, so will include day of week as an another strata. The random assignment of observations to treatment will be replicable using the associated seed number. We intend to use the stratarand command in Stata to carry out the randomization.
|
After
We carry out randomization in Stata to assign football clubs and rental properties to the 6 text variations in a way so that we have balance in the number of observations across each manipulation within each strata. Randomization is stratified at the country-region level. For example for rental listings we intend to using cities as that is readily available on the ad. For football, we will use county or district. We will combine small regions together into its own Strata. Additionally we will send the emails over Mon - Friday day of the week, so will include day of week as an another strata. That is the strata will be at the region-day level. The random assignment of observations to treatment will be replicable using the associated seed number. We intend to use the stratarand command in Stata to carry out the randomization.
For countries with a large sample size, we may employ two different organizational accounts for the deployments to limit the potential of being detected as spam. In this case, we will also randomize over observations assigned to the organizational accounts.
|
|
Field
Intervention (Hidden)
|
Before
|
After
In addition to the initial intervention, we intend to do a follow-up intervention, although this will not be randomized. In the follow-up intervention, we will message all those who do not respond within 3 business days a follow-up reminder message that will inform the coach that we are still looking for a team or the landlord/realtor that we are still looking for a property. When sending these follow-ups, we will keep the language proficiency and language signal to be consistent with the initial treatment assignment. For example, if a realtor initially receives an email in English with a language learning signal and does not respond after 3 business days, they will receive a follow-up in English with the language learning signal included. The follow-up intervention may be informative about the effects of preserverance by our treatment conditions. We select 3 business days before doing a follow-up as based on prior research and our pilot testing we find that the overwhelming majority of responses to an email in these contexts are received within the first 3 business days.
For ethical reasons, we will send a message to everyone that responds within 3 business days thanking them for their response and notifying that were are no longer searching for a team or an appartment.
|
|
Field
Secondary Outcomes (End Points)
|
Before
The secondary outcomes are as follows:
1) indicator variable for whether the response is in the local language (versus English),
2) indicator variable for whether the response is negative,
3) indicator variable for whether the response is making further inquiries
4) whether the response is received within 24 hours
|
After
The secondary outcomes are as follows:
1) indicator for opening the email when observed
2) indicator variable for whether the response is unconditionally positive (yes),
3) indicator variable for whether the response is making further inquiries (yes, but...),
4) indicator variable for whether the response has a positive sentiment (yes or yes, but...)
5) indicator variable for whether the response is negative (no),
6) indicator variable for whether the response is received within 24 hours
|
|
Field
Secondary Outcomes (Explanation)
|
Before
For whether the local language is used, a value of 1 is assigned to responses that are written in the country's primary language, and 0 otherwise (e.g., English).
For whether the response is negative, a value of 1 is assigned to the responses where the email sender is in any way denied permission to join for trial practice or denied viewing of a property and is 0 otherwise.
We further distinguish between a positive response and a response with further inquiries. For example, for football coaches: "Where did you play before?" or for landlords: "Where did you find this advertisement?". Such inquiries are coded as 1 and are 0 otherwise for the further inquiries indicator variable.
We will compare the email sent date to the reply date to determine whether a response was delivered within 24 hours
|
After
1) We can observe a measure of whether the email is opened when the emails are deployed using an email plugin. This is 1 if the person opened the email, and 0 otherwise. We may not have this variable for Austria and Hungary as we may not be able to use email plugin in those countries, but rather contact forms associated with the posted listings.
2) The response is positive when the respondent welcomes us to come for practice or see the property unconditionally on other factors. For example, "The property is available, I can show you on Tuesday at 5 pm" or "You can come for practice this Friday at 6 pm" would be an unconditional positive response. We code this as "Yes"
3) We further distinguish between a positive response and a response with further inquiries. For example, for football coaches: "Where did you play before?" or for landlords/realtors: "Where did you find this advertisement?". We code this as a "Yes, but ..." Such inquiries are coded as 1 and are 0 otherwise for the further inquiries indicator variable.
4) We will aggregate our "yes" and "yes, but ..." classification into one positive response variable. We may not have a lot of variation in the data between secondary outcomes 2) and 3), so this aggregating would be more informative.
5) For whether the response is negative, a value of 1 is assigned to the responses where the email sender is in any way denied permission to join for trial practice or denied viewing of a property and is 0 otherwise. For example, "The property is not available" is coded as "No"
6) We will compare the email sent date to the reply date to determine whether a response was delivered within 24 hours
|