Local Language Acquisition and Social Integration

We examine the degree to which acquiring proficiency in the local language can increase social opportunities for foreigners. We will send thousands of requests to join an amateur football club and view a rental property using a foreign-sounding name, randomly varying whether the message is sent in proficient English, broken local language (A1/A2), or proficient local language. Additionally, when English or a broken local language is used, we will have an additional treatment condition which will mention in a subtle way that the person is taking a local language class. We will also have a control group where the message is sent in the native language using a native-sounding name. We will then compare response rates by football coaches and landlords/realtors across the different message variations. Such an experimental design will enable us to study whether signaling efforts to learn the local language and acquire some degree of proficiency in the local language will increase social opportunity in joining a football club and finding a place to live.

2024-10-21

2025-08-31

The primary outcomes are as follows:

1) indicator variable for whether the football coach responds to the email,
2) indicator variable for whether the realtor or landlord (listing poster) responds to the email

The outcome variable is coded as 1 if there is any type of response and is 0 otherwise. We wait at least 3 business days to allow sufficient time for the realtor/landlord or coach before recording the response.

The secondary outcomes are as follows:
1) indicator for opening the email when observed
2) indicator variable for whether the response is unconditionally positive (yes),
3) indicator variable for whether the response is making further inquiries (yes, but...),
4) indicator variable for whether the response has a positive sentiment (yes or yes, but...)
5) indicator variable for whether the response is negative (no),
6) indicator variable for whether the response is received within 24 hours

1) We can observe a measure of whether the email is opened when the emails are deployed using an email plugin. This is 1 if the person opened the email, and 0 otherwise. We may not have this variable for Austria and Hungary as we may not be able to use email plugin in those countries, but rather contact forms associated with the posted listings.

2) The response is positive when the respondent welcomes us to come for practice or see the property unconditionally on other factors. For example, "The property is available, I can show you on Tuesday at 5 pm" or "You can come for practice this Friday at 6 pm" would be an unconditional positive response. We code this as "Yes"

3) We further distinguish between a positive response and a response with further inquiries. For example, for football coaches: "Where did you play before?" or for landlords/realtors: "Where did you find this advertisement?". We code this as a "Yes, but ..." Such inquiries are coded as 1 and are 0 otherwise for the further inquiries indicator variable.

4) We will aggregate our "yes" and "yes, but ..." classification into one positive response variable. We may not have a lot of variation in the data between secondary outcomes 2) and 3), so this aggregating would be more informative.

5) For whether the response is negative, a value of 1 is assigned to the responses where the email sender is in any way denied permission to join for trial practice or denied viewing of a property and is 0 otherwise. For example, "The property is not available" is coded as "No"

6) We will compare the email sent date to the reply date to determine whether a response was delivered within 24 hours

The study takes place across 4 countries in central Europe: Austria, Hungary, Czech Republic, and Slovakia. We send emails to amateur football clubs asking them to join for trial practice and send requests to view a rental property. To assess whether taking efforts to learn the local language as a foreigner increases opportunities to socially integrate across these two dimensions, we compare response rates across various text variations.

We will have 6 different message manipulations: (1) local name and proficient local language, (2) foreign name and proficient local language, (3) foreign name, broken local language, and taking local language class, (4) foreign name and broken local language, (5) foreign name, English, and taking a local language class, (5) foreign name, English, and taking a local language class, and (6) foreign name and English. The control group is the text version where a native-sounding name and the native language are used, remaining are 5 different treatment conditions that use a foreign-sounding name.

The messages that employ the broken local language were constructed after consulting online language teachers, native speakers of the language, and chatGPT. We first wrote both the broken and proficient messages in English and then had online language teachers translate the messages into German, Czech, Slovak, and Hungarian. Even for writing the messages in broken English, we consulted an online English language teacher.

Not available

We carry out randomization in Stata to assign football clubs and rental properties to the 6 text variations in a way so that we have balance in the number of observations across each manipulation within each strata. Randomization is stratified at the country-region level. For example for rental listings we intend to using cities as that is readily available on the ad. For football, we will use county or district. We will combine small regions together into its own Strata. Additionally we will send the emails over Mon - Friday day of the week, so will include day of week as an another strata. That is the strata will be at the region-day level. The random assignment of observations to treatment will be replicable using the associated seed number. We intend to use the stratarand command in Stata to carry out the randomization.

For countries with a large sample size, we may employ two different organizational accounts for the deployments to limit the potential of being detected as spam. In this case, we will also randomize over observations assigned to the organizational accounts.

The unit of randomization is a football club or an advertisement for a rental property listing. The randomization is carried out within a day of week (message send day) - country-region strata.

No

The randomization is carried out within day-country-region Stratas. We plan to have around 5 x 4 x 5 = 100 strata, with several 100s of observations per strata.

We expect to have between 2500 - 4000 football clubs and between 6000 - 10000 rental property advertisements.

As we have 6 text versions, we expect to have between 400 - 650 observations per treatment arm for the football experiment, and at least 1000 observations per treatment for the housing experiment.

We will compute the MDES using the PowerUp software provided by Dong, N. and Maynard, R. A. (2013). We will compute the MDES to compare the control group (native, proficient local language) to one of the 5 treatment conditions as these are our primary comparisons of interest in our study. Assuming we have 40 blocks (regions) and around 21 observations per block, and assuming constant treatment effects, a significance level of 5, two-tailed test, partial R^2 for the blocks as 0.1, and setting the power to 80%, we achieve an MDES of 0.18 SD. Further including covariates such that the R^2 increases to 0.2 decreases the MDES to 0.17 SD. Assuming an average response rate of 40%, the MDES for the response rate differential is around 8 percentage points when comparing across two groups. Increasing the sample size to a plausible 4000 observations decreases the MDES to 0.14 SD or 7 percentage points. We expect the power to be larger in the housing experiment than the football experiment due to a larger sample size. Our sample size in the housing experiment may be sufficiently large to have a MDES of 0.1 SD or around 5 percentage points. These effect sizes are aligned with prior literature which typically finds large effects when comparing the response rate of football coaches or the callback rate of hiring managers across foreign-sounding names and native-sounding names.