Local Language Acquisition and Social Integration

We examine the degree to which acquiring proficiency in the local language can increase social opportunities for foreigners. We will send thousands of requests to join an amateur football club and view a rental property using a foreign-sounding name, randomly varying whether the message is sent in proficient English, broken local language (A1/A2), or proficient local language. Additionally, when English or a broken local language is used, we will have an additional treatment condition which will mention in a subtle way that the person is taking a local language class. We will also have a control group where the message is sent in the native language using a native-sounding name. We will then compare response rates by football coaches and landlords/realtors across the different message variations. Such an experimental design will enable us to study whether signaling efforts to learn the local language and acquire some degree of proficiency in the local language will increase social opportunity in joining a football club and finding a place to live.

2024-10-21

2025-08-31

The primary outcomes are as follows:

1) indicator variable for whether the football coach responds to the email,
2) indicator variable for whether the landlord responds to the email

The outcome variable is coded as 1 if there is any type of response and is 0 otherwise.

The secondary outcomes are as follows:

1) indicator variable for whether the response is in the local language (versus English),
2) indicator variable for whether the response is negative,
3) indicator variable for whether the response is making further inquiries
4) whether the response is received within 24 hours

For whether the local language is used, a value of 1 is assigned to responses that are written in the country's primary language, and 0 otherwise (e.g., English).

For whether the response is negative, a value of 1 is assigned to the responses where the email sender is in any way denied permission to join for trial practice or denied viewing of a property and is 0 otherwise.

We further distinguish between a positive response and a response with further inquiries. For example, for football coaches: "Where did you play before?" or for landlords: "Where did you find this advertisement?". Such inquiries are coded as 1 and are 0 otherwise for the further inquiries indicator variable.

We will compare the email sent date to the reply date to determine whether a response was delivered within 24 hours

The study takes place across 4 countries in central Europe: Austria, Hungary, Czech Republic, and Slovakia. We send emails to amateur football clubs asking them to join for trial practice and send requests to view a rental property. To assess whether taking efforts to learn the local language as a foreigner increases opportunities to socially integrate across these two dimensions, we compare response rates across various text variations.

We will have 6 different message manipulations: (1) local name and proficient local language, (2) foreign name and proficient local language, (3) foreign name, broken local language, and taking local language class, (4) foreign name and broken local language, (5) foreign name, English, and taking a local language class, (5) foreign name, English, and taking a local language class, and (6) foreign name and English. The control group is the text version where a native-sounding name and the native language are used, remaining are 5 different treatment conditions that use a foreign-sounding name.

The messages that employ the broken local language were constructed after consulting online language teachers, native speakers of the language, and chatGPT. We first wrote both the broken and proficient messages in English and then had online language teachers translate the messages into German, Czech, Slovak, and Hungarian.

We will collect the data sequentially and not all at the same time. We will first implement the housing experiment for Czechia, and then do Slovak, Austria, and Hungary soon after. We expect there to be around a 2 week gap in the data collection for each country. This is because that rental market moves quickly, so we need to focus on deploying the messages as soon as the data on all the listings is collected, thus making it difficult to implement all countries at the same time.

We will do the football experiment in the summer when it is the appropriate season for football. A foreigner will typically look for a rental property first, and then explore other social activities. For the football experiment, we will still implement it sequentially one country at a time, but should have a minimal delay between countries as the data on clubs can be collected in advance without becoming outdated.

We carry out randomization in Stata to assign football clubs and rental properties to the 6 text variations in a way so that we have balance in the number of observations across each manipulation within each strata. Randomization is stratified at the country-region level (e.g., county level in Hungary). Additionally we will send the emails over Mon - Friday day of the week, so will include day of week as an another strata. The random assignment of observations to treatment will be replicable using the associated seed number. We intend to use the stratarand command in Stata to carry out the randomization.

The unit of randomization is a football club or an advertisement for a rental property listing. The randomization is carried out within a day of week (message send day) - country-region strata.

No

The randomization is carried out within day-country-region Stratas. We plan to have around 5 x 4 x 5 = 100 strata, with several 100s of observations per strata.

We expect to have between 2500 - 4000 football clubs and between 6000 - 10000 rental property advertisements.

As we have 6 text versions, we expect to have between 400 - 650 observations per treatment arm for the football experiment, and at least 1000 observations per treatment for the housing experiment.

We will compute the MDES using the PowerUp software provided by Dong, N. and Maynard, R. A. (2013). We will compute the MDES to compare the control group (native, proficient local language) to one of the 5 treatment conditions as these are our primary comparisons of interest in our study. Assuming we have 40 blocks (regions) and around 21 observations per block, and assuming constant treatment effects, a significance level of 5, two-tailed test, partial R^2 for the blocks as 0.1, and setting the power to 80%, we achieve an MDES of 0.18 SD. Further including covariates such that the R^2 increases to 0.2 decreases the MDES to 0.17 SD. Assuming an average response rate of 40%, the MDES for the response rate differential is around 8 percentage points when comparing across two groups. Increasing the sample size to a plausible 4000 observations decreases the MDES to 0.14 SD or 7 percentage points. We expect the power to be larger in the housing experiment than the football experiment due to a larger sample size. Our sample size in the housing experiment may be sufficiently large to have a MDES of 0.1 SD or around 5 percentage points. These effect sizes are aligned with prior literature which typically finds large effects when comparing the response rate of football coaches or the callback rate of hiring managers across foreign-sounding names and native-sounding names.

Analysis Plan