Experimental Design Details
We carried out an experiment in a medium-sized Mexican company to assess the effectiveness of an ethics intervention. The company functions as a pawnshop that offers loans while holding collateral like jewelry, for example. If the loan is not paid back the item is sold. The company is a nonprofit organization with the purpose of offering loans at an interest rate lower than profit-based financial institutions. It has 206 branches in 9 different states in Mexico and in Mexico City, and approximately 1.000 employees.
The program we evaluated was the communication of 7 values contained in the ethics code of the company through emails. The company has an ethics code that every new employee must read. After reading the document they must sign a letter in which they agree to abide by the code. The experiment was designed to evaluate if communicating the values contained in the code had an impact on the employees´ understanding of the code.
The company´s code of ethics is called “Values. Document of institutional culture” and is divided into two main parts: The first reviews the institutional philosophy through its mission, vision, principles and values while the second part is the code of conduct that contains eight main points namely, conciliation of interests, use and protection of information, integrity, respect for people and their dignity, selection and relationship with suppliers, responsible use of resources and services, safety and work climate and, last, the Board of Integrity and Good Practices.
The ethical program evaluated consisted of sending an email fortnightly to every employee in the treatment group (we explain how the groups were formed below). The email included an explanation of the nature of a value included in the ethical code along with an example of how that value might be applied in practice. A total of 7 emails were sent as part of the experiment.
The first step in the experiment design was deciding the level of randomization. It is possible to randomize at the individual level, which would amount to randomizing employees. Alternatively, randomization can be at a group level, which in this case it would mean the company´s branches. We decided to randomize at the branch level mainly because we were worried about possible spillovers in the treatment process.
Spillovers are indirect effects of the program on subjects that do not belong to the treatment group. This could happen if randomization was at employee level and an employee who received the program email discusses its contents with an employee who did not. Then the employee might learn about the values contained in the email and receive an indirect effect of the program evaluated. If there are spillovers the results of the randomized experiment might be biased.
Therefore, we conducted a clustered experiment where the clusters are the company’s branches. The experiment by clusters significantly mitigates the risk of having indirect effects since the employees of the company have contact with others mainly within each branch, while communication between employees of different branches is uncommon.
The next step was defining which branches would be in the treatment and control groups, respectively. The random allocation of units (branches) between the control and treatment groups was done through the random function of the Stata statistical software program. This function is based on a seed number from which the random number sequence is generated. To make the randomization process more transparent, it was decided to use the number corresponding to the winning number of the National Lottery of Mexico in the draw prior to the start of the experiment as the seed number. The number is 51609 and corresponds to the prize of the draw with date August 23, 2016.
For the allocation of branches to the treatment and control groups, the branches were arranged in alphabetical order, after which each branch was assigned the random number generated by Stata and based on the seed number 51609. The next step was to order branches based on this number from lowest to highest. Finally, the first half of this list (103 branches) was assigned to the treatment group and the second half (another 103 branches) to the control group.
In order to measure the effect of the program, we used a set of 7 vignettes developed by the human resources department of the company with the help of an external consultant. Each vignette was designed to measure the practical understanding of each of the 7 values that were sent via emails to employees. Each vignette has three different courses of action from which the employee must choose. This design stems from the company´s concern that employees might not understand how to use the code of ethics even if they read it. The assumption is that if the employee really understands the ethics code, he or she will be able to select the course of action in each vignette that is consistent with the values that the specific code relates to. We decided to use a measure developed by the company itself because it is meaningful and useful for the organization itself. Appendix 2 contains the vignettes we used.
We sent the vignettes to all company employees across both the control (those who did not receive emails about ethics) and treatment (those who received emails about ethics) branches, and we analyzed the significance of the EPi variable on the responses to these vignettes, after controlling for socio-economic variables of the employees and the corporate ethical values and location of the branches where they work, in order to evaluate the impact of the implemented ethical program.
The impact we are measuring is the Intention to Treat (ITT) which in this case, is the average effect of sending the email to the employees and not the effect of having the employees reading it. This is because it is not feasible for the researchers to verify that the employees read the emails carefully.
We also included other variables such as demographic data including gender, age, job tenure, education level, marital status and location of the branch and an ethical climate survey to assess if the emails had an impact on how employees understood the code as measured by the vignettes. The ethical climate survey was adapted from the work of Hunt et al (1989).
Another known risk in experimental designs is attrition. It happens when the researcher cannot collect data on the outcomes of the experiment because she/he is not able to reach some of the individuals in the control or treatment groups. In particular, attrition is of concern when it occurs at different rates in the treatment and comparison groups
In order to limit attrition, we sent reminders to employees to complete the survey as did the human resources department periodically. Consequently, 95% of employees answered the survey and of those who did not, the majority (60%) was employees who left the company during the experiment. It is important to note that we had at least one response from every single branch in the company and the results are not compromised by attrition.