Mentoring in the Workplace: Treatment versus Selection

We analyze the effectiveness of a workplace mentoring program where employees opt-into mentoring or are randomly assigned a mentor. More details are provided in the design field.

2019-05-27

2019-12-20

Log revenue-per-call (RPC), an indicator for worker turnover, log completed tenure, the firm’s internal adherence to schedule measure (e.g. time spent working whilst at work), and the firm’s internal engagement metrics (online surveys asking for willingness to recommend employment at firm, comfort with leadership, etc.).

Agent’s weekly RPC is a measure of sales productivity that removes demand variation outside of the worker’s control. RPC is the primary productivity measure used by the firm, combining both agent’s firm-specific knowledge and their individual effort. Worker turnover measures whether the interventions changed the agents’ propensity to leave the firm. Log of completed tenure is a different measure of retention that has been used in the prior literature and the attendance measure provides an adjacent measure of agent effort. Finally, engagement measures are hypothesized to be forward looking measures of productivity.

Please see the hidden field.

Seasoned sales agents are invited to apply as internal mentors to incoming recruits (the firm “qualifies” mentors as having sufficient sales experience). New mentorship opportunities are periodically announced, and prior mentors are permitted to re-enter the mentor pool. The firm communicates that serving as a mentor is a useful first step to being considered for a managerial position. New mentors complete a survey asking them about their personality, interests, work preferences, and values. Mentors are randomly assigned with probability ½ to receive a set of instructions emphasizing that mentoring is about teaching protégés how to do the job. The remaining mentors receive instructions emphasizing that mentoring is about providing protégés support.

Sales agents are hired in batches (cohorts). Newly hired sales agents complete two weeks of training, primarily in a classroom or listening in on other agents’ sales calls. New agents then complete the same personality and preference survey that mentors take. At the end of their two-week training, each cohort of agents is eligible for randomization into a mentoring treatment arm. Any mentoring relationship commences as soon as the agent completes their training. The randomization procedure is as follows
Cohort Level Randomization: The initial level of randomization is cohorts of new hires (potential protégés). Each cohort (a group of new hires who are joining the firm at the same time, are in the same training group, and will be working in the same sales division and office location) will be randomized into one of two conditions: Random mentoring or Voluntary mentoring. 40% of the cohorts will be in the Random mentoring group and 60% of the cohorts will be in the Voluntary group.

Within Cohort Randomization:
For cohorts in Random mentoring, new hires will receive a mentor with probability 1/2. This will be communicated privately between sales floor staff and the individual workers. Agents in Random cohorts who do not receive a mentor will not receive communication regarding the program.

For cohorts in Voluntary mentoring, sales floor staff verbally explain the firm’s mentorship program, answer questions, and provide each agent a confidential ballot where they can decide whether or not to enter a lottery which randomly determines whether the agent is allocated a randomly assigned mentor, or no mentor at all. Of the agents who enter the lottery, approximately 1/2 will be assigned a mentor. Agents who choose not to be mentored will never be assigned a mentor.

Compliance Tracking: The firm’s training staff will track whether mentors and protégés meet. This tracking will be aided by worksheets. Upon completion of the worksheets, the firm will reward “kudos” points that can be accumulated to purchase items from the company store.
As mentioned earlier, mentors may participate more than once, however they will never have more than one protégé at a time.

Eligible protégés and mentors will each take an electronic survey at the end of the formal program. The survey for protégés will ask about the protégé’s initial excitement when told about the mentoring program, their perceived engagement with their mentor, and an estimate of the effectiveness of mentoring. This question will be phrased as: “What was your average RPC last week? What do you think your average RPC would have been had you not been working with a mentor?”

The survey for mentors will ask about the protégé’s enthusiasm for the mentorship program and an estimate of the mentor’s perceived treatment effect on the protégé. This question will be phrased as: “If your protégé had not received mentoring, his/her RPC would have been [40% lower --- slider --- 40% higher]. Note that numbers greater than zero mean that mentoring was not effective for improving protégé performance. Please be candid, as your responses will not be shared with management.”



Edit June 4, 2019:
To assess the potential for spillovers, we have revisited the design in consultation with the company such that there will be "hold out" cohorts for one division-office who never receive mentoring. Any cohorts/individuals who are switching brands also will be held-out. Work from Home cohorts will also present a possible "hold out" group for comparison and all cohorts in a smaller third office (which no longer exist, but for whom historical data is available) were "hold out" cohorts who knew nothing about mentoring.

A "sentiment survey" will be administered to all agents in their 5th week on the sales floor. This will be one week after mentored agents finish hiring. We will gather information on their feelings towards the onbaording process and ask questions, common in the literature, to solicit their sentiment towards the firm, their perceptions of their ability, their enthusiasm about the job, etc. We will use this survey to test for spillovers based on survey responses.

Randomization done by computer. Participants will be informed if randomized in.

Clustered randomization of cohorts in a first level, with individual randomization within the cohort. See design details.

Yes

The exact sample size is stochastic and depends on the firm’s actual hiring. We have 46 planned clusters.

In one office, the firm has projected 269 new hires in 22 cohorts. There are 350 new hires in 24 cohorts projected in the second office.

Please see design field.

Using pre-intervention data to estimate the intra-class correlation coefficient and residual variation, the minimum detectable effect size for log RPC between those randomized into and out of mentoring is 0.07.