Performance in the New Era of Work: A Study of Employers' Perceptions of Employee Performance in Remote and In-Person Environments

In this study, participants act as employers tasked with evaluating work products completed by employees under either remote or in-person conditions. Each participant reviews eight work samples, all of which are of similar quality, ensuring that differences in evaluation stem from perceptions rather than actual performance disparities. Importantly, participants are informed of the work mode in which each product was completed. By analyzing their evaluations, this study aims to determine whether employers exhibit a preference for in-person work, despite the consistency in work quality.

Participants in this study take on the role of an "employer" responsible for evaluating work completed by "employees" who were assigned to either remote or in-person conditions. Employees were given two real-world site selection challenges and tasked with generating innovative and actionable solutions. Each participant will review eight work products—each created by a different employee under one of the two work modes.

Critically, the quality of all work products is controlled to be similar, ensuring that any differences in employer evaluations arise from perceived biases rather than actual disparities in performance. Participants will be explicitly informed of the work mode in which each product was completed before making their assessments. They will rate each work product based on criteria such as creativity, feasibility, and overall effectiveness, as well as provide qualitative feedback. By comparing evaluations across work modes, this study aims to investigate whether employers demonstrate a preference for in-person work, even when objective work quality remains constant.

2025-03-13

2025-03-19

Employer Evaluations (A score) of Work Product: Ratings of the overall quality of each work product, including clarity, feasibility, and relevance to the site selection challenge.

The key outcome variables of interest in this experiment are derived from employer evaluations of each work product based on the following metrics:

Number of Ideas – The total count of unique, non-repetitive ideas provided in the work product. (Response Type: Text entry)

Variability of Ideas – The extent to which the work product presents diverse approaches and perspectives. (Response Type: Slider: 0 = no variability, 10 = extremely varied)

Consistency in Quality of Ideas – The degree to which the ideas are well-developed and maintain a consistent level of quality throughout the work product. (Response Type: Slider: 0 = very inconsistent, 10 = highly consistent)

Specificity – The level of detail and precision in the ideas, where more specific and detailed suggestions are considered more actionable. (Response Type: Slider: 0 = very vague, 10 = highly specific)

Feasibility – The likelihood that the proposed ideas can be realistically implemented using existing resources, technology, and strategies. (Response Type: Slider: 0 = not feasible, 10 = highly feasible)

Novelty – The originality and innovativeness of the proposed ideas, focusing on their uniqueness in addressing unmet needs. (Response Type: Slider: 0 = not novel, 10 = highly novel)

Demand – The extent to which the ideas align with current market trends and meet existing or emerging demands. (Response Type: Slider: 0 = no demand, 10 = high demand)

Business Value – The potential for the ideas to create utility for a commercial organization by contributing to strategic goals, profitability, or competitive advantage. (Response Type: Slider: 0 = no value, 10 = high value)

Experimental Design
This study employs a between-subjects experimental design to investigate whether employers exhibit a preference for in-person work over remote work when evaluating employee-generated work products. Participants, acting as "employers," will assess eight work products created by real participants ("employees") who completed site selection tasks either remotely or in person.

Key Design Features:

Participant Recruitment: The study will recruit 40 participants from Connect Research and 10 participants from a commercial real estate company, ensuring a diverse sample with both general and industry-specific perspectives.
Controlled Work Quality: The work products presented to participants will be of similar quality, ensuring that differences in evaluations arise from perceived biases rather than actual disparities in performance.
Work Mode Disclosure: Employers will be explicitly informed whether each work product was completed in-person or remotely before making their assessments.
Evaluation Metrics: Employers will evaluate work products based on predefined criteria, including the number of unique ideas, variability, consistency in quality, specificity, feasibility, novelty, demand, and business value. Ratings will be collected using a combination of text entry and slider scales.
Randomized Work Product Order: The order in which participants receive work products will be randomized to control for order effects.

Clarification of Random Assignments
This study follows a within-subjects design, where each participant evaluates work products from both treatment conditions (remote and in-person). However, the employees (work product creators) were randomly assigned to either the remote or in-person condition when completing their tasks.

Random Assignment of Employees:

Employees who generated the work products were randomly assigned to complete their tasks either remotely or in person.
This ensures that differences in work mode are not due to self-selection but rather due to experimental manipulation.
Employer Evaluations (Participants in the Study):

Participants (employers) do not experience a separate treatment and control group but instead evaluate work products from both conditions.
Each participant assesses 8 work products (4 remote, 4 in-person) presented in a randomized order to control for order effects.
Treatment and Control Groups Defined by Work Products:

Treatment Group: Work products created by employees who completed their tasks remotely.
Control Group: Work products created by employees who completed their tasks in person.

This study employs a randomized work product order design to control for order effects and minimize biases in employer evaluations. The randomization method is implemented as follows:

Work Product Order Randomization:

Each participant will evaluate eight work products (four completed in-person and four completed remotely).
The order of presentation of these work products will be randomized for each participant to prevent any systematic biases based on sequence effects.
Work Mode Distribution:

Participants will review an equal number of remote and in-person work products (4 remote, 4 in-person).
The randomization process ensures that no participant sees all remote or all in-person work products consecutively, reducing potential anchoring or contrast effects.

The unit of randomization in this study is at the individual participant level (i.e., each employer evaluating work products)

No

0

50 employers

200 evaluations of remote work products (50 participants × 4 remote work products each)
200 evaluations of in-person work products (50 participants × 4 in-person work products each)

Effect Size (Cohen’s d): 0.40 Difference in Ratings: ~0.4 points on a 0-10 scale Percentage Difference: 4% change relative to the 10-point scale Thus, this study is powered to detect at least a 4% difference in employer evaluations between remote and in-person work products, assuming standard deviations remain stable.