Experimental Design Details
Experimental Set-up
In this field experiment, we focus on the evaluation of ideas within companies. Therefore, we recruited innovation managers in our partner company and asked them to evaluate real ideas that other employees of our partner firm had proposed. We designed an online evaluation interface with the survey tool Qualtrics, where we will also collect the data. Qualtrics offers several ways to customize the functionality and appearance of a survey, which we used to adjust the survey flow to our needs and the visual appearance to the corporate design of our industrial partner.
Innovation managers come from all over the world and can access the evaluation tool online through a personal link sent to their corporate email address. Each innovation manager is asked to evaluate 48 ideas. The online evaluation consists of three parts: a short introductory welcome screen, the main part for idea evaluation (48 evaluation screens; one for each idea), and a short exit survey of 16 questions. The text of the welcome screen, an example of the evaluation screen and the text of exit questionnaire are given in the appendix.
Ideas are retrieved from the company’s idea management system and their content (idea title and description) is not changed. Each idea is presented on an individual evaluation screen that contains the following:
- Short request to evaluate the idea (“Please evaluate the following idea.”)
- Information on the idea proposer depending on the treatment condition (see “Treatment Conditions”)
- Idea title
- Idea description (between 40 and 999 characters)
- Five questions to rate the ideas (see “Variables and Measurement”)
- Text field for open comments
The innovation managers who evaluate the ideas are not aware that they are experimental subjects. Moreover, idea proposers are not aware that the ideas they submitted to the company’s idea management system are used in our field experiment. Finally, our contact persons in the partner firm are also not aware of the experimental manipulation.
In our communication, we took great care not to reveal our research questions (the effect of blind evaluation and the effect of the relationship between idea proposer and evaluator), our research approach (conducting a field experiment) or the experimental manipulation (blinding the idea proposer). Instead, we communicated that we were conducting a joint study (company and academic researchers) to unlock the intrapreneurial spirit and improve our industrial partner’s idea evaluation.
Treatment Conditions
Innovation managers evaluate ideas under two conditions:
(1) Blind evaluation: The innovation manager receives no information about the idea proposer. Before the idea title and the idea description, no information on the idea proposer is provided and instead “N/A” is displayed.
(2) Non-blind evaluation: The innovation manager receives information about the idea proposer. Before the idea title and the idea description, the idea proposer’s name (first name and last name), organizational unit (abbreviation and full name) and location (city and country) are displayed.
We use a within-subject design implying that each innovation manager evaluates ideas under both conditions (blind and non-blind). Ideas are randomly assigned to one of the conditions with the help of the built-in randomization function in Qualtrics.
Idea Selection and Randomization
Each innovation manager evaluates 48 ideas, which are randomly picked from a larger pool of 412 ideas. The ideas were submitted by employees through a dedicated idea management platform used by our industrial partner. In total, we retrieved 570 ideas that were submitted between February 6, 2019 and October 7, 2019 to the idea management system. We eliminated 15 duplicate ideas and 5 ideas with blank idea descriptions. For our experimental manipulation (blinding) to work credibly, we need early-stage ideas that have not passed through our partner firm and ideally have not be seen by the innovation managers before. To arrive at an appropriate set of ideas, we took the following steps:
(1) We restricted the set of ideas to ideas from the past six months (from April, 8 2019 to October 7, 2019).
(2) We excluded ideas that have already progressed in the internal implementation process, e.g. by having a coach assigned to develop the idea further (6 ideas).
(3) We excluded ideas that contained links or other additional material, e.g. a text document or presentation slides (42 ideas).
(4) We excluded ideas with missing information on the proposer’s gender, organizational unit or location.
(5) We excluded ideas that were proposed by one of the innovation managers who evaluate the ideas.
To ensure that each innovation manager evaluates ideas from proposers with diverse backgrounds, we rely on stratified random assignment of the ideas. Each idea is assigned to one of 20 strata based on the idea proposer’s gender (2 strata) and organizational unit (10 strata). Note that we have not stratified based on the idea proposer’s location. We then randomly pick ideas from each stratum with a built-in function in Qualtrics. The number of ideas picked from each stratum is roughly proportional to the stratum size, although we slightly oversample small strata. For example, we oversample ideas that were proposed by women from organizational units from which only few ideas have been proposed (because a majority of the ideas is proposed by male employees).