Changing the System, Not the Seeker

Last registered on June 14, 2021

Pre-Trial

Trial Information

General Information

Title
Changing the System, Not the Seeker
RCT ID
AEARCTR-0007685
Initial registration date
May 18, 2021

Initial registration date is when the trial was registered.

It corresponds to when the registration was submitted to the Registry to be reviewed for publication.

First published
May 18, 2021, 9:41 AM EDT

First published corresponds to when the trial was first made public on the Registry after being reviewed.

Last updated
June 14, 2021, 4:20 PM EDT

Last updated is the most recent time when changes to the trial's registration were published.

Locations

Primary Investigator

Affiliation
Boston University

Other Primary Investigator(s)

PI Affiliation
University of Oregon
PI Affiliation
GIL - World Bank
PI Affiliation
GIL - World Bank

Additional Trial Information

Status
In development
Start date
2021-05-18
End date
2023-03-31
Secondary IDs
Abstract
We will assess the effect of multiple organization-level treatments on the propensity of investors to invest in a startup. We will assess this variable in multiple ways including evaluation on a scale, and more qualitative evaluation.
External Link(s)

Registration Citation

Citation
Goldstein, Markus et al. 2021. "Changing the System, Not the Seeker." AEA RCT Registry. June 14. https://doi.org/10.1257/rct.7685-1.1
Sponsors & Partners

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information
Experimental Details

Interventions

Intervention(s)
Researchers have designed three interventions which an investment organization making investments in early-stage startups will apply to treatment investors as they evaluate startups.
1) Prompting consistent inquiry
2) Evaluating demonstrated competence
3) Sharing prior evaluations
Intervention Start Date
2021-05-25
Intervention End Date
2022-09-30

Primary Outcomes

Primary Outcomes (end points)
Dependent Variable: The dependent variable is propensity to invest in a startup.
Primary Outcomes (explanation)
Dependent Variable: The dependent variable is Yp – the propensity to invest in a startup by the paired program. The four paired programs will take place in four geographic regions and include entrepreneurs from across those regions: Sub-Saharan Africa, India, MENA and Latin America. This will result in a total sample of eight programs - four treatment programs and four control programs – with one treatment and one control program in each location. We use fixed effects for the paired program in all regressions. (We will only include fixed effects for the investor in pooled regressions when we join up the sample with the professional investors.)
We will measure the dependent variable using 4 methods.
For the first treatment – prompting consistent enquiry – our primary dependent variable will be qualitative, following Kanze et al. (2018).
For the second treatment – evaluating demonstrated competence – our primary dependent variable will be scales, inspired by Clingingsmith and Shane’s (2018) dependent variable.
1. Scales: Each trainee investor will evaluate each startup on a scale. The baseline evaluation takes place on a 6-point scale. Thereafter, evaluators will use a 24-point scale (control group) and a 32-point scale (treatment group). All scale evaluations are normalized by the program using a z-score.
2. Binary: Each trainee investor will know that the top 2 rated startups will receive investment. Therefore, each trainee investor will carefully consider who they place in the top 2.
3. Qualitative: Each trainee investor will be asked what additional information they need from the startup. Trainee investors will also ask for additional information in conversations, and combined, this will form a secondary dependent variable. All responses will be coded as “promotion-focused” or “prevention-focused”. We will assess the proportion of promotion vs. prevention-focused questions (Kanze et al. 2018).
4. Performance-reward bias: Normalized scale (Male normalized qualitative proportion – Female normalized qualitative proportion) – see Castilla (2008). Intuitively, if you get the same scale rank, what is the difference between the qualitative score by the gender of the entrepreneur?

Secondary Outcomes

Secondary Outcomes (end points)
Secondary Outcomes (explanation)

Experimental Design

Experimental Design
We will assess the effect of multiple treatments on the propensity of investors to invest in a startup. We will assess this variable in multiple ways including evaluation on a scale, and more qualitative evaluation.
Experimental Design Details
TRAINEE INVESTORS
We will assess the main hypothesis using the regressions:
Y_ip=α_1 y_0p±α_2 F_ip±α_3 〖T1〗_ip ±〖 α〗_4 〖〖T1×F〗_ip±X_j±L_p±ε_ip〗_
Y_ip=〖β_1 y_0p± β〗_2 F_ip±β_3 〖T2〗_ip ±〖 β〗_4 〖〖T2×F〗_ip±X_j±L_p±ε_ip〗_
This is an ANCOVA regression, which we plan to use to increase our statistical power, following McKenzie (2012). Y0 is the baseline measurement of the dependent variable – before the program starts. i is the stage of measurement – the ranking variable over three time periods. We will measure the dependent variable three times during each program and pool the three measurements. (We will also measure the dependent variable at specific points in time, which will give us less statistical power, but will produce directional effects over time, which we also plan to analyze.)
Dependent Variable: The dependent variable is Yp – the propensity to invest in a startup by the paired program. The four paired programs will take place in four geographic regions and include entrepreneurs from across those regions: Sub-Saharan Africa, India, MENA and Latin America. This will result in a total sample of eight programs - four treatment programs and four control programs – with one treatment and one control program in each location. We use fixed effects for the paired program in all regressions. (We will only include fixed effects for the investor in pooled regressions when we join up the sample with the professional investors.)
We will measure the dependent variable using 4 methods.
For the first treatment – prompting consistent enquiry – our primary dependent variable will be qualitative, following Kanze et al. (2018).
For the second treatment – evaluating demonstrated competence – our primary dependent variable will be scales, inspired by Clingingsmith and Shane’s (2018) dependent variable.
Scales: Each trainee investor will evaluate each startup on a scale. The baseline evaluation takes place on a 6-point scale. Thereafter, evaluators will use a 24-point scale (control group) and a 32-point scale (treatment group). All scale evaluations are normalized by the program using a z-score.
Binary: Each trainee investor will know that the top 2 rated startups will receive investment. Therefore, each trainee investor will carefully consider who they place in the top 2.
Qualitative: Each trainee investor will be asked what additional information they need from the startup. Trainee investors will also ask for additional information in conversations, and combined, this will form a secondary dependent variable. All responses will be coded as “promotion-focused” or “prevention-focused”. We will assess the proportion of promotion vs. prevention-focused questions (Kanze et al. 2018).
Performance-reward bias: Normalized scale (Male normalized qualitative proportion – Female normalized qualitative proportion) – see Castilla (2008). Intuitively, if you get the same scale rank, what is the difference between the qualitative score by the gender of the entrepreneur?

Gender. In the model, F is our gender variable – a female-led company. In our context, we define a female-led company as the founder that the investor interacts with identifies as female. We made this choice because other researchers have found the gender of the person pitching a startup to be meaningful to evaluations of that startup (e.g., Brooks at al. 2015, Kanze et al. 2018). Because our interventions focus on changing the organizational evaluation process after an interaction with a founder, we find this the most apt definition.
In practice, we know that many teams have multiple co-founders, who may be present during the program. To implement our definition, we have three measures, but the first is our preferred measure:
Binary: Was the female founder present in the interaction with the investor?
Binary: Does the company have a female founder on their founding team sheet for investors?
Scale (percentage): How much was the company represented by a female founder in interactions? We will note which founder speaks in each workshop during the program.

Intervention 1: For intervention 1 on prompting confident inquiry, we will have a sample size of at least 216 investor decisions on startups led by a female founder (within 720 total investor decisions on all startups). We will assess the effect of this treatment using a between subject design. Using 1 pre measure and 3 post measures in ANCOVA requires a sample size of 212 total decisions on females (across treatment and control) for a total power calculation of 0.85. We therefore estimate that we will see a significant positive effect of the treatment on the qualitative dependent variable primarily (but we will look for positive and significant effects on other DVs too). We expect a positive and significant sign on a4 on all DVs.
Intervention 2: For intervention 2 on evaluating using demonstrated competence, we will have the same sample size as intervention 1, with the same power calculations. We therefore estimate that we will see a significant positive effect of the treatment on the scale dependent variable primarily (but we will look for positive and significant effects on other DVs too). We expect a positive and significant sign on b4 on all DVs.
Intervention 3: TBD.
Control variables. In the equations where we do not use investor fixed effects, we will include potential control variables at the investor level including the gender of the investor (Greenberg & Mollick, 2018), and experience of the investor (Clingingsmith & Shane, 2018), which we will measure by a binary variable: having participated in another accelerator, and continuous variables: years of experience in the market.
In the equations where we do not use startup fixed effects, we will include control variables for the startups including:
Evidence of their underlying quality: whether they were accepted by Connect or waitlisted (binary); their average score in Connect due diligence (1-4 continuous)
Evidence of their maturity: whether they were in the most popular geographic market for the program from the finalist group i.e., Egypt for MENA (binary); number of founders (categorical); total employees (categorical); the log of funds raised (continuous).

We have additional hypotheses and data drawing from professional investors - attached.
Randomization Method
randomization done in office by a computer
Randomization Unit
individual investor

Planned Number of Clusters
Each trainee investor is a cluster and will make at least 9 decisions.
Each professional investor is a cluster and will make at least 3 decisions.
Planned Number of Observations
At least 1,500 individual investor decisions, from at least 200 individual investors.
Sample size (or number of clusters) by treatment arms *
For treatment 1 and treatment 2, I will have a sample size of at least 1,500 individual investor decisions, and 456 on female founders. These stem from at least 200 individual investors.
Power calculation: Minimum Detectable Effect Size for Main Outcomes
The minimum detectable effect size for the ANCOVA calculations is 0.225.
Was the treatment clustered?
Yes

Experiment Characteristics

Sample size: planned number of clusters
200 investors
Sample size: planned number of observations
1,500
Sample size (or number of clusters) by treatment arms
100 investors treatment, 100 investors control
Minimum detectable effect size for main outcomes (accounting for sample design and clustering)
The minimum detectable effect size for the ANCOVA calculations is 0.225.
IRB

Institutional Review Boards (IRBs)

IRB Name
Boston University IRB
IRB Approval Date
2020-08-21
IRB Approval Number
5690X
Analysis Plan

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Post-Trial

Post Trial Information

Study Withdrawal

There is information in this trial unavailable to the public. Use the button below to request access.

Request Information

Intervention

Is the intervention completed?
No
Data Collection Complete
Data Publication

Data Publication

Is public data available?
No

Program Files

Program Files
Reports, Papers & Other Materials

Relevant Paper(s)

Reports & Other Materials