Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures

We implement a training intervention for judges who participate in the external evaluation of early-stage venture applications during the semi-finalist stage of a large-scale venture competition program.

The training focuses on "problem identification" as a key criterion for assessing the quality of entrepreneurial ventures. Judges are randomly assigned at the individual level to one of two conditions. In the control condition, judges receive standard instructions that reiterate the competition’s evaluation guidelines. In the treatment condition, judges receive guidance emphasizing the importance of clearly defining and validating the problem the venture seeks to address.

This intervention examines whether training expert evaluators to prioritize problem formulation influences their assessments and the feedback they provide to ventures. Specifically, we test whether a problem-focused evaluation approach leads to greater variation in judges’ recommendations and produces feedback that ventures perceive as more constructive and actionable.

The training intervention is embedded at three key touchpoints in the evaluation process to ensure treated judges receive consistent reinforcement of the problem-centric approach and control judges receive standard onboarding guidelines set by the institution:

1. Email Instructions: Judges receive an email inviting them to log in to the evaluation platform. This email includes a link to a Docsend file containing updated evaluation instructions. Specifically, the second slide of the onboarding materials lays out the "Executive Summary" of what is discussed in the rest of the standard onboarding deck for control judges and the "NEW Guideline: Problem-Identification" for treated judges. In another word, the control slide lays out a summary of the default guidelines for onboarding, and the treatment slide outlines the importance of problem identification in assessing early-stage ventures and provides examples of ventures that solve a problem vs not solving a problem. The rest of the onboarding materials stay the same for both the control and the treatment groups.

2. Pop-up Instructions on the Evaluation Platform: When judges access the evaluation platform and begin evaluations of the first venture assigned, they are shown a pop-up message reiterating the evaluation guidelines. The control group receives message that summarizes the onboarding deck ("Executive Summary") and the treatment group receives the message that explicitly instruct judges on how to assess ventures through a problem-centric lens. Additionally, treated judges are presented with two illustrative examples—one of a venture that has clearly identified and validated a problem and another of a venture with a poorly articulated or unvalidated problem—to help them recognize key differences ("NEW Guideline: Problem-Identification").

3. Evaluation Rubric Guidance: Within the evaluation rubric, control judges have the standard rubric with no tips or hint but the treated judges receive an in-line prompt embedded in blue-font text for each application they review. The prompt states: "Tip: As you evaluate, focus on how well each team has defined and validated the problem they are tackling. Please take a moment to carefully review the instructions at the top of the page."

By integrating training at these three stages, the intervention ensures that judges are continually reminded to focus on problem definition and validation throughout the evaluation process. This design allows us to assess whether problem-focused training influences judges' assessments, their likelihood of providing extreme recommendations, and the quality of the feedback they offer to ventures.

2025-02-12

2025-03-16

The experiment evaluates how a problem-centric training intervention influences judges' assessments of early-stage ventures. The key outcome variables include both quantitative ratings and qualitative feedback provided by judges:

1. Quantitative Ratings (Numerical Scores 1-5 on Evaluation Criteria)

Judges score ventures based on the following dimensions:
- Problem & Customer Definition: The problem is clearly identified, the solution aligns with the identified problem, and the customer
has been well defined and understood.
- Solution/Product/MVP: The team has a solution, product, or MVP that they have tested with their customers.
- Business Model: There is a business/financial model that appears both achievable and sustainable.
- Impact: The idea, once implemented, will have a material impact on the world as measured by directly or
indirectly improving the quality of peoples’ lives.
- Overall Recommendation: Overall, how strongly do you recommend that this startup moves forward to the PIC Finalist round?

2. Qualitative Feedback (Open-Ended Responses)

Judges provide written feedback on venture applications, with specific prompts to capture their evaluation reasoning:
- Strengths: What are 1-2 strengths of this team?
- Areas for Improvement: What are 1-2 areas where this team can improve?
Clarification on Problem Definition: If the judge rates the Problem & Customer Definition below a 3 (on a 5-point scale), they are prompted to explain what could improve it.

3. Optional Connection Question

To measure judges’ willingness to engage further with ventures: Yes, I’m open to connecting with this team. Please share my contact info if they are interested.

Several primary outcomes require constructed measures, detailed as follows:

1. Correlation Between Overall Recommendation and Evaluation Criteria

To assess whether a problem-focused evaluation approach shifts how judges weigh different criteria in their final recommendation, we will examine the correlation between:
Overall Recommendation and individual evaluation scores for:
Problem & Customer Definition
Solution/Product/MVP
Business Model
Impact

2. Feedback Length

We will compare the average word count of qualitative feedback provided by judges in the treatment and control groups.

3. Feedback Quality

- Constructiveness/Concreteness: NLP/LLM-assisted analysis of unstructured text. The comparison will evaluate the specificity of judges' feedback, particularly in relation to problem identification.
- Focus on Problem Definition and Alignment: Again textual analysis would uncover nuances on whether the feedback explicitly discuss whether the problem is well-articulated and validated, and whether it highlights the connection (or misalignment) between the problem identified and the proposed solution.

We will measure completion rates for each judge and time spent by each judge on each venture assigned.

We will also categorize venture teams based on whether their applications fall under the opportunity discovery vs market creation means, using the technologies specified, customer/market defined, early tractions, etc.

We will also categorize judges based on their expertise (VCs, entrepreneurs, operators) and explore potential heterogeneities.

We will also measure venture teams' responses to the feedback judges give via a post-competition survey: Venture teams will report on how they find judges' feedback helpful (or not) and how the additional signal (or noise) affect their decision-making, iteration process, and plan to modify their idea and execution strategy.

We will measure completion rates for each judge and time spent by each judge on each venture assigned.

We will also categorize venture teams based on whether their applications fall under the opportunity discovery vs market creation means, using the technologies specified, customer/market defined, early tractions, etc.

We will also categorize judges based on their expertise (VCs, entrepreneurs, operators) and explore potential heterogeneities.

We will also measure venture teams' responses to the feedback judges give via a post-competition survey: Venture teams will report on how they find judges' feedback helpful (or not) and how the additional signal (or noise) affect their decision-making, iteration process, and plan to modify their idea and execution strategy.

This study evaluates whether training expert judges to prioritize problem identification influences their assessments of early-stage ventures. We conduct a randomized controlled trial within a large-scale U.S.-based venture competition program, where judges evaluate semi-finalist applications.

Judges are randomly assigned at the individual level to either a treatment group or a control group:

Treatment Group: Judges receive additional training emphasizing problem identification and why it is important to focus on solving a "right" problem at the first place when evaluating early-stage ventures.
Control Group: Judges receive standard evaluation/onboarding instructions without any additional emphasis on problem formulation.

The intervention aims to assess whether a problem-focused evaluation approach affects:
Judges’ scoring, as well as the weight they assign to problem definition relative to other evaluation criteria leading to their final recommendations.
The quality, length, and focus of qualitative feedback provided to ventures.
Ventures’ perceptions of feedback usefulness, measured through post-competition surveys.

The study contributes to research on the evaluation process of entrepreneurial innovations by testing whether structured training improves expert assessors’ ability to recognize and support early-stage ventures with well-defined problems.

This study evaluates whether training expert judges to prioritize problem identification influences their assessments of early-stage ventures. We conduct a randomized controlled trial within a large-scale U.S.-based venture competition program, where judges evaluate semi-finalist applications.

Judges are randomly assigned at the individual level to either a treatment group or a control group:

Treatment Group: Judges receive additional training emphasizing problem identification and why it is important to focus on solving a "right" problem at the first place when evaluating early-stage ventures. This training is reinforced through (1) email instructions with linked guidelines, (2) pop-up instructions on the evaluation platform with illustrative examples, and (3) an in-line prompt within the evaluation rubric.
Control Group: Judges receive standard evaluation/onboarding instructions without any additional emphasis on problem formulation.

The intervention aims to assess whether a problem-focused evaluation approach affects:
Judges’ scoring, as well as the weight they assign to problem definition relative to other evaluation criteria leading to their final recommendations.
The quality, length, and focus of qualitative feedback provided to ventures.
Ventures’ perceptions of feedback usefulness, measured through post-competition surveys.

The study contributes to research on the evaluation process of entrepreneurial innovations by testing whether structured training improves expert assessors’ ability to recognize and support early-stage ventures with well-defined problems.

The randomization done in office by a computer (R script).

Individual judges

No

229 Judges

1,488 judge-venture pairs

114 judges control, 115 judges treatment