Judging the Problem: A Problem-Centric Approach to Evaluating Early-Stage Ventures

We implement a training intervention for judges who participate in the external evaluation of early-stage venture applications during the semi-finalist stage of a large-scale venture competition program.

The training focuses on "problem identification" as a key criterion for assessing the quality of entrepreneurial ventures. Judges are randomly assigned at the individual level to one of two conditions. In the control condition, judges receive standard instructions that reiterate the competition’s evaluation guidelines. In the treatment condition, judges receive guidance emphasizing the importance of clearly defining and validating the problem the venture seeks to address.

This intervention examines whether training expert evaluators to prioritize problem formulation influences their assessments and the feedback they provide to ventures. Specifically, we test whether a problem-focused evaluation approach leads to greater variation in judges’ recommendations and produces feedback that ventures perceive as more constructive and actionable.

2025-02-12

2025-03-16

The experiment evaluates how a problem-centric training intervention influences judges' assessments of early-stage ventures. The key outcome variables include both quantitative ratings and qualitative feedback provided by judges:

1. Quantitative Ratings (Numerical Scores 1-5 on Evaluation Criteria)

Judges score ventures based on the following dimensions:
- Problem & Customer Definition: The problem is clearly identified, the solution aligns with the identified problem, and the customer
has been well defined and understood.
- Solution/Product/MVP: The team has a solution, product, or MVP that they have tested with their customers.
- Business Model: There is a business/financial model that appears both achievable and sustainable.
- Impact: The idea, once implemented, will have a material impact on the world as measured by directly or
indirectly improving the quality of peoples’ lives.
- Overall Recommendation: Overall, how strongly do you recommend that this startup moves forward to the PIC Finalist round?

2. Qualitative Feedback (Open-Ended Responses)

Judges provide written feedback on venture applications, with specific prompts to capture their evaluation reasoning:
- Strengths: What are 1-2 strengths of this team?
- Areas for Improvement: What are 1-2 areas where this team can improve?
Clarification on Problem Definition: If the judge rates the Problem & Customer Definition below a 3 (on a 5-point scale), they are prompted to explain what could improve it.

3. Optional Connection Question

To measure judges’ willingness to engage further with ventures: Yes, I’m open to connecting with this team. Please share my contact info if they are interested.

Several primary outcomes require constructed measures, detailed as follows:

1. Correlation Between Overall Recommendation and Evaluation Criteria

To assess whether a problem-focused evaluation approach shifts how judges weigh different criteria in their final recommendation, we will examine the correlation between:
Overall Recommendation and individual evaluation scores for:
Problem & Customer Definition
Solution/Product/MVP
Business Model
Impact

2. Feedback Length

We will compare the average word count of qualitative feedback provided by judges in the treatment and control groups.

3. Feedback Quality

- Constructiveness/Concreteness: NLP/LLM-assisted analysis of unstructured text. The comparison will evaluate the specificity of judges' feedback, particularly in relation to problem identification.
- Focus on Problem Definition and Alignment: Again textual analysis would uncover nuances on whether the feedback explicitly discuss whether the problem is well-articulated and validated, and whether it highlights the connection (or misalignment) between the problem identified and the proposed solution.

We will measure completion rates for each judge and time spent by each judge on each venture assigned.

We will also categorize venture teams based on whether their applications fall under the opportunity discovery vs market creation means, using the technologies specified, customer/market defined, early tractions, etc.

We will also categorize judges based on their expertise (VCs, entrepreneurs, operators) and explore potential heterogeneities.

We will also measure venture teams' responses to the feedback judges give via a post-competition survey: Venture teams will report on how they find judges' feedback helpful (or not) and how the additional signal (or noise) affect their decision-making, iteration process, and plan to modify their idea and execution strategy.

We will measure completion rates for each judge and time spent by each judge on each venture assigned.

We will also categorize venture teams based on whether their applications fall under the opportunity discovery vs market creation means, using the technologies specified, customer/market defined, early tractions, etc.

We will also categorize judges based on their expertise (VCs, entrepreneurs, operators) and explore potential heterogeneities.

We will also measure venture teams' responses to the feedback judges give via a post-competition survey: Venture teams will report on how they find judges' feedback helpful (or not) and how the additional signal (or noise) affect their decision-making, iteration process, and plan to modify their idea and execution strategy.

This study evaluates whether training expert judges to prioritize problem identification influences their assessments of early-stage ventures. We conduct a randomized controlled trial within a large-scale U.S.-based venture competition program, where judges evaluate semi-finalist applications.

Judges are randomly assigned at the individual level to either a treatment group or a control group:

Treatment Group: Judges receive additional training emphasizing problem identification and why it is important to focus on solving a "right" problem at the first place when evaluating early-stage ventures.
Control Group: Judges receive standard evaluation/onboarding instructions without any additional emphasis on problem formulation.

The intervention aims to assess whether a problem-focused evaluation approach affects:
Judges’ scoring, as well as the weight they assign to problem definition relative to other evaluation criteria leading to their final recommendations.
The quality, length, and focus of qualitative feedback provided to ventures.
Ventures’ perceptions of feedback usefulness, measured through post-competition surveys.

The study contributes to research on the evaluation process of entrepreneurial innovations by testing whether structured training improves expert assessors’ ability to recognize and support early-stage ventures with well-defined problems.

Not available

The randomization done in office by a computer (R script).

Individual judges

No

229 Judges

1,488 judge-venture pairs

114 judges control, 115 judges treatment