Experimental Design
Experimental Design (Public)*: Our approach to estimating the causal impact of the program follows these steps.
From the pool of applications (N = 78), we:
a) exclude proposals with the lowest potential (Low potential),
b) secure grants for the most promising projects (High potential), and
c) randomize grants among projects with intermediate potential.
Projects with intermediate potential often involve uncertainty. The project pool also confounds heterogeneous risk-return profiles with similar expected value, making it unreliable to base selection decisions solely on judges' scores. Randomly assigning grants to these projects is hence more efficient and fairer.
An independent evaluation agency grades each submitted proposal and categorizes them as High, Intermediate, or Low potential. Projects are scored on 11 criteria, including innovativeness and budget coherence, using a scale from 0 to 3. Projects scoring 0 or 1 in five or more criteria qualify as very low potential and are rejected (N=19). Projects scoring 3 in eight or more criteria receive grants directly (N=3). The remaining projects (N=56) enter a pool for random grant allocation. Within this pool, projects are divided into four subgroups based on projected investment. One project from each group is selected by lottery until funds run out.
The selection process described above was announced publicly and outlined in Decree 57/2023, approved on May 24, 2023, and published in the Diario Oficial de Extremadura (DOE nº 77) on April 22, 2024.
The random assignment of projects to Treatment and Control was implemented in December 2024, in Extremadura, under the supervision of a Notary. All the companies in the intermediate group were divided into four groups based on the amount of the requested subsidy. Since there were 56 companies in this group, each pool contained 14 companies. The order of the pools was randomized, and then one company was drawn from each pool until the total amount of the aid program was exhausted. In total, 22 projects from the group of intermediate potential projects were assigned to Treatment and 34 to Control. The total amount of the funds allocated was EUR 3.093,080, with an average allocated grant of EUR 137,941 [Max = EUR 200,000, Median = EUR 135,348, Min = EUR 52,734]. The funds of the grants assigned to the treatment group are expected to be disbursed from February to late April or early May 2025.
We will evaluate the program’s impact on firms with intermediate potential projects, focusing on outcomes such as innovation, technology adoption, exports, employment, sales, investments and profits. Using ANCOVA, we will measure the after the disbursement of funds, controlling for baseline values of the outcomes of interest, and stratification variables. Survey attrition at baseline was 7%. We aim to complement survey data with data from other sources, such as from the Banco de España and Orbis. Data may also be collected after longer periods of time, to study long-term effects.
Besides the standard frequentist estimation mentioned above, we will implement a Bayesian impact evaluation, incorporating the project-based prior beliefs of evaluators and program-based prior beliefs of relevant stakeholders about the program’s expected impact. We aim to determine whether the lack of estimated impact of R&D programs reported in the literature using frequentist methods indicates that there is truly no impact or whether the frequentist approach is inadequate for estimating the impact of interventions in highly uncertain settings, with highly heterogeneous risk-return project profiles, where the factors driving a project's success may be inversely valued by policymakers.
We extend the method proposed by Iacovone et al. (2023), which uses policymakers' and academics' priors about a program, in the following way. In addition to integrating stakeholders' priors (e.g., policymakers, academics) regarding the overall effectiveness of the R&D program, we elicit the full subjective distribution of evaluators’ prior beliefs about each project under treatment and control status. This allows us to leverage the unique insights evaluators have about the individual projects they review in each potential state, as they are expected to hold more precise priors about the risk-return profile than prior beliefs constructed with aggregate information from the pool of projects as it is often done. We collect bivariate priors, capturing both (i) the probability of a project’s success and (ii) the distribution of its potential effects, both with and without the grant. This approach is particularly relevant in environments where multiple projects may have similar expected returns yet exhibit substantial heterogeneity in risk-return profiles.
To gain precision in the evaluators’ priors, we measure evaluators’ attitudes toward risk, ambiguity, and time preferences, enabling us to capture how they weight risk and return in their scoring and eventually controlling for them when eliciting priors.
Finally, we will compare aggregate priors of the evaluators to the aggregate priors of the relevant stakeholders.