Impact Evaluation of an Intervention on Small and Medium Enterprises in Chile

Impulso Chileno is a program created by a Chilean NGO aimed to help high-potential Chilean firms grow faster by giving them business training, one-on-one mentoring, and a sizable monetary transfer. This program has been implemented yearly since 2018, and its fifth iteration will be evaluated with a randomized control trial.

The NGO identifies high-potential small and medium enterprises (SMEs) through a business plan competition in which entrepreneurs present their business, its financial statements, and a business plan detailing how they would invest the monetary transfer.

Firms that participate in the contest face a selection process with several stages. Ultimately, an evaluation committee will choose the 500 firms they believe show the most promise. During this year of the program, 250 of the 500 finalist firms will be selected at random from and assigned to the treatment group; the remaining 250 finalists will form a control group.

The treatment group will receive the three primary components of the program:
A monetary transfer
This cash transfer can range between 3 and 5 million Chilean pesos (approximately USD 3,500 to USD 5,800). The average value of the transfer is USD 5,300.
This transfer can only be used to buy physical assets, working capital, digital tools, and fund marketing campaigns.
A business training program
The training program consists of 30 hours of online, group-based, real-time business training sessions.
The Business Administration School of the Pontificia Universidad Católica de Chile (PUC Chile) delivers this program.
Individual mentoring.
At least eight one-on-one mentoring sessions, which have a free-form structure and usually last about an hour.
The mentors usually lead or work for larger Chilean companies in the same sector as the mentees. They are volunteers from Corporación Simon de Cirene’s mentoring network.


Additionally, two optional online talks were offered to entrepreneurs in both the treatment and control groups throughout the year to maintain a base level of engagement with both groups and minimize the risk of attrition bias. The subjects for these talks were: (1) empowerment, (2) business experiences and (3) work-life balance.

2023-01-20

2023-07-31

Monthly sales, costs, and profits from survey data; Number of employees and type (family members, formal employees, full time vs part-time employment, unpaid helpers). If available from tax data: sales, taxable profits, costs.

Monthly sales, costs, and profits from survey data; Number of employees and type (family members, formal employees, full time vs part-time employment, unpaid helpers). From tax data: sales, taxable profits, costs. From unemployment insurance data: number of formal employees. We will also report the Anderson Index of these outcomes.

From survey data
Sales
Monthly Sales for November
Yearly Sales - An estimate of the sales in the last 12 months.
Costs
Monthly Costs per Item - Asked how much money was spent during November for the following items: merchandise, payment of infrastructure services (electricity, water, etc.), gasoline, payment of debts and interest rates, cost of salaries and wages, renting equipment, establishment’s rent, cost of telecommunication services, tax payments, repair and maintenance of equipment, professional services (i.e. lawyers, accountants), transportation costs and others.
Profits
Monthly Profits - Profits for November are calculated by subtracting monthly costs from their estimated monthly sales. This calculation is done via survey software and shown to the entrepreneur, so they can verify it. Changes are made to each sub-section if the entrepreneur sees a problem with the final result.
Yearly Profits - Directly estimated by the entrepreneur, taking into account the last 12 months.
Number of Employees
Number of Employees - how many employees did the business have during November. An employee is someone who works in their business and receives a recurring salary or wage.
Number of Full-Time and Part-Time Employees - how many of the employees are full-time, and how many hours a week do the part-time employees work relative to a typical 44 hour a week full-time job.
Number and Percent of Formal Employees - how many of these employees are formal employees. By this, we mean that they are currently subscribed to social security and health care systems through this business.
Number and Percent of Related Employees - how many of these employees are family members of the entrepreneur?
Number of unpaid employees - apart from the employees numbered, how many other people help out at the business that does not receive a recurring wage or salary. As an example, we can think of the son of an entrepreneur who is involved in the business but doesn’t get paid as a helper.

From tax data
Tax data for 2021 and 2022 for baseline, and 2023 for follow-up. If there is an impact in 2023, we will study the impact in 2024, then up to 2029 and furthermore 2033-34.
Yearly reported profit
Yearly reported cost
Yearly reported sales
Number of employees

Business assets, networks, and business practices knowledge. The number of hours worked by the entrepreneur; entrepreneur’s compensation. From tax data: transactions, number of vendors/buyers.

From survey data
Entrepreneur’s
Compensation - Track whether the entrepreneur gives himself a salary and its amount. Include business profits withdrawals made with other than monthly frequency.
Weekly Hours Worked - How many hours a day does the entrepreneur work on his business for each day of the week. An indicator variable is created if the sum total of hours worked to surpass 48 hours, which is the mark at which the ILO defines excessive working hours.
Savings - Number of weeks of household consumption the entrepreneur could afford with the amount of money he has saved.
Access to Healthcare and Social Security - Check whether the entrepreneur is currently subscribed to healthcare and social security services and if these are public or private.
Business Practices
Digital Tools and Online Presence - Indicator based on whether the entrepreneur partakes in a number of different business practices related to the areas of usage of digital tools for marketing and record-keeping, along with presence in different social networks and online stores.
Marketing - Indicator based on whether the entrepreneur partakes in 7 different business practices related to the areas of Marketing
Record Keeping - Indicator based on whether the entrepreneur partakes in 8 different business practices related to the areas of record keeping
Financial Planning - Indicator based on whether the entrepreneur partakes in 8 different business practices related to the areas of financial planning
Stock Control - Indicator based on whether the entrepreneur partakes in 3 different business practices related to the areas of buying and keeping control of stock.
Learning
The Business Administration School of PUC Chile developed a 10-question diagnostic test that asks questions on the topics covered during the training sessions. The entrepreneurs will fill out a selection of these questions during the in-person surveys to measure changes in learning about business administration throughout the program
Business Assets
Current Business Assets
Networks
Number of people the entrepreneurs can contact for business-related questions

From Tax data:
Number of vendors: number of firms from where the firm buys in the last year
Number of buyers: number of different buyers in the previous year
Number of transactions: number of sales and buys in the previous year

Additional Data From Implementing Organization
Spending on Investments and Capital - Entrepreneurs assigned to the treatment group need to send all invoices to the Luksic Foundation to ensure all purchases with the money handed out were made in the pre-approved categories. We plan to create a database with these invoices to complement our analysis and see which types of investments were the most common and most effective.
Attendance to Training Sessions - We will keep track of individual entrepreneurs’ assistance to the Training Sessions to complement our analysis.
Topics covered during Mentoring Sessions - Mentors will fill out a form each time a mentoring session happens. In this form, we will know the session's length and the topics covered during that time. We aim to use this information to complement our analysis and see which topics were considered the most useful for entrepreneurs.
This additional data will be used for compliance (the first two) or give qualitative data that might be useful in unpacking channels (the third)

The Impulso Chileno program is divided into two stages. The first stage consists of a business plan competition to identify high-potential Chilean small and medium enterprises (SMEs). The program's second stage is where the top-rated firms from the competition receive the intervention itself: a business training course, individual mentoring, and a large cash transfer to invest in their businesses.
Our impact evaluation randomly assigned the top 500 firms participating in the competition into treatment and control groups to measure the program’s effect on the firm's growth.

Business Plan Competition:
The business plan competition was held between September 2022 and January 2023. During this period, 21,780 firms applied to the contest through an online application that gathers basic firm and entrepreneur-level data and their business plan proposal.
Firms had to be formal and have annual sales within a pre-specified bracket [~$7,600 to $380,000 USD] to qualify as a high-potential SME. Firms that did not comply with these prerequisites were disqualified. Firms that fit these criteria were scored based on their business model, entrepreneur’s profile, and business plan proposal.
The top-rated 2,500 firms were then asked to submit tax statements for the previous four months and a 3-minute video explaining their business model and business plan. An evaluation committee (formed by representatives of the implementation partners) watched the videos, reviewed the documents, and re-ranked the contenders to select the top 700 of them as semi-finalists.
Each semi-finalist had a 30-minute interview with members of the evaluation committee, where entrepreneurs presented their business plan proposals. Based on this interview, the 700 firms were re-ranked once again to find the top 500 finalists for the program.
Randomized Assignment:
Toward the end of January 2023, 250 finalists were randomly assigned to a treatment group, while the rest were assigned to the control group. Between March and July 2023, the treatment group received the three components of the program: a monetary transfer, business training and mentoring. Meanwhile, the control group received none of these components.
This randomization was stratified based on the entrepreneur’s gender and the macro-zone from where the business is operated. A macro-zone is an aggregation of different Chilean regions (analogous to US states), which split the country into 6 distinct areas; North, Center, Center-South, South, and Austral Macrozones, along with the Metropolitan region of Santiago de Chile.
Data Collection:
In-person baseline surveys were carried out at each finalist's place of work throughout December 2022 and January 2023, before the randomized selection process was carried out (N=500). Follow-up surveys will be conducted 12 months later. Most of the medium-term effects of the program will be measured using these surveys. We plan to use administrative tax data to measure the short and long-term impact of Impulso Chileno.
Impact Evaluation - OLS:
We will estimate the average treatment effects of the program by conducting an Ordinary Least Squares (OLS) regression, where the assignment variable will serve as the key predictor. To enhance the precision of our analysis, we will employ Machine Learning algorithms, specifically, Post-Double Selection LASSO to select controls in the main regressions; and causal forests to study heterogeneous effects, thereby adopting a data-driven approach.
In specific, regarding causal forests, we will study treatment heterogeneities estimating the Conditional Average Treatment Effect (CATE) including, in the causal forest, the following baseline characteristics: entrepreneurs’ gender, age of the entrepreneur, entrepreneur’s education level, business sector, years since business creation, and region of operation. Furthermore, we will use the predictions on the expected treatment effect for each individual, given the covariates, to investigate treatment heterogeneity.

We will divide the sample in two groups, that is, top and bottom half of the predictions. Then, we will report the balance test for the CATE and p-values adjusted for multiple hypothesis testing. Finally, using that input, we will further detect covariate differences between the two groups to know which are the groups that benefit the most from participating in the Impulso Chileno Program.

To increase power, we will control for the baseline value of dependent variables if available. We will also control by strata and survey specific differences (such as in-person or virtual follow-up).

We will also analyze (without the experimental variation) the program implementation (type of investment funded, attendance to training sessions).

Note that in a companion paper (Assessing Entrepreneurship Potential: Evaluating Video Submissions for Selection Process) we will analyze how applicant’s videos affect the selection process.

Not available

The randomization process was made through a series of 12 public lotteries. Each of the lotteries corresponded to different strata given our stratification design. For example, there was a lottery for female entrepreneurs that hail from the Austral macro-zone, where half of the lottery’s participants were chosen at random to participate in the treatment group. The main implementing partner lived-streamed this event and had a public notary to certify the transparency and fairness of it.

Randomization done at the individual firm level, stratifying by gender and macro-zone.

No

No clustering.

500 observations in total for the randomized control trial, with an extra 200 observations which will be used in the regression discontinuity design.

250 firms in the treatment group, 250 firms in the control group.

Under the current specifications of our research design, we estimate that we will be able to find a minimum detectable effect (MDE) of 0.249 standard deviations in monthly sales for the firms in the trial. The top 300 participants of last year’s iteration of Impulso Chileno reported an average of monthly sales and profits to be around 5 million chilean pesos (CLP), which is around $5,300 USD. The standard deviation (sd) of monthly sales is 5.6 million CLP, so an MDE of 0.25 sd increase in monthly sales constitutes a increase of 1.4 million CLP ($1,500 USD) in monthly sales. This translates to a 28% increase in monthly sales. This randomized control trial had its power calculations parametrized using a power of 80%, a significance level of 5%, a sample size of 500, a correlation between yearly sales of 60%, an adoption rate of 90% and an attrition rate of 30%. Parameters such as correlations, adoption and attrition rates are set as a midway point between observed data from previous iterations of Impulso Chileno and data collected from a metaanalysis compiled by David McKenzie and Cristopher Woodruff in 2012.