Impact Evaluation of an Intervention on Small and Medium Enterprises in Chile

Impulso Chileno is a program created by a Chilean NGO to help high-potential Chilean firms grow faster by giving them business training, one-on-one mentoring, and a sizable monetary transfer. This program has been implemented yearly since 2018, and its fifth iteration will be evaluated with a randomized control trial.

The NGO identifies high-potential small and medium enterprises (SMEs) through a business contest in which entrepreneurs present their business, its financial statements, and a business plan for what they would do with their monetary endowment.

Firms that participate in the contest face a selection process with several stages. Ultimately an evaluation committee will choose the 500 firms they believe show the most promise. 250 of these firms will be randomly assigned to the treatment group, while the rest will form a control group.

The treatment group will then receive the three primary components of the program:
(1) A monetary transfer:
- This cash transfer can range between 3 and 5 million Chilean pesos (approximately USD 3,500 to USD 5,800).
- The average value of the transfer is USD 5,300. This transfer can only be used to buy physical assets, working capital, digital tools, and fund marketing campaigns.

(2) A business training program:
- The training program consists of 30 hours of online, group-based, real-time business training sessions.
- The Business Administration School of the Pontificia Universidad Católica de Chile (PUC Chile) imparts this program.

(3) Individual mentoring:
- At least eight one-on-one mentoring sessions, which have a free-form structure and usually last about an hour.
- The mentors usually lead or work for larger Chilean companies in the same sector as the mentees. They are volunteers from Corporación Simon de Cirene’s mentoring network.

Meanwhile, the control group will receive none of these components. However, we plan on offering three optional online talks to entrepreneurs on both the treatment and control groups throughout the year to maintain a base level of engagement with both groups and minimize the risk of attrition bias. The proposed subjects for these talks are (1) empowerment, (2) business experiences and (3) work-life balance. We expect these talks to not have a significant impact on outcome variables, but we will keep track of entrepreneur’s attendance to the events in order to control for them.

2023-01-20

2023-07-31

Monthly sales, costs, and profits from survey data; Number of employees and type (family members, formal employees, full time vs part-time employment, unpaid helpers). If available from tax data: sales, taxable profits, costs.

Sales (survey)
- Monthly Sales for November - Estimated using three different approaches, then verified by the entrepreneur.
- Other Monthly Sales - Directly asked from August to October.
- Yearly Sales - An estimate of the sales in the last 12 months.

Costs (survey)
- Monthly Costs per Item - Asked how much money was spent during November for the following items: merchandise, payment of infrastructure services (electricity, water, etc.), gasoline, payment of debts and interest rates, cost of salaries and wages, renting equipment, establishment’s rent, cost of telecommunication services, tax payments, repair and maintenance of equipment, professional services (ie. lawyers, accountants), transportation costs and others.

Profits (survey)
- Monthly Profits - Profits for November are calculated by subtracting monthly costs from their estimated monthly sales. This calculation is done via survey software and shown to the entrepreneur so they can verify it. Changes are made to each sub-section if the entrepreneur sees a problem with the final result.
- Yearly Profits - Directly estimated by the entrepreneur taking into account the last 12 months.

Number of Employees
- Number of Employees - how many employees did the business have during November. An employee is someone who works in their business and receives a recurring salary or wage.
- Number of Full-Time and Part-Time Employees - how many of the employees are full-time, and how many hours a week do the part-time employees work relative to a typical 44 hour a week full-time job.
- Number and Percent of Formal Employees - how many of these employees are formal employees. By this, we mean that they are currently subscribed to social security and health care systems through this business.
- Number and Percent of Related Employees - how many of these employees are family members of the entrepreneur?
- Number of unpaid employees - apart from the employees numbered, how many other people help out at the business that does not receive a recurring wage or salary. As an example, we can think of the son of an entrepreneur who is involved in the business but doesn’t get paid as a helper.

Tax data (if available)
- Yearly reported profit
- Yearly reported cost
- Yearly reported sales
- Unemployment insurance data
- Number of formal employees

Business assets, networks, and business practices knowledge. The number of hours worked by the entrepreneur; entrepreneur’s compensation. If available from tax data: transactions, number of vendors/buyers.

Entrepreneur’s...
- Compensation - Track whether or not the entrepreneur gives himself a salary and its amount. Include business profits withdrawals made with other than monthly frequency.
- Weekly Hours Worked - How many hours a day does the entrepreneur work on his business for each day of the week. An indicator variable is created if the sum total of hours worked surpass 48 hours, which is the mark at which the ILO defines excessive working hours.
- Savings - Number of weeks of household consumption the entrepreneur could afford with the amount of money he has saved.
- Access to Healthcare and Social Security - Check whether the entrepreneur is currently subscribed to healthcare and social security services and if these are public or private.

Business Practices:
- Digital Tools and Online Presence - Indicator based on whether or not the entrepreneur partakes in a number different business practices related to the areas of usage of digital tools for marketing and record-keeping, along with presence in different social networks and online stores.
- Marketing - Indicator based on whether or not the entrepreneur partakes in 7 different business practices related to the areas of Marketing
- Record Keeping - Indicator based on whether or not the entrepreneur partakes in 8 different business practices related to the areas of record keeping
- Financial Planning - Indicator based on whether or not the entrepreneur partakes in 8 different business practices related to the areas of financial planning
- Stock Control - Indicator based on whether or not the entrepreneur partakes in 3 different business practices related to the areas of buying and keeping control of stock.

Learning:
- The Business Administration School of PUC Chile developed a 10-question diagnostic test that asks questions on the topics covered during the training sessions. The entrepreneurs will fill out a selection of these questions during the in-person surveys to measure changes in learning about business administration throughout the program

Business Assets:
- Current Business Assets

Networks
- Number of people the entrepreneurs can contact for business-related questions

Monitoring Data
- Spending on Investments and Capital - Entrepreneurs assigned to the treatment group need to send all invoices to the Luksic Foundation to ensure all purchases with the money handed out were made in the pre-approved categories. We plan to create a database with these invoices to complement our analysis and see which types of investments were the most common and most effective.
- Attendance to Training Sessions - We will keep track of individual entrepreneurs’ assistance to the Training Sessions to complement our analysis.
- Topics Covered during Mentoring Sessions - Mentors will fill out a form each time a mentoring session happens. In this form, we will know the session's length and the topics covered during that time. We aim to use this information to complement our analysis and see which topics were considered the most useful for entrepreneurs.

Tax data (if available)
- Number of vendors: number of firms from where the firm buys in the last year
- Number of buyers: number of different buyers in the previous year
- Number of transactions: number of sales and buys in the previous year

The Impulso Chileno program is divided into two stages. The first stage consists of a business plan competition to identify high-potential Chilean small and medium enterprises (SMEs). The program's second stage is where the top-rated firms from the competition receive the intervention itself: a business training course, individual mentoring, and a large cash transfer to invest in their businesses.

Our impact evaluation will randomly assign the top 500 firms participating in the competition into treatment and control groups to measure the program’s effect on the firm's growth.

Business Plan Competition:

The business plan competition will be held between September 2022 and January 2023. During this period, we expect around 20,000 firms to apply to the contest through an online application that gathers basic firm and entrepreneur-level data and their business plan proposal.

Firms must be formal and have annual sales within a pre-specified bracket [~$7,600 to $380,000 USD] to qualify as a high-potential SME. Firms that do not comply with these prerequisites are disqualified. Firms who fit these criteria will be scored based on their business model, entrepreneur’s profile, and business plan proposal.

The top-rated 2,500 firms will then be asked to submit tax statements for the previous four months and a 3-minute video explaining their business model and business plan. An evaluation committee (formed by representatives of the implementation partners) will watch the videos, review the documents, and re-rank the contenders to select the top 700 of them as semi-finalists.

Each semi-finalist will have a 30-minute interview with members of the evaluation committee, where entrepreneurs will present their business plan proposal. Based on this interview, the 700 firms will be re-ranked once again to find the top 500 finalists for the program.

Randomized Assignment:

Toward the end of January 2023, 250 finalists will be randomly assigned to a treatment group, while the rest will form a control group. Between February and July 2023, the treatment group will receive the three components of the program: a monetary transfer, business training and mentoring. Meanwhile, the control group will receive none of these components.

This randomization will be stratified based on the entrepreneur’s gender and the macro-zone from where the business is operated from. A macro-zone is an aggregation of different Chilean regions (analogous to US states), which split the country into 6 distinct areas; North, Center, Center-South, South and Austral Macrozones, along with the Metropolitan region of Santiago de Chile.

Data Collection:

In-person baseline surveys will be carried out at each finalist's place of work throughout December 2022 and January 2023, before the randomized selection process is carried out (N=500). Follow-up surveys will be conducted 12 months later. Most of the medium-term effects of the program will be measured using these surveys. We plan to use administrative data from unemployment insurance and tax authorities to measure the long-term impact of Impulso Chileno, if available.

Regression Discontinuity Design

Apart from the randomized control trial involving the top 500 firms, we will also survey and gather data from the next 200 firms who did not reach the business plan competition’s threshold to become finalists. We plan on using the scores and ranking within the business plan competition as the running variable in a regression discontinuity design, comparing the 200 non-finalist with the 250 control group firms. This data will be used to evaluate the contest’s effectiveness in identifying top firms and comparing the program’s impact on the average firm to those firms closer to the cut off.

Not available

The randomization process will be made through a series of 12 public lotteries. Each of the lotteries corresponds to a different strata given our stratification design. For example, there will be a lottery for female entrepreneurs that hail from the Austral macro-zone, where half of the lottery’s participants will be chosen at random to participate in the treatment group. The main implementing partner intends to live-stream this event and to have a public notary certify the transparency and fairness of it.

Randomization done at the individual firm level, stratifying by gender and macro-zone.

No

No clustering.

500 observations in total for the randomized control trial, with an extra 200 observations which will be used in the regression discontinuity design.

250 firms in the treatment group, 250 firms in the control group.
200 firms below the ranking threshold for randomization, which receive no treatment components.

Under the current specifications of our research design, we estimate that we will be able to find a minimum detectable effect (MDE) of 0.249 standard deviations in monthly sales for the firms in the trial. The top 300 participants of last year’s iteration of Impulso Chileno reported an average of monthly sales and profits to be around 5 million chilean pesos (CLP), which is around $5,300 USD. The standard deviation (sd) of monthly sales is 5.6 million CLP, so an MDE of 0.25 sd increase in monthly sales constitutes a increase of 1.4 million CLP ($1,500 USD) in monthly sales. This translates to a 28% increase in monthly sales. This randomized control trial had its power calculations parametrized using a power of 80%, a significance level of 5%, a sample size of 500, a correlation between yearly sales of 60%, an adoption rate of 90% and an attrition rate of 30%. Parameters such as correlations, adoption and attrition rates are set as a midway point between observed data from previous iterations of Impulso Chileno and data collected from a metaanalysis compiled by David McKenzie and Cristopher Woodruff in 2012.