Kartu Prakerja Impact Evaluation

Overall impact evaluation: Receipt of the Kartu Prakerja jobs training and cash transfer program among eligible applicants

The Kartu Prakerja (or Pre-Employment Card) is a newly launched Government of Indonesia (GoI) program that aims to address three interconnected problems: 1) skills gaps that may prevent workers from entering into good jobs, 2) the difficulties unemployed or informal workers face meeting their consumption needs, and 3) providing financial assistance to help support households and allowing the them to smooth consumption during their job search. The program also will support take-up of the digital financial tools that facilitate full participation in a modern economy. Kartu Prakerja enrolls eligible applicants through a lottery system. Beneficiaries can enroll in a wide variety of training courses through an online platform and receive cash transfers upon completing a course. The program provides the cash transfers electronically to bank or e-wallet accounts.

2021-01-25

2023-12-31

The six primary outcomes we will examine are:
• Labor force outcomes (employed; new job or business since program application; wage; monthly household income, hours worked; job search and activity preparing a new business; job satisfaction, confidence in business skills, child care responsibilities, sectoral change subject to data availability)
• Consumption smoothing (asset sales, loans, transfers, migration, and self-reported and subjective consumption)
• Psychometric outcomes (depression; self-efficacy)
• Digital skills and comfort (use of internet in job, comfort with and usage of e-money, preferences for e-money vs. phone credits as survey compensation, use of platforms)
• Types of trainings chosen (courses selected, amount of training budget spent).
• Approval of government COVID response and preferences about government programs

We will also examine ‘first stage’ outcomes that measure program usage (program uptake, obtaining training certificates, total hours spent in training courses).

See below.

Heterogeneity: In addition to the overall impact evaluation, we plan to analyze two important dimensions of heterogeneity
• Multiple applications: for households that apply in batch X, compare impacts for winners from batch X with impacts from those who lose batch X but reapply and win in a subsequent batch. We will focus on batches for which there is immediate and/or short delay between the announcement of one batch and the application to the next batch.
• Gender: for households where multiple family members apply in the same batch, how do outcomes differ if a male vs. female applicant is randomly selected as the winner in that batch.

Note that for these outcomes, we will consider the primary outcomes above, as well as additional primary outcomes from the administrative data (number of reapplications, type of training course chosen, share of available training budget spend, and time to first stipend disbursal).

When possible, depending on timing of data source, we will do heterogeneity based on whether those randomized to receive the program are currently / recently getting the stipend as of the time of the survey.

We will also do secondary heterogeneity analysis based on demographics (age, education, gender, rural/urban, java/off java, and baseline occupation if available).

Descriptive Analysis of Program Uptake: We will also provide descriptive analysis of program uptake using baseline data matched to administrative data for eligible populations, including marginal value of income (consumption, wages), baseline internet access (cell phone coverage, smartphone/laptop ownership), gender / recent maternity status, previous employment, disability, and other demographics.

The Kartu Prakerja enrollment lottery randomizes eligible applicants to receive the program (treatment group) or not (control group). Those who are not selected in a given wave of this lottery are allowed to reapply. The first component of this research is to compare the outcomes of eligible applicants selected to receive the program to eligible applicants not selected to the program, to accurately calculate the program’s impacts. We extend this analysis to understand determinants of program takeup and heterogeneous effects along two key dimensions: a) heterogeneity based on the number of times people have previously applied and b) for families with multiple applicants, heterogeneity based on which family member is randomly selected.

Data
We study eligible program applicants from 2020 and 2021 using 1) Kartu Prakerja administrative data, 2) Indonesian national sample surveys (Sakernas and Susenas from August 2020-September 2021 as outcomes, and prior Sakernas and Susenas from 2018-2020 as baseline), and 3) program-specific surveys.

Kartu Prakerja is an ideal research context because eligible applicants to are offered the program through a lottery which randomizes eligible applicants, at the individual level, to receive the program (treatment group) or not (control group). Our study employs Kartu Prakerja’s lottery randomization, and multiple data sources to generate evidence, focusing on eligible applicants to the 2020 and 2021 enrollment cohorts.

Given Kartu Prakerja’s lottery system, we can compare the outcomes of eligible applicants selected to receive the program to eligible applicants not selected to the program, using the outcome data described below, to accurately calculate the program’s impacts. The enrollment lottery is conducted in waves, and applicants are screened for eligibility prior to randomization. Eligible applicants who were not selected in a given lottery wave are allowed to reapply and participate in a subsequent wave.

To account for this design, we will use an instrumental variable approach, where winning the lottery is an instrument for receiving Kartu Prakerja. The first stage regression will predict ultimate receipt of the Kartu Prakerja program by the time the outcome data was collected based on being selected in a given lottery wave. To analyze all lottery waves together, we simply stack the data as if we were running each wave one by one, and run a joint version above the instrumental variable equations with lottery wave fixed effects, clustering standard errors by individual to take into account the fact that a given individual can enter a lottery multiple times, but only has a single outcome variable in a given dataset. Using current program data that we have received from PMO on the number of eligible applicants and beneficiaries selected in each wave from PMO, we have already assessed that this instrumental variable approach yields a strong first stage.

We can extend the impact evaluation analysis to understand determinants of program take-up. First, we can explore the characteristics of those who apply, are selected, and engage most in training; whether treatment effects differ for different types of beneficiaries; how different types of beneficiaries use the cash transfers; and the characteristics of those who obtain the highest benefit from Kartu Prakerja. Using regression interactions and machine learning techniques, we will analyze heterogeneity by baseline demographic characteristics, as well as competency and motivation metrics from the application form.

Kartu Prakerja’s lottery also presents a second strategy for studying take-up decisions. Given the lottery is conducted in waves, and applicants not selected in a given wave can reapply and participate in the next lottery wave, by examining the number of lottery waves an eligible applicant attempts, we can explore whether these “hurdles” are an effective screening device for ensuring that the highest-potential applicants enter the program, whether they screen out those most in need, or whether selection effects vary by outcome (different implications for labor vs. consumption vs. financial inclusion impacts). Since multiple family members can apply, and who is chosen is random, we can also use this to study the differential impacts of the program for different types of family members.

(2) Data
We study eligible program applicants from 2020 and 2021 using 1) Kartu Prakerja administrative data, 2) Indonesian national sample surveys, and 3) program-specific surveys. Administrative data contains baseline characteristics and which trainings participants choose. The SUSENAS and SAKERNAS national sample surveys are administered twice a year by the GoI to a random sample of households across Indonesia, including some individuals who have applied for Kartu Prakerja. SAKERNAS asks about labor outcomes (employment, wages, hours), SUSENAS focuses on consumption (spending patterns, food security), and both contain measures relevant to financial inclusion. We are also conducting additional surveys for more detailed information. As part of this research, we are expanding the scope and coverage of these surveys to include additional questions and also provide comparable data for the control group.

Computer random number generator

Individual

No

N/A; individual randomization

Administrative Data: Approximately 45,000,000 applicants Surveys we conduct: Approximately 40,000 beneficiaries Government surveys: All applicants who are matched to national sample surveys

Approximately 45,000,000 applicants in batches 1-17; approximately 5 million of which were chosen to receive the program; only a small subset of these will be covered in survey data, see above

210901+PraKerja+Analysis+Plan.pdf