Can artificial intelligence prevent passive waste? Evidence from early warning alarms in Brazilian public procurement.

Intervention. We evaluate the impact of sending information revealing textual patterns in procurement documents that indicate the presence of passive waste in public procurement in the Brazilian public sector. More precisely, within a sample of purchase acts with at least one early warning alarm, we randomly allocate e-mails to the purchase acts' managers that reveal textual patterns suggesting:
- Opportunities for efficiency gains (treatment 1).
- Unnecessary restrictions to competition (treatment 2).

2022-07-18

2024-07-17

Primary outcomes. We will estimate the effect of our two treatments on three primary procurement outcomes. First, to compute the percentage price change caused by our treatments, we will use the logarithm of the auction price as an outcome. Second, we will estimate treatment effects on the auction rebate, which measures the percentage discount of the auction concerning its reference price. Third, we estimate treatment effects on value for money measures, which compares the prices in our sample with average prices of the same goods outside our sample.

Primary outcomes. We follow Bandiera et al. (2009) and use hedonic regressions to estimate quality-adjusted prices.

Secondary outcomes. We estimate treatment effects on the quality of the goods and auction competitiveness. If we can merge the procurement and contract management phases accurately, we plan to include time- and cost- overrun measures in the analysis.

We take several steps to compute our complementary outcomes. First, following the recent work in progress of Dimas Fazio, we plan to measure the quality of procured goods using information about the brands purchased. Second, we measure auction competitiveness using the number of bids and the number of bidders.

Experimental design. We randomly allocate e-mails to public managers revealing early warning alarms based of textual patterns in procurement documents that suggest the presence of passive waste. We divide the early warning alarms in two groups: one suggesting opportunities for efficiency gains (treatment 1) and another indicating unnecessary competition restrictions (treatment 2).

We used a clustered randomization design at the purchase act level. In the Brazilian procurement system, documents regulating the procurement auctions are published at the purchase act level, which are groups of procurement auctions managed by the same organization. In this case, by construction, the textual patterns suggesting the presence of passive waste affect all auctions within a purchase act. Therefore, while our unit of analysis is the procurement auction, our two treatments are randomized at the purchase act level.

Our population of interest is the set of auctions belonging to purchase acts with at least one early warning alarm during the two years of impact evaluation. We expect a sample with 257 purchase acts (number of clusters) and 822 auctions (number of observations).

Not available

Randomization method. For legal and logistical reasons, the Brazilian Court of Audits (TCU) randomizes the treatment allocation and sends the messages to our receiving organization (DNIT). The randomization process happens in four steps. First, TCU's artificial intelligence tools select each purchase act with at least one early-warning alarm. Second, the software Microsoft Power BI generates a data entry identified by the pair (purchase act ID, type of alarm), uniquely identifying each group of early earnings alarms within each purchase acts. Third, the software allocates each entry to control and treatment groups and prepares an automated e-mail for each distinct alarm in the data entries assigned to the treatment group. Fourth, for each purchase act, the chief of DNIT's internal control receives the automated e-mails and forwards them to the local office manager implementing the purchase act.

We describe a didactical example to increase our readers' understanding. Assume that the purchase act with identifier PA1 has five early warning alarms belonging to two groups: two text patterns suggesting restriction to competition (RC) and three indicating opportunities for efficiency gains (EG). In this case, the software generates two data entries - i.e., PA1-RC and PA1-EG - and two randomizations. If we assume that the software randomly allocates PA1-RC to the treatment group and PA1-EG to the control group, the manager of the purchase act receives two e-mails, one for each early warning alarm suggesting restriction to competition (RC).

Randomization unit. Our unit of randomization is the unique identifier of a group of alarms within a purchase act, implying that we are randomizing our two treatments at the purchase act level.

Yes

Planned number of clusters. We estimate the number of clusters based on monthly frequencies of early warning alarms among DNIT's purchase acts in the thirty-six months between July 2019 and June 2022. The average number of purchase acts with at least one early warning alarm per month and non-missing data for our primary outcomes is 10.72 in the entire sample. The same averages are 3.52 in the sample of purchase acts with at least one alarm suggesting unnecessary restrictions to competition (sample 1) and 8.16 in the one with at least one alarm suggesting opportunities for efficiency gains (sample 2). Therefore, for an experimental evaluation of two years of duration, we expect 257 purchase acts in the entire sample, 85 in sample 1, and 196 in sample 2.

Such projections are conservative because they still do not include purchase acts in the Regime Diferenciado de Contratação (RDC), a procurement modality created to accelerate selected infrastructure projects.

Planned number of observations. Given that we estimate an average of 3.2 auctions per purchase acts without missing variable for the outcomes in our sample, the estimates about the number of clusters imply an expected sample size of 822 auctions in the full sample, 272 in sample 1, and 627 in sample 2.

Sample size (or the number of clusters) by treatment arms. As we divide the purchase acts evenly among control and treatment groups, we expect 129 purchase acts in at least one of the treatments, 43 purchase acts in treatment 2, and 98 purchase acts in treatment 2.