Information Gathering and Contracting Outcomes in Online Procurement

Buyers in a large online market for services procurement will see different prompts that provide either control messages or suggested best practices for hiring. We will measure how these prompts change whether buyers adopt staged hiring, which entails formally defining milestones that are smaller than a full project. We will also measure whether treated buyers change the number of bidders they evaluate. Auxiliary data analysis will seek to understand how these treatments change buyers’ time use during the formal contracting/negotiation phase and subsequent time spent communicating after a bidder begins working.

The platform employs 16 recruiters who are specialists in the procurement process and who assist buyers with writing job advertisements and attracting applicants. Each week the director of the recruiter program will communicate a randomly assigned message that each recruiter will use when chatting with clients. There are 2 active treatments and 1 control that the company has agreed to deploy at the time of pre-registration. Outside of these messages, the recruiters have been instructed to continue interacting with clients in a manner nearly identical to their day-to-day tasks. The following variations will be put in place:

Group 1 – Interact with clients per current company protocol. The regular interaction with buyers involves a chat popup on the site where a recruiter offers a personal introduction and helps the buyer to attract and screen freelancers for projects. The recruiter will then chat with freelancers who may be interested, and the recruiter subsequently responds to each sides’ needs and helps to serve a matchmaking function. In addition, the recruiter will display an introductory message to client while in the queue, prior to their first interaction, that reads as follows:
“My role as a Recruiter is to find you great freelancers to bid on your project. I can also help you refine your project description, review bids from qualified freelancers, and select the best freelancer for your project.”

Group 2 – Interact with clients per current company protocol. In addition, display an expanded message to client while in the queue, including an explanation of the benefits of setting up a small milestone:

“My role as a Recruiter is to find you great freelancers to bid on your project. I can also help you refine your project description, review bids from qualified freelancers, and select the best freelancer for your project.

Expert Tip: Create a small paid milestone task to get a preview of a freelancer’s work – and working style – early on.”

Group 3 – Interact with clients per current company protocol. In addition, display an expanded message to client while in the queue, including an explanation of the benefits of milestones and screening/evaluating multiple freelancers:

“My role as a Recruiter is to find you great freelancers to bid on your project. I can also help you refine your project description, review bids from qualified freelancers, and select the best freelancer for your project.

Expert Tip: Create a small paid milestone task to get a preview of a freelancer’s work – and working style – early on.

Not sure which freelancer to choose? Invite more than one freelancer to complete an initial milestone task. Then hire the freelancer who delivers the best result to work on your project!”

Each recruiter will be assigned to use 1 message each week for 4 or more weeks. The company leadership wants to see all recruiters in all conditions as part of the project, and the week of July 12 message assignment will allocate any recruiter to a new messaging condition if they have not cycled through that condition previously. Otherwise recruiters will be assigned at random for the week of July 12. Messages will not be encouraged on weekends, only week days.

At the time of pre-registration, our data use agreement with the platform has not been finalized, so we have not been able to assess power or likely effect sizes from an initial pilot that was conducted. Also at the time of pre-registration, there is discussion with the platform around offering a subsidy to estimate the takeup elasticity of these suggestions, but this has not been finalized.

2020-06-10

2020-07-24

Project fill rates, log project revenue, project success, client retention, number of project milestones, number of freelancers evaluated, number of freelancers offered milestone contracts.

Number of freelancers evaluated on future projects, number of milestones on future projects, buyer time use and extent of chat (to measure effort costs), measures of freelancer quality (to measure match composition change). The exact details of these measures are not yet known because we do not yet have access to data.

Please see details under intervention for randomization.

The main issue with data handling and the design is specifying how to deal with buyers who have multiple interactions with recruiters during the experiment. For these buyers, we will code their treatment based on the maximum extent of information they have seen previously. For example, a buyer who sees message 1, message 3, then message 2 will have treatment coded as 1, 3, 3 because treatment 3 contains more information than treatment 2.

In office by computer

Clustered by recruiter.

Yes

There are 16 individual units where treatment will be assigned, with different treatment assignment each week.

We do not have access to data to include this information, as it depends on the inflow of buyers to the site. We cannot release the planned sample size without consent of the company in a data use agreement.

The expected number of clusters per treatment (recruiter-weeks) for a 4 week run-time is 21. We cannot disclose the sample size without consent of the company in a data use agreement.