|
Field
Experimental Design (Public)
|
Before
Applicants to the conference will grade submissions as peer reviewers. We start from the list of applicants’ affiliations and stratify them into two tiers based on subject specific university rankings. For each tier, we pair reviewers from different institutions of similar quality. We will also consider stratifying by reviewer seniority depending on the total pool of applicants by the closure of the call for papers. We then randomize institutions to either blind or not blind grading. We end up with a set of pairs where one reviewer is assigned to blind and the other reviewer is assigned to non-blind grading. This methodology also ensures that reviewers from the same institutions are assigned to the same treatment arm.
We stratify submissions into two tiers using the same university rankings. We will create as many blocks as the total number of reviewer pairs, and we form two types of blocks. Every block will contain 8 submissions, randomly chosen from both strata in different proportions. Type 1 blocks will contain mostly submissions coming from highly ranked institutions. Type 2 blocks will contain mostly submissions coming from less highly ranked institutions. For each reviewer strata, we randomly allocate a block of either type to each pair. For each couple, we only randomize blocks that do not contain submissions from the institutions where reviewers are affiliated with. This ensures that reviewers will not receive papers from their own institution. Our goal is to have blocks of both types allocated to both types of reviewers to explore possible heterogeneity across the institutional ranking of reviewers and submitters. Reviewers submit their grades within 14 days after receiving the grading package.
|
After
We compare reviews for the same extended abstract by blind and non-blind reviewers. We will use the whole body of submissions to a PhD Conference in Economics organized in Paris. Submissions are open to PhD students and early post-docs from any university.
Applicants to the conference will grade submissions as peer reviewers. We start from the list of applicants’ affiliations and stratify them into two tiers based on subject specific university rankings. For each tier, we randomize the affiliations of the reviewers into either blind or non blind grading. This methodology ensures that reviewers from the same institution are assigned to the same treatment arm.
We match blind and non blind reviewers from different institutions to make pairs. Pairs are formed such that reviewers from the top tier are matched with reviewers from the second tier, and remaining reviewers from the second tier are paired with someone from the same tier. This pairing methodology allows us to maximize the number of couples of each type and explore potential heterogeneity across couples.
We randomize papers to blocks and randomly assign them to couples. We will create as many blocks as the total number of reviewer pairs. Every block will contain 8 submissions, randomly chosen. For each couple, we only randomize blocks that do not contain submissions from the institutions where reviewers are affiliated with. This ensures that reviewers will not receive papers from their own institution. Reviewers submit their grades within 14 days after receiving the grading package.
|