Abstract
The objective of this study is to test whether to reduce the bias of sequential order on player's performance by information provision. By using a large set of data on piano players and piano competition judges in Japan. In this setting, we can clearly show the order of performance and the probability to win at the competition are negatively correlated, despite the fact that the order of performance is determined strictly by lottery. More specifically, the first few in the order of performance have statistically significantly lower scores. This occurred in all levels of competitions, including advanced and finals, and in fact there were much anecdotal evidence on SNS. It is widely known among players and their instructors that the first to perform in the performance order is less likely to win.
In fact, it has been observed in other music competitions, such as violin as well as piano, and in all other contexts, including figure skating, business pitches, idol contests, etc. This is referred to as the sequential order effect in the literature. There are two leading hypotheses about why the order of performance matters. The first hypothesis is judges' fatigue. This may occur because judges would exhaust more physically at the later of the competition and they are more likely they are to make permissive choices (Hockey, 2013). The second hypothesis is caused by "calibration," which occurs when judges try to make consistent decisions throughout the competition. Since the distribution of scores is not known in advance, it is not initially clear how many grades correspond to a perfect score. For this reason, they try to avoid extreme judgments and evaluations in the early stages. However, in the later stages, they try to maintain consistency by giving higher scores to those that do better based on their criteria, and these have been found to be common in "contest" or "competition" contexts (Antipov & Pokryshevskaya, 2017; Bian et al., 2021). The same may occur not only in competitions in music and sports, but also in job interviews, etc. There is a wide range of areas in which order is seen to influence performance. The results of this study are expected to have a significant impact on society in terms of enhancing the fairness of evaluation.
In our experiments, we will examine whether an information provision due to calibration can reduce the bias of sequential order. Specifically, 109 competitions with 12,673 players (as treatment group) are randomly selected from 230 competitions nationwide from late May to July, with five fixed judges per competition (121 control competitions with 11,386 players). Based on the date and location of competition, we will run a stratified cluster randomized trial. The intervention consists of using data from past competitions to show through Youtube video and one-page letter sized paper that the first few competitors in the order of performance have lower probability to win. In addition, to accurately estimate the effect of the information provision, we will measure the fatigue of each judge by using "Fatigue and Stress Measurement System" developed by Hitachi Systems, Ltd.