Primary Outcomes (explanation)
Accuracy of Earnings Prediction:
To assess the accuracy of earnings prediction, we compare the participants' answers with the actual archival data. The accuracy of participant $j$'s earnings prediction after reviewing firm $i$'s year $t$ conference call is:
\begin{align}\label{eq:epspred}
\text{Accuracy}_{ijt} = -1 \times \left| \mathbbm{1}(\text{Increase})_{it} - \left(\frac{\text{Answer}_{ijt}}{10}+0.5\right) \right|
\end{align}
$\mathbbm{1}(\text{Increase})_{it}$ is an indicator that equals one when firm $i$'s year $t+1$ EPS is larger than that of year $t$ and zero otherwise. As a participant's answer ($\text{Answer}_{ijt}$) ranges from $-5$ to $+5$, we scale it with ten and add back 0.5 to make it distributed between 0 and 1. As a participant makes a more correct prediction, the term $\left| \mathbbm{1}(\text{Increase})_{it} - \left(\frac{\text{Answer}_{ijt}}{10}+0.5\right) \right|$ becomes closer to zero. If a participant makes a completely incorrect prediction, this expression takes a value of 1. We multiply it by $-1$ to make higher values indicate a higher accuracy.
Additionally, we also use AI to evaluate the text quality.
*****
Accuracy of Sentiment:
Along with earnings prediction, participants are asked to assess whether the news is positive or negative for the company. Their answers span from $-5$ to $+5$, identical to the earnings prediction. We use the same formula as Equation \ref{eq:epspred} to evaluate the accuracy. However, instead of $\mathbbm{1}(\text{Increase})_{it}$ being an indicator for EPS improvement, we use cumulative stock returns around the earnings announcement dates (from one day before to one day after the announcement) as a benchmark. $\mathbbm{1}(\text{Increase})_{it}$ takes a value of one when announcement returns are positive and zero otherwise.
****
Portfolio returns and Sharpe Ratios for investments:
We measure investment efficiency by comparing the participants' answers with hypothetical maximum realized returns and Sharpe ratios. Participants provided their asset allocation plan for the stocks (Stocks A and B) and cash holdings. Their allocation amounts should add up to \$1,000. We compute relative investment weights $w_A$, $w_B$, and $w_C$ by scaling the amount invested in Stock A, Stock B, and cash by \$1,000 ($w_A+w_B+w_C=1$).
Participants must provide their asset allocation plans for both short-term and long-term. We define short-term returns as the daily returns measured from when the market opens on the trading day immediately following the earnings conference call until it market closes on the same day. Long-term returns are measured as cumulative returns from when the market opens on the trading day immediately following the earnings conference call, extending up to 252 trading days thereafter. We obtain participant $j$'s realized portfolio ($m$) return and Sharpe ratio after reading time $t$ earnings call information in both investment scenarios.