Do You Trust Your Own Voice? – An Experimental Approach to the Strategic Use of Audio Deepfakes

Subjects participate in a laboratory-online experiment and are assigned to the roles Speaker (Trustee) or Listener (Trustor) and engage in a trust game. Listeners are assigned to one of the treatments, varying the type of message they receive: Control (no message), Text (transcribed message), Authentic (authentic voice message), Artificial-1 (artificial voice message, young woman), Artificial-2 (middle aged woman), Artificial-3 (young male), Artificial-4 (middle aged man).

Subjects for the laboratory part of the experiment are recruited from the university’s subject pool and are assigned to the role Speaker. They record an audio message (authentic message) and select an artificial voice (artificial message), before engaging in two rounds of trust game decisions. The strategy method is used to elicit the trust game decisions for both message types. Subjects also state their expectations about the respective trust game decisions of the Listeners and rate the available artificial voices.
Subjects for the online part of the experiment are recruited from Prolific and are assigned to the role Listener and are randomly assigned to one of the seven treatments. Subjects state their initial trust game decision (pre-message), before either: wait 10 seconds (Control), read a message (transcribed message; Text), listen to an authentic message (Authentic), or listen to an artificial message (Artificial-1/2/3/4). Subjects make the trust game decision one more time (post-message). There’s a 50% chance the initial pre-message trust game decision is payoff relevant and a 50% chance the latter post-message trust game decision is payoff relevant. Subjects in the Authentic and Artificial treatments also rate whether they perceived the audio message as authentic or artificial.

2024-12-09

2025-01-31

Trust game decisions (Amount transferred and amount transferred back)
Perceived authenticity of audio messages
Voice preference of artificial voices

Trust game decisions measure the trust and trustworthiness in respect to the message type. Speakers’ trust game decisions are elicited using the strategy method, which allows for a complete mapping of potential responses. Listeners’ trust game decisions are elicited using an initial decision before the exposure to the message and a decision after the exposure to capture deviations from individual baselines.
Perceived authenticity of audio messages measures the human detection rate of audio deepfakes, the believability of the used audio messages, as well as discloses potential influences on the trust game decisions.
Voice preference of artificial voices assess whether subjects base their choice of artificial voice on their perception of trustworthiness or characteristics such as gender and age.

Expectations of trust game decisions
Perceived trustworthiness ratings of artificial voices
General trust and risk preferences
Demographic factors

Expectations of trust game decisions analyze the alignment between Speaker expectations and actual Listener behavior in respect to the message type.
Perceived trustworthiness ratings of artificial voices assess whether subjects base their choice of artificial voice on their perception of trustworthiness or characteristics such as gender and age.
General trust and risk preferences assess individual differences in baseline trust and risk.
Demographic factors (including age, gender, English understanding and speaking level) are examined to measure their influence on trust game decisions and the evaluation of authentic and artificial messages.

The experiment consists of a trust game with one-sided pre-play communication and involves two roles: Speaker (Trustee) and Listener (Trustor). The study is split in three parts:
Part 1 (Laboratory, Speaker Role): Subjects assigned to the Speaker role record an authentic audio message and select an artificial audio message. They participate in trust game decisions using the strategy method and provide ratings for artificial voices. General trust, risk preferences, and demographic information are also elicited.
Part 2 (Online, Listener Role): Subjects assigned to the Listener role are recruited online and randomly assigned to one of seven treatments: Control (no message), Text (transcribed message), Authentic (authentic voice message), Artificial-1 (artificial voice message, young woman), Artificial-2 (middle aged woman), Artificial-3 (young male), Artificial-4 (middle aged man).. They make trust game decisions before and after receiving the message and classify the message as either authentic or artificial. General trust, risk preferences, and demographic information are also elicited.
Part 3 (Laboratory, Speaker Role): Speakers review aggregated trust game decision statistics based on message type and make an additional decision on which message type to use for a new trust game interaction, elicited using the strategy method.

Subjects engage in a trust game with one-sided pre-play audio communication, assigned to the roles of Speaker (Trustee) and Listener (Trustor). Both - Speaker and Listener - are endowed with 10 points.
Trust Game Procedure:
- The Listener transfers an integer amount between 0 and 10 points to the Speaker.
- The transferred amount is tripled by the experimenter.
- The Speaker decides on the percentage of the tripled amount to return to the Listener using the strategy method.
- Payoffs: Speaker: 10 points + the portion of the tripled amount they keep. Listener: 10 points + the amount returned by the Speaker.
The experiment is conducted in three parts:
Part 1 (Speakers, Laboratory): Subjects are recruited from the university’s subject pool and assigned to the Speaker role. The experiment proceeds as follows:
- General Trust Game Instructions.
- Authentic Message Recording: Subjects record an audio message (maximum of 15-seconds) directed to one Listener. Private information is not allowed in the message.
- Artificial Message Selection: Subjects select one of four generated artificial messages directed to another Listener. Artificial messages differ only in tone of voice (e.g., gender, age); the content is identical to the authentic message.
- Trust Game Decisions (Strategy Method): Decisions are elicited for both the authentic message and artificial message conditions. The order of message types is randomized. Payoff Mechanism: One decision is randomly selected for payment.
- Expectations: Subjects state their expectations about the amount the Listener will transfer for each message type. Expectations are incentivized based on accuracy.
- Voice Evaluations: Subjects rate the trustworthiness of the four artificial voices.
- Additional Elicitations: General trust and risk preferences. Basic demographic factors.

Part 2 (Listeners, Online via Prolific): Subjects are recruited online and assigned to the Listener role. They are randomly assigned to one of seven treatments:
- Control: No message.
- Text: A transcribed version of the Speaker’s authentic message.
- Authentic: The authentic voice message recorded by the Speaker.
- Artificial-1/2/3/4: The respective artificial message.

The procedure is as follows:
- Audio Check: Performed only for subjects in the Authentic and Artificial-1/2/3/4 treatments.
- General Trust Game Instructions.
- Trust Game Decisions: Listeners first state their initial trust game decision (pre-message) without any message. Then:
- Control: Wait 10 seconds.
- Text: Read the transcribed message.
- Authentic/Artificial-1/2/3/4: Listen to the respective message.
Listeners make the trust game decision one more time (post-message). Payoff Mechanism: There is a 50% chance that either the pre-message or the post-message decision is payoff-relevant.
- Message Classification: Listeners in the Authentic and Artificial-1/2/3/4 treatments classify the received audio message as authentic or artificial (incentivized).
- Additional Elicitations: General trust and risk preferences. Basic demographic factors.

Part 3 (Speakers, Laboratory): Speakers from Part 1 complete the following tasks:
- Feedback on Listener Decisions: Speakers are shown statistics summarizing Listener trust game decisions by message type.
- Message Type Decision: Speakers choose which message type (authentic or artificial) to send to a previously uninvolved Listener.
- Trust Game Decision: Speakers make a trust game decision using the strategy method for the chosen message type. Speakers state their expectations for the Listener's transfer amount. Expectations are incentivized based on accuracy.

Individual subjects in the role of Speaker are randomly recruited from the university’s subject pool. Due to known technical difficulties in reliably generating audio deepfakes, the planned number of subjects in the role of Speaker is doubled in order to ensure having sufficient technically sound audio deepfakes. Individual subjects in the role of Listener are recruited on Prolific and randomly assigned to the treatments.

Individual subjects in the role of Speaker are randomly recruited from the university’s subject pool. Individual subjects in the role of Listener are recruited on Prolific and randomly assigned to the treatments.

No

-

40 Speaker observations (5 audio messages + 1 text message each; 240 messages) 2800 Listener observations (10 observations for each message + control treatment)

Sample size of 80 Speaker (in order to get 40 technically sound artificial audio messages).
Listeners and the respective treatment:
- Control: 400 subjects
- Text: 400 subjects
- Authentic: 400 subjects
- Artificial-1: 400 subjects
- Artificial-2: 400 subjects
- Artificial-3: 400 subjects
- Artificial-4: 400 subjects

Using G*Power with alpha = 0.05, beta = 0.2 and medium effect sizes - Mann-Whitney U test: 134 subjects (67 per group) - Chi-square test of independence: 207 subjects - Pearson correlation: 82 subjects