Models were much better at spotting evasive language in earnings-call answers than at judging whether a speaker sounded unconfident, according to a new arXiv preprint. In direct tests against human labels, GPT-5 reached 87.4 on textual evasion using macro F1, a score that weighs the classes evenly, and 89.5 on accuracy. The audio model GPT-Audio reached 58.5 on macro F1 and 38.2 for the unconfident class.
The benchmark behind the comparison is called DualEvasion. It contains 505 annotated question-answer pairs from 60 earnings calls, each carrying two independent judgments: one about textual evasion and another about vocal confidence. The study was designed to test whether those channels provide complementary information, rather than treating a shaky-sounding answer as automatically evasive.
That separation mattered. The text and vocal labels disagreed in 162 cases, or 32.1% of the sample. The four combinations were 302 answers judged direct and confident, 67 direct but unconfident, 95 evasive but confident, and 41 both evasive and unconfident. In other words, direct language could coexist with an unconfident delivery, while an evasive answer could sound confident.
A benchmark that keeps voice and wording apart
The data came from earnings-call recordings obtained through the EarningsCall API under a commercial license and aligned with DefeatBeta transcripts using WhisperX timestamps and speaker metadata. The authors started with 3,592 calls with available audio, then narrowed the set using three rules: answers from a single executive, at least four question-answer pairs per call, and responses shorter than 300 seconds. The final selection covered 60 calls and 49 unique tickers.
Two financial domain experts independently labeled every pair, then settled the final labels through adjudication. A separate check used five financial experts who independently rated a random subset of 52 responses for vocal confidence, producing Fleiss' kappa of 0.713, an agreement score across several raters. Those validation labels did not change the final labels, which came from the original two experts.
The gap was clearest in audio
On the text task, GPT-5 posted 87.4 macro F1 and 89.5 accuracy. Gemini-3-Flash was close, with 84.7 macro F1 and 89.1 accuracy. Agreement scores tell a similar story: GPT-5's Cohen's kappa with annotators was 0.813, compared with 0.866 for annotator-to-annotator agreement.
Audio was a different test. GPT-Audio's macro F1 was 58.5, and its F1 for answers classified as unconfident was 38.2. Gemini-3-Flash's vocal agreement with annotators was 0.225 on Cohen's kappa, far below the 0.774 agreement between human annotators. The contrast indicates that success with written evasion did not automatically carry over to reading confidence from a voice.
Speaker context helped, but only modestly
The researchers then asked whether the models were using raw acoustic values instead of judging a speaker against that person's usual baseline. After speaker-centering, Gemini-2.5-Flash's mean absolute acoustic correlation with the unconfident label fell from .195 to .032. The reported p-value was below .01. The authors interpret the change as showing reliance on absolute acoustic values rather than speaker-specific baselines.
That insight did not translate into a large performance jump. Speaker-level normalization raised unconfident F1 by 4.7 points for Gemini-2.5-Flash, 2.9 for Gemini-3-Flash, and 1.0 for GPT-Audio. The paper describes those gains as modest and says the gap with human performance remains.
Finally, the study explored whether the labels tracked post-earnings stock volatility. In call-level regressions, audio-based unconfidence generally explained more volatility variance than textual evasion, while combining the two dimensions produced the highest R2, the measure used for explained variance. The associations were stronger over v3 and v7 than over v30. The authors stress that this was exploratory and does not show that a vocal cue or evasive wording caused volatility.
The evidence is bounded by the benchmark's size and construction. It contains 505 pairs from 60 calls, and the route from 3,592 calls with available audio to 60 selected calls across 49 tickers may limit how broadly the results apply. The market analysis also uses call-level regressions on this small sample, and exact R2 values are not reported.
Taken together, the preprint points to two separate tasks. Textual evasion was the easier signal for the tested systems, while vocal-confidence detection remained weaker and speaker-aware adjustments helped only modestly. The document is marked arXiv version 1 and dated 28 August 2026.
Paper data and sources
Original title: A Shaky Voice Is Not Always a Dodge: Benchmarking Textual and Vocal Evasion Detection in Earnings Calls
Authors: Mirae Kim, Seonghun Jeong, Youngjun Kwak
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text