An AI model that used speech-act information alongside the words themselves produced the strongest results in tests designed to forecast when online conversations would derail. Its lead was clearest with limited training data, but the advantage did not transfer evenly from one dataset to another.
An arXiv version 2 preprint dated 9 September 2026 describes the modeling study. It asks whether speech-act information can act as a low-dimensional pragmatic representation, or a compact description of what language is doing in context, that could reduce lexical noise and improve forecasting generalizability.
Reading more than the wording
To create the extra signal, researchers prompted open-source gpt-oss-120b with a taxonomy of 50 speech-act verbs plus an other label. The signal was fed into the forecasting systems alongside the text. Hparallel used speech-act detection as an auxiliary learning task; Hsequential prioritized speech-act probabilities in a GRU-based sequence model and trained the two tasks together, without an intermediate speech-act-only phase.
The tests used WIKI, CMV and GITHUB dialogue datasets. WIKI contained 4,188 conversations, CMV 19,578, and GITHUB 202 derailed dialogues plus 696 civil dialogues. The data were split 60 percent for training, 20 percent for validation and 20 percent for testing, while GITHUB dialogues were cropped to their 10 most recent utterances.
The evaluation combined turn-level probabilities into a conversation-level forecast and tracked performance across several training sizes using an area-under-the-learning-curve measure. Per-timestep predictions were assessed from one to five steps from the target event, with AULC calculated across training subsets of 300, 500, 1,000, 2,500 and 11,802 dialogues. Results were averaged over five random seeds.
The edge was clearest with less data
On the in-domain tests, Hparallel had the strongest Dynamicmean AULC result for all three datasets. The measure combined the model's turn-by-turn probabilities into a conversation-level forecast while tracking performance across training-set sizes. Its Macro-F1 and AUPRC scores were 62.50 ± 0.98 and 66.85 ± 0.71 on WIKI; 65.29 ± 0.45 and 67.94 ± 0.26 on CMV; and 71.55 ± 3.47 and 66.84 ± 1.65 on GITHUB.
That lead was clearest in the low-data and turn-by-turn analysis. All models improved as dialogues moved closer to the target event, and Hparallel had the highest per-timestep performance, especially with limited training data. At the largest reported training size, 11,802 dialogues, the text-based PLM comparator marginally outperformed Hparallel.
The advantage did not travel evenly
The picture changed when a model trained on one dataset was tested on another. Transfers into GITHUB favored Hsequential, followed by Hparallel. For GITHUB to CMV, Hablation had the highest Macro-F1 and Hparallel the highest AUPRC, while the peak Macro-F1 was 36.19.
The two metrics also told different parts of the story. Macro-F1, which depends on a decision threshold, fell more sharply under domain shift than AUPRC, which is threshold independent. Distance analyses found GITHUB to be the most divergent dataset and slightly closer to WIKI than to CMV. Hparallel performed best between the more similar WIKI and CMV datasets.
The authors link that pattern to the way the two architectures use the signal. They interpret Hsequential's emphasis on speech-act probabilities as useful on more divergent transfers, while Hparallel's combination of speech-act and textual features appears better suited to the more similar WIKI and CMV datasets.
The signal still needs testing
The speech-act labels were checked against manual labels, but the check covered only 85 sentences from 30 randomly selected responses, including 10 responses from each dataset. One author supplied the manual labels. Agreement, measured with Cohen's Kappa, was 0.53 across 51 fine-grained classes and 0.88 after the labels were grouped into five broad categories. The study describes those results as moderate agreement at fine granularity and almost perfect agreement at the broad level.
The main pattern also held in two robustness tests. When BERT replaced RoBERTa-large, Hparallel remained strongest in-domain and speech-act-based models remained strongest in the reported cross-domain tests. In a GraphNLI ablation, the version with speech-act supervision reported higher performance in both in-domain and cross-domain settings.
The evidence remains bounded by the evaluation design. The study compared three datasets, and one of its cross-domain directions, GITHUB to CMV, was weak. Comparisons used paired t-tests with a p-value threshold below 0.05, but the authors note that five random seeds make normality testing difficult and give some significance analyses limited statistical power.
The document is an arXiv version 2 preprint dated 9 September 2026, and its conclusion provides an anonymous code-repository URL.
Paper data and sources
Original title: Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecasting
Authors: Angela Yifei Yuan, Christine De Kock, Christopher Leckie
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text