Systems built to spot hallucinated text in image-conditioned answers performed only moderately in a four-language evaluation, and the order at the top was less secure than a single leaderboard suggested. The best average score was 0.58 for character-level correlation, 0.46 for correlation that also required the right hallucination label, and 0.51 for overlap between predicted and annotated spans. Those results were 30 to 40 points above the reported baselines, but most mean scores were below 0.4 and no system exceeded 0.6 on any language or metric.
The task asked each participant to identify hallucinated character spans in a response conditioned on an image and assign every span to one of five categories. In practical terms, a system had to identify both the location and category of a hallucination.
A broad test and controlled generation
The SHEEP dataset contained 20,000 samples across four languages. It combined outputs from five large vision-language models with 1,600 human-written items, and used both random sampling and LLM-judge-assisted preselection. The shared task drew 27 teams and 623 submissions across the same four target languages.
Responses came from four open models at 8-billion-parameter scale and one larger Gemma3-27B model. The generation setup used a temperature of 0.7, capped outputs at 512 tokens and used five random seeds per input. The task's training phase began with about 15,200 annotated samples, while evaluation ended on a hidden test set of about 4,800 samples, with roughly 1,200 per language.
Three ways to judge a detector
The organizers separated three parts of the job. IoU measured whether a system's predicted span overlapped the annotated span, without using its confidence or category. Corr measured whether the system's character-by-character confidence pattern matched annotator agreement. Corrlbl went a step further by requiring the hallucination to be put in the correct category.
Corr was calculated as a Spearman correlation between the gold character-level vector and the predicted vector. Spearman correlation is a way of measuring whether two ranked patterns line up. When either vector was constant, the benchmark used exact-match scoring for whether a hallucination was present or absent.
The leaderboard was not a clean verdict
A point estimate did not settle the race. The evaluation team drew 25,000 bootstrap samples, repeatedly resampling the test data, to calculate 95% confidence intervals for scores and ranks. It also reported the probability that one submission outranked the next-ranked submission. In English, the top team's mean rank was 2.9, but its 95% rank interval ran from first to tenth.
That uncertainty was not confined to English. Intervals for leading systems extended as far as 10 positions in Chinese, 14 in Italian and French, and 15 in English. The implication is that small differences near the top should be treated cautiously, because resampling could move systems several places.
On the reported correlation and IoU measures, TÜRKSAT led English, while vroom-vroom led French, Italian and Chinese. TÜRKSAT posted 0.55 correlation and 0.48 IoU in English. vroom-vroom posted 0.58 and 0.52 in French, 0.56 and 0.48 in Italian, and 0.61 and 0.53 in Chinese.
Example choice changed the comparisons
The comparison found different ranking stability depending on which examples were used. Rankings from Human-written and Silver examples were highly similar, with Spearman correlations ranging from .861 to .973. Rankings involving Random examples were generally less consistent, with the strongest sensitivity in Chinese on the label-conditioned correlation measure. Reported score levels also varied by language, metric and evaluation strategy.
The blind spot in empty predictions
Another result complicates the scores: systems often returned no marked hallucination span. Empty predictions appeared in 34.87% of 4,898 test instances, while 18.51% of the total test set had empty annotations. The difference does not by itself show that systems were wrong, because an empty prediction may be an appropriate abstention, but the benchmark did not directly settle that question.
For test cases without marked hallucinations, empty annotations ranged from 15.5% to 25.5%, while empty system predictions ranged from 30.7% to 46.0%. The authors suggest adding an explicit empty-instance or abstention metric so that a detector is judged on whether declining to mark a span was appropriate, not only on its other scores.
What the results can support
The study's conclusion is narrower than a conventional winner's list. It shows how these systems compared on this benchmark, but it does not establish that any one system is reliably superior outside the reported data. Nor can the benchmark establish performance across broader real-world settings.
The benchmark has limited coverage of languages, domains and source model outputs, and its static design may permit leakage or overfitting even though test annotations were kept private. The overview's narrower message is that fine-grained multilingual detection remains difficult and that leaderboard positions should be reported with uncertainty alongside results from more than one example-construction strategy.
The analysis comes from an arXiv version 2 preprint dated 28 August 2026.
Paper data and sources
Original title: Overview of SHROOM-Visions 2026: A Shared Task on Hallucination Detection in Large Vision-Language Models
Authors: Raúl Vázquez, Aman Sinha, Chuyuan Li et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text