The gap between knowing and doing
Large language models can often describe what a new Chinese internet word means, yet still fail when asked to reconstruct the linguistic form that produced it. That is the central finding of an arXiv preprint built around a benchmark of 4,759 Chinese neologisms. On the benchmark's direct definition task, Kimi-K2.5 led the field with 67.74% accuracy, while most open-source models scored below 40%.
But the study points to a more specific weakness than simple unfamiliarity. In a second tier of category-specific tests, all 18 evaluated models beat their own overall Tier 1 scores, and Kimi-K2.5 reached 81.94%. The two tiers ask different kinds of questions, however, so the jump cannot be treated as a causal test of one prompting method against another.
A test of word formation
CNeo-Bench divides the material into five top-level categories and nine subcategories, then separates description from operation. Tier 1 asks for a definition. Tier 2 asks the model to work with a word's source form: three subcategories use open-ended restoration and exact string matching, while six use multiple-choice questions with four options. The benchmark supplies example contexts when the mapping from a form to an answer is not unique.
The dataset passed through a filtering stage that yielded 4,917 entries. Requiring at least five returned RedNote posts removed 158 entries, leaving the final 4,759. Three part-time annotators labeled the entries into nine subcategories. Their raw three-way agreement was 78%, with Fleiss' kappa, an agreement measure, at 0.69.
The models were run under tightly specified conditions: temperature 0, greedy decoding, no thinking, and a single generation for each item. GPT, DeepSeek and Kimi systems were accessed through official APIs; the other models ran on a single H800.
Where the models fell short
GPT-5.1 scored 50.89% overall on Tier 1, below every Chinese-focused frontier model in the comparison. The gap was especially wide in two subcategories highlighted by the paper. For Pinyin abbreviations, GPT-5.1 scored 47.40%, against 76.56% for Kimi-K2.5. For unconventional characters, the figures were 21.74% and 82.61%.
The stronger Tier 2 results came in a format that gave models more structure, including multiple-choice options and, for some items, example context. The authors interpret the pattern as evidence that some knowledge becomes accessible when the task provides that support. Since Tier 1 and Tier 2 differ in format and demands, the result is an association within this benchmark, not evidence that context alone caused the improvement.
To look past the headline scores, the researchers tracked whether a model got both parts of the job right. They placed each item in one of four outcome groups: correct on both tiers, correct only on the definition, correct only on the operation, or wrong on both. The operation-failure rate was the share of definition-correct items that failed the operation task.
The gap was large in the categories that require a model to recover a particular form. For Chinese homophones, operation-failure rates ranged from 24.2% for DeepSeek-V3.2 to 54.1% for Gemma3-27B; Kimi-K2.5 was at 27.9%. For unconventional characters, the rate was 38.9% for Kimi-K2.5 and 57.1% for DeepSeek-V3.2, while other models were above 50%.
Across the three open-ended subcategories, the dominant Cell B failure was explanation or paraphrase. A model would give a meaning-equivalent response instead of reproducing the requested source form. The pattern matters because a definition can be semantically adequate while still missing the sound, character or other formal construction that the diagnostic asks it to manipulate.
Examples helped, but not enough
The researchers also created a hard subset of 1,058 entries, equal to 22.23% of the benchmark. These were items that GPT-5.1, DeepSeek-V3.2 and Kimi-K2.5 all missed when asked, without examples, to generate a Tier 1 definition. The shared misses show that the benchmark contains difficulty that was not limited to one system.
Examples recovered some of those cases. One-shot prompting brought back 37% to 50% of the hard items across models, while three-shot prompting recovered 53% to 67%. Even after three examples, 33% to 47% remained unrecovered. The examples were generated by an independent LLM and manually filtered, so the result shows a change under the study's prompting setup, not a tested solution to the underlying weakness.
A focused warning, not a general verdict
There are limits to what the scores can establish. Tier 1 was judged by an LLM on binary correctness, with empty or degenerate outputs forced to zero. On a stratified sample of 1,000 instances, the judge agreed with human annotators 95% of the time. That validation supports the benchmark's scoring check, but the study does not establish general Chinese-language competence or human-level performance beyond this collection.
The authors suggest that repeated failures may reflect missing exposure or weak activation of the relevant forms, and point to explicit representation of phonetic or visual substrates as a possible direction. Those explanations remain interpretations: the benchmark diagnoses the gap but does not test mitigation strategies.
The manuscript is an arXiv preprint, and it says the data and code will be released upon acceptance. For readers testing Chinese-language AI systems, the practical message is narrow but useful: recognizing a neologism's meaning is not the same as being able to reconstruct how it was made.
Paper data and sources
Original title: CNeo-Bench: Diagnosing Large Language Models on Chinese Neologisms
Authors: Kaiyan Zhao, Zhongtao Miao, Zheyong Xie et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text