A preprint reports that SMART had the best WER among the comparison methods it lists, while scoring 96.72 F1@50 on Large-scale KSL and 59.77 F1@50 on DS KSL. The system is designed to handle two linked jobs: sign-language recognition in continuous video and sign spotting, which identifies where signs occur. The findings come from benchmark model tests, so they do not by themselves measure communication outcomes for people who use sign language.
A system built around shared timing
SMART combines motion descriptions generated by a multimodal large language model, or MLLM, as auxiliary semantic cues with video-text alignment, a Multi-Scale Temporal Adapter, or MSTA, and a CSLR-aware CSFormer module for spotting. Its recognition branch uses a CLIP visual encoder with Spatial Adapters, Prefix embeddings and MSTA, followed by convolutional and recurrent sequence modeling, NormLinear and CTC. BERT text embeddings supply semantic supervision.
The evaluation used benchmark video collections: 8,257 PHOENIX14-T videos, 20,654 CSL-Daily videos, 35,987 Large-scale KSL videos and 28,250 DS KSL videos. The datasets cover German, Chinese and Korean data. The KSL datasets provide temporal boundary annotations for spotting, and the supplied spotting results are reported for those datasets rather than PHOENIX14-T or CSL-Daily.
Recognition was scored with Word Error Rate, or WER, where lower values are better. Spotting was scored with F1, a measure based on precision and recall. The paper reports SMART as best in WER across all four datasets; its row lists 17.58, 19.50, 26.50, 26.20, 0.64, 0.48, 27.27 and 23.92.
Where the numbers moved
On the two KSL tests, the spotting numbers were more specific. Large-scale KSL is reported at 96.72 F1@50 and 0.48 WER, while DS KSL is reported at 59.77 F1@50 and 22.93 WER. The supplied analysis gives point estimates only and reports no confidence intervals or repeated-run variability.
An internal comparison focused on CSFormer, the spotting module. In the reported test comparison, F1@50 changed from 7.37 to 96.72 on Large-scale KSL and from 5.90 to 59.77 on DS KSL. On DS KSL, WER changed from 23.92 to 22.93. The figures describe differences between the listed model configurations on these benchmark tests.
The recognition ablation compared several settings. On PHOENIX14-T, Dev WER changed from 18.60 for the baseline to 17.88 in the MSTA comparison. The alignment comparison was 19.80 versus 19.75 Test WER. The joint MSTA-plus-alignment configuration reported 17.58 Dev WER and 19.50 Test WER.
Other tests examined training choices. On PHOENIX14-T, batch size 8 was reported as the practical upper bound and best setting, with 17.58 Dev WER and 19.50 Test WER. InfoNCE was associated with 20.76 Test WER versus 19.80 without alignment, while SigLIP was associated with 19.75. In the tuning comparison, MSTA in the last four layers with a reduction factor of 32 gave 17.58 Dev WER; the last-eight-layer setting gave 19.39 Test WER and 17.88 Dev WER, while LoRA gave 24.40 Dev and 25.20 Test WER.
Spotting ablations compared configurations after CSLR features were included. Test F1@50 moved from 94.50 to 95.44 on Large-scale KSL and from 58.54 to 59.68 on DS KSL. The cross-attention variant reached 97.06 and 59.81, while the full-model DS KSL Test WER was 22.93 versus 23.72 for the listed comparison.
A benchmark result with clear limits
The results should be read as offline benchmark evidence. The supplied analysis reports no confidence intervals, standard deviations, repeated-run variability or inferential tests. It also says that generalization beyond the represented German, Chinese and Korean benchmark domains was not tested. For the KSL datasets, validation data were derived from the original training data in an 8:2 split. There was no direct measure of human communication, accessibility or user outcomes.
The authors interpret gloss-level recognition guidance and dense temporal supervision as complementary for continuous sign-language understanding. Whether that combination holds across additional languages, domains, signers and independently reproduced runs remains an open question.
The work reports support from IITP, funded by the Korean government's Ministry of Science and ICT, through the ITRC and ICAN programs, with each listed at 50%. The front matter identifies the document as arXiv:2608.25493v2, dated 31 August 2026.
Paper data and sources
Original title: SMART: MLLM-guided Temporal Alignment for Unifying Sign Language Recognition and Spotting
Authors: Eunjee Choi, JungHoon Sung, Seongwhan Cho et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text