Instruction fine-tuning was the top-performing condition for language models generating English glossaries grounded in Classical Sanskrit verses, according to an arXiv preprint. The strongest listed result came from phi-4, which scored 0.716 on Jaccard, a measure of overlap between generated and reference glossary phrases, and 0.787 on Meaning Faithfulness, an embedding-based measure of how closely generated meanings matched the references. The instruction-fine-tuned condition ranked highest for each listed language model on both measures, ahead of prompting and the study's alignment and sequence-to-sequence baselines.
The task is narrower than translating a verse from scratch. Given a Sanskrit sloka and its English translation, the system must recover meaningful Sanskrit phrase boundaries and attach meanings grounded in that translation. The setup is designed to reflect traditional patha commentary.
A benchmark built from two Sanskrit traditions
The benchmark contains 31,316 sloka, translation and glossary triples drawn from the Valmiki Ramayana and Srimad Bhagavatam. Each example pairs a Sanskrit verse with an English translation and a glossary mapping Sanskrit phrases to meanings. The annotations were converted into key-value dictionaries, with light normalization of Unicode and whitespace while preserving the linguistic structure.
The researchers used 25,050 instances for training, 3,133 for validation and 3,133 for testing. Contiguous verses and near-duplicate glossary entries were kept from crossing the splits to reduce leakage between the material used to build the systems and the material used to assess them.
The comparison covered gemma-3n-E4B-it, gemma-3-12b-it, phi-4 and Qwen3.5-9B. It also included FastAlign and ByT5-Sanskrit sequence-to-sequence and pipeline systems as baselines. The language models were tested with zero-shot prompts, prompts containing one, five or 10 examples, and instruction fine-tuning using LoRA.
The gain extended beyond familiar glossary entries
For phi-4, seen-key accuracy was 21.53% with zero-shot prompting and 87.85% with instruction fine-tuning. For unseen keys, it was 6.30% with zero-shot prompting and 66.45% with instruction fine-tuning. The authors interpret the similar gains on seen and unseen keys as evidence of generalizable segmentation and glossing behavior.
The fine-tuning used LoRA with shared settings including a scaling parameter of 16, 0.05 dropout, a learning rate of 3 times 10 to the minus 5, 0.15 warmup, BF16 precision, gradient clipping at 1.0 and 10 training epochs. Ranks, target modules, batch sizes and sequence lengths varied by model. The reported setup used memory-constrained LoRA rather than full fine-tuning.
Finding the right phrase boundary was still the hard part
A closer look at the weakest outputs pointed to a specific failure mode. The qualitative review examined 174 samples from the bottom 5% by Meaning Faithfulness. Over-segmentation, in which a phrase is split into too many pieces, accounted for 70% of the reviewed cases, or 122 samples. General segmentation problems appeared in 22 cases, both over- and under-segmentation in 12, and under-segmentation in nine. Four cases involved semantic or translation errors, while five had no identified issue.
The segmentation ablation showed how much this bottleneck matters. Meaning Faithfulness was reported as 0.787 without gold segmentation and 0.872 with gold segmentation. Predicted segmentation added little. Gold segmentation is an oracle-style condition, however, because it supplies information that a deployable system would have to discover for itself.
The authors identify morphology and compound-boundary detection as the primary bottleneck. Their conclusion emphasizes that instruction fine-tuning, segmentation supervision and changes in word order all mattered to performance.
Shuffling the verse exposed sensitivity to syntax
The models also struggled when the words within each sloka line were randomly shuffled. Every reported model scored lower after the perturbation. For Qwen3.5-9B, Jaccard fell from 0.693 to 0.394. Under the shuffled condition, gemma-3-12b-it scored 0.587 on Jaccard and 0.750 on Meaning Faithfulness; phi-4 scored 0.632 and 0.768; and Qwen3.5-9B scored 0.394 and 0.682.
The benchmark scores are automated measures, not direct human judgments. Jaccard uses surface overlap between generated and reference key sets. Meaning Faithfulness matches keys using normalized string similarity, with a 0.7 threshold, then compares meaning embeddings with cosine similarity; unmatched keys contribute zero. These choices can penalize morphologically equivalent ways of resolving Sanskrit phrase boundaries, and the supplied analysis does not show that the embedding score agrees with human judgments.
What the result does, and does not, establish
The evidence comes from a computational benchmark based only on the Valmiki Ramayana and Srimad Bhagavatam, both described as narrative works with comparatively regular syntax. It therefore does not establish transfer to additional Sanskrit genres, ornate Kavya or Vedic prose, and it does not demonstrate robustness to every form of free Sanskrit word order.
The comparisons are descriptive and nonrandomized, so they show differences between model conditions rather than a causal effect of instruction fine-tuning or segmentation. The error review covered only the bottom 5% of samples, and the gold-segmentation result is an upper-bound style reference rather than a directly comparable system. The analysis leaves open whether larger budgets or full fine-tuning would close the remaining segmentation gap.
The document is an arXiv preprint, identified as version 2 dated 1 September 2026. Its abstract states that the data and code are available. The work reports partial support from the GCP Research Grant for Gemma and India's National Language Translation Mission, Bhashini, with GPU support from Svarupa.
Paper data and sources
Original title: Padamitra: Grounded Glossary Generation for Classical Sanskrit
Authors: Manoj Balaji Jagadeeshan, Sai Pragnaan Marala, Pawan Goyal
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-25
DOI: Not available
Original paper · Full text