A new preprint reports that a language model generally performed better on long-context tests when it was fine-tuned with the same sparse attention and KV-cache policy used during inference. The comparison was against sequence-parallel fine-tuning, an approach that keeps exact attention during training. The authors describe matched-policy fine-tuning as the stronger option across the evaluated setups, although PopQA may be an exception.
One of the clearest gaps appeared under the h2ono2k cache policy on NQ. At a 64k-token context, the matched-policy setup scored 45.2, compared with 16.2 for sequence-parallel training. At 128k tokens, the corresponding scores were 38.5 and 14.5.
The comparison addresses the paper's central engineering question: whether training under the same sparse cache policy helps when that policy is used at inference. The supplied evidence is limited to the model, benchmarks and configurations tested here.
A targeted engineering test
The experiments used one Qwen3-4B-Instruct-25078 checkpoint and the Helmet long-context benchmark. The main evaluation covered four tasks at both 64k and 128k tokens, using five cache policies and two chunk sizes, along with eight additional tasks at 128k.
The proposed method is designed to work with arbitrary KV-cache policies. Its components include nested activation checkpointing, CPU offloading and a linear KV-cache buffer recurrence, a way of updating the cache information carried through the training process.
In the reported matched-policy arm, the cache length was 32,768 tokens. The tests used chunks of 1,024 or 2,048 tokens, 8-bit KV buffers, four devices and an effective batch size of eight. The model used LoRA-only training with rank 16 and alpha 16, AdamW, a base learning rate of 0.0005 and up to five epochs.
The comparison asked how much performance changed when the sparse policy used at inference was also present during fine-tuning, rather than using sequence or context parallelism with exact attention during training.
The details changed the outcome
The choice of cache policy mattered. H2O variants generally outperformed lastrec and smart-lastrec, while the plain h2o variant tended to trail normalized and original versions. A 2,048-token chunk performed about as well as a 1,024-token chunk and ran faster in the reported comparisons.
The two training strategies also produced notably different output lengths. The paper defines R as the generated output's length divided by the target answer's length. It also uses p128 to indicate an output that reaches exactly 128 tokens. Matched-policy outputs had R close to 1, while sequence-parallel outputs had much larger ratios and p128 was often close to 100 percent, suggesting that many outputs hit the maximum length used in the diagnostic.
That alignment came at a computational cost. On 128k-token datasets with 2,048-token chunks, the reported training-update overhead was about 25 to 30 percent compared with exact training.
A result sensitive to the yardstick
The ranking was not identical under every scoring rule. With SubEM, sequence-parallel models often scored better. The authors argue that SubEM can reward verbose outputs and recommend against using it for Helmet, making the choice of metric an important part of how these results are read.
The paper also describes a LongBench V2 experiment, with 402 cases for training, 20 for validation and 81 for testing. But the supplied results section is explicitly marked as pending, so it provides no LongBench V2 performance result to compare.
The evidence remains narrow because the experiments used one reported model and LoRA-only training, while the supplied LongBench V2 results are pending. The main comparisons are reported without formal significance testing or confidence intervals.
For readers assessing this engineering approach, the practical message is specific: test whether fine-tuning with the inference-time cache policy improves the same benchmarks. The paper's comparison is bounded by the evaluated model, benchmarks and configurations, and includes a possible PopQA exception.
The work is identified in the supplied front matter as arXiv version 2, dated 28 September 2026. No funding or conflict-of-interest statement is reported in the supplied text.
Paper data and sources
Original title: Learning how to Forget: Fine-tuning for Long-Context Sparse Attention
Authors: Matthias Seeger, Zeyu Zhang, Vihang Patil et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-20
DOI: Not available
Original paper · Full text