A preprint reports that language models trained on generated writing trajectories built from scientific papers scored higher on an automated writing average than a direct supervised fine-tuning setup. The reported difference was 2.44 points, or a 4.7 percent gain, for the model using the 4B-generated paper data.
The approach starts with a finished paper but expands it into a sequence of writing steps. A record begins with a writing request and a plan, then pairs a deliberation with each retained section and its verbatim text, before adding a review of the abstract and the abstract itself.
Turning papers into writing trails
The same paper-derived construction was used in two other ways. It created instruction examples for supervised fine-tuning and tasks based on held-out papers, with rubrics and checklists for the PAW-Bench writing benchmark.
The scale was large: 1.8 million quality-filtered arXiv papers produced a 60 billion-token continued-pretraining corpus, described as roughly twice the source text. The reverse construction also produced about 200,000 supervised fine-tuning samples and 2,940 PAW-Bench academic writing tasks.
For the model comparisons, each trajectory mixture used 30 billion tokens from a generator and 20 billion FineWeb-Edu tokens, for a 50 billion-token training set. The controls used the same budget with FineWeb-Edu alone or with raw paper text plus FineWeb-Edu.
Higher writing scores in an automated test
The writing tests favored the paper-derived data in the main comparison. Under DeepWriting supervised fine-tuning, the Our-Data-4B model recorded the highest average among the continued-pretraining models, 2.44 points above direct fine-tuning, a reported 4.7 percent gain.
The paper also tested whether changing the fine-tuning mix altered the result. Replacing 10,000 DeepWriting samples with paper-derived samples corresponded to a rise in PAW-Bench from 55.42 to 63.73 and in checklist pass rate from 0.529 to 0.650. Adding trajectory continued pre-training yielded a PAW-Bench score of 64.13 and an overall score of 54.38.
Those writing results came from a fixed subset rather than the full benchmark: 300 tasks were sampled without replacement from 2,940 PAW-Bench tasks, with seed 42 used for every model. The same responses were then rescored with Gemini 3.1 Pro and Seed 2.0 Pro. Under every judge, all five paper-data configurations scored above all five reference configurations on the average.
Reasoning held up, but the test remains narrow
The paper-data training did not show the same drop on its reasoning checks as FineWeb-Edu-only continued pre-training. The direct-fine-tuning reasoning average was 46.32, while FineWeb-Edu-only continued pre-training reached 44.30. Replacing 30 billion tokens with paper-derived data produced averages from 45.66 to 46.82, which the paper describes as close to direct fine-tuning.
Additional tests examined paper comprehension. On Qasper, the Our-Data-27B model was more than 3 points above direct fine-tuning, while Plain-Paper continued pre-training was nearly 4 points below it. All three paper-data models outperformed both comparators on QASA.
On LongBench v2, the Our-Data-9B and Our-Data-27B models were ahead of direct fine-tuning in both reported context brackets. The differences were 2.6 and 2.2 points for the 9B data, and 6.0 and 3.3 points for the 27B data. Plain-Paper continued pre-training fell 5.06 points when the bracket widened to include contexts up to 64k tokens.
There is a clear boundary to what the study can establish. The continued-pretraining experiments used one base model, Qwen2.5-7B, and one mixture of 30 billion trajectory tokens plus 20 billion general-text tokens. Larger models and other mixture ratios were not evaluated.
The work remains an arXiv preprint, version 2 dated September 7, 2026. Within the tested setup, the reported signal is an association between paper-derived trajectories and higher writing scores, with reasoning results staying close to direct fine-tuning.
Paper data and sources
Original title: Unfolding Scientific Papers into Multi-Turn Generation Trajectories for Continued Pre-Training
Authors: Qiankai Xu, Qiguang Chen, Zixin Su et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text