An arXiv preprint describes LiDAR-SAM2, a framework that turns synchronized RGB video into labels for LiDAR point clouds, the 3D scan points used by the system, and then uses those labels to train an interactive LiDAR-only segmentation model. The design is meant to avoid human LiDAR annotation during training and to produce a consistent object track from a single click per object. The framework is built for 4D segmentation, keeping labels tied to objects across time.
From video masks to 3D tracks
To create the labels, the system applies SAM2 to synchronized RGB streams from multiple views. It projects each view’s masks into LiDAR coordinates, then combines them across views and time. Tracks from different views are merged when they share LiDAR points. For temporal propagation, the method uses ego-motion and majority voting within small 3D cells, called voxels, to extend and fuse mask IDs from frame to frame.
Training is split into two stages. The first learns frame-by-frame alignment between video-derived information and LiDAR; the second learns temporal propagation and consistency at the object level.
Scores rise, but the gap remains
The evaluation used SemanticKITTI 64-beam LiDAR scans synchronized with two forward-facing RGB cameras. The researchers trained on the dataset’s training split and reported results on its validation split. For semantic segmentation, they used mIoU, an average overlap score across 19 classes. For 4D panoptic segmentation, they used LSTQ, which reports association and classification components alongside separate IoU scores for stuff and thing categories.
A before-and-after check of the pseudo-labels showed an average of 22.12 segments before aggregation, with recall of 13.59% and precision of 91.90%. After aggregation, the average rose to 73.84 segments and recall to 79.58%, while precision fell to 85.93%. In plain terms, the method recovered far more segments after aggregation, while precision was lower but remained high. These are reported point estimates; the paper gives no uncertainty intervals or inferential tests for the comparison.
The downstream semantic results favored LiDAR-SAM2 labels over SAM2 labels, but not the full-ground-truth condition. With MinkU, mIoU was 55.2 using LiDAR-SAM2 labels, 29.8 using SAM2 labels, and 64.8 with full ground truth. With PTv2, the corresponding scores were 61.3, 26.4, and 69.0. In both tested backbones, models trained with LiDAR-SAM2 labels had higher reported mIoU than models trained with SAM2 labels, while still trailing models trained with full ground truth.
On the 4D panoptic measure, LiDAR-SAM2-label training again scored below full-ground-truth training. LSTQ was 46.9 versus 62.7 for 4D-PLS, 57.7 versus 67.0 for 4D-StOP, and 64.8 versus 70.5 for Mask4Former, respectively. Across all three backbones, the machine-made labels retained substantial reported performance, but they did not match the reference condition. As with the semantic results, no uncertainty interval or inferential test was reported.
Visual inspection offered a more favorable qualitative picture: the paper reports clean object boundaries and identities that stayed consistent over time. That assessment was not quantified, so it complements rather than replaces the numerical comparisons.
The practical question is still open
The tool test simulated annotation on four-frame sweeps. It used 0.2 m voxels for stuff components, kept carried masks only when their intersection-over-union, or IoU, exceeded 0.8, and reported roughly ten initial point prompts per frame. The framework is presented as supporting a single click per object, but this protocol does not establish that one click is enough for every object or for a complete scene.
The evidence is limited to one benchmark configuration: SemanticKITTI’s 64-beam scans with two forward-facing cameras. It does not show how the approach would perform on other datasets, domains, camera setups, or calibration conditions. The paper also refers readers to supplementary material for details of the range-view segmentation loss, but the supplied text does not state whether that material is available.
The authors interpret the results as support for a practical interactive LiDAR-only tool that could lower annotation burden with minimal prompting. But the supplied experiments do not directly measure human annotation time or cost, so that claim remains an interpretation rather than a direct workflow measurement. The document is an arXiv preprint, version 2, dated 30 September 2026. Its acknowledgement lists support from the Ministry of Education of the Republic of Korea and the National Research Foundation of Korea under the stated NRF grant.
Paper data and sources
Original title: Bootstrapping a 4D LiDAR Annotation Tool from Video Foundation Models
Authors: Jihun Kim, Hyun-Kurl Jang, Hyemin Yang et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-26
DOI: Not available
Original paper · Full text