A new audio AI preprint reports a clear split in what two learned representations, compact numerical summaries of audio, are good at. The transformation embedding, zT, performed best when the system had to find a matching processing setup or transfer a processing style between different instruments. The processed-audio embedding, zy, was stronger when a downstream probe, a small model trained to read information from an embedding, had to estimate the processing chain and its settings.
The model kept two views of the same sound
The question behind the comparison was straightforward: should an audio representation describe the processing itself, or the sound after processing? The framework compared three training objectives: processing consistency, description alignment, and equivariance via forward prediction. It produced both zT and zy, giving the researchers a common basis for retrieval, probing and style-transfer tests.
How the comparison was built
Training used stereo music audio at 44.1 kHz from MedleyDB, MoisesDB and Mixing Secrets. For evaluation, the researchers used the MUSDB test subset and 67 Linux plugins with randomized parameters, covering processor chains from one to eight effects. The pipeline pre-rendered 1.5 million subbatches of 64 tuples, drew 16 subbatches into a full batch of 1,024, used random crops lasting 5 to 10 seconds, and trained for 100,000 steps on one NVIDIA H100 GPU.
The training comparison was an ablation, meaning the researchers switched each objective on or off rather than tuning a separate strength for every loss. The three loss weights were restricted to 0 or 1, producing seven non-trivial combinations. Each query was paired with a same-processing positive from a different source, plus hard negatives built from the query's exact source. The setup tests a trade-off between making processing portable across sources and retaining source information, but it does not establish that any individual objective caused the observed differences.
Retrieval favored the full combination
Retrieval favored the full mix of objectives. The combination of processing consistency, description alignment and forward-prediction equivariance reached 60.4% average accuracy. It found the matching processing in 87.3% of cases using the same stem, 63.2% within the same instrument, 45.5% across stems, 48.7% across instruments and 57.3% in the random condition. AFx-Rep and Fx-Encoder++ averaged 45.2% and 36.0%.
The full mix did not win every retrieval setting. The paper labels processing consistency LT, description alignment LP and forward-prediction equivariance Ly. Among single-objective versions, description alignment was strongest for within-source and random retrieval, while processing consistency was strongest in the cross-source conditions. LT + LP was the strongest pair, and the authors report a slight retrieval drop when Ly was added.
Longer chains were not a simple obstacle. Retrieval accuracy improved with chain length through the six-stage evaluation, then saturated quickly. Processor generalization gaps were small, but the source gap was much larger, and the authors identified unseen-source performance as the main bottleneck.
The embeddings kept different information
That split became clearer in probing tests. With the embeddings frozen, three-layer MLP probes read out chain and parameter information. For the full objective combination, chain estimation scored 53.8 on intersection over union, an overlap measure for predicted and actual chains; 18.1 on exact match; and 60.9 on F1, a balance of precision and recall. Parameter estimation had a mean absolute error of 17.5, the average size of the prediction error. Reported source-classification accuracies were 89.4 from zT and 95.1 from zy.
The source-classification results also show why complete separation is difficult. The authors report the lowest source leakage for zT when processing consistency was used, but note that some leakage is expected because audio processors are signal-dependent. In other words, the source can remain part of a useful representation even when the model is trying to emphasize the processing.
The gap widened when the source changed
zT's advantage returned in style transfer, where the system carried a processing effect from one source condition to another. For the full objective combination, the reported MRSTFT distance, a measure of output difference, was 0.390 ±0.010 for same-stem transfer, 0.575 ±0.016 within an instrument and 0.647 ±0.018 across instruments. The ± figures are reported 95% confidence intervals, and lower distance represents closer outputs. Within-stem results were comparable, with the reported paired-test value above 0.2; under source mismatch, zT outperformed zy and AFx-Rep, with the reported value below 0.001.
An oracle comparison showed how much depends on knowing the input. In cross-instrument retrieval, blind zT scored 48.7%, an estimated non-blind version scored 45.6%, non-blind zy scored 22.9%, and an oracle zT given the true input scored 85.9%. The oracle result indicates substantial headroom, but blind recovery of the transformation embedding remains an input-estimation problem.
A benchmark with clear boundaries
The study's reach is narrower than its numbers might suggest. The authors identify limited source diversity, Fx-normalization that requires source categories, structured processor descriptions unavailable for some plugins, and blind zT recovery as practical limitations. They leave open whether one model can capture source content, the transformation and the transformed audio together. No human or perceptual listening evaluation was reported, so the metrics do not show which output listeners would prefer. The results therefore apply to the rendered audio, processors, preprocessing and benchmark tasks evaluated in this work, rather than establishing performance for broader audio conditions.
The document is an arXiv version 1 preprint dated 28 August 2026. Its central message is a choice of tool for the job: zT for distance-based retrieval and cross-source style transfer, and zy for probe tasks that estimate processing details. The paper does not present a single representation that captures every aspect of source, transformation and transformed audio at once.
Paper data and sources
Original title: Exploring the Design Space of Representation Learning for Audio Transformations
Authors: Sungho Lee, Marco Martínez-Ramírez, Junghyun Koo et al.
Journal/Repository: arXiv
Status: Preprint, not yet peer-reviewed
First online: 2026-08-28
DOI: Not available
Original paper · Full text