Perplexity AI Researchers Detail LoRA Recipe For Converting Four Language Models To Diffusion

Diffusion language models generate text by refining a fully masked sequence in parallel rather than producing one token after another, which in principle allows faster and more flexible output than the usual autoregressive approach. Building one from scratch has mostly meant expensive pretraining or fine-tuning every parameter of an existing model. The paper published on arXiv on October 30, 2026, proposes a cheaper middle path called BYOD, for Build Your Own DLM.

The method attaches LoRA adapters, small trainable additions that sit on top of a frozen base model, to four openly available models: Gemma 2 9B, Llama 3.1 8B, Qwen2.5 7B and Ministral 8B. The authors say this is the first reported conversion of a Mistral-family model into a diffusion generator. None of the base model’s own weights change during conversion, only the adapters do, and those adapters make up between 4.1 and 5.8 percent of the resulting model’s final parameter count. Each conversion runs for 25,000 iterations, needs at most 102.4 million tokens, and finishes in 3.1 to 4.0 hours on a single Google Colab GPU, a budget closer to a weekend side project than a training run requiring a data center.

All four converted models can generate text from a fully masked starting sequence and were shown in a controlled attention test to use context from both directions, the defining behavior of a diffusion model rather than a left-to-right one. The strongest conversion, built on Gemma, reached 50.6 percent average accuracy across nine evaluation tasks, close to the 51.9 percent scored by LLaDA 8B Instruct, an existing diffusion model trained from scratch, in the same test harness.

The paper is upfront about where the method still falls short. When generating 128 tokens in just 64 steps, a setting chosen to keep diffusion fast, BYOD’s perplexity ranged from 8.3 to 11.1, well above the 4.1 LLaDA achieved and far above the 2.2 to 3.0 the original autoregressive parent models scored on the same measure. The authors frame this gap, not the conversion cost, as the problem future work needs to close before adapter-based diffusion conversion becomes a practical substitute rather than a research demonstration.

The numbers

The adapters span 4.1 to 5.8 percent of each model’s final parameter count, each conversion needs at most 102.4 million tokens, BYOD-Gemma reached 50.6 percent average accuracy against LLaDA’s 51.9 percent on the same nine tasks, and generation quality at low step counts remains the open problem the authors flag for future work.