Paper Compiles LLM Routing Decisions Into Parallel Inference for RAG

A model that acts as a controller inside a structured pipeline still has to speak its decisions out loud, one token after another, even when the decision itself is simple: which of several document sections to open next. In hierarchical retrieval augmented generation, that pattern repeats at every level of a document tree, and the paper argues the serial decoding used to enforce it is not actually required.

The authors frame the problem around two existing options. A model can emit free-form JSON describing which sections it wants, which is quick but fails often enough to need repair. Or it can be forced through grammar-constrained generation, which is reliable but strictly serial, so each branch of the decision waits for the one before it to finish. The paper’s claim is that for this kind of bounded-horizon control, the set of reachable next actions forms a shallow tree that can be worked out in one prefill and one further probe pass, rather than walked one step at a time. A lemma in the paper is used to argue that the compiled version produces the identical policy, in exact arithmetic, to the slower serial method it replaces.

The method was tested across four document-disjoint data splits and at two model scales, run primarily on a MetaX device with a repeat run on an H800 chip to check that the ordering of results held across hardware.

The numbers

The paper reports that compiled control matched the serial, token by token policy on between 84 and 89 percent of questions across the tested splits, and states that every case where the two methods diverged was a floating point near-tie rather than a genuine disagreement. Compared with the slower token-wise serial decoding baseline, the compiled approach is reported to cut control latency by 35 to 47 percent. Against an idealized chunked decoding engine, a stronger baseline that already batches some of the work, the reported saving narrows to 6 to 7 percent. The submission itself runs 5 pages with 2 figures and 1 table, according to its listing.

The gap between the 35 to 47 percent figure and the 6 to 7 percent figure is the detail worth sitting with: most of the available speed-up comes from avoiding naive serial decoding at all, and only a smaller slice comes from this specific compilation trick once a team has already built a reasonably efficient batched pipeline. That matters for anyone deciding whether to adopt a new inference technique or simply fix an unoptimized baseline first.