DeepSeek Cloud Model Ties Local LLM in Compliance Extraction Test

A team of four researchers, led by Khuong Nguyen-An, submitted a paper on September 23, 2026 describing RoPA Manager, a system meant to solve a narrow but pressing compliance problem in Vietnam. Under Law No. 91/2025/QH15 and its implementing decree, which became binding on January 1, 2026, organizations operating in the country must keep detailed Records of Processing Activities describing how they collect and use personal data. Writing those records by hand is slow, and sending the underlying documents to a cloud-hosted large language model can conflict with data sovereignty rules that many organizations now have to follow.

RoPA Manager answers that tension by combining hybrid retrieval, lexical ranking over tsvector, dense vector search and Reciprocal Rank Fusion, with locally deployed language models that never send source documents off premises. To test it, the authors built a benchmark drawn from 32 organizations, covering 77 processing activities across 12 field groups. End to end extraction against that benchmark reached token coverage between roughly 50 and 55 percent of the reference labels, a figure the paper is careful to separate from the scorer’s own robustness rating.

The most notable comparison sits inside that benchmark. Across 32 paired scenarios run on a single 24 GB GPU, a locally hosted Qwen3.5 model showed no statistically significant difference from the cloud-based DeepSeek model the team used as a comparison point, with the gap measured at 0.20 percentage points in DeepSeek’s favor and a p-value of 0.72. A third model, Gemma-4-31B, performed significantly worse in the same test, which the authors flag with a p-value under 0.01.

The numbers

Two independent experts reviewed 1,558 of the benchmark’s reference values, a little over a third of the full set, and found no incorrect entries, reaching 99.68 percent agreement between reviewers. The automated scorer itself, tested separately on perturbed data without calling any model, reached an F1 score of 0.9493. Those two figures measure different things: one checks whether human experts agree with the benchmark’s labels, the other checks whether the scoring code can tell a right answer from a wrong one, and the paper is explicit that neither substitutes for end-to-end extraction accuracy, which the token coverage figures address instead. The work is due to appear at the VNICT 2026 conference in Hanoi in November.