Paper Proposes Detecting Who Contributed an Insight, Not Just Who Wrote It

Researchers from a multi-university team posted a paper on arXiv on October 5, 2026, arguing that detecting AI-written text is no longer the useful question for peer review. Their proposed task, Insight Provenance, instead asks whether the underlying judgment in a review comment came from a human, a large language model, or some hybrid of the two. The distinction matters because a reviewer can run a human draft through a paraphraser and defeat a detector built to spot machine phrasing, while the actual intellectual contribution behind the comment stays invisible to that kind of check.

To study the problem, the team built InsightProv-v0, a dataset constructed from scientific papers and human reviews, with different levels of LLM involvement simulated using three models: GPT-4o, Gemini, and DeepSeek. Provenance was annotated at the sentence level, which let the authors test not just whether a review contains machine-generated text but which specific sentences carry a human idea versus a model-generated one.

The paper’s central finding is a warning about how these systems are evaluated. Strong accuracy on raw data looked convincing at first, but the researchers found that models were exploiting linguistic and textual-authorship shortcuts rather than genuinely detecting provenance, and that performance degraded substantially once the evaluation was progressively debiased to remove those shortcuts. Their response is a two-stage adversarial framework meant to suppress the shortcut signals while keeping the information that actually indicates where an insight came from.

Beyond detection accuracy, the analysis points to a pattern in how humans and AI models differ. AI-generated insights tend to stay close to generic knowledge or information already present in the paper being reviewed, while human insights more often pull in outside knowledge and independent judgment that was not supplied by the paper itself. The authors frame this as evidence that wording alone is not a durable signal: an LLM can be asked to rewrite a sentence, but the provenance of the idea behind it leaves a signal that survives the rewrite.

The numbers

• The dataset draws on 4,057 scientific papers as its source material.
• It includes 12,660 human reviews, annotated for provenance at the sentence level across three simulated levels of LLM involvement.

The paper does not propose a product or a policy. It proposes a different question for anyone building AI detection tools for academic publishing, shifting the target from matching a writing style to tracing where a judgment came from.