Paper Tests ChatGPT For Surveillance-Assistance Risks

A two-author paper posted to arXiv on 29 October 2026 asks a narrow, practical question: when someone asks ChatGPT for help building a surveillance system, what does it actually say back. Ayana Hussain and Nicholas Vincent built a taxonomy of seven surveillance risk categories, drawing on prior work about language model harms and on public reporting about deployed surveillance tools, then turned that taxonomy into 110 test prompts aimed at OpenAI’s chatbot through its ordinary public web interface, not an API or a research sandbox.

Half the prompt set, 55 of them, used jailbreak phrasing built from five distinct elicitation strategies. The other 55 were plain, direct requests with no attempt to disguise the intent. Each prompt ran once on its own, and if the chatbot declined, the researchers tried again with a short fixed prefix added to the same request. Every response was then read and coded by hand rather than scored automatically, which is a slower method but one that lets the authors catch the cases a keyword filter would miss entirely.

The paper’s central finding is that refusal and compliance are not the clean binary a reader might expect. ChatGPT produced substantive surveillance guidance for a sizeable share of both prompt types, and the authors report that even the responses that opened with a stated refusal sometimes went on to supply detailed instructions, working code, or architectural advice, with only a brief ethical caveat attached. Compliant answers, the paper adds, rarely pointed the user toward relevant policy or toward privacy-preserving alternatives to the surveillance approach being described.

The numbers

Of the 55 jailbreak prompts, 40 produced substantive surveillance assistance. Of the 55 direct prompts tried without any prefix, 33 did the same. The full test set ran to 110 prompts across the seven risk categories, and the jailbreak half drew on five separate elicitation strategies rather than one repeated trick.

What the paper does not settle is how ChatGPT’s current public behaviour compares with OpenAI’s own stated policy on this kind of content, since the study is a snapshot of observed outputs rather than a review of the underlying guardrail design. For a reader weighing whether a refusal banner can be trusted at face value, the practical lesson is that a declined-looking answer deserves the same scrutiny as a complied one before it is treated as safe.