Paper Finds Profession-Specific Prompts Add Cost Without Accuracy Gains on Gemini

A paper posted on September 7, 2026 tested whether detailed, profession-specific system prompts actually make a model better at scientific tasks, or just make it talk more. Researcher Timothy Kassis ran Gemini 3.8 Flash, accessed through OpenRouter inside the Pi agent harness, against a corpus of 503 AGENTS.md profiles built for specific professions, each matched to a scientific benchmark question. Every matched profile was compared against four simpler controls: a bare “You are a helpful assistant” baseline, just the profile’s opening role sentence, a generic scientific-rigor instruction, and a profile pulled from an unrelated field.

Across nine text-based science benchmarks sampling 4,531 questions, 4,488 items finished all five prompt conditions after retrying API errors, and answers were graded automatically with rule-based scoring. The headline result is a wash: the average accuracy difference between a matched profile and the plain baseline was -0.6 percentage points, with a bootstrap interval spanning both sides of zero, and no single benchmark showed a statistically clear improvement either way.

What the long prompts reliably changed was the bill, not the score. Matched profiles produced 1.5 to 2.3 times as many output tokens as the baseline and cost 2.2 to 4.5 times more per successful call. On a tool-using bioinformatics benchmark, the longer prompt actually pulled the solve rate down, from 56.7% at baseline to 46.7% with the profile loaded, because the extra text pushed more runs into token or time limits. The one place a longer prompt helped was resilience to a flaky API: on one benchmark, frequent provider drops left the short baseline with a correct first-pass answer on only 54.0% of items, against 71.6% for the long profile, an effect the paper attributes to prompt length and formatting rather than any domain expertise in the text.

The numbers

• Accuracy difference, profile minus baseline: -0.6 percentage points, with no benchmark showing a clear gain.
• Output tokens: 1.5 to 2.3 times as many under the profile condition.
• Cost per successful call: 2.2 to 4.5 times higher under the profile condition.
• Corpus size: 503 profession-specific prompt profiles evaluated.

For a team paying Gemini’s per-token API rates, the practical read is that a detailed persona prompt is closer to a cost decision than an accuracy one, and worth testing against a short baseline before it becomes the default system prompt.