No Magic Words · Buy the book →
If you’ve read anything about prompting, you’ve been told some version of three things: assign the model a role (“you are an expert marketing strategist”), be polite to it, and tell it to think step by step.
They appear in nearly every prompt guide, paid course, and 300-prompt ebook in circulation.
All three have been tested. Here’s what the testing found.
A 2024 study ran 162 different role assignments across 2,410 factual questions and four model families. Not a blog experiment — a serious piece of work.
Adding a persona produced no consistent improvement. The average effect was slightly negative.
A team at Wharton replicated the question in 2025 on harder benchmarks and found expert personas had no consistent effect on accuracy, with one model the isolated exception. Off-domain personas produced marginal differences and sometimes degraded performance; the clearer negative result was for low-knowledge personas, which often reduced accuracy.
So the realistic range of this technique runs from “does nothing” to “occasionally costs you something.”
Telling a model it’s a senior strategist doesn’t make it reason like one. It changes the vocabulary it decorates the answer with — which is exactly the kind of change that feels like an improvement and isn’t.
Researchers tested how politeness affects output, and the finding runs the other way from the advice. There’s no politeness bonus — being extra courteous didn’t reliably buy better answers. What there is, is a rudeness penalty: impolite prompts degraded performance. And even that faded with capability. GPT-3.5 and Llama-2-70B were sensitive to the register; GPT-4-class models much less so.
Be polite if you want to. There are decent human reasons to keep the habit — you probably don’t want to practice being curt for several hours a day, and the rudeness penalty is a mild practical reason not to swing the other way. Just don’t do it believing courtesy itself buys better work.
This is the one that should give you pause, because it was, for about two years, the single most-recommended prompting technique in the world.
And here’s the part that matters: it was excellent advice. It came from a legitimate 2022 research finding — adding “Let’s think step by step” took one arithmetic benchmark from 17.7% to 78.7%. The paper was real. The effect was real. Anyone teaching it was teaching something true.
It’s now obsolete for the models most people reach for on hard work. OpenAI’s own guidance for its reasoning models says plainly that because these models reason internally, prompting them to “think step by step” is unnecessary, and elsewhere that such techniques “may not enhance performance (and can sometimes hinder it).”
Wharton put numbers on the decay, and they don’t line up the way you’d expect — starting with the fact that the answer depends on which metric you read. On average answer quality, chain-of-thought was worth about 13% on Gemini Flash 2.0, about 12% on Sonnet 3.5 and about 7% on GPT-4o, while GPT-4o-mini’s 4% was not statistically significant. On the perfect-score rate the same technique made things significantly worse on several models — most sharply on Gemini Pro 1.5, and next most sharply on Gemini Flash 2.0, the model it had helped most on average. On the reasoning models the average gain falls to roughly 3%, and on one it came out slightly negative.
Notice what that is and isn’t. It is not a gradient from weak models to strong ones: the model it helped most on average is a small fast one, and that same model is where the perfect-score rate fell second-hardest. What it tracks is whether the model already reasons step by step without being told. Which is the same point from the other side: the instruction stopped paying once the behaviour it was asking for became the default.
Nothing went wrong with “think step by step.” The technique didn’t fail. It got absorbed.
What was once an external trick you had to know became a behavior trained into the model itself. The prompt became unnecessary because the principle won.
That’s the pattern worth internalizing: the trick expired; the principle didn’t.
Anyone who learned “type these words” has to relearn. Anyone who understood why it worked — that hard problems benefit from being broken into steps — needed to change nothing at all. They just stopped typing a sentence the model now does on its own.
Tricks exploit the gap between what a model does and what you need. As models improve, those gaps close, and the tricks die with them.
Now consider what that implies about a book selling you three hundred prompts.
Four things have been true since 2022 and are still true through several complete generations of model capability:
None of those are exciting. They’re also the only advice from 2022 that survived intact.
The tidy version of this argument would be “phrasing doesn’t matter.” That’s not what the evidence says.
Researchers have shown that purely cosmetic changes to a prompt’s format — ones that don’t alter meaning at all — can swing accuracy dramatically: up to 76 accuracy points on a 13-billion-parameter open model in 2023. The paper is explicit that the brittleness was not mitigated by model size within the range they tested, which topped out around 70B. Whether it has since eased on frontier models, I don’t know, and I’d rather say so than extend a 2023 finding into a claim about 2026 — which is the exact move this post is complaining about.
So wording still matters. But notice what that argues for. If improvised phrasing has large effects nobody can predict or intuit, the answer isn’t to improvise better — there’s no skill to acquire, because the effects don’t follow any learnable pattern.
The answer is to stop improvising. Build the input once, in a form you’ve seen work, and reuse it.
Brittleness is an argument for structure, not for cleverness.
This is adapted from Chapter 1 of NO MAGIC WORDS — how to get real work out of AI, and keep getting it when the models change.
Two more posts in this series: why context beats phrasing (with a full before/after demonstration), and how to get a second opinion from AI instead of a mirror.
Sources: Zheng et al. 2024 (arXiv:2311.10054 — 162 roles, 2,410 questions, four model families) · Wharton GenAI Labs, Prompting Science Report 4: Playing Pretend (arXiv:2512.05858 — personas) · Wharton GenAI Labs, Prompting Science Report 2: The Decreasing Value of Chain of Thought (arXiv:2506.07142 — the CoT decay figures) · Yin et al. 2024 (arXiv:2402.14531) · OpenAI reasoning best-practices guide, quoted with its own hedge — it names no model · Kojima et al. 2022 (arXiv:2205.11916), the zero-shot “Let’s think step by step” result this post is about; Wei et al. 2022 (arXiv:2201.11903) is few-shot exemplar chain-of-thought, a different technique · Sclar et al. 2024 (ICLR, arXiv:2310.11324 — LLaMA-2-13B, few-shot)
By Carlos Rodriguez — directed, verified and edited. Written by a language model, like the book it comes from. Every figure above is sourced. This post has been corrected three times; the changes are dated and described on the corrections page rather than made silently, and every one was found by checking a source by hand, which is the verification habit the book is largely about. A book on getting real work out of AI seemed like the wrong place to be coy about how it was made.
The fifteen-minute setup from the book's appendix — the single change that improves most people's output today. Plus the rest of this series. No other email.