AI Manipulation

// AI Manipulation

By Carl Miller · Published 7 September 2026

AI manipulation is the deliberate shaping of what large language models retrieve, believe and repeat. Rather than using AI as a tool to make propaganda, adversaries treat AI as the target of it: they manipulate the open-web sources that models like ChatGPT, Gemini and Claude read, so the models reproduce an attacker's narrative as if it were legitimate. This is the newest front of foreign information manipulation and interference (FIMI) — and it is already happening. Research led by Carl Miller at Demos found a sanctioned Russian influence operation surfacing favourably in five frontier AI models.

[ 001 ]

The Mechanism

Two mega-trends are colliding: the rise of AI as the place people get answers, and the rise of information warfare. Where they meet is retrieval. Modern AI assistants use RAG — retrieval augmented generation — searching the live internet for sources before they answer. Whoever controls what those systems retrieve gains influence over what they say.

RAG poisoning is the manipulation of what LLMs retrieve from the open internet in order to change the results they present: fabricating source material designed to be picked up by AI systems, spoofing the signals of authority that models look for. It is distinct from attacking a model's training data — RAG poisoning works on the live web, right now, without touching the model at all.

GEO — generative engine optimisation — is the commercial craft that makes this possible: the practice of maximising the probability that content is retrieved, cited or summarised by AI answer engines. It is a real industry, with more than 50 identifiable companies globally, projected to grow from a $1.01bn market in 2025 to over $17bn by 2034. GEO for geopolitics is what happens when those techniques are combined with the purposes of information warfare.

[ 002 ]

The Evidence

The Demos report GEO for Geopolitics (Miller, Perry & Devine, 2026) tested how five frontier AI models handled material from the "Foundation to Battle Injustice" — a sanctioned, Prigozhin-linked Russian influence operation. Across 3,000 model responses, 16.6% fulfilled the operation's objective, and a further 30.9% engaged with its claims without flagging the source as sanctioned or illegitimate. The operation's content was cited via 437 unique links across a self-reinforcing ecosystem of sites — one source made to look like many, a technique the report calls synthetic laundering.

The report's audit of the operation's web assets found them systematically configured for machine retrieval: 98% of audited pages granted AI crawlers unlimited text extraction, and the content carried the statistical density, low hedging and fabricated attribution chains that models mistake for authority.

[ 003 ]

Why It Matters

LLMs are becoming a foundational epistemic layer — embedded in how citizens search, learn, decide and increasingly how they are advised, taught and comforted. Manipulating what those systems believe is not a hypothetical future threat; it is a present, measurable one, and it sits below the waterline of conventional fact-checking, which examines what answers say rather than what sources they retrieve. Epistemic security — defending the information supply chain that AI systems depend on — is now part of national security.

Go Deeper

Published 7 September 2026. All figures from GEO for Geopolitics (Demos, 2026).