Insider Brief
- Generative AI is making phishing campaigns more convincing and personalized, weakening traditional defenses that rely on obvious content-based warning signs.
- AI-assisted phishing can combine public information, compromised accounts and deepfake audio or video to impersonate trusted people and support realistic social-engineering attacks.
- Organizations can reduce the risk by combining behavioral detection, phishing-resistant authentication, independent verification procedures and employee training focused on security processes.
In January 2024, a finance employee at Arup, the London-based engineering firm behind the Sydney Opera House, received an email from someone claiming to be the company’s UK-based CFO. The message requested a confidential transaction. The employee was skeptical. He asked for a video call to confirm.
On that call were the CFO and several colleagues he recognized. They looked and sounded right. Over the course of the conversation, the employee made 15 wire transfers totaling $25.6 million to five bank accounts in Hong Kong. Every person on that call, other than the employee himself, was an AI-generated deepfake, built from publicly available video and audio of Arup executives scraped from earnings calls, conferences, and company footage. The company’s CIO later described the incident as technology-enhanced social engineering rather than a traditional cyberattack.
The Arup case is a specific, well-documented instance of a broader mechanism. SK NEXUS’s coverage of social engineering and how people get hacked covers the underlying psychology that phishing has always exploited, before AI made the exploitation easier to scale.
This article covers how generative AI has changed the mechanics of phishing, what automated personalization looks like at scale, why filters built for an earlier era of attacks are structurally blind to this one, and what detection and training approaches currently exist.
How Generative AI Rewrote the Economics of Phishing
Email security training has spent two decades teaching people to spot the same tells. Awkward phrasing and generic greetings. Spelling errors and sender addresses that don’t quite match. Those tells depended on phishing being written by someone working outside their native language, under time pressure, at volume. Generative AI removes every one of those constraints at once.
Microsoft’s 2025 Digital Defense Report found that AI-generated phishing emails achieve a 54% click-through rate, compared to 12% for manually written phishing. Microsoft describes it as the most significant change in phishing the company observed in the past year. IBM’s X-Force research found generative AI has cut the time required to draft a convincing phishing email from roughly 16 hours to about five minutes. That is close to a 200x increase in attacker output per hour worked.
Similarly, Hoxhunt’s 2026 Phishing Trends Report tracked the share of AI-assisted phishing in its detection network rising from under 5% in November 2025 to 56% in December, a 14-fold jump in a single month. That share has since settled closer to 40%.
What Automated Personalization Looks Like
Spear phishing has always required research. Learning who a target reports to. What vendors a company uses. What an executive’s writing style sounds like. That research used to be the bottleneck, but that’s not the case anymore.
A general-purpose language model can draft a fluent, contextually appropriate email in seconds. Purpose-built criminal tools go further. WormGPT, first sold on underground forums in 2023 and rebuilt in multiple successor versions since, was fine-tuned specifically on phishing templates, malware code, and exploit writeups, stripped of the safety guardrails present in commercial models. Newer variants run on top of commercial open-weight models and are marketed on Telegram with subscription pricing starting around €60. A 2026 successor called KawaiiGPT, distributed freely on GitHub, takes roughly five minutes to configure and can generate a functional spear-phishing lure on request.
These tools do not need to invent information about a target. They assemble it through LinkedIn connections, recent company announcements, executive names pulled from public filings, and writing samples scraped from published emails or interviews all feed into a message that references real people and real context. The output reads like it was written by someone who actually knows the organization. In a functional sense, it was.
Deepfake audio and video extend the same personalization into real-time interaction. Arup’s CIO later said that out of curiosity, he tried deepfaking himself using free, open-source tools after the incident. It took him about 45 minutes.
Why Signature-Based Email Filters Can’t See This Threat
Traditional email security operates on pattern recognition. That approach assumes an attacker’s output looks different from legitimate mail in some detectable way.
AI-generated phishing can bypass many traditional email defenses. Some campaigns pass DMARC checks and come from compromised legitimate accounts, making the sender appear genuine. Business email compromise can also contain no attachment or malicious link, leaving signature-based filters with little to detect.
Polymorphic campaigns make detection harder by changing subject lines, sender names, and message structures across different emails. KnowBe4’s 2025 phishing research found this type of variation across observed attacks.
The final hurdle that must be crossed is the behavioral one. In many instances, traditional email filters have judged emails based on each single message and not according to whether the request is consistent with the behavior of the user. That makes plausible requests from legitimate accounts particularly difficult to identify.
What Organizations Can Do About It
No single control stops AI-generated phishing. The organizations best positioned against it are combining behavioral detection, authentication that can’t be phished, verification procedures built for deepfakes, and training that teaches judgment rather than pattern-spotting.
Shift Detection From Content to Behavior
Security vendors are increasingly focusing on communication patterns rather than message content alone. Behavioral detection systems learn how a sender typically communicates, including the types of requests they make and when and how they usually send them. The system can then flag unusual behavior even when the message contains no obvious technical warning signs.
Deploy Phishing-Resistant Authentication
Phishing-resistant multi-factor authentication, including FIDO2 security keys and passkeys, addresses a key weakness in credential-based attacks. Even if an employee enters their credentials on a convincing fake login page, those credentials cannot be reused to access the real account. This makes it much harder for adversary-in-the-middle phishing kits to capture and reuse valid authentication credentials.
Separate Verification Channels for Financial Requests
For deepfake-enabled attacks such as the one reported at Arup, the recommended controls are largely established. Channel separation in payment authorization was already recommended for business email compromise before deepfakes became a concern.
It requires a payment request received through one channel to be verified through a separate, independently initiated channel. Deepfakes add a new challenge because seeing and hearing someone on a video call can no longer, by itself, serve as independent verification.
Retrain Employees Around Procedure, Not Typos
Programs that focus on typos and generic greetings may be less effective as AI-generated phishing becomes more polished. Awareness training can instead focus on procedural discipline, such as verifying unusual requests through a second channel regardless of how convincing the original message appears. Manufactured urgency can also be treated as a reason to pause and verify a request.
These controls are already familiar to many security teams. Organizations that require out-of-band verification for wire transfers and use phishing-resistant MFA may be better positioned to handle AI-generated phishing than those that rely mainly on employees spotting poor grammar or obvious mistakes.
For readers looking to go deeper on adjacent AI security risks, AI Insider’s coverage of prompt injection as an enterprise attack surface and data poisoning as a supply chain risk covers related ground in how generative AI is reshaping the enterprise threat model.
