The era of poorly written phishing emails with obvious spelling mistakes is over. Large language models allow attackers to generate unlimited volumes of personalised, grammatically perfect, contextually relevant phishing content. Here's what AI-assisted phishing looks like — and why traditional detection is failing.
For years, the primary advice for spotting phishing was to look for spelling mistakes, generic greetings ("Dear Customer"), and poor English. These indicators reflected the reality that most phishing was written by non-native English speakers using templates, with little personalisation and minimal attention to quality.
That reality ended in 2023. Large language models — the same AI technology underpinning ChatGPT — can generate grammatically perfect, stylistically sophisticated phishing emails in seconds, at no meaningful cost, in any language, with any level of personalisation. The old detection heuristics no longer apply.
What AI-Assisted Phishing Looks Like Now
Personalisation at scale: An attacker who scrapes LinkedIn for a company's staff list, their roles, their recent posts, and their connection networks can prompt an LLM to generate personalised emails for each target — referencing real projects they've mentioned publicly, their actual job title, their actual manager's name. This level of personalisation previously required significant manual research per target. With AI, it's automated across hundreds of targets simultaneously.
Perfect language quality: Grammatical errors, unnatural phrasing, and stilted language — once reliable red flags — are no longer present in AI-generated phishing. The output reads naturally because it was generated by the same models producing legitimate business content.
Contextual relevance: AI can generate phishing content timed to current events — a "tax return reminder" during NZ tax season, a "payment verification" during a period of EOFY activity, a "security alert" referencing a recent public cyber incident. The relevance increases the likelihood that recipients will engage.
AI Voice Phishing (Vishing)
Beyond email, AI voice cloning technology allows attackers to create audio deepfakes of known individuals. In 2024, a finance worker at a Hong Kong company was instructed by a voice they believed was their CFO — on a video call — to transfer HKD 200 million ($38.5 million NZD). All participants in the "video call" were deepfaked. The call was convincing enough that the employee transferred the funds without hesitation.
This technology, previously requiring significant resources, is now accessible through free and low-cost consumer tools. NZ businesses with executives who have publicly available audio or video content are potential targets.
What This Means for Detection
Detection based on content quality is now unreliable. Technical controls — email authentication (DMARC/SPF/DKIM), email security filtering based on sender reputation and behaviour, DNS filtering, and link detonation — remain effective because they don't rely on content quality. Process controls — verbal verification of payment requests, standard procedures for account changes — remain effective because they bypass the email channel entirely.
AI has removed the most visible indicators of phishing. The defences that remain effective are technical (email authentication, URL filtering) and procedural (verification processes that don't rely on email alone). Quality of writing is no longer a reliable signal.
