Something We Should Never Automate

Large language models will enable the mass production of ruinous scams that are currently too labor-intensive to produce

The awful “Pig Butchering” scam is about to go mainstream.

The reason we’ve all heard from fake princes offering fake riches in exchange f

or our real bank numbers is that those emails are basically free to send. Zero marginal labor and no real overhead means that one sorry victim can pay for millions of failed gambits.

There are other online scams that few of us have faced because they’re much too labor-intensive to mass produce profitably. But large language models are on the cusp of making them dangerously affordable and borderline ubiquitous.

One goes by the charming name of “pig-butchering.” It involv­es befriending a victim via WhatsApp, discord, a dating app, or some other place where strangers sometimes strike up digital friendships. Lonely immigrants are often targeted by gangs in their home countries. They stalk their victims on social media – then reach out and chummily pretend to come from the same region, or share some distant acquaintance.

After weeks of building trust, the scammer shares a hot crypto tip. The victim is persuaded to make a small, experimental bet through a legitimate-looking exchange -- and seems to win big! They may even withdraw some profits. And now the “fattening” phase is in full swing, with the victim targeted for slaughter weeks or months down the line.

Trust deepens through a relentlessly patient series of messages. The scammer comes off as empathetic, plugged-in, perhaps smitten, and sometimes scolding. Enabled, emboldened, and seduced by phantom profits, the victim bets bigger and bigger until they’re truly all-in.

And then their fake profits, their fake friend, and their actual life savings all vanish. Forbes documented a recent seven-figure con in which the scammer and victim swapped 271,000 thousand words of text. That equates to an eight-hundred-page book.

Even the greediest con artist can only pull that off so many times per year. And again – this is why most of us have never faced this. But when chatbots start replacing workers at scale, one of the most vulnerable job titles out there will be Pig Butchering Scammer.

Yes, it’s hard to think of a more deserving group of disruptees. But their replacements will work 24/7 for free. And they’ll each do the work of armies of flesh-and-blood conpersons. As costs collapse, the revenue threshold for profitable attacks will also plummet. Eventually the most vulnerable people with the tiniest savings will enter the sights of the digital butchers.

Meanwhile, more powerful media will enter the equation. Synthetic voices are pretty obvious for now – particularly when it comes to conveying empathy, vulnerability, charisma, or playfulness. But as millions of friendships are feigned by crooks & marketers, and then ranked by engagement, duration and profitability, nuances like these will be burnt into algorithms and ruthlessly optimized. Realtime, photorealistic video will come along next, allowing the weaponization of beauty, microexpressions, and more.

Perhaps the darkest side of this will be the intuitions we evolve to protect ourselves. It’s bad enough that I could miss out on a fortune by instinctively shunning all the real Nigerian princes who actually want to enrich me. But what if we learn to blot out the countless subtle signals of human connection that the avatar armies learn to mimic – the way we unconsciously ignore most advertising? If we wall ourselves off in this way both offline and online? With humans as well as bots?

There’s so much about generative AI that excites me. But this boom will bring some cavernous pitfalls along with it, and we can’t allow them to blindside us. 

• • •

And ONE LAST THING - if you found this post valuable, and/or enjoy my substack in general, please forward it around, or recommend this to your friends. I’m starting to develop an audience here - and any help readers might provide with this will be deeply appreciated !!!

Oh and one TRULY last thing - if you’re curious about the prompt that generated today’s tasteful illustration, it’s Midjourney, “photo of a pig dressed like a butcher, in the style of Judd Apatow, holding a cleaver, cutting meat, white apron, pile of meat, editorial quality, stylish costume design, movie still --ar 3:2 --v 5”