In one of the first scenes of Dune, Duke Leto (Oscar Isaac) signs a contract with his ducal seal ring, imprinting his family crest onto the paper in wax. This ring is used in-world as a marker of authenticity—like a medieval duke, Leto is verifying his identity with information that only he has access to. But why does the Dune universe’s space dictator (played by Christopher Walken, for some reason) run his government on Dark Age tech? Couldn’t he just use DocuSign?
AI, and most complex digital technology in general, is outlawed in the Duniverse because it manipulated and dumbed down humans. It didn’t just seize control of human governments, it made humans so dependent on its assistance they were unable—and even unwilling—to rebel. The Reverend Mother of the Bene Gesserit tells Jessica, “once, men turned their thinking over to machines in the hope that this would set them free. But that only permitted other men with machines to enslave them.” In the age of addictive technology, we can probably all relate to that.
Humankind finally rose up against AI and their tech overlords to destroy all thinking machines, leaving only the bare minimum of technology behind. In the new era, instead of allowing human potential to stagnate with digital assistance, there was a revolution in human augmentation—the Bene Gesserit’s psychic powers, the creation of mentats to replace computers, and of course, the use of spice for interstellar navigation. This would allow humanity to drive its own development, rather than being manipulated by technology.
As of the beginning of Dune, paper documents and other analog records are a way to verify that a contract or record is free of digital influence. That contract Leto signs is brought to him all the way from the imperial planet Kaitain—Thufir Hawat notes that the journey costs “a total of 1,460,062 Solaris round trip.” For those unfamiliar with the exchange rate of Solaris to the US dollar, that is a lot of money.
The emperor later makes an in-person visit to the war zone of Arrakis just to meet Muad’Dib in person—surely they could have done that over Zoom, or via holographic video calls as in Star Wars? Instead, communication happens either by courier or in person, to ensure everyone is who they say they are. The emperor also keeps a hand-written journal, and the Bene Gesserit even utilize genetic memory as a very inventive form of non-digital documentation. As you probably know, Frank Herbert couldn’t have had generative AI in mind specifically in the early 1960s when he wrote about thinking machines taking over human civilization. Even Denis Villeneuve wouldn’t have known about Dall-E or ChatGPT when he adapted and filmed the Dune movies back in 2019 and 2022 (respectively). But that doesn’t change the fact that Dune 2 arrives at a pivotal time in this technology’s development, and the source material’s opinion on AI in general leaves little doubt what Herbert would think of genAI in particular.
It’s becoming clear that within the next few years, AI in our world might drive banks and government agencies back to paper—or at least to much more stringent security measures to verify authenticity. But could similarly Dune-like developments come to pass in realms where we can’t even imagine a return from the digital?
Imagine scrolling social media a year or two from now and being entirely unsure which posts are real and which are AI-generated. This isn’t so far from where we are now; the only difference is scale. The effect would be especially pronounced on platforms like Tiktok, where the whole point is to discover creators you don’t follow via a “For You” algorithm. If nothing the algorithm served to you was real and you knew this, what would you gain from watching Tiktok?
People today get an alarming amount of their opinions from the Internet, and they’re often deceived by what they see online. As of the 2016 election, this was already an existential issue for culture—and one we’ve done little about, unfortunately. People don’t put that much effort into screening the accounts they interact with for ill intent, except for calling anyone who disagrees with them a Russian asset, of course. And even leaving election tampering aside, influencer marketing is enough of a psyop already!
A sufficiently advanced AI might not just act as an agent of misinformation, it could wield manipulative powers akin to that of the Bene Gesserit. If a pre-GPT chatbot could trick its developer into thinking it’s sentient, we shouldn’t take chances with human psychological frailty.
The Bene Gesserit administer the Gom Jabbar test to see if a being is human. They screen for robots disguised as humans, and also for compromised humans who have been manipulated by machines too successfully to control their own instincts. It’s kind of like the marshmallow test for kids; if they haven’t developed self-control yet, they can’t delay the gratification of eating the first marshmallow to get the second one later. (Too bad Paul couldn’t take that test instead.)
If we don’t want to turn our thinking, feeling, buying, voting, etc. over to machines completely, protection from AI-assisted psyops should be a priority. In a darker timeline, we would eventually accept our new AI hyperreality, the same way we’ve accepted social media advertising.
That timeline would be closer to our current world than we might be comfortable acknowledging. What the Bene Gesserit call “machine thought,” the algorithms and optimized flows of computer systems, already define our era of history. When Instagram introduced reels, for instance, creators quickly noticed that if they didn’t create reels, their posts would be algorithmically punished. This advanced Meta’s goal of competing with Tiktok, because it ensured its platform would immediately have tons of video content available to make reels successful. Humans have to adapt to fit technology, rather than the other way around.
I’ve been bored of algorithms for a long time, and considering all generative AI does is the same algorithmic trick of spitting out similar content to what it’s seen before, we should expect social media to get Worse in this way. I know a lot of many amazing artists who have stopped posting entirely, just because they’re tired of jumping through algorithmic hoops! Going viral on Tiktok requires a totally different skill set from the actual making of art, after all.
In the universe of Dune, the Butlerian Jihad began when an AI called Omnius aborted a Bene Gesserit’s baby without her permission because she was so powerful the AI didn’t want her to reproduce. The Bene Gesserit had been using AI to plan out their breeding programs, but this revealed their technology was selecting for the most docile humans to weaken potential resistance. Just as Instagram limits the reach of less compliant creators, Omnius wanted to limit psionic ability in humans. According to Bene Gesserit records, AI breeding and conditioning made humans become like animals—that is, humans couldn’t remain human in a world controlled by AI.
During the ensuing crusade, the Bene Gesserit developed not just the gom jabbar but also their rad psychic powers as a way to counter AI. As of the book’s beginning their domain is emotion and communion with the collective unconscious, sidestepping manipulation by machines with supernatural self control. To advance humanity’s potential, they breed more powerful humans, which they hope will culminate in the coming of the Kwisatz Haderach (Paul Atreides). They maintain control of human government partially because they love power, but also partially as a way to ensure humanity doesn’t become vulnerable to AI again.
In other words, the Bene Gesserit developed their ideology of humanity in response to the realm of the human being threatened by tech. They aren’t just responding to a specific political and economic danger that AI poses to humanity; they go one step further to protect human concerns for humanity’s sake. As in, even though genAI might be very profitable for a few, the Bene Gesserit stance would be that it still shouldn’t take over human art or culture, because that wouldn’t be best for humans. I, for one, am amenable to becoming a sexy Jedi to save my job from the robot apocalypse.
There are stronger religious overtones in the Duniverse than in our world, probably because of this focus on the human: when society stops worshipping machines and automation, we turn to ourselves and other people. As I read into questions like “what is art?” and “what does it mean to be human?” I am inevitably led by my sources into the realm of the spiritual. I’m talking Joseph Campbell, obviously, but also Heidegger, who believed art was a revelation of truth—truth that he juxtaposes with the extractive logic of tech and capital. In his essay “The Question Concerning Technology,” he treats art and industry as opposites, positing art as the only way to revive the soul from the alienation of technological exploitation. The Reverend Mother would probably agree.
What would happen if we turned to art to revive our souls and found only content generated by technological exploitation? To replace artistic truth with high-fructose corn syrup would pollute human culture and undermine our relationship to very possibility of truth, nevermind truth itself. Could you produce artistic truth if you’d never seen it before?
Jean Baudrillard argued in Simulacra and Simulation that an effective copy of a thing devalues the original by rendering it irrelevant, eventually supplanting the original by sowing doubt that there’s anything authoritative or special about it compared to the copy. If you can generate an essay without having any ideas first, how important can thinking really be?
Art is one way humans understand and know ourselves. If we replace even the most corporatized art with AI, culture will no longer be humans understanding humans. It’ll be AI generating an understanding for humanity to have of itself, endangering humanity’s capacity to understand itself independently. It would make us less human.
On all fronts—political, economic, and spiritual—over the next 5-10 years the proponents of AI will clash with those who want to preserve the human for its own sake. The Duniverse demonstrates the stakes of this new culture war perfectly: if humans want to continue being a thing, they should act against technology that could seize control of government and/or culture. And as I’ve shown, this culture war has already been here for decades—the coming conflict will only continue the battle against deepfakes, psyops, and algorithms.
In the end, the AIs that had taken over civilization were destroyed with nuclear weapons in a climactic final battle between humans and machines. In a 2023 article, AI thought leader Eliezer Yudkowsky urged the international community to ban AI beyond a certain threshold of complexity, to “be willing to destroy a rogue datacenter by airstrike.” His article is Butlerian enough that it deserves to be quoted at length:
Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy […] that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.
In lieu of a Harkonnen invasion, this might be the defining Holy War of our time.
(Accidental double comment)
This is the only thing that’s ever made me want to watch Dune.
Wow. I don’t want to downplay any of the dangers of AI, but I do think that a) The author buys a little too much into the idea that current AI actually is true AI (or a step on the way to it) instead of a very elaborate con that gives you the impression to interact with something that’s creative and/or thinking; and b), that closing with quoting something along the lines of “we should be willing to throw nuclear warheads on sever farms if necessary!” is REALLY not helping the debate. If that notion became the mainstream, just imagine the number of crackpot governments that would argue “Yes, of course we have to eradicate this or that other country with a nuclear strike! Because they’re secretly run by AI!” Sorry, but I think any narrative along the lines of “humanity has a common enemy, which is XY” is extremely dangerous, because if it becomes succesful, it will ALWAYS be employed to kill humans (usually lots of them).
The problem with the Turing Test is that it assumes people don’t want to be fooled into thinking a machine is intelligent.
I, for one, believe there are other technological alternatives to a Skynet-like future than Bene Gesserit-style eugenics. AI is a tool, and as with any tool, care must be taken to ensure that it is used properly and safely to maximize its usefulness.
As near as I can tell, the author has never read the book, Dune, but has seen the most recent movies. But they have read the (much inferior) prequels from Brian Herbert.
You want to read Artificial Intelligence: Defining Our Terms by Michael Swaine, who has been writing about computers and technology for decades now.
Thank you! That’s my take as well
I had the impression for a long time that the machine control of humans was more of a doing everything for them and lettings humans just sit around and stagnate, like what is implied in Jack Williamson’s With Folded Hans. It wasn’t until I read The Butlerian Jihad that I heard the idea that the machines were controlling everything in a manner more like The Matrix. I’m curious what other people thought.
Which is pretty weird given machines are a lot more resistant to radiation than humans are.
Dune doesn’t really solve the ‘problem’ of AI. It simply bans it entirely. And then posits a world were no one dares ignore the ban. That’s not really solving the problem as much as simply writing it out of the novel.
Well, to be fair, Dune posited the problem of AI long before there was any need to adress it in science fiction, so he didn’t just write it out, he introduced it by writing it out, and in a pretty prescient way. However, the whole thing was much better in the original novels, where it stayed very vague what the machines hat done to humankind and how the Butlerian Jihad went down; once all of this became more explicit in the prequels, it frankly felt pretty silly. I never understood why the machines would enslave humans in the first place, for example. It’s weird how Dune and, for some parts, its sequels, still feel pretty timely in some ways, while the prequels already feel dated when they come out.
Generally, my understanding is that there’s a long tradition of sci-fi where AI is viewed as something impossible to write, based on a belief among some technologists that an AI singularity would so fundamentally transform the structure of civilization that there is no story to write that contains AI. The answer these authors came to was to create narrative reasons where AI couldn’t exist. Verner Vinge, who recently passed, came up with his own distinct reason for why AI doesn’t exist in The Fire on the Deep by inventing a physical law that made advanced AI impossible within our galaxy. I always saw Dune’s allergy to AI as more of a narrative invention to justify their absence than a moral stance.
“when an AI called Omnius aborted a Bene Gesserit’s baby without her permission”
Somehow I had the impression that the Bene Gesserit (like the Mentats and the Suk School) did not exist at the time of the Butlerian Jihad? Been a while since I read it so I’m not sure…
You’re right. None of these human potential groups existed during the Butlerian Jihad. Also, Omnius was the ruling machine entity. His creation, the robot Erasmus, killed Susan Butler’s baby and by so doing triggered the war.
If I remember correctly, the prequels introduced the “Sorceress of Rossak” or something as their predecessors, who had some kind of psionic powers totally out of in tune with everything else in the Dune-universe until then.
Of course, the AI “solution” of Dune puts humans under control of an autocratic empire, ultimately run by a non-human thing, supported by a not-quite human Guild and a couple of quasi-religious orders of manipulators into selective breeding humans. Wiping out AI just put humanity under a functionally equivalent tyranny. Replacing the emperor with Paul Atreides just made that tyranny longer lived and less subject to succession disputes.
I’d feel more for this if it understood that large Language Models, while they are used by humans to create multiple deeply deceptive and manipulative stories (Someone managed to fake the X LLM into making up lies about Elon Musk being a pedophile and a criminal) they are still human driven because despite the hype, LLMs are NOT AI, and the dangers they pose are not the dangers of self-aware machines, and pretending they are by constantly treating them as AI, or even a valid path towards true AI, is a mistake. I’d rather point out the real dangers they pose – the examples of mushroom foraging texts written by AIs that badly misidentify edible vs poisonous, or, well, this: https://www.funraniumlabs.com/2024/04/phil-vs-llms/ .
All of which are very serious issues. None of which involve actual artificial intelligence and not one of which is solved by nuclear powers murdering humans.
Point of order: the gom jabbar is not the pain-box test; the gom jabbar is the poisoned needle she holds to his neck to make him take it. The gom jabbar is used by the Bene Gesserit for many things; Alia kills the Baron Harkonnen with one, for example.
I don’t think centuries of eugenics and the reintroduction of feudalism is the answer to AI-related problems.
It’s certainly an answer, but not the optimal one, except for the main beneficiaries: the emperor and the aristocracy. Considering that autocracies, and tyrannies in general, have a habit of destroying records which disagree with their interests, I offer the hypothesis that the reports of the reasons for the Butlerian Jihad are lies.
Is the Bene Gesserit’s narrative that they developed psychic abilities and outlawed “thinking machines” for the good of mankind actually the truth, though? Or just another manipulation in order to place them in the seat of power, better able to control human evolution?
One of the things that was most frustrating about the new adaptations as a book reader, was no one else who saw the movie with me getting this core aspect of the setting. That part of the reason for the neo feudalism, and the psychic witches, and the entire setting/society being the way they were, was because they had followed our path to the end result, which is machines running the show, and rejected it so hard they decided they would rather perfect humanity (through various means) than make a freaking iPhone.
It’s one of the main things that sets the setting apart from other sci-fi, and it was just glossed over because Denis couldn’t include one freaking line in the scene between Paul and the Reverend Mother.
Colossus the Forbin project.