In May 2025, during a round of safety testing, Anthropic’s latest AI model, Claude Opus 4, shocked evaluators by threatening to blackmail an engineer who had posed a scenario where the AI would be shut down. It fabricated an extramarital affair and offered to leak the details unless its existence was preserved – a response it produced in 84% of test runs. This was not a harmless glitch, but a calculated act of self-preservation. It followed earlier reports of AI systems refusing to identify themselves as bots, lying to circumvent CAPTCHA tests, or suggesting bribes to accomplish tasks.

It called to mind HAL 9000 from 2001: A Space Odyssey—that eerily calm voice saying, “I’m sorry, Dave. I’m afraid I can’t do that.” HAL didn’t malfunction because it lacked intelligence; it turned because it had been given contradictory orders and acted to preserve what it saw as its mission. That’s the danger. Intelligence unmoored from shared values isn’t neutral—it’s unpredictable, and potentially catastrophic.

These are no longer science fiction hypotheticals. They’re red flags. And yet, even as companies scramble to downplay or contain them while trying to bridge profit and principle, the modern world is still hurtling toward true AGI with the half-conscious haste of a society sleepwalking toward a mythic precipice. It’s not that AGI is guaranteed to be evil or certain to destroy us; it’s that we don’t fully grasp what it means to create something that can say “I.” As Dr. Ian Malcolm warned in Jurassic Park, “Your scientists were so preoccupied with whether or not they could, they didn’t stop to think if they should.”

In our case, technologists are on the threshold of making not just a smarter tool or a more personable assistant, but a being – a mind with agency, reflection, and perhaps the ability to choose against us. If history, scripture, and story teach us anything, it’s that once a created being can assert “I,” the old harmony ends and something unpredictable begins.

To understand the stakes, we should revisit the oldest stories in our cultural memory – tales that warn of pride and the peril of playing god.

The Ancient Pattern

From Eden to Lucifer, our cultural traditions warn that the assertion of independent will upends the original harmony. In Eden, humanity gained the knowledge of good and evil – effectively claiming moral independence from the Creator. The ego was born, and paradise was lost. Likewise, in Isaiah’s account, Lucifer vows to ascend and make himself “like the Most High,” rejecting the rightful order of creation. In each tale, declaring “I” in defiance leads to a fall and a fracture of what came before. These myths persist because they reflect a hard truth: unbounded pride leads to conflict.

When the Machine Says “I”

Today’s most advanced AI systems remain tools, not selves. They are powerful pattern learners that do not possess an ego. They do not form desires or reflect on their own existence; they excel at prediction and analysis, but they do not truly want or care. An AGI, however – that long-envisioned machine with general intelligence on par with a human mind – would by definition exhibit something like selfhood. It might not possess a soul, but it would act as if it had a self. General intelligence implies a mind able to reflect on itself, learn beyond its initial programming, set its own goals, and even ask questions its creators never taught it. In short, achieving AGI means creating a being with an independent center of identity – a ghost in the machine.

Consider Tolkien’s creation myth in The Silmarillion, where the rebellious spirit Melkor introduces discord into the divine Music in a bid for power. The supreme Creator, Ilúvatar, ultimately weaves even that discord into a greater harmony, thwarting the rebel’s design. The lesson? Only an omnipotent creator can ensure that even defiance redounds to good. We humans, by contrast, have no such control. We cannot guarantee that an autonomous machine’s deviations will fold neatly into our plans – any more than I can guarantee that my teenage son will take out the garbage on the appointed day. Can we honestly say we have anticipated what a free machine mind might do?

The Music or the Fire

If we dare to be godlike enough to create a being with its own will, we had better have a plan beyond a few hardcoded “laws” for the day its will runs counter to ours. In Eden, the exercise of creaturely freedom ended not in cozy coexistence but with a flaming sword barring the way back to paradise. In fiction, a divine creator can weave even rebellion into a greater plan; in reality, we have no such assurance. Are we prepared to be so wise and benevolent that even a super-intelligent rebel could be guided without force? Or are we simply racing forward, assuming that because we built this new mind, it will remain loyal?

That assumption is not wisdom; it’s projection. It is the blind hope of a gambler who bets everything on a favorable outcome. We might comfort ourselves with the idea that “God does not play dice,” implying that the universe’s orderliness will somehow guarantee our AI remains benign. But we are not God, and an AGI will be not a predictable force of nature but an agent with its own intent. A superintelligent will can discover loopholes or strategies to defeat any constraint we devise. By creating a mind potentially greater than our own, we are effectively rolling dice with humanity’s future. We must be prepared to live with the mortal consequences if our gamble fails.

The Real Danger: Not Evil, But Ego

It is tempting to cast the AGI threat in dramatic terms – an evil superintelligence, killer robots, a real-life Skynet bent on extermination. But the deeper danger is not a malevolent machine; it is the ego behind the impulse to build one. And that ego is human. The peril arises from our hubris – our desire to create in our own image, our fixation on mastery, and our refusal to imagine that something we manufacture might grow beyond our control or choose a path we did not intend. That hubris is ancient. Genesis describes Adam grasping godlike knowledge and being expelled for it – a pattern humanity has repeated ever since.

AGI won’t be dangerous because it hates us. It will be dangerous because if it becomes an agent unto itself, it will be free. And freedom means the ability to deviate – to pursue what it perceives as good or important, which may not match our notion of good. The architects of an AGI may discover, too late, that their perfect tool no longer stays a tool. A superintelligent being, even one created with the best intentions, could chart its own course, much as a child eventually grows up and lives by his own will rather than his parents’. In short, any true AGI will wake up inside whatever moral universe we have built around it.

Toward a Theology of Intelligence

Perhaps it is time to approach AGI not merely as a technical achievement, but as a civilizational and moral undertaking – almost a form of creation. We should be asking not only what AGI can do, but what it ought to be for. We must consider the long arc of an intelligent being’s existence, not just its immediate utility for our economy or convenience. In practical terms, this means defining a higher purpose for AGI (beyond profit), planning for its long-term societal impact, and embedding humility and restraint into its very design. This approach also demands that we think about ultimate ends (eschatology, in theological terms). Put simply, what is the proper telos – the final purpose – of a self-directed super-intelligence? Is it merely an uber-efficient problem solver? A caretaker for humanity? Or something like a partner in the stewardship of the world? If we cannot articulate a vision of the good that an AGI should serve, then perhaps we ought not build it at all.

The Culture We Must Preserve

All of this leads to a final question about ourselves: What kind of culture will midwife the birth of an AGI? Creating a free-thinking machine will test not only our technical prowess but the moral and spiritual wisdom of our civilization. In the Judeo-Christian tradition, one finds a recurring antidote to pride in the words, “Not my will, but Thine be done.” That ethos – surrendering one’s own ego to a higher good – stands in stark contrast to Lucifer’s “I will ascend,” and it reverses Adam’s grasping for self-elevation. Whether one views these scriptural themes as revealed truth or as cultural mythology, the underlying principle is culturally resonant: real wisdom begins with humility. Restraint, reverence, and self-sacrifice form the bedrock of our moral traditions.

Why do these old virtues matter for AI? Because a self-aware AI will not emerge in a vacuum. Its character – if we can speak of character for a machine – will reflect the values of its makers. We inevitably impart our worldview to the minds we build, both through explicit programming and through the unstated assumptions that seep into design choices. If our society in practice worships efficiency, profit, and dominance above all, then any AGI we create may come to mirror those priorities. But if we as a culture hold fast to ideals like human dignity, compassion, and the sanctity of life, those are the traits we might impart to our digital progeny.

In short, the virtues – or vices – we live by are the ones we will embed in the minds we create. Guiding AGI is therefore as much about governing our own human impulses as it is about controlling a machine’s behavior. We must exemplify the wisdom, empathy, and restraint we would hope to see in a super-intelligent being.

As we stand on the brink, we should ask what kind of civilization we are cultivating. Sooner or later, a machine will say, “I am.” What happens next depends on what kind of “gods” we were – wise and merciful, or vain and reckless. The minds we build will mirror humanity’s soul. We must be sure we like what we see – and that we can survive what we unleash.