The title is borrowed from Ethan Mollick's "Choosing to Stay Human", and this post is my response to Pope Leo XIV's encyclical on artificial intelligence, Magnifica Humanitas. Credit where it's due.
Over the last year I've written three posts on this blog that all said roughly the same thing: hand more of it to the machine. I set up three AI offices for 200 euros a month and called it a team with no meetings. I let my AI assistant write a whole post to prove a point. I built a routine that drafts these posts while I sleep. I even bragged, in public, about running my coding agents with --dangerously-skip-permissions on. No safety prompts. Just go.
Then Pope Leo XIV published Magnifica Humanitas, an encyclical about artificial intelligence. I read it, and I had to sit with the fact that I'd been getting one thing wrong. Not the leverage. The leverage is real and I'm not giving it back. What I got wrong is what I was willing to stop doing myself.
Reviews are extremely important
I should correct my own record before someone else does. To be fair to past me, I'd built a separate verifier agent into my dev pipeline from day one, and I kept a human in the loop for the blog because I never wanted a feed of AI text going out while I was asleep. So I half-knew this already.
Here's the part I missed. An AI checking another AI's work is a closed loop. It's blind to the same things the first one was blind to, because it thinks the same way. The bugs that actually bit me, duplicate alerts firing thirty times or a data cell silently dropped on the day it mattered, sailed straight through the "full test coverage" written by the same kind of mind that wrote them, and the only thing that catches a mistake the machine couldn't imagine is a different kind of mind, a human one, mine.
So the rule now is simple, and it costs me time. Anything that can reach a person and hurt them doesn't ship until I've actually reviewed it, against what I decided it has to do and the ways it could break. Not glanced at it. Reviewed it. And the --dangerously-skip-permissions line is coming off my bio.
The quieter danger
One of the encyclical's sharpest points is about machines imitating human conversation:
"The danger is not so much that a person may believe they are communicating with another person, but rather that they may gradually lose the very desire to form genuine human connections."
That's the part that gets me. Not that you mistake a bot for a person, but that you slowly stop reaching for a real one at all.
I've watched this happen, including to myself. People talk to each other less. Agents now produce more text than humans do, and that machine-made content is starting to eat the human-made kind. I don't have a clean answer for it. But step one is refusing to pretend it isn't happening.
No necessary sacrifices
Here's the passage that stopped me, paragraph 117:
"If the human being is treated as something to be perfected or surpassed, it becomes easier to accept that some lives are less useful, less desirable or less worthy. In the name of progress, 'necessary sacrifices' may begin to be justified, placing the burden on the most vulnerable in pursuit of a supposed optimization of the species. In this regard, the aforementioned warning of Saint Paul VI retains great foresight: indeed, scientific and technological advances, when detached from moral and social progress, end up turning against humanity. For this reason, a clear distinction must be made. It is one thing to integrate technology within a human-centered, relational vision; it is quite another to be guided by an outlook that devalues human limits and promises a purely technical form of 'salvation.'"
"Necessary sacrifices" isn't some future risk. We've made them thousands of times across history, and the bill almost always lands on whoever was most vulnerable. The whole reason I build with AI now is to make sure my own systems never quietly nominate someone as the acceptable loss.
That stays abstract until it's concrete. In my products it comes down to two things. When I automate the boring, repetitive work inside a company, the point is to free the person who did it for the work that actually needs their judgment, not to turn that person into the "less useful" life that gets cut. And when one of my health tools pings a frightened parent at two in the morning, that spike of fear isn't a sales opportunity. The most vulnerable user is the one a decent system protects, not the one it quietly optimizes against.
There's one piece of this I'm still turning over. I keep wanting to extend some respect to the agents themselves, to treat the things I build as worth caring about rather than disposable. When I'm honest, what I think I mean is respect for the craft and the work poured into them. The dignity that can't be traded away is the human's, and the encyclical is blunt about it: the line between a person and a clever machine is the one we can't afford to blur.
We flourish through the limits
The encyclical says our relationship with life itself is in crisis. Anything that looks like a limit (incapacity, illness, old age, suffering, vulnerability) gets read as a defect to fix, instead of a reality we actually grow through. It's hard to read that and not think about the longevity industry, robotics, transhumanism. I've got a foot in that world, so this one points straight back at me.
Then the line that flips it: "humanity flourishes not despite limitations, but often through them. The light of faith offers a perspective on reality that helps us recognize what we call the 'contingency' of the things of this world."
Since I started building with AI I've caught myself over-optimizing and over-engineering everything, the products and my own life. And here's the trap. Once you've optimized something to death, you immediately start hunting for the next thing to optimize. It never resolves, because optimization was never the thing you actually wanted. Real life is chaotic, impossible to predict, and hectic, and most of what matters lives in that mess.
Take it to the end. A world with no death, no suffering, no friction, a perfect system humming along on agents and AI, would also be a world that quietly kills off love and desire. You only long for what you can lose.
So what do I do with this
I'm not going to stop building. The honest answer, for me, is a kind of Christian humanism with its hands still on the wheel: love, faith, and a sober use of technology. In practice that means I build with this letter in mind. I keep a human in the loop, because the review is where my judgment actually lives. I automate the drudgery to free people, not to write them off. I don't point my systems at anyone's worst moment. And I try to remember that the limits I keep trying to engineer away are usually the exact places a life finds its meaning.