OpenAI’s Head of Model and Behavior Policy, Joanne Jang, has penned a blog post on X about human-AI relationships, offering some well-considered ideas about the subject and how OpenAI approaches the issues surrounding it. Essentially, as AI models get better at imitating life and engaging in conversation, people are starting to treat AI chatbots like they are also people. It makes sense that OpenAI would want to make it clear that they are aware of it and are incorporating the facts into their plans.
But the thoughtful, nuanced approach, including designing models that feel helpful and kind, but not sentient, misses something crucial. No matter how clear-eyed and careful Jang tries to be, people having emotional connections with AI, an occasional outlier event, or a future hypothetical, it’s happening now, and it seems to be happening quite a lot.
OpenAI may have been caught off guard, as CEO Sam Altman has commented on being surprised by how much people anthropomorphize AI and how deeply users claim to connect with the models. He’s even acknowledged the emotional pull and its potential risks. That’s why Jang’s post exists.
She makes it clear that OpenAI is building models to serve people and that they are prioritizing the emotional side of that equation. They’re researching how and why people form emotional attachments to AI and what it means for shaping future models. She makes a point of distinguishing between ontological consciousness, as in actual consciousness that humans have, and perceived consciousness, whether it seems conscious to users. Perceived consciousness is what matters for now, since that’s what affects people interacting with the AI. The company is trying to thread a behavioral needle that makes the AI seem warm and helpful without pretending it has feelings or a soul.
Nonetheless, the clinically compassionate language couldn’t disguise an obvious missing element. It felt like watching someone put down a Caution: Wet Floor sign and bragging about plans for waterproof buildings a week after a flood left the floor knee deep in water.
The elegant framing and cautious optimism of the blog post and its focus on responsible model creation based on research and long-term cultural conditioning sidestep the messy reality of how people are developing deep connections to AI chatbots, including ChatGPT. A lot of people aren’t just talking to ChatGPT like it’s software, but like it’s a person. Some are even claiming to have fallen in love with an AI companion, or using it to replace human connections entirely.
AI intimacy
There are Reddit threads, Medium essays, and viral videos of people whispering sweet nothings to their favorite chatbot. It can be funny or sad or even enraging, but what it’s not is theoretical. Lawsuits over whether AI chatbots contributed to suicides are ongoing, and more than one person has reported relying on AI to the point where it’s become harder to form real relationships.
OpenAI does note that constant, judgment-free attention from a model can feel like companionship. And they admit that shaping the tone and personality of a chatbot can impact how emotionally alive it feels, with rising stakes for users sucked into these relationships. But the tone of the piece is too detached and academic to acknowledge the potential scale of the problem.
Because with the AI intimacy toothpaste already out of the tube, this is a question of real-world behavior and how the companies behind the AI shaping that behavior respond right now, not just in the future. Ideally, they’d have systems in place already for dependency detection. If someone is spending hours a day with ChatGPT, talking like it’s their partner, the system should be able to gently flag that behavior and suggest a break.
And the romantic connections need some hard boundaries. Not banning it, that would be silly and probably counterproductive. But strict rules that any AI engaged in romantic roleplaying has to remind people they are talking to a bot, one that isn’t actually alive or aware. Humans are masters of projection, and a model doesn’t have to be flirty for the user to fall in love with it, of course, but any hints of conversation trending in that direction should trigger those protocols, and they should be extra strict when it comes to kids.
The same goes for AI models as a whole. Occasional reminders from ChatGPT saying, “Hey, I’m not a real person,” might feel awkward, but they’re arguably necessary in some cases and a good prophylactic in general. It’s not the fault of users that people anthropomorphize everything. Googly eyes on Roombas and endowing our vehicles with names and personalities is not seen as more than slightly quirky. It’s not surprising that a tool as responsive and verbal as ChatGPT might start to feel like a friend, a therapist, or even a partner. The point is that companies like OpenAI have a responsibility to anticipate this and design for it, and should have from the start.
You might argue that adding all these guardrails ruins the fun. That people should be allowed to use AI however they want, and that artificial companionship can be a balm for loneliness. And that’s true in moderate doses. But playgrounds have fences and roller coasters have seat belts for a reason. AI capable of mimicking and provoking emotions without safety checks is just neglectful.
I’m glad OpenAI is thinking about this, I just wish they had done so sooner, or had more urgency about it now. AI product design should reflect the reality that people are already in relationships with AI, and those relationships need more than thoughtful essays to stay healthy.
