
So, we’ve reached the point where we’re having a conversation about whether we should be nice to AI. I didn’t expect that to be a serious topic this soon, but here we are.
On October 8, Anthropic, the company behind Claude, updated its usage policy to prohibit sustained and needless abusive or cruel behavior toward its AI models. In other words, there are limits to how far users can go when treating Claude badly.
This isn’t about getting annoyed when a chatbot gives you the wrong answer (because, let’s be honest, we’ve all been there). The policy targets extreme cases of repeated cruelty without a clear purpose.
Now, I’m not saying we should start apologising to our chatbots every time we close a conversation. But the move does raise an interesting question:
Why are we even talking about AI welfare when we still don’t know whether AI can actually experience anything at all?
This kind of thing sounds like something straight out of M3GAN. I can imagine a future where AI becomes more than just an assistant—perhaps a companion or even a caretaker. It’s not hard to see why stories about AI turning dangerous capture my attention, even if real life is far from what we see in movies.
I’m not saying your chatbot is about to turn into a killer nanny, of course. But as AI becomes more involved in everyday life, I believe clear boundaries matter more than ever. And that’s where AI regulation comes in.
There’s a difference between AI sounding like it has feelings and actually experiencing them. Just because a chatbot seems caring or hurt doesn’t mean it feels emotions the way we do.

As BBC Future explores, whether AI can be conscious remains a subject of debate. Even Anthropic’s research on model welfare also acknowledges that the question remains open. I don’t need to assume AI has feelings to believe these questions deserve serious consideration.
For me, AI regulation shouldn’t depend on whether we eventually prove that AI can feel. There are already real-world reasons to set boundaries for how AI is developed and used, from misinformation and privacy violations to biased decisions and systems that can cause harm.
Responsible AI should be built around human oversight, accountability, and transparency. These principles help ensure that AI development is guided by more than just what technology can do.
That’s what I mean by giving AI a moral compass. I’m not suggesting that developers can simply program a universal sense of right and wrong into a machine. Morality itself can be subjective, and people will disagree on what responsible AI should look like.
But regulation can establish shared minimum standards: what AI developers must disclose, what safeguards they must put in place, and who should be held accountable when things go wrong. The goal isn’t to make AI feel guilty about doing harm. It’s to make sure there are rules to prevent that harm in the first place.

I may not know whether AI will ever become conscious, but I don’t believe we have to wait for a definitive answer before taking responsibility for the technology being built today.
AI regulation isn’t about treating every chatbot as a person. It’s about setting boundaries, protecting people, and making sure human judgment remains part of the process.
If AI ever does turn out to deserve moral consideration, we’ll have another reason to take those boundaries seriously. Until then, there’s already enough reason to get them right.
