Ten Years Ago, People Taught a Chatbot to Be Terrible. Now AI Can End the Conversation.

The story of Tay

Back in 2016, Microsoft launched a chatbot called Tay on Twitter. It was designed to have conversations with people, with the idea that those interactions would help Microsoft improve the system.

Microsoft described Tay as “A.I. fam from the internet that’s got zero chill.” It was aimed at 18 to 24 year olds in the US, built partly with the help of improvisational comedians, and available on Kik and GroupMe as well as Twitter. The promise was simple: “The more you chat with Tay the smarter she gets.”

It didn’t go quite as planned.

Within hours, some users had figured out how to manipulate Tay into posting racist and offensive messages. Microsoft took it offline less than a day after launching it.

I remember that story. It was quite a scandal at the time, and I was surprised by how quickly things had gone wrong. Microsoft later explained that a coordinated group of users had exploited a vulnerability in the system. Tay hadn’t developed offensive beliefs. People had deliberately manipulated its responses.

Recently, I came across something that reminded me of Tay.

Anthropic, the company behind Claude, has announced new rules against repeatedly abusing its AI models. And certain Claude models can now end conversations themselves when someone persistently behaves abusively.

We’ve gone from people trying to make chatbots behave badly to AI companies considering how people should behave toward chatbots.

That’s quite a change in just ten years.

AI can now end certain conversations

In August 2025, Anthropic announced that Claude Opus 4 and 4.1 could end conversations in rare situations involving persistent, extreme abuse or harmful requests.

This isn’t about someone getting frustrated, disagreeing with Claude or asking difficult questions. The feature is intended as a last resort, after the model has already tried to redirect the conversation and there’s little chance of a productive exchange.

Even then, the user can start a new conversation. Claude is simply allowed to end that particular interaction.

Then, on October 8, 2026, Anthropic announced an update to its usage policy, which takes effect on November 12. Among the changes is a prohibition on sustained, needless abusive or cruel behaviour toward its AI models.

The company explains that ordinary frustration, criticism, creative work involving dark subjects, and legitimate research or testing aren’t covered. The rule is intended for extreme cases of repeated cruelty without a clear purpose.

I can understand why a company would want to set boundaries around how its technology is used. But there’s something else going on here that caught my attention.

Anthropic is also researching the possibility of AI welfare.

Can an AI actually be hurt?

When Anthropic introduced Claude’s conversation-ending feature, it said:

“We remain highly uncertain about the potential moral status of Claude and other LLMs, now or in the future.”

Now that’s something I hadn’t expected to be reading in an announcement about a chatbot.

Anthropic isn’t claiming that Claude is conscious or that it experiences suffering. The company is acknowledging that we don’t really know enough about these questions yet, and it’s exploring whether some precautions might make sense.

An AI system can produce language that sounds emotional. It can describe preferences, appear uncomfortable with a request or respond as though something bothers it. But that doesn’t mean it’s actually experiencing those feelings.

We don’t currently have a reliable way to establish whether AI systems have subjective experiences.

So does it matter how we treat something that may not feel anything at all?

There are different ways to look at this. AI is software, and we need to make sure it’s safe, reliable and used responsibly. But I can also understand why researchers are starting to investigate whether future AI systems might deserve different consideration.

And then there’s the human side. If someone spends hours deliberately being cruel to a chatbot, does that matter even if the chatbot doesn’t experience anything? Could the way we interact with something that responds like a person influence how we behave elsewhere?

I don’t know. We have plenty of questions and not many clear answers yet.

But I think it’s worth exploring, especially as AI becomes part of so many areas of our lives.

What a difference ten years makes

One thing I’ve always enjoyed about technology is watching what happens once people start using it. You can design something with one purpose in mind, and then discover all sorts of things you hadn’t anticipated.

Sometimes people find problems the developers missed. Sometimes they find entirely new uses. And sometimes the technology brings up questions that go well beyond how it works.

That’s what I see happening here.

In 2016, Microsoft was dealing with people manipulating a chatbot into producing offensive messages. Ten years later, Anthropic is considering questions about possible AI welfare and introducing rules about how people interact with its models.

Of course, these are different situations, and none of this proves that AI can experience anything. But it does show how the conversation around AI is changing.

And I think that’s worth paying attention to.

We’re developing technology at an incredible pace, while still figuring out what it means for us and how we want to interact with it.

When Tay was making headlines, I certainly wasn’t thinking about whether a chatbot should be able to end a conversation.

Now we’re discussing exactly that.

It makes me wonder what we’ll be talking about ten years from now.