Why AI Gets It Wrong, and How to Know When to Trust It

Have you ever had a conversation with AI where it gets something wrong, you correct it, and somehow it just keeps getting worse? You explain again, it apologizes, gives you another answer, and that one isn’t quite right either. Sometimes starting a completely new chat works better than trying to fix the original one.

But why does that happen? And how can something capable of writing code, analyzing research and solving complicated problems also make such basic mistakes?

Part of the answer is in how language models work. They’re very different from traditional computer programs, and some of the characteristics that make them so useful are also responsible for their weaknesses.

The same things that make AI useful can make it unreliable

Traditional software follows instructions written by programmers. If you enter the same information under the same conditions, you generally get the same result. The software can have bugs, of course, but its rules are explicitly defined.

Large language models, like those behind ChatGPT and Claude, work differently.

They’re trained on enormous amounts of information, learning patterns in language, facts, relationships and reasoning. When you ask a question, they generate a response using what they’ve learned and the information available to them.

That’s why you can ask the same tool to explain a scientific paper, write a poem, help with a business plan or create a computer program. Nobody had to write individual instructions for every possible question.

But learning patterns isn’t the same as having a verified database of facts.

Anthropic explains this particularly well in its AI Capabilities and Limitations course. It identifies four properties of language models, each with strengths and weaknesses.

Predicting what comes next. Language models generate responses piece by piece, predicting what should follow. This gives them remarkable flexibility with language, but it also means they can produce something that sounds completely reasonable without it being true.

Knowledge. They have learned from vast amounts of information, but their knowledge is uneven. They’re generally stronger on common, well-documented subjects than obscure or recent facts. Tools with web access can retrieve newer information, although finding information doesn’t automatically mean interpreting it correctly.

Working memory. AI can follow a conversation and use earlier messages as context. That’s incredibly useful until the conversation gets long, important details are lost, or an incorrect assumption starts influencing everything that follows.

Steerability, or following instructions. You can tell AI what you want and how you want it done. But unlike conventional software executing a precise instruction, a language model interprets your request. It can misunderstand what you meant, follow the wrong part of an instruction or become overly agreeable.

I think understanding these differences makes AI much less mysterious. When something goes wrong, there’s often a reason for it. And once you recognize the type of problem, you have a better idea of how to fix it.

Why AI can sound so sure when it’s wrong

In September 2025, OpenAI published research explaining why language models hallucinate. A hallucination is when AI generates information that isn’t true, sometimes with complete confidence.

The researchers used an interesting comparison with school exams.

Imagine you’re taking a test. If you leave an answer blank, you get zero. If you guess, you might get lucky. So even when you don’t know the answer, guessing can improve your score.

Many AI evaluations have worked in a similar way. Models are rewarded for getting answers right, but aren’t always sufficiently rewarded for acknowledging uncertainty.

OpenAI demonstrated this with two models answering simple factual questions.

The older o4-mini model answered 24% correctly but got 75% wrong. It declined to give a specific answer only 1% of the time.

The newer GPT-5 thinking mini model answered 22% correctly, slightly fewer than the older model. But it gave wrong answers only 26% of the time, declining to answer the remaining 52%.

So the older model technically scored a little better on correct answers, while being wrong almost three times as often.

That makes quite a difference when you’re relying on the answer to make a decision.

And it’s not just theoretical. In October 2025, a study coordinated by the European Broadcasting Union and led by the BBC evaluated more than 3,000 answers from ChatGPT, Copilot, Gemini and Perplexity to news-related questions. Journalists found that 45% contained at least one significant issue, including inaccurate information, misleading sources or missing context.

That figure doesn’t mean 45% of everything AI says is wrong. The study specifically examined news-related answers, but it demonstrates why a convincing response isn’t necessarily a reliable one.

AI has improved, and so have our ways of using it

The technology is improving quickly.

Models have become better at reasoning, following instructions, recognizing uncertainty and using tools to check their answers.

For example, OpenAI reported in 2025 that its GPT-5 thinking model had a 65% lower hallucination rate than its earlier o3 model, measured across the factual claims in its answers to prompts resembling real ChatGPT conversations. That’s a substantial improvement, although it is a company-reported result from a specific test, not a guarantee of accuracy across every task.

Modern AI assistants can also search the internet, read documents, run calculations and work with other software rather than relying entirely on information learned during training.

But a better model doesn’t eliminate the need to check its work. And there’s another part of this that matters: how we use AI.

Anthropic’s March 2026 Economic Index found that people who had used Claude for at least six months had about a 10% higher success rate in their conversations than newer users, as rated by Claude itself. The researchers said this could reflect more sophisticated early adopters, but it could also be evidence of learning by doing.

Still, it raises an interesting point. Using AI effectively is a skill we can develop, not just a matter of having access to the newest model.

Why starting a fresh chat can actually help

Remember that conversation that kept going wrong?

AI uses the history of your conversation to decide what to say next. If it made an incorrect assumption earlier, that assumption may still be influencing later answers. Even after you’ve corrected it, the original mistake remains in the conversation.

The longer you keep trying to fix things, the more complicated that history can become.

A fresh conversation removes those previous exchanges. You can provide the correct information and start again without all the earlier wrong turns. It won’t solve every problem, and you’ll need to provide any important context again, but it can help.

Felix Rieseberg, who leads engineering for Claude.ai, Cowork and Claude Code Desktop at Anthropic, takes this idea further.

In an October 2026 interview, he explained that he rarely wants the same AI session to judge its own work.

Instead, he starts several separate sessions and asks each one to prove the work wrong: that the facts are wrong, that the thinking is wrong, or that something was misunderstood. He then has another session evaluate which criticisms are valid.

You can do something similar using different AI models, such as ChatGPT, Claude and Gemini. Rather than asking one model to confirm that another is correct, ask it to find errors, challenge assumptions and identify anything that lacks evidence.

There’s an important distinction, though. Two or three AI models agreeing with each other doesn’t prove they’re right. They may have learned from similar sources or make similar mistakes.

Felix also emphasizes evidence. For fact-checking, he asks AI to find primary sources for every number he mentions, which gives you something concrete to check.

That makes much more sense to me than repeatedly asking the original AI, “Are you sure?”

First-principles thinking: what do we actually know?

Another concept worth understanding is first-principles thinking.

Dianne Penn, Head of Product for Anthropic’s AI Research and Labs teams, discussed it in a July 2026 interview on Lenny’s Podcast. She identified first-principles thinking as one of the main qualities her team looks for when hiring product managers.

The idea is fairly simple: instead of accepting how something has always been done, go back to the basic facts and work out what actually makes sense.

For example, imagine asking AI how to fix a slow website. It recommends rebuilding the entire site.

You could accept that recommendation and start planning a rebuild.

Or you could question it. What exactly is making the website slow? Is it the hosting, the images, the plugins or something in the code? What evidence supports the recommendation? Could the problem be solved without rebuilding everything?

That’s first-principles thinking in practice.

It’s not specifically an AI technique, and it doesn’t replace fact-checking. But I think it’s particularly valuable when working with AI because it helps us question the assumptions behind an answer, not just the answer itself.

And AI can help with that too. Ask it to identify what it’s assuming, what information is missing, and what evidence would change its recommendation.

So how do we know when to trust AI?

Anthropic has another course called AI Fluency: Framework and Foundations, developed with professors Rick Dakan and Joseph Feller. It teaches four skills for working with AI: Delegation, Description, Discernment and Diligence.

Two are especially relevant here.

Discernment means evaluating what AI gives you. Is the answer accurate, relevant and complete? Does the reasoning make sense? Is anything missing?

Diligence is about taking responsibility for how you use it. Have you chosen the right tool? Have you checked the important information? Are you comfortable standing behind the result?

You don’t need to investigate every sentence AI writes.

If you’re brainstorming ideas or looking for a different way to phrase something, a quick review may be enough.

If you’re making business decisions, preparing client work, publishing research or dealing with legal, financial or health information, the consequences of getting something wrong are much greater. That’s when checking original sources, reviewing calculations and getting appropriate professional advice becomes important.

A useful starting point is to ask yourself three questions:

1. What do I actually know to be true?
2. What is AI assuming or telling me that I haven’t verified?
3. How important would it be if this answer were wrong?

You can even ask another AI to help investigate those questions, as long as you don’t confuse its agreement with proof.

In Stack Overflow’s 2026 Developer Survey, 48% of respondents said they trusted AI when they could easily verify its output. Only 6.6% trusted it for many tasks, including important work decisions.

I think that’s a useful distinction.

Two free courses I’d recommend

If you use AI regularly, I highly recommend these two courses from Anthropic. Both are free, and you don’t need a technical background to benefit from them.

AI Capabilities and Limitations↗ (opens in a new tab) explains how language models work, why they behave the way they do, and where their limitations come from.

AI Fluency: Framework and Foundations↗ (opens in a new tab) focuses on working effectively with AI, evaluating its output and using it responsibly.

The lessons apply beyond Claude. They’re useful for anyone working with language models.

I love what AI makes possible, and I don’t think we need to choose between being enthusiastic about it and understanding its limitations.

The models will continue to improve. But knowing how they work, recognizing when something doesn’t add up, and asking better questions are skills worth developing ourselves.

Maybe the question isn’t simply whether we can trust AI. It’s whether we know enough about what we’re asking it to do to recognize when it might be wrong.