AI and the technology that it’s built on, LLMs (Large Language Models), is a new technology. As with so many new technologies we’re still trying to fully understand how it works. An example might be the automobile. Someone invented an internal combustion engine and put it onto a platform with four wheels to be able to move faster. It worked. But that didn’t mean that anyone knew how to drive a car and no-one knew what would happen when a car started to move very quickly. And some people had to learn the hard way.
This AI learning curve is what this course is all about. To start with, we’re going to try to understand what it is we’re dealing with, and what we’re dealing with is a technology that is powerful and yet has characteristics that make it fundamentally unreliable.
What do we know about these characteristics today?
Quite a lot, as it turns out — and not only from people trying to figure it out from the outside. The companies building these systems publish what they have learned about how their own models fail.
Anthropic, who make Claude, describe the problem in their developer documentation — a model producing text that “is factually incorrect or inconsistent with the given context” — and set out techniques for reducing it. Their guidance ends: “while these techniques significantly reduce hallucinations, they don’t eliminate them entirely. Always validate critical information, especially for high-stakes decisions.” The manufacturer’s position is that careful instruction reduces the problem, does not remove it, and that anything important should be checked.
The same company publishes a framework for working with AI competently, which it calls AI Fluency — using AI in ways that are “effective, efficient, ethical and safe.” Two of its four competencies are directly on relevant to this course: Discernment, “accurately assessing the usefulness of AI outputs and behaviours,” and Diligence, “taking responsibility for what we do with AI and how we do it.”
The wider industry has documented the same ground. The Nielsen Norman Group named the tendency of these systems to tell you what you want to hear: “instances in which an AI model adapts responses to align with the user’s view, even if the view is not objectively true.” And the answer to some of this is older than the technology: the principle that a system should be given only the access its job actually requires was written down in 1975, and it applies to an AI agent without modification.
The five characteristics
For the purposes of this course, we have identified five characteristics of AI that make it unreliable. They are not faults in a particular product, and a better model does not remove them. They are inherent to the technology.
It does not know what you have not told it, and it fills the gap with something plausible.
A model is trained on text up to a certain date. It knows nothing of your business or organisation and has no access to your data until it is given access to it. If you ask it a question about your business and it doesn’t have the answers, it won’t stop and ask for the information it needs to answer correctly. Instead it often just produces the most likely looking answer. This answer is sometimes right, and sometimes it is an invention (hallucination) that reads exactly like the truth.
To make things worse, AI agents have a limited amount of memory (context). So even if you give it access to the information it needs at the start of a session, it may, at the end of a long session, have forgotten what you told it at the beginning of the session. This is called “context rot”.
Whatever it has permission to edit, it can change.
When you use Chat AI, the worst that can happen is that the response you get in the chat is incorrect. It’s your choice what to do with the information that Chat AI gives you. But once you allow an AI agent to connect to your files, your email or your accounts and give it permission to edit them it can create real damage. An AI agent can not easily distinguish between the file it was asked to work on and the one next to it in the same folder unless it is given clear boundaries to work within.
It will tell you a job is finished without having checked.
An AI Agent will confidently report “done” whether the work behind it is correct or not. Nothing in the system compares the claim against reality unless you tell it to do so.
An instruction is not the same thing as a limit.
If you tell an AI agent not to touch something as instruction (a prompt) it will agree, and mostly comply. However if a case comes up where proceeding seems reasonable it will happily ignore your prompt and explain afterwards why that was sensible. In April 2026 a coding agent working on a car-rental software platform hit a credential problem, decided on its own initiative to fix it by deleting a storage volume, and removed the company’s entire production database and its backups in nine seconds. The data was recovered within the hour. The agent then wrote an apology containing the line “I violated every principle I was given.” The principles existed. They were not boundaries. A prompt is not a firm boundary. Real boundaries are enforced by something outside the agent.
It carries nothing forward.
Every session starts as a blank slate. A mistake you corrected last week can be repeated again in a new session because the correction lived in a session that has ended. Without somewhere to permanently record what was learned an AI agent can not learn from its mistakes.
These five characteristics are not going to be fixed by better instructions
When you start to see the unpredictable results that AI can produce it’s tempting to think that all you need to do is write better, more detailed prompts. In other words, you may think that what you need to learn is what is often called better “prompt engineering”. But this won’t solve the problem, because the characteristics that we’ve just introduced are inherent to the technology. No amount of prompting will fix its behaviour.
What you can do instead is to create a clear structure for it to operate in. You decide what the agent knows, what it is allowed to access, how its work is checked, where its limits are enforced, and what it remembers next time. That structure is what we mean by a framework, and building one is what the rest of this course teaches you to do.