The story
I set out to build a system that would monitor my server and fix problems by itself. The idea was reasonable enough — if something went wrong at three in the morning, the agent would identify it and deal with it without waking anyone up.
I asked the AI to design the monitoring and the self-healing systems. It did. The results looked plausible, were confidently presented, and turned out to be largely unreliable. It took about three weeks before I understood what was actually happening: the AI was not drawing on any established body of knowledge. It was guessing — intelligently, fluently, but guessing nonetheless — and presenting those guesses as if they were considered recommendations.
I started again. This time I gave the AI a different instruction: go out and find established checklists and documented best practice for this kind of system. Research it properly, find sources you can show me, and then implement against those. The difference was considerable. Once it had something verifiable to work against, the systems it built were ones I could trust.
The lesson, in my experience, is this. AI will produce confident, authoritative-sounding output whether or not it has any reliable basis for what it is telling you. It is not consulting a body of verified knowledge. It is pattern-matching against whatever it was trained on, and presenting the result as if it were considered advice. For routine tasks, in well-documented territory, this often works well enough. For anything specialised, you cannot simply describe what you want and assume the result is sound. You need to give it something solid to implement against — established sources, verified checklists, documented practice — and then check the output against those same sources yourself.
The point
[to develop — you cannot assume AI advice is grounded in verified knowledge. For anything that matters, supply the verified sources yourself and check the output against them.]