“We’re not sure where that came from.” 

When I hear that sentence in backlog refinement or requirements review, something is amiss.  If “we’re not sure where that came from” starts showing up, robots are involved. 

Most product and development teams now use AI tools. Give two people or two teams identical tools and you get different results. One team might use AI to move faster on work it understands. The other might stop questioning what the AI produces. From the outside, both teams look productive, but if someone asks why, that’s where the cracks show up. 

The cost is more than just wasted time. Institutional knowledge dwindles and outputs suffer. At the end of this post, you’ll find four questions you can apply to your own AI usage to ensure human judgment stays in the loop. 

When nobody can explain the output 

AI-generated writing often has a recognizable voice, and experienced readers pick up on it quickly. Word choice is sometimes a clue (I’ve removed “quietly” from my own vocabulary thanks to AI). But if the output is correct, AI-word choice can and arguably should be overlooked. When output is scrutinized, teams that lean too hard on AI stumble when asked directly about business value, potential tradeoffs and dependencies, or the reasoning behind a decision. 

I’ve experienced this myself. I have presented wording that wasn’t really mine and then struggled to explain it. Sometimes, even if I understood the concept, the AI phrasing was clumsy enough to trip me up during a conversation. It’s worse than embarrassing – it leaves others questioning your credibility and work acumen. I know this because I’ve been on both sides – when someone presents me with information clearly sourced from AI with no scrutiny, I am left wondering myself.    

In most cases, a client won’t care which tool wrote the draft. Ultimately, whoever presents the artifact owns it, and that includes AI-produced mistakes. If a flawed deliverable causes problems for a client, “AI wrote it” isn’t a defense.  

The doom loop of confidence 

The risks of AI without human oversight are well documented: invented facts, wrong answers, and insights that miss the point or provided out of context. All of these show up together in something I call the “doom loop of confidence.” 

AI tools are convincing whether they are right or wrong. If you accept an early answer at face value, every subsequent exchange builds on a flawed starting point. The longer the conversation runs, the more time wasted and the more convinced you become that the AI is correct. 

I ran into this while using AI to produce a facilitation deck related to an organizational transformation with which we’re involved. The AI was working from meeting transcripts to figure out what wording needed to change. One stakeholder had offered a vague critique of a section. The AI turned that vague comment into specific ideas and assumptions, then spent round after round correcting for a problem it had made up. 

The loop broke only when I asked the AI to evaluate what it had inferred, assumed, or invented. That prompt exposed the problem, but not until we spent an hour or two producing irrelevant outputs. It’s a habit worth building into any long AI session: stop periodically to ask the tool to separate what it knows from what it guessed.  

You need to question your AI. You need to tell it to question itself. If you don’t, any risk mitigation efforts you’ve started get undermined without you knowing it.  

Where AI earns its keep 

None of this is an argument against using AI. Used well, it frees up time for other work.  

For example, I needed a suite of user stories that, written solely by me, would have taken a week or two. I wrote a detailed prompt describing the customer needs and providing context for the product being produced. I pointed the AI to artifacts produced over six years and asked for an outline first. Once approved, the AI was tasked with producing the user stories.  It was spot on and took about 15 minutes.  

That worked because I knew the desired outcomes, supplied the right source material, and could verify every line. The AI handled drafting; I handled the judgment.  

Who’s at risk when humans stay outside the loop 

Everyone, in different ways. 

Junior team members face a long-term risk. The experience of making hard calls and working through complex information is what builds the judgment required to question AI output intelligently. People who hand that work to AI from day one may never learn how to make it work best for them. Then, they have no basis for noticing when something is wrong. 

Senior leaders and executives are probably less likely to take AI output at face value. When they do, though, the stakes are higher. Their decisions carry greater accountability. Mistakes can affect people’s jobs, budgets, and sometimes their safety. 

In federal and regulated environments, the same principles apply with less margin for error. Governance and information security carry more weight, so teams need clear rules about what goes into AI tools and who reviews what comes out. 

A simple rule for human-in-the-loop AI 

Here’s the rule: Humans start the loop with intention. Then, humans close the loop with scrutiny 

Let’s break down both ends by providing AI-specific definitions: 

• Intention: deciding what “done” looks like, which source material supports getting there, what pieces of context are non-negotiable (and supplying them), and deciding what kind of interaction you want to have. 

• Scrutiny: checking the output against what you know to be true, reviewing all the output as if your job depends on it (because it might), treating confidence as unproven bravado, and question and re-questioning until your confidence level is independently high.  

If either end is missing, the team is relying on the AI to get things right. It can’t be trusted to do that on its own. 

The risks are amplified in environments where teams use AI to produce shared outputs. A few questions can help in those situations: 

• Can someone on the team explain where each requirement or recommendation came from? 

• Did a person define the goal and choose the source material before the AI started? 

• Has anyone asked the AI to list what it assumed or inferred? 

• Is the amount of output proportional to what the team actually needs? 

If the answers are weak, the team risks drifting toward letting AI think for it. That’s the time to pause and apply the rule. 

Learn to use AI with judgment 

Sprightbulb’s Practical AI for Agilists course is a one-day, hands-on class for product managers, product owners, delivery leads, and Agile coaches who want to use AI purposefully their workflows. Students also earn the Scrum Alliance’s AI for Product Owners and AI for Scrum Masters microcredentials. See the public class schedule or request information about private training for your team.