The Day a Refund Went Wrong
Picture this: a customer asks your local business's chatbot, "Can I get a refund on this order?" The agent checks the order status, reviews your refund policy, confirms the item hasn't shipped, and then—without asking—calls the refund function. The API returns success. Logs look clean:
tool: refund_order
order_id: 20260710001
result: successBut the customer only asked if they *could* get a refund. They never said "do it." The tool call worked. The business action was wrong.
This isn't just a hypothetical—it's the kind of failure that happens when local businesses rush to deploy AI agents without thinking through governance. And it's exactly what a new framework called Knowledge-driven Computing (KDC) aims to fix.
What Logs Miss
When something goes wrong, you can usually trace which tool ran and what it returned. But try answering these questions from a typical log:
- What did the agent think the customer wanted?
- Why did it decide to act instead of just answering?
- Which policy said a refund was allowed?
- Why did it assume "can I" meant "please do it"?
- What risk level was this action?
- Who authorized the call?
Logs record what happened, not why it happened. In traditional software, the "why" is baked into the code—a refund button leads to a confirmation step, then a backend check, then the transaction. The logic is static and auditable.
But AI agents decide at runtime. The same phrase "can I get a refund" could be a question, a request, or just a comparison of options. When choices happen in real time, the reasoning behind them needs to be captured too.
Introducing the Reasoning Object
KDC proposes something called a Reasoning Object—a structured record of why an agent made a decision. It's not a prompt (that's just input), not a trace (that's just performance metrics), and not a full chain-of-thought (that's too messy and often unreliable).
A Reasoning Object for a high-stakes action should include:
- Goal: What the task was supposed to accomplish
- Context: User, order, time, session, system state
- Knowledge cited: Which policies or data versions were used
- Memory used: Any past preferences or precedents
- Evidence: Facts, documents, or human confirmations
- Key judgments: The steps from goal to conclusion
- Conclusion type: Is this a fact, a suggestion, a plan, or an action request?
- Uncertainty: Was there enough information?
- Risk: What happens if this is wrong?
- Action recommendation: Should a capability be invoked?
- Feedback: Did reality confirm or refute the judgment?
In our refund example, the object should separate two things:
- Fact: This order qualifies for a refund.
- Authorization: The user explicitly requested a refund.
The first can be supported by order data and policy. The second needs a clear user confirmation. Mixing them up is how mistakes happen. If the system tags the task as "eligibility inquiry" and marks "user confirmation" as a prerequisite for action, the control layer can stop the agent from jumping from advice to execution.
From Judgment to Action: The Causal Chain
A governed AI action shouldn't leave just a tool call. It should form a chain:
Business goal → Knowledge & evidence → Reasoning Object → Capability recommendation → Policy & permission check → Capability call → Execution result → Real-world feedbackThe Reasoning Object sits between knowledge and action, acting as the anchor. Auditors can see that a refund wasn't triggered because the model "felt like it," but because a specific goal was identified, a certain policy version was cited, order facts were checked, risk was assessed, and an action was recommended.
It also makes rejection a normal outcome. If evidence is missing or the user hasn't confirmed, the object can say "cannot act." The system doesn't wait for the tool to fail—it refuses upfront.
Not every query needs this heavy structure. Low-risk read-only calls can skip it. But for refunds, payments, permission changes, external notifications, approvals, and irreversible actions, you need at least the equivalent of goal, evidence, risk, and authorization records.
Tool, Skill, Capability: Three Layers of Execution
KDC also argues we need to separate three concepts that often get lumped together:
- Tool: How to execute a specific operation (e.g.,
get_order_status,refund_order) - Skill: How to organize multiple steps around a business goal (e.g., "handle order refund" includes checking status, verifying policy, getting confirmation, executing refund, handling exceptions)
- Capability: The governance wrapper—what conditions allow this action to be performed, who owns it, what's the risk level, what are the prerequisites and rollback plans
Too often, teams just give the model a tool and hope the prompt keeps it in line. But prompts are fragile. If the refund tool is wide open, you're relying on a probabilistic model to always remember the rules. That's not a safety net.
Capabilities bring key constraints into the system itself:
- Permissions: Is the current user allowed to do this?
- Risk level: Does the amount or scenario need extra checks?
- Policy: Are all prerequisites met?
- Human-in-the-loop: Do we need explicit user confirmation or manager approval?
- Rate limiting & circuit breakers: Stop abnormal calls
- Transactions & compensation: Avoid half-done states
- Audit trail: Who decided and why
- Lifecycle: Is this capability in beta or stable?
In the refund case, the control plane can enforce that a refund capability requires a clear user confirmation. Even if the agent messes up or the skill misses a step, the capability won't execute.
The Agent's Real Role
KDC doesn't see the agent as the all-in-one container. It's not responsible for everything—understanding, memory, reasoning, tool selection, execution, error handling, and auditing all at once. That's a recipe for unmanageable complexity.
Instead, the agent is a coordinator. It can interpret goals, gather context, choose skills, and recommend actions. But high-impact judgments need to be recorded in a Reasoning Object, and high-risk actions need to pass through capability governance. The agent can suggest, but it doesn't get to act just because it reasoned.
Practical Steps for Local Businesses
If you're running a local SEO agent—whether it handles bookings, refunds, or service requests—here's how to start building guardrails:
- Pick one high-impact action (e.g., issuing a refund, changing an appointment, sending a confirmation email).
- Map the causal chain: Goal → knowledge → reasoning → capability selection → permission check → user confirmation → execution → feedback.
- Create a minimal AI Action Record: For each important action, log the goal, the evidence used, the risk level, who authorized it, and the real-world result.
- Look for four gaps:
- Causal gap: You see the tool call but not the reasoning.
- Semantic gap: The tool has parameters but no business meaning or owner.
- Governance gap: Key rules live only in the prompt, not in system policy.
- Feedback gap: The API succeeds but no one verifies the real-world outcome.
You don't need a fancy new platform. You can start with your existing logs and add a few structured fields. The goal is to be able to answer: Why did this action happen? Who allowed it? On what basis? And what was the result?
Calling Tools Isn't the Hard Part
Getting a model to call a function is easy. The hard part is making sure it calls the right function at the right time for the right reason—and stops when it shouldn't.
For local businesses, this matters even more. A refund mistake or an unauthorized booking can cost you a customer and your reputation. The KDC framework reminds us that success isn't just about execution—it's about judgment, authorization, and accountability.
So next time you're tempted to let your AI agent loose, ask yourself: If it calls the refund tool and it returns success, is that the same as doing the right thing? Probably not.
Comments (0)
Please sign in to post a comment.
Don't have an account? Create one
No comments yet. Be the first to comment!