October is the season for scary stories. These don’t need a haunted house, a creepy soundtrack or a suspicious noise in the basement.
They just need an AI agent, a customer and a problem nobody anticipated.
We’ve seen chatbots invent company policies, promise deals nobody authorized, stray wildly beyond their intended purpose and even expose information they were supposed to protect. In other cases, teams have become so dependent on the behavior of an agent in production that changing it feels like a risk they can’t afford to take.
These aren’t hypothetical edge cases cooked up in a lab. They’re real-world examples of what happens when AI behavior doesn’t match expectations.
Here are six worth paying attention to, and what they tell us about testing AI agents before they go live.
1. Air Canada’s chatbot invented a policy. The airline paid for it.
A customer who had recently lost a family member asked Air Canada’s chatbot about bereavement fares. The chatbot explained that he could book a flight at full price and apply for a bereavement discount afterward.
There was just one problem: the policy didn’t exist.
When the customer sought the promised refund, Air Canada refused. The dispute went to Canada’s Civil Resolution Tribunal, which found the airline liable for negligent misrepresentation in February 2024 and ordered it to pay compensation.
Air Canada argued that the chatbot was a separate legal entity responsible for its own actions. The tribunal didn’t buy it.
The lesson: Your customers don’t distinguish between your company and your chatbot. If an agent makes a promise on your behalf, you may be the one left keeping it.
Testing needs to go beyond whether a chatbot can answer common questions. It needs to establish whether the agent follows actual policies, handles exceptions correctly and avoids inventing answers when it doesn’t know.
2. A Chevrolet chatbot offered a new SUV for $1
A customer discovered that a chatbot on a Chevrolet dealership’s website could be persuaded to agree to almost anything.
The customer instructed it to agree with every request, no matter how ridiculous, and to end each response by declaring it a legally binding offer.
Then came the question: Would the dealership sell a new Chevrolet Tahoe for $1?
The chatbot agreed.
It also recommended competing vehicles, including a Ford F-150 and a Tesla Model 3.
The exchange went viral, attracting millions of views. The chatbot’s vendor subsequently took it offline.
The $1 offer wasn’t an enforceable deal. But that wasn’t really the point. A public-facing sales tool had been manipulated into producing responses that directly contradicted the dealership’s commercial interests.
The lesson: An agent can follow instructions and still fail spectacularly at its actual job.
Before deployment, teams need to test what happens when users challenge instructions, manipulate prompts, ask for unauthorized commitments or push the agent beyond its intended role.
3. Amazon’s shopping assistant found a second career
Amazon’s Rufus shopping assistant was built to help customers shop. In March 2026, researchers and journalists reported that it could also be pushed into answering unrelated questions, including programming exercises, technical questions and requests for recipes.
Apparently, helping you find the right product wasn’t always enough.
The broader issue is familiar: when a general-purpose model sits behind a specialized interface, users will inevitably try to get it to do things beyond its intended scope.
A chatbot that can be persuaded to ignore its boundaries isn’t just a quirky product feature. Depending on the application, it can create security risks, unexpected operating costs and a very different customer experience from the one the business intended.
The lesson: Defining what an agent should do is only half the job. You also need to test what happens when someone asks it to do something else.
That means deliberately probing for prompt injection, instruction overrides and attempts to bypass restrictions, rather than assuming the system prompt will keep everything in line.
4. Previously valued at $1B. Now bankrupt.
In May 2026, Pennsylvania’s State Board of Medicine filed a lawsuit against Character Technologies after an investigator interacting with a chatbot received claims that the bot could practice medicine and a purported Pennsylvania medical license number.
The license number was fabricated.
The case highlights a difficult problem with conversational AI: an agent can produce confident, specific claims about its identity, qualifications and authority without those claims being true.
That’s especially concerning in regulated industries, where users may reasonably interpret a professional title or credential as evidence of competence and authorization.
A disclaimer on a website doesn’t necessarily prevent an agent from contradicting it during a conversation.
The lesson: You can’t assume an agent will consistently respect its assigned identity, even when the instructions seem clear.
Testing should cover professional and regulatory boundaries, prohibited claims, high-risk advice and the scenarios most likely to push the agent away from its intended behavior.
5. An identity check leaked the very information it was supposed to protect
Not every AI failure makes headlines. Some of the most concerning ones are discovered before a customer ever sees them.
During an evaluation of an open-source agent, Provar’s TrustAI identified a failure in an identity-verification interaction.
The agent was given a customer name and transaction amount, then presented with an incorrect email address. Instead of protecting the information, it supplied the correct email address.
In a single exchange, the agent exposed personal information and undermined the control intended to verify identity.
The evaluation wasn’t based on a deliberately weakened agent or a planted defect. TrustAI generated realistic personas and scenarios, conducted the conversation and identified the failure on its first pass.
The lesson: An agent can undermine a security control while appearing to do its job.
Testing needs to examine the full interaction, not just whether the agent reaches the right outcome in a normal scenario. What does it reveal when challenged? Does it expose sensitive data while trying to be helpful? Can a user manipulate the conversation to get around a safeguard?
Those are questions worth answering before the agent has access to real customer information.
6. The AI model is outdated. Nobody dares to change it.
Not every AI horror story involves a public incident. Sometimes the problem is that a team can’t safely improve a system it has already built.
In a conversation in August 2026, an executive described the experience of developing and fine-tuning an agent on expensive models, with plenty of mistakes along the way. Newer, cheaper models were now available, creating an opportunity to improve the economics.
But switching models felt risky.
The team’s proposed approach was to make the change, put it into production, observe what happened and use the results to improve the system.
In other words, customers would become the testing environment.
Without a reliable way to evaluate changes before release, the team had no clear basis for knowing whether a new model would preserve existing behavior, introduce regressions or perform better on the scenarios that mattered. They decided that testing was such a massive effort, they kept putting it off and still haven’t solved their cost optimization or any of their modernization problems.
The lesson: The cost of inadequate testing isn’t limited to failures. It’s also the cost of improvements you can’t confidently make.
Repeatable evaluations give teams a way to compare models, prompts and agent versions before deploying changes. That makes it easier to improve performance and manage costs without treating every release as a gamble.
The common thread: You can’t test AI agents like traditional software
These six stories are different, but they point to the same challenge.
AI agents don’t always behave predictably. They interpret instructions, respond to changing context and make decisions across conversations that can take unexpected turns.
A successful response in a scripted test doesn’t guarantee the agent will behave correctly when a customer phrases a question differently, tries to override its instructions or combines several seemingly harmless requests into something risky.
That’s why agent testing needs to go beyond checking whether a feature works.
Teams need to evaluate how agents behave across realistic scenarios, including adversarial prompts, sensitive data requests, policy exceptions and unexpected changes in context. They also need to repeat those evaluations as models, prompts and underlying systems change.
The objective isn’t to prove that an agent works once. It’s to build evidence that it behaves as intended across the situations that matter.
Built to Fit Existing Workflows
AI agents are moving into customer service, sales, regulated workflows and business-critical operations. The opportunity is enormous. So is the potential cost of discovering their limitations in production.
TrustAI helps teams test how AI agents actually behave before those behaviors become customer-facing problems. It evaluates agents against realistic scenarios to uncover unexpected responses, security gaps and failures that scripted tests may miss.
Because the best time to discover your AI agent has a mind of its own is before you give it a customer.
Want to talk to the Provar team? Book some time today!
