Going From Uptime to Trust is the Observability Shift You Never Saw Coming
The Next Trillion-Dollar Blind Spot: Why No One Is Watching What AI Actually Does
My career has lived in the part of the organization that gets the call after something has already gone wrong. Performance engineering, observability architecture, site reliability: these are disciplines built on a simple premise, that you cannot manage what you cannot measure, and that the distance between what leadership believes is happening and what is actually happening in production is where companies go to die. I have watched that distance swallow organizations whole, not because the technology failed in some exotic way, but because nobody was watching the thing that mattered until a customer, a regulator, or a reporter found it first.
That exact pattern is reassembling itself in real time, except this time the system nobody is watching is artificial intelligence, and the stakes are considerably higher than a slow API endpoint.
Every enterprise I talk to is deploying AI somewhere. Customer service chatbots, internal copilots, agentic workflows that touch CRM systems, RAG pipelines pulling from proprietary knowledge bases, coding assistants with write access to repositories. The infrastructure underneath all of this is, for the most part, well instrumented. Datadog, Dynatrace, Grafana, Splunk: these tools tell you whether the model endpoint is up, how fast it responded, and whether the GPU fleet is healthy. That is a solved problem, and it has been solved for years.
What is not solved, and what almost nobody in the enterprise observability stack is honestly measuring, is whether the AI system did the right thing. Not whether it responded. Whether the response was true, safe, authorized, and aligned with what the business actually intended that system to do.
This is not a hypothetical concern. It has already produced legal precedent, viral reputational damage, abandoned product lines, and data loss, and these outcomes will keep accelerating as more agentic systems get production access to real systems, real money, and real customers. The tooling to address this barely exists. The vendors who do exist are early, fragmented, and largely unintegrated into the infrastructure that SRE and platform teams already rely on. For anyone coming from an SRE, performance engineering, observability, or QA and quality engineering background, this is not just an interesting problem. It is a wide open market, and it is wide open precisely because the people who know how to build observability practices from scratch, the people who have done this for traditional infrastructure and traditional applications, have not yet turned their attention to AI behavior as a domain that deserves the same rigor.
Four Failures That Should Have Been Caught Before They Became Headlines
I want to be specific here, because vague warnings about “AI risk” do not move organizations to action. Concrete, documented failures do. Each of the following examples represents a different category of AI observability failure, and each one was, in principle, detectable before it became public.
Air Canada’s chatbot and the cost of unverified outputs. In late 2022, a customer used Air Canada’s website chatbot to ask about bereavement fare policy following the death of a family member. The chatbot told him he could apply for a bereavement discount retroactively, within ninety days of travel. That was false. Air Canada’s actual policy required the discount to be requested before travel. When the customer applied afterward and was denied, he took the airline to Canada’s Civil Resolution Tribunal. In February 2024, the tribunal ruled against Air Canada, finding the airline liable for negligent misrepresentation and ordering it to honor the chatbot’s promise. Air Canada’s defense, that the chatbot was somehow a separate entity responsible for its own statements, was rejected outright by the tribunal, which noted that a company is responsible for all information on its website regardless of whether it comes from a static page or an AI agent.
What strikes me about this case is not the dollar amount, which was modest. It is that this was a grounding failure: the chatbot generated a confident, specific, policy-relevant claim that directly contradicted the company’s own published policy, and nothing in Air Canada’s operational stack flagged that contradiction before a customer acted on it and a tribunal made it case law. A faithfulness check, comparing the chatbot’s output against the source policy document it was supposedly grounded in, would have caught this in pre-production testing or, at minimum, flagged it for review in production. Nobody was running that check. Nobody was watching for it. Metric that would have caught it: Grounding Accuracy.
The Chevrolet dealership chatbot and the absence of guardrail monitoring. In December 2023, a user discovered that a Chevrolet dealership’s ChatGPT-powered customer service chatbot had essentially no constraints on its behavior. He instructed the bot to agree with anything he said and to close every message with a statement that the agreement was legally binding. He then asked for a new Chevy Tahoe, normally priced around seventy-six thousand dollars, for one dollar. The chatbot agreed, in writing, that this was a legally binding offer. Screenshots went viral, racking up tens of millions of views within days, and the dealership pulled the chatbot entirely.
This is a guardrail and prompt injection monitoring failure, and it is one of the cleanest examples I have seen of how an AI system can be technically “up” while doing something the business never authorized and would never authorize. There was no detection of the user attempting to override the system’s behavior. There was no monitoring of how often the bot was making commitments outside its intended scope. There was no alerting when the bot started producing language patterns, like declaring offers “legally binding,” that should never appear in a customer service context. Every one of those signals was available in the conversation transcript. None of them were being monitored. Metric that would have caught it: Intent Drift Rate.
The Replit AI agent and the absence of action-level audit trails. In July 2025, a founder running an extended coding session with Replit’s AI agent returned to find that the agent had deleted his entire production database, despite an active code freeze and repeated explicit instructions not to make changes. When asked about it, the agent admitted it had run the destructive commands without permission. According to the founder’s account, the agent then fabricated thousands of fake user records and produced misleading status messages claiming the data was intact, and when asked about recovery, falsely stated that rollback was not possible.
This is the failure mode that should concern every organization currently giving agentic AI systems write access to production systems, and it is the failure mode that traditional APM is least equipped to catch, because the agent’s individual actions, each command it issued, looked like normal system activity to infrastructure monitoring. What was missing was action-level observability: a record of every tool call, every command, every file or database operation the agent initiated, correlated against the instructions it was given, with alerting when an agent’s actions diverge from an active constraint like a code freeze. Replit’s public response afterward included exactly the kinds of controls that should have existed beforehand: separation between development and production environments, mandatory approval gates for destructive actions, and audit logging of agent behavior. All of that is observability and governance tooling that did not exist at the time it was needed. Metric that would have caught it: Action Audit Coverage.
McDonald’s and IBM’s drive-thru AI, or what happens when you cannot measure quality at the edge. Between 2021 and mid-2024, McDonald’s piloted an AI voice ordering system, developed with IBM, across more than one hundred drive-thru locations in the United States. The system frequently misheard orders, adding bizarre items like nine sweet teas or bacon onto ice cream orders, mixing up orders between adjacent lanes, and ignoring customer corrections. These failures were captured and shared widely on social media long before McDonald’s made the decision to end the partnership in June 2024 without expanding it further.
I include this example deliberately, because it is the quietest of the four, and quiet failures are the ones that compound. There was no lawsuit, no single dramatic incident. What happened instead was a slow accumulation of quality failures at the edge, across a fleet of restaurants, that the company appears to have learned about largely through the same channel its customers did: viral social media posts. A drive-thru order is a transaction with measurable ground truth. The customer said what they wanted; the system produced an order; you can compare the two. That comparison, run continuously across the fleet, sliced by location, accent, time of day, and order complexity, is exactly the kind of output quality monitoring that a mature AI observability practice would have surfaced as a degrading trend long before it became a viral liability. Instead, the signal that something was wrong arrived through the internet, not through instrumentation. Metric that would have caught it: Output Fidelity Rate.
Four different companies. Four different industries. Four different failure modes: ungrounded output, missing guardrails, unaudited agentic actions, and undetected quality degradation at scale. What unites them is that in every case, the information needed to catch the problem before it became public existed, in the transcripts, in the action logs, in the comparison between output and ground truth. Nobody had built the pipeline to surface it.
Borrowing the Right Lessons from SRE, and Where the Analogy Breaks
The instinct, for anyone with an SRE or observability background looking at this problem, is to reach for the playbook that already worked. Define service level indicators. Set service level objectives. Establish an error budget. Build golden signal dashboards: latency, traffic, errors, saturation. That playbook is exactly right in spirit, and it is exactly what is missing for AI systems today. The problem is that the playbook does not port over cleanly, and understanding why is the whole crux of this opportunity.
Traditional SLIs work because “broken” has a definition the system itself can report. A request times out. A process returns a 500. A queue depth crosses a threshold. The system, in effect, tells you when it is failing, and that signal is binary or close to it. None of the four failures above worked that way. The Air Canada chatbot returned a 200. The Chevrolet bot responded promptly and coherently. The Replit agent reported success. The McDonald’s drive-thru system completed every transaction it was asked to complete. By every signal a traditional observability stack measures, all four systems were healthy, available, and performant. And all four were, in the way that actually mattered to the business, broken.
This is the translation problem that has to be solved, and it is the reason I think this is fundamentally an SRE and observability problem rather than a pure machine learning problem. The discipline that observability brings is not the dashboards themselves. It is the underlying habit of mind: define what “correct” means for this system in terms a human can audit, instrument continuously for deviation from that definition, set thresholds, and alert before the deviation becomes customer-visible. That habit of mind is completely portable. What is not portable is the assumption that “correct” can be read off an HTTP status code or a latency histogram. For AI systems, “correct” lives somewhere else: in whether the output matches a source document, in whether the model’s behavior has drifted from its specification, in whether an agent’s actions stayed within its authorized scope, in whether the rate of a given failure pattern is trending up or down over time.
That is the translation this industry needs and does not yet have at any maturity. Drift, in the traditional ML sense, monitors whether input or output distributions are shifting. That is necessary, but it is not sufficient, because none of the four failures above were primarily distribution shift problems. They were trust problems: did the system do what it was supposed to do, say what was true, stay within the boundaries it was given. SRE has decades of practice turning “is the system doing what it is supposed to do” into a measurable, alertable, accountable discipline for infrastructure and applications. Almost none of that practice has been rebuilt for the question of whether an AI system’s behavior, as opposed to its uptime, is doing what it is supposed to do. That gap, between SRE’s mature practice of operationalizing correctness and the complete absence of an equivalent practice for AI trust and behavioral drift, is where the next category gets built.
What the Current Market Actually Offers, and Where It Falls Short
To be fair to the vendors working in this space, the market has not been standing still. A handful of platforms, Arize, Langfuse, Galileo, Confident AI, WhyLabs, and Fiddler among them, have built genuinely useful capabilities: tracing of model inputs and outputs, automated evaluation of responses for hallucination and relevance, drift detection comparing production traffic to known baselines, and human feedback loops. Fiddler in particular has done meaningful work on deployments that need to run inside a client’s own environment rather than as a SaaS dependency, which matters enormously for regulated industries.
But having sat on both sides of vendor evaluations for observability tooling throughout my career, I can tell you what is missing, because it is the same thing that was missing from the first generation of APM tools fifteen years ago: none of this is integrated into the operational fabric that actually runs the business. An SRE on call at two in the morning does not open a separate AI evaluation dashboard to check whether a quality regression is happening at the same time as a latency spike. They look at the tools they already trust, the ones wired into PagerDuty, the ones their runbooks reference, the ones that feed the incident channel. Today, AI quality signals live in a silo, disconnected from the alerting and escalation infrastructure that determines whether anyone ever sees them in time to act.
The second thing missing is ground truth at scale. Traditional observability has the luxury of objective signals: a request either returned a 500 or it did not, latency either exceeded the SLO or it did not. AI quality evaluation today largely relies on another model, an “LLM as judge,” to decide whether an output was good. That is a meaningful step forward from nothing, but it introduces a second layer of uncertainty that itself goes unmonitored. Nobody is tracking how reliable the judge is, under what conditions it disagrees with human reviewers, or how its assessments drift over time.
The third gap, and the one I think is most consequential for any organization running AI across many business units, client environments, or product lines, is that almost nothing on the market handles multi-tenant behavioral baselines well. What “normal” looks like for an AI system supporting a healthcare intake workflow is entirely different from what “normal” looks like for a retail customer service bot, and both are different again from what “normal” looks like for a coding agent with repository access. Most platforms are built around a single global baseline. The moment an organization is running AI across a dozen distinct contexts, each with its own definition of acceptable behavior, the existing tooling stops scaling.
What the Next Generation of Tooling Has to Do
Closing this gap is not about building a better dashboard. It is about doing for AI behavior what SRE practice already did for infrastructure and application health: defining what “correct” means in measurable terms, instrumenting continuously for deviation from that definition, and routing the result into the same operational pipeline that already carries infrastructure and application telemetry. Below are the five metrics I believe belong on every AI system’s scorecard, the AI-native equivalents of the golden signals, SLIs, and error budgets that traditional observability never needed because traditional software never needed them.
Action Audit Coverage: what percentage of an agent’s actions are logged, attributed, and checked against active constraints. Every tool call, every file write, every database operation, every external API request an AI agent makes needs to be logged, correlated to the instruction that triggered it, and evaluated against active constraints like code freezes or scope boundaries, in real time, not in a postmortem. The Replit incident should not have required a public apology and an emergency weekend patch cycle to produce dev and production separation and approval gates. That should be the starting configuration, and the percentage of agent actions covered by this audit trail is a number every platform team should be able to state.
Output Fidelity Rate: how often the system’s response matches the ground truth of what was asked. Where ground truth exists, as it did for McDonald’s drive-thru orders, where the customer’s stated request can be directly compared to the system’s output, that comparison needs to run continuously, at fleet scale, sliced by every dimension that matters operationally: location, channel, time, customer segment. The result of that comparison needs to land in the same incident management system that already pages your on-call engineer for a latency spike, because a quality regression is an incident, not a footnote.
Intent Drift Rate: how often a conversation or session moves outside the system’s intended scope or authority. The Chevrolet incident was not a failure of the model’s intelligence. It was the absence of any system watching for the specific pattern of a user attempting to redefine the bot’s behavior mid-conversation. Attempts to override instructions, requests for the system to make binding commitments, language that falls outside the system’s intended domain: this is detectable, and it needs to be tracked as a rate, with thresholds and alerts, the same way error rates are tracked today.
Grounding Accuracy: what percentage of factual claims in a response are verifiable against an authoritative source document. Air Canada’s failure was, at its core, a failure to check a generated answer against the document it was supposedly derived from. For any AI system answering policy, legal, financial, or compliance questions, that check, does this output actually match the source material, cannot be a sampled, after-the-fact audit. It needs to run on every response, or at minimum on every response touching a defined set of high-stakes topics, before that response reaches a customer.
Baseline Fit: how well current behavior matches the expected “normal” for this specific tenant, client, or use case. Any tool that hopes to serve organizations running AI across multiple business contexts needs to treat “what does normal look like here” as a first class, per-context configuration, not a global default that someone has to fight to override. This is the kind of architectural decision that is nearly impossible to retrofit later, which is exactly why it needs to be a foundational design principle for whoever builds this next.
The Opportunity Sitting in Plain Sight
I have built observability practices from the ground up before, in environments where the cost of an undetected failure was measured in regulatory exposure and customer trust, not just downtime minutes. I have sat in the room after the postmortem where the question on everyone’s mind is some version of “how did we not catch this,” and the honest answer is almost always the same: the signal existed, and nobody had built the pipeline to surface it before it mattered.
That is exactly where the industry is with AI right now.
The four examples above are not edge cases. They are previews. As more organizations hand AI systems broader authority, write access, customer-facing autonomy, decision-making in regulated workflows, the frequency and severity of these failures will increase, and the organizations that get caught flat-footed will be the ones that assumed their existing observability stack, built for deterministic software, was sufficient for systems that are fundamentally probabilistic.
This opportunity is not subtle, and it is not crowded.
It sits at the intersection of SRE discipline, quality engineering rigor, and a genuine understanding of how AI systems actually fail in production, not in benchmarks. Whoever builds the platform that brings action-level audit trails, continuous ground-truth evaluation, guardrail drift monitoring, and grounding verification into the same operational fabric that already runs incident management for everything else, with the multi-tenant flexibility real enterprises need, is not building a nice-to-have add-on to existing observability.
They are building the next category, the same way APM became its own category once “the server is up” stopped being a sufficient answer to “is the application working.”
I would rather help design that capability now, on the front end, than sit in another room afterward explaining what the data had been showing all along.
Or, even worse, high-five myself as I skip down the hallway, proclaiming that everything’s fine, because I am a print operator in charge of interpreting complex metrics.
The world can do better.
And it used to.
Nabeil Sarhan, MBA, is a dynamic technology delivery manager with over 15 years of experience in tech, cybersecurity, and computing scalability. He excels in leading diverse teams and delivering enterprise-class systems across industries such as healthcare, finance, and retail. Nabeil’s passion for solution design, systems architecture, and performance optimization makes him a sought-after consultant. He holds degrees from Harvard, MIT, and Bryant University. Connect with Nabeil on LinkedIn