How to Tell a Last-Mile AI System Is Failing

Users who cannot go elsewhere do not complain, so the usual failure signals never arrive. Which proxies to instrument before launch, why every one of them is gameable, and the single control that actually detects confident wrong answers.

Published 2026-09-04 · By Shahzad Asghar

A commercial product tells you when it is failing. Users leave, complaints arrive, revenue moves, and someone notices within a quarter. Those signals are so reliable that most monitoring practice quietly assumes them.

At the last mile, none of them arrive.

The people a humanitarian or public service AI system serves usually cannot go elsewhere. They frequently have no channel to complain through, no expectation that complaining would help, and good reasons not to draw attention to themselves. The absence of complaints is not evidence that the system works. It is the expected condition whether the system works or not.

That leaves a system that can degrade for months while every dashboard stays green. This is about detecting that, and about what to build before launch so detection is possible at all.

The Last-Mile AI Framework covers designing for these conditions and why AI pilots fail covers the transition to production. This is the part after that: the system is live, and you need to know whether it is doing what you think.

The two silent failures

People stop using it, and you read that as low demand. Usage falls. The obvious interpretation is that the need was overstated. The other interpretation — that the system failed people badly enough that word spread — produces exactly the same graph. Nothing in the usage data distinguishes them, and the second is at least as likely.

People keep using it and receive wrong answers. Worse, because the numbers look healthy. Someone told they are ineligible when they are eligible has no way to know the system was wrong, and often no route to challenge it. Volume stays up. Satisfaction is unmeasured. The error rate is invisible because nobody with the standing to detect it is looking.

Both failures are consistent with a green dashboard. That is the problem to solve.

Signals that do exist

None of these are as good as a complaint. Together they are usually enough.

Where people abandon. Not whether they finish, but the step at which they stop. A cluster of abandonment at one point is the most legible failure signal available, and it points at a specific screen, prompt, or question rather than at the system in general.

The fallback rate. How often does the system hand over to a person? This needs care in both directions. A rising rate means the system is failing more often. A falling rate is ambiguous: it may be improving, or the handover path may have broken, or staff may have stopped answering. Monitor the fallback path itself, separately, as a system in its own right.

Repeat contacts from the same person. Somebody asking the same question three times in a week did not get an answer the first two times, whatever the system recorded. This is the closest available proxy for dissatisfaction and it is usually already in the logs.

Who stopped appearing. Break usage down by language, channel, location, and time of day, then watch for a group that quietly disappears. Aggregate volume can hold steady while one language cohort drops to zero. That cohort is where the system broke, and the total will never show it.

Time compared to the manual baseline. If the process previously took four days and the system now takes six hours, that is worth knowing — and it is worth re-measuring, because these figures decay. The comparison is only meaningful if someone recorded the baseline before launch, which is the step most often skipped.

The trap in all of them

Every metric above can be improved by the system doing less.

Abandonment falls if you shorten the flow by asking fewer questions and giving worse answers. The fallback rate falls if handover quietly stops working. Repeat contacts fall if people give up. Response time improves if the model stops checking anything.

This is why no metric can be a target on its own, and why the next section is not optional.

The control that actually works

Sample the outputs and have a person read them.

Take a fixed number each week — twenty is enough to start — chosen at random rather than from the flagged pile, and have somebody who understands the domain read the exchange and judge whether the answer was right. Record the judgement. Track it over time.

This is unglamorous, it does not scale elegantly, and it is the only method that detects the failure mode where confident wrong answers reach people who cannot contest them. Every automated signal is a proxy. This is the measurement.

Two conditions make it work. The reviewer must be independent of whoever built the system, because the author is the worst judge of their own output. And the sample must be random, because reviewing only escalated cases measures the escalation path rather than the system.

That is the same logic as information technology audit in the age of AI: once you cannot inspect the mechanism, you test the output on a real population rather than reasoning about the design.

What to instrument before launch

Most of this cannot be added retrospectively, because the data was never captured.

Log the step, not just the outcome — an interaction record that only stores the final answer cannot tell you where people abandoned.

Record the baseline before the system replaces anything. How long did it take, how often was it wrong, how many people did it reach. Once the old process is gone, that number is unrecoverable and every later claim of improvement becomes unfalsifiable.

Segment from day one. Retrofitting language, channel and location breakdowns onto logs that never carried them is not possible.

Instrument the fallback path separately, so a broken handover looks different from a system that needs fewer handovers.

Name who reviews the samples and who can stop the system, before either is needed. A stopping condition with no named owner is a sentence in a document.

Go and ask

Every signal above is inference. The only ground truth is the people using it.

Periodically, someone should sit with actual users — not staff, not a proxy group — and watch them use the system. It is expensive, it does not fit a monitoring dashboard, and it surfaces things no log contains: that the phrasing is confusing in the second language, that the flow assumes a literacy level many users do not have, that people found a workaround months ago and nobody upstream knew.

In consumer software this is routine user research. At the last mile it is skipped far more often, usually because the users are hard to reach — which is the same reason the other signals are missing, and therefore exactly why it matters more here rather than less.

The short version

Assume the absence of complaints means nothing. Instrument abandonment, fallback, repeat contacts and cohort drop-off before launch, because they cannot be added later. Treat every one of them as gameable and pair them with weekly human review of a random sample. Keep a pre-launch baseline so improvement claims stay falsifiable. And go and watch someone use it, periodically, for as long as the system is live.

A system serving people who cannot complain needs more monitoring than a commercial product, not less. It usually gets far less, and that asymmetry is where the harm accumulates.

For the design constraints underneath this, see last-mile AI and AI in humanitarian operations. For delivered examples, AI for refugee operations.

Frequently asked questions

How do you monitor an AI system when users cannot give feedback?

Through proxies, paired with human review. Log where people abandon, how often the system falls back to a person, how often the same individual returns with the same question, and whether any group has quietly stopped using it. Then read a random sample of outputs every week, because every proxy can be improved by the system doing less.

Why is the absence of complaints not a good sign?

Because complaint channels assume the user has an alternative, the standing to object, and a reason to believe it will help. At the last mile, often none of those hold. Silence is the expected condition whether the system works or not, so it carries almost no information.

What should be logged before an AI system launches?

The step-level path rather than only the final outcome, cohort attributes such as language and channel, the fallback path as a separate signal, and the pre-launch baseline for time, error rate and reach. None of these can be reconstructed afterwards.

How many outputs should be reviewed by a human?

Enough to notice a change, sampled randomly rather than from flagged cases, and read by somebody independent of the team that built the system. Twenty a week is a workable starting point for most services. Consistency over time matters more than volume.

What is the most common monitoring mistake in humanitarian AI?

Treating steady usage as success. Volume can hold while one language group drops to zero, or while people receive confident wrong answers they have no way to contest. Segmented metrics and sampled human review are what separate those cases from a working system.

Can you add monitoring to a system that is already live?

Partly. Fallback rates and repeat contacts can often be derived from existing logs. Step-level abandonment, cohort segmentation and the pre-launch baseline usually cannot, which is why they belong in the build rather than the improvement plan.

← All articles