An AI agent that acts is an alarm that fires

Everyone is finding out this year that an agent which can act is not the same as an agent anyone lets act. The surveys all come out the same way. Lots of pilots, far fewer things actually running, and quality is the reason people give for the gap.

I have built the version of this problem that existed before anyone said the word agent. Alert systems for vehicle fleets. It fails in a way I recognise.

The alarm problem

Build something that detects conditions and it will detect them. A week in, the operator has three hundred notifications a day and four of them matter. By week three she is clearing the list without reading it. The system is now worse than having nothing, because everyone upstairs believes the fleet is being watched.

Detection was never the difficult part. The difficult part is that an alert nobody acts on is noise, and noise teaches people to stop looking at whatever channel it comes through. Once that has happened you cannot undo it by improving the detection.

An agent is the same thing with the stakes raised. It does not tell you about a condition, it does something about it. The equivalent of a dismissed alert is a colleague having to undo the action.

The handoff is the product

On the telematics platforms we treated alarms, geofences and ticketing as one chain rather than three features. A geofence says what counts as the condition. An alarm says it happened. A ticket says who owns it now.

That third link is the one that gets cut. It is the least interesting thing to specify and it is the only reason the first two are worth building.

Most agents I have looked at ship with two links. It decides, it acts, and then the trail goes cold. No record of why, nobody named as responsible, no obvious route for a person who thinks it got this one wrong. Teams put observability in and call that covered, but knowing what the agent did tells you nothing about who answers for it.

Autonomy is a dial

The question is never whether the agent can do the thing. It is what happens the tenth time it does the thing badly.

Which makes autonomy a setting, and one that should be visible: suggest and wait, act and tell someone, act quietly and log it. Different actions inside the same product deserve different settings. The setting should be allowed to move once you have evidence, which means somebody has to be collecting the evidence, which is the step that gets skipped.

Put it in fleet terms. Drafting a ticket is a suggestion. Closing one is an action. Sending a technician across the city is an action that costs diesel and somebody's afternoon. Nobody would give those three the same permissions, and yet software teams hand their agent one level for everything and then act surprised.

Trust gets withdrawn quietly

I wrote a teardown a while back on why developers adopt AI coding assistants and then stop trusting them for the parts that matter, while carrying on using them for the parts that do not. Nobody raises a ticket to say they have lost confidence. The usage numbers stay healthy while the product hollows out underneath them.

Agents will go the same way and the same dashboard will hide it. Seats and sessions look fine while every user quietly shrinks the list of things they will let it touch.

What I would measure

How many of the actions it took got reversed by a person. How many of its suggestions were accepted without an edit. How wide the set of permissions a given user leaves it after three months, compared with the first week.

And the one that carries over directly from fleet work: what share of alerts ended as a closed ticket rather than a dismissal. That number tells you whether you built a colleague or another notification.