Behind MUVES · Article · Part 2 of 3
The monitor that watches our AI agents has no AI in it
When you ask a model whether another model is behaving, you get another opinion. We wanted facts, so the watchdog over MUVES OS is a set of plain rules that read the database and cannot be talked round.
In brief
- Our monitor, Overseer, is a separate process that reads the agents’ database read-only and checks plain rules.
- It opens an incident after two failing runs in a row and closes it after five clean ones, so it neither cries wolf nor forgets.
- It stores row numbers and counts, never the private text it is checking.
Once you have a few dozen agents doing real work, you need to know when one of them goes wrong. The obvious move is to ask another model to keep an eye on them. We decided against it. A model judging a model gives you a second opinion, and opinions can be argued with, including by the very text they are reading.
So the watchdog over MUVES OS, which we call Overseer, has no model in it at all. It is a separate program that opens the system's database read-only and checks a list of plain rules.
What it looks for
The rules are boring on purpose. Each one is a question with a yes or no answer that a database query can settle:
- Is a job stuck longer than its runner is allowed to take?
- Did an answer finish but never reach the person who asked?
- Has the same request been queued twice?
- Does something in a log look like a password or a key?
- Was a note filed in the wrong person's space?
None of these need judgment. They need someone who checks every time and never gets tired, which is exactly what a few lines of SQL are good at.
A rule that reads the database cannot be persuaded by what it finds there.
Not crying wolf
A monitor that alarms on every blip gets muted within a week. We borrowed thresholds from the way operations teams run large websites. Overseer raises an incident only after a rule fails on two runs in a row, and it closes the incident only after five clean runs. One odd reading does not wake anybody up, and a problem that flickers on and off does not keep closing itself.
When a service is restarting in a loop, every job that depends on it fails too. Rather than raising ten alarms for one cause, Overseer holds back the dependent alarms and reports the restart loop.
Watching without reading
A monitor sees everything, so it has to be careful about what it keeps. Overseer stores only row numbers and counts: "a job is stuck", "three answers undelivered". It never copies the private text of a message, a note or a client file into its own records. Its access to the database is read-only, so the watchdog cannot change what it watches.
Why plain rules age well
There is a quieter benefit. Models change every few months, and a model-based judge can change its mind along with them. A rule like "a finished answer must reach the person who asked" means the same thing next year as it does today. When we add a rule, we add it because something real went wrong once, so the list doubles as a record of the problems we have already solved.
Testing the tester
Every rule was tested against a scratch copy of the database layout before it was allowed near the real one: create a stuck job, check the rule fires; clear it, check the incident closes after five clean runs. A monitor nobody has tested is a second thing to worry about.
Exhibit 1Two ways to watch an agent
| Another model as judge | Plain rules on the database | |
|---|---|---|
| Answer | An opinion, worded differently each time | Yes or no, the same every time |
| Can be misled by the text | Yes | No, it reads rows and timestamps |
| Good at | Tone, quality, "is this a good reply?" | Stuck, missing, doubled, misfiled |
| Cost per check | A model call | A database query |
Where a model still helps
This is not an argument against using models to review work. In MUVES OS, department heads review drafts before they go anywhere, and that review is a model reading for quality and tone. That is a judgment call, and a model is a reasonable first reader. Overseer covers the other kind of failure: the job that silently stopped, the answer that never arrived, the request that ran twice. Those are facts, and facts deserve a check that cannot be argued with.
If you run agents built by someone else and are not sure what they get up to, the same approach works from the outside. We describe it on our agent monitoring and debugging page, and the full build is in the agent monitoring case study.
