What is a useful way of doing "all ok" notifications?

Question

Who watches the watchers

A monitoring system (e.g. nagios) usually has a good way of producing warnings and change-notifications. But the worst thing that can happen is to miss an event because the monitoring was down.

What do you mean "an issue", we haven't got any report-- oh, wait...

So, what is a meaningful way of saying "hi, nothing to report, and monitoring is up"?

In some cases I chose a regular "hello world" notification email, but that tends to quickly overload users' attention, making them less sensitive to the important notifications from the same source (and it keeps flooding the inbox).

Have been considering a periodical report - in a preset interval (say 7 days) send an "all ok" update, that we are still good. That is better, but leaves a worst-case 7 day window of uncertainty.

A more radical version is the "Fire-alarm drill" producing false alarms to test the actual preparedness and response time. Though a great exercise, users hate this and it doesn't solve the issue per se.

Or you could have a monitoring-system-monitoring-system. But that adds overhead and complexity and I'm looking for a leaner way.

The solution should be passive. I don't want to ask users to check on some status page (or even multiple status pages) just to see that there is nothing to report.

Or does it require an another way of organizing the process?

On the other hand (actual event notification), I really like the notification behavior of smartd which has a diminishing mode, where messages are sent at doubling intervals until solved.

The original message about this issue was sent at Sun Apr 27
Another message will be sent in 64 days if the problem persists.

Answer 1

I don't think you're actually solving a problem by sending "Everything's ok" E-Mails. Even if they're sent daily, you can't be sure that they will be missed if they suddenly stop. But the main concern I see with those passive E-Mails (passive as they don't force any action) is that users numb themselves to E-Mails from the notification system. They're more likely to ignore an important notification from the system when they're used to receiving a daily e-mail.

Periodical reports, weekly or monthly can be sent, but IMHO not as part of the monitoring system itself but for information purposes. E.g. as statistical overview of up and downtimes.

Please don't do the fire alarm drill. People will get tired of the false alarms and you'll be at least as famous as this guy.

Or you could have a monitoring-system-monitoring-system

Although this seems as a little too much overhead at first, this might be the best solution. You'll get notifications only if something's really wrong. Therefore e-mails from the notification system will get more attention.

Before you ask: "But who monitors the monitoring-system-monitoring system?": they can monitor each other (btw: we used to do this in our company with mmonit). You're of course right, this adds technical overhead. But IMO technical overhead is better than relying on manual tasks, e.g. checking the system everyday manually or to have people read daily e-mails.