Who watches the watchers
A monitoring system (e.g. nagios) usually has a good way of producing warnings and change-notifications. But the worst thing that can happen is to miss an event because the monitoring was down.
What do you mean "an issue", we haven't got any report-- oh, wait...
So, what is a meaningful way of saying "hi, nothing to report, and monitoring is up"?
In some cases I chose a regular "hello world" notification email, but that tends to quickly overload users' attention, making them less sensitive to the important notifications from the same source (and it keeps flooding the inbox).
Have been considering a periodical report - in a preset interval (say 7 days) send an "all ok" update, that we are still good. That is better, but leaves a worst-case 7 day window of uncertainty.
A more radical version is the "Fire-alarm drill" producing false alarms to test the actual preparedness and response time. Though a great exercise, users hate this and it doesn't solve the issue per se.
Or you could have a monitoring-system-monitoring-system. But that adds overhead and complexity and I'm looking for a leaner way.
The solution should be passive. I don't want to ask users to check on some status page (or even multiple status pages) just to see that there is nothing to report.
Or does it require an another way of organizing the process?
On the other hand (actual event notification), I really like the notification behavior of smartd which has a diminishing mode, where messages are sent at doubling intervals until solved.
The original message about this issue was sent at Sun Apr 27
Another message will be sent in 64 days if the problem persists.