In a previous article, I showed to how to install Supervisor and Superlance on CentOS. While Superlance is great for telling you when one of your child threads goes down, what about when Supervisor itself goes down? That is where Monit1 comes in.
In this tutorial, we will install Monit from the EPEL2 repository, and then configure it to send us an email alert if Supervisor goes offline. It will also automatically restart Supervisor for you if it crashes.
I got 5.1 from EPEL (5.8 is the latest):
$ yum install monit
Edit /etc/monit.conf, add:
set mailserver localhost set alert firstname.lastname@example.org
You should set mailserver to point to your SMTP, and alert to point to your email address that will receive Monit alerts.
Create the file /etc/monit.d/supervisor with the following content:
check process cron with pidfile /var/run/supervisord.pid group supervisor start program = "/etc/init.d/supervisord start" stop program = "/etc/init.d/supervisord stop" if 5 restarts within 5 cycles then timeout
The configuration will monitor the Supervisord PID file (/var/run/supervisord.pid), alerting when that file disappears due to a crash or changes due to a restart. There is a lot more your can do with Supervisor service monitoring, but this is the most basic config to get you up-and-running monitoring Supervisor.
$ service monit start
Now just for fun, try this and see what happens:
$ service supervisord stop