Puppet Nagios configuration files location
/etc/nagios
nrpe.cfg , nrpe_local.cfg
/usr/lib/nagios/plugins
/etc/puppetlabs/mcollective/
/etc/puppetlabs/activemq/
checking puppet load
/usr/lib64/nagios/plugins/check_load -w 18,12,6 -c 30,20,10
To check Puppet_DAS or https "command[check_das"
/usr/lib/nagios/plugins/check_http -H localhost -u "/diagnostics" -S -p 8192 -j HEAD -t 40
fixing Puppet_ActiveMQ_JVM_Memory related errors
These type of errors generally means the activemq memory heap has ‘jumped it’s boundaries’ and crashed (or crippled) the mcollective app - and it needs to be restarted , but better to recycle all with dopuppet restart
1. check Nagios to see if there is a history - note any in the ticket
2. do a status check (if you like…) and then restart all things puppet with:
# /usr/local/doozer/dopuppet status
# /usr/local/doozer/dopuppet restart
the ‘dopuppet’ command will status/start/stop/restart whatever puppet processes that node needs like mcollective stuff.
dopuppet is a great command to ensure you are looking at ‘all things puppet' ; it does status without the need to become root
other trouble shooting paths :-
cd /var/run/puppetlabs/activemq/
ls
mv activemq.pid /var/tmp/
/etc/init.d/pe-activemq start
/etc/init.d/pe-activemq status
service mcollective restart
service pe-mcollective restart
cd /etc/puppetlabs/mcollective/
server-patch.sh
ls -l /etc/puppetlabs/activemq/
cd /opt/puppetlabs/
cd server/
cd bin
cd /opt/puppetlabs/server/bin
./keytool -list -keystore /etc/puppetlabs/activemq/broker.ks
/cd /etc/nagios
ls -l
cd /usr/lib/nagios/plugins
./check_puppet_jvm_fd -w 10 -c 50
cat check_puppet_jvm_mem
./check_puppet_jvm_mem -w 10 -c 90
ps -ef |grep -i java
cd /opt/puppetlabs/server/apps/java/lib/jvm/java/jre
cd lib
ls -al
cat /etc/nagios/puppet.jmxnagios
-------------------------------------------------------------------------------------
Swap space usage is high exceeding configured thresholds for 15 minutes.
verify swap is higly used by puppet java process , ( ps aux --sort -pcpu ) ; if so
/usr/local/doozier/dopuppet restart
once restarted # swapoff -a; swapon -a
