Wednesday, August 10, 2016

puppet Trouble shooting



Puppet  Nagios configuration files location
/etc/nagios
        nrpe.cfg , nrpe_local.cfg
/usr/lib/nagios/plugins
/etc/puppetlabs/mcollective/
 /etc/puppetlabs/activemq/


checking puppet load
/usr/lib64/nagios/plugins/check_load -w 18,12,6 -c 30,20,10

To check Puppet_DAS or https    "command[check_das"
/usr/lib/nagios/plugins/check_http -H localhost -u "/diagnostics" -S -p 8192 -j HEAD -t 40




fixing Puppet_ActiveMQ_JVM_Memory  related  errors 

These type of errors  generally means the activemq memory heap has ‘jumped it’s boundaries’ and crashed (or crippled) the mcollective app - and it needs to be restarted , but  better to  recycle all with dopuppet restart
1. check Nagios to see if there is a history - note any in the ticket
2. do a status check (if you like…) and then restart all things puppet with:
# /usr/local/doozer/dopuppet status
# /usr/local/doozer/dopuppet restart

 the ‘dopuppet’ command will status/start/stop/restart whatever puppet processes that node needs like mcollective stuff.
dopuppet is a great command to ensure you are looking at ‘all things puppet' ; it does status without the need to become root

other trouble shooting paths :-
   cd /var/run/puppetlabs/activemq/
         ls
       mv activemq.pid /var/tmp/
       /etc/init.d/pe-activemq start
       /etc/init.d/pe-activemq status

 service mcollective restart  
 service pe-mcollective restart

  cd   /etc/puppetlabs/mcollective/
  server-patch.sh

  ls   -l    /etc/puppetlabs/activemq/
  cd /opt/puppetlabs/
          cd server/
                 cd bin    
  cd /opt/puppetlabs/server/bin
  ./keytool -list  -keystore /etc/puppetlabs/activemq/broker.ks

 /cd /etc/nagios
            ls -l
 cd /usr/lib/nagios/plugins
       ./check_puppet_jvm_fd -w 10 -c 50
       cat check_puppet_jvm_mem
       ./check_puppet_jvm_mem -w 10 -c 90
  ps -ef |grep -i java
       cd /opt/puppetlabs/server/apps/java/lib/jvm/java/jre
       cd lib
       ls -al
   cat /etc/nagios/puppet.jmxnagios
-------------------------------------------------------------------------------------
Swap space usage is high exceeding configured thresholds for 15 minutes. 
verify swap  is higly used by puppet  java process , ( ps aux --sort -pcpu ) ; if so
/usr/local/doozier/dopuppet restart
once restarted # swapoff -a; swapon -a