Zabbix

From Newroco tech docs
Jump to navigationJump to search

unformatted & untested notes

Installing Zabbix 3.x on Ubuntu LTS with PostgreSQL

Setting up PostgreSQL

As root do

  1. su postgres
  2. psql
  3.  
  4.  

Enter

  1. psql -U <the username you just created>
  2.  
  3. => create database zabbix;
  4. CREATE DATABASE
  5. => \q
  6.  
  7. cd /usr/share/doc/zabbix-server-pgsql
  8. gunzip create.sql.gz
  9. psql -U <username> zabbix < create.sql
  10.  
  11.  

username/password for DB needs adding in both UI conf and in /etc/zabbix/zabbix-server.conf

installing Zabbix server on Ubuntu 14.04

  1. # wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb
  2. # dpkg -i zabbix-release_3.0-1+trusty_all.deb
  3. # apt-get update
  4. # apt-get install zabbix-agent zabbix-server-pgsql
  5.  
  6.  
  7. wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update && apt-get install zabbix-agent zabbix-server-pgsql
  8.  

Then edit conf file to suit environment

Using auto-discovery

Zabbix auto-discovery helps find both devices you want to monitor and rogue devices. Discovery can be be through a number of techniques and used to add the discovered devices to the monitoring.

Basic using ICMP ping

The simplest discovery method is to use ICMP ping. Go to the Configuration menu and select Discovery, then click New Discovery Rule. Give it a suitably descriptive name e.g. "branch office 1 workstations", add an IP range to search through, then click New next to Checks. From the Check type drop down list choose ICMP ping, then click Add in that section, and then the Add button at the bottom of the page.

Note if adding multiple discovery rules that they run consecutively, so if any are for a large subnet then it will take a long while to run through every IP address before the next rule is tried.

enabling auto-registration of devices

Auto registration combines well with discovery to streamline registration of new devices with your Zabbix server. To enable use the web UI and go to Configuration > Actions and use the Event Source dropdown box to select Auto registration. Then click Create Action, name it "Auto register action" then in Operations click New, select Operation type as Add to host group and choose the host group as Discovered hosts.

NB can make different actions for auto registration using Conditions - particularly useful for pushing all end user devices into their own group by metadata in their agent conf file.

installing agent on Ubuntu 14.04

  1. # wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb
  2. # dpkg -i zabbix-release_3.0-1+trusty_all.deb
  3. # apt-get update
  4. # apt-get install zabbix-agent
  5.  
  6. wget http://repo.zabbix.com/zabbix/3.0/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.0-1+trusty_all.deb && dpkg -i zabbix-release_3.0-1+trusty_all.deb && apt-get update
  7. apt-get install zabbix-agent
  8.  

Then edit conf file to suit environment

  1. vi /etc/zabbix/zabbix_agentd.conf

and set to zabbix server IP

  1. Server=ip.add.re.ss
  2. ServerActive=ip.add.re.ss

And comment out

  1. Hostname=

and restart zabbix-agent

  1. # service zabbix-agent restart

TO BE CONSIDERED:

  • making repo per site with zabbix-agent bundled with working conf file (added benefit is repo could have builds for many OSes)
  • or adding DNS records based on source IP, and having conf file same for all e.g. zabbixproxy.newro.co, zabbixserver.newro.co, then DNS records point to correct internal IP, then use single repo for all customer sites

installing agent through salt on Ubuntu 16.04 target server

On salt master server use the following commands(where '*' means all minions of the salt master server):

  1.  
  2. sudo salt '*' cmd.run 'wget http://repo.zabbix.com/zabbix/3.4/ubuntu/pool/main/z/zabbix-release/zabbix-release_3.4-1+xenial_all.deb'
  3. sudo salt '*' cmd.run 'dpkg -i zabbix-release_3.4-1+xenial_all.deb'
  4.  

Where 3.4 is the version of zabbix and xenial is the version of ubuntu on the minion server. Feel free to change them to fit your needs.

  1.  
  2. sudo salt '*' cmd.run 'apt-get update'
  3. sudo salt '*' cmd.run 'apt-get -y install zabbix-agent'
  4.  

Next we have to edit /etc/zabbix/zabbix_agentd.conf, you can use this command to see the uncommented lines of the file:

  1.  
  2. sudo salt '*' cmd.run 'grep "^[^#;]" /etc/zabbix/zabbix_agentd.conf'
  3.  

Now we have to set Server=<zabix-server ip>, ServerActive=<zabbix-server ip> and comment Hostname= line:

  1.  
  2. sudo salt '*' cmd.run 'sed -i 's/Server=127.0.0.1/Server=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf' #this command replaces "Server=127.0.0.1" with "Server=<zabbix-server ip>"
  3. sudo salt '*' cmd.run 'sed -i 's/ServerActive=127.0.0.1/ServerActive=<zabbix-server ip>/g' /etc/zabbix/zabbix_agentd.conf'
  4. sudo salt '*' cmd.run 'sed -i 's/Hostname=/#Hostname=/g' /etc/zabbix/zabbix_agentd.conf'
  5.  

And now restart zabbix agent:

  1.  
  2. sudo salt '*' cmd.run 'service zabbix-agent restart'
  3. sudo salt '*' cmd.run 'systemctl enable zabbix-agent.service'
  4.  

Info When adding the agent on zabbix Gui the Hostname must correspond otherwise zabbix server will not receive data from the agents

Install zabbix proxy

Info Install zabbix-proxy on different server than zabbix-server

Update the repos for the latest packeges

  1.  
  2. # wget http://repo.zabbix.com/zabbix/<version>/ubuntu/pool/main/z/zabbix-release/zabbix-release_<version>+<ubuntu version>_all.deb
  3. # sudo dpkg -i zabbix-release_<version>+<ubuntu-version>_all.deb
  4. # sudo apt-get update
  5.  

Info

Use a different database from zabbix sever


Install database

  1. apt-get install postgresql postgresql-contrib

Create the base DB & allow web app access

Set up the database and user,

  1.  
  2. # su postgres
  3. # psql -d template1
  4.  
  5. CREATE USER user_db_zabbix_proxy CREATEDB;
  6. CREATE DATABASE zabbix_proxy OWNER user_db_zabbix_proxy;
  7. \q
  8.  
  9. # exit
  10.  

Notice: If the server does not work without a password or it requires you can add it:

  1.  
  2. su postgres
  3. psql -d template1
  4. ALTER USER user_db_zabbix_proxy WITH PASSWORD 'type_your_password';
  5. \q
  6.  
  7. #exit
  8.  

Edit PostgreSQL settings to allow local connections for yourdbuser to the nextcloud database (on all DB servers):

  1.  
  2. # vi /etc/postgresql/<version number>/main/pg_hba.conf
  3.  
  4. # "local" is for Unix domain socket connections only
  5. local all user_db_zabbix_proxy trust
  6. local all all peer
  7.  


Then restart postgresql

  1.  
  2. service postgresql restart
  3.  


Install zabbix proxy with desired database

  1.  
  2. apt-get install zabbix-proxy-pgsql
  3.  
  4. apt-get upgrade
  5.  


Creating proxy database

  1.  
  2. zcat /usr/share/doc/zabbix-proxy-pgsql/schema.sql.gz | psql -U user_db_zabbix_proxy zabbix_proxy
  3.  

Start service zabbix-proxy

  1.  
  2. systemctl start zabbix-proxy
  3. systemctl enable zabbix-proxy
  4.  

Edit the configuration file

  1.  
  2. # vi /etc/zabbix/zabbix_proxy.conf
  3. ProxyMode=0
  4. Server=94.130.33.45
  5. Hostname=compendium_zbxProxy ###same name will be used when added on the web interface
  6. DBHost=localhost
  7. DBName=zabbix_proxy
  8. DBUser=user_db_zabbix_proxy
  9. DBPassword=password_db
  10.  

Restart zabbix-proxy

  1. systemctl restart zabbix-proxy


Info If it is needed to be installed over WAN create port forwarding rule on both server on for ports 10050 and 10051 Notice: The installation must be on the host.

  1. iptables -t nat -I PREROUTING -p tcp -d <host-ip> --dport 10050 -j DNAT --to-destination <zabbix-proxy-ip>:10050
  2. iptables -t nat -I PREROUTING -p tcp -d <host-ip> --dport 10051 -j DNAT --to-destination <zabbix-proxy-ip>:10051
  3. sudo iptables -I FORWARD -m state -d <zabbix-proxy-network>/24 --state NEW,RELATED,ESTABLISHED -j ACCEPT

adding kvm specific monitoring

requires python-libvirt (probably already installed)

https://github.com/bushvin/zabbix-kvm-res


UserParameter=custom.kvm.pool.discover,/usr/local/bin/zabbix-kvm-res.py --resource pool --action list UserParameter=custom.kvm.pool.size[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action $1 --pool $2 UserParameter=custom.kvm.pool.active[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action active --pool $1 UserParameter=custom.kvm.pool.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource pool --action UUID --pool $1 UserParameter=custom.kvm.net.discover,/usr/local/bin/zabbix-kvm-res.py --resource net --action list UserParameter=custom.kvm.net.active[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action active --net $1 UserParameter=custom.kvm.net.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource net --action UUID --net $1 UserParameter=custom.kvm.domain.discover,/usr/local/bin/zabbix-kvm-res.py --resource domain --action list UserParameter=custom.kvm.domain.active[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action active --domain $1 UserParameter=custom.kvm.domain.uuid[*],/usr/local/bin/zabbix-kvm-res.py --resource domain --action UUID --domain $1


zabbix agent

http://docswiki.newro.co/index.php/KVM_monitoring_xml

Monitoring ISCSI volume for read only

Create a file with the following script:

  1.  
  2. #!/bin/bash
  3.  
  4. if [ $# -eq 1 ]; then
  5. file=$1
  6. else
  7. echo "Wrong number of parameters!"
  8. exit 1;
  9. fi
  10.  
  11. touch $file &> /dev/null
  12. if [ $? -eq 0 ]; then
  13. rm $file &> /dev/null
  14. echo "1"
  15. else
  16. echo "0"
  17. fi
  18.  

Add a line like this in /etc/zabbix/zabbix_agentd.conf:

  1.  
  2. UserParameter=custom.checkro[*],/path/to/script/file $1
  3.  

And restart the agent

  1.  
  2. service zabbix-agent restart
  3.  

To call the script from zabbix GUI, create an Item and in the Key field put "custom.checkro[/path/to/iscsivolume/test-file]".

User Parameter for zabbix-agent

For all Linux servers

  1. UserParameter=ServiceStatus[*],service status | grep Active
  2. UserParameter=CoreNum,grep processor /proc/cpuinfo | wc -l
  3. UserParameter=system.sw.debianupdates,apt-get dist-upgrade -s |sed -n 's/^\([0-9]\+\) upgraded.*/\1/p'

Only for physical servers, not on VMs

  1. UserParameter=CPUtemp,cat /sys/class/thermal/thermal_zone0/temp

RabbitMQ

  1. UserParameter=AliveRabbit,ps -aux | grep [r]abbitMqDaemon.py | wc -l

Reset password for user on GUI

Zabbix reset password


Setting Host Zabbix Proxy and Group

To set up the proxy, first create the Group Host, if it doesn't exists. Afterwards create the Hosts, with zabbix-proxy and http proxy. After it's done go to Zabbix -> Administration -> Proxies and add up your proxy and proxy hosts which it will supervise. If it works it should be displayed available.


Check in /etc/zabbix/zabbix_proxy.conf the following:

Hostname -> must be the same in the file and on the monitoring zabbix website.

Server ip -> must be the server ip on which the zabbix website is hosted.

Update zabbix-agent with salt

This is a salt recipe for upgrading zabbix-agent on multiple servers:

  1. create-upgrade-zabbix-agent:
  2. file.managed:
  3. - name: /tmp/upgrade-zabbix-agent.sh
  4. - mode: 755
  5. - contents: |
  6. #!/bin/bash
  7. codename=`lsb_release -c | cut -f2`
  8. wget https://repo.zabbix.com/zabbix/4.2/ubuntu/pool/main/z/zabbix-release/zabbix-release_4.2-1+"$codename"_all.deb
  9. yes | dpkg -i zabbix-release_4.2-1+"$codename"_all.deb
  10. apt-get update
  11. apt-get -y -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install --only-upgrade zabbix-agent
  12. systemctl restart zabbix-agent
  13. rm zabbix-release_4.2-1+"$codename"_all.deb
  14.  
  15. run-upgrade-zabbix-agent:
  16. cmd.run:
  17. - name: /tmp/upgrade-zabbix-agent.sh
  18.  
  19. delete-upgrade-script:
  20. cmd.run:
  21. - name: rm /tmp/upgrade-zabbix-agent.sh

POSTGRESQL_DB_FREE_SPACE

POSTGRESQL_DB_FREE_SPACE[1]

CHECKUPS for backups

Steps to reproduce:

1. Create an item using vfs.file.time[<directory or file>, modify] it will monitorize the folder or file if it changed it's date or if a newer sub-file or sub-folder changed it's date. If it's a backup check for the backup date, if the time it's the creation date of the imported folder and not the current date, you should monitor the whole folder and not the "current" which is the symlink, it will check for new files or folders inside.

2. At unit use unixtime to convert from epoch to unix date and time.

3. For schedule checkups use Scheduling and set update interval to 0 so it will select the schedule checkup.

4. For trigger use the item created and use fuzzytime as function and set it up at fuzzytime requested time in seconds 86400 is 24h for checkup ( just complete the requested fields) and set the value equal to 0, it will check if the file changed in the last previous 24h if not it will send you a message with a PROBLEM state.

5. Do tests to be sure everything works ok. Done.


Dependency triggers

To set dependency triggers set the requested trigger to be dependent on what you would like and if that trigger occurs the dependent trigger won't be triggered. The serie of triggers need to have a distance of time between them so it won't have an conflict when it request the data from the first to the last trigger.

MYSQL monitoring

1.Import this template on your zabbix WEB UI

2. Create a new user in your MYSQL DB and grant the required privileges:

  1. CREATE USER 'zabbix'@'localhost' IDENTIFIED WITH mysql_native_password BY '<password>';
  2. GRANT USAGE,REPLICATION CLIENT,PROCESS,SHOW DATABASES,SHOW VIEW ON *.* TO 'zabbix'@'%';

3.Create a new file on /etc/zabbix/ called .my.cnf and paste:

  1. [mysql]
  2. user=zabbix
  3. password=<password>
  4. [mysqladmin]
  5. user=zabbix
  6. password=<password>

4. On /etc/zabbix/zabbix_agentd.d create a new file called userparameter_mysql.conf anp paste:

# Flexible parameter to determine database or table size. On the frontend side, use keys like mysql.size[zabbix,history,data].
# Key syntax is mysql.size[<database>,<table>,<type>]. # Database may be a database name or "all". Default is "all". # Table may be a table name or "all". Default is "all". # Returns value in bytes.
UserParameter=mysql.size[*],bash -c 'echo "select sum($(case "$3" in both|"") echo "data_length+index_length";; data|index) echo "$3_length";; free) echo "data_free";; esac)) from information_schema.tables$([[ "$1" = "all" || ! "$1" ]] || echo " where table_schema=\"$1\"")$([[ "$2" = "all" || ! "$2" ]] || echo "and table_name=\"$2\"");" | /usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -N'

# MySQL version and status
UserParameter=mysql.ping,mysqladmin --defaults-extra-file=/etc/zabbix/.my.cnf ping | grep -c alive
UserParameter=mysql.version,mysql --defaults-extra-file=/etc/zabbix/.my.cnf -V

# Flexible parameter to grab global variables. On the frontend side, use keys like mysql.status[Com_insert].
UserParameter=mysql.extended_status[*],/usr/bin/mysqladmin --defaults-extra-file=/etc/zabbix/.my.cnf extended-status variables |awk {'print $$2"| "$$4'} | grep "$1|" | awk {'print $$2'}

# Replication monitoring
UserParameter=mysql.slave_lagging,/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -Bse "show slave status\\G" | grep Seconds_Behind_Master | awk '{print $$2}' | sed -e 's/^NULL$/-1/; s/![0-9]+/-1/' | awk '{if($2 ~ /d/) {print 100} else {print $2}}'

UserParameter=mysql.slave_status[*],/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -e "show slave status\\G"|grep -w $1 | awk '{print $$2}'

UserParameter=mysql.slave_status2[*],/usr/bin/mysql --defaults-extra-file=/etc/zabbix/.my.cnf -e "show slave status\\G"|grep -i $1 | awk '{print $$2}' 1>> /dev/null 2>> /dev/null && echo yes || echo no

5. Restart zabbix and mysql services:

systemctl restart mysql.service
systemctl restart zabbix-agent.service