240px-Diamond warning sign (Vienna Convention style).svg.png Content of this wiki is DEPRECATED 240px-Diamond warning sign (Vienna Convention style).svg.png

Servery nms

Z old-wiki.siliconhill.cz
(Rozdíly mezi verzemi)
Přejít na: navigace, hledání
(Linux servers)
(Linux servers)
Řádka 81: Řádka 81:
 
Apache is not running on $server                         {$server:proc.num[httpd].last(0)}<1                  Average
 
Apache is not running on $server                         {$server:proc.num[httpd].last(0)}<1                  Average
 
Configured max number of opened files is too low on $server {$server:kernel.maxfiles.last(0)}<512          Information
 
Configured max number of opened files is too low on $server {$server:kernel.maxfiles.last(0)}<512          Information
Configured max number of processes is too low on $server {$server:kernel.maxproc.last(0)}<256 Information
+
Configured max number of processes is too low on $server {$server:kernel.maxproc.last(0)}<256                       Information
Email (SMTP) server is down on $server                  {$server:net.tcp.service[smtp].last(0)}=0 Average
+
Email (SMTP) server is down on $server                  {$server:net.tcp.service[smtp].last(0)}=0         Average
FTP server is down on $server                              {$server:net.tcp.service[ftp].last(0)}=0 Average
+
FTP server is down on $server                              {$server:net.tcp.service[ftp].last(0)}=0         Average
Host information was changed on $server                {$server:system.uname.diff(0)}>0 Information
+
Host information was changed on $server                {$server:system.uname.diff(0)}>0             Information
Hostname was changed on $server {$server:system.hostname.diff(0)}>0 Information
+
Hostname was changed on $server                                 {$server:system.hostname.diff(0)}>0 Information
IMAP server is down on $server {$server:net.tcp.service[imap].last(0)}=0 Average
+
IMAP server is down on $server                                 {$server:net.tcp.service[imap].last(0)}=0 Average
Inetd is not running on $server {$server:proc.num[inetd].last(0)}<1 Average
+
Inetd is not running on $server                                 {$server:proc.num[inetd].last(0)}<1 Average
Lack of free memory on server $server {$server:vm.memory.size[free].last(0)}<10000 Average
+
Lack of free memory on server $server                         {$server:vm.memory.size[free].last(0)}<10000 Average
Lack of free swap space on $server {$server:system.swap.size[,free].last(0)}<100000 High
+
Lack of free swap space on $server                         {$server:system.swap.size[,free].last(0)}<100000 High
Low free disk space on $server volume / {$server:vfs.fs.size[/,pfree].last(0)}<10 High
+
Low free disk space on $server volume /                         {$server:vfs.fs.size[/,pfree].last(0)}<10 High
Low free disk space on $server volume /home {$server:vfs.fs.size[/home,pfree].last(0)}<10 High
+
Low free disk space on $server volume /home                 {$server:vfs.fs.size[/home,pfree].last(0)}<10 High
Low free disk space on $server volume /opt {$server:vfs.fs.size[/opt,pfree].last(0)}<10 High
+
Low free disk space on $server volume /opt                 {$server:vfs.fs.size[/opt,pfree].last(0)}<10 High
Low free disk space on $server volume /tmp {$server:vfs.fs.size[/tmp,pfree].last(0)}<10 High
+
Low free disk space on $server volume /tmp                 {$server:vfs.fs.size[/tmp,pfree].last(0)}<10 High
Low free disk space on $server volume /usr {$server:vfs.fs.size[/usr,pfree].last(0)}<10 High
+
Low free disk space on $server volume /usr                 {$server:vfs.fs.size[/usr,pfree].last(0)}<10 High
Low free disk space on $server volume /var {$server:vfs.fs.size[/var,pfree].last(0)}<10 High
+
Low free disk space on $server volume /var                 {$server:vfs.fs.size[/var,pfree].last(0)}<10 High
Low number of free inodes on $server volume / {$server:vfs.fs.inode[/,pfree].last(0)}<10 High
+
Low number of free inodes on $server volume /                 {$server:vfs.fs.inode[/,pfree].last(0)}<10 High
Low number of free inodes on $server volume /home {$server:vfs.fs.inode[/home,pfree].last(0)}<10 High
+
Low number of free inodes on $server volume /home           {$server:vfs.fs.inode[/home,pfree].last(0)}<10 High
Low number of free inodes on $server volume /opt {$server:vfs.fs.inode[/opt,pfree].last(0)}<10 High
+
Low number of free inodes on $server volume /opt       {$server:vfs.fs.inode[/opt,pfree].last(0)}<10 High
Low number of free inodes on $server volume /tmp {$server:vfs.fs.inode[/tmp,pfree].last(0)}<10 High
+
Low number of free inodes on $server volume /tmp       {$server:vfs.fs.inode[/tmp,pfree].last(0)}<10 High
Low number of free inodes on $server volume /usr {$server:vfs.fs.inode[/usr,pfree].last(0)}<10 High
+
Low number of free inodes on $server volume /usr         {$server:vfs.fs.inode[/usr,pfree].last(0)}<10 High
Mysql is not running on $server {$server:proc.num[mysqld].last(0)}<1 Average
+
Mysql is not running on $server                                 {$server:proc.num[mysqld].last(0)}<1 Average
News (NNTP) server is down on $server {$server:net.tcp.service[nntp].last(0)}=0 Average
+
News (NNTP) server is down on $server                         {$server:net.tcp.service[nntp].last(0)}=0 Average
POP3 server is down on $server {$server:net.tcp.service[pop].last(0)}=0 Average
+
POP3 server is down on $server                                 {$server:net.tcp.service[pop].last(0)}=0 Average
Processor load is too high on $server {$server:system.cpu.load[,avg1].last(0)}>5 Average
+
Processor load is too high on $server                         {$server:system.cpu.load[,avg1].last(0)}>5 Average
Server $server is unreachable {$server:status.last(0)}=2 High
+
Server $server is unreachable                                 {$server:status.last(0)}=2 High
SSH server is down on $server {$server:net.tcp.service[ssh].last(0)}=0 Average
+
SSH server is down on $server                                 {$server:net.tcp.service[ssh].last(0)}=0 Average
Sshd is not running on $server {$server:proc.num[sshd].last(0)}<1 Average
+
Sshd is not running on $server                                 {$server:proc.num[sshd].last(0)}<1 Average
Syslogd is not running on $server {$server:proc.num[syslogd].last(0)}<1 Average
+
Syslogd is not running on $server                         {$server:proc.num[syslogd].last(0)}<1 Average
Too many processes on $server {$server:proc.num[].last(0)}>300 High
+
Too many processes on $server                                 {$server:proc.num[].last(0)}>300 High
Too many processes running on $server {$server:proc.num[,,run].last(0)}>10 Average
+
Too many processes running on $server                         {$server:proc.num[,,run].last(0)}>10 Average
Too may users connected on server $server {$server:system.users.num.last(0)}>50 Average
+
Too may users connected on server $server                 {$server:system.users.num.last(0)}>50 Average
Version of zabbix_agent(d) was changed on $server {$server:agent.version.diff(0)}>0 Average
+
Version of zabbix_agent(d) was changed on $server       {$server:agent.version.diff(0)}>0 Average
WEB (HTTP) server is down on $server {$server:net.tcp.service[http].last(0)}=0 Average
+
WEB (HTTP) server is down on $server                         {$server:net.tcp.service[http].last(0)}=0 Average
Zabbix_agentd is not running on $server {$server:proc.num[zabbix_agentd].last(0)}<1 Average
+
Zabbix_agentd is not running on $server                         {$server:proc.num[zabbix_agentd].last(0)}<1 Average
Zabbix_server is not running on $server {$server:proc.num[zabbix_server].last(0)}<1 Average
+
Zabbix_server is not running on $server                         {$server:proc.num[zabbix_server].last(0)}<1 Average
$server has just been restarted {$server:system.uptime.last(0)}<600 Information
+
$server has just been restarted                                 {$server:system.uptime.last(0)}<600 Information
 
</pre>
 
</pre>
  

Verze z 1. 3. 2009, 02:04

Obsah

Obecné informace

NMS je Network monitoring system. Hlavním cílem tohoto serveru je, monitorování dostupnosti a performance veškerého provozu na Strahově.


Cíle

  • performance a alarm monitoring serveru a sitovych prvku

Cílem je prozkoumat, zda je dany spravovaný server aktivní, pokud aktivní je, je vše v pořádku, v opačném případě zašle systém e-mail správci, zástupci a technickému manažerovi. Systém dále zašle SMS správci a jeho zástupci o nekativitě PC.

Monitorované platformy

Switche a routre

  • Cisco Catalyst 6509
  • Cisco 3750
  • Cisco 2950
  • Cisco 2960
  • Cisco Air Ap 1242AG-E-K9

Servery

  • HP Proliant 145 G2
  • HP Proliant DL380 G5
  • Různe x86a x64 servery
  • Dell ???

Jina zarizeni

  • KVM
  • APC Symmetra LX 16kVA

Operacni systemy

  • Debian GNU Linux Lenny
  • Debian GNU Linux 4.0 Etch
  • Debian GNU Linux 3.1 Sarge
  • FreeBSD 6.3
  • Microsoft Windows Server 2003 Standart x64
  • Microsoft Windows Server 2008

Komunikační protokoly vyuzivany k monitoringu

  • ICMP (ping)
  • SNMP (161/UDP)
  • SYSLOG (514/UDP)
  • ZABBIX AGENT (10050/TCP)
  • ZABBIX TRAP (10051/TCP)

Alarm monitoring

Dostupnost sítovych zarizeni

  • Monitorovaní stavu portů u jednotlivých propojů na páteři a blocích pomocí syslog správ nebo SNMP trapů
  • Monitorování dostupnosti síťových prvkú pomocí ICMP pingu
  • Seznam síťových prvků a monitorovaných interfaců:[xls]

Dostupnost serverů

  • Dostupnost serverů se ověřuje pomoci ICMP pingu.
  • Seznam monitorovaných serverů: [xls]

Dostupnost služeb

  • Monitorování dostupnosti TCP/UDP služeb je zabezpečen pomocí požadavků na daný TCP/UDP port a kontrolou bežíciho procesu
  • Seznam ověřovaných služeb: [xls]

Performance monitoring

  • Zátež CPU (load)
  • Vyuziti pameti
  • Mereni teplot
  • Chybovost sitovych interfacu
  • Stav baterie APC

Severity

  • Information
  • Warning
  • Average
  • High
  • Disaster

Default triggers

Linux servers

Name	Expression	Severity
/etc/inetd.conf has been changed on server $server              {$server:vfs.file.cksum[/etc/inetd.conf].diff(0)}>0	Warning
/etc/passwd has been changed on server $server                  {$server:vfs.file.cksum[/etc/passwd].diff(0)}>0	        Average
/etc/services has been changed on server $server        	{$server:vfs.file.cksum[/etc/services].diff(0)}>0	Average
/usr/bin/ssh has been changed on server $server         	{$server:vfs.file.cksum[/usr/bin/ssh].diff(0)}>0	Average
/usr/sbin/sshd has been changed on server $server       	{$server:vfs.file.cksum[/usr/sbin/sshd].diff(0)}>0	Average
/vmlinuz has been changed on server $server	                {$server:vfs.file.cksum[/vmlinuz].diff(0)}>0	        Warning
Apache is not running on $server	                        {$server:proc.num[httpd].last(0)}<1                   	Average
Configured max number of opened files is too low on $server	{$server:kernel.maxfiles.last(0)}<512           	Information
Configured max number of processes is too low on $server	{$server:kernel.maxproc.last(0)}<256                        Information
Email (SMTP) server is down on $server                  	{$server:net.tcp.service[smtp].last(0)}=0	        Average
FTP server is down on $server                              	{$server:net.tcp.service[ftp].last(0)}=0	        Average
Host information was changed on $server                 	{$server:system.uname.diff(0)}>0	             Information
Hostname was changed on $server	                                {$server:system.hostname.diff(0)}>0	Information
IMAP server is down on $server	                                {$server:net.tcp.service[imap].last(0)}=0	Average
Inetd is not running on $server	                                {$server:proc.num[inetd].last(0)}<1	Average
Lack of free memory on server $server	                        {$server:vm.memory.size[free].last(0)}<10000	Average
Lack of free swap space on $server	                        {$server:system.swap.size[,free].last(0)}<100000	High
Low free disk space on $server volume /	                        {$server:vfs.fs.size[/,pfree].last(0)}<10	High
Low free disk space on $server volume /home	                {$server:vfs.fs.size[/home,pfree].last(0)}<10	High
Low free disk space on $server volume /opt	                {$server:vfs.fs.size[/opt,pfree].last(0)}<10	High
Low free disk space on $server volume /tmp	                {$server:vfs.fs.size[/tmp,pfree].last(0)}<10	High
Low free disk space on $server volume /usr	                {$server:vfs.fs.size[/usr,pfree].last(0)}<10	High
Low free disk space on $server volume /var	                {$server:vfs.fs.size[/var,pfree].last(0)}<10	High
Low number of free inodes on $server volume /	                {$server:vfs.fs.inode[/,pfree].last(0)}<10	High
Low number of free inodes on $server volume /home           	{$server:vfs.fs.inode[/home,pfree].last(0)}<10	High
Low number of free inodes on $server volume /opt        	{$server:vfs.fs.inode[/opt,pfree].last(0)}<10	High
Low number of free inodes on $server volume /tmp        	{$server:vfs.fs.inode[/tmp,pfree].last(0)}<10	High
Low number of free inodes on $server volume /usr	        {$server:vfs.fs.inode[/usr,pfree].last(0)}<10	High
Mysql is not running on $server	                                {$server:proc.num[mysqld].last(0)}<1	Average
News (NNTP) server is down on $server	                        {$server:net.tcp.service[nntp].last(0)}=0	Average
POP3 server is down on $server	                                {$server:net.tcp.service[pop].last(0)}=0	Average
Processor load is too high on $server	                        {$server:system.cpu.load[,avg1].last(0)}>5	Average
Server $server is unreachable	                                {$server:status.last(0)}=2	High
SSH server is down on $server	                                {$server:net.tcp.service[ssh].last(0)}=0	Average
Sshd is not running on $server	                                {$server:proc.num[sshd].last(0)}<1	Average
Syslogd is not running on $server	                        {$server:proc.num[syslogd].last(0)}<1	Average
Too many processes on $server	                                {$server:proc.num[].last(0)}>300	High
Too many processes running on $server	                        {$server:proc.num[,,run].last(0)}>10	Average
Too may users connected on server $server	                {$server:system.users.num.last(0)}>50	Average
Version of zabbix_agent(d) was changed on $server       	{$server:agent.version.diff(0)}>0	Average
WEB (HTTP) server is down on $server	                        {$server:net.tcp.service[http].last(0)}=0	Average
Zabbix_agentd is not running on $server	                        {$server:proc.num[zabbix_agentd].last(0)}<1	Average
Zabbix_server is not running on $server	                        {$server:proc.num[zabbix_server].last(0)}<1	Average
$server has just been restarted	                                {$server:system.uptime.last(0)}<600	Information

Notifikace

Notifikace sou rozesílane emailem pres smtp server service.sh.cvut.cz na adresu $server@sh.cvut.cz a přez GSM modul formou SMS (zatim neimplementovano).

Format email notifikace

Head>To:		$HOSTALIAS@sh.cvut.cz
Head>From:		nms@sh.cvut.cz
Head>Subject:	$HOSTALIAS - $ALARMTYPE
Body>Notification Type: $NOTIFICATIONTYPE$
Body>Host: $HOSTALIAS$
Body>State: $HOSTSTATE$ for $HOSTDURATION$
Body>Alarmtype: $ALARMTYPE
Body>Address: $HOSTADDRESS$
Body>Info: $HOSTOUTPUT$
Body>Date/Time: $LONGDATETIME$
Body>ACK by: $HOSTACKAUTHOR$
Body>Comment: $HOSTACKCOMMENT$
Body>Severity: $SEVERITY

Format SMS notifikace

  • Maximálně 160 znaků.
Head>To:		$ADMIN_NUMBER
Head>From:		$NMS_NUMBER

Body>Host: $HOSTALIAS
Body>State: $HOSTSTATE$ for $HOSTDURATION$
Body>Date/Time: $LONGDATETIME$
Body>Alarmtype: $ALARMTYPE
Body>Severity: $SEVERITY

Informace o serveru nms.sh.cvut.cz

  • Operační systém : Debian Linux Lenny
  • Hardware : Dell Power Edge
  • Správce : James
  • Zabbixadmin : Netopier

Software

  • zabbix 1.4.2 - activity and performance monitoring
  • cacti 0.7.8c - monitoring sitoveho provozu na portech cisco routru

Hostované služby

  • monitoring serverů

Administrátorský (root) přístup na server

Římek Jakub - správce serveru
Leonov Alexander - zabbix admin

Ruzne navody

Nastaveni zabbix agenta

/etc/zabbix/zabbix_agent.conf

Server=147.32.127.234
Timeout=3

/etc/zabbix/zabbix_agentd.conf

Server=147.32.127.234
ServerPort=10051
Hostname=$hostname_serveru.sh.cvut.cz
ListenPort=10050
ListenIP=$ipadresa_serveru
StartAgents=5
Timeout=3

Nastaveni cisco routru

Global settings:

logging 147.32.127.234
snmp-server enable traps
snmp-server enable authentication
snmp-server enable linkdown linkup
snmp-server enable coldstart warmstart
snmp-server enable traps chassis
snmp-server enable traps module
snmp-server enable traps ospf state-change
snmp-server enable traps ospf errors
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps cpu threshold

Interface settings:

logging event link-status
snmp trap link-status
Jmenné prostory

Varianty
Akce