Verze z 1. 3. 2009, 02:07

Obecné informace

NMS je Network monitoring system. Hlavním cílem tohoto serveru je, monitorování dostupnosti a performance veškerého provozu na Strahově.

Cíle

performance a alarm monitoring serveru a sitovych prvku

Cílem je prozkoumat, zda je dany spravovaný server aktivní, pokud aktivní je, je vše v pořádku, v opačném případě zašle systém e-mail správci, zástupci a technickému manažerovi. Systém dále zašle SMS správci a jeho zástupci o nekativitě PC.

Monitorované platformy

Switche a routre

Cisco Catalyst 6509
Cisco 3750
Cisco 2950
Cisco 2960
Cisco Air Ap 1242AG-E-K9

Servery

HP Proliant 145 G2
HP Proliant DL380 G5
Různe x86a x64 servery
Dell ???

Jina zarizeni

KVM
APC Symmetra LX 16kVA

Operacni systemy

Debian GNU Linux Lenny
Debian GNU Linux 4.0 Etch
Debian GNU Linux 3.1 Sarge
FreeBSD 6.3
Microsoft Windows Server 2003 Standart x64
Microsoft Windows Server 2008

Komunikační protokoly vyuzivany k monitoringu

ICMP (ping)
SNMP (161/UDP)
SYSLOG (514/UDP)
ZABBIX AGENT (10050/TCP)
ZABBIX TRAP (10051/TCP)

Alarm monitoring

Dostupnost sítovych zarizeni

Monitorovaní stavu portů u jednotlivých propojů na páteři a blocích pomocí syslog správ nebo SNMP trapů
Monitorování dostupnosti síťových prvkú pomocí ICMP pingu
Seznam síťových prvků a monitorovaných interfaců:[xls]

Dostupnost serverů

Dostupnost serverů se ověřuje pomoci ICMP pingu.
Seznam monitorovaných serverů: [xls]

Dostupnost služeb

Monitorování dostupnosti TCP/UDP služeb je zabezpečen pomocí požadavků na daný TCP/UDP port a kontrolou bežíciho procesu
Seznam ověřovaných služeb: [xls]

Performance monitoring

Zátež CPU (load)
Vyuziti pameti
Mereni teplot
Chybovost sitovych interfacu
Stav baterie APC

Severity

Information
Warning
Average
High
Disaster

Default triggers

Linux servers

Name	                                                        Expression	                                        Severity
/etc/inetd.conf has been changed on server $server              {$server:vfs.file.cksum[/etc/inetd.conf].diff(0)}>0	Warning
/etc/passwd has been changed on server $server                  {$server:vfs.file.cksum[/etc/passwd].diff(0)}>0	        Average
/etc/services has been changed on server $server        	{$server:vfs.file.cksum[/etc/services].diff(0)}>0	Average
/usr/bin/ssh has been changed on server $server         	{$server:vfs.file.cksum[/usr/bin/ssh].diff(0)}>0	Average
/usr/sbin/sshd has been changed on server $server       	{$server:vfs.file.cksum[/usr/sbin/sshd].diff(0)}>0	Average
/vmlinuz has been changed on server $server	                {$server:vfs.file.cksum[/vmlinuz].diff(0)}>0	        Warning
Apache is not running on $server	                        {$server:proc.num[httpd].last(0)}<1                   	Average
Configured max number of opened files is too low on $server	{$server:kernel.maxfiles.last(0)}<512           	Information
Configured max number of processes is too low on $server	{$server:kernel.maxproc.last(0)}<256                    Information
Email (SMTP) server is down on $server                  	{$server:net.tcp.service[smtp].last(0)}=0	        Average
FTP server is down on $server                              	{$server:net.tcp.service[ftp].last(0)}=0	        Average
Host information was changed on $server                 	{$server:system.uname.diff(0)}>0	                Information
Hostname was changed on $server	                                {$server:system.hostname.diff(0)}>0	                Information
IMAP server is down on $server	                                {$server:net.tcp.service[imap].last(0)}=0	        Average
Inetd is not running on $server	                                {$server:proc.num[inetd].last(0)}<1	                Average
Lack of free memory on server $server	                        {$server:vm.memory.size[free].last(0)}<10000	        Average
Lack of free swap space on $server	                        {$server:system.swap.size[,free].last(0)}<100000	High
Low free disk space on $server volume /	                        {$server:vfs.fs.size[/,pfree].last(0)}<10	        High
Low free disk space on $server volume /home	                {$server:vfs.fs.size[/home,pfree].last(0)}<10	        High
Low free disk space on $server volume /opt	                {$server:vfs.fs.size[/opt,pfree].last(0)}<10	        High
Low free disk space on $server volume /tmp	                {$server:vfs.fs.size[/tmp,pfree].last(0)}<10	        High
Low free disk space on $server volume /usr	                {$server:vfs.fs.size[/usr,pfree].last(0)}<10	        High
Low free disk space on $server volume /var	                {$server:vfs.fs.size[/var,pfree].last(0)}<10	        High
Low number of free inodes on $server volume /	                {$server:vfs.fs.inode[/,pfree].last(0)}<10	        High
Low number of free inodes on $server volume /home           	{$server:vfs.fs.inode[/home,pfree].last(0)}<10	        High
Low number of free inodes on $server volume /opt        	{$server:vfs.fs.inode[/opt,pfree].last(0)}<10	        High
Low number of free inodes on $server volume /tmp        	{$server:vfs.fs.inode[/tmp,pfree].last(0)}<10	        High
Low number of free inodes on $server volume /usr	        {$server:vfs.fs.inode[/usr,pfree].last(0)}<10	        High
Mysql is not running on $server	                                {$server:proc.num[mysqld].last(0)}<1	                Average
News (NNTP) server is down on $server	                        {$server:net.tcp.service[nntp].last(0)}=0	        Average
POP3 server is down on $server	                                {$server:net.tcp.service[pop].last(0)}=0	        Average
Processor load is too high on $server	                        {$server:system.cpu.load[,avg1].last(0)}>5 	        Average
Server $server is unreachable	                                {$server:status.last(0)}=2	                        High
SSH server is down on $server	                                {$server:net.tcp.service[ssh].last(0)}=0	        Average
Sshd is not running on $server	                                {$server:proc.num[sshd].last(0)}<1	                Average
Syslogd is not running on $server	                        {$server:proc.num[syslogd].last(0)}<1	                Average
Too many processes on $server	                                {$server:proc.num[].last(0)}>300	                High
Too many processes running on $server	                        {$server:proc.num[,,run].last(0)}>10	                Average
Too may users connected on server $server	                {$server:system.users.num.last(0)}>50	                Average
Version of zabbix_agent(d) was changed on $server       	{$server:agent.version.diff(0)}>0	                Average
WEB (HTTP) server is down on $server	                        {$server:net.tcp.service[http].last(0)}=0	        Average
Zabbix_agentd is not running on $server	                        {$server:proc.num[zabbix_agentd].last(0)}<1	        Average
Zabbix_server is not running on $server	                        {$server:proc.num[zabbix_server].last(0)}<1	        Average
$server has just been restarted	                                {$server:system.uptime.last(0)}<600	                Information

Notifikace

Notifikace sou rozesílane emailem pres smtp server service.sh.cvut.cz na adresu $server@sh.cvut.cz a přez GSM modul formou SMS (zatim neimplementovano).

Format email notifikace

Head>To:		$HOSTALIAS@sh.cvut.cz
Head>From:		nms@sh.cvut.cz
Head>Subject:	$HOSTALIAS - $ALARMTYPE
Body>Notification Type: $NOTIFICATIONTYPE$
Body>Host: $HOSTALIAS$
Body>State: $HOSTSTATE$ for $HOSTDURATION$
Body>Alarmtype: $ALARMTYPE
Body>Address: $HOSTADDRESS$
Body>Info: $HOSTOUTPUT$
Body>Date/Time: $LONGDATETIME$
Body>ACK by: $HOSTACKAUTHOR$
Body>Comment: $HOSTACKCOMMENT$
Body>Severity: $SEVERITY

Format SMS notifikace

Maximálně 160 znaků.

Head>To:		$ADMIN_NUMBER
Head>From:		$NMS_NUMBER

Body>Host: $HOSTALIAS
Body>State: $HOSTSTATE$ for $HOSTDURATION$
Body>Date/Time: $LONGDATETIME$
Body>Alarmtype: $ALARMTYPE
Body>Severity: $SEVERITY

Informace o serveru nms.sh.cvut.cz

Operační systém : Debian Linux Lenny
Hardware : Dell Power Edge
Správce : James
Zabbixadmin : Netopier

Software

zabbix 1.4.2 - activity and performance monitoring
cacti 0.7.8c - monitoring sitoveho provozu na portech cisco routru

Hostované služby

monitoring serverů

Administrátorský (root) přístup na server

Římek Jakub - správce serveru
Leonov Alexander - zabbix admin

Ruzne navody

Nastaveni zabbix agenta

/etc/zabbix/zabbix_agent.conf

Server=147.32.127.234
Timeout=3

/etc/zabbix/zabbix_agentd.conf

Server=147.32.127.234
ServerPort=10051
Hostname=$hostname_serveru.sh.cvut.cz
ListenPort=10050
ListenIP=$ipadresa_serveru
StartAgents=5
Timeout=3

Nastaveni cisco routru

Global settings:

logging 147.32.127.234
snmp-server enable traps
snmp-server enable authentication
snmp-server enable linkdown linkup
snmp-server enable coldstart warmstart
snmp-server enable traps chassis
snmp-server enable traps module
snmp-server enable traps ospf state-change
snmp-server enable traps ospf errors
snmp-server enable traps config-copy
snmp-server enable traps config
snmp-server enable traps cpu threshold

Interface settings:

logging event link-status
snmp trap link-status

Servery nms