Monitoring of servers and applications is an integral part in enterprise-grade environments. It does not matter what type of applications you are utilizing or how complex they are. They just have to work!

This article provides an easy-to-implement approach to monitor periodically executed programms / scripts, local and networked services. Even embedded and self-sufficient systems can benefit!

1. Description of the Problem

You are running a business and you are required to provide working network services or there are cronjobs running to perform regular jobs. For sure you want to get notified if things go wrong. For this task a lot of commercial and open source software solutions are available. In this article we do not want to compare monitoring solutions or to describe how to setup one solution or another.

We want to show up a strategy that will help you to be successful to keep your services up and running.

2. Standards and Independance

When selecting monitoring solutions do not overrate graphical user interfaces and promises of nice-looking advertising materials. It is very important, that your monitoring system is flexible and supports the integration of custom monitoring sensors, the definition of rules how to analyse results of returned data and the notification of responsible persons when issues have been detected.

Avoid to adapt your IT infrastructure to fit the needs of a monitoring system. Instead you should utilize solutions that are able to connect to custom systems and protocols. This will ensure that you will be able to change the supplier or the software product, if you do realize that your monitoring system does not fit your needs or necessities to extend your solution are too expensive.

When you are about to buy new software components or development is performed inhouse you should have a look at the specification whether and how the new software is monitorable. Today, software systems are very often complex and hard to administer. If software is not able to tell you details about their innermost and virtual well-being, it will be very hard to identify problems and to predict the load behaviour for the next several months.

Nowadays, software has to provide easy-to-use interfaces to monitor (and to control) internal states of components and subsystems!

2.1. Define Standards for Applications

As mentioned before: Nowadays, software has to provide easy-to-use interfaces to monitor (and to control) internal states of components and subsystems! If there are no standards available that describe how to monitor your applications you have to define them!

Applications developed inhouse are easy to customize. Depending on your field of application internal states may be externalized by making available process data via databases, network interfaces (e.g. via HTTP, HTTPs, SMTP, etc.), filesystems, ... . It is not required to invent new protocols or structures for a new type XML document. Just let your applications generate a list of some important key-value pairs representing the state of processes and the connectivity to external systems. 

You should be able to identify a set of parameters that are important to know for every application. And if you did identify multiple sets this is no problem. In this case you will be able to categorize applications and they will fit to predefined classes of applications. However, for every application you will find a number of state information that is unique but required. 

Please take a look at the following example. It contains data common for every process. And additional parameters will be added at the end of the list. We did choose a simple text format to distribute the state information.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16

# example of status information
status=OK
time=2010-09-02_14:16:48
startup=2010-08-27_13:39:16
hostname=myhostname-200v.linux.rz.linux-support.com
hostaddress=10.191.122.94
instance=waf_secmaster
mem_total=99811328
mem_free=74077736
http-backend01=TRUE
http-backend02=TRUE
http-backend03=TRUE
database01=TRUE
database02=TRUE
swversion=4.1.0.3
 

This kind of state data is retrievable via standard and custom protocols by virtually every client application. It does not require to implement or to inculde additional parsers and it is even readable by human beings.

Collecting this kind of information and analysing historical data will enable you to identify quirks and problems of your systems.

2.2. Standards for Virtual Machines

As mentioned in the previous section you just have to identify status information, that is required for monitoring your applicaitons. It does not matter what runtime environment your applications are utilizing. Programming languages like Java, Smalltalk or Python have ready lots of runtime status information. It is a simple task for developers to collect the specified data and to assemble and return text-based status documents at runtime. 

2.3. Standards for Servers

It does not matter whether you are monitoring real servers or virtualized containers. Virtualized guest systems provide nearly the same features as real servers do. So differences for monitoring these systems are marginal. 

Collecting status information will be performed by software agents or your monitoring system itself. You just have to define the data you are interesed in. Keep in mind, that monitoring servers and the reliable identification of problems is much more complex than observing a set of applications. If not considered before, you may (let) analyse contents of logfiles and the state of hardware devices to get a more comprehensive picture.

2.4. Closed Source Applications

Software provided by third-party vendors does not know what you are interested in. In this case you could delegate the provisioning of status information to software agents. Software agents are able to utilize features provided by operating systems, virtual machines or they have to try to guess. 

3. Whom to Inform?

Whom to inform when problems have been identified? Depending on predefined rules your monitoring system will be able to send notification by email, SMS, fax, SNPP, ... . You should also consider to place notifications in issue tracking systems to ensure that messages will not be collected by email inboxes while responsible persons are on holliday.

4. Self-Healing and Robust Systems

If you are running highly available services or you are responsible for self-sufficient systems you should develop strategies to implement processes for unassisted problem analysis and features for self-healing. Immediately after your system detects that it is not able to bring services up again responsible persons have to be informed.