How to customise TSAMP start and stop scripts to make monitoring agents cluster aware

I’ve integrated Tivoli Storage Automation (TSA) and DB2 .  The  solution is based around a two node TSAMP solution which is “automation software”,  running as part of a clustered Reliable Scalable Cluster Technology (RSCT) environment. (RSCT  is the “cluster software”)

Read More on Tivoli Storage Automation for Multiplatforms(SA MP) and the shared disk approach

TSAMP uses a set of automation scripts to manage DB2 that is, start, stop and monitor them. TSAMP responds to an unexpected change in state for individual resources (eg DB2 instance failure), a server failure/crash, a network/NIC related outage

To confirm the TSA integration with DB2 , you can check the DB2 dbm configuration . If you’re using Linux check :

db2 get dbm cfg | grep 'Cluster manager'

One of the challenges in integrating TSAMP with DB2 is  supporting  services need to be  made cluster-aware. Some examples are monitoring and backups.

In a non clustered environment , there may be a monitoring agent installed monitoring DB2. Monitoring is based on DB2 as a single unit. In the cluster scenario, there is the added complication of correlating the state of one DB2 instance  (Node 1) with the state of another  DB2 instance (Node 2).

There are multiple approaches  to this problem. The approach is dependant on the monitoring platform. You may have an agentless remote monitoring system in place. The solution discussed here  is for situations where a monitoring agent is installed on every OS

For monitoring , the solution applied is to customise the TSA start and stop scripts. The TSA scripts are located on  /usr/sbin/rsct/sapolicies/db2  . They are refreshed at every DB2  version upgrade, therefore tight management is required.

As well as managing the TSA scripts at every DB2 version upgrade, there’ll be a requirement to manage monitoring agent upgrades. The upgrade will probably reset the defaults including the automatic start up at OS start

The basic principle is to take out the monitoring agent from the standard start \ stop process. i.e don’t automatically start the db2  monitoring agent when the OS starts.  Use the clustering software to manage the stop \ start of the db2 agent monitor.

The files  customised are db2V10_start.ksh   and db2V10_stop.ksh.  When the TSA cluster software is running and  is attempting to initiate db2start on Node 1 , one of the scripts it will execute is db2V10_start.ksh.

When there is a failover , one of the scripts TSA will use is db2V10_stop.ksh.  These scripts can be exploited to stop and start various supporting services such as monitoring.

In the situation of a server crash i.e the db2V10_stop.ksh doesn’t run , the server on start up won’t start the monitoring agent , as you’ve already taken it out of the /etc/init.d /   section

The next step is to look at correlating the different states of the Cluster Nodes and avoid false alerts. I’ll discuss this process in a future post.

I’ll also write a post about the monitoring which comes with RSCT. The RSCT monitoring supplies loads of  detail on every aspect of the cluster state  

Read More on DB2 clustering and high availability

TSAMP maintenance and diagnostics

TSAMP Cheat Sheet for DBA managing DB2 clustering