Analyzing a DB2 trap file

Usually  when the DB2 instance ends unexpectedly, trap files are created .There is usually one trap file capturing  the root cause . A number of other files are created, supporting data about DB2 processes. The dbdiag.log file indicates which files to analyze.

By default , the files are created in “<INATANCE_HOME>/sqllib/db2dump/FODC*”. A typical list of files are :

19880.000.core

19880.20.000.db2pd.alldbs.txt 

19880.20.000.trap.txt 

db2eventlog.000.crash 

db2pd.dbptnmem.txt  OSSNAPS    

sysctl.txt

19880.20.000.cos.txt

19880.20.000.rawstack.txt

chkconfig.txt

db2pd.dbmcfg.txt

db2pd.memory.txt

procmaps.txt

top.txt

In the  DB2 instance db2diag.log , various messages may appear:

2011-11-10-10.42.24.462022+000 E38459E1556         LEVEL: Critical
PID     : 22233                TID  : 183312050528 PROC : db2sysc
INSTANCE: INSTANCE_HOME               NODE : 000
EDUID   : 97                   EDUNAME: db2logmgr (MYDB)
FUNCTION: DB2 UDB, oper system services, sqloEDUCodeTrapHandler, probe:10
MESSAGE : ADM14001C  An unexpected and critical error has occurred: "Trap". The           instance may have been shutdown as a result. "Automatic" FODC (First Occurrence Data Capture) has been invoked and diagnostic information  has been recorded in directory       "/INSTANCE_HOME/sqllib/db2dump/FODC_Trap_2011-11-10-10.42.18.403802/". Please look in this directory for detailed evidence about what  happened and contact IBM support if necessary to diagnose the problem.
DATA #1 : Signal Number Recieved, 4 bytes
11
DATA #2 : Siginfo, 128 bytes

2011-11-10-10.42.24.485359+000 I40016E481          LEVEL: Error
PID     : 17630                TID  : 183083405184 PROC : db2vend (db2logmgr.meth1 - 97 (W
INSTANCE: INSTANCE_HOME               NODE : 000
FUNCTION: DB2 UDB, oper system services, DB2VEND_main, probe:1040
MESSAGE : ZRC=0x870F0009=-2029060087=SQLO_EOF "the data does not exist" DIA8506C Unexpected end of file was reached.
DATA #1 : String, 63 bytes
Fenced vendor process pipe failure, terminating vendor process.

2011-11-10-10.42.24.488118+000 E40498E547          LEVEL: Severe
PID     : 22231                TID  : 183089752416 PROC : db2wdog
INSTANCE: INSTANCE_HOME               NODE : 000
EDUID   : 2                    EDUNAME: db2wdog
FUNCTION: DB2 UDB, base sys utilities, sqleWatchDog, probe:20
MESSAGE : ADM0503C  An unexpected internal processing error has occurred.  ALL DB2 PROCESSES ASSOCIATED WITH THIS INSTANCE HAVE BEEN SHUTDOWN.  Diagnostic information has been recorded.  Contact IBM Support for further assistance.

In the above example, the db2diag.log entry shows that the function sqloEDUCodeTrapHandler returned a Signal number of 11. This points towards a segmentation fault\violation and the database signal handler has caught the signal.

A segmentation violation is an error occurring where an active Linux program tries to access memory not allocated to the program. It terminates with a segmentation violation.

When you look in the trap file, you should see that it contains the same signal number as was indicated by the db2diag.log entry.

On Linux platforms, this information in the trap files is displayed as:

This is a snippet of the trap file

Signal #11 (SIGSEGV): si_addr is 0x0000000000000000, si_code is 0x00000001 (SEGV_MAPERR:Address not mapped to object.)
Data seg top [sbrk(0)] = 0x00000000006BB000
Cur data size (bytes)  = 0xFFFFFFFF
Cur stack size (bytes) = 0x00A00000

Read More on DIAGLEVEL , diagnostic levels and db2diag.log

DIAGLEVEL db2 - DBA DB2

DB2 – Managing Diagnostic Logs - DBA DB2

db2diag.log file delete on Linux and DB2 - DBA DB2