1、Program terminated with signal 11, Segmentation fault.

定位思路：

1.1 查看堆栈

            Program terminated with signal 11, Segmentation fault.

            #0  0x00007f8e0fa57d00 in pthread_mutex_lock () from /lib64/libpthread.so.0
            Missing separate debuginfos, use: debuginfo-install glibc-2.17-307.el7.1.x86_64 libgcc-4.8.5-39.el7.x86_64 libstdc++-4.8.5-39.el7.x86_64
            (gdb) bt
            #0  0x00007f8e0fa57d00 in pthread_mutex_lock () from /lib64/libpthread.so.0
            #1  0x00007f8e0fd5ad66 in ACE_OS::mutex_lock(pthread_mutex_t*) ()
               from /mnt/wyyong2/dailytest/asr/ims/IMS37_RedHat_Build9004/lib64/libACE.so.5.7.1
            #2  0x000000000042e9de in ACE_OS::thread_mutex_lock (m=0x90) at ../../../share/ACE/ace/OS_NS_Thread.inl:3417
            #3  0x0000000000435e21 in ACE_OS::recursive_mutex_lock (m=0x90) at ../../../share/ACE/ace/OS_NS_Thread.inl:691
            #4  0x0000000000435ed2 in ACE_Recursive_Thread_Mutex::acquire (this=0x90)
                at ../../../share/ACE/ace/Recursive_Thread_Mutex.inl:37
            #5  0x000000000043a7ab in ACE_Guard<ACE_Recursive_Thread_Mutex>::acquire (this=0x7f8dee7fb3a0)
                at ../../../share/ACE/ace/Guard_T.inl:12
            #6  0x0000000000437f12 in ACE_Guard<ACE_Recursive_Thread_Mutex>::ACE_Guard (this=0x7f8dee7fb3a0, l=...)
                at ../../../share/ACE/ace/Guard_T.inl:38
            #7  0x0000000000449389 in ISP_Reactor_Acceptor<SRM_Ev_Handler, 62>::clear_timer_evh (this=0x0)
                at ../../common/ISP_Com/ISP_Reactor_Acceptor.cpp:529
            #8  0x0000000000446f4c in SRM_Ev_Task::svc (this=0x7ffff3ac27a0)
                at /home/jenkins/workspace/YYDHDLMRMREDHAT_7c08eca4-3973-4435-a6b0-4a5e721482c0/source/core/server/MRM/SRM_Ev_Task.cpp:288
            #9  0x00007f8e0fd82a87 in ACE_Task_Base::svc_run(void*) ()
               from /mnt/wyyong2/dailytest/asr/ims/IMS37_RedHat_Build9004/lib64/libACE.so.5.7.1
            #10 0x00007f8e0fd84051 in ACE_Thread_Adapter::invoke() ()
               from /mnt/wyyong2/dailytest/asr/ims/IMS37_RedHat_Build9004/lib64/libACE.so.5.7.1
            #11 0x00007f8e0fa55ea5 in start_thread () from /lib64/libpthread.so.0
            #12 0x00007f8e0e4b78dd in clone () from /lib64/libc.so.6

1.2 查看代码段

代码段1：

      283:// 检查返回值，如果这里错误，那么需要设置全局退出标记，退出进程
      284:if ( ret != 0 )
      285:{
      286:  GET_INFO_MNGR()->set_exit_flag(true);
      287:  acceptor_->clear_timer_evh();
      288:  return ret;
      289:}

代码段2：

      526:template<class RCVR, int MAX_RCVR>
      527:int ISP_Reactor_Acceptor<RCVR, MAX_RCVR>::clear_timer_evh(void)
      528:{
      529:      ACE_GUARD_RETURN (ACE_Recursive_Thread_Mutex, monitor, lock_, -1);

      531:      std::list<ACE_Event_Handler*>::iterator pi = timer_evh_list_.begin();

查看lock_ 是否为null；查看堆栈 ACE_OS::recursive_mutex_lock (m=0x90)可知，lock_ 不为空；

1.3 查看代码：

发现一个疑点

是否由于时assert导致；但是这里的ISP_ASSERT_LOGERR 较为复杂，从编译入手，使用 -E 查看预编译后代码：

      160798    ret = acceptor_->open(local, backlog);
      160799    if ( ret != 0 )
      160800    {
      160801 
      160802      if ( !(ret == 0) ) { if ( ISP_Logger::instance() ) ISP_Logger::instance()->log_error ("MRM_Ev_Task::svc() open eve       nt acceptor failed, code = %d", ACE_OS::last_error()); (0); }

发现这里并没有assert 使用，原因时release版本屏蔽了assert：
同时本地验证 assert 产生的coredump；

      Core was generated by `./a.out'.
      Program terminated with signal 6, Aborted.
      #0  0x00007fba9869831e in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:67
      67      int res = INLINE_SYSCALL (tgkill, 3, pid, selftid, sig);
      Missing separate debuginfos, use: debuginfo-install glibc-2.12-1.212.el6_10.3.x86_64 libgcc-4.4.7-16.el6.x86_64 libstdc++-4.4.7-16.el6.x86_64
      (gdb) bt
      #0  0x00007fba9869831e in raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:67
      #1  0x00007fba98699765 in abort () at abort.c:91
      #2  0x00007fba986918ce in __assert_fail_base (fmt=<value optimized out>, assertion=0x4006d1 "0", file=0x4006c8 "test.cpp", 
          line=<value optimized out>, function=<value optimized out>) at assert.c:94
      #3  0x00007fba98691990 in __assert_fail (assertion=0x4006d1 "0", file=0x4006c8 "test.cpp", line=7, 
          function=0x4006d3 "int main()") at assert.c:103
      #4  0x00000000004005c1 in main ()

在linux上 assert产生的coredump发送的时信号6，注意区分

1.4 回到堆栈信息

看是否可以挖掘到可用的信息
这里查看到，对应代码

      #8  0x0000000000446f4c in SRM_Ev_Task::svc (this=0x7ffff3ac27a0)
    at /home/jenkins/workspace/YYDHDLMRMREDHAT_7c08eca4-3973-4435-a6b0-4a5e721482c0/source/core/server/MRM/SRM_Ev_Task.cpp:288

查看：

      (gdb) f 8
      #8  0x0000000000446f4c in SRM_Ev_Task::svc (this=0x7ffff3ac27a0)
          at /home/jenkins/workspace/YYDHDLMRMREDHAT_7c08eca4-3973-4435-a6b0-4a5e721482c0/source/core/server/MRM/SRM_Ev_Task.cpp:288
      288   /home/jenkins/workspace/YYDHDLMRMREDHAT_7c08eca4-3973-4435-a6b0-4a5e721482c0/source/core/server/MRM/SRM_Ev_Task.cpp: 没有那个文件或目录.
      (gdb) p acceptor_
      $1 = (SRM_Ev_Acceptor *) 0x0

发现原来是 acceptor_ 为空，但是调用了对象的函数，那为什么还可以调用呢？解释对象为空时，可调用成员函数

      250             ret = acceptor_->open(local, backlog);
      251             if ( ret != 0 )
      252             {
      253                  delete acceptor_, acceptor_ = NULL;
      254                  ISP_ASSERT_LOGERR(ret == 0,
      255                     ("MRM_Ev_Task::svc() open event acceptor failed, code = %d", ACE_OS::last_error()));