[asterisk-dev] segfault in asterisk 1.6.1.6 chan_iax2 after approx. 75.000 calls.

[Robert Verspuy]

I'm running Asterisk 1.6.1.6 since 19 days now,
but I've had 3 times a segfault on the same address.

Sep 17 13:03:57 switch02 kernel: asterisk[13597]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 00000000420b1440 error 4
     In the mean time 75.517 calls were setup.
Sep 22 19:07:13 switch02 kernel: asterisk[23982]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 0000000042142440 error 4
     In the mean time  76.725 calls were setup.
Sep 28 14:26:16 switch02 kernel: asterisk[2777]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 0000000042088440 error 4

When running gdb on the coredump:

Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
#0  _ao2_find (c=0x2aaa00000002, arg=0x420884d0, flags=OBJ_POINTER) at 
astobj2.c:712
712        return __ao2_callback(c,flags, cb_fn, arg, NULL, NULL, 0, NULL);
(gdb) bt
#0  _ao2_find (c=0x2aaa00000002, arg=0x420884d0, flags=OBJ_POINTER) at 
astobj2.c:712
#1  0x00002aaac090e2ae in __find_callno (callno=1, dcallno=26995, 
sin=0x4208ce80, new=0, sockfd=14, return_locked=0, check_dcallno=1) at 
chan_iax2.c:2450
#2  0x00002aaac091b570 in socket_process (thread=0x7c0eaa0) at 
chan_iax2.c:2582
#3  0x00002aaac0926459 in iax2_process_thread (data=0x7c0eaa0) at 
chan_iax2.c:10933
#4  0x00000000004f28ac in dummy_start (data=<value optimized out>) at 
utils.c:968
#5  0x0000003b07a06367 in start_thread () from /lib64/libpthread.so.0
#6  0x0000003b06ed309d in clone () from /lib64/libc.so.6

It's a live server running production traffic. So I don't have much 
possibilities to easily test a different version.
Also the issue only happens to me once a week (about once every approx. 
75.000 calls).
The segfault didn't happen on the most busiest times (sunday), but just 
on the more quiet days.

The last time the server had 50 calls, 97 channels (43 chan_ss7 
channels, 46 sip channels and 8 iax2 channels)

Software running:
  - CentOS 5 (latest updates as of 11 sept 2009)
  - asterisk 1.6.1.6
  - chan_ss7 1.2.1
  - dahdi-linux 2.2.0.2
  - dahdi-tools 2.2.0
  - wanpipe 3.5.6

Is there any other information I can get from the core file to 
understand what's going wrong,
and how to fix it?

Thnx,
Robert Verspuy

-- 
*Exa-Omicron*
Patroonsweg 10
3892 DB Zeewolde
Tel.: 088-OMICRON (66 427 66)
http://www.exa-omicron.nl