[asterisk-dev] segfault in asterisk 1.6.1.6 chan_iax2 after approx. 75.000 calls.

[David Vossel]

----- "Robert Verspuy" <robert at exa-omicron.nl> wrote:

> I'm running Asterisk 1.6.1.6 since 19 days now,
> but I've had 3 times a segfault on the same address.
> 
> Sep 17 13:03:57 switch02 kernel: asterisk[13597]: segfault at 
> 00002aaa0000000a rip 0000000000435c36 rsp 00000000420b1440 error 4
>      In the mean time 75.517 calls were setup.
> Sep 22 19:07:13 switch02 kernel: asterisk[23982]: segfault at 
> 00002aaa0000000a rip 0000000000435c36 rsp 0000000042142440 error 4
>      In the mean time  76.725 calls were setup.
> Sep 28 14:26:16 switch02 kernel: asterisk[2777]: segfault at 
> 00002aaa0000000a rip 0000000000435c36 rsp 0000000042088440 error 4
> 
> When running gdb on the coredump:
> 
> Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.
> Program terminated with signal 11, Segmentation fault.
> #0  _ao2_find (c=0x2aaa00000002, arg=0x420884d0, flags=OBJ_POINTER) at
> 
> astobj2.c:712
> 712        return __ao2_callback(c,flags, cb_fn, arg, NULL, NULL, 0,
> NULL);
> (gdb) bt
> #0  _ao2_find (c=0x2aaa00000002, arg=0x420884d0, flags=OBJ_POINTER) at
> 
> astobj2.c:712
> #1  0x00002aaac090e2ae in __find_callno (callno=1, dcallno=26995, 
> sin=0x4208ce80, new=0, sockfd=14, return_locked=0, check_dcallno=1) at
> 
> chan_iax2.c:2450
> #2  0x00002aaac091b570 in socket_process (thread=0x7c0eaa0) at 
> chan_iax2.c:2582
> #3  0x00002aaac0926459 in iax2_process_thread (data=0x7c0eaa0) at 
> chan_iax2.c:10933
> #4  0x00000000004f28ac in dummy_start (data=<value optimized out>) at
> 
> utils.c:968
> #5  0x0000003b07a06367 in start_thread () from /lib64/libpthread.so.0
> #6  0x0000003b06ed309d in clone () from /lib64/libc.so.6
> 
> It's a live server running production traffic. So I don't have much 
> possibilities to easily test a different version.
> Also the issue only happens to me once a week (about once every
> approx. 
> 75.000 calls).
> The segfault didn't happen on the most busiest times (sunday), but
> just 
> on the more quiet days.
> 
> The last time the server had 50 calls, 97 channels (43 chan_ss7 
> channels, 46 sip channels and 8 iax2 channels)
> 
> Software running:
>   - CentOS 5 (latest updates as of 11 sept 2009)
>   - asterisk 1.6.1.6
>   - chan_ss7 1.2.1
>   - dahdi-linux 2.2.0.2
>   - dahdi-tools 2.2.0
>   - wanpipe 3.5.6
> 
> Is there any other information I can get from the core file to 
> understand what's going wrong,
> and how to fix it?
> 
> Thnx,
> Robert Verspuy

I investigated this for a bit and was not able to identify the problem immediately based upon the back trace.  If you could, and haven't already, this should be documented on our bug tracker issues.asterisk.org.  Thanks for reporting this!

~Vossel