[asterisk-bugs] [Asterisk 0015997]: segfault in 1.6.1.6 in _ao2_find, called from chan_iax2 after approx. 75.000 calls

Mon Oct 5 05:08:20 CDT 2009

A NOTE has been added to this issue. 
====================================================================== 
https://issues.asterisk.org/view.php?id=15997 
====================================================================== 
Reported By:                exarv
Assigned To:                russell
====================================================================== 
Project:                    Asterisk
Issue ID:                   15997
Category:                   Channels/chan_iax2
Reproducibility:            have not tried
Severity:                   crash
Priority:                   normal
Status:                     assigned
Asterisk Version:           1.6.1.6 
JIRA:                        
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-10-01 02:37 CDT
Last Modified:              2009-10-05 05:08 CDT
====================================================================== 
Summary:                    segfault in 1.6.1.6 in _ao2_find, called from
chan_iax2 after approx. 75.000 calls
Description: 
I'm running Asterisk 1.6.1.6 since 19 days now,
but I've had 3 times a segfault on the same address.

Sep 17 13:03:57 switch02 kernel: asterisk[13597]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 00000000420b1440 error 4
     In the mean time 75.517 calls were setup.
Sep 22 19:07:13 switch02 kernel: asterisk[23982]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 0000000042142440 error 4
     In the mean time  76.725 calls were setup.
Sep 28 14:26:16 switch02 kernel: asterisk[2777]: segfault at 
00002aaa0000000a rip 0000000000435c36 rsp 0000000042088440 error 4

It's a live server running production traffic. So I don't have much 
possibilities to easily test a different version.
Also the issue only happens to me once a week (about once every approx. 
75.000 calls).
The segfault didn't happen on the most busiest times (sunday), but just 
on the more quiet days.

The last time the server had 50 calls, 97 channels (43 chan_ss7 
channels, 46 sip channels and 8 iax2 channels)

Software running:
  - CentOS 5 (latest updates as of 11 sept 2009)
  - asterisk 1.6.1.6
  - chan_ss7 1.2.1
  - dahdi-linux 2.2.0.2
  - dahdi-tools 2.2.0
  - wanpipe 3.5.6

====================================================================== 

---------------------------------------------------------------------- 
 (0111835) exarv (reporter) - 2009-10-05 05:08
 https://issues.asterisk.org/view.php?id=15997#c111835 
---------------------------------------------------------------------- 
Just had another crash:

Oct  5 11:19:20 switch02 kernel: asterisk[18505]: segfault at
00002aaa0000000a rip 0000000000435c36 rsp 0000000042388440 error 4
97660 calls and 6 days, 21 hours after the last crash

gdb:
Core was generated by `/usr/sbin/asterisk -f -vvvg -c'.
Program terminated with signal 11, Segmentation fault.
https://issues.asterisk.org/view.php?id=0  _ao2_find (c=0x2aaa00000002,
arg=0x423884d0, flags=OBJ_POINTER) at
astobj2.c:712
712		return __ao2_callback(c,flags, cb_fn, arg, NULL, NULL, 0, NULL);

(gdb) bt
https://issues.asterisk.org/view.php?id=0  _ao2_find (c=0x2aaa00000002,
arg=0x423884d0, flags=OBJ_POINTER) at
astobj2.c:712
https://issues.asterisk.org/view.php?id=1  0x00002aaac090e2ae in __find_callno
(callno=1, dcallno=17150,
sin=0x4238ce80, new=0, sockfd=14, return_locked=0, check_dcallno=1) at
chan_iax2.c:2450
https://issues.asterisk.org/view.php?id=2  0x00002aaac091b570 in socket_process
(thread=0x1a07940) at
chan_iax2.c:2582
https://issues.asterisk.org/view.php?id=3  0x00002aaac0926459 in
iax2_process_thread (data=0x1a07940) at
chan_iax2.c:10933
https://issues.asterisk.org/view.php?id=4  0x00000000004f28ac in dummy_start
(data=<value optimized out>) at
utils.c:968
https://issues.asterisk.org/view.php?id=5  0x0000003b07a06367 in start_thread ()
from /lib64/libpthread.so.0
https://issues.asterisk.org/view.php?id=6  0x0000003b06ed309d in clone () from
/lib64/libc.so.6

(gdb) print iax_transfercallno_pvts
$1 = (struct ao2_container *) 0x2aaa00000002

(gdb) ptype iax_transfercallno_pvts
type = struct ao2_container {
    ao2_hash_fn *hash_fn;
    ao2_callback_fn *cmp_fn;
    int n_buckets;
    int elements;
    int version;
    struct bucket buckets[0];
} *

(gdb) print iax_transfercallno_pvts.elements
Cannot access memory at address 0x2aaa00000016 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-10-05 05:08 exarv          Note Added: 0111835                          
======================================================================