[asterisk-bugs] [Asterisk 0011790]: iax2 deadlock?

[noreply at bugs.digium.com]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=11790 
====================================================================== 
Reported By:                mostyn
Assigned To:                russell
====================================================================== 
Project:                    Asterisk
Issue ID:                   11790
Category:                   Channels/chan_iax2
Reproducibility:            always
Severity:                   major
Priority:                   normal
Status:                     assigned
Asterisk Version:           1.4.17 
SVN Branch (only for SVN checkouts, not tarball releases): N/A  
SVN Revision (number only!):  
Disclaimer on File?:        N/A 
Request Review:              
====================================================================== 
Date Submitted:             01-17-2008 19:23 CST
Last Modified:              01-19-2008 15:49 CST
====================================================================== 
Summary:                    iax2 deadlock?
Description: 
We have an asterisk 1.4.17 machine that appears to trigger an IAX deadlock
several times per day.  We have recently increased the number of clients
using the machine, the issue did not appear when the load was lower.  At
the moment have about 160 IAX peer/user pairs (of which about 130 are
online) and 1000 SIP peer/user pairs (of which about 575 are online).  We
are attempting to migrate from an asterisk 1.2 installation, which handled
this load OK.  

When the issue occurs, "iax2 show peers" gradually shows more and more
peers as UNKNOWN.  At the same time, SIP user/peers also become more
lagged.  CPU load jumps from http://bugs.digium.com/view.php?id=14#20% to above
100% (SMP machine). The asterisk
console had a message saying the following: 

NOTICE[10020]: chan_iax2.c:6699 socket_read: Out of idle IAX2 threads for
I/O, pausing!

If we issue an "iax2 reload", it does not seem to help and "iax2 show
peers" does not show any output at that point.  We have to restart asterisk
to get going again.  

We attempted to increase iaxthreadcount and iaxmaxthreadcount in iax.conf,
but they appear to max out at 256 even if the iax.conf settings are higher.
 

Based on suggestions by russellb on the irc channel, we changed
MAX_PEER_BUCKET from 1 to 563 in chan_iax2.c and recompiled, with thread
debugging enabled.  We are still experiencing the issue.  I will attach the
output of "core show locks" from when the issue is occuring.  

CPU: dual intel xeon 5160 3GHz
OS: debian 4.0r2 amd64
RAM: 4G
====================================================================== 

---------------------------------------------------------------------- 
 russell - 01-19-08 15:49  
---------------------------------------------------------------------- 
In an email, you indicated that you had moved your production systems to
Asterisk 1.2.  Please let me know if you are able to reproduce this error
on a test system.  It may be helpful for me to log in and look at the
situation with gdb.

Another thing you can do to provide potentially useful debug information
is to run Asterisk under valgrind.  Only do this in a test environment,
though, as valgrind slows things down a lot.

See doc/valgrind.txt for more information. 

Issue History 
Date Modified   Username       Field                    Change               
====================================================================== 
01-19-08 15:49  russell        Note Added: 0080905                          
======================================================================