[asterisk-bugs] [Asterisk 0011790]: iax2 deadlock?

[noreply at bugs.digium.com]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=11790 
====================================================================== 
Reported By:                mostyn
Assigned To:                russell
====================================================================== 
Project:                    Asterisk
Issue ID:                   11790
Category:                   Channels/chan_iax2
Reproducibility:            always
Severity:                   major
Priority:                   normal
Status:                     feedback
Asterisk Version:           1.4.17 
SVN Branch (only for SVN checkouts, not tarball releases): N/A  
SVN Revision (number only!):  
Disclaimer on File?:        N/A 
Request Review:              
====================================================================== 
Date Submitted:             01-17-2008 19:23 CST
Last Modified:              01-19-2008 18:01 CST
====================================================================== 
Summary:                    iax2 deadlock?
Description: 
We have an asterisk 1.4.17 machine that appears to trigger an IAX deadlock
several times per day.  We have recently increased the number of clients
using the machine, the issue did not appear when the load was lower.  At
the moment have about 160 IAX peer/user pairs (of which about 130 are
online) and 1000 SIP peer/user pairs (of which about 575 are online).  We
are attempting to migrate from an asterisk 1.2 installation, which handled
this load OK.  

When the issue occurs, "iax2 show peers" gradually shows more and more
peers as UNKNOWN.  At the same time, SIP user/peers also become more
lagged.  CPU load jumps from http://bugs.digium.com/view.php?id=14#20% to above
100% (SMP machine). The asterisk
console had a message saying the following: 

NOTICE[10020]: chan_iax2.c:6699 socket_read: Out of idle IAX2 threads for
I/O, pausing!

If we issue an "iax2 reload", it does not seem to help and "iax2 show
peers" does not show any output at that point.  We have to restart asterisk
to get going again.  

We attempted to increase iaxthreadcount and iaxmaxthreadcount in iax.conf,
but they appear to max out at 256 even if the iax.conf settings are higher.
 

Based on suggestions by russellb on the irc channel, we changed
MAX_PEER_BUCKET from 1 to 563 in chan_iax2.c and recompiled, with thread
debugging enabled.  We are still experiencing the issue.  I will attach the
output of "core show locks" from when the issue is occuring.  

CPU: dual intel xeon 5160 3GHz
OS: debian 4.0r2 amd64
RAM: 4G
====================================================================== 

---------------------------------------------------------------------- 
 mostyn - 01-19-08 18:01  
---------------------------------------------------------------------- 
I rolled back to asterisk 1.2 on an different machine yesterday, so the
asterisk 1.4.17 machine is free for stress testing.  However, I'm not sure
how to generate enough test clients/calls to reproduce the error, do you
know of any good IAX call generators, or should I just setup another
asterisk machine? 

Issue History 
Date Modified   Username       Field                    Change               
====================================================================== 
01-19-08 18:01  mostyn         Note Added: 0080908                          
======================================================================