[asterisk-bugs] [Asterisk 0014625]: Random deadlocks leading to lockup on 1.4.23.1

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=14625 
====================================================================== 
Reported By:                acunningham
Assigned To:                tilghman
====================================================================== 
Project:                    Asterisk
Issue ID:                   14625
Category:                   General
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     acknowledged
Asterisk Version:           1.4.23 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-03-08 19:36 CDT
Last Modified:              2009-03-25 19:22 CDT
====================================================================== 
Summary:                    Random deadlocks leading to lockup on 1.4.23.1
Description: 
Random channels seem to be giving deadlocks on a 1.4.23.1 system:

# tail -f /var/log/asterisk/full | grep -i deadlock
[Mar  8 20:35:37] DEBUG[4249] channel.c: Avoiding initial deadlock for
channel '0x8779688'
[Mar  8 20:35:37] DEBUG[4249] channel.c: Avoiding initial deadlock for
channel '0x87ff698'
[Mar  8 20:35:42] DEBUG[4249] channel.c: Avoiding initial deadlock for
channel '0x8314468'
[Mar  8 20:35:46] DEBUG[4249] channel.c: Avoiding initial deadlock for
channel '0x861ddd0'


I've also seen "Avoiding deadlock" as well as "Avoiding initial deadlock",
though these are less common. The system is continually busy at medium load
(tens of calls). Over a period of days, the problem gets worse and worse
until Asterisk stops handling calls and needs to be killed with "kill -9".
Compiling with DEBUG_THREADS and running "core show locks" always shows:

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <file> <line num> <function> <lock name> <lock addr> (times locked)
===
=======================================================================

with no further output. I've tried the patch in ticket
http://bugs.digium.com/view.php?id=13116 but this
doesn't help.

I'm not sure what debugs would be useful, but if someone can advise what
would be useful I'm happy to take them.
====================================================================== 

---------------------------------------------------------------------- 
 (0102208) acunningham (reporter) - 2009-03-25 19:22
 http://bugs.digium.com/view.php?id=14625#c102208 
---------------------------------------------------------------------- 
The problem has occurred again, and looks more like deadlocks than file
descriptors.

The CPU load on the machine was rising over time, and when I loaded the
Asterisk console and ran "core show locks" it produced no output (not even
the headers). I ran it again and the Asterisk console locked up. Loading a
new shell and running "asterisk -rx 'core show locks'" locked up as well.
It took a "kill -9" on the main Asterisk process to free them. Attached in
the next minute or so will be gdb4.txt from just before killing Asterisk.

We do regularly connect to the manager interface. Perhaps the file
descriptors problem is caused by the deadlocks problem? My theory (and it
is just a theory) is that when a process connects to the manager interface
it hangs due to the deadlocks problem. This then consumes a file
descriptor. A minute later another connection is made, consuming another
file descriptor, and so on.

I'll try your DEBUG_FD_LEAKS patch after hours tonight. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-03-25 19:22 acunningham    Note Added: 0102208                          
======================================================================