[asterisk-bugs] [Asterisk 0018629]: Possible deadlock on 1.6.2.12, 14 and 15

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
https://issues.asterisk.org/view.php?id=18629 
====================================================================== 
Reported By:                justintonation
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   18629
Category:                   Core/General
Reproducibility:            random
Severity:                   crash
Priority:                   normal
Status:                     feedback
Asterisk Version:           1.6.2.15 
JIRA:                        
Regression:                 No 
Reviewboard Link:            
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2011-01-17 01:22 CST
Last Modified:              2011-01-22 05:01 CST
====================================================================== 
Summary:                    Possible deadlock on 1.6.2.12, 14 and 15
Description: 
After hours to days the system appears to reach a deadlocked state.  SIP
phones lose registration, in-flight calls continue, no new calls can be
established.  Need a "service asterisk restart" to get going again.

Problem is usually first observed when phones lose registration (120
seconds between registrations).

System has never been observed to reach this state overnight or on a
weekend so it appears to only happen when the system is actively handling
calls.


====================================================================== 

---------------------------------------------------------------------- 
 (0130894) justintonation (reporter) - 2011-01-22 05:01
 https://issues.asterisk.org/view.php?id=18629#c130894 
---------------------------------------------------------------------- 
I'm a little reluctant to add this info as I'm not sure how well it relates
to what has gone before.  But FWIW...

I finally had a chance to lock the users out of the building for a few
hours and I finished compiling and configuring a system on another server
from source.  Release info as above with Asterisk 1.6.2.15.  I built with
DONT-OPTIMIZE and DEBUG-THREADS.

The system started intermittently dropping SIP registrations immediately
(with no calls) and when I did a few incoming calls to a queue of four
extensions the system locked up almost immediately.

I took the required dumps BUT I am by no means certain that they relate to
the same issue as originally reported.

I recompiled without DEBUG-THREADS and as far as I was able to tell with
some simple tests the system worked normally at that point.  Of course
previously the deadlock was never observed until we had a normal workload
on the system.

By way of an experiment I tried building 1.6.2.8 with both DONT-OPTIMIZE
and DEBUG-THREADS and also had sip registration issues (though no lockup). 
Build with just DONT-OPTIMIZE everything again appeared to work normally. 
I ONLY rebuilt Asterisk in these experiments, not DAHDI and libpri, so I
got a warning about possibly incompatible modules - all seemed to work
despite that.

I did not formally monitor resource use but when I ran top a few times I
never saw more than 5% CPU busy in the DONT-OPTIMIZE configurations and
about 10% in the DONT-OPTIMIZE and DEBUG-THREADS configuration.

So the upshot is I've uploaded the files but have no idea if they are
related to the original assumed deadlock.

I hope to run some more tests before people get back to work on Monday and
am still mulling over what configuration to leave in production for next
week (if I change it at all).  The frustrating thing is that a
DEBUG-THREADS compile seems to be unusable in our config for whatever
reasons, so I probably will never get a backtrace of a real life occurrence
of the problem. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2011-01-22 05:01 justintonation Note Added: 0130894                          
======================================================================