[asterisk-bugs] [Asterisk 0016811]: Heavy locking in manager.c results in eventual crash and loss of CLI commands and intense CPU load

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
https://issues.asterisk.org/view.php?id=16811 
====================================================================== 
Reported By:                aragon
Assigned To:                qwell
====================================================================== 
Project:                    Asterisk
Issue ID:                   16811
Category:                   Core/ManagerInterface
Reproducibility:            always
Severity:                   crash
Priority:                   normal
Status:                     feedback
Target Version:             1.4.38
Asterisk Version:           SVN 
JIRA:                       SWP-899 
Regression:                 No 
Reviewboard Link:            
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2010-02-11 08:29 CST
Last Modified:              2010-09-28 17:04 CDT
====================================================================== 
Summary:                    Heavy locking in manager.c results in eventual crash
and loss of CLI commands and intense CPU load
Description: 
We run some scripts connecting to Asterisk AMI to display realtime Asterisk
statistics.  When our AMI stuff is connected to Asterisk versions 1.4.25 to
1.4.30rc2 there is heavy duty locking and CPU load spiking which will
degrade voice quality, kill the Asterisk CLI commands like core show locks,
core show channels etc... and I have seen the load in HTOP reach 300 until
Asterisk is forcibly restarted. Asterisk 1.4.24 is not effected so we don't
think it is our scripts...

Core show locks and gdb asterisk --pid=`pidof asterisk` 'thread apply all
bt' is attached as txt files
======================================================================
Relationships       ID      Summary
----------------------------------------------------------------------
related to          0016754 [patch] Multiple segfaults in leave_voi...
====================================================================== 

---------------------------------------------------------------------- 
 (0127487) aragon (reporter) - 2010-09-28 17:04
 https://issues.asterisk.org/view.php?id=16811#c127487 
---------------------------------------------------------------------- 
On each call would call up AGI command but AGI takes a long time to start
and delays call processing.  The long delay would cause locks to be
excessive and the more calls processed the more the CPU would spike causing
even longer processing delays of calls further causing longer locks.
I don't know why Leif related this issue to a crash in voicemail...

The fix was to replace agi with fastagi to remove the delay caused by agi
startup times.

AMI would monitor things like agent status, extension busy status, etc...

I'd like to help you fix this too but I couldn't wait for a developer to
get around to it and now that this is fixed with fastagi I cannot reliably
reproduce the problem.  Therefore feel free to use any existing debug info
attached to this report to fix the bug on your own or close the ticket. 
But I do think this issue is closely related to
https://issues.asterisk.org/view.php?id=16081 which I can reliably
reproduce.
Both issues are related to slow agi startup times since I have resolved
both issues by dumping agi in favor of fastagi 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2010-09-28 17:04 aragon         Note Added: 0127487                          
======================================================================