[asterisk-bugs] [Asterisk 0016811]: Heavy locking in manager.c results in eventual crash and loss of CLI commands and intense CPU load

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
https://issues.asterisk.org/view.php?id=16811 
====================================================================== 
Reported By:                aragon
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   16811
Category:                   Core/ManagerInterface
Reproducibility:            always
Severity:                   crash
Priority:                   normal
Status:                     acknowledged
Asterisk Version:           1.4.30-rc2 
JIRA:                       SWP-899 
Regression:                 No 
Reviewboard Link:            
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2010-02-11 08:29 CST
Last Modified:              2010-03-10 15:24 CST
====================================================================== 
Summary:                    Heavy locking in manager.c results in eventual crash
and loss of CLI commands and intense CPU load
Description: 
We run some scripts connecting to Asterisk AMI to display realtime Asterisk
statistics.  When our AMI stuff is connected to Asterisk versions 1.4.25 to
1.4.30rc2 there is heavy duty locking and CPU load spiking which will
degrade voice quality, kill the Asterisk CLI commands like core show locks,
core show channels etc... and I have seen the load in HTOP reach 300 until
Asterisk is forcibly restarted. Asterisk 1.4.24 is not effected so we don't
think it is our scripts...

Core show locks and gdb asterisk --pid=`pidof asterisk` 'thread apply all
bt' is attached as txt files
======================================================================
Relationships       ID      Summary
----------------------------------------------------------------------
related to          0016754 [patch] Multiple segfaults in leave_voi...
====================================================================== 

---------------------------------------------------------------------- 
 (0119242) aragon (reporter) - 2010-03-10 15:24
 https://issues.asterisk.org/view.php?id=16811#c119242 
---------------------------------------------------------------------- 
It happened again now using 1.4.30rc3
CPU load so high it took minutes to login to Asterisk with SSH.
After some patience I was able to attach with gdb asterisk -pid=
Uploading 10032010gdbasterisk-pid_threadapplyallbt.txt although I don't
know if this will be helpful because Asterisk does not deadlock completely.
 Everything just times out because the CPU is so overloaded by what appears
to be a very long lock which spikes the CPU.
After I disconnected gdb the CPU load went back to normal and core show
locks did not show any lock.

Peers are monitored with qualify=yes so when CPU maxes out I lose all my
peers. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2010-03-10 15:24 aragon         Note Added: 0119242                          
======================================================================