[asterisk-bugs] [Asterisk 0014918]: Asterisk 1.4.24.1 deadlock

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=14918 
====================================================================== 
Reported By:                aragon
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   14918
Category:                   Channels/chan_local
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     new
Target Version:             1.4.25
Asterisk Version:           1.4.24 
Regression:                 Yes 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-04-16 16:12 CDT
Last Modified:              2009-04-27 14:11 CDT
====================================================================== 
Summary:                    Asterisk 1.4.24.1 deadlock
Description: 
Asterisk locks randomly

This is the first lock on this system since upgrading to 1.4.24.1
yesterday

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <file> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 3086154656 (do_devstate_changes  started at [  363]
devicestate.c ast_device_state_engine_init())
=== ---> Tried and failed to get Lock http://bugs.digium.com/view.php?id=0
(channel.c): MUTEX 1101
channel_find_locked &c->lock 0x9469908 (0)
=== -------------------------------------------------------------------
===
=== Thread ID: 19549088 (do_monitor started at [16693] chan_sip.c
restart_monitor())
=== ---> Lock http://bugs.digium.com/view.php?id=0 (chan_sip.c): MUTEX 16545
do_monitor &iflock 0x4eb92e0
(1)
=== ---> Waiting for Lock http://bugs.digium.com/view.php?id=1 (pbx.c): MUTEX
2158 ast_extension_state_del
&(&hints)->lock 0x8166948 (1)
=== --- ---> Locked Here: pbx.c line 2194 (ast_add_hint)
=== -------------------------------------------------------------------
===
=== Thread ID: 30337952 (pbx_thread started at [ 2663] pbx.c
ast_pbx_start())
=== ---> Lock http://bugs.digium.com/view.php?id=0 (channel.c): MUTEX 2751
ast_write &chan->lock 0x9469908
(1)
=== ---> Lock http://bugs.digium.com/view.php?id=1 (chan_local.c): MUTEX 320
local_write &p->lock 0x9476338
(1)
=== -------------------------------------------------------------------
===
=== Thread ID: 30829472 (session_do started at [ 2475] manager.c
accept_thread())
=== ---> Lock http://bugs.digium.com/view.php?id=0 (manager.c): RDLOCK 2235
process_message &actionlock
0x8164460 (1)
=== ---> Lock http://bugs.digium.com/view.php?id=1 (pbx.c): MUTEX 4850
ast_add_extension2 &con->lock
0x95a6218 (1)
=== ---> Lock http://bugs.digium.com/view.php?id=2 (pbx.c): MUTEX 2194
ast_add_hint &(&hints)->lock
0x8166948 (1)
=== ---> Tried and failed to get Lock http://bugs.digium.com/view.php?id=3
(channel.c): MUTEX 1101
channel_find_locked &c->lock 0x9469908 (0)
=== -------------------------------------------------------------------
===
=== Thread ID: 25209760 (pbx_thread started at [ 2663] pbx.c
ast_pbx_start())
=== ---> Waiting for Lock http://bugs.digium.com/view.php?id=0 (chan_sip.c):
MUTEX 4648 sip_alloc &iflock
0x4eb92e0 (1)
=== --- ---> Locked Here: chan_sip.c line 16545 (do_monitor)
=== -------------------------------------------------------------------
===
=== Thread ID: 64269216 (session_do started at [ 2475] manager.c
accept_thread())
=== ---> Lock http://bugs.digium.com/view.php?id=0 (manager.c): RDLOCK 2235
process_message &actionlock
0x8164460 (1)
=== ---> Tried and failed to get Lock http://bugs.digium.com/view.php?id=1
(channel.c): MUTEX 1101
channel_find_locked &c->lock 0x9469908 (0)
=== -------------------------------------------------------------------
===
=======================================================================


====================================================================== 

---------------------------------------------------------------------- 
 (0103834) zerohalo (reporter) - 2009-04-27 14:11
 http://bugs.digium.com/view.php?id=14918#c103834 
---------------------------------------------------------------------- 
I think I may have found the source of the problem with our situation. It
appears that we have been doing sip reloads daily from cron (when peers are
100% realtime). Monitoring mysql qps pointed me here, graph attached. When
it reached critical mass, perhaps mysql couldn't reply in a timely manner,
hence the dns lookup of the peername. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-04-27 14:11 zerohalo       Note Added: 0103834                          
======================================================================