[asterisk-bugs] [Asterisk 0016321]: asterisk deadlocks and calls will stop queueing.

[Asterisk Bug Tracker]

The following issue has been UPDATED. 
====================================================================== 
https://issues.asterisk.org/view.php?id=16321 
====================================================================== 
Reported By:                valon24
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   16321
Category:                   Channels/chan_agent
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     new
Asterisk Version:           SVN 
JIRA:                        
Regression:                 Yes 
Reviewboard Link:            
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-11-25 11:31 CST
Last Modified:              2009-12-01 12:43 CST
====================================================================== 
Summary:                    asterisk deadlocks and calls will stop queueing.
Description: 
Our system will go fine for days processing calls queueing them and then
deliver the calls to agents, we will then see the problem at random, it
will happen sometimes every 3 or 4 days and then sometimes 3 times in one
day.

The problem will force us to restart asterisk.

A call will come into the inbound route and then the queue application is
called.
after the queue application is call there is just dead air and nothing
more will happen with that call. Once it happens no other calls will be
processed they will just keep hitting the queue application and go to
silence.

Normally in the CLI we will see something like:

[Nov 23 14:35:18] VERBOSE[4930] logger.c:     -- Executing
[909 at from-internal:18] Queue("SIP/192.168.6.132-c8170bc0",
"909|t|1712|1712") in new stack

[Nov 23 14:35:18] VERBOSE[4930] logger.c:     -- Started music on hold,
class 'acc_1', on SIP/192.168.6.132-c8170bc0


After the system deadlocks we will only see:


[Nov 23 14:36:14] VERBOSE[5009] logger.c:     -- Executing
[903 at from-internal:18] Queue("SIP/192.168.6.132-c8114300",
"903|t|1709|1709") in new stack

I have will attach the output of core show locks and so everyone can see
the full log but this is normally what we see each time we see the issue

This happened 3 times yesterday:

=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <file> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 1097607488 (do_devstate_changes  started at [  363]
devicestate.c ast_device_state_engine_init())
=== ---> Lock https://issues.asterisk.org/view.php?id=0 (devicestate.c): MUTEX
289 do_state_change
&(&devstate_cbs)->lock 0x75e390 (1)
=== ---> Lock https://issues.asterisk.org/view.php?id=1 (chan_agent.c): MUTEX
304 agent_devicestate_cb
&(&agents)->lock 0x2aaaba9a24f0 (1)
=== ---> Waiting for Lock https://issues.asterisk.org/view.php?id=2
(chan_agent.c): MUTEX 313
agent_devicestate_cb &p->lock 0x2aaaaceecd80 (1)
=== --- ---> Locked Here: chan_agent.c line 498 (agent_read)
=== -------------------------------------------------------------------
===
=== Thread ID: 1106180416 (session_do           started at [ 2522]
manager.c accept_thread())
=== ---> Lock https://issues.asterisk.org/view.php?id=0 (manager.c): RDLOCK 2281
process_message &actionlock
0x76b720 (1)
=== ---> Waiting for Lock https://issues.asterisk.org/view.php?id=1
(app_queue.c): MUTEX 4781
manager_queues_status &(&queues)->lock 0x2aaab6129bd0 (1)
=== --- ---> Locked Here: app_queue.c line 2842 (try_calling)
=== -------------------------------------------------------------------
===
=== Thread ID: 1112938816 (pbx_thread           started at [ 2623] pbx.c
ast_pbx_start())
=== ---> Lock https://issues.asterisk.org/view.php?id=0 (channel.c): MUTEX 2061
__ast_read &chan->lock 0x2768a68
(1)
=== ---> Lock https://issues.asterisk.org/view.php?id=1 (chan_agent.c): MUTEX
498 agent_read &p->lock
0x2aaaaceecd80 (1)
=== ---> Waiting for Lock https://issues.asterisk.org/view.php?id=2
(chan_agent.c): MUTEX 1669 agent_logoff
&(&agents)->lock 0x2aaaba9a24f0 (1)
=== --- ---> Locked Here: chan_agent.c line 304 (agent_devicestate_cb)
=== -------------------------------------------------------------------
===
=== Thread ID: 1129191744 (pbx_thread           started at [ 2623] pbx.c
ast_pbx_start())
=== ---> Lock https://issues.asterisk.org/view.php?id=0 (app_queue.c): MUTEX
2842 try_calling &(&queues)->lock
0x2aaab6129bd0 (1)
=== ---> Waiting for Lock https://issues.asterisk.org/view.php?id=1
(chan_agent.c): MUTEX 1456 agent_request
&(&agents)->lock 0x2aaaba9a24f0 (1)
=== --- ---> Locked Here: chan_agent.c line 304 (agent_devicestate_cb)
=== -------------------------------------------------------------------






====================================================================== 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-12-01 12:43 lmadsen        Asterisk Version         1.4.27 => SVN       
2009-12-01 12:43 lmadsen        Regression               No => Yes           
2009-12-01 12:43 lmadsen        Description Updated                          
======================================================================