[asterisk-bugs] [Asterisk 0014112]: Thread deadlock causes Asterisk to stop routing calls to agents, agents unable to change status

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=14112 
====================================================================== 
Reported By:                sroberts
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   14112
Category:                   Channels/chan_agent
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     feedback
Asterisk Version:           1.4.19 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2008-12-19 07:16 CST
Last Modified:              2009-03-02 15:49 CST
====================================================================== 
Summary:                    Thread deadlock causes Asterisk to stop routing
calls to agents, agents unable to change status
Description: 
We run several call centres on Asterisk. Our queue servers are using
Asterisk 1.4.19 on CentOS 4.6. The busier sites can take anywhere between
5000 and 10000 calls per day.

We are experiencing a problem whereby occasionally Asterisk will stop
routing calls to agents. If one opens the console and issues a "show
channels" command the channel list does not finish displaying, it only
shows a portion of the channels. The console then becomes unresponsive.
Calls then continue to build up in the queue, and only restarting Asterisk
fixes it.

I have attached a dump of "core show locks". I could not get any further
info this time, as Asterisk must generally be restarted immediately when
this happens to keep the call centre operating. If you look for Thread id
3006892960 you will see that this thread has locked the list of agents
while a whole host of other threads are waiting for this mutex to be
released. I ran "core show locks" 4 times about 2 seconds apart before
restarting and in each case that lock was still being held with many other
threads waiting for it.

The lock in question is in agent_logoff in chan_agent.c. It seems that the
list of agent channels is locked, it then matches the agent channel it is
looking for and then falls into one of the following two loops:

while (p->owner && ast_channel_trylock(p->owner)) {
    ast_mutex_unlock(&p->lock);
    usleep(1);
    ast_mutex_lock(&p->lock);
}

or

while (p->chan && ast_channel_trylock(p->chan)) {
    ast_mutex_unlock(&p->lock);
    usleep(1);
    ast_mutex_lock(&p->lock);
}


It seems as though it is unable to obtain the lock for p->owner or
p->chan, so it keeps looping trying to get the lock. The problem is that
the number of threads waiting for the agent list to be unlocked starts
growing and growing as more and more calls pour into the queue. Obviously
because the list of agents is locked, calls can't be routed. 

This problem seems to happen every 2 to 3 weeks and generally occurs only
when the box is receiving a lot of calls.

Any suggestions/advice would be greatly appreciated.




====================================================================== 

---------------------------------------------------------------------- 
 (0101028) sroberts (reporter) - 2009-03-02 15:49
 http://bugs.digium.com/view.php?id=14112#c101028 
---------------------------------------------------------------------- 
Hi, the lockup occurred again today, and on Friday. I managed to get a gdb
backtrace and have that Asterisk compiled with optimizations off.
Unfortunately, though I didn't get all the threads, as there were more
lines than I had console window. I didn't get the chance to do it again, as
there were already 150 calls stuck in the queue and more pouring in. If it
happens again I'll get the full trace. 

That being said, it's exactly the same lockup now. Always
agent_devicestate_cb waiting for a lock which is held by ast_hangup. While
this is waiting, it holds the lock for the list of channels, which prevents
calls from being routed to agents.

If you look at gdb_threads_2 it looks like thread 182 is the culprit here.
In ast_hangup there is a ast_channel_lock(chan); statement. Now in
__ast_pthread_mutex_lock(), there is a do/while loop that seemingly will
wait until the lock is acquired. Now I may be on the complete wrong track
here, but is it not possible that two threads are trying to update the
agent channel and hang it up at the same time? 

I see that agent_devicestate_cb is only called when chan_agent is
loaded/unloaded. Now we make changes to the agents during the day and call
a reload of chan_agent.so and app_queue.so. Could it be that triggering
this reload while an agent channel is being hung up could cause this lock? 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-03-02 15:49 sroberts       Note Added: 0101028                          
======================================================================