[asterisk-bugs] [Asterisk 0014112]: Thread deadlock causes Asterisk to stop routing calls to agents, agents unable to change status

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=14112 
====================================================================== 
Reported By:                sroberts
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   14112
Category:                   Channels/chan_agent
Reproducibility:            random
Severity:                   major
Priority:                   normal
Status:                     feedback
Asterisk Version:           1.4.19 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2008-12-19 07:16 CST
Last Modified:              2009-02-25 01:43 CST
====================================================================== 
Summary:                    Thread deadlock causes Asterisk to stop routing
calls to agents, agents unable to change status
Description: 
We run several call centres on Asterisk. Our queue servers are using
Asterisk 1.4.19 on CentOS 4.6. The busier sites can take anywhere between
5000 and 10000 calls per day.

We are experiencing a problem whereby occasionally Asterisk will stop
routing calls to agents. If one opens the console and issues a "show
channels" command the channel list does not finish displaying, it only
shows a portion of the channels. The console then becomes unresponsive.
Calls then continue to build up in the queue, and only restarting Asterisk
fixes it.

I have attached a dump of "core show locks". I could not get any further
info this time, as Asterisk must generally be restarted immediately when
this happens to keep the call centre operating. If you look for Thread id
3006892960 you will see that this thread has locked the list of agents
while a whole host of other threads are waiting for this mutex to be
released. I ran "core show locks" 4 times about 2 seconds apart before
restarting and in each case that lock was still being held with many other
threads waiting for it.

The lock in question is in agent_logoff in chan_agent.c. It seems that the
list of agent channels is locked, it then matches the agent channel it is
looking for and then falls into one of the following two loops:

while (p->owner && ast_channel_trylock(p->owner)) {
    ast_mutex_unlock(&p->lock);
    usleep(1);
    ast_mutex_lock(&p->lock);
}

or

while (p->chan && ast_channel_trylock(p->chan)) {
    ast_mutex_unlock(&p->lock);
    usleep(1);
    ast_mutex_lock(&p->lock);
}


It seems as though it is unable to obtain the lock for p->owner or
p->chan, so it keeps looping trying to get the lock. The problem is that
the number of threads waiting for the agent list to be unlocked starts
growing and growing as more and more calls pour into the queue. Obviously
because the list of agents is locked, calls can't be routed. 

This problem seems to happen every 2 to 3 weeks and generally occurs only
when the box is receiving a lot of calls.

Any suggestions/advice would be greatly appreciated.




====================================================================== 

---------------------------------------------------------------------- 
 (0100688) sroberts (reporter) - 2009-02-25 01:43
 http://bugs.digium.com/view.php?id=14112#c100688 
---------------------------------------------------------------------- 
Hi, 

So I managed to get the go-ahead and upgraded to 1.4.22. Unfortunately, it
seems that there is now a lock occurring elsewhere.

If you look at threads 3085781920 and 3008134048, it seems to me that an
agent channel is being hung up, yet the hangup doesn't complete as it can't
get hold of a lock. It then also locks up in agent_devicestate_cb, which
locks the agent list.

Now we use call files to log agents in and out and to pause/unpause them.
So what happens when one of these deadlocks occurs is that agents start
trying to log off and log in again, which causes even more threads to wait
on the agent list. I've attach two more core locks, taken about 30 seconds
apart as well as a gdb backtrace of the threads.

Unfortunately, I can't get authorization to compile Asterisk with
optimizations off as my superiors are worried about performance
implications. What, if any are the performance implications of compiling
with optimizations off? I'd like to do this, however without some
motivation as to why it won't affect performance I can't do this. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-02-25 01:43 sroberts       Note Added: 0100688                          
======================================================================