[asterisk-dev] [Code Review] [patch] Use of MASTER_CHANNEL causes a race condition ending in a deadlock among channels

[Russell Bryant]

----- Original Message -----
> On 110301 0812, Russell Bryant wrote:
> > Yes, this is part of the deadlock we are seeing as well. The thread
> > you
> > are pointing to is blocking waiting on a channel lock.
> >
> >> === --- ---> Locked Here: channel.c line 3636 (__ast_read)
> >
> > It says it's locked in a thread that called ast_read(). What is that
> > thread blocking on, though?
> 
> The thread is a pbx_thread in WaitForSilence application. Why it would
> go into that while holding a channel lock, is beyond my understanding.
> 
> > By the way, which issue are you referring to? #28643 isn't a valid
> > mantis issue number.
> 
> Russel, I am sorry, I messed up the ticket number. It's #18823.

I took a look at the backtrace on the issue.  This is the exact same situation we were looking at.

The pbx thread does not have the channel locked going into the waitforsilence application.  It is locking it in ast_read().  It's stuck in res_timing_timerfd.  This seems to be happening often enough that is seriously concerning.  There is something wrong with either res_timing_timerfd (or the underlying timerfd API that it uses).  This is not going to be an easy issue to figure out.  If we had a test environment that reliably reproduced it, that would be a great start.

For you or anyone else that hits this same issue, the only workaround right now is to install DAHDI and use the res_timing_dahdi module instead of res_timing_timerfd.

Thread 30 (process 25423):
#0  0x00007f7ddbc3af7b in read () from /lib/libc.so.6
#1  0x00007f7dcb5f1288 in timerfd_timer_ack (handle=50, quantity=1) at res_timing_timerfd.c:166
#2  0x000000000055ec11 in ast_timer_ack (handle=0x11c7220, quantity=1) at timing.c:169
#3  0x0000000000473351 in __ast_read (chan=0xe46558, dropaudio=0) at channel.c:3706
#4  0x000000000047568a in ast_read (chan=0xe46558) at channel.c:4151

--
Russell Bryant
Digium, Inc.  |  Engineering Manager, Open Source Software
445 Jan Davis Drive NW   -    Huntsville, AL 35806  -  USA
jabber: rbryant at digium.com    -=-    skype: russell-bryant
www.digium.com -=- www.asterisk.org -=- blogs.asterisk.org