[asterisk-dev] Deadlock in pthread_exit due to lazy binding with libgcc

[Yousf Ateya]

Dear,

I started to see a  strange deadlock in some asterisk nodes. For every
call, when calling pthread_exit from pbx_thread, the caller thread is stuck
inside pthread_exit.

After a while, there will be tens-of-thousands of threads having the same
backtrace. After some googling, I found this happens because of the default
lazy linking of gcc linker.

Related issue of stackoverflow:
http://stackoverflow.com/questions/11954527/dlopen-malloc-deadlock

Tried to recompile asterisk using:
export LDFLAGS=-Wl,-z,now
./configure && make && make install

and this deadlock problem didn't happen again; the problem cause is lazy
binding with libgcc.

Shall we add this option by default or add it in menuselect?

I am using Asterisk 13.4 compiled on Ubuntu 14.04 64 bit with gcc 4.8.2,
but probably this applies to other OSs/compilers.


============
Deadlock backtrace
============

Thread 6138 (Thread 0x2acfc684b700 (LWP 12259)):
#0  __lll_lock_wait () at
../nptl/sysdeps/unix/sysv/linux/x86_64/lowlevellock.S:135
#1  0x00002ace66628672 in _L_lock_953 () from
/lib/x86_64-linux-gnu/libpthread.so.0
#2  0x00002ace666284da in __GI___pthread_mutex_lock (mutex=0x2ace64b1b968
<_rtld_global+2312>) at ../nptl/pthread_mutex_lock.c:114
#3  0x00002ace6490c34e in _dl_open (file=0x2ace66630ccf "libgcc_s.so.1",
mode=-2147483647, caller_dlopen=0x2ace6662ea43 <pthread_cancel_init+35>,
nsid=-2, argc=1, argv=0x7fff1f751498, env=0x1907320)
    at dl-open.c:613
#4  0x00002ace670c30f2 in do_dlopen (ptr=ptr at entry=0x2acfc684ad80) at
dl-libc.c:87
#5  0x00002ace64907ff4 in _dl_catch_error (objname=0x2acfc684ad60,
errstring=0x2acfc684ad70, mallocedp=0x2acfc684ad50, operate=0x2ace670c30b0
<do_dlopen>, args=0x2acfc684ad80) at dl-error.c:187
#6  0x00002ace670c31b2 in dlerror_run (args=0x2acfc684ad80,
operate=0x2ace670c30b0 <do_dlopen>) at dl-libc.c:46
#7  __GI___libc_dlopen_mode (name=<optimized out>, mode=<optimized out>) at
dl-libc.c:163
#8  0x00002ace6662ea43 in pthread_cancel_init () at
../nptl/sysdeps/pthread/unwind-forcedunwind.c:52
#9  0x00002ace6662ec0c in _Unwind_ForcedUnwind (exc=0x2acfc684bd70,
stop=stop at entry=0x2ace6662cbc0 <unwind_stop>, stop_argument=0x2acfc684ae60)
at ../nptl/sysdeps/pthread/unwind-forcedunwind.c:129
#10 0x00002ace6662cd40 in __GI___pthread_unwind (buf=<optimized out>) at
unwind.c:129
#11 0x00002ace66627535 in __do_cancel () at pthreadP.h:280
#12 __pthread_exit (value=value at entry=0x0) at pthread_exit.c:29
#13 0x000000000057a523 in pbx_thread (data=data at entry=0x2acf0c171588) at
pbx.c:6773
#14 0x00000000005d734a in dummy_start (data=<optimized out>) at utils.c:1237
#15 0x00002ace66626182 in start_thread (arg=0x2acfc684b700) at
pthread_create.c:312
#16 0x00002ace6708747d in clone () at
../sysdeps/unix/sysv/linux/x86_64/clone.S:111


-- 
Yousf Ateya,
StarkBits
www.starkbits.com

-- 


This e-mail message is intended only for the use of the intended recipient(s).
The information contained therein may be confidential or privileged,
and its disclosure or reproduction is strictly prohibited.
If you are not the intended recipient, please return it immediately to its sender 
at the above address and destroy it. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20150715/d3b1278c/attachment.html>