[asterisk-bugs] [Asterisk 0015109]: Abort by memory allocator, possibly in moh_files_generator

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
https://issues.asterisk.org/view.php?id=15109 
====================================================================== 
Reported By:                jvandal
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   15109
Category:                   Resources/res_musiconhold
Reproducibility:            random
Severity:                   block
Priority:                   normal
Status:                     acknowledged
Target Version:             1.4.27
Asterisk Version:           1.4.24 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-05-14 10:49 CDT
Last Modified:              2009-07-03 15:20 CDT
====================================================================== 
Summary:                    Abort by memory allocator, possibly in
moh_files_generator
Description: 
I have a server running with Asterisk 1.4.24.1 where it randomly segfault
for "unknown" reason.

I'm not sure if this is related to moh_files_generator function or with
filestream_descructor.

Let me know what needed in order to fix this crash, if GDB traces aren't
enough. 

Asterisk is compiled with DONT_OPTIMIZE and others flag needed for "gdb".

======================================================================
Relationships       ID      Summary
----------------------------------------------------------------------
related to          0014958 Segfault Asterisk 1.4.24.1
related to          0015123 out of bounds crash and core dump
has duplicate       0015195 double free or corruption (!prev) in mo...
====================================================================== 

---------------------------------------------------------------------- 
 (0107400) aragon (reporter) - 2009-07-03 15:20
 https://issues.asterisk.org/view.php?id=15109#c107400 
---------------------------------------------------------------------- 
Russell

I'm really confused.
You said the valgrind output is normal because Asterisk was stopped.
But Asterisk was not stopped, in fact it crashed and I caught the crash in
Valgrind. I caught that data here
https://issues.asterisk.org/file_download.php?file_id=23158&type=bug

I have been trying to find ways of reproducing the bug and I think I am
close.  I'm pretty sure it is caused during a reload.  And now I am
suspecting that this crash only involves musiconhold if an mp3 file is
played back in files mode and not a wav file.  I have been trying to
reproduce a crash with that hypothesis by removing all mp3 files from
/var/lib/asterisk/moh/ and only using stock wav files.  With that test in
place and no Valgrind running so I could more easily reproduce the problem
I ran into two crashes today.
Neither of these crashes bt evidenced any crash caused by moh.
1. Another crash caused by the bug in 15396
2. A crash during a reload

This is the backtrace from 2.
(Apologies in advance to Leif for posting full bt in bug notes but the bt
data is small...)

https://issues.asterisk.org/view.php?id=0  0x00727827 in _dl_close_worker ()
from /lib/ld-linux.so.2
(gdb) bt
https://issues.asterisk.org/view.php?id=0  0x00727827 in _dl_close_worker ()
from /lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=1  0x00728487 in _dl_close () from
/lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=2  0x0087ece4 in dlclose_doit () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=3  0x00722e46 in _dl_catch_error () from
/lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=4  0x0087f2cc in _dlerror_run () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=5  0x0087ed1a in dlclose () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=6  0x080c1ff3 in load_dynamic_module ()
https://issues.asterisk.org/view.php?id=7  0x080c338c in load_resource ()
https://issues.asterisk.org/view.php?id=8  0x080c3dea in load_modules ()
https://issues.asterisk.org/view.php?id=9  0x0807212c in main ()
(gdb) bt full
https://issues.asterisk.org/view.php?id=0  0x00727827 in _dl_close_worker ()
from /lib/ld-linux.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=1  0x00728487 in _dl_close () from
/lib/ld-linux.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=2  0x0087ece4 in dlclose_doit () from
/lib/libdl.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=3  0x00722e46 in _dl_catch_error () from
/lib/ld-linux.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=4  0x0087f2cc in _dlerror_run () from
/lib/libdl.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=5  0x0087ed1a in dlclose () from
/lib/libdl.so.2
No symbol table info available.
https://issues.asterisk.org/view.php?id=6  0x080c1ff3 in load_dynamic_module ()
No symbol table info available.
https://issues.asterisk.org/view.php?id=7  0x080c338c in load_resource ()
No symbol table info available.
https://issues.asterisk.org/view.php?id=8  0x080c3dea in load_modules ()
No symbol table info available.
https://issues.asterisk.org/view.php?id=9  0x0807212c in main ()
No symbol table info available.
(gdb) thread apply all bt

Thread 1 (process 15353):
https://issues.asterisk.org/view.php?id=0  0x00727827 in _dl_close_worker ()
from /lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=1  0x00728487 in _dl_close () from
/lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=2  0x0087ece4 in dlclose_doit () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=3  0x00722e46 in _dl_catch_error () from
/lib/ld-linux.so.2
https://issues.asterisk.org/view.php?id=4  0x0087f2cc in _dlerror_run () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=5  0x0087ed1a in dlclose () from
/lib/libdl.so.2
https://issues.asterisk.org/view.php?id=6  0x080c1ff3 in load_dynamic_module ()
https://issues.asterisk.org/view.php?id=7  0x080c338c in load_resource ()
https://issues.asterisk.org/view.php?id=8  0x080c3dea in load_modules ()
https://issues.asterisk.org/view.php?id=9  0x0807212c in main ()

And this looks exactly like my Valgrind file.
So my hypothesis looks more provable and the crash/memory abort occurs
during a reload (maybe moh is just incidental).  Some more days trying to
reproduce the crash without mp3 (wav only) files in /var/lib/asterisk/moh/
I think will finish proving my hypothesis.

Does the valgrind info provide any clue why asterisk would crash during a
normal reload? 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-07-03 15:20 aragon         Note Added: 0107400                          
======================================================================