[asterisk-bugs] [Asterisk 0014436]: Asterisk crashes in ast_channels_free with "free(): invalid pointer"

[Asterisk Bug Tracker]

A NOTE has been added to this issue. 
====================================================================== 
http://bugs.digium.com/view.php?id=14436 
====================================================================== 
Reported By:                Censo
Assigned To:                
====================================================================== 
Project:                    Asterisk
Issue ID:                   14436
Category:                   Core/Channels
Reproducibility:            random
Severity:                   crash
Priority:                   normal
Status:                     feedback
Asterisk Version:           1.6.0.5 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases): N/A 
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2009-02-09 09:05 CST
Last Modified:              2009-02-11 06:26 CST
====================================================================== 
Summary:                    Asterisk crashes in ast_channels_free with "free():
invalid pointer"
Description: 
Asterisk has crashed under load after 20 hours of work, glibc reports
freeing of invalid pointer.
Asterisk was compiled without optimizations and with malloc_debug on.
====================================================================== 

---------------------------------------------------------------------- 
 (0099876) Censo (reporter) - 2009-02-11 06:26
 http://bugs.digium.com/view.php?id=14436#c99876 
---------------------------------------------------------------------- 
I investigated source of some problems, preceding crash and found some
inconsistency in mutex handling in Asterisk.
   These logs appeared the same second when Asterisk crashed:

1. [Feb  9 15:29:59] ERROR[7222]
/root/dev/pbx/asterisk-1.6.0.5/include/asterisk/lock.h: pbx.c line 6424
(ast_explicit_goto): Error obtaining mutex: Invalid argument
2. [Feb  9 15:29:59] ERROR[7222]
/root/dev/pbx/asterisk-1.6.0.5/include/asterisk/lock.h: pbx.c line 6437
(ast_explicit_goto): mutex '&chan->lock_dont_use' freed more times than
we've locked!
3. [Feb  9 15:29:59] ERROR[7222]
/root/dev/pbx/asterisk-1.6.0.5/include/asterisk/lock.h: pbx.c line 6437
(ast_explicit_goto): Error releasing mutex: Invalid argument

   It is not clear why mutex get corrupted, but:

1. When trying to lock it in '__ast_pthread_mutex_lock',
pthread_mutex_lock is called (lock.h:427) and if result is 'wrong',
t->reentrancy (lock.h:437) is _not_ incremented.
2. __ast_pthread_mutex_unlock is called and it decrements usage of the
mutex every time! It does not care whether mutex was right or wrong. Since
the mutex was not taken at step one, it reports wrong reentrancy.
3. And only at last '__ast_pthread_mutex_unlock' tries to unlock the
mutex, reporting one more error.

   I think, since return value for both lock/unlock functions is not
checked, it is needed to keep track of last "lock" operation to prevent
false reentrancy decrements. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-02-11 06:26 Censo          Note Added: 0099876                          
======================================================================