[asterisk-bugs] [Asterisk 0013778]: asterisk blocked at startup between main/asterisk.c/loader.c/load_modules and manager.c/loader.c/ast_module_reload

Wed Jun 3 13:00:27 CDT 2009

The following issue is now READY FOR TESTING. 
====================================================================== 
https://issues.asterisk.org/view.php?id=13778 
====================================================================== 
Reported By:                hotsblanc
Assigned To:                seanbright
====================================================================== 
Project:                    Asterisk
Issue ID:                   13778
Category:                   Core/General
Reproducibility:            always
Severity:                   major
Priority:                   normal
Status:                     ready for testing
Asterisk Version:           Older 1.4 
Regression:                 No 
SVN Branch (only for SVN checkouts, not tarball releases):  1.4  
SVN Revision (number only!):  
Request Review:              
====================================================================== 
Date Submitted:             2008-10-24 15:18 CDT
Last Modified:              2009-06-03 13:00 CDT
====================================================================== 
Summary:                    asterisk blocked at startup between
main/asterisk.c/loader.c/load_modules and manager.c/loader.c/ast_module_reload
Description: 
- using asterisk-1.4.11 (there is no easy way for me right now to update to
SVN header for 1.4 or 1.6 trunk)
- OS is Linux 2.6.18_pro500-440epx_eval (ppc amcc)

when starting an AMI client right after asterisk and issuing an AMI
reload, asterisk gets locked and cannot for instance process any SIP or AMI
message. 

we do launch an AMI reload at startup to update few asterisk configuration
files on the fly after getting some dynamic information from external
processes (and asterisk gets started just before our AMI client).

- putting traces in the code, at startup main/asterisk.c calls

loader.c/load_modules(0)

which locks the module_list with

AST_LIST_LOCK(&module_list);

[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack for thread 805449056
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 0
/usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 1
/usr/sbin/asterisk(load_modules+0x50) [0x10067b90]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 2
/usr/sbin/asterisk(main+0x1928) [0x1002790c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 3 /lib/libc.so.6
[0xfb99134]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 4 /lib/libc.so.6
[0xfb9935c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK load_modules
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK module_register

before the load_modules method can finish, the AMI client 'reload' call
ends up calling loader.c/ast_module_reload which locks the same linked
module list head object:

AST_LIST_LOCK(&module_list);

[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack for thread 806479024
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 0
/usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 1
/usr/sbin/asterisk(ast_module_reload+0x8c) [0x10066810]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 2 /usr/sbin/asterisk
[0x10042d78]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 3
/usr/sbin/asterisk(ast_cli_command+0xb0) [0x10046ae4]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 4 /usr/sbin/asterisk
[0x10072474]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 5 /usr/sbin/asterisk
[0x10073290]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 6 /usr/sbin/asterisk
[0x10074750]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 7 /usr/sbin/asterisk
[0x10074798]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 8 /usr/sbin/asterisk
[0x100b45cc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 9 /lib/libpthread.so.0
[0xffa9758]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === ast_module_reload before
LOCK

the list_lock call in loader.c/ast_module_reload method never returns
and loader.c/load_modules method never returns either, so none of these 2
methods end up calling

AST_LIST_UNLOCK(&module_list);

this happens to me 4 out of 5 times when starting my AMI client after
asterisk with no real dependency between the 2 startup but the fact that
the asterisk process is up linux wise.

work-around: call init_manager() after load_modules(0) in main/asterisk.c
so that my AMI client fails creating the TCP manager socket and keeps on
trying; that way whenever the client TCP socket gets created and AMI reload
gets called, load_modules method call is ensured to have ended.

or ... would there be another command I could issue first to know that
asterisk is fully loaded, .e.g main code done and asterisk in main loop?
======================================================================
Relationships       ID      Summary
----------------------------------------------------------------------
parent of           0015189 [patch] #exec script can't access manag...
====================================================================== 

---------------------------------------------------------------------- 
 (0105939) seanbright (manager) - 2009-06-03 13:00
 https://issues.asterisk.org/view.php?id=13778#c105939 
---------------------------------------------------------------------- 
hotsblanc,

I know it has been a while since this bug was addressed, but if you could
try the second patch in issue 15189 (the one called
06032009_15189_deferred_reloads.diff) and report back if that solves the
deadlock issue you were seeing, that would be great.

Thanks. 

Issue History 
Date Modified    Username       Field                    Change               
====================================================================== 
2009-06-03 13:00 seanbright     Note Added: 0105939                          
2009-06-03 13:00 seanbright     Status                   assigned => ready for
testing
======================================================================