[asterisk-bugs] [Asterisk 0013778]: asterisk blocked at startup between main/asterisk.c/loader.c/load_modules and manager.c/loader.c/ast_module_reload
Asterisk Bug Tracker
noreply at bugs.digium.com
Thu Jun 4 09:53:59 CDT 2009
A NOTE has been added to this issue.
======================================================================
https://issues.asterisk.org/view.php?id=13778
======================================================================
Reported By: hotsblanc
Assigned To: seanbright
======================================================================
Project: Asterisk
Issue ID: 13778
Category: Core/General
Reproducibility: always
Severity: major
Priority: normal
Status: closed
Asterisk Version: Older 1.4
Regression: No
SVN Branch (only for SVN checkouts, not tarball releases): 1.4
SVN Revision (number only!):
Request Review:
Resolution: fixed
Fixed in Version:
======================================================================
Date Submitted: 2008-10-24 15:18 CDT
Last Modified: 2009-06-04 09:53 CDT
======================================================================
Summary: asterisk blocked at startup between
main/asterisk.c/loader.c/load_modules and manager.c/loader.c/ast_module_reload
Description:
- using asterisk-1.4.11 (there is no easy way for me right now to update to
SVN header for 1.4 or 1.6 trunk)
- OS is Linux 2.6.18_pro500-440epx_eval (ppc amcc)
when starting an AMI client right after asterisk and issuing an AMI
reload, asterisk gets locked and cannot for instance process any SIP or AMI
message.
we do launch an AMI reload at startup to update few asterisk configuration
files on the fly after getting some dynamic information from external
processes (and asterisk gets started just before our AMI client).
- putting traces in the code, at startup main/asterisk.c calls
loader.c/load_modules(0)
which locks the module_list with
AST_LIST_LOCK(&module_list);
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack for thread 805449056
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 0
/usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 1
/usr/sbin/asterisk(load_modules+0x50) [0x10067b90]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 2
/usr/sbin/asterisk(main+0x1928) [0x1002790c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 3 /lib/libc.so.6
[0xfb99134]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === stack 4 /lib/libc.so.6
[0xfb9935c]
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK load_modules
[Oct 24 11:16:11] DEBUG[12543] loader.c: === LOCK module_register
before the load_modules method can finish, the AMI client 'reload' call
ends up calling loader.c/ast_module_reload which locks the same linked
module list head object:
AST_LIST_LOCK(&module_list);
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack for thread 806479024
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 0
/usr/sbin/asterisk(print_trace+0x38) [0x100661dc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 1
/usr/sbin/asterisk(ast_module_reload+0x8c) [0x10066810]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 2 /usr/sbin/asterisk
[0x10042d78]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 3
/usr/sbin/asterisk(ast_cli_command+0xb0) [0x10046ae4]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 4 /usr/sbin/asterisk
[0x10072474]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 5 /usr/sbin/asterisk
[0x10073290]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 6 /usr/sbin/asterisk
[0x10074750]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 7 /usr/sbin/asterisk
[0x10074798]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 8 /usr/sbin/asterisk
[0x100b45cc]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === stack 9 /lib/libpthread.so.0
[0xffa9758]
[Oct 24 11:16:12] DEBUG[12565] loader.c: === ast_module_reload before
LOCK
the list_lock call in loader.c/ast_module_reload method never returns
and loader.c/load_modules method never returns either, so none of these 2
methods end up calling
AST_LIST_UNLOCK(&module_list);
this happens to me 4 out of 5 times when starting my AMI client after
asterisk with no real dependency between the 2 startup but the fact that
the asterisk process is up linux wise.
work-around: call init_manager() after load_modules(0) in main/asterisk.c
so that my AMI client fails creating the TCP manager socket and keeps on
trying; that way whenever the client TCP socket gets created and AMI reload
gets called, load_modules method call is ensured to have ended.
or ... would there be another command I could issue first to know that
asterisk is fully loaded, .e.g main code done and asterisk in main loop?
======================================================================
Relationships ID Summary
----------------------------------------------------------------------
parent of 0015189 [patch] #exec script can't access manag...
======================================================================
----------------------------------------------------------------------
(0105983) svnbot (reporter) - 2009-06-04 09:53
https://issues.asterisk.org/view.php?id=13778#c105983
----------------------------------------------------------------------
Repository: asterisk
Revision: 199054
_U branches/1.6.2/
U branches/1.6.2/include/asterisk/_private.h
U branches/1.6.2/main/asterisk.c
U branches/1.6.2/main/loader.c
------------------------------------------------------------------------
r199054 | seanbright | 2009-06-04 09:53:58 -0500 (Thu, 04 Jun 2009) | 54
lines
Merged revisions 199051 via svnmerge from
https://origsvn.digium.com/svn/asterisk/trunk
................
r199051 | seanbright | 2009-06-04 10:31:24 -0400 (Thu, 04 Jun 2009) | 47
lines
Merged revisions 199022 via svnmerge from
https://origsvn.digium.com/svn/asterisk/branches/1.4
........
r199022 | seanbright | 2009-06-04 10:14:57 -0400 (Thu, 04 Jun 2009) |
40 lines
Safely handle AMI connections/reload requests that occur during
startup.
During asterisk startup, a lock on the list of modules is obtained by
the
primary thread while each module is initialized. Issue 13778 pointed
out a
problem with this approach, however. Because the AMI is loaded before
other
modules, it is possible for a module reload to be issued by a
connected client
(via Action: Command), causing a deadlock.
The resolution for 13778 was to move initialization of the manager to
happen
after the other modules had already been lodaded. While this fixed
this
particular issue, it caused a problem for users (like FreePBX) who
call AMI
scripts via an #exec in a configuration file (See issue 15189).
The solution I have come up with is to defer any reload requests that
come in
until after the server is fully booted. When a call comes in to
ast_module_reload (from wherever) before we are fully booted, the
request is
added to a queue of pending requests. Once we are done booting up, we
then
execute these deferred requests in turn.
Note that I have tried to make this a bit more intelligent in that it
will not
queue up more than 1 request for the same module to be reloaded, and
if a
general reload request comes in ('module reload') the queue is flushed
and we
only issue a single deferred reload for the entire system.
As for how this will impact existing installations - Before 13778, a
reload
issued before module initialization was completed would result in a
deadlock.
After 13778, you simply couldn't connect to the manager during startup
(which
causes problems with #exec-that-calls-AMI configuration files). I
believe this
is a good general purpose solution that won't negatively impact
existing
installations.
(closes issue https://issues.asterisk.org/view.php?id=15189)
(closes issue https://issues.asterisk.org/view.php?id=13778)
Reported by: p_lindheimer
Patches:
06032009_15189_deferred_reloads.diff uploaded by seanbright
(license 71)
Tested by: p_lindheimer, seanbright
Review: https://reviewboard.asterisk.org/r/272/
........
................
------------------------------------------------------------------------
http://svn.digium.com/view/asterisk?view=rev&revision=199054
Issue History
Date Modified Username Field Change
======================================================================
2009-06-04 09:53 svnbot Checkin
2009-06-04 09:53 svnbot Note Added: 0105983
======================================================================
More information about the asterisk-bugs
mailing list