[asterisk-dev] Asterisk 11 crashing on Hangup()

[Pavel Troller]

Hi for the second time,

> Hi!
>   On a highly loaded system, I see occasional crashes. gdb shows:
> (gdb) bt
> #0  0x000000000048a451 in ast_bridged_channel (chan=0x3ba74d8) at channel.c:7403
> #1  0x000000000047b6b5 in ast_set_hangupsource (chan=0x3ba74d8, source=0x5f3fd3 "dialplan/builtin", force=0) at channel.c:2721
> #2  0x000000000054ff3d in pbx_builtin_hangup (chan=0x3ba74d8, data=0x4498f040 "16") at pbx.c:10624
> #3  0x000000000052fa05 in pbx_exec (c=0x3ba74d8, app=0x1c682f0, data=0x4498f040 "16") at pbx.c:1594
> #4  0x000000000053a674 in pbx_extension_helper (c=0x3ba74d8, con=0x0, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=43, label=0x0, callerid=0x7f3f6d0c3450 "00919820014701", action=E_SPAWN, found=0x4499169c,
>     combined_find_spawn=1) at pbx.c:4447
> #5  0x000000000053dc70 in ast_spawn_extension (c=0x3ba74d8, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=43, callerid=0x7f3f6d0c3450 "00919820014701", found=0x4499169c, combined_find_spawn=1) at pbx.c:5563
> #6  0x00007f3f773e31e2 in _macro_exec (chan=0x3ba74d8, data=0x449943c0 "incept,,", exclusive=0) at app_macro.c:412
> #7  0x00007f3f773e438f in macro_exec (chan=0x3ba74d8, data=0x449943c0 "incept,,") at app_macro.c:585
> #8  0x000000000052fa05 in pbx_exec (c=0x3ba74d8, app=0x1df5820, data=0x449943c0 "incept,,") at pbx.c:1594
> #9  0x000000000053a674 in pbx_extension_helper (c=0x3ba74d8, con=0x0, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=129, label=0x0, callerid=0x7f3f6d0c3450 "00919820014701", action=E_SPAWN, found=0x44996a3c,
>     combined_find_spawn=1) at pbx.c:4447
> #10 0x000000000053dc70 in ast_spawn_extension (c=0x3ba74d8, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=129, callerid=0x7f3f6d0c3450 "00919820014701", found=0x44996a3c, combined_find_spawn=1) at pbx.c:5563
> #11 0x00007f3f773e31e2 in _macro_exec (chan=0x3ba74d8, data=0x44999760 "stddest,.*,y,DAHDI/g2,,3,8,20,0,,0,,EF", exclusive=0) at app_macro.c:412
> #12 0x00007f3f773e438f in macro_exec (chan=0x3ba74d8, data=0x44999760 "stddest,.*,y,DAHDI/g2,,3,8,20,0,,0,,EF") at app_macro.c:585
> #13 0x000000000052fa05 in pbx_exec (c=0x3ba74d8, app=0x1df5820, data=0x44999760 "stddest,.*,y,DAHDI/g2,,3,8,20,0,,0,,EF") at pbx.c:1594
> #14 0x000000000053a674 in pbx_extension_helper (c=0x3ba74d8, con=0x0, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=4, label=0x0, callerid=0x4e13d50 "m\225\003", action=E_SPAWN, found=0x4499bed0, combined_find_spawn=1)
>     at pbx.c:4447
> #15 0x000000000053dc70 in ast_spawn_extension (c=0x3ba74d8, context=0x3ba8328 "macro-incept", exten=0x3ba8378 "s", priority=4, callerid=0x4e13d50 "m\225\003", found=0x4499bed0, combined_find_spawn=1) at pbx.c:5563
> #16 0x000000000053f467 in __ast_pbx_run (c=0x3ba74d8, args=0x0) at pbx.c:6038
> #17 0x0000000000541095 in pbx_thread (data=0x3ba74d8) at pbx.c:6368
> #18 0x000000000059d8c8 in dummy_start (data=0x230c280) at utils.c:1028
> #19 0x00007f3f9564d427 in start_thread () from /lib64/libpthread.so.0
> #20 0x00007f3f9685ba0d in clone () from /lib64/libc.so.6
> (gdb) frame 0
> #0  0x000000000048a451 in ast_bridged_channel (chan=0x3ba74d8) at channel.c:7403
> 7403            if (bridged && ast_channel_tech(bridged)->bridged_channel)
> (gdb) print *bridged
> $1 = {tech = 0x0, tech_pvt = 0x0, music_state = 0x0, generatordata = 0x0, generator = 0x0, bridged_channel = 0x3ba74d8, ....
> 
>   As you can see, bridged->tech is 0, which causes the second term in the condition to segfault.
>   I don't know, why it's zero, but I've done this:
> 
> Index: channel.c
> ===================================================================
> --- channel.c	(revision 376616)
> +++ channel.c	(working copy)
> @@ -7400,7 +7400,7 @@
>  {
>  	struct ast_channel *bridged;
>  	bridged = ast_channel_internal_bridged_channel(chan);
> -	if (bridged && ast_channel_tech(bridged)->bridged_channel)
> +	if (bridged && ast_channel_tech(bridged) && ast_channel_tech(bridged)->bridged_channel)
>  		bridged = ast_channel_tech(bridged)->bridged_channel(chan, bridged);
>  	return bridged;
>  }

It didn't help. It crashes later in the same function - it tries to fill in
the hangupsource string but it fails for the following reason:

(gdb) frame 0
#0  0x000000000059f3ac in __ast_string_field_ptr_grow (mgr=0x7fb9e123a3c0, pool_head=0x7fb9e123a358, needed=17, ptr=0x7fb9e123a3b0) at utils.c:1759
1759            size_t space = (*pool_head)->size - (*pool_head)->used;
(gdb) print *mgr
$11 = {last_alloc = 0x0, embedded_pool = 0x0}
(gdb) print *pool_head
$12 = (struct ast_string_field_pool *) 0x0
(gdb) print *ptr
$13 = 0x867106 ""

And it is because __field_mgr_pool in the channel is also zero.
So, I think it's not a good idea to patch the immediate points of the
crash, but to find, why the channel is already partially destroyed
when the dial plan calls Hangup() after Dial() returns and the dialplan
logic doesn't find anything more to do.

WIth regards, Pavel

> 
>   The crashes are not frequent (coming for about every 30000th call), so we must
> wait a while to see, whether it helped or another crash will happen somewhere
> for the same reason.
> 
> What do you think about this problem ?
> 
> With regards,
>   Pavel
> 
> --
> _____________________________________________________________________
> -- Bandwidth and Colocation Provided by http://www.api-digital.com --
> 
> asterisk-dev mailing list
> To UNSUBSCRIBE or update options visit:
>    http://lists.digium.com/mailman/listinfo/asterisk-dev