[asterisk-dev] [Code Review] Deadlock x2 Race on pickup and a race on hint remove

Thu Apr 14 01:51:54 CDT 2011

> On 2011-04-13 14:35:59, David Vossel wrote:
> >

ao2_callback should be called with no elements in the container locked this is not always the case and deadlocks happen almost all deadlocks i see out in the wild are ao2_callback related.

> On 2011-04-13 14:35:59, David Vossel wrote:
> > /branches/1.8/main/channel.c, lines 6243-6249
> > <https://reviewboard.asterisk.org/r/1166/diff/5/?file=16131#file16131line6243>
> >
> >     locking order is
> >     
> >     Channels container then the individual channels.
> >     
> >     If you hold the channels lock before grabbing a channel lock, no dead lock avoidance should take place while grabbing the channel lock.

In a perfect world .... the problem is if you see the lock output above the pesky ao2_callback in another thread does not play nice it grabs a channel lock then runs ao2_callback against channels grabing the channels container causing a collision here.

> On 2011-04-13 14:35:59, David Vossel wrote:
> > /branches/1.8/main/pbx.c, lines 4269-4275
> > <https://reviewboard.asterisk.org/r/1166/diff/5/?file=16132#file16132line4269>
> >
> >     No dead lock avoidance is necessary here.  The container lock held before the element lock is the correct locking order.

yes this is the correct locking order i wanted to initially swap it arround once again ao2_callback does not follow this rule nice when removing a hint the hint is held then callback grabs the hints and a deadlock is born this avoids it not fixes the underlying problem.

- irroot

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/1166/#review3332
-----------------------------------------------------------

On 2011-04-13 13:29:53, irroot wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/1166/
> -----------------------------------------------------------
> 
> (Updated 2011-04-13 13:29:53)
> 
> 
> Review request for Asterisk Developers, Russell Bryant and Brett Bryant.
> 
> 
> Summary
> -------
> 
> Can some one give me a good reason why the bellow order should not be reversed ??
> 
> ao2_callback runs on various threads holding container locks we must make sure we
> lock the element before locking a container i have a feeling asterisk could be riddled
> with these deadlocks / race conditions ....
> 
> /* disclaimer the first lock output is from a patched app_directed_pickup 
>  *
>  * the changes use a callback the same as features.c
>  *
>  */
> 
> 
> =======================================================================
> === Currently Held Locks ==============================================
> =======================================================================
> ===
> === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
> ===
> === Thread ID: 0xab37bb70 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
> === ---> Lock #0 (channel.c): MUTEX 6241 ast_do_masquerade channels 0x89de1d0 (1)
> === ---> Waiting for Lock #1 (channel.c): MUTEX 6244 ast_do_masquerade original 0x92fa3a8 (1)
> === --- ---> Locked Here: app_directed_pickup.c line 322 (pickup_by_group)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0xab597b70 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
> === ---> Waiting for Lock #0 (app_directed_pickup.c): MUTEX 155 pickup_do target 0x92fa3a8 (1)
> === --- ---> Locked Here: app_directed_pickup.c line 322 (pickup_by_group)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0xab04bb70 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
> === ---> Lock #0 (app_directed_pickup.c): MUTEX 322 pickup_by_group target 0x92fa3a8 (1)
> === ---> Lock #1 (cdr.c): RDLOCK 1142 post_cdr &(&be_list)->lock 0x81f1cc8 (1)
> === ---> Waiting for Lock #2 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x89de1d0 (1)
> === --- ---> Locked Here: channel.c line 6241 (ast_do_masquerade)
> === -------------------------------------------------------------------
> ======================================================================= 
> 
> 
> =======================================================================
> === Currently Held Locks ==============================================
> =======================================================================
> ===
> === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
> ===
> === Thread ID: 0xb3c41b70 (tps_processing_function started at [  451] taskprocessor.c ast_taskprocessor_get())
> === ---> Lock #0 (pbx.c): MUTEX 4269 handle_statechange hints 0x97a4b48 (1)
> === ---> Waiting for Lock #1 (pbx.c): MUTEX 4270 handle_statechange hint 0xb04cf7c0 (1)
> === --- ---> Locked Here: pbx.c line 4523 (ast_remove_hint)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0xb09f6b70 (do_monitor           started at [24621] chan_sip.c restart_monitor())
> === ---> Lock #0 (chan_sip.c): MUTEX 24119 handle_request_do &netlock 0xb309fdc0 (1)
> === ---> Lock #1 (chan_sip.c): MUTEX 7593 find_call p 0xffb8d48 (1)
> === ---> Waiting for Lock #2 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x97a4b48 (1)
> === --- ---> Locked Here: pbx.c line 4269 (handle_statechange)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0xb05c3b70 (netconsole           started at [ 1344] asterisk.c listener())
> === ---> Lock #0 (loader.c): MUTEX 673 ast_module_reload &reloadlock 0x81f5280 (1)
> === ---> Lock #1 (loader.c): MUTEX 711 ast_module_reload &(&module_list)->lock 0x81f5208 (1)
> === ---> Lock #2 (pbx.c): MUTEX 4523 ast_remove_hint hint 0xb04cf7c0 (1)
> === ---> Waiting for Lock #3 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x97a4b48 (1)
> === --- ---> Locked Here: pbx.c line 4269 (handle_statechange)
> === -------------------------------------------------------------------
> ===
> =======================================================================
> 
> 
> This addresses bugs 18654 and 19091.
>     https://issues.asterisk.org/view.php?id=18654
>     https://issues.asterisk.org/view.php?id=19091
> 
> 
> Diffs
> -----
> 
>   /branches/1.8/channels/chan_sip.c 313524 
>   /branches/1.8/main/channel.c 313524 
>   /branches/1.8/main/pbx.c 313524 
> 
> Diff: https://reviewboard.asterisk.org/r/1166/diff
> 
> 
> Testing
> -------
> 
> Ok this makes things a little more intresting need some more work
> 
> =======================================================================
> === Currently Held Locks ==============================================
> =======================================================================
> ===
> === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
> ===
> === Thread ID: 0xabd35b70 (do_monitor           started at [24621] chan_sip.c restart_monitor())
> === ---> Lock #0 (chan_sip.c): MUTEX 24119 handle_request_do &netlock 0xb30f3dc0 (1)
> === ---> Lock #1 (channel.c): MUTEX 6233 ast_do_masquerade original 0xab409a30 (1)
> === ---> Waiting for Lock #2 (channel.c): MUTEX 6254 ast_do_masquerade channels 0xa1d31c0 (1)
> === --- ---> Locked Here: channel.c line 6254 (ast_do_masquerade)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0xaab97b70 (pbx_thread           started at [ 5038] pbx.c ast_pbx_start())
> === ---> Lock #0 (channel.c): MUTEX 6254 ast_do_masquerade channels 0xa1d31c0 (1)
> === ---> Waiting for Lock #1 (channel.c): MUTEX 6233 ast_do_masquerade original 0xab409a30 (1)
> === --- ---> Locked Here: channel.c line 6233 (ast_do_masquerade)
> === -------------------------------------------------------------------
> =======================================================================
> 
> 
> Thanks,
> 
> irroot
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20110414/9251bd1b/attachment-0001.htm>