<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/1166/">https://reviewboard.asterisk.org/r/1166/</a>
</td>
</tr>
</table>
<br />
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">ao2_callback should be called with no elements in the container locked this is not always the case and deadlocks happen almost all deadlocks i see out in the wild are ao2_callback related.</pre>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<p style="margin-top: 0;">On April 13th, 2011, 2:35 p.m., <b>David Vossel</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
<thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
<a href="https://reviewboard.asterisk.org/r/1166/diff/5/?file=16131#file16131line6243" style="color: black; font-weight: bold; text-decoration: underline;">/branches/1.8/main/channel.c</a>
<span style="font-weight: normal;">
(Diff revision 5)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="4"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">int ast_do_masquerade(struct ast_channel *original)</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">6243</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="cm">/* lock the original channel to determine if the masquerade is require or not */</span></pre></td>
<th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6243</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="cm">/* lock the original channel<span class="hl"> with deadlock avoidance</span> to determine if the masquerade is require or not */</span></pre></td>
</tr>
<tr>
<th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">6244</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ast_channel_lock</span><span class="p">(</span><span class="n">original</span><span class="p">)<span class="hl">;</span></span></pre></td>
<th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6244</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "><span class="hl"> </span><span class="k"><span class="hl">while</span></span><span class="hl"> </span><span class="p"><span class="hl">(</span></span><span class="n">ast_channel_<span class="hl">try</span>lock</span><span class="p">(</span><span class="n">original</span><span class="p">)<span class="hl">)</span></span><span class="hl"> </span><span class="p"><span class="hl">{</span></span></pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6245</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_unlock</span><span class="p">(</span><span class="n">channels</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6246</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">usleep</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6247</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_lock</span><span class="p">(</span><span class="n">channels</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6248</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="p">}</span></pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#f0f0f0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">6245</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#f0f0f0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">6249</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
</tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">locking order is
Channels container then the individual channels.
If you hold the channels lock before grabbing a channel lock, no dead lock avoidance should take place while grabbing the channel lock.</pre>
</blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">In a perfect world .... the problem is if you see the lock output above the pesky ao2_callback in another thread does not play nice it grabs a channel lock then runs ao2_callback against channels grabing the channels container causing a collision here.</pre>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<p style="margin-top: 0;">On April 13th, 2011, 2:35 p.m., <b>David Vossel</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
<thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
<a href="https://reviewboard.asterisk.org/r/1166/diff/5/?file=16132#file16132line4269" style="color: black; font-weight: bold; text-decoration: underline;">/branches/1.8/main/pbx.c</a>
<span style="font-weight: normal;">
(Diff revision 5)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="4"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">static int handle_statechange(void *datap)</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#f0f0f0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">4269</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_lock</span><span class="p">(</span><span class="n">hints</span><span class="p">);</span></pre></td>
<th bgcolor="#f0f0f0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4269</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_lock</span><span class="p">(</span><span class="n">hints</span><span class="p">);</span></pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">4270</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_lock</span><span class="p">(</span><span class="n">hint</span><span class="p">)<span class="hl">;</span></span></pre></td>
<th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4270</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "><span class="hl"> </span><span class="k"><span class="hl">while</span></span><span class="hl"> </span><span class="p"><span class="hl">(</span></span><span class="n">ao2_<span class="hl">try</span>lock</span><span class="p">(</span><span class="n">hint</span><span class="p">)<span class="hl">)</span></span><span class="hl"> </span><span class="p"><span class="hl">{</span></span></pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4271</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_unlock</span><span class="p">(</span><span class="n">hints</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4272</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">usleep</span><span class="p">(</span><span class="mi">1</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4273</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">ao2_lock</span><span class="p">(</span><span class="n">hints</span><span class="p">);</span></pre></td>
</tr>
<tr>
<th bgcolor="#b1ebb0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2"></font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#b1ebb0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4274</font></th>
<td bgcolor="#c5ffc4" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="p">}</span></pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#f0f0f0" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">4271</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
<th bgcolor="#f0f0f0" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">4275</font></th>
<td bgcolor="#ffffff" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "></pre></td>
</tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">No dead lock avoidance is necessary here. The container lock held before the element lock is the correct locking order.</pre>
</blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">yes this is the correct locking order i wanted to initially swap it arround once again ao2_callback does not follow this rule nice when removing a hint the hint is held then callback grabs the hints and a deadlock is born this avoids it not fixes the underlying problem.
</pre>
<br />
<p>- irroot</p>
<br />
<p>On April 13th, 2011, 1:29 p.m., irroot wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: url('https://reviewboard.asterisk.orgrb/images/review_request_box_top_bg.png'); background-position: left top; background-repeat: repeat-x; border: 1px black solid;">
<tr>
<td>
<div>Review request for Asterisk Developers, Russell Bryant and Brett Bryant.</div>
<div>By irroot.</div>
<p style="color: grey;"><i>Updated 2011-04-13 13:29:53</i></p>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Can some one give me a good reason why the bellow order should not be reversed ??
ao2_callback runs on various threads holding container locks we must make sure we
lock the element before locking a container i have a feeling asterisk could be riddled
with these deadlocks / race conditions ....
/* disclaimer the first lock output is from a patched app_directed_pickup
*
* the changes use a callback the same as features.c
*
*/
=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0xab37bb70 (pbx_thread started at [ 5038] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 6241 ast_do_masquerade channels 0x89de1d0 (1)
=== ---> Waiting for Lock #1 (channel.c): MUTEX 6244 ast_do_masquerade original 0x92fa3a8 (1)
=== --- ---> Locked Here: app_directed_pickup.c line 322 (pickup_by_group)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xab597b70 (pbx_thread started at [ 5038] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (app_directed_pickup.c): MUTEX 155 pickup_do target 0x92fa3a8 (1)
=== --- ---> Locked Here: app_directed_pickup.c line 322 (pickup_by_group)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xab04bb70 (pbx_thread started at [ 5038] pbx.c ast_pbx_start())
=== ---> Lock #0 (app_directed_pickup.c): MUTEX 322 pickup_by_group target 0x92fa3a8 (1)
=== ---> Lock #1 (cdr.c): RDLOCK 1142 post_cdr &(&be_list)->lock 0x81f1cc8 (1)
=== ---> Waiting for Lock #2 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x89de1d0 (1)
=== --- ---> Locked Here: channel.c line 6241 (ast_do_masquerade)
=== -------------------------------------------------------------------
=======================================================================
=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0xb3c41b70 (tps_processing_function started at [ 451] taskprocessor.c ast_taskprocessor_get())
=== ---> Lock #0 (pbx.c): MUTEX 4269 handle_statechange hints 0x97a4b48 (1)
=== ---> Waiting for Lock #1 (pbx.c): MUTEX 4270 handle_statechange hint 0xb04cf7c0 (1)
=== --- ---> Locked Here: pbx.c line 4523 (ast_remove_hint)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb09f6b70 (do_monitor started at [24621] chan_sip.c restart_monitor())
=== ---> Lock #0 (chan_sip.c): MUTEX 24119 handle_request_do &netlock 0xb309fdc0 (1)
=== ---> Lock #1 (chan_sip.c): MUTEX 7593 find_call p 0xffb8d48 (1)
=== ---> Waiting for Lock #2 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x97a4b48 (1)
=== --- ---> Locked Here: pbx.c line 4269 (handle_statechange)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xb05c3b70 (netconsole started at [ 1344] asterisk.c listener())
=== ---> Lock #0 (loader.c): MUTEX 673 ast_module_reload &reloadlock 0x81f5280 (1)
=== ---> Lock #1 (loader.c): MUTEX 711 ast_module_reload &(&module_list)->lock 0x81f5208 (1)
=== ---> Lock #2 (pbx.c): MUTEX 4523 ast_remove_hint hint 0xb04cf7c0 (1)
=== ---> Waiting for Lock #3 (astobj2.c): MUTEX 657 internal_ao2_callback c 0x97a4b48 (1)
=== --- ---> Locked Here: pbx.c line 4269 (handle_statechange)
=== -------------------------------------------------------------------
===
=======================================================================
</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Ok this makes things a little more intresting need some more work
=======================================================================
=== Currently Held Locks ==============================================
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0xabd35b70 (do_monitor started at [24621] chan_sip.c restart_monitor())
=== ---> Lock #0 (chan_sip.c): MUTEX 24119 handle_request_do &netlock 0xb30f3dc0 (1)
=== ---> Lock #1 (channel.c): MUTEX 6233 ast_do_masquerade original 0xab409a30 (1)
=== ---> Waiting for Lock #2 (channel.c): MUTEX 6254 ast_do_masquerade channels 0xa1d31c0 (1)
=== --- ---> Locked Here: channel.c line 6254 (ast_do_masquerade)
=== -------------------------------------------------------------------
===
=== Thread ID: 0xaab97b70 (pbx_thread started at [ 5038] pbx.c ast_pbx_start())
=== ---> Lock #0 (channel.c): MUTEX 6254 ast_do_masquerade channels 0xa1d31c0 (1)
=== ---> Waiting for Lock #1 (channel.c): MUTEX 6233 ast_do_masquerade original 0xab409a30 (1)
=== --- ---> Locked Here: channel.c line 6233 (ast_do_masquerade)
=== -------------------------------------------------------------------
=======================================================================</pre>
</td>
</tr>
</table>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Bugs: </b>
<a href="https://issues.asterisk.org/view.php?id=18654">18654</a>,
<a href="https://issues.asterisk.org/view.php?id=19091">19091</a>
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/branches/1.8/channels/chan_sip.c <span style="color: grey">(313524)</span></li>
<li>/branches/1.8/main/channel.c <span style="color: grey">(313524)</span></li>
<li>/branches/1.8/main/pbx.c <span style="color: grey">(313524)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/1166/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>