<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:x="urn:schemas-microsoft-com:office:excel" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=Content-Type content="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<style>
<!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:SimSun;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"Lucida Sans Unicode";
panose-1:2 11 6 2 3 5 4 2 2 4;}
@font-face
{font-family:"\@SimSun";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:"Courier New";}
.MsoChpDefault
{mso-style-type:export-only;}
@page Section1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.Section1
{page:Section1;}
/* List Definitions */
@list l0
{mso-list-id:1366099249;
mso-list-type:hybrid;
mso-list-template-ids:1516958224 253107058 134807555 134807557 134807553 134807555 134807557 134807553 134807555 134807557;}
@list l0:level1
{mso-level-start-at:0;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-font-family:SimSun;
mso-bidi-font-family:"Times New Roman";}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-GB link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal>Masquerading is one of the difficult areas in understanding
Asterisk. I think I may have found a problem with it that is leading to a
double free of a channel, specifically where the same channel is both the clone
channel for a local channel optimisation, and the original channel for an AMI
Redirect. However, for a number of reasons (see below), I don’t
think I’m yet in a position to submit a formal bug report. What I
would appreciate is some views on whether I’m on the right track,
suggestions of how to verify it in lab conditions, and some more insight into
masquerading (particularly the reason for having separate prepare and execute
steps). Obviously, if someone recognizes they have fixed this, that is
also useful information.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Local channel optimisation involves masquerading the channel
bridged from one side of the local channel (original) into the other side of
the channel (clone). The basic current hypothesis is that, because Local
channel optimisation uses an implicit do_masquerade, and releases locks after
the preparation step, it is possible for an Redirect applied to the clone
channel to intervene, but the optimise to still try to complete. The Redirect
takes the optimise clone channel and masquerades it into a newly created channel.
It explicitly does the do_masquerade, retaining locks through the whole
process. I speculate that this confuses the upstream channel of the
optimise clone, which sees a hangup because of the Redirect masquerade, but
given the right timing, by the time that it tries to hang it up, it is dealing
with a channel that has had information overwritten by the optimise masquerade.
The net effect is a double free on the optimise clone channel, with a resulting
call of abort().<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>The prepare stage of masquerading checks to see whether the
original channel is an original for any other masquerade and similarly whether
the clone is a clone for any other masquerade. However, my speculation is
that it should also check to see whether the original is a clone for another
one, and probably vice versa.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Details of the scenario<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>We have a call that is set up as an Originate into a local
channel that then runs the Queue application. The ;1 side of that local
channel then runs a second local channel that dials a SIP device. From
now on, I don’t think the fact that there is a first local channel
matters. It is the thread that is running its ;1 side that ends up
issuing abort and there is no involvement of the ;2 side. In the CLI we
see:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>[2010-05-11
10:02:18.249] VERBOSE[10673] app_dial.c: --
SIP/siptrunk1-b6791708 answered Local/99999999999@xxx-e908;2<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'>Which
I believe starts the optimise sequence, although it requires an ast_write to
actually start the masquerade, and the trace doesn’t show where that happens.
I hypothesize that the prepare step of the optimise masquerade happens in this
interval.<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<pre>[2010-05-11 10:02:18.273] VERBOSE[10674] pbx.c: -- Executing [702@yyy:1] NoOp("Local/99999999999@xxx-e908;1) in new stack<o:p></o:p></pre>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:"Courier New"'><o:p> </o:p></span></p>
<p class=MsoNormal>Which is the first step of the Redirected dialplan,
indicating that a complete masquerade of the ;1 side of the second local
channel has happened. We wouldn’t have initiated the Redirect in
this way if we had known the call was answered, so there is a race condition
here, but one we would have coped with.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Further steps in this bit of dialplan are here, ending in a
ParkedCall call, which I don’t think is specifically relevant.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Then things get messy. Note this thread is the one
that should be hanging up because of the Redirect masquerade – it is the thread
running Dial, which is dialing the channel that is becoming a zombie:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<pre>[2010-05-11 10:02:18.278] WARNING[10672] channel.c: Channel 'SIP/siptrunk1-b6791708' may not have been hung up properly<o:p></o:p></pre><pre><o:p> </o:p></pre><pre><o:p> </o:p></pre><pre>[2010-05-11 10:02:18.279] VERBOSE[10674] features.c: -- Channel Local/99999999999@xxx;1 connected to parked call 702<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>This channel name is that of the channel that is being optimised out (although its original physical channel structure is the optimise clone).<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>At this point 10672 aborts, in the channel free from the ast_hangup from the normal end of call processing in app_dial.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>The dump shows that it is SIP/siptrunk1-b6791708 that it is trying to hang up. It has softhangup set and no bridge. This is the optimised out state of the local channel. tech_pvt is still set (consistent with the original SIP channel). My speculation is that the SIP channel never hung up, but this is actually the optimise clone channel that has had the cloning completed after it got soft hung up in its role as original channel for the Redirect masquerade.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>On this theory, checking for both masqr and masq in the preparation stage for the masquerade may avoid the problem, but as the comments say, masquerading is a seriously wacked out operation, and I don’t understand it well enough to be sure.<o:p></o:p></pre><pre><o:p> </o:p></pre><pre>The other question is, when do you actually need to separate the preparation for the masquerade from the do_masquerade. If the local channel had done both, with the locks held, there would have been no gap in which things could go wrong.<o:p></o:p></pre>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Reasons for not submitting this as a bug report yet are:<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>it has only happened once in several weeks of use and we
don’t know an efficient way of reproducing it;<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>it was on a customer system which means, for example,
we can’t run valgrind on it;<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>that system is based on 1.6.1.0;<o:p></o:p></p>
<p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span
style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'>
</span></span><![endif]>going to a later version of Asterisk is likely to cost
several man weeks in terms of re-integrating local modifications and re-running
the system test, and we are reluctant to do that for anything except the next
long terms stable release.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>Also, I cannot see any code changes in the areas relating to
our current hypothesis.<o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal>As this is a public mailing list, please ignore the part of
the following relating to confidentiality.<o:p></o:p></p>
<p class=MsoNormal><span style='font-family:"Lucida Sans Unicode","sans-serif"'>--
</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:"Lucida Sans Unicode","sans-serif"'>Dave
Woolley</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:"Lucida Sans Unicode","sans-serif"'>BTS
Holdings Plc</span><span style='font-family:"Arial","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:"Lucida Sans Unicode","sans-serif"'>Tel:
+44 (0)20 8401 9000 Fax: +44 (0)20 8401 9100</span><span style='font-family:
"Arial","sans-serif"'><o:p></o:p></span></p>
<p class=MsoNormal><span style='font-family:"Arial","sans-serif"'><a
href="http://www.bts.co.uk/"><span style='font-family:"Lucida Sans Unicode","sans-serif";
color:blue'>http://www.bts.co.uk</span></a></span><span style='font-family:
"Lucida Sans Unicode","sans-serif"'> </span><o:p></o:p></p>
<p class=MsoNormal><o:p> </o:p></p>
</div>
This email and its attachments may be confidential and are intended solely for the use of the individual to whom it is addressed. Any views or opinions expressed are solely those of the author and do not necessarily represent those of the company. If you are not the intended recipient of this email, you must take no action based upon it, nor must you copy or show it to anyone. Please contact the sender if you believe you have received this email in error. In accordance with English Law, email communications may be monitored. All reasonable precautions have been taken to ensure that no viruses are present in this email; however, the company cannot accept responsibility for loss or damage arising from the use of this email. We recommend that you subject this email to your own virus checking procedures. BTS Holdings PLC is registered in England 1517630, VAT No 523 5092 66. Registered office, BTS House, Manor Road, Wallington, SM6 0DD</body>
</html>