<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal">Hello, <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">We use Asterisk extensively for conferencing – for the last 8 years or so this has been the 1.4/1.6/1.8 releases running chan_sip and meetme for up to around 350 concurrent users. Right around that number DAHDI hit’s a hard coded memory
limit and kicks allocation errors in the log.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[Jun 22 10:04:13] WARNING[9095] app_meetme.c: Unable to open DAHDI pseudo channel: Cannot allocate memory<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">In order to support our growing user count we recently upgraded to 13.1-cert6 with pjsip and replaced meetme with confbridge. During all of our UAT and load testing everything seemed to be fine, there were no perceived audio quality issues
or any logs that would indicate an issue. Unfortunately now that we’re in production I’m getting consistent complaints that the audio from participants is cutting in and out. It only seems to occur while under load with > 350 users but that is anecdotal at
best. This is not a simple networking issue, we’ve pretty much ruled that out with various performance testing. That was not the case initially and we had incrementing UDP packet receive errors which we’ve eliminated with a bit of tuning.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There are numerous architectural differences between the two installations and so far I have not been successful in determining the root cause. I’m reaching out to the community and the developers for insight and feedback hoping there is
prior experience with this issue and how to resolve it. As you can see below the most significant difference is probably the use of VMware on the new install. I’ve tuned the ESXi host and guest per VMware’s recommendations for latency and jitter (full cpu/mem
reservations) with no improvement. With all of the reading I’ve done I suspect my issue may come down to a timing source and VMware not providing a reliable clock. It seems they allow a backlog of interrupts and if it hasn’t caught up in 60s they are simply
dropped. <o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Before I rip apart the environment and rebuild on physical I’d like to try and confirm that hypothesis. In the past this was a simple matter of running dahdi_test which would report the accuracy. I’m not sure how to interpret the results
of “timing test” in the Asterisk CLI. If I increase the number of ticks per second the results are erratic while under load. I’m using the timerfd module in Asterisk with a 1000HZ tick kernel and high res timers enabled. I’ve tried both hpet and tsc as system
clock sources, both exhibit the same breaks in audio. It sounds like someone presses the mute button in the middle of a sentence.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Any insight is appreciated!<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here are the specs on the new install:<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Physical HW Cisco UCS Blade (UCSB-B200-M3)<o:p></o:p></p>
<p class="MsoNormal">vMware ESXI 5.5<o:p></o:p></p>
<p class="MsoNormal">VM Guest 4 vCPU w/ 32G of RAM tuned for latency/jitter (sensitivity=high) and full cpu/memory reservations.<o:p></o:p></p>
<p class="MsoNormal">VM OS Redhat EL7 kernel 3.10.0-327.13.1.el7.x86_64 with tickless disabled e.g nohz=off and 1000HZ.
<o:p></o:p></p>
<p class="MsoNormal">Asterisk 13.1-cert6 using the timerfd module.
<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Regards<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">Robert McGilvray</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"><br>
SS&C GlobeOp <br>
Associate Director, IT Network Security<br>
<br>
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">GlobeOp Financial Services
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">|
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">1565 Front Street
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">|
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">Yorktown Hts NY 10598</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"><br>
<br>
t: </span><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">+1 (914)-293-3584
</span><span lang="EN-GB" style="font-size:8.0pt;font-family:"Arial","sans-serif";color:black"> </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">|
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">f: +1 (914)-293-3510<br>
</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">rmcgilvr@globeop.com</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC"> | </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><span style="color:#0A85C7"><a href="http://www.ssctech.com/"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">www.ssctech.com</span></a></span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">
|</span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">
</span><u><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7"><a href="http://www.sscglobeop.com/"><span style="color:#0A85C7">www.sscglobeop.com</span></a></span></u><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D">Follow us: </span><span style="color:#0A85C7"><a href="http://twitter.com/GlobeOp" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">Twitter</span></a></span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC"> |
</span><a href="http://www.facebook.com/pages/SSC-Technologies-Inc/191750415876" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">Facebook</span></a><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#CCCCCC">
| </span><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#1F497D"> </span><a href="http://www.linkedin.com/company/globeop-financial-services" target="_blank"><span style="font-size:8.0pt;font-family:"Arial","sans-serif";color:#0A85C7">LinkedIn</span></a><o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
This email with all information contained herein or attached hereto may contain confidential and/or privileged information intended for the addressee(s) only. If you have received this email in error, please contact the sender and immediately delete this email
in its entirety and any attachments thereto.
</body>
</html>