<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>This is helpful, so I understand how some people are using lastms in the database.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I also have a multiserver solution, and I use INVITE (Dial) to see if the peer is registered. If the user is not reachable the Dial will return a status from the other box that can be handled.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>My experience is that a multi-server solution can't scale past limits in the range listed below the way it works now unless you never experience a network issue that makes 25% or more of the peers go unreachable:<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>10 servers, 2000 qualified peers per server<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>120 second qualify interval<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>qualify=3000<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>default retransmit time of 1 second<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Database load split between two reasonably powerful MySQL servers with circular replication enabled<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>at this scale, as soon as you have an event that takes down a large percentage of peers the database activity will back up, the qualify options packets will not be processed in real time, and the system will become unstable, with all peers flapping between reachable/unreachable, even after the network event is gone. It simply won't recover until you restart asterisk.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>I am sure the limits vary with environment, but there is a limit with the current strategy, and it is caused by excessive database updates of lastms during periods with high unreachable peer counts.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Have you seen this yet? What is the scale of your deployment?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0in 0in 0in 4.0pt'><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> asterisk-dev-bounces@lists.digium.com [mailto:asterisk-dev-bounces@lists.digium.com] <b>On Behalf Of </b>Leandro Dardini<br><b>Sent:</b> Friday, November 15, 2013 2:26 PM<br><b>To:</b> Asterisk Developers Mailing List<br><b>Subject:</b> Re: [asterisk-dev] scalability issue with realtime lastms update on qualify state change<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p><br>Il 15/nov/2013 22:00 "Damon Estep" <<a href="mailto:damon@soho-systems.com">damon@soho-systems.com</a>> ha scritto:<br>><br>> peer_lastms in memory tells you that, not lastms in the realtime database.<br>><br>> As far as I can see the lastms value in the realtime database is loaded only when the peer is created and quickly overwritten as soon as the first qualify is sent.<o:p></o:p></p><p>I have a realtime multiserver solution so I need to rely on database lastms otherwise how can I know if a peer usually registered on the other server is reachable or not?<o:p></o:p></p><p>Not only... on the web interface, the lastms is really pretty...<span style='color:#1F497D'><o:p></o:p></span></p><p>><br>> On Nov 15, 2013, at 1:29 PM, "Leandro Dardini" <<a href="mailto:ldardini@gmail.com">ldardini@gmail.com</a>> wrote:<br>><br>>><br>>> Il 15/nov/2013 21:11 "Damon Estep" <<a href="mailto:damon@soho-systems.com">damon@soho-systems.com</a>> ha scritto:<br>>> ><br>>> > I have run into an issue with 1.8.15 with mysql realtime peers, qualify=3000, and about 2000 peers.<br>>> ><br>>> > <br>>> ><br>>> > In the event of a network degradation event (high packet loss, network down) the system gets into an unusable state.<br>>> ><br>>> > <br>>> ><br>>> > let's say you have 2000 peers with qualify=yes (or 2000), and they are all reachable.<br>>> ><br>>> > <br>>> ><br>>> > Qualify frequency is 60 seconds (user definable), qualify unreachable is 10 seconds, and default retransmit timer of 1 second (both hardcoded in sip.h)<br>>> ><br>>> > <br>>> ><br>>> > 1000 peers go offline due to a network event<br>>> ><br>>> > <br>>> ><br>>> > The database has to be updated 1000 times in 60 seconds to mark the lastms -1 for these 1000 peers<br>>> ><br>>> > <br>>> ><br>>> > The same 1000 peers are now on a OPTIONS query schedule of once per 10 seconds, with 6 retransmits at 1 second intervals, total of 7 packets every 10 seconds for 1000 peers, or 700 new packets per second.<br>>> ><br>>> > <br>>> ><br>>> > So, we experienced a major network issue, and our response is to increase the load on the asterisk server dramatically by updating the database 1000 times and starting a new campaign to reach unreachable peers that are offline.<br>>> ><br>>> > <br>>> ><br>>> > In practice we see that when this happens, the calculated RTT time for peers that are not part of the network outage starts to increase (delay somewhere in hardware, code, who knows, but it does increase), and many of them start flapping between unreachable and reachable.<br>>> ><br>>> > <br>>> ><br>>> > The database query load goes through the roof, the number of packets coming out of the asterisk box goes through the roof, and asterisk will not recover until it is restarted, even after the network event is cleared.<br>>> ><br>>> > <br>>> ><br>>> > This is a new issue that came into play when lastms was added to the realtime database and the qualify code started updating it on every state change. 1.2 before lastms would handle this event gracefully, 1.8 wont, can't comment on 1.2 or 1.6.<br>>> ><br>>> > <br>>> ><br>>> > My thinking is that the lastms value in the db has little to no value. The only time I see it used is when a realtime peer is built from the database on registration. If the lastms value is -1 the peer registers and is set to unreachable, gets an option query and becomes reachable right away.<br>>> ><br>>> > <br>>> ><br>>> > rtupdate=no will stop the lastms updates, but it also stops the registration data from being updated in the database, which has unintended consequences.<br>>> ><br>>> > <br>>> ><br>>> > Also, the 10 second qualify timer when unreachable might make sense in small environments, but is way too aggressive for larger environments. It needs to be user configurable.<br>>> ><br>>> > <br>>> ><br>>> > Before I start patching, can anyone tell me what value lastms has and why it needs to be in the database?<br>>> ><br>>> > <br>>><br>>> I disagree about the no value for lastms. How can I know if a peer is reachable without the lastms field?<br>>><br>>> Leandro<br>>><br>>> -- <br>>> _____________________________________________________________________<br>>> -- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com">http://www.api-digital.com</a> --<br>>><br>>> asterisk-dev mailing list<br>>> To UNSUBSCRIBE or update options visit:<br>>> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>><br>><br>> --<br>> _____________________________________________________________________<br>> -- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com">http://www.api-digital.com</a> --<br>><br>> asterisk-dev mailing list<br>> To UNSUBSCRIBE or update options visit:<br>> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><o:p></o:p></p></div></div></body></html>