<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"><base href="x-msg://810/"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space; "><br><div><div>16 nov 2013 kl. 00:57 skrev Damon Estep <<a href="mailto:damon@soho-systems.com">damon@soho-systems.com</a>>:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I typically have 250 concurrent channels active per server.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">RTP is not handled in the sip channel, but qualify and the realtime database updates for SIP peers are.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">High RTP load won't affect SIP scheduling. High SIP load will.<o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Apples and Oranges.</span></div></div></div></blockquote>Yes, there's one single thread in chan_sip handling all the Qualify and inbound sip messages. If we moved the database updates to a background thread the situation would improve. Then the monitor thread would just move on to the next message, letting another thread wait for the database.</div><div><br></div><div>/O<br><blockquote type="cite"><div lang="EN-US" link="blue" vlink="purple" style="font-family: Helvetica; font-size: medium; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; "><div class="WordSection1" style="page: WordSection1; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "><o:p></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:asterisk-dev-bounces@lists.digium.com" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a><span class="Apple-converted-space"> </span>[mailto:asterisk-<a href="mailto:dev-bounces@lists.digium.com" style="color: purple; text-decoration: underline; ">dev-bounces@lists.digium.com</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Leandro Dardini<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, November 15, 2013 4:52 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Asterisk Developers Mailing List<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [asterisk-dev] scalability issue with realtime lastms update on qualify state change<o:p></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">I still have some doubts. When asterisk is bridging a simple phone call, a rtp voice packet leaves the asterisk box every 20ms, so it is 50 packets every second. My system has an average of 40 contemporary calls, so there are 2000 packets leaving the box every second and asterisk can keep them all running. I am sure the problem is what you explain, but I am not sure the source of the problem is the one you identify. <o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Leandro<o:p></o:p></div></div></div><div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></p><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">2013/11/16 Damon Estep <<a href="mailto:damon@soho-systems.com" target="_blank" style="color: purple; text-decoration: underline; ">damon@soho-systems.com</a>><o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Remember that when 1000 peers are offline there are an extra 7000 packets being SENT every 10s to try and reach them. Asterisk gets behind in processing the GOOD responses</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">It is too busy talking to listen :)</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The high load starts before the network outage comes back up and gets worse once it comes up, here is why:</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The network outage in this case is a regional event, impacting 20-30% of users.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">20-30% of users go unreachable; database is updated 400-600 times in a short period, no problem yet, but a higher MySQL load for a minute.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">400-600 peers go into fast qualify mode with no response, 7 packets each every 10 seconds, no responses.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Other 1400-1600 peers are engaged in normal OPTIONS dialogs, but the calculated RTTs(lastms) start climbing, due to the increased asterisk packet activity (network conditions for these peers have not changed, but observed lastms values progressively increase to well over 2 seconds).</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Now most of the peers are beyond maxms according to asterisk, but are not really according to network conditions.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Flapping between reachable/unreachable starts for all peers. as packets coming in are processed the ones that process in less than 2s are reachable, the rest are unreachable. Different ones every cycle.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Network recovers, and nothing changes, database load is way high, peers will flap for hours. In one case from 3AM to 5AM before we responded and restarted. The network event was 15 minutes, the impact was two hours, and the solution was a restart.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I can duplicate it on demand, but it is not very end user friendly to do so :(</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I am sure setting qualify higher (3000-5000) and modifying the hard coded DEFAULT_FREQ_NOTOK (from 10 to 60 or so) would also get things under control.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Either way, I have no use for lastms in the database, so why waste cycles writing it?</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">BTW, we upgraded from 1.2 to 1.8 and that is when this started. A packet capture during one of the events showed what was happening. 1.2 was doing everything the same, on the same schedules, with the same user count, but was not updating lastms in the DB.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a><span class="Apple-converted-space"> </span>[mailto:<a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Leandro Dardini<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, November 15, 2013 4:10 PM</span><o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br><b>To:</b><span class="Apple-converted-space"> </span>Asterisk Developers Mailing List<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [asterisk-dev] scalability issue with realtime lastms update on qualify state change<o:p></o:p></div></div></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Please read again your answer, because something is wrong ... on one side you are saying asterisk starts getting behind having to track the reponse of 7000 packets being sent every 10 seconds ... but if no answers are received due to network issue, is asterisk busy doing what? I don't think asterisk is becoming busy waiting for something not arriving. On the other side, you are blaming writes to the database for the slowdown. Are you seeing high load on the mysql database while the network is down or when the network is just back up?<o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">However the solution is pretty simple. Just patch chan_sip to not update lastms and test the system. I am pretty interested in the result.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Leandro<o:p></o:p></div></div></div><div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></p><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">2013/11/15 Damon Estep <<a href="mailto:damon@soho-systems.com" target="_blank" style="color: purple; text-decoration: underline; ">damon@soho-systems.com</a>><o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">It is not 1000 queries in 60 seconds, it starts as 1000 MORE queries in 60 seconds and compounds as it starts thrashing.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Thousands of other queries also taking place all the time. WAY over 300/second total.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Not DNS related, DNS is local, hostnames are IPs anyways, so no DNS needed.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">We have done the port 5060 block test, and can duplicate this very easily.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Tables are InnoDB, binary logged, and circular replication.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Remember that when 1000 peers are offline there are an extra 7000 packets being sent every 10s to try and reach them. Asterisk gets behind in processing the good responses, and the calculated RTT time for GOOD peers starts exceeding the maxms threshold, then the GOOD peers go offline too. Big snowball comes rolling down the hill and all is lost, all peers start flapping reachable/unreachable, and a quick restart of asterisk calms it down.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">The best confirmation that this is the issue is that it was resolved by patching chan_sip.c so it does not update lastms when state changes or when a qualify is not answered at all. Only updates on registration or registration expire.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Still doing more testing, but pretty confident this is the answer.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I understand that you are using the lastms in the realtime db for call routing, so my solution won't work for you. Probably need a configuration flag like rtupdatequalify=yes|no</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a><span class="Apple-converted-space"> </span>[mailto:<a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Leandro Dardini<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, November 15, 2013 3:47 PM</span><o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br><b>To:</b><span class="Apple-converted-space"> </span>Asterisk Developers Mailing List<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [asterisk-dev] scalability issue with realtime lastms update on qualify state change<o:p></o:p></div></div></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">My system are far less populated, so I think to be far from hitting your problems, but still, even after having read again your case, I can't believe asterisk is trashing that way. 1000 queries in 60 seconds are nothing compared with the load I have on the database even with far less peers. I had mysql databases (not voip related) running flawless with 300 queries every second... <o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Are you sure the trashing of your asterisk is not due to something other? I experienced problems with asterisk (not realtime) when the DNS servers were not reachable due to network outage... It will be nice to make a test... firewalling port 5060 inbound, so asterisk will think all peers are unreachable... I suspect it will not trash...<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Which kind of table type are you using? Are you still using MyISAM? Having a lots of contemporary "write" to a MyISAM table can bring a lots of slowdowns... just move to InnoDB.<o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">Leandro<o:p></o:p></div><div><p class="MsoNormal" style="margin: 0in 0in 12pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></p><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; ">2013/11/15 Damon Estep <<a href="mailto:damon@soho-systems.com" target="_blank" style="color: purple; text-decoration: underline; ">damon@soho-systems.com</a>><o:p></o:p></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">This is helpful, so I understand how some people are using lastms in the database.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I also have a multiserver solution, and I use INVITE (Dial) to see if the peer is registered. If the user is not reachable the Dial will return a status from the other box that can be handled.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">My experience is that a multi-server solution can't scale past limits in the range listed below the way it works now unless you never experience a network issue that makes 25% or more of the peers go unreachable:</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">10 servers, 2000 qualified peers per server</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">120 second qualify interval</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">qualify=3000</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">default retransmit time of 1 second</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Database load split between two reasonably powerful MySQL servers with circular replication enabled</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">at this scale, as soon as you have an event that takes down a large percentage of peers the database activity will back up, the qualify options packets will not be processed in real time, and the system will become unstable, with all peers flapping between reachable/unreachable, even after the network event is gone. It simply won't recover until you restart asterisk.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">I am sure the limits vary with environment, but there is a limit with the current strategy, and it is caused by excessive database updates of lastms during periods with high unreachable peer counts.</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); ">Have you seen this yet? What is the scale of your deployment?</span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><span style="font-size: 11pt; font-family: Calibri, sans-serif; color: rgb(31, 73, 125); "> </span><o:p></o:p></div><div style="border-style: none none none solid; border-left-width: 1.5pt; border-left-color: blue; padding: 0in 0in 0in 4pt; "><div><div style="border-style: solid none none; border-top-width: 1pt; border-top-color: rgb(181, 196, 223); padding: 3pt 0in 0in; "><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; ">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; "><span class="Apple-converted-space"> </span><a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a><span class="Apple-converted-space"> </span>[mailto:<a href="mailto:asterisk-dev-bounces@lists.digium.com" target="_blank" style="color: purple; text-decoration: underline; ">asterisk-dev-bounces@lists.digium.com</a>]<span class="Apple-converted-space"> </span><b>On Behalf Of<span class="Apple-converted-space"> </span></b>Leandro Dardini<br><b>Sent:</b><span class="Apple-converted-space"> </span>Friday, November 15, 2013 2:26 PM<br><b>To:</b><span class="Apple-converted-space"> </span>Asterisk Developers Mailing List<br><b>Subject:</b><span class="Apple-converted-space"> </span>Re: [asterisk-dev] scalability issue with realtime lastms update on qualify state change</span><o:p></o:p></div></div></div><div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; "><br>Il 15/nov/2013 22:00 "Damon Estep" <<a href="mailto:damon@soho-systems.com" target="_blank" style="color: purple; text-decoration: underline; ">damon@soho-systems.com</a>> ha scritto:<br>><br>> peer_lastms in memory tells you that, not lastms in the realtime database.<br>><br>> As far as I can see the lastms value in the realtime database is loaded only when the peer is created and quickly overwritten as soon as the first qualify is sent.<o:p></o:p></p><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">I have a realtime multiserver solution so I need to rely on database lastms otherwise how can I know if a peer usually registered on the other server is reachable or not?<o:p></o:p></p><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">Not only... on the web interface, the lastms is really pretty...<o:p></o:p></p><p style="margin-right: 0in; margin-left: 0in; font-size: 12pt; font-family: 'Times New Roman', serif; ">><br>> On Nov 15, 2013, at 1:29 PM, "Leandro Dardini" <<a href="mailto:ldardini@gmail.com" target="_blank" style="color: purple; text-decoration: underline; ">ldardini@gmail.com</a>> wrote:<br>><br>>><br>>> Il 15/nov/2013 21:11 "Damon Estep" <<a href="mailto:damon@soho-systems.com" target="_blank" style="color: purple; text-decoration: underline; ">damon@soho-systems.com</a>> ha scritto:<br>>> ><br>>> > I have run into an issue with 1.8.15 with mysql realtime peers, qualify=3000, and about 2000 peers.<br>>> ><br>>> > <br>>> ><br>>> > In the event of a network degradation event (high packet loss, network down) the system gets into an unusable state.<br>>> ><br>>> > <br>>> ><br>>> > let's say you have 2000 peers with qualify=yes (or 2000), and they are all reachable.<br>>> ><br>>> > <br>>> ><br>>> > Qualify frequency is 60 seconds (user definable), qualify unreachable is 10 seconds, and default retransmit timer of 1 second (both hardcoded in sip.h)<br>>> ><br>>> > <br>>> ><br>>> > 1000 peers go offline due to a network event<br>>> ><br>>> > <br>>> ><br>>> > The database has to be updated 1000 times in 60 seconds to mark the lastms -1 for these 1000 peers<br>>> ><br>>> > <br>>> ><br>>> > The same 1000 peers are now on a OPTIONS query schedule of once per 10 seconds, with 6 retransmits at 1 second intervals, total of 7 packets every 10 seconds for 1000 peers, or 700 new packets per second.<br>>> ><br>>> > <br>>> ><br>>> > So, we experienced a major network issue, and our response is to increase the load on the asterisk server dramatically by updating the database 1000 times and starting a new campaign to reach unreachable peers that are offline.<br>>> ><br>>> > <br>>> ><br>>> > In practice we see that when this happens, the calculated RTT time for peers that are not part of the network outage starts to increase (delay somewhere in hardware, code, who knows, but it does increase), and many of them start flapping between unreachable and reachable.<br>>> ><br>>> > <br>>> ><br>>> > The database query load goes through the roof, the number of packets coming out of the asterisk box goes through the roof, and asterisk will not recover until it is restarted, even after the network event is cleared.<br>>> ><br>>> > <br>>> ><br>>> > This is a new issue that came into play when lastms was added to the realtime database and the qualify code started updating it on every state change. 1.2 before lastms would handle this event gracefully, 1.8 wont, can't comment on 1.2 or 1.6.<br>>> ><br>>> > <br>>> ><br>>> > My thinking is that the lastms value in the db has little to no value. The only time I see it used is when a realtime peer is built from the database on registration. If the lastms value is -1 the peer registers and is set to unreachable, gets an option query and becomes reachable right away.<br>>> ><br>>> > <br>>> ><br>>> > rtupdate=no will stop the lastms updates, but it also stops the registration data from being updated in the database, which has unintended consequences.<br>>> ><br>>> > <br>>> ><br>>> > Also, the 10 second qualify timer when unreachable might make sense in small environments, but is way too aggressive for larger environments. It needs to be user configurable.<br>>> ><br>>> > <br>>> ><br>>> > Before I start patching, can anyone tell me what value lastms has and why it needs to be in the database?<br>>> ><br>>> > <br>>><br>>> I disagree about the no value for lastms. How can I know if a peer is reachable without the lastms field?<br>>><br>>> Leandro<br>>><br>>> --<span class="Apple-converted-space"> </span><br>>> _____________________________________________________________________<br>>> -- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" target="_blank" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br>>><br>>> asterisk-dev mailing list<br>>> To UNSUBSCRIBE or update options visit:<br>>> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br>><br>><br>> --<br>> _____________________________________________________________________<br>> -- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" target="_blank" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br>><br>> asterisk-dev mailing list<br>> To UNSUBSCRIBE or update options visit:<br>> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><o:p></o:p></p></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br>--<br>_____________________________________________________________________<br>-- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" target="_blank" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br><br>asterisk-dev mailing list<br>To UNSUBSCRIBE or update options visit:<br> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div></div></div></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br>--<br>_____________________________________________________________________<br>-- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" target="_blank" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br><br>asterisk-dev mailing list<br>To UNSUBSCRIBE or update options visit:<br> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "> <o:p></o:p></div></div></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><br>--<br>_____________________________________________________________________<br>-- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" target="_blank" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br><br>asterisk-dev mailing list<br>To UNSUBSCRIBE or update options visit:<br> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><o:p></o:p></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif; "><o:p> </o:p></div></div></div></div>--<span class="Apple-converted-space"> </span><br>_____________________________________________________________________<br>-- Bandwidth and Colocation Provided by<span class="Apple-converted-space"> </span><a href="http://www.api-digital.com" style="color: purple; text-decoration: underline; ">http://www.api-digital.com</a><span class="Apple-converted-space"> </span>--<br><br>asterisk-dev mailing list<br>To UNSUBSCRIBE or update options visit:<br> <a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" style="color: purple; text-decoration: underline; ">http://lists.digium.com/mailman/listinfo/asterisk-dev</a></div></blockquote></div><br><div apple-content-edited="true">
<span class="Apple-style-span" style="border-collapse: separate; color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: 2; text-align: auto; text-indent: 0px; text-transform: none; white-space: normal; widows: 2; word-spacing: 0px; -webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; -webkit-text-decorations-in-effect: none; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0; "><div>---</div><div>* Olle E Johansson - <a href="mailto:oej@edvina.net">oej@edvina.net</a></div><div>* Cell phone +46 70 593 68 51, Office +46 8 96 40 20, Sweden</div><div><br class="webkit-block-placeholder"></div></span><br class="Apple-interchange-newline">
</div>
<br></body></html>