<div dir="ltr"><div class="gmail_default" style="font-family:arial narrow,sans-serif"><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, Mar 2, 2016 at 2:56 AM, Ross Beer <span dir="ltr"><<a href="mailto:ross.beer@outlook.com" target="_blank">ross.beer@outlook.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div><div dir="ltr">Hi George,<br> <br>I have re-built the 'c1bf014ea08cf66835a6f000e2bd6c7da588da6b' commit and PJSIP and Asterisk hasn't crashed after reload. However it did take 25 mins to load.<br> <br>As requested I have opened a ticket for the realtime issue:<br> <br><a href="https://issues.asterisk.org/jira/browse/ASTERISK-25826" target="_blank">https://issues.asterisk.org/jira/browse/ASTERISK-25826</a></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:'arial narrow',sans-serif;display:inline">Got it, thanks.</div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"><br> <br>Basically, I think this could be resolved by a configuration option that stops sourcery/pjsip loading all peers at start-up as this is not needed for the current setup. This has been discussed before on the mailing list however it doesn't look like it progresses any further.<br></div></div></blockquote><div><br></div><div><span style="font-family:'arial narrow',sans-serif">If you're up for trying something, you can comment out the qualify_and_schedule_all function <div class="gmail_default" style="font-family:'arial narrow',sans-serif;display:inline">in </div>line<div class="gmail_default" style="font-family:'arial narrow',sans-serif;display:inline">s</div> 1135<div class="gmail_default" style="font-family:'arial narrow',sans-serif;display:inline">-1147</div> of res/res_pjsip/pjsip_options.c, then comment out its 2 references on lines 1245 and 1281. If that drops your startup times, then we know we're on the right track.</span><br></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"> <br>I would like to thank you for all of your help tying to identify the issue and hope that we can resolve it soon.<br></div></div></blockquote><div><br></div><div><div class="gmail_default" style="font-family:'arial narrow',sans-serif">No worries!</div></div><div> <br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div dir="ltr"> <br>Kind regards,<br> <br>Ross<br> <br><div><hr>From: <a href="mailto:george.joseph@fairview5.com" target="_blank">george.joseph@fairview5.com</a><br>Date: Tue, 1 Mar 2016 16:27:06 -0700<div><div class="h5"><br>To: <a href="mailto:asterisk-dev@lists.digium.com" target="_blank">asterisk-dev@lists.digium.com</a><br>Subject: Re: [asterisk-dev] Asterisk Segfault After PJSIP Commit 5241<br><br><div dir="ltr"><div style="font-family:arial narrow,sans-serif"><br></div><div><br><div>On Tue, Mar 1, 2016 at 3:07 PM, Ross Beer <span dir="ltr"><<a href="mailto:ross.beer@outlook.com" target="_blank">ross.beer@outlook.com</a>></span> wrote:<br><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div><div dir="ltr">ok,<br> <br>That took 15 mins to load and then crashed. This will be due to the <span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap">pjsip_dlg_create_uas_and_inc_lock <font color="#000000" face="Calibri">commit.</font></span></div></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">It should not have crashed. That commit had the fix for it. If it did crash with that commit, open a Jira issue and attach a full backtrace.</div></div><div> </div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div dir="ltr"><br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri"></font></span> <br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri">However 15 mins to start is a long time and would cause issues in a production environment.</font></span></div></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif;display:inline">Would you open a Jira issue on the realtime problem (if one isn't already open).</div></div><div><div style="font-family:"arial narrow",sans-serif;display:inline">I'm starting to look at alternatives.</div></div><div><div style="font-family:"arial narrow",sans-serif;display:inline"><br></div></div><div><div style="font-family:"arial narrow",sans-serif;display:inline"><br></div></div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div dir="ltr"><br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri"></font></span> <br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri">Thank you for your help here,</font></span><br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri"></font></span> <br><span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap"><font color="#000000" face="Calibri">Ross</font></span><br><br> <br><div><hr>From: <a href="mailto:george.joseph@fairview5.com" target="_blank">george.joseph@fairview5.com</a><br>Date: Tue, 1 Mar 2016 14:02:38 -0700<div><div><br>To: <a href="mailto:asterisk-dev@lists.digium.com" target="_blank">asterisk-dev@lists.digium.com</a><br>Subject: Re: [asterisk-dev] Asterisk Segfault After PJSIP Commit 5241<br><br><div dir="ltr"><div style="font-family:"arial narrow",sans-serif"><br></div><div><br><div>On Tue, Mar 1, 2016 at 1:04 PM, Ross Beer <span dir="ltr"><<a href="mailto:ross.beer@outlook.com" target="_blank">ross.beer@outlook.com</a>></span> wrote:<br><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div><div dir="ltr">Hi George,<br> <br>Using a development test box for testing!!<br> <br>Asterisk 13.7.2 with no cache takes 4:12 to load, that with PJSIP Commit 5240<br></div></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">Ok, try this combination...</div><div style="font-family:"arial narrow",sans-serif">"git checkout c1bf014ea08cf66835a6f000e2bd6c7da588da6b"</div><div style="font-family:"arial narrow",sans-serif">pjproject from trunk.</div><div style="font-family:"arial narrow",sans-serif">with caching.</div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">The commit I referenced is the one that handles the <span style="color:rgb(53,53,53);font-family:monospace;white-space:pre-wrap">pjsip_dlg_create_uas_and_inc_lock</span></div><div style="font-family:"arial narrow",sans-serif"></div><br></div><div><br></div><div><br></div><div> </div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div dir="ltr"> <br>Qualify time on the aor is set to zero, I guess a query could be made to check for a value greater than zero instead of loading all endpoints.<br> <br>Ross<br> <br><div><hr>From: <a href="mailto:george.joseph@fairview5.com" target="_blank">george.joseph@fairview5.com</a><br>Date: Tue, 1 Mar 2016 12:45:28 -0700<div><div><br>To: <a href="mailto:asterisk-dev@lists.digium.com" target="_blank">asterisk-dev@lists.digium.com</a><br>Subject: Re: [asterisk-dev] Asterisk Segfault After PJSIP Commit 5241<br><br><div dir="ltr"><div style="font-family:"arial narrow",sans-serif"><br></div><div><br><div>On Tue, Mar 1, 2016 at 12:21 PM, Ross Beer <span dir="ltr"><<a href="mailto:ross.beer@outlook.com" target="_blank">ross.beer@outlook.com</a>></span> wrote:<br><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid">
<div><div dir="ltr">Hi George,<br> <br>No endpoints are qualified, there are 20,000 endpoints with only 75 static contacts defined in the aors. The database is a MySQL cluster.<br> <br>With the current Asterisk 13 branch with cache disabled and the latest PJSIP it takes 5 mins and then before finishing it crashes.<br> <br>With Asterisk 13.7.2 with cache it takes around 1 1/2 min to load, however due to the bug with PJSIP Commit 5241 asterisk crashes when using TLS devices.<br></div></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">Try 13.7.2 without the cache. I'm trying to understand where the time is being spent. I know it will crash because of that bug. You're not doing this on a production system are you??</div></div><div> </div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div dir="ltr"> <br>The main issue here is that the endpoints are loaded as soon as PJSIP loads, ideally endpoints would only be loaded once a device registers or attempts to make a call. Much in the same way as Asterisk 1.8 chan_sip manages realtime.<br> <br>There is no need to load the endpoints as they are not qualified.<br></div></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">How do you know they're not qualified if you don't load them? :)</div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">Time to load up a database with 20,000 endpoints I guess.</div></div><div> </div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div><div dir="ltr"> <br>Ross<br> <br><div><hr>From: <a href="mailto:george.joseph@fairview5.com" target="_blank">george.joseph@fairview5.com</a><br>Date: Tue, 1 Mar 2016 11:58:15 -0700<span><br>To: <a href="mailto:asterisk-dev@lists.digium.com" target="_blank">asterisk-dev@lists.digium.com</a><br>Subject: Re: [asterisk-dev] Asterisk Segfault After PJSIP Commit 5241<br><br></span><div dir="ltr"><div style="font-family:"arial narrow",sans-serif"><br></div><div><div><div><br><div>On Tue, Mar 1, 2016 at 11:38 AM, Michael Ulitskiy <span dir="ltr"><<a href="mailto:mulitskiy@acedsl.com" target="_blank">mulitskiy@acedsl.com</a>></span> wrote:<br><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><u></u>
<div style="font-family:"Sans Serif";font-size:11pt;font-style:normal;font-weight:400">
<p style="text-indent:0px">Hello,</p>
<p style="text-indent:0px"> </p>
<p style="text-indent:0px">Please see this discussion <a href="http://lists.digium.com/pipermail/asterisk-dev/2015-October/075122.html" target="_blank">http://lists.digium.com/pipermail/asterisk-dev/2015-October/075122.html</a></p>
<p style="text-indent:0px">I guess you're talking about the same problem.</p></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">It's possible.</div><br></div><div> <br></div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div style="font-family:"Sans Serif";font-size:11pt;font-style:normal;font-weight:400">
<p style="text-indent:0px"> </p>
<p style="text-indent:0px">Michael</p><span>
<p style="text-indent:0px"> </p>
<p style="text-indent:0px">On Tuesday, March 01, 2016 06:26:27 PM Ross Beer wrote:</p>
<p style="text-indent:0px">> Hi George,</p>
<p style="text-indent:0px">> </p>
<p style="text-indent:0px">> We need to store contacts in realtime for our system. However not all endpoints are registered only about 200, yet asterisk loops through every endpoint which has been defined. It does this if contacts are in realtime or not.</p>
<p style="text-indent:0px">> </p>
<p style="text-indent:0px">> Its almost like pjsip is loading them to check if they need to be qualified etc.</p>
<p style="text-indent:0px">> </p>
<p style="text-indent:0px">> Asterisk 1.8 only put things into cache once they were accessed, is this an option for sourcery?</p></span></div></blockquote><div><br></div><div><div style="font-family:"arial narrow",sans-serif">Well, in order to initiate qualify of contacts, Asterisk does have to "access" them all so I'm not quite sure what the problem is.</div><br></div><div><span style="font-family:"arial narrow",sans-serif">Can we reset to a known config and see what happens?</span><br></div><div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">pjproject from the published 2.4.5 tarball.</div><div style="font-family:"arial narrow",sans-serif">Asterisk from the published 13.7.2 tarball.</div><div style="font-family:"arial narrow",sans-serif">Disable memory_cache altogether in sorcery.conf.<br></div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">See what happens.</div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">Give me an estimate of how many endpoints and aors there are in the database, how many of those aors have static contacts defined, and what's your qualify interval.</div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">An idea of your database setup would help as well. Same server, local, remote, etc.</div><div style="font-family:"arial narrow",sans-serif"><br></div><div style="font-family:"arial narrow",sans-serif">Let's solve 1 problem at a time.</div></div><div><br></div><div> <br></div><blockquote style="padding-left:1ex;border-left-color:rgb(204,204,204);border-left-width:1px;border-left-style:solid"><div style="font-family:"Sans Serif";font-size:11pt;font-style:normal;font-weight:400"><span>
<p style="text-indent:0px"><br></p></span></div></blockquote></div></div></div></div></div><span>
<br>--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a></span></div> </div></div>
<br>--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br></blockquote></div><br></div></div>
<br>--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a></div></div></div> </div></div>
<br>--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br></blockquote></div><br></div></div>
<br>--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a></div></div></div> </div></div>
<br>--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br></blockquote></div><br></div></div>
<br>--
_____________________________________________________________________
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" target="_blank">http://www.api-digital.com</a> --
asterisk-dev mailing list
To UNSUBSCRIBE or update options visit:
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a></div></div></div> </div></div>
<br>--<br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
asterisk-dev mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-dev" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-dev</a><br></blockquote></div><br></div></div>