<div dir="ltr"><div dir="ltr"><br></div><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, Apr 2, 2020 at 11:34 AM Nick Olsen <<a href="mailto:nick@floridavirtualsolutions.com">nick@floridavirtualsolutions.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Paddy, It's pretty easy to spot from the CLI.<div><br></div><div>A voicemail gets called. And the screen basically stops scrolling from there. Eventually you'll get the "Task processors exceeded 500 queued tasks" or something like that. And maybe channels attempting to hangup due to lack of RTP (If you have no-rtp timers configured).</div><div><br></div><div>Once you find the problem mailbox, You can call it via any method and it'll deadlock every time as soon as it tries to play the mailboxes unavail greeting. I've never had it occur when there is no unavail greeting. Each case deleting the problem recording from the database fixes the issue, And subsequent recordings for the same mailbox have no issue.</div></div></blockquote><div><br></div><div>Given that the issue appears to be related to specific rows and not the database in general, you might want to get a backtrace while the system is locked as Josh suggested earlier. Once you get the backtraces, open an issue ar <a href="https://issues.asterisk.org">https://issues.asterisk.org</a>.</div><div><br></div><div><a href="https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace">https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace</a><br></div><div>NOTE: you do NOT need to recompile with the DEBUG_THREADS, MALLOC_DEBUG, DONT_OPTIMIZE or BETTER_BACKTRACES but the Asterisk binaries need to still have the symbols in them (un-stripped).</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><br clear="all"><div><div dir="ltr"><div dir="ltr"><b>Nick Olsen</b><div>Network Engineer</div><div>Office: 321-408-5000 x103</div><div>Mobile: 321-794-0763</div><div><img src="http://dl.floridavirtualsolutions.com/emaillogo/logo.png"><br></div></div></div></div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Apr 1, 2020 at 9:04 PM Paddy Grice <<a href="mailto:paddy@wizaner.com" target="_blank">paddy@wizaner.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><u></u>
<div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">Hi All</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">This sounds just like a problem I have had and still
investigating having moved to 16.9 using chan_sip. </font></span><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">I am still trying
to repeat the problem it looks from debug that the issue is either voicemail of
call transfer but I cant consistently repeat it. </font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">Voicemail is using ODBC and I just imported the data from
the old system into the new database.</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">Nick - if you have any more info I would be grateful
</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">TIA</font></span></div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;"></font></span> </div>
<div dir="ltr" align="left"><span><font face="Arial" color="#0000ff" size="2" style="--darkreader-inline-color:#337dff;">Paddy</font></span></div>
<div dir="ltr" align="left"><span></span> </div>
<div dir="ltr" align="left">
<hr>
</div>
<div dir="ltr" align="left"><font face="Tahoma" size="2"><b>From:</b> asterisk-users
[mailto:<a href="mailto:asterisk-users-bounces@lists.digium.com" target="_blank">asterisk-users-bounces@lists.digium.com</a>] <b>On Behalf Of </b>Nick
Olsen<br><b>Sent:</b> 01 April 2020 18:54<br><b>To:</b> Asterisk Users Mailing
List - Non-Commercial Discussion<br><b>Subject:</b> Re: [asterisk-users] PJSIP
Lockup<br></font><br></div>
<div></div>
<div dir="ltr">We ultimately found this to be a voicemail issue. The voicemail is
held in MYSQL as well (via ODBC). And we found when attempting to playback a
customers voicemail unavail greeting is when the deadlock would occur
(Immediately, every time. Throwing the same "task processors" errors, And
making pjsip completely unresponsive). We had imported a number of greetings
from a legacy asterisk system and the vast majority of them worked. When we
deleted the row containing the customers unavail greeting (making asterisk
revert to read the mailbox number) all issues went away. If we re-record the
customers unavail greeting it works fine and the problem doesn't reoccur. This
was one out of ~250 voicemails imported.
<div><br></div>
<div>Since then we've done a few more migrations and they've all gone smooth
with the exception of the most recent one. ~50% of the imported greetings have
caused asterisk to deadlock. We've been checking them now at time of
migration.</div>
<div><br></div>
<div>What I can't figure out is what it doesn't like about the greeting. It was
on a previous asterisk system working fine. The row looks identical to a working
one. The only thing I can guess is something about the blob for the recording
goes wrong. It would be nice if asterisk handled that more
gracefully. </div>
<div><br></div>
<div>I post this mostly just for internet history. To hopefully help the next
guy out who has this same issue.</div>
<div><br clear="all">
<div>
<div dir="ltr">
<div dir="ltr"><b>Nick Olsen</b>
<div>Network Engineer</div>
<div>Office: 321-408-5000 x103</div>
<div>Mobile: 321-794-0763</div>
<div><img src="http://dl.floridavirtualsolutions.com/emaillogo/logo.png"><br></div></div></div></div><br></div></div><br>
<div class="gmail_quote">
<div class="gmail_attr" dir="ltr">On Mon, Mar 2, 2020 at 8:29 PM Joshua C. Colp
<<a href="mailto:jcolp@sangoma.com" target="_blank">jcolp@sangoma.com</a>>
wrote:<br></div>
<blockquote class="gmail_quote" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)">
<div dir="ltr">
<div dir="ltr">On Mon, Mar 2, 2020 at 4:24 PM Nick Olsen <<a href="mailto:nick@floridavirtualsolutions.com" target="_blank">nick@floridavirtualsolutions.com</a>> wrote:<br></div>
<div class="gmail_quote">
<blockquote class="gmail_quote" style="padding-left:1ex;margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204)">
<div dir="ltr">Thanks for the info, Joshua.
<div><br></div>
<div>Does PJSIP handle database access the same way Chan_sip did? We had a
number of boxes running chan_sip referencing the same mysql server without
issue.</div>
<div><br></div>
<div>We're going to attempt to get a backtrace on the next occurance. We're
also going to run a local copy of the database on the same physical asterisk
instance and have the system reference it. Just to "throw everything at the
wall".</div></div></blockquote>
<div><br></div>
<div>It uses the same underlying API and layer. It can do more frequent
database access though due to queries and because PJSIP is
multithreaded.</div></div>
<div><br></div>-- <br>
<div dir="ltr">
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div dir="ltr">
<div style="font-family:tahoma,sans-serif"><font color="#073763" style="--darkreader-inline-color:#97cdf8;">Joshua C.
Colp</font></div>
<div style="font-family:tahoma,sans-serif"><font color="#073763" style="--darkreader-inline-color:#97cdf8;">Asterisk
Technical Lead</font></div>
<div style="font-family:tahoma,sans-serif"><font color="#073763" style="--darkreader-inline-color:#97cdf8;">Sangoma
Technologies</font></div>
<div style="font-family:tahoma,sans-serif"><font color="#073763" style="--darkreader-inline-color:#97cdf8;">Check us out
at <a href="http://www.sangoma.com" target="_blank">www.sangoma.com</a> and <a href="http://www.asterisk.org" target="_blank">www.asterisk.org</a></font><br></div></div></div></div></div></div></div></div></div></div></div>--
<br>_____________________________________________________________________<br>--
Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br><br>Check
out the new Asterisk community forum at: <a href="https://community.asterisk.org/" rel="noreferrer" target="_blank">https://community.asterisk.org/</a><br><br>New to Asterisk?
Start here:<br> <a href="https://wiki.asterisk.org/wiki/display/AST/Getting+Started" rel="noreferrer" target="_blank">https://wiki.asterisk.org/wiki/display/AST/Getting+Started</a><br><br>asterisk-users
mailing list<br>To UNSUBSCRIBE or update options visit:<br> <a href="http://lists.digium.com/mailman/listinfo/asterisk-users" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-users</a></blockquote></div></div>
-- <br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
Check out the new Asterisk community forum at: <a href="https://community.asterisk.org/" rel="noreferrer" target="_blank">https://community.asterisk.org/</a><br>
<br>
New to Asterisk? Start here:<br>
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+Started" rel="noreferrer" target="_blank">https://wiki.asterisk.org/wiki/display/AST/Getting+Started</a><br>
<br>
asterisk-users mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-users" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-users</a></blockquote></div>
-- <br>
_____________________________________________________________________<br>
-- Bandwidth and Colocation Provided by <a href="http://www.api-digital.com" rel="noreferrer" target="_blank">http://www.api-digital.com</a> --<br>
<br>
Check out the new Asterisk community forum at: <a href="https://community.asterisk.org/" rel="noreferrer" target="_blank">https://community.asterisk.org/</a><br>
<br>
New to Asterisk? Start here:<br>
<a href="https://wiki.asterisk.org/wiki/display/AST/Getting+Started" rel="noreferrer" target="_blank">https://wiki.asterisk.org/wiki/display/AST/Getting+Started</a><br>
<br>
asterisk-users mailing list<br>
To UNSUBSCRIBE or update options visit:<br>
<a href="http://lists.digium.com/mailman/listinfo/asterisk-users" rel="noreferrer" target="_blank">http://lists.digium.com/mailman/listinfo/asterisk-users</a></blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr" style="font-size:12.8px"><div dir="ltr" style="font-size:12.8px"><div style="font-family:tahoma,sans-serif;font-size:small"><span style="color:rgb(7,55,99)">George Joseph</span><br></div></div><div dir="ltr" style="font-size:small"></div><div style="font-family:tahoma,sans-serif;font-size:small"><span style="color:rgb(7,55,99)">Asterisk Software Developer</span><br></div><span style="color:rgb(7,55,99);font-family:tahoma,sans-serif;font-size:small">direct/fax +1 256 428 6012</span><br><div style="font-family:tahoma,sans-serif;font-size:small"><font color="#073763" style="--darkreader-inline-color:#97cdf8;">Check us out at</font> <a href="http://www.sangoma.com/" style="color:rgb(17,85,204)" target="_blank">www.sangoma.com</a> and <a href="http://www.asterisk.org" target="_blank">www.asterisk.org</a><br></div><div style="font-family:tahoma,sans-serif;font-size:small"><img src="cid:ii_k3abte590" alt="image.png" width="184" height="32" style="margin-right: 0px;"></div></div></div></div></div></div></div></div></div>