<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal">Thank you Joshua.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">I enabled better debugging and we’re running tests inhouse trying to replicate issue. This morning, I see the following happened during our internal load test.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Would this indicate system is too busy or a potential issue?<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[09/28 09:23:10.805] ERROR[3083] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.800] ERROR[3035] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.810] ERROR[3073][C-000001c3] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.810] ERROR[3153] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.814] ERROR[3083] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.818] ERROR[3035] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.814] ERROR[2884][C-0000018b] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.820] ERROR[3076][C-000001c1] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.812] ERROR[2896][C-00000190] frame.c: Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.819] ERROR[3073][C-000001c3] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.821] ERROR[3153] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.831] ERROR[2884][C-0000018b] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.834] ERROR[3076][C-000001c1] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:23:10.830] ERROR[2896][C-00000190] frame.c: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x55d42936e8d8 (0)<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">….<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[09/28 09:31:55.372] ERROR[3181] : Got 9 backtrace records<o:p></o:p></p>
<p class="MsoNormal"># 0: [0x55d428652177] asterisk utils.c:2454 __ast_assert_failed()<o:p></o:p></p>
<p class="MsoNormal"># 1: [0x55d4284bb742] asterisk astobj2.c:588 __ao2_ref()<o:p></o:p></p>
<p class="MsoNormal"># 2: [0x55d42856c10b] asterisk frame.c:349 ast_frdup()<o:p></o:p></p>
<p class="MsoNormal"># 3: [0x55d4284e7ecd] asterisk bridge_channel.c:1074 ast_bridge_channel_queue_frame()<o:p></o:p></p>
<p class="MsoNormal"># 4: [0x7faa79aabf40] bridge_softmix.so bridge_softmix.c:1939 softmix_mixing_loop()<o:p></o:p></p>
<p class="MsoNormal"># 5: [0x7faa79aac54d] bridge_softmix.so bridge_softmix.c:2047 softmix_mixing_thread()<o:p></o:p></p>
<p class="MsoNormal"># 6: [0x55d42864ef9b] asterisk utils.c:1299 dummy_start()<o:p></o:p></p>
<p class="MsoNormal"># 7: [0x7fab064b96db] libpthread.so.0 pthread_create.c:463 start_thread()<o:p></o:p></p>
<p class="MsoNormal"># 8: [0x7fab056ab71f] libc.so.6 clone.S:97 clone()<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[09/28 09:31:55.378] WARNING[2911][C-00000194] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__b0e7e0a4-e07d-4315-9692-d6208e2d7ac9-00000018;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.427] WARNING[3332][C-0000012c] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__14b6ae97-794b-44ea-83fd-6a17e02b43f1-00000095;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.441] WARNING[3095][C-000001cb] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__6c2e9575-ae69-40e8-b45e-b756ea9c6a0e-00000055;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.536] ERROR[3187] : Got 9 backtrace records<o:p></o:p></p>
<p class="MsoNormal"># 0: [0x55d428652177] asterisk utils.c:2454 __ast_assert_failed()<o:p></o:p></p>
<p class="MsoNormal"># 1: [0x55d4284bb742] asterisk astobj2.c:588 __ao2_ref()<o:p></o:p></p>
<p class="MsoNormal"># 2: [0x55d42856c10b] asterisk frame.c:349 ast_frdup()<o:p></o:p></p>
<p class="MsoNormal"># 3: [0x55d4284e7ecd] asterisk bridge_channel.c:1074 ast_bridge_channel_queue_frame()<o:p></o:p></p>
<p class="MsoNormal"># 4: [0x7faa79aabf40] bridge_softmix.so bridge_softmix.c:1939 softmix_mixing_loop()<o:p></o:p></p>
<p class="MsoNormal"># 5: [0x7faa79aac54d] bridge_softmix.so bridge_softmix.c:2047 softmix_mixing_thread()<o:p></o:p></p>
<p class="MsoNormal"># 6: [0x55d42864ef9b] asterisk utils.c:1299 dummy_start()<o:p></o:p></p>
<p class="MsoNormal"># 7: [0x7fab064b96db] libpthread.so.0 pthread_create.c:463 start_thread()<o:p></o:p></p>
<p class="MsoNormal"># 8: [0x7fab056ab71f] libc.so.6 clone.S:97 clone()<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[09/28 09:31:55.557] WARNING[3195][C-00000130] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__9cb014dc-f022-4b36-8abb-81a184ad6ea8-00000071;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.585] WARNING[3323][C-0000016c] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__6055546c-d560-4074-bbc5-3a8b33b6759b-00000092;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.612] WARNING[3076][C-000001c1] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__46a6eaad-3f85-4941-aa45-92ab814b6b0f-0000004f;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.637] ERROR[2898] : Got 8 backtrace records<o:p></o:p></p>
<p class="MsoNormal"># 0: [0x55d428652177] asterisk utils.c:2454 __ast_assert_failed()<o:p></o:p></p>
<p class="MsoNormal"># 1: [0x55d4284bb742] asterisk astobj2.c:588 __ao2_ref()<o:p></o:p></p>
<p class="MsoNormal"># 2: [0x7faa79aa75c0] bridge_softmix.so bridge_softmix.c:257 softmix_process_write_audio()<o:p></o:p></p>
<p class="MsoNormal"># 3: [0x7faa79aabef7] bridge_softmix.so bridge_softmix.c:1934 softmix_mixing_loop()<o:p></o:p></p>
<p class="MsoNormal"># 4: [0x7faa79aac54d] bridge_softmix.so bridge_softmix.c:2047 softmix_mixing_thread()<o:p></o:p></p>
<p class="MsoNormal"># 5: [0x55d42864ef9b] asterisk utils.c:1299 dummy_start()<o:p></o:p></p>
<p class="MsoNormal"># 6: [0x7fab064b96db] libpthread.so.0 pthread_create.c:463 start_thread()<o:p></o:p></p>
<p class="MsoNormal"># 7: [0x7fab056ab71f] libc.so.6 clone.S:97 clone()<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[09/28 09:31:55.638] WARNING[2872][C-00000189] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__b6b99750-c9aa-420c-9e9f-fa41e9a99fa9-0000000d;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.706] WARNING[3092][C-000001ca] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__43f4512f-4009-448f-9ccf-4d3ada975f74-00000054;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.764] WARNING[3231][C-00000161] channel.c: Exceptionally long voice queue length queuing to CBAnn/IS__57f2e3a7-6c86-423f-b35f-25048ef68591-0000007d;1<o:p></o:p></p>
<p class="MsoNormal">[09/28 09:31:55.771] ERROR[3208] : Got 8 backtrace records<o:p></o:p></p>
<p class="MsoNormal"># 0: [0x55d428652177] asterisk utils.c:2454 __ast_assert_failed()<o:p></o:p></p>
<p class="MsoNormal"># 1: [0x55d4284bb742] asterisk astobj2.c:588 __ao2_ref()<o:p></o:p></p>
<p class="MsoNormal"># 2: [0x7faa79aa75c0] bridge_softmix.so bridge_softmix.c:257 softmix_process_write_audio()<o:p></o:p></p>
<p class="MsoNormal"># 3: [0x7faa79aabef7] bridge_softmix.so bridge_softmix.c:1934 softmix_mixing_loop()<o:p></o:p></p>
<p class="MsoNormal"># 4: [0x7faa79aac54d] bridge_softmix.so bridge_softmix.c:2047 softmix_mixing_thread()<o:p></o:p></p>
<p class="MsoNormal"># 5: [0x55d42864ef9b] asterisk utils.c:1299 dummy_start()<o:p></o:p></p>
<p class="MsoNormal"># 6: [0x7fab064b96db] libpthread.so.0 pthread_create.c:463 start_thread()<o:p></o:p></p>
<p class="MsoNormal"># 7: [0x7fab056ab71f] libc.so.6 clone.S:97 clone()<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b>From:</b> asterisk-users <asterisk-users-bounces@lists.digium.com>
<b>On Behalf Of </b>Joshua C. Colp<br>
<b>Sent:</b> Thursday, September 23, 2021 12:03 PM<br>
<b>To:</b> Asterisk Users Mailing List - Non-Commercial Discussion <asterisk-users@lists.digium.com><br>
<b>Subject:</b> Re: [asterisk-users] Any thoughts on Asterisk 16.17.0 outputting FRACK refcount related messages<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal">On Thu, Sep 23, 2021 at 1:59 PM Dan Cropp <<a href="mailto:dan@amtelco.com">dan@amtelco.com</a>> wrote:<o:p></o:p></p>
</div>
<div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We have an extremely busy/large customer. They run fine most of the time, but periodically asterisk will output FRACK refcount related messages. It doesn’t seem to be related
to the volume, because it’s not breaking during their peak times.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">When this happens, the system becomes unstable and they have to restart to get things resolved.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">To give an idea of the instability, we have seen INVITE/Trying responses in SIP messaging logs.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We tell Asterisk to answer via AMI, but Asterisk never sends the OK (even 24 seconds later it hasn’t sent).<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">Eventually the other send CANCEL of the call.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto"> <o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">We’ve now captured 4 different days where something like the following occurs.<o:p></o:p></p>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">1) Is there a good way to tell if this may be fixed in Asterisk 16.20.0 (short of upgrading)?<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">You could examine the changes between the two and see if any issues seem relevant.<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"> <o:p></o:p></p>
</div>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<div>
<p class="MsoNormal" style="mso-margin-top-alt:auto;mso-margin-bottom-alt:auto">2) Would this be something I should submit as an asterisk issue? Unfortunately, site is so busy capturing the debug will be very difficult (if not impossible) due to amount of
data.<o:p></o:p></p>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Without a backtrace, or debug logs it'd be really hard to look at. The message just means that an object was referenced a lot. Why that is - who knows. It could be that the system can't keep up for some reason, or there's an off-nominal
path, or something is deadlocked, or something else.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">-- <o:p></o:p></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#073763">Joshua C. Colp</span><span style="font-family:"Tahoma",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#073763">Asterisk Technical Lead</span><span style="font-family:"Tahoma",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#073763">Sangoma Technologies</span><span style="font-family:"Tahoma",sans-serif"><o:p></o:p></span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Tahoma",sans-serif;color:#073763">Check us out at
<a href="http://www.sangoma.com" target="_blank">www.sangoma.com</a> and <a href="http://www.asterisk.org" target="_blank">
www.asterisk.org</a></span><span style="font-family:"Tahoma",sans-serif"><o:p></o:p></span></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<span style="font-size:10.0pt"><br>
This email is intended only for the use of the party to which it is addressed and may contain information that is privileged, confidential, or protected by law. If you are not the intended recipient you are hereby notified that any dissemination, copying or
distribution of this email or its contents is strictly prohibited. If you have received this message in error, please notify us immediately by replying to the message and deleting it from your computer.
</span>
</body>
</html>