[asterisk-bugs] [JIRA] (ASTERISK-27321) Asterisk Crashing with FRACK Errors and Serious Network Trouble

Wed Oct 18 11:44:21 CDT 2017

    [ https://issues.asterisk.org/jira/browse/ASTERISK-27321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=239430#comment-239430 ] 

Steven Sedory commented on ASTERISK-27321:
------------------------------------------

So looking at the log, I noticed that about 20 minutes after one of our users unplugged his phone, the Serious Network Error came up, and nine seconds later, the first FRACK.

[2017-10-18 06:32:31] NOTICE[2636] chan_sip.c: Peer '235' is now UNREACHABLE!  Last qualify: 23
[2017-10-18 06:32:31] VERBOSE[2559] chan_sip.c: Extension Changed 235[ext-local] new state Unavailable for Notify User 201 
[2017-10-18 06:53:14] WARNING[2636] chan_sip.c: sip_xmit of 0x7f6784903580 (len 759) to 76.168.37.93:59972 returned -2: Interrupted system call
[2017-10-18 06:53:14] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:15] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678421a740 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:15] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:16] WARNING[2636] chan_sip.c: sip_xmit of 0x7f678417cbc0 (len 760) to 76.168.37.93:59972 returned -2: No such file or directory
[2017-10-18 06:53:16] ERROR[2636] chan_sip.c: Serious Network Trouble; __sip_xmit returns error for pkt data
[2017-10-18 06:53:25] ERROR[7153] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x22c55a0 (0)
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: Got 21 backtrace records
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #0: [0x603fb2] /usr/sbin/asterisk(__ast_assert_failed+0x88) [0x603fb2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #1: [0x45d11a] /usr/sbin/asterisk() [0x45d11a]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #2: [0x45d147] /usr/sbin/asterisk() [0x45d147]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #3: [0x45e4f2] /usr/sbin/asterisk() [0x45e4f2]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #4: [0x45e729] /usr/sbin/asterisk(__ao2_link+0x43) [0x45e729]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #5: [0x45eb9c] /usr/sbin/asterisk() [0x45eb9c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #6: [0x45ee3f] /usr/sbin/asterisk(__ao2_callback+0x5f) [0x45ee3f]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #7: [0x7f672298a04c] /usr/lib64/asterisk/modules/chan_sip.so(+0x6d04c) [0x7f672298a04c]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #8: [0x7f6722989d8e] /usr/lib64/asterisk/modules/chan_sip.so(+0x6cd8e) [0x7f6722989d8e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #9: [0x4da4cc] /usr/sbin/asterisk(ast_cli_command_full+0x274) [0x4da4cc]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #10: [0x54d66e] /usr/sbin/asterisk() [0x54d66e]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #11: [0x5534fe] /usr/sbin/asterisk() [0x5534fe]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #12: [0x553e34] /usr/sbin/asterisk() [0x553e34]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #13: [0x5542ff] /usr/sbin/asterisk() [0x5542ff]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #14: [0x5ed787] /usr/sbin/asterisk() [0x5ed787]
[2017-10-18 06:53:26] VERBOSE[7153] logger.c: #15: [0x600bb4] /usr/sbin/asterisk() [0x600bb4]

> Asterisk Crashing with FRACK Errors and Serious Network Trouble
> ---------------------------------------------------------------
>
>                 Key: ASTERISK-27321
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-27321
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Channels/chan_sip/General
>    Affects Versions: 13.17.0
>         Environment: FreePBX 13.0.192.16 and Asterisk 13.17.0, proxmox 4.4 on Dell R720, local RAID volume. Using TCP and obscure port for SIP. UDP 5060 still open/enabled, but firewalled to only allow Anveo Direct servers.
>            Reporter: Steven Sedory
>            Assignee: Unassigned
>            Severity: Critical
>         Attachments: backtrace.txt, backtrace-with-debuginfo.txt, best-backtrace.txt, full
>
>
> Running FreePBX 13.0.192.16 and Asterisk 13.17.0
> I have previously posted about this issue in the freepbx and astersk forums. Here are those links:
> https://community.asterisk.org/t/asterisk-freepbx-crashing-and-frack-errors/72159
> https://community.freepbx.org/t/consistent-asterisk-freepbx-crash-issue/43682/1
> Host: Dell R720 with 2x Xeon E5-2620 2.00GHz (6 Core) and 64GB RAM DDR3 ECC), local PERC storage.
> Hypervisor: Proxmox 4.4-1.
> Network: using onboard Quad NIC. Bridge “vmbr0” points to “bond0” as the bridge port, and bond0 has eth0 and eth1 in it in “active-backup” mode, each going to one of our two core switches. Using Cisco 3560G. Switch ports are in trunk mode, with native vlan set to our management vlan. VMs are tagged to our public facing vlan, for direct internet access.
> VMs are running FreePBX/Asterisk versions mentioned above. Each have 4GB RAM fixed with ballooning disabled, 4 cores (2 sockets, 2 cores; have tried with NUMA enabled and disabled) with type “Default (kvm64)”, NIC using E1000 model, vdisk is 300G presented as ide0 as a raw image on a local LVM-Thin volume.
> Endpoints: All endpoints are NAT’d. We use TCP for SIP with an obscure port (not 5060 or near that). RTP traffic on our VSP’s required port range is allowed as well. All other traffic is dropped per the FreePBX firewall.
> In summary, what is happening is that we get a bunch of errors like this:
> [2017-09-28 02:05:18] ERROR[7061] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> [2017-09-28 02:05:24] ERROR[6934] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> [2017-09-28 02:05:28] ERROR[7107] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> and right before and after, we have most of our peers go unreachable. Sometime Asterisk will crash afterwards, sometimes not.
> The issue happens intermittently, but seems to happen more frequently on the VMs that have more peers/endpoints (100+). I don’t think we’ve had it happen on any VMs that had less than 100 peers/endpoints.
> We recently chopped a server that had about 130 endpoints into two of 110 and 20. More accurately, we moved 110 off server A to server B, leaving 20 on server B. Before that move, we were experiencing FRACK! errors every day (anywhere from 20-300, usually all within a 20 minute window or so). Once the 110 were moved to server B, server A has never again had FRACK! errors or asterisk crashes. Server B however is having them now, just much less often then when all 130 endpoints were on server A. My assumption for that is due to the slightly lower endpoint total on the VM.
> This morning was one of those instances. We had 193 errors, identical to the three I posted above (minus the ERROR[number] being different). AND, we had a crash afterwards. Here is the backtrace: http://pastebin.freepbx.org/view/8cccc15f2
> So I come to you, the asterisk community, for help. I first posted on the FreePBX forum, and was directed here.
> I understand this may point to a memory issue, but what is strange is that the Dell iDrac log doesn’t show any memory errors in it. Perhaps there are errors but iDrac just isn’t seeing them to report them. I’m hoping someone out there can parse through the backtrace and give me a clear answer to what the problem is. Thanks in advance.

--
This message was sent by Atlassian JIRA
(v6.2#6252)