[asterisk-bugs] [JIRA] (ASTERISK-27321) Asterisk Crashing with FRACK Errors and Serious Network Trouble

[Richard Mudgett (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-27321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=239388#comment-239388 ] 

Richard Mudgett commented on ASTERISK-27321:
--------------------------------------------

The backtrace still does not have any symbolic information in it.  Before trying again you need to verify that you can get backtraces with the needed symbolic information.

>From your backtrace thread 1 (at the end of the file):
{noformat}
Thread 1 (Thread 0x7f855c0ca700 (LWP 15151)):
#0  0x00007f853ba85a43 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
#1  0x00007f853ba6ac66 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
#2  0x00007f853baee0c6 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
#3  0x00007f853baa886c in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
#4  0x00000000005c23de in ast_sched_runq ()
#5  0x00007f853baec405 in ?? () from /usr/lib64/asterisk/modules/chan_sip.so
#6  0x0000000000602d4c in ?? ()
#7  0x00007f85daa07aa1 in start_thread () from /lib64/libpthread.so.0
#8  0x00007f85d9d8f93d in clone () from /lib64/libc.so.6
{noformat}

>From ASTERISK-27346 core.FreePBX-2017-10-15T15-45-36-0500-brief.txt thread 1 (at the end of the file):
{noformat}
Thread 1 (Thread 0x7480ddc0 (LWP 4428)):
#0  0x728afef4 in iks_filter_add_rule () from /usr/lib/arm-linux-gnueabihf/libiksemel.so.3
#1  0x70bc621c in custom_connection_handler (opt=0x2262554, var=0x2266e28, obj=0x226330c) at chan_motif.c:2681
#2  0x00115d70 in aco_process_var (type=0x70bd9380 <endpoint_option>, cat=0x2266c30 "g********gmailcom", var=0x2266e28, obj=0x226330c) at config_options.c:743
#3  0x00115e60 in aco_process_category_options (type=0x70bd9380 <endpoint_option>, cfg=0x2263a90, cat=0x2266c30 "g********gmailcom", obj=0x226330c) at config_options.c:756
#4  0x001150c4 in process_category (cfg=0x2263a90, info=0x70bd944c <cfg_info>, file=0x70bd93c0 <jingle_conf>, cat=0x2266c30 "g********gmailcom", preload=0) at config_options.c:521
#5  0x001153ac in internal_process_ast_config (info=0x70bd944c <cfg_info>, file=0x70bd93c0 <jingle_conf>, cfg=0x2263a90) at config_options.c:560
#6  0x001159a0 in aco_process_config (info=0x70bd944c <cfg_info>, reload=0) at config_options.c:686
#7  0x70bc69bc in load_module () at chan_motif.c:2751
#8  0x00177550 in start_resource (mod=0x2181788) at loader.c:986
#9  0x001782e0 in load_resource_list (load_order=0x7ed4a794, global_symbols=0, mod_count=0x7ed4a78c) at loader.c:1238
#10 0x00178aa8 in load_modules (preload_only=0) at loader.c:1373
#11 0x0006f3d4 in asterisk_daemon (isroot=1, runuser=0x7ed4b988 "asterisk", rungroup=0x7ed4b978 "asterisk") at asterisk.c:4693
#12 0x0006e864 in main (argc=8, argv=0x7ed4ccb4) at asterisk.c:4443
{noformat}

See the difference?

To verify if you have symbolic information available, you can attach gdb to a running asterisk by "sudo gdb asterisk <asterisk-process-id>".  At the gdb command prompt you can then type "bt" to see the backtrace of the current thread.  If you have symbols available, the displayed backtrace will have function names, line numbers, and function parameter values similar to the example above.  To exit gdb issue the "quit" command.

[1] man gdb for gdb specific documentation
[2] https://wiki.asterisk.org/wiki/display/AST/Getting+a+Backtrace

> Asterisk Crashing with FRACK Errors and Serious Network Trouble
> ---------------------------------------------------------------
>
>                 Key: ASTERISK-27321
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-27321
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Channels/chan_sip/General
>    Affects Versions: 13.17.0
>         Environment: FreePBX 13.0.192.16 and Asterisk 13.17.0, proxmox 4.4 on Dell R720, local RAID volume. Using TCP and obscure port for SIP. UDP 5060 still open/enabled, but firewalled to only allow Anveo Direct servers.
>            Reporter: Steven Sedory
>            Assignee: Unassigned
>            Severity: Critical
>         Attachments: backtrace.txt, backtrace-with-debuginfo.txt, best-backtrace.txt
>
>
> Running FreePBX 13.0.192.16 and Asterisk 13.17.0
> I have previously posted about this issue in the freepbx and astersk forums. Here are those links:
> https://community.asterisk.org/t/asterisk-freepbx-crashing-and-frack-errors/72159
> https://community.freepbx.org/t/consistent-asterisk-freepbx-crash-issue/43682/1
> Host: Dell R720 with 2x Xeon E5-2620 2.00GHz (6 Core) and 64GB RAM DDR3 ECC), local PERC storage.
> Hypervisor: Proxmox 4.4-1.
> Network: using onboard Quad NIC. Bridge “vmbr0” points to “bond0” as the bridge port, and bond0 has eth0 and eth1 in it in “active-backup” mode, each going to one of our two core switches. Using Cisco 3560G. Switch ports are in trunk mode, with native vlan set to our management vlan. VMs are tagged to our public facing vlan, for direct internet access.
> VMs are running FreePBX/Asterisk versions mentioned above. Each have 4GB RAM fixed with ballooning disabled, 4 cores (2 sockets, 2 cores; have tried with NUMA enabled and disabled) with type “Default (kvm64)”, NIC using E1000 model, vdisk is 300G presented as ide0 as a raw image on a local LVM-Thin volume.
> Endpoints: All endpoints are NAT’d. We use TCP for SIP with an obscure port (not 5060 or near that). RTP traffic on our VSP’s required port range is allowed as well. All other traffic is dropped per the FreePBX firewall.
> In summary, what is happening is that we get a bunch of errors like this:
> [2017-09-28 02:05:18] ERROR[7061] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> [2017-09-28 02:05:24] ERROR[6934] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> [2017-09-28 02:05:28] ERROR[7107] astobj2.c: FRACK!, Failed assertion bad magic number 0x0 for object 0x3e7c690 (0)
> and right before and after, we have most of our peers go unreachable. Sometime Asterisk will crash afterwards, sometimes not.
> The issue happens intermittently, but seems to happen more frequently on the VMs that have more peers/endpoints (100+). I don’t think we’ve had it happen on any VMs that had less than 100 peers/endpoints.
> We recently chopped a server that had about 130 endpoints into two of 110 and 20. More accurately, we moved 110 off server A to server B, leaving 20 on server B. Before that move, we were experiencing FRACK! errors every day (anywhere from 20-300, usually all within a 20 minute window or so). Once the 110 were moved to server B, server A has never again had FRACK! errors or asterisk crashes. Server B however is having them now, just much less often then when all 130 endpoints were on server A. My assumption for that is due to the slightly lower endpoint total on the VM.
> This morning was one of those instances. We had 193 errors, identical to the three I posted above (minus the ERROR[number] being different). AND, we had a crash afterwards. Here is the backtrace: http://pastebin.freepbx.org/view/8cccc15f2
> So I come to you, the asterisk community, for help. I first posted on the FreePBX forum, and was directed here.
> I understand this may point to a memory issue, but what is strange is that the Dell iDrac log doesn’t show any memory errors in it. Perhaps there are errors but iDrac just isn’t seeing them to report them. I’m hoping someone out there can parse through the backtrace and give me a clear answer to what the problem is. Thanks in advance.



--
This message was sent by Atlassian JIRA
(v6.2#6252)