[asterisk-bugs] [JIRA] (ASTERISK-27532) ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
Richard Mudgett (JIRA)
noreply at issues.asterisk.org
Tue Jan 16 19:15:50 CST 2018
[ https://issues.asterisk.org/jira/browse/ASTERISK-27532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=241584#comment-241584 ]
Richard Mudgett edited comment on ASTERISK-27532 at 1/16/18 7:15 PM:
---------------------------------------------------------------------
Hi,
We did upgrade to 13.18.4 and we are not seeing frack errors but we are still facing the issue . All of sudden between 9:30 to 10:30 AM ET the system CPU utilization goes to 100% . We are using queuemerics with asterisk to provide dynamic agent login and pause functionality. During high CPU utilization we saw multiple hung channels for queuemetrics context and high CPU IO wait. Attached the snippet of active channels and top command
See [^log_2018-01-15_top.txt]
See [^log_2018-01-15_core_show_channels.txt]
We are getting below intermittent warning in fulllogs
{noformat}
[2018-01-12 10:07:35] WARNING[20604] taskprocessor.c: The 'stasis-core-control' task processor queue reached 500 scheduled tasks again.
[2018-01-12 10:07:35] WARNING[20711][C-0001131a] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks again
{noformat}
Please help us to diagnose this furthur.
Regards.
Mohit
Edit: Extracted the pasted logs and attached them as text files per issue guidelines. Please *do not* paste large logs into issue comments it makes working with issues more difficult.
{quote}
Be courteous. Do not paste debug output in the description or a comment, instead please attach any debugging output as text files and reference them by file name.
{quote}
was (Author: mtaneja712):
Hi,
We did upgrade to 13.18.4 and we are not seeing frack errors but we are still facing the issue . All of sudden between 9:30 to 10:30 AM ET the system CPU utilization goes to 100% . We are using queuemerics with asterisk to provide dynamic agent login and pause functionality. During high CPU utilization we saw multiple hung channels for queuemetrics context and high CPU IO wait. Attached the snippet of active channels and top command
[root at is01-astrsk-001 ~]# top
top - 10:33:15 up 6 days, 10:40, 4 users, load average: 270.12, 223.60, 137.14
Tasks: 370 total, 2 running, 368 sleeping, 0 stopped, 0 zombie
Cpu0 : 5.0%us, 3.0%sy, 0.0%ni, 0.0%id, 91.3%wa, 0.0%hi, 0.7%si, 0.0%st
Cpu1 : 12.9%us, 1.4%sy, 0.0%ni, 0.0%id, 85.4%wa, 0.0%hi, 0.3%si, 0.0%st
Cpu2 : 2.6%us, 1.6%sy, 0.0%ni, 0.0%id, 95.7%wa, 0.0%hi, 0.0%si, 0.0%st
Cpu3 : 4.7%us, 1.4%sy, 0.0%ni, 0.0%id, 93.6%wa, 0.0%hi, 0.3%si, 0.0%st
Mem: 12198032k total, 10398464k used, 1799568k free, 262508k buffers
Swap: 786428k total, 368k used, 786060k free, 9059464k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
2288 asterisk 20 0 3151m 186m 17m S 31.9 1.6 1766:26 asterisk
25923 asterisk 20 0 356m 51m 9480 S 1.0 0.4 1:24.58 php
19 root 20 0 0 0 0 R 0.3 0.0 0:27.75 events/0
1788 root 30 10 431m 9448 1680 S 0.3 0.1 7:10.84 uniloader
2643 root 20 0 1222m 13m 2428 S 0.3 0.1 13:05.13 fail2ban-server
26456 root 20 0 15264 1536 992 R 0.3 0.0 0:00.17 top
1 root 20 0 19360 1372 1132 S 0.0 0.0 0:01.03 init
2 root 20 0 0 0 0 S 0.0 0.0 0:00.97 kthreadd
3 root RT 0 0 0 0 S 0.0 0.0 0:02.73 migration/0
4 root 20 0 0 0 0 S 0.0 0.0 0:08.20 ksoftirqd/0
5 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/0
6 root RT 0 0 0 0 S 0.0 0.0 0:00.62 watchdog/0
7 root RT 0 0 0 0 S 0.0 0.0 0:03.28 migration/1
8 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/1
9 root 20 0 0 0 0 S 0.0 0.0 0:10.83 ksoftirqd/1
10 root RT 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/1
11 root RT 0 0 0 0 S 0.0 0.0 0:02.86 migration/2
12 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/2
13 root 20 0 0 0 0 S 0.0 0.0 0:09.57 ksoftirqd/2
14 root RT 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/2
15 root RT 0 0 0 0 S 0.0 0.0 0:02.99 migration/3
16 root RT 0 0 0 0 S 0.0 0.0 0:00.00 stopper/3
17 root 20 0 0 0 0 S 0.0 0.0 1:02.08 ksoftirqd/3
18 root RT 0 0 0 0 S 0.0 0.0 0:00.54 watchdog/3
20 root 20 0 0 0 0 S 0.0 0.0 2:53.73 events/1
21 root 20 0 0 0 0 S 0.0 0.0 0:27.22 events/2
22 root 20 0 0 0 0 S 0.0 0.0 0:43.18 events/3
23 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/0
24 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/1
25 root 20 0 0 0 0 S 0.0 0.0 0:00.00 events/2
[root at is01-astrsk-001 ~]# asterisk -rx "core show channels verbose"
Channel Context Extension Prio State Application Data CallerID Duration Accountcode PeerAccount BridgeID
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770209|NONE|NO 00:17:31
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +14156997561 00:22:25
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770338|NONE|NO 00:15:23
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +14156997561 00:23:30
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770476|1515770 00:13:05
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770190|NONE|NO 00:17:51
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +18036178832 00:23:38
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770425|NONE|NO 00:13:55
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +12128249578 00:24:12
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770069|1515770 00:19:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770591|NONE|NO 00:11:09
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770583|NONE|NO 00:11:17
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770661|NONE|NO 00:09:59
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770756|NONE|NO 00:08:24
Local/945189 at from-qu from-queue 670029 1 Up AppQueue (Outgoing Line) 670000 00:39:06 fecb1220-3479-49ee-b
Local/945189 at from-qu macro-dialout-trunk s 23 Up Dial SIP/hqcm02/945189,300,M(s +19378230681 00:39:06 b0b10faf-0cdd-46fd-a
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515771160|1515771 00:01:41
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771248|NONE|NO 00:00:13
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771047|NONE|NO 00:03:33
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771160|NONE|NO 00:01:40
SIP/hqcm02-00006ee4 macro-dialout-trunk s 1 Up AppDial (Outgoing Line) 43874 01:04:46 70a9b39a-2489-40af-9
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770917|1515770 00:05:43
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770054|NONE|NO 00:20:06
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770119|NONE|NO 00:19:01
SIP/hqcm02-0000705b macro-dialout-trunk s 1 Up AppDial (Outgoing Line) 945189 00:39:06 b0b10faf-0cdd-46fd-a
Local/943141 at inbound macro-dialout-trunk s 23 Up Dial SIP/hqcm02/943141,300, 1111111111 01:15:38 6aad3735-fe37-44ca-8
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770236|NONE|NO 00:17:04
Local/943141 at inbound queuemetrics 11 7 Up ChanSpy Local/945180 at from-queue 01:15:38
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770306|NONE|NO 00:15:54
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770157|NONE|NO 00:18:23
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770259|1515770 00:16:41
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770412|NONE|NO 00:14:09
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515769875|1515769 00:23:06
SIP/hqcm02-00006e79 from-pstn 1 Up AppDial (Outgoing Line) 943141 01:15:38 6aad3735-fe37-44ca-8
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770704|NONE|NO 00:09:16
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770643|NONE|NO 00:10:18
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770608|NONE|NO 00:10:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770989|NONE|NO 00:04:31
SIP/hqcm02-000070b7 macro-dialout-trunk s 1 Up AppDial (Outgoing Line) 945518 00:27:59 aa146340-799b-47f7-9
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770949|NONE|NO 00:05:11
SIP/dc01hqccm003-000 ext-queues 680017 39 Up Queue 680017,t,,,,,,,, +19733772424 01:08:05 70a9b39a-2489-40af-9
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771077|NONE|NO 00:03:03
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771125|NONE|NO 00:02:16
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771220|NONE|NO 00:00:40
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771089|NONE|NO 00:02:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769896|NONE|NO 00:22:44
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769975|NONE|NO 00:21:25
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770110|NONE|NO 00:19:10
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770008|NONE|NO 00:20:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770229|NONE|NO 00:17:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770102|NONE|NO 00:19:19
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770287|NONE|NO 00:16:13
SIP/dc01hqccm003-000 ext-queues 670029 40 Up Queue 670029,t,,,,,,,, +12127522705 00:28:14 1e8ba47f-102c-4552-8
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770049|NONE|NO 00:20:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770401|NONE|NO 00:14:19
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770359|NONE|NO 00:15:01
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770504|NONE|NO 00:12:36
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770449|NONE|NO 00:13:31
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515769989|1515769 00:21:11
SIP/hqcm02-00007038 from-pstn 1 Up AppDial (Outgoing Line) 942908 00:42:39 aaa86d15-54e9-4b3c-b
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770110|1515770 00:19:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770542|NONE|NO 00:11:58
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770679|NONE|NO 00:09:41
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770855|NONE|NO 00:06:45
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770612|NONE|NO 00:10:49
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770927|NONE|NO 00:05:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770603|NONE|NO 00:10:57
SIP/dc01hqccm003-000 ext-queues 670029 40 Up Queue 670029,t,,,,,,,, +19378230681 00:39:57 534dbde6-9d00-497f-b
SIP/hqcm02-00006e15 from-pstn 1 Up AppDial (Outgoing Line) 943421 01:31:51 c1e07564-b9eb-4102-a
Local/943867 at from-qu from-queue 670029 1 Up AppQueue (Outgoing Line) 670000 00:54:10 5abe34c8-5289-45e0-b
Local/942908 at inbound queuemetrics 11 7 Up ChanSpy Local/945514 at from-queue 00:42:39
Local/943867 at from-qu macro-dialout-trunk s 23 Up Dial SIP/hqcm02/943867,300,M(s +12896886667 00:54:10 e563f888-e242-4d6d-9
Local/942908 at inbound macro-dialout-trunk s 23 Up Dial SIP/hqcm02/942908,300, 1111111111 00:42:39 aaa86d15-54e9-4b3c-b
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770576|1515770 00:11:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769892|NONE|NO 00:22:48
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770576|1515770 00:11:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770132|NONE|NO 00:18:48
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769880|NONE|NO 00:23:00
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769936|NONE|NO 00:22:04
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770089|NONE|NO 00:19:31
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770348|NONE|NO 00:15:12
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770426|NONE|NO 00:13:54
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770157|1515770 00:18:23
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770351|NONE|NO 00:15:10
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770489|NONE|NO 00:12:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770561|NONE|NO 00:11:40
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770065|1515770 00:19:55
SIP/dc01hqccm003-000 ivr-2 s 11 Up BackGround custom/csmainivr_20170713 +14166438784 00:00:05
SIP/dc01hqccm003-000 ivr-2 s 11 Up BackGround custom/csmainivr_20170713 +15129847276 00:00:06
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670036-201801 +13033021763 00:00:09
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +16172531610 00:00:18
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770760|NONE|NO 00:08:20
SIP/dc01hqccm003-000 ivr-2 s 11 Up BackGround custom/csmainivr_20170713 +16467396603 00:00:20
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13175383334 00:01:20
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +12177531190 00:01:29
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +12038458229 00:01:30
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +12026381616 00:01:32
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +16174570431 00:01:55
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771173|NONE|NO 00:01:27
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770990|1515770 00:04:30
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770976|1515770 00:04:44
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770579|1515770 00:11:22
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769906|NONE|NO 00:22:34
Local/943656 at inbound macro-dialout-trunk s 23 Up Dial SIP/hqcm02/943656,300, 1111111111 00:51:06 5d881f59-d2c7-48aa-b
Local/943656 at inbound queuemetrics 11 7 Up ChanSpy Local/945516 at from-queue 00:51:06
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770123|NONE|NO 00:18:57
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769855|NONE|NO 00:23:26
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770244|NONE|NO 00:16:57
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770307|NONE|NO 00:15:53
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670031-201801 +15086836306 00:00:31
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680010-201801 +14109620202 00:00:31
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770178|NONE|NO 00:18:02
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670031-201801 +15129847276 00:00:36
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770418|NONE|NO 00:14:02
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +12038458229 00:00:37
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670070-201801 +16105036549 00:00:44
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +16174570431 00:00:50
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680036-201801 +34936261940 00:00:58
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670010-201801 +19057856991 00:01:04
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +14166438784 00:01:12
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770590|NONE|NO 00:11:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770706|NONE|NO 00:09:14
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770883|NONE|NO 00:06:17
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680036-201801 +34936261940 00:02:09
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770967|NONE|NO 00:04:53
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770958|NONE|NO 00:05:02
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670016-201801 +12177531190 00:02:48
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680010-201801 +14109620202 00:02:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771132|NONE|NO 00:02:08
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771230|NONE|NO 00:00:30
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670070-201801 +16105036549 00:02:24
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +14166438784 00:02:37
Local/670000 at from-in sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670069-201801 670000 00:04:08
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770529|1515770 00:12:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770020|NONE|NO 00:20:40
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769966|NONE|NO 00:21:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770289|NONE|NO 00:16:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770150|NONE|NO 00:18:30
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770403|NONE|NO 00:14:17
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770461|NONE|NO 00:13:20
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770364|NONE|NO 00:14:56
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515769948|1515769 00:21:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770451|NONE|NO 00:13:29
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770542|NONE|NO 00:11:58
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770686|NONE|NO 00:09:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770864|NONE|NO 00:06:36
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16467396603 00:02:54
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770632|NONE|NO 00:10:29
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770944|NONE|NO 00:05:16
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680038-201801 +17348877134 00:04:13
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670016-201801 +12177531190 00:03:43
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771198|NONE|NO 00:01:02
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670010-201801 +16475355438 00:03:15
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771184|NONE|NO 00:01:17
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +4412277690 00:04:29
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670031-201801 +15086836306 00:04:31
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769893|NONE|NO 00:22:47
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769984|NONE|NO 00:21:16
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770138|NONE|NO 00:18:43
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769889|NONE|NO 00:22:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770219|NONE|NO 00:17:21
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770096|NONE|NO 00:19:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770280|NONE|NO 00:16:20
Local/943421 at inbound queuemetrics 11 7 Up ChanSpy Local/945511 at from-queue 01:31:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770138|NONE|NO 00:18:42
Local/943421 at inbound macro-dialout-trunk s 23 Up Dial SIP/hqcm02/943421,300, 1111111111 01:31:51 c1e07564-b9eb-4102-a
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770396|NONE|NO 00:14:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770267|NONE|NO 00:16:33
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770354|NONE|NO 00:15:07
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770491|NONE|NO 00:12:49
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770111|1515770 00:19:10
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770562|NONE|NO 00:11:39
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770064|1515770 00:19:56
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +4412277690 00:05:15
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770597|NONE|NO 00:11:03
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +14107666456 00:05:20
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770924|NONE|NO 00:05:37
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680032-201801 +15614141053 00:05:31
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +4412277690 00:05:42
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770598|NONE|NO 00:11:02
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770765|NONE|NO 00:08:15
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16365374002 00:05:44
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670010-201801 +16475355438 00:04:22
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +17348877134 00:06:11
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670031-201801 +15086836306 00:06:19
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515771080|1515771 00:03:00
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771179|NONE|NO 00:01:22
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13016520281 00:07:44
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13054364501 00:08:05
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16783334296 00:08:21
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +12483700910 00:08:25
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13016520281 00:08:49
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +4144629149 00:09:04
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770990|1515770 00:04:30
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16467396603 00:04:36
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770577|1515770 00:11:24
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +12172573702 00:04:47
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +14166438784 00:05:02
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769907|NONE|NO 00:22:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770075|NONE|NO 00:19:45
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770126|NONE|NO 00:18:54
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769862|NONE|NO 00:23:18
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770192|NONE|NO 00:17:49
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770077|NONE|NO 00:19:43
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770333|NONE|NO 00:15:27
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680032-201801 +19405358503 00:06:00
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770476|1515770 00:13:05
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770179|NONE|NO 00:18:01
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770257|NONE|NO 00:16:43
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770485|NONE|NO 00:12:55
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770590|NONE|NO 00:11:11
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +4412277690 00:06:25
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680032-201801 +19405358503 00:06:46
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770903|NONE|NO 00:05:57
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770655|NONE|NO 00:10:05
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16365374002 00:06:48
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +17348877134 00:07:00
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +14166438784 00:07:03
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680032-201801 +19405358503 00:07:05
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770963|NONE|NO 00:04:57
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16783334296 00:07:14
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +12483700638 00:07:32
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680032-201801 +19405358503 00:07:37
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q930000-201801 +12029862700 00:10:47
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16783334296 00:09:08
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q930000-201801 +12029862700 00:09:11
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13054364501 00:09:26
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +4144629149 00:10:14
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771149|NONE|NO 00:01:52
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13054364501 00:10:26
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771242|NONE|NO 00:00:18
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771105|NONE|NO 00:02:36
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +4144629149 00:12:16
SIP/hqcm02-00006f92 from-pstn 1 Up AppDial (Outgoing Line) 943656 00:51:06 5d881f59-d2c7-48aa-b
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13175081024 00:12:47
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770920|1515770 00:05:40
Local/670000 at from-in from-internal-xfer 670000 1 Up (None) (Empty) 00:01:16 fecb1220-3479-49ee-b
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670002-201801 +17574819000 00:12:49
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770594|1515770 00:11:06
Local/670000 at from-in sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670069-201801 670000 00:01:16
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770158|NONE|NO 00:18:22
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770300|NONE|NO 00:16:00
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770266|1515770 00:16:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770409|NONE|NO 00:14:12
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +19417402901 00:17:05
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770468|NONE|NO 00:13:12
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13175081024 00:17:28
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770379|NONE|NO 00:14:42
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770214|1515770 00:17:26
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770452|NONE|NO 00:13:28
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770589|NONE|NO 00:11:11
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770694|NONE|NO 00:09:26
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770609|NONE|NO 00:10:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770872|NONE|NO 00:06:29
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770634|NONE|NO 00:10:27
SIP/hqcm02-00006f3e macro-dialout-trunk s 1 Up AppDial (Outgoing Line) 943867 00:54:10 e563f888-e242-4d6d-9
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770608|NONE|NO 00:10:53
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770986|NONE|NO 00:04:34
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +14156997561 00:19:51
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770944|NONE|NO 00:05:16
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771038|NONE|NO 00:03:42
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771065|NONE|NO 00:03:15
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771123|NONE|NO 00:02:17
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771220|NONE|NO 00:00:40
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771082|NONE|NO 00:02:58
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13015207599 00:14:16
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +17574819000 00:14:30
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13015207599 00:15:19
SIP/dc01hqccm003-000 macro-hangupcall s 4 Up Hangup (Empty) +4412273788 00:52:43
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +17574819000 00:15:41
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670008-201801 +14156997561 00:15:53
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680038-201801 +19417402901 00:16:06
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +13175081024 00:16:06
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16102937912 00:16:22
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16145145317 00:16:22
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +17178874519 00:16:39
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769894|NONE|NO 00:22:47
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769951|NONE|NO 00:21:49
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770225|NONE|NO 00:17:15
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770099|NONE|NO 00:19:22
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770287|NONE|NO 00:16:14
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770512|1515770 00:12:28
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 +16102937912 00:18:32
SIP/hqcm02-00007142 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670029-201801 945621 00:20:34
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770397|NONE|NO 00:14:23
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770357|NONE|NO 00:15:03
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770494|NONE|NO 00:12:46
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515769956|1515769 00:21:44
Local/942922 at custom- queuemetrics 28 5 Up System echo "1515770430|NONE|NO 00:13:52
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770448|NONE|NO 00:13:33
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770563|NONE|NO 00:11:37
Local/945518 at from-qu macro-dialout-trunk s 23 Up Dial SIP/hqcm02/945518,300,M(s +12127522705 00:27:59 aa146340-799b-47f7-9
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770542|NONE|NO 00:11:59
Local/945518 at from-qu from-queue 670029 1 Up AppQueue (Outgoing Line) 670000 00:27:59 1e8ba47f-102c-4552-8
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770807|NONE|NO 00:07:33
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670017-201801 +12128249578 00:21:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770768|NONE|NO 00:08:12
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q670031-201801 +15129847276 00:20:07
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515771183|NONE|NO 00:01:17
SIP/dc01hqccm003-000 macro-hangupcall s 4 Up Hangup (Empty) +4413155558 00:56:53
SIP/dc01hqccm003-000 ext-queues 670029 40 Up Queue 670029,t,,,,,,,, +12896886667 00:57:00 5abe34c8-5289-45e0-b
SIP/dc01hqccm003-000 sub-record-check recordcheck 19 Up MixMonitor 2018/01/12/q680015-201801 +19417402901 00:21:07
Local/25 at queuemetric queuemetrics 25 8 Up System echo "1515770576|1515770 00:11:24
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515769913|NONE|NO 00:22:27
Local/22 at queuemetric queuemetrics 22 4 Up System echo "1515770131|NONE|NO 00:18:49
303 active channels
291 active calls
90631 calls processed
We are getting below intermittent warning in fulllogs
[2018-01-12 10:07:35] WARNING[20604] taskprocessor.c: The 'stasis-core-control' task processor queue reached 500 scheduled tasks again.
[2018-01-12 10:07:35] WARNING[20711][C-0001131a] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks again
Please help us to diagnose this furthur.
Regards.
Mohit
> ERROR[5383][C-00003919] astobj2.c: Excessive refcount 100000 reached on ao2 object 0x142f648
> --------------------------------------------------------------------------------------------
>
> Key: ASTERISK-27532
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-27532
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: . I did not set the category correctly.
> Affects Versions: 13.17.0
> Environment: Sangoma Freepbx official distro FreePBX 13.0.192.16
> Asterisk 13.17.0
> Reporter: Mohit Taneja
> Assignee: Mohit Taneja
> Labels: fax, webrtc
> Attachments: console.txt, log_2018-01-15_core_show_channels.txt, log_2018-01-15_top.txt
>
>
> Hi
> We are recently installed Freepbx distro “FreePBX 13.0.192.16” running “Asterisk 13.17.0” on a x86_64 running Linux. From last 2 weeks the CPU suddenly goes to 100% utilization everyday between 9:50 AM EDT to 10:30 AM EDT.
> In the full logs we saw below error during the high CPU utilization. Once the error stop appearing the CPU agin goes to normal
> I have disabled CDR Reports and Call Recording Report modules but the problem is still happening
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list