[asterisk-bugs] [JIRA] (ASTERISK-26831) res_rtp_asterisk: Race condition when RTCP and WebRTC is used
Badalian Vyacheslav (JIRA)
noreply at issues.asterisk.org
Tue Mar 14 06:30:13 CDT 2017
[ https://issues.asterisk.org/jira/browse/ASTERISK-26831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=235772#comment-235772 ]
Badalian Vyacheslav edited comment on ASTERISK-26831 at 3/14/17 6:30 AM:
-------------------------------------------------------------------------
i have 52GB refs logs.... now i try use refcounter.py and will post results here...
also i look to full debug log but it's not have any error or warnings. debug level 0 becouse it's porduction server and i can't increment level.
full log:
{code}
[2017-03-14 10:58:37] VERBOSE[6154][C-0000156b] netsock2.c: Using SIP RTP TOS bits 184
[2017-03-14 10:58:37] VERBOSE[6154][C-0000156b] netsock2.c: Using SIP RTP CoS mark 5
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:21] ExecIf("SIP/yuvchepizhnaya_wof-00002a28", "0?Set(CONNECTEDLINE(name,i)=CID:78007552771)") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:22] GotoIf("SIP/yuvchepizhnaya_wof-00002a28", "0?customtrunk") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:23] Dial("SIP/yuvchepizhnaya_wof-00002a28", "SIP/sbc/79064924469,300,") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] netsock2.c: Using SIP RTP TOS bits 184
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] netsock2.c: Using SIP RTP CoS mark 5
[2017-03-14 10:58:38] VERBOSE[6125][C-00001566] res_rtp_asterisk.c: 0x62600094e100 -- Probation passed - setting RTP source address to 10.216.88.62:58296
[2017-03-14 10:58:38] VERBOSE[6125][C-00001566] res_rtp_asterisk.c: 0x62600094e100 -- Probation passed - setting RTP source address to 10.216.88.62:58296
[2017-03-14 10:58:56] Asterisk 13.14.0 built by root @ ds-asterisk05.tcsbank.ru on a x86_64 running Linux on 2017-03-13 13:54:32 UTC
{code}
On our load (200-300 calls) we get 1-3 crashes in day. Its critical bug for us . Also we have premium digium support.
Crush log:
{code}
==4251==ERROR: AddressSanitizer: SEGV on unknown address 0x60e0610000e4 (pc 0x7f442d1f1818 bp 0x6290003b826c sp 0x7f442e77e478 T81)
#0 0x7f442d1f1817 in srtp_get_stream (/lib64/libsrtp.so.1+0x6817)
#1 0x7f442d1f2c78 in srtp_protect_rtcp (/lib64/libsrtp.so.1+0x7c78)
#2 0x7f442d4212ad in ast_srtp_protect /home/pbs.vbadalyan/asterisk-13.14.0/res/res_srtp.c:440
#3 0x7f4416f35a6e in __rtp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2305
#4 0x7f4416f42d14 in rtcp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2330
#5 0x7f4416f42d14 in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3282
#6 0x7f4416f4405c in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3349
#7 0x74c214 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
#8 0x7f440119b4bd in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
#9 0x7c22af in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
#10 0x7f44323e5dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
#11 0x7f44316c573c in clone (/lib64/libc.so.6+0xf773c)
AddressSanitizer can not provide additional info.
SUMMARY: AddressSanitizer: SEGV (/lib64/libsrtp.so.1+0x6817) in srtp_get_stream
Thread T81 created by T0 here:
#0 0x7f44345624a8 in pthread_create (/lib64/libasan.so.3+0x314a8)
#1 0x7cc4b4 in ast_pthread_create_stack /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1288
#2 0x7f44010cf9cc in restart_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29542
#3 0x7f440119c3af in restart_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29529
#4 0x7f440119c3af in load_module /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:35256
#5 0x65ad3f in start_resource /home/pbs.vbadalyan/asterisk-13.14.0/main/loader.c:1029
#6 0x65c2c7 in start_resource /home/pbs.vbadalyan/asterisk-13.14.0/main/loader.c:1022
#7 0x65c2c7 in load_resource_list /home/pbs.vbadalyan/asterisk-13.14.0/main/loader.c:1293
#8 0x65fd77 in load_modules /home/pbs.vbadalyan/asterisk-13.14.0/main/loader.c:1454
#9 0x438663 in asterisk_daemon /home/pbs.vbadalyan/asterisk-13.14.0/main/asterisk.c:4690
#10 0x438663 in main /home/pbs.vbadalyan/asterisk-13.14.0/main/asterisk.c:4446
#11 0x7f44315efb34 in __libc_start_main (/lib64/libc.so.6+0x21b34)
==4251==ABORTING
{code}
was (Author: slavon):
i have 52GB refs logs.... now i try use refcounter.py and will post results here...
also i look to full debug log but it's not have any error or warnings. debug level 0 becouse it's porduction server and i can't increment level.
full log:
{code}
[2017-03-14 10:58:37] VERBOSE[6154][C-0000156b] netsock2.c: Using SIP RTP TOS bits 184
[2017-03-14 10:58:37] VERBOSE[6154][C-0000156b] netsock2.c: Using SIP RTP CoS mark 5
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:21] ExecIf("SIP/yuvchepizhnaya_wof-00002a28", "0?Set(CONNECTEDLINE(name,i)=CID:78007552771)") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:22] GotoIf("SIP/yuvchepizhnaya_wof-00002a28", "0?customtrunk") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] pbx.c: Executing [s at macro-dialout-trunk:23] Dial("SIP/yuvchepizhnaya_wof-00002a28", "SIP/sbc/79064924469,300,") in new stack
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] netsock2.c: Using SIP RTP TOS bits 184
[2017-03-14 10:58:37] VERBOSE[6157][C-0000156c] netsock2.c: Using SIP RTP CoS mark 5
[2017-03-14 10:58:38] VERBOSE[6125][C-00001566] res_rtp_asterisk.c: 0x62600094e100 -- Probation passed - setting RTP source address to 10.216.88.62:58296
[2017-03-14 10:58:38] VERBOSE[6125][C-00001566] res_rtp_asterisk.c: 0x62600094e100 -- Probation passed - setting RTP source address to 10.216.88.62:58296
[2017-03-14 10:58:56] Asterisk 13.14.0 built by root @ ds-asterisk05.tcsbank.ru on a x86_64 running Linux on 2017-03-13 13:54:32 UTC
{code}
On our load (200-300 calls) we get 1-3 crashes in day. Its critical bug for us . Also we have premium digium support.
Crush log:
{code}
{code}
> res_rtp_asterisk: Race condition when RTCP and WebRTC is used
> -------------------------------------------------------------
>
> Key: ASTERISK-26831
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-26831
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Resources/res_rtp_asterisk
> Affects Versions: 13.14.0
> Reporter: Badalian Vyacheslav
> Assignee: Unassigned
> Severity: Minor
>
> Race condition. With this story, I am fighting for more than a month. I tried different options but the idea of an end. There is only a heavy load (more than 200 simultaneous calls WebRTC) after 50 000-100 000 calls.
> Please help. It is difficult to disentangle this tangle. The error for a long time (many versions).
> It may have an idea why this might happen? This is because rtp_stop or because rtp_destroy?
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x6290016c326c at pc 0x7f007c288b1b bp 0x7f00404275b0 sp 0x7f0040426d58
> WRITE of size 64 at 0x6290016c326c thread T467
> #0 0x7f007c288b1a (/lib64/libasan.so.3+0x5eb1a)
> #1 0x7f0074fdea26 in memcpy /usr/include/bits/string3.h:51
> #2 0x7f0074fdea26 in ast_srtp_protect /home/pbs.vbadalyan/asterisk-13.14.0/res/res_srtp.c:438
> #3 0x7f005c6e65de in __rtp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2305
> #4 0x7f005c6f5e8d in rtcp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2330
> #5 0x7f005c6f5e8d in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3286
> #6 0x7f005c6f6ecc in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3357
> #7 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
> #8 0x7f00469ad02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
> #9 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
> #10 0x7f007a4c2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
> #11 0x7f00797a273c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x603002c6d820 at pc 0x7f55310f2ff3 bp 0x7f552b6f2240 sp 0x7f552b6f19e8
> READ of size 3 at 0x603002c6d820 thread T88
> #0 0x7f55310f2ff2 (/lib64/libasan.so.3+0x3cff2)
> #1 0x7f552fbae4a4 (/lib64/libjansson.so.4+0x74a4)
> #2 0x7f552fbad3f7 (/lib64/libjansson.so.4+0x63f7)
> #3 0x7f552fbad8f3 (/lib64/libjansson.so.4+0x68f3)
> #4 0x7f552fbad723 (/lib64/libjansson.so.4+0x6723)
> #5 0x7f552fbadb8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
> #6 0x64fd24 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
> #7 0x64fef8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
> #8 0x7f5511e81f26 in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3295
> #9 0x7f5511e82ea4 in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3360
> #10 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
> #11 0x7f54fc11a02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
> #12 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
> #13 0x7f552f34edc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
> #14 0x7f552e62e73c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> ERROR: AddressSanitizer: heap-use-after-free on address 0x6030003739c0 at pc 0x7f30ccac6e05 bp 0x7f30bec2e2b0 sp 0x7f30bec2da58
> READ of size 19 at 0x6030003739c0 thread T738
> #0 0x7f30ccac6e04 in __asan_memcpy (/lib64/libasan.so.2+0x8ce04)
> #1 0x7f30cb530571 (/lib64/libjansson.so.4+0x5571)
> #2 0x7f30cb532887 in json_string_nocheck (/lib64/libjansson.so.4+0x7887)
> #3 0x7f30cb531907 (/lib64/libjansson.so.4+0x6907)
> #4 0x7f30cb531723 (/lib64/libjansson.so.4+0x6723)
> #5 0x7f30cb531b8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
> #6 0x64c2e4 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
> #7 0x64c4b8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
> #8 0x7f30addd8d77 in ast_sockaddr_copy /home/pbs.vbadalyan/asterisk-13.14.0/include/asterisk/netsock2.h:145
> #9 0x7f30addd8d77 in ast_rtcp_read /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:4217
> #10 0x7f30addd93cc (/usr/lib/asterisk/modules/res_rtp_asterisk.so+0x2a3cc)
> #11 0x73d1e3 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
> #12 0x7f30972c78d5 in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
> #13 0x7ae78f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
> #14 0x7f30cacd2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
> #15 0x7f30c9fb373c in clone (/lib64/libc.so.6+0xf773c)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list