[asterisk-bugs] [JIRA] (ASTERISK-26831) res_rtp_asterisk: Race condition when RTCP and WebRTC is used

Badalian Vyacheslav (JIRA) noreply at issues.asterisk.org
Thu Apr 20 09:13:59 CDT 2017


    [ https://issues.asterisk.org/jira/browse/ASTERISK-26831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=236618#comment-236618 ] 

Badalian Vyacheslav edited comment on ASTERISK-26831 at 4/20/17 9:13 AM:
-------------------------------------------------------------------------

Compiled patch 7
{code}
git fetch https://gerrit.asterisk.org/asterisk refs/changes/43/5343/7 && git checkout FETCH_HEAD
{code}

1. Have deadlocked channels 
{code}
ds-asterisk05*CLI> core show channels
Channel              Location             State   Application(Data)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/ivmakevit23393_w (None)               Down    (None)
SIP/ivmakevit23393_w (None)               Down    (None)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/raburenkov_wof-0 (None)               Down    (None)
SIP/raburenkov_wof-0 (None)               Down    (None)
SIP/vyumex_wof-00003 (None)               Down    (None)
SIP/yuadomracheva_wo (None)               Down    (None)
SIP/1_stub-000035df  (None)               Down    (None)
SIP/1_stub-000035de  (None)               Down    (None)
SIP/1_stub-000035dd  (None)               Down    (None)
SIP/1_stub-000035dc  (None)               Down    (None)
SIP/1_stub-000035db  (None)               Down    (None)
SIP/1_stub-000035da  (None)               Down    (None)
SIP/1_stub-000035cd  (None)               Down    (None)
SIP/1_stub-000035cc  (None)               Down    (None)
SIP/1_stub-000035cb  (None)               Down    (None)
SIP/1_stub-000035e0  (None)               Down    (None)
SIP/1_stub-000035d9  (None)               Down    (None)
SIP/1_stub-000035d8  (None)               Down    (None)
SIP/1_stub-000035d7  (None)               Down    (None)
SIP/1_stub-000035d6  (None)               Down    (None)
SIP/1_stub-000035d5  (None)               Down    (None)
SIP/1_stub-000035d4  (None)               Down    (None)
SIP/1_stub-000035d2  (None)               Down    (None)
SIP/1_stub-000035c9  (None)               Down    (None)
SIP/iadavidenko_wof- (None)               Down    AppDial2((Outgoing Line))
SIP/ndmargaryan_wof- (None)               Down    (None)
48 active channels
0 active calls
6798 calls processed
ds-asterisk05*CLI>

{code}

2. Have errors in logs
{code}
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x60b00005b588 (0) at line 122 in stasis_message_create_full of stasis_message.c
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: [2017-04-19 14:12:59] ERROR[2257][C-00000518]: stasis_message.c:122 stasis_message_create_full: Excessive refcount 100000 reached on ao2 object 0x60b00005b588
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: [2017-04-19 14:12:59] ERROR[2257][C-00000518]: stasis_message.c:122 stasis_message_create_full: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x60b00005b588 (0)

{code}

and asterisk restart randomly
{code}
Apr 20 16:50:52 ds-asterisk05.tcsbank.ru asterisk[32080]: [2017-04-20 16:50:52] ERROR[3649]: manager.c:6310 process_message: Unable to process manager action 'Originate'. Asterisk is shutting down.

{code}


was (Author: slavon):
Compiled patch 7
{code}
git fetch https://gerrit.asterisk.org/asterisk refs/changes/43/5343/7 && git checkout FETCH_HEAD
{code}

1. Have deadlocked channels 
{code}
ds-asterisk05*CLI> core show channels
Channel              Location             State   Application(Data)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/ivmakevit23393_w (None)               Down    (None)
SIP/ivmakevit23393_w (None)               Down    (None)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/ndmargaryan_wof- (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/altumanyan_wof-0 (None)               Down    (None)
SIP/raburenkov_wof-0 (None)               Down    (None)
SIP/raburenkov_wof-0 (None)               Down    (None)
SIP/vyumex_wof-00003 (None)               Down    (None)
SIP/yuadomracheva_wo (None)               Down    (None)
SIP/1_stub-000035df  (None)               Down    (None)
SIP/1_stub-000035de  (None)               Down    (None)
SIP/1_stub-000035dd  (None)               Down    (None)
SIP/1_stub-000035dc  (None)               Down    (None)
SIP/1_stub-000035db  (None)               Down    (None)
SIP/1_stub-000035da  (None)               Down    (None)
SIP/1_stub-000035cd  (None)               Down    (None)
SIP/1_stub-000035cc  (None)               Down    (None)
SIP/1_stub-000035cb  (None)               Down    (None)
SIP/1_stub-000035e0  (None)               Down    (None)
SIP/1_stub-000035d9  (None)               Down    (None)
SIP/1_stub-000035d8  (None)               Down    (None)
SIP/1_stub-000035d7  (None)               Down    (None)
SIP/1_stub-000035d6  (None)               Down    (None)
SIP/1_stub-000035d5  (None)               Down    (None)
SIP/1_stub-000035d4  (None)               Down    (None)
SIP/1_stub-000035d2  (None)               Down    (None)
SIP/1_stub-000035c9  (None)               Down    (None)
SIP/iadavidenko_wof- (None)               Down    AppDial2((Outgoing Line))
SIP/ndmargaryan_wof- (None)               Down    (None)
48 active channels
0 active calls
6798 calls processed
ds-asterisk05*CLI>

{code}

2. Have errors in logs
{code}
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x60b00005b588 (0) at line 122 in stasis_message_create_full of stasis_message.c
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: [2017-04-19 14:12:59] ERROR[2257][C-00000518]: stasis_message.c:122 stasis_message_create_full: Excessive refcount 100000 reached on ao2 object 0x60b00005b588
Apr 19 14:12:59 ds-asterisk05.tcsbank.ru asterisk[11220]: [2017-04-19 14:12:59] ERROR[2257][C-00000518]: stasis_message.c:122 stasis_message_create_full: FRACK!, Failed assertion Excessive refcount 100000 reached on ao2 object 0x60b00005b588 (0)

{code}

> res_rtp_asterisk: Race condition when RTCP and WebRTC is used
> -------------------------------------------------------------
>
>                 Key: ASTERISK-26831
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26831
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Resources/res_rtp_asterisk
>    Affects Versions: 13.14.0, 13.15.0
>            Reporter: Badalian Vyacheslav
>            Assignee: Unassigned
>            Severity: Minor
>         Attachments: backtrace.txt
>
>
> Race condition. With this story, I am fighting for more than a month. I tried different options but the idea of an end. There is only a heavy load (more than 200 simultaneous calls WebRTC) after 50 000-100 000 calls.
> Please help. It is difficult to disentangle this tangle. The error for a long time (many versions).
> It may have an idea why this might happen? This is because rtp_stop or because rtp_destroy?
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x6290016c326c at pc 0x7f007c288b1b bp 0x7f00404275b0 sp 0x7f0040426d58
> WRITE of size 64 at 0x6290016c326c thread T467
>     #0 0x7f007c288b1a  (/lib64/libasan.so.3+0x5eb1a)
>     #1 0x7f0074fdea26 in memcpy /usr/include/bits/string3.h:51
>     #2 0x7f0074fdea26 in ast_srtp_protect /home/pbs.vbadalyan/asterisk-13.14.0/res/res_srtp.c:438
>     #3 0x7f005c6e65de in __rtp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2305
>     #4 0x7f005c6f5e8d in rtcp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2330
>     #5 0x7f005c6f5e8d in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3286
>     #6 0x7f005c6f6ecc in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3357
>     #7 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #8 0x7f00469ad02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #9 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #10 0x7f007a4c2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #11 0x7f00797a273c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x603002c6d820 at pc 0x7f55310f2ff3 bp 0x7f552b6f2240 sp 0x7f552b6f19e8
> READ of size 3 at 0x603002c6d820 thread T88
>     #0 0x7f55310f2ff2  (/lib64/libasan.so.3+0x3cff2)
>     #1 0x7f552fbae4a4  (/lib64/libjansson.so.4+0x74a4)
>     #2 0x7f552fbad3f7  (/lib64/libjansson.so.4+0x63f7)
>     #3 0x7f552fbad8f3  (/lib64/libjansson.so.4+0x68f3)
>     #4 0x7f552fbad723  (/lib64/libjansson.so.4+0x6723)
>     #5 0x7f552fbadb8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
>     #6 0x64fd24 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
>     #7 0x64fef8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
>     #8 0x7f5511e81f26 in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3295
>     #9 0x7f5511e82ea4 in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3360
>     #10 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #11 0x7f54fc11a02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #12 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #13 0x7f552f34edc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #14 0x7f552e62e73c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> ERROR: AddressSanitizer: heap-use-after-free on address 0x6030003739c0 at pc 0x7f30ccac6e05 bp 0x7f30bec2e2b0 sp 0x7f30bec2da58
> READ of size 19 at 0x6030003739c0 thread T738
>     #0 0x7f30ccac6e04 in __asan_memcpy (/lib64/libasan.so.2+0x8ce04)
>     #1 0x7f30cb530571  (/lib64/libjansson.so.4+0x5571)
>     #2 0x7f30cb532887 in json_string_nocheck (/lib64/libjansson.so.4+0x7887)
>     #3 0x7f30cb531907  (/lib64/libjansson.so.4+0x6907)
>     #4 0x7f30cb531723  (/lib64/libjansson.so.4+0x6723)
>     #5 0x7f30cb531b8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
>     #6 0x64c2e4 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
>     #7 0x64c4b8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
>     #8 0x7f30addd8d77 in ast_sockaddr_copy /home/pbs.vbadalyan/asterisk-13.14.0/include/asterisk/netsock2.h:145
>     #9 0x7f30addd8d77 in ast_rtcp_read /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:4217
>     #10 0x7f30addd93cc  (/usr/lib/asterisk/modules/res_rtp_asterisk.so+0x2a3cc)
>     #11 0x73d1e3 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #12 0x7f30972c78d5 in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #13 0x7ae78f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #14 0x7f30cacd2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #15 0x7f30c9fb373c in clone (/lib64/libc.so.6+0xf773c)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)



More information about the asterisk-bugs mailing list