[asterisk-bugs] [JIRA] (ASTERISK-26831) res_rtp_asterisk: Race condition when RTCP and WebRTC is used

Badalian Vyacheslav (JIRA) noreply at issues.asterisk.org
Tue May 9 09:48:58 CDT 2017


    [ https://issues.asterisk.org/jira/browse/ASTERISK-26831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=236929#comment-236929 ] 

Badalian Vyacheslav edited comment on ASTERISK-26831 at 5/9/17 9:47 AM:
------------------------------------------------------------------------

locks_x*.txt - 20-40 concurent calls. 

We do sipmle webrtc call to MOH

2853% CPU!! 
{code}
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 72545 asterisk  20   0 18.558g 1.812g  21448 S  2853  1.4 117:04.45 asterisk
{code}

{code}
m1-asterisk01t*CLI> core show channels
Channel              Location             State   Application(Data)
SIP/user36_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user16_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user20_stub-0000 s at macro-hangupcall:3 Up      GotoIf(1?theend)
SIP/user40_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user3_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user22_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user1_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user19_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user14_stub-0000 s at macro-hangupcall:7 Up      Hangup()
SIP/user12_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user5_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user41_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user38_stub-0000 (None)               Ringing AppDial2((Outgoing Line))
SIP/user34_stub-0000 (None)               Down    (None)
SIP/user11_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user8_stub-00000 (None)               Down    AppDial2((Outgoing Line))
SIP/user31_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user1000_stub-00 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user15_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user28_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user44_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user32_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user37_stub-0000 s at macro-hangupcall:5 Up      ExecIf(0?Set(CDR(recordingfile
SIP/user18_stub-0000 (None)               Ringing AppDial2((Outgoing Line))
SIP/user26_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user21_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user35_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user9_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user4_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user2_stub-00000 s at from-internal:1    Down    (None)
SIP/user43_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user30_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user13_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user39_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user33_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user17_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user42_stub-0000 s at macro-hangupcall:7 Up      Hangup()
SIP/user6_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user29_stub-0000 s at macro-hangupcall:5 Up      ExecIf(0?Set(CDR(recordingfile
SIP/user27_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user10_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
42 active channels
34 active calls
{code}


was (Author: slavon):
locks_x*.txt - 20-40 concurent calls. 

2853% CPU!! 
{code}
   PID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND
 72545 asterisk  20   0 18.558g 1.812g  21448 S  2853  1.4 117:04.45 asterisk
{code}

{code}
m1-asterisk01t*CLI> core show channels
Channel              Location             State   Application(Data)
SIP/user36_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user16_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user20_stub-0000 s at macro-hangupcall:3 Up      GotoIf(1?theend)
SIP/user40_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user3_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user22_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user1_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user19_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user14_stub-0000 s at macro-hangupcall:7 Up      Hangup()
SIP/user12_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user5_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user41_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user38_stub-0000 (None)               Ringing AppDial2((Outgoing Line))
SIP/user34_stub-0000 (None)               Down    (None)
SIP/user11_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user8_stub-00000 (None)               Down    AppDial2((Outgoing Line))
SIP/user31_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user1000_stub-00 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user15_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user28_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user44_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user32_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user37_stub-0000 s at macro-hangupcall:5 Up      ExecIf(0?Set(CDR(recordingfile
SIP/user18_stub-0000 (None)               Ringing AppDial2((Outgoing Line))
SIP/user26_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user21_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user35_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user9_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user4_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user2_stub-00000 s at from-internal:1    Down    (None)
SIP/user43_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user30_stub-0000 (None)               Down    AppDial2((Outgoing Line))
SIP/user13_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user39_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user33_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user17_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user42_stub-0000 s at macro-hangupcall:7 Up      Hangup()
SIP/user6_stub-00000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user29_stub-0000 s at macro-hangupcall:5 Up      ExecIf(0?Set(CDR(recordingfile
SIP/user27_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
SIP/user10_stub-0000 766 at from-internal:2  Up      MusicOnHold(,5)
42 active channels
34 active calls
{code}

> res_rtp_asterisk: Race condition when RTCP and WebRTC is used
> -------------------------------------------------------------
>
>                 Key: ASTERISK-26831
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26831
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Resources/res_rtp_asterisk
>    Affects Versions: 13.14.0, 13.15.0
>            Reporter: Badalian Vyacheslav
>            Assignee: Unassigned
>            Severity: Minor
>         Attachments: backtrace.txt, locks1.txt, locks2.txt, locks_x2.txt, locks_x3.txt, locks_x4.txt, locks_x.txt
>
>
> Race condition. With this story, I am fighting for more than a month. I tried different options but the idea of an end. There is only a heavy load (more than 200 simultaneous calls WebRTC) after 50 000-100 000 calls.
> Please help. It is difficult to disentangle this tangle. The error for a long time (many versions).
> It may have an idea why this might happen? This is because rtp_stop or because rtp_destroy?
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x6290016c326c at pc 0x7f007c288b1b bp 0x7f00404275b0 sp 0x7f0040426d58
> WRITE of size 64 at 0x6290016c326c thread T467
>     #0 0x7f007c288b1a  (/lib64/libasan.so.3+0x5eb1a)
>     #1 0x7f0074fdea26 in memcpy /usr/include/bits/string3.h:51
>     #2 0x7f0074fdea26 in ast_srtp_protect /home/pbs.vbadalyan/asterisk-13.14.0/res/res_srtp.c:438
>     #3 0x7f005c6e65de in __rtp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2305
>     #4 0x7f005c6f5e8d in rtcp_sendto /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:2330
>     #5 0x7f005c6f5e8d in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3286
>     #6 0x7f005c6f6ecc in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3357
>     #7 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #8 0x7f00469ad02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #9 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #10 0x7f007a4c2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #11 0x7f00797a273c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> =ERROR: AddressSanitizer: heap-use-after-free on address 0x603002c6d820 at pc 0x7f55310f2ff3 bp 0x7f552b6f2240 sp 0x7f552b6f19e8
> READ of size 3 at 0x603002c6d820 thread T88
>     #0 0x7f55310f2ff2  (/lib64/libasan.so.3+0x3cff2)
>     #1 0x7f552fbae4a4  (/lib64/libjansson.so.4+0x74a4)
>     #2 0x7f552fbad3f7  (/lib64/libjansson.so.4+0x63f7)
>     #3 0x7f552fbad8f3  (/lib64/libjansson.so.4+0x68f3)
>     #4 0x7f552fbad723  (/lib64/libjansson.so.4+0x6723)
>     #5 0x7f552fbadb8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
>     #6 0x64fd24 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
>     #7 0x64fef8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
>     #8 0x7f5511e81f26 in ast_rtcp_write_report /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3295
>     #9 0x7f5511e82ea4 in ast_rtcp_write /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:3360
>     #10 0x740c64 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #11 0x7f54fc11a02d in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #12 0x7b217f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #13 0x7f552f34edc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #14 0x7f552e62e73c in clone (/lib64/libc.so.6+0xf773c)
> {code}
> {code}
> ERROR: AddressSanitizer: heap-use-after-free on address 0x6030003739c0 at pc 0x7f30ccac6e05 bp 0x7f30bec2e2b0 sp 0x7f30bec2da58
> READ of size 19 at 0x6030003739c0 thread T738
>     #0 0x7f30ccac6e04 in __asan_memcpy (/lib64/libasan.so.2+0x8ce04)
>     #1 0x7f30cb530571  (/lib64/libjansson.so.4+0x5571)
>     #2 0x7f30cb532887 in json_string_nocheck (/lib64/libjansson.so.4+0x7887)
>     #3 0x7f30cb531907  (/lib64/libjansson.so.4+0x6907)
>     #4 0x7f30cb531723  (/lib64/libjansson.so.4+0x6723)
>     #5 0x7f30cb531b8c in json_vpack_ex (/lib64/libjansson.so.4+0x6b8c)
>     #6 0x64c2e4 in ast_json_vpack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:823
>     #7 0x64c4b8 in ast_json_pack /home/pbs.vbadalyan/asterisk-13.14.0/main/json.c:814
>     #8 0x7f30addd8d77 in ast_sockaddr_copy /home/pbs.vbadalyan/asterisk-13.14.0/include/asterisk/netsock2.h:145
>     #9 0x7f30addd8d77 in ast_rtcp_read /home/pbs.vbadalyan/asterisk-13.14.0/res/res_rtp_asterisk.c:4217
>     #10 0x7f30addd93cc  (/usr/lib/asterisk/modules/res_rtp_asterisk.so+0x2a3cc)
>     #11 0x73d1e3 in ast_sched_runq /home/pbs.vbadalyan/asterisk-13.14.0/main/sched.c:783
>     #12 0x7f30972c78d5 in do_monitor /home/pbs.vbadalyan/asterisk-13.14.0/channels/chan_sip.c:29514
>     #13 0x7ae78f in dummy_start /home/pbs.vbadalyan/asterisk-13.14.0/main/utils.c:1235
>     #14 0x7f30cacd2dc4 in start_thread (/lib64/libpthread.so.0+0x7dc4)
>     #15 0x7f30c9fb373c in clone (/lib64/libc.so.6+0xf773c)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)



More information about the asterisk-bugs mailing list