[asterisk-bugs] [JIRA] (ASTERISK-26445) rtp: Deadlock in getting payload code

[Ross Beer (JIRA)]

     [ https://issues.asterisk.org/jira/browse/ASTERISK-26445?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ross Beer updated ASTERISK-26445:
---------------------------------

    Comment: was deleted

(was: I've just had the same deadlock without using the hangup handlers accessing RTP data:

{noformat}
Thread 10 (Thread 0x7f4168af2700 (LWP 45870)):
#0  0x00007f42dcfb08ad in __lll_lock_wait () at /lib64/libpthread.so.0
#1  0x00007f42dcfaca48 in pthread_rwlock_rdlock () at /lib64/libpthread.so.0
#2  0x000000000053ce3a in __ast_rwlock_rdlock (filename=0x6f1eab "rtp_engine.c", line=924, func=0x6f2f50 <__PRETTY_FUNCTION__.17039> "ast_rtp_codecs_payload_code", t=0x7f40701f4a40, name=0x6f2204 "&codecs->codecs_lock") at lock.c:911
#3  0x0000000000598671 in ast_rtp_codecs_payload_code (codecs=0x7f40701f4a20, asterisk_format=1, format=0x2a1f640, code=0) at rtp_engine.c:924
#4  0x00007f417ba8bd09 in ast_rtp_read (hdrlen=12, len=172, rtpheader=0x7f3fb02eb520, instance=0x7f3fb03904f8) at res_rtp_asterisk.c:4301
#5  0x00007f417ba8bd09 in ast_rtp_read (instance=0x7f3fb03904f8, rtcp=<optimized out>) at res_rtp_asterisk.c:4502
#6  0x0000000000596d9a in ast_rtp_instance_read (instance=0x7f3fb03904f8, rtcp=0) at rtp_engine.c:482
#7  0x00007f41785845af in chan_pjsip_read (ast=0x7f3fb00f1638) at chan_pjsip.c:695
#8  0x00000000004b7940 in __ast_read (chan=0x7f3fb00f1638, dropaudio=0) at channel.c:3898
#9  0x00000000004b93eb in ast_read (chan=0x7f3fb00f1638) at channel.c:4305
#10 0x000000000048659b in bridge_handle_trip (bridge_channel=0x7f40580088a8) at bridge_channel.c:2416
#11 0x0000000000486a46 in bridge_channel_wait (bridge_channel=0x7f40580088a8) at bridge_channel.c:2586
#12 0x0000000000487131 in bridge_channel_internal_join (bridge_channel=0x7f40580088a8) at bridge_channel.c:2732
#13 0x000000000046dbcf in ast_bridge_join (bridge=0x7f4058003e08, chan=0x7f3fb00f1638, swap=0x0, features=0x7f4168aedfd0, tech_args=0x0, flags=(AST_BRIDGE_JOIN_PASS_REFERENCE | AST_BRIDGE_JOIN_INHIBIT_JOIN_COLP)) at bridge.c:1712
#14 0x000000000050e3e0 in ast_bridge_call_with_flags (chan=0x7f3fb00f1638, peer=0x7f4058006438, config=0x7f4168aef000, flags=0) at features.c:672
#15 0x000000000050e4b2 in ast_bridge_call (chan=0x7f3fb00f1638, peer=0x7f4058006438, config=0x7f4168aef000) at features.c:711
#16 0x00007f41969cb608 in dial_exec_full (chan=0x7f3fb00f1638, data=0x7f4168aef540 "PJSIP/23712*201,,b(vh-add-distinctivering-headers^1^1)", peerflags=0x7f4168aef400, continue_exec=0x0) at app_dial.c:3158
#17 0x00007f41969cb999 in dial_exec (chan=0x7f3fb00f1638, data=0x7f4168aef540 "PJSIP/23712*201,,b(vh-add-distinctivering-headers^1^1)") at app_dial.c:3210
#18 0x00000000005879a6 in pbx_exec (c=0x7f3fb00f1638, app=0x36cdd00, data=0x7f4168aef540 "PJSIP/23712*201,,b(vh-add-distinctivering-headers^1^1)") at pbx_app.c:485
#19 0x00000000005753db in pbx_extension_helper (c=0x7f3fb00f1638, con=0x0, context=0x7f3fb00f1ff0 "crossservers", exten=0x7f3fb00f2040 "23712*201", priority=18, label=0x0, callerid=0x7f4058002c40 "01524389900", action=E_SPAWN, found=0x7f4168af1be4, combined_find_spawn=1) at pbx.c:2884
#20 0x00000000005788a2 in ast_spawn_extension (c=0x7f3fb00f1638, context=0x7f3fb00f1ff0 "crossservers", exten=0x7f3fb00f2040 "23712*201", priority=18, callerid=0x7f4058002c40 "01524389900", found=0x7f4168af1be4, combined_find_spawn=1) at pbx.c:4110
#21 0x000000000057950b in __ast_pbx_run (c=0x7f3fb00f1638, args=0x0) at pbx.c:4285
#22 0x000000000057ac37 in pbx_thread (data=0x7f3fb00f1638) at pbx.c:4605
#23 0x0000000000600915 in dummy_start (data=0x7f3fb031b2a0) at utils.c:1235
#24 0x00007f42dcfa861a in start_thread () at /lib64/libpthread.so.0
#25 0x00007f42dc2e45fd in clone () at /lib64/libc.so.6
{noformat}

This happens once every 20,000 calls or so.)

> rtp: Deadlock in getting payload code
> -------------------------------------
>
>                 Key: ASTERISK-26445
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26445
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Core/RTP
>    Affects Versions: 13.11.2
>         Environment: Fedora 23
>            Reporter: Ross Beer
>            Assignee: Unassigned
>         Attachments: backtrace-threads_2016-10-05-CLEAN.txt, backtrace-threads-CLEAN.txt
>
>
> Asterisk deadlocks and no further processing can occur.
> This looks to be related to rtp_engine.c
> {noformat}
> #2  0x000000000053ce5a in __ast_rwlock_rdlock (filename=0x6f1e4b "rtp_engine.c", line=924, func=0x6f2ef0 <__PRETTY_FUNCTION__.16829> "ast_rtp_codecs_payload_code", t=0x7fe9945032f0, name=0x6f21a4 "&codecs->codecs_lock") at lock.c:911
> {noformat}
> I think this may have been caused by using hangup handlers on the channel. One before a dial and then another in the Dial b option subroutine. The hangup handlers call the same routine that updates the hangup reason for each channel:
> {noformat}
> #5  0x000000000058cac5 in ast_func_read (chan=0x7f4cc420fdb8, function=0x7f4d38cae0e0 "CHANNEL(rtcp,all)", workspace=0x7f4d38cad0c0 "", len=4096) at pbx_functions.c:617
> #6  0x0000000000590bb0 in pbx_substitute_variables_helper_full (c=0x7f4cc420fdb8, headp=0x7f4cc4210580, cp1=0x7f4d38caf260 "CDR(mos)=${CHANNEL(rtcp,all)};cr=${CHANNEL(audioreadformat)};cw=${CHANNEL(audiowriteformat)};cn=${CHANNEL(audionativeformat)}", cp2=0x7f4d38caf349 "", count=8182, used=0x7f4d38caf238) at pbx_variables.c:693
> #7  0x000000000059112b in pbx_substitute_variables_helper (c=0x7f4cc420fdb8, cp1=0x7f4d38caf260 "CDR(mos)=${CHANNEL(rtcp,all)};cr=${CHANNEL(audioreadformat)};cw=${CHANNEL(audiowriteformat)};cn=${CHANNEL(audionativeformat)}", cp2=0x7f4d38caf340 "CDR(mos)=", count=8191) at pbx_variables.c:790
> #8  0x0000000000575257 in pbx_extension_helper (c=0x7f4cc420fdb8, con=0x0, context=0x7f4cc4210770 "record-hangupcause", exten=0x7f4cc42107c0 "s", priority=4, label=0x0, callerid=0x7f4b60004a70 "<< PRIVATE INFORMATION REMOVED >>>", action=E_SPAWN, found=0x7f4d38cb1924, combined_find_spawn=1) at pbx.c:2873
> #9  0x00000000005788a2 in ast_spawn_extension (c=0x7f4cc420fdb8, context=0x7f4cc4210770 "record-hangupcause", exten=0x7f4cc42107c0 "s", priority=4, callerid=0x7f4b60004a70 "<< PRIVATE INFORMATION REMOVED >>>", found=0x7f4d38cb1924, combined_find_spawn=1) at pbx.c:4110
> {noformat}
> From the backtrace (attached) it looks like calling channel and the audio formats causes a lock.



--
This message was sent by Atlassian JIRA
(v6.2#6252)