[asterisk-bugs] [JIRA] (ASTERISK-30157) Deadlock between bridge_channel_internal_join and ao2 channel traversal

Asterisk Team (JIRA) noreply at issues.asterisk.org
Tue Jul 26 06:20:09 CDT 2022


    [ https://issues.asterisk.org/jira/browse/ASTERISK-30157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=259797#comment-259797 ] 

Asterisk Team commented on ASTERISK-30157:
------------------------------------------

Thanks for creating a report! The issue has entered the triage process. That means the issue will wait in this status until a Bug Marshal has an opportunity to review the issue. Once the issue has been reviewed you will receive comments regarding the next steps towards resolution. Please note that log messages and other files should not be sent to the Sangoma Asterisk Team unless explicitly asked for. All files should be placed on this issue in a sanitized fashion as needed.

A good first step is for you to review the [Asterisk Issue Guidelines|https://wiki.asterisk.org/wiki/display/AST/Asterisk+Issue+Guidelines] if you haven't already. The guidelines detail what is expected from an Asterisk issue report.

Then, if you are submitting a patch, please review the [Patch Contribution Process|https://wiki.asterisk.org/wiki/display/AST/Patch+Contribution+Process].

Please note that once your issue enters an open state it has been accepted. As Asterisk is an open source project there is no guarantee or timeframe on when your issue will be looked into. If you need expedient resolution you will need to find and pay a suitable developer. Asking for an update on your issue will not yield any progress on it and will not result in a response. All updates are posted to the issue when they occur.

Please note that by submitting data, code, or documentation to Sangoma through JIRA, you accept the Terms of Use present at [https://www.asterisk.org/terms-of-use/|https://www.asterisk.org/terms-of-use/].

> Deadlock between bridge_channel_internal_join and ao2 channel traversal
> -----------------------------------------------------------------------
>
>                 Key: ASTERISK-30157
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-30157
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Core/Bridging, Core/Channels
>    Affects Versions: 18.9.0
>            Reporter: N A
>            Severity: Major
>
> This has happened several times now that I've begun to pick out a pattern of this deadlock consistently happening in the same place, so here we are:
> The deadlock is always on a channel (which then blocks the entire channels container), which basically grinds Asterisk to a halt as existing calls may complete but no new channels can be created.
> core show locks:
> {noformat}
> [2022-07-26 10:45:51] ERROR[7806]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:299 __ast_pthread_mutex_lock: channel.c line 922 (__ast_channel_alloc_ap): Deadlock? waited 16665 sec for mutex 'channels'?
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_bt_get_addresses+0x1d) [0x5603f1ac6716]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xb5) [0x5603f1b81517]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_lock+0x10b) [0x5603f1ab9091]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xb6c6e) [0x5603f1af8c6e]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_channel_alloc+0xde) [0x5603f1af8f56]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_unreal_new_channels+0x227) [0x5603f1b3d14c]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xf7195) [0x5603f1b39195]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0xc8cfa) [0x5603f1b0acfa]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_request+0x51) [0x5603f1b0b3ca]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x103352) [0x5603f1b45352]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_dial_prerun+0x77) [0x5603f1b45845]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x17395e) [0x5603f1bb595e]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_pbx_outgoing_app_predial+0xb9) [0x5603f1bb64a6]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_pbx_outgoing_app+0x5d) [0x5603f1bb63e7]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/lib/asterisk/modules/pbx_spool.so(+0x4f24) [0x7f2083a3df24]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x215820) [0x5603f1c57820]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f209292afa3]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:307 __ast_pthread_mutex_lock: astobj2_container.c line 310 (internal_ao2_traverse): 'channels' was locked here.
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_bt_get_addresses+0x1d) [0x5603f1ac6716]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ast_pthread_mutex_lock+0xb5) [0x5603f1b81517]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_lock+0x10b) [0x5603f1ab9091]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x79982) [0x5603f1abb982]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(__ao2_callback_data+0x4d) [0x5603f1abbd37]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_channel_callback+0x54) [0x5603f1afa0c8]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_channel_get_by_name_prefix+0x46) [0x5603f1afa7f9]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(ast_parse_device_state+0x4e) [0x5603f1b42caa]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x10102b) [0x5603f1b4302b]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x1014d1) [0x5603f1b434d1]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x1018aa) [0x5603f1b438aa]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /usr/sbin/asterisk(+0x215820) [0x5603f1c57820]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libpthread.so.0(+0x7fa3) [0x7f209292afa3]
> [2022-07-26 10:45:51] ERROR[10667]: lock.c:76 __dump_backtrace: /lib/x86_64-linux-gnu/libc.so.6(clone+0x3f) [0x7f20923beeff]
> voip*CLI> core show locks
> =======================================================================
> === 18.9.0
> === Currently Held Locks
> =======================================================================
> ===
> === <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
> ===
> === Thread ID: 0x7f2083d11700 LWP:31458 (do_devstate_changes  started at [  621] devicestate.c ast_device_state_engine_init())
> === ---> Lock #0 (astobj2_container.c): MUTEX 310 internal_ao2_traverse self 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1abb982] asterisk astobj2_container.c:315 internal_ao2_traverse()
>         [0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
>         [0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
>         [0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
>         [0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
>         [0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
>         [0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
>         [0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 1290 ast_channel_by_name_cb chan 0x7f206c4c0060 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1afa189] asterisk channel.c:1291 ast_channel_by_name_cb()
>         [0x5603f1abba0a] asterisk astobj2_container.c:326 internal_ao2_traverse()
>         [0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
>         [0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
>         [0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
>         [0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
>         [0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
>         [0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
>         [0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067d04700 LWP:31499 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f62ba660 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067c87700 LWP:31500 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070ae46e0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067c0a700 LWP:31501 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20614274c0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067b8d700 LWP:31502 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078e8a4a0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067b10700 LWP:31503 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2076c18840 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067a93700 LWP:31504 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207d5d6f10 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067a16700 LWP:31505 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088b673f0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067999700 LWP:31506 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2085c8de80 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206791c700 LWP:31507 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f208cf92ef0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206789f700 LWP:31508 (iax2_process_thread  started at [12720] chan_iax2.c start_network_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f748d6d0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f20697c4700 LWP:443 (worker_start         started at [ 1111] threadpool.c worker_thread_start())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207c82fce0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
>         [0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
>         [0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
>         [0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
>         [0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
>         [0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
>         [0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
>         [0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
>         [0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
>         [0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2067ffa700 LWP:3871 (worker_start         started at [ 1111] threadpool.c worker_thread_start())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f206d746c80 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
>         [0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
>         [0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
>         [0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
>         [0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
>         [0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
>         [0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
>         [0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
>         [0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
>         [0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f209008c700 LWP:21787 (worker_start         started at [ 1111] threadpool.c worker_thread_start())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207e9f3d30 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
>         [0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
>         [0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
>         [0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
>         [0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
>         [0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
>         [0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
>         [0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
>         [0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
>         [0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f20650b1700 LWP:2130 (default_tps_processing_function started at [  226] taskprocessor.c default_listener_start())
> === ---> Waiting for Lock #0 (astobj2_container.c): MUTEX 308 internal_ao2_traverse self 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2065228700 LWP:6593 (pbx_thread           started at [ 4736] pbx.c ast_pbx_start())
> === ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f206d6426c0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
>         [0x5603f1aeb2c8] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
>         [0x5603f1acca08] asterisk bridge.c:1725 ast_bridge_join()
>         [0x5603f1c9cf55] asterisk features.c:684 ast_bridge_call_with_flags()
>         [0x5603f1c9d113] asterisk features.c:715 ast_bridge_call()
>         [0x7f20812bd146] app_dial.so app_dial.c:3413 dial_exec_full()
>         [0x7f20812bda11] app_dial.so app_dial.c:3482 dial_exec()
>         [0x5603f1bbbc19] asterisk pbx_app.c:492 pbx_exec()
>         [0x5603f1ba42f6] asterisk pbx.c:2948 pbx_extension_helper()
>         [0x5603f1ba868f] asterisk pbx.c:4210 ast_spawn_extension()
>         [0x5603f1ba949f] asterisk pbx.c:4386 __ast_pbx_run()
>         [0x5603f1baae48] asterisk pbx.c:4711 pbx_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f206d539fc0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1aeb2fb] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
>         [0x5603f1acca08] asterisk bridge.c:1725 ast_bridge_join()
>         [0x5603f1c9cf55] asterisk features.c:684 ast_bridge_call_with_flags()
>         [0x5603f1c9d113] asterisk features.c:715 ast_bridge_call()
>         [0x7f20812bd146] app_dial.so app_dial.c:3413 dial_exec_full()
>         [0x7f20812bda11] app_dial.so app_dial.c:3482 dial_exec()
>         [0x5603f1bbbc19] asterisk pbx_app.c:492 pbx_exec()
>         [0x5603f1ba42f6] asterisk pbx.c:2948 pbx_extension_helper()
>         [0x5603f1ba868f] asterisk pbx.c:4210 ast_spawn_extension()
>         [0x5603f1ba949f] asterisk pbx.c:4386 __ast_pbx_run()
>         [0x5603f1baae48] asterisk pbx.c:4711 pbx_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f206c4c0060 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1afa189] asterisk channel.c:1291 ast_channel_by_name_cb()
>         [0x5603f1abba0a] asterisk astobj2_container.c:326 internal_ao2_traverse()
>         [0x5603f1abbd37] asterisk astobj2_container.c:421 __ao2_callback_data()
>         [0x5603f1afa0c8] asterisk channel.c:1275 ast_channel_callback()
>         [0x5603f1afa7f9] asterisk channel.c:1433 ast_channel_get_by_name_prefix()
>         [0x5603f1b42caa] asterisk devicestate.c:295 ast_parse_device_state()
>         [0x5603f1b4302b] asterisk devicestate.c:376 _ast_device_state()
>         [0x5603f1b434d1] asterisk devicestate.c:464 do_state_change()
>         [0x5603f1b438aa] asterisk devicestate.c:540 do_devstate_changes()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206541c700 LWP:6598 (bridge_channel_ind_thread started at [ 1892] bridge.c bridge_impart_internal())
> === ---> Waiting for Lock #0 (bridge_channel.c): MUTEX 235 ast_bridge_channel_lock_bridge bridge 0x7f206d6426c0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
>         [0x5603f1ae3c77] asterisk bridge_channel.c:236 ast_bridge_channel_lock_bridge()
>         [0x5603f1ada376] asterisk bridge_basic.c:296 basic_hangup_hook()
>         [0x5603f1ae50b3] asterisk bridge_channel.c:613 ast_bridge_channel_kick()
>         [0x5603f1aea8b9] asterisk bridge_channel.c:2595 bridge_handle_trip()
>         [0x5603f1aeaf78] asterisk bridge_channel.c:2803 bridge_channel_wait()
>         [0x5603f1aeb9de] asterisk bridge_channel.c:2975 bridge_channel_internal_join()
>         [0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: bridge_channel.c line 2877 (bridge_channel_internal_join)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f20669fa700 LWP:6599 (bridge_channel_ind_thread started at [ 1892] bridge.c bridge_impart_internal())
> === ---> Lock #0 (bridge_channel.c): MUTEX 2877 bridge_channel_internal_join bridge_channel->bridge 0x7f208d2dcef0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1ae3793] asterisk bridge.h:484 _ast_bridge_lock()
>         [0x5603f1aeb2c8] asterisk bridge_channel.c:2879 bridge_channel_internal_join()
>         [0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (bridge_channel.c): MUTEX 2879 bridge_channel_internal_join bridge_channel->chan 0x7f206c4c0060 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1aeb2fb] asterisk bridge_channel.c:2881 bridge_channel_internal_join()
>         [0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (bridge_channel.c): MUTEX 2885 bridge_channel_internal_join peer 0x7f206d539fc0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1aeb349] asterisk bridge_channel.c:2886 bridge_channel_internal_join()
>         [0x5603f1accdd5] asterisk bridge.c:1798 bridge_channel_ind_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: bridge_channel.c line 2879 (bridge_channel_internal_join)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2066082700 LWP:6632 (worker_start         started at [ 1111] threadpool.c worker_thread_start())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207deb3e30 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
>         [0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
>         [0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
>         [0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
>         [0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
>         [0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
>         [0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
>         [0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
>         [0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
>         [0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2069747700 LWP:6633 (worker_start         started at [ 1111] threadpool.c worker_thread_start())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088ca6da0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f206bc30016] chan_pjsip.so chan_pjsip.c:563 chan_pjsip_new()
>         [0x7f206bc3ce0b] chan_pjsip.so chan_pjsip.c:3134 chan_pjsip_incoming_request()
>         [0x7f206bf0a4ed] res_pjsip_session.so res_pjsip_session.c:4418 handle_incoming_request()
>         [0x7f206bf07446] res_pjsip_session.so res_pjsip_session.c:4043 new_invite()
>         [0x7f206bf07cec] res_pjsip_session.so res_pjsip_session.c:4135 handle_new_invite_request()
>         [0x7f206bf08ee1] res_pjsip_session.so res_pjsip_session.c:4247 session_on_rx_request()
>         [0x7f20930f858f] libasteriskpj.so.2 sip_endpoint.c:930 pjsip_endpt_process_rx_data()
>         [0x7f206bfcc242] res_pjsip.so pjsip_distributor.c:956 distribute()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c4bd4e] asterisk threadpool.c:1354 execute_tasks()
>         [0x5603f1c41f6c] asterisk taskprocessor.c:1239 ast_taskprocessor_execute()
>         [0x5603f1c498b4] asterisk threadpool.c:367 threadpool_execute()
>         [0x5603f1c4b4f6] asterisk threadpool.c:1137 worker_active()
>         [0x5603f1c4b260] asterisk threadpool.c:1057 worker_start()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f20643ce700 LWP:7806 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f207f2d2898 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb61c4] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
>         [0x5603f1bb6129] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
>         [0x7f2083a3e052] pbx_spool.so pbx_spool.c:443 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207d0d3590 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb61c4] asterisk pbx.c:8444 ast_pbx_outgoing_exten_predial()
>         [0x5603f1bb6129] asterisk pbx.c:8423 ast_pbx_outgoing_exten()
>         [0x7f2083a3e052] pbx_spool.so pbx_spool.c:443 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2065516700 LWP:10658 (iax2_process_thread  started at [ 1626] chan_iax2.c find_idle_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f208d1bf620 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206670c700 LWP:10667 (iax2_process_thread  started at [ 1626] chan_iax2.c find_idle_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070ac7120 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206668f700 LWP:10689 (iax2_process_thread  started at [ 1626] chan_iax2.c find_idle_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078eb8cf0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f20680f4700 LWP:10690 (iax2_process_thread  started at [ 1626] chan_iax2.c find_idle_thread())
> === ---> Lock #0 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f205ca4c040 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x7f2080fbdbbb] chan_iax2.so chan_iax2.c:5901 ast_iax2_new()
>         [0x7f2080fdfbf8] chan_iax2.so chan_iax2.c:11493 socket_process_helper()
>         [0x7f2080fe519b] chan_iax2.so chan_iax2.c:12073 socket_process()
>         [0x7f2080fe5851] chan_iax2.so chan_iax2.c:12184 iax2_process_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #1 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206444b700 LWP:20245 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f205647e9f8 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20567c5740 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2064fb7700 LWP:20580 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x5603f73717a8 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f6d92510 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2066883700 LWP:20849 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2078e3ef88 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2078dfb650 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206641e700 LWP:21042 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f208886ffa8 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2088b7cde0 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2066324700 LWP:21379 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2070621b68 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070bfce20 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2066518700 LWP:21625 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f205735b548 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f20574cfa00 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2066789700 LWP:21839 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2053cd3658 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2048f8ac00 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f206622a700 LWP:22010 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f2070b2b8c8 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f2070bf6880 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2065f88700 LWP:22282 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x7f207cf88398 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x7f207cc72a50 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> === Thread ID: 0x7f2064ebd700 LWP:22553 (attempt_thread       started at [  475] pbx_spool.c launch_service())
> === ---> Lock #0 (dial.c): MUTEX 439 ast_dial_prerun &(&dial->channels)->lock 0x5603f7a5bf08 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1b4581f] asterisk dial.c:440 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Lock #1 (channel.c): MUTEX 920 __ast_channel_alloc_ap tmp 0x5603f6dbfa10 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c3f] asterisk channel.c:922 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === ---> Waiting for Lock #2 (channel.c): MUTEX 922 __ast_channel_alloc_ap channels 0x5603f3b4e080 (1)
>         [0x5603f1b81517] asterisk lock.c:272 __ast_pthread_mutex_lock()
>         [0x5603f1ab9091] asterisk astobj2.c:241 __ao2_lock()
>         [0x5603f1af8c6e] asterisk channel.c:924 __ast_channel_alloc_ap()
>         [0x5603f1af8f56] asterisk channel.c:966 __ast_channel_alloc()
>         [0x5603f1b3d14c] asterisk core_unreal.c:1204 ast_unreal_new_channels()
>         [0x5603f1b39195] asterisk core_local.c:1027 local_request_with_stream_topology()
>         [0x5603f1b0acfa] asterisk channel.c:6347 request_channel()
>         [0x5603f1b0b3ca] asterisk channel.c:6442 ast_request()
>         [0x5603f1b45352] asterisk dial.c:368 begin_dial_prerun()
>         [0x5603f1b45845] asterisk dial.c:441 ast_dial_prerun()
>         [0x5603f1bb595e] asterisk pbx.c:8264 pbx_outgoing_attempt()
>         [0x5603f1bb64a6] asterisk pbx.c:8507 ast_pbx_outgoing_app_predial()
>         [0x5603f1bb63e7] asterisk pbx.c:8487 ast_pbx_outgoing_app()
>         [0x7f2083a3df24] pbx_spool.so pbx_spool.c:437 attempt_thread()
>         [0x5603f1c57820] asterisk utils.c:1572 dummy_start()
>         [0x7f209292afa3] libpthread.so.0 pthread_create.c:487 start_thread()
>         [0x7f20923beeff] libc.so.6 clone.S:97 clone()
> === --- ---> Locked Here: astobj2_container.c line 310 (internal_ao2_traverse)
> === -------------------------------------------------------------------
> ===
> =======================================================================
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.2#6252)



More information about the asterisk-bugs mailing list