[asterisk-bugs] [JIRA] (ASTERISK-29841) APP Queue Deadlock
André Emerich (JIRA)
noreply at issues.asterisk.org
Thu Jan 6 10:13:07 CST 2022
André Emerich created ASTERISK-29841:
----------------------------------------
Summary: APP Queue Deadlock
Key: ASTERISK-29841
URL: https://issues.asterisk.org/jira/browse/ASTERISK-29841
Project: Asterisk
Issue Type: Bug
Security Level: None
Components: Applications/app_queue
Affects Versions: 13.38.3
Environment: VMWare Virtual Machine
16 x vCPU
16 GB RAM
Reporter: André Emerich
Severity: Major
Hi,
We are facing a problem with app_queue.
>From time to time Asterisk stops accepting the command "queue show" and it is not possible to give any more commands of this type. As it is a production environment, we enabled DEBUG_THREAD for a short moment and managed to get the evidence below:
=======================================================================
=== 13.38.3
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x7f8994743700 LWP:23185 (default_tps_processing_function started at [ 228] taskprocessor.c default_listener_start())
=== ---> Lock #0 (app_queue.c): MUTEX 2430 device_state_cb q 0x3206420 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/lib64/asterisk/modules/app_queue.so(+0x1c334) [0x7f89adab9334]
/usr/sbin/asterisk() [0x5d6435]
/usr/sbin/asterisk() [0x5d6740]
/usr/sbin/asterisk(ast_taskprocessor_execute+0x167) [0x5f2d27]
/usr/sbin/asterisk() [0x5f2de0]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== ---> Waiting for Lock #1 (astobj2_container.c): MUTEX 608 internal_ao2_iterator_next iter->c 0x31b6270 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/sbin/asterisk() [0x45dc4d]
/usr/sbin/asterisk(__ao2_find+0x2d) [0x45e35d]
/usr/lib64/asterisk/modules/app_queue.so(+0x13a8e) [0x7f89adab0a8e]
/usr/lib64/asterisk/modules/app_queue.so(+0x162f5) [0x7f89adab32f5]
/usr/sbin/asterisk(ast_cli_command_full+0x689) [0x4d9209]
/usr/sbin/asterisk() [0x55e04d]
/usr/sbin/asterisk() [0x54f5d8]
/usr/sbin/asterisk() [0x55d15b]
/usr/sbin/asterisk() [0x5f5eed]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 9255 (__queues_show)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f894782f700 LWP:24184 (handle_tcptls_connection started at [ 918] tcptls.c ast_tcptls_server_root())
=== ---> Lock #0 (app_queue.c): MUTEX 9255 __queues_show queues 0x31b6270 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/lib64/asterisk/modules/app_queue.so(+0x1567b) [0x7f89adab267b]
/usr/sbin/asterisk(ast_cli_command_full+0x689) [0x4d9209]
/usr/sbin/asterisk() [0x55e04d]
/usr/sbin/asterisk() [0x54f5d8]
/usr/sbin/asterisk() [0x55d15b]
/usr/sbin/asterisk() [0x5f5eed]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== ---> Waiting for Lock #1 (app_queue.c): MUTEX 9261 __queues_show q 0x3206420 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/lib64/asterisk/modules/app_queue.so(+0x9760) [0x7f89adaa6760]
/usr/lib64/asterisk/modules/app_queue.so(+0x9983) [0x7f89adaa6983]
/usr/sbin/asterisk(ast_bridge_call_with_flags+0x112) [0x50ffd2]
/usr/lib64/asterisk/modules/app_queue.so(+0x2aad7) [0x7f89adac7ad7]
/usr/lib64/asterisk/modules/app_queue.so(+0x2ce38) [0x7f89adac9e38]
/usr/sbin/asterisk(pbx_exec+0xcc) [0x591e7c]
/usr/sbin/asterisk() [0x585a8f]
/usr/sbin/asterisk() [0x587b82]
/usr/sbin/asterisk() [0x5896db]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 2430 (device_state_cb)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f89463ad700 LWP:29630 (pbx_thread started at [ 4687] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (app_queue.c): MUTEX 1848 set_queue_variables q 0x3206420 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/lib64/asterisk/modules/app_queue.so(+0x9760) [0x7f89adaa6760]
/usr/lib64/asterisk/modules/app_queue.so(+0x9983) [0x7f89adaa6983]
/usr/sbin/asterisk(ast_bridge_call_with_flags+0x112) [0x50ffd2]
/usr/lib64/asterisk/modules/app_queue.so(+0x2aad7) [0x7f89adac7ad7]
/usr/lib64/asterisk/modules/app_queue.so(+0x2ce38) [0x7f89adac9e38]
/usr/sbin/asterisk(pbx_exec+0xcc) [0x591e7c]
/usr/sbin/asterisk() [0x585a8f]
/usr/sbin/asterisk() [0x587b82]
/usr/sbin/asterisk() [0x5896db]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 2430 (device_state_cb)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f8945bdd700 LWP:30356 (pbx_thread started at [ 4687] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (app_queue.c): MUTEX 1848 set_queue_variables q 0x3206420 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/lib64/asterisk/modules/app_queue.so(+0x9760) [0x7f89adaa6760]
/usr/lib64/asterisk/modules/app_queue.so(+0x9983) [0x7f89adaa6983]
/usr/sbin/asterisk(ast_bridge_call_with_flags+0x112) [0x50ffd2]
/usr/lib64/asterisk/modules/app_queue.so(+0x2aad7) [0x7f89adac7ad7]
/usr/lib64/asterisk/modules/app_queue.so(+0x2ce38) [0x7f89adac9e38]
/usr/sbin/asterisk(pbx_exec+0xcc) [0x591e7c]
/usr/sbin/asterisk() [0x585a8f]
/usr/sbin/asterisk() [0x587b82]
/usr/sbin/asterisk() [0x5896db]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 2430 (device_state_cb)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f8947fff700 LWP:31161 (pbx_thread started at [ 4687] pbx.c ast_pbx_start())
=== ---> Waiting for Lock #0 (astobj2_container.c): MUTEX 336 internal_ao2_traverse self 0x31b6270 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/sbin/asterisk() [0x45dc4d]
/usr/sbin/asterisk(__ao2_find+0x2d) [0x45e35d]
/usr/lib64/asterisk/modules/app_queue.so(+0x13a8e) [0x7f89adab0a8e]
/usr/lib64/asterisk/modules/app_queue.so(+0x162f5) [0x7f89adab32f5]
/usr/sbin/asterisk(ast_cli_command_full+0x689) [0x4d9209]
/usr/sbin/asterisk() [0x55e04d]
/usr/sbin/asterisk() [0x54f5d8]
/usr/sbin/asterisk() [0x55d15b]
/usr/sbin/asterisk() [0x5f5eed]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 9255 (__queues_show)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f8944bc0700 LWP:2185 (handle_tcptls_connection started at [ 918] tcptls.c ast_tcptls_server_root())
=== ---> Waiting for Lock #0 (astobj2_container.c): MUTEX 608 internal_ao2_iterator_next iter->c 0x31b6270 (1)
/usr/sbin/asterisk(__ast_pthread_mutex_lock+0x7d) [0x53cf3d]
/usr/sbin/asterisk(__ao2_lock+0x8e) [0x45bb4e]
/usr/sbin/asterisk() [0x45dc4d]
/usr/sbin/asterisk(__ao2_find+0x2d) [0x45e35d]
/usr/lib64/asterisk/modules/app_queue.so(+0x13a8e) [0x7f89adab0a8e]
/usr/lib64/asterisk/modules/app_queue.so(+0x162f5) [0x7f89adab32f5]
/usr/sbin/asterisk(ast_cli_command_full+0x689) [0x4d9209]
/usr/sbin/asterisk() [0x55e04d]
/usr/sbin/asterisk() [0x54f5d8]
/usr/sbin/asterisk() [0x55d15b]
/usr/sbin/asterisk() [0x5f5eed]
/usr/sbin/asterisk() [0x605dea]
/usr/lib64/libpthread.so.0(+0x7ea5) [0x7f8ae852dea5]
/usr/lib64/libc.so.6(clone+0x6d) [0x7f8ae75cb9fd]
=== --- ---> Locked Here: app_queue.c line 9255 (__queues_show)
Is there a patch or something to fix this?
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list