[asterisk-bugs] [JIRA] (ASTERISK-29046) pbx: Deadlock when doing a reload, while simultaneously doing an ExtensionState on a pattern match hint that ends up adding an extension

[Joshua C. Colp (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-29046?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=251803#comment-251803 ] 

Joshua C. Colp commented on ASTERISK-29046:
-------------------------------------------

This deadlock is between two threads:

{noformat}
Thread 114 (Thread 0x7f9400df4700 (LWP 2958)): -> Has hints locked (in ast_add_hint), but wants contexts
#0  0x00007f9454e8e54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9454e89eb6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f9454e89daf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000053728e in __ast_pthread_mutex_lock ()
#4  0x000000000057fde6 in ast_rdlock_contexts ()
#5  0x0000000000570b28 in ast_hint_extension ()
#6  0x00000000005738e7 in ast_get_hint ()
#7  0x000000000054fac9 in manager_state_cb ()
#8  0x00000000005715cc in execute_state_callback ()
#9  0x0000000000573597 in ast_add_hint ()
#10 0x000000000057cb8b in add_priority ()
#11 0x000000000057d338 in ast_add_extension2_lockopt ()
#12 0x000000000057cbf9 in ast_add_extension2 ()
#13 0x00007f94504bf02f in pbx_load_config () from /usr/lib64/asterisk/modules/pbx_config.so
#14 0x00007f94504bfed6 in pbx_load_module () from /usr/lib64/asterisk/modules/pbx_config.so
#15 0x00007f94504c00ce in reload () from /usr/lib64/asterisk/modules/pbx_config.so
#16 0x0000000000534ea4 in ast_module_reload ()
#17 0x00000000004ce4e0 in handle_core_reload ()
#18 0x00000000004d6703 in ast_cli_command_full ()
#19 0x00000000004d6846 in ast_cli_command_multiple_full ()
#20 0x0000000000453ca7 in netconsole ()
#21 0x00000000005fd81c in dummy_start ()
#22 0x00007f9454e87ea5 in start_thread () from /lib64/libpthread.so.0
#23 0x00007f94541638dd in clone () from /lib64/libc.so.6

Thread 112 (Thread 0x7f940173b700 (LWP 923)): -> Has contexts read locked (in ast_add_extension), but wants hints
#0  0x00007f9454e8e54d in __lll_lock_wait () from /lib64/libpthread.so.0
#1  0x00007f9454e89eb6 in _L_lock_941 () from /lib64/libpthread.so.0
#2  0x00007f9454e89daf in pthread_mutex_lock () from /lib64/libpthread.so.0
#3  0x000000000053728e in __ast_pthread_mutex_lock ()
#4  0x000000000045b293 in __ao2_lock ()
#5  0x000000000057328c in ast_add_hint ()
#6  0x000000000057d625 in ast_add_extension2_lockopt ()
#7  0x000000000057cbf9 in ast_add_extension2 ()
#8  0x000000000057bf15 in ast_add_extension ()
#9  0x000000000057107e in internal_extension_state_extended ()
#10 0x00000000005710ee in ast_extension_state ()
#11 0x000000000054c081 in action_extensionstate ()
#12 0x000000000054df61 in process_message ()
#13 0x000000000054e889 in do_message ()
#14 0x000000000054eccb in session_do ()
#15 0x00000000005ea1e5 in handle_tcptls_connection ()
#16 0x00000000005fd81c in dummy_start ()
#17 0x00007f9454e87ea5 in start_thread () from /lib64/libpthread.so.0
#18 0x00007f94541638dd in clone () from /lib64/libc.so.6
{noformat}

> pbx: Deadlock when doing a reload, while simultaneously doing an ExtensionState on a pattern match hint that ends up adding an extension
> ----------------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: ASTERISK-29046
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-29046
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: PBX/General
>    Affects Versions: 13.22.0
>         Environment: FreePBX V15 with asterisk 13.22.0, Running on Virtual machine.
> 32 GB RAM, 8 Core CPU.Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz
>            Reporter: Ramarajan
>         Attachments: core-asterisk-running-2020-08-27T00-22-17-0400-brief.txt, core-asterisk-running-2020-08-27T00-22-17-0400-full.txt, core-asterisk-running-2020-08-27T00-22-17-0400-locks.txt, core-asterisk-running-2020-08-27T00-22-17-0400-thread1.txt
>
>
> Asterisk hangs when doing the core reload (apply config in FreePBX).
> This issue occurs very frequently when we do multiple reloads at a time.
> During the problematic time, asterisk not accepting any calls, registrations, and not even responding to any SIP messages. 
> The only solution to get it up and running again is to kill the asterisk process and start it over again.
> I have attached the core-dump files for your reference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)