[asterisk-bugs] [JIRA] (ASTERISK-27314) Crash/Deadlock with realtime peers (MySQL)
Cyril Ramière (JIRA)
noreply at issues.asterisk.org
Wed Oct 4 18:06:38 CDT 2017
Cyril Ramière created ASTERISK-27314:
----------------------------------------
Summary: Crash/Deadlock with realtime peers (MySQL)
Key: ASTERISK-27314
URL: https://issues.asterisk.org/jira/browse/ASTERISK-27314
Project: Asterisk
Issue Type: Bug
Security Level: None
Components: General
Affects Versions: 14.6.2, 14.5.0, 15.0.0
Environment: Linux Ubuntu 16.04.3 LTS (4.4.0-96-generic)
Reporter: Cyril Ramière
Severity: Critical
Attachments: core-asterisk-running-2017-10-04T15-15-31+0200-locks.txt, core-asterisk-running-2017-10-04T15-15-31+0200-thread1.txt
Hi,
I think I found a problem with asterisk > 13.x (tested on 14.5.0 & 14.6.2 & 15.0.0) & MySQL realtime.
When there are two (or more) incoming calls made almost simultaneously, or the server is "under load", the asterisk server stops responding to INVITEs almost instantaneously, locks one CPU to 100%, then acts strange (it is not fully locked but it doesn't accept new calls, while keeping active calls up)
Found out that when asterisk is "stuck", doing "realtime mysql status" in the console causes a segfault or freeze the console input.
After digging through the problem, it seems that the "sippeers => mysql,asterisk,sipUsers" line in extconfig.conf is causing all the troubles.
Note : sql schema should be fine, I used the file path/to/src/asterisk-x.x.x/contrib/realtime/mysql/mysql_config.sql, connection to the server is fine too.
Even with an empty table the problem happens.
I made my tests with SIPP to reproduce the issue, here is the command (or you can just "load" the server with real calls):
(limit=10 calls at rate 3calls/sec to exten 1000)
sipp -sn uac -d 600000 -s "1000" 10.1.1.104 -l 10 -r 3 -mp 5606
Configuration is :
Ubuntu 16.04.3 LTS (4.4.0-96-generic)
- sip.conf : my trunk
- extensions.conf : static dialplan for test + switch realtime (not used for the test)
- extconfig.conf : use for MOH, extensions, peers (problem here)
- res_mysql.conf : our SQL server
The exact same configurations is fine on our older asterisks V13.9.1 (no problems).
Here are the "core show locks" result, dumps attached to the ticket.
core show locks
=======================================================================
=== 14.6.2
=== Currently Held Locks
=======================================================================
===
=== <pending> <lock#> (<file>): <lock type> <line num> <function> <lock name> <lock addr> (times locked)
===
=== Thread ID: 0x7f355af35700 LWP:62038 (do_devstate_changes started at [ 646] devicestate.c ast_device_state_engine_init())
=== ---> Lock #0 (res_config_mysql.c): RDLOCK 154 find_database &(&databases)->lock 0x7f351830c310 (1)
main/backtrace.c:59 __ast_bt_get_addresses() (0x472b94+1D)
main/lock.c:866 __ast_rwlock_rdlock() (0x56364c+D8)
addons/res_config_mysql.c:155 find_database()
addons/res_config_mysql.c:329 realtime_mysql()
main/config.c:3257 ast_load_realtime_all_fields() (0x4fc94c+B4)
main/config.c:3291 ast_load_realtime_fields() (0x4fcbaa+32)
main/config.c:3340 ast_load_realtime() (0x4fccca+11E)
channels/chan_sip.c:5445 realtime_peer_by_name()
channels/chan_sip.c:5632 realtime_peer()
channels/chan_sip.c:5747 sip_find_peer_full()
channels/chan_sip.c:5786 sip_find_peer()
channels/chan_sip.c:30367 sip_devicestate()
main/devicestate.c:382 _ast_device_state()
main/devicestate.c:472 do_state_change()
main/devicestate.c:565 do_devstate_changes()
main/utils.c:1233 dummy_start()
:0 start_thread()
libc.so.6 clone() (0x7f355e172370+6D)
=== ---> Waiting for Lock #1 (res_config_mysql.c): MUTEX 157 find_database &cur->lock 0x3049f88 (1)
main/backtrace.c:59 __ast_bt_get_addresses() (0x472b94+1D)
main/lock.c:258 __ast_pthread_mutex_lock() (0x5615bb+D8)
addons/res_config_mysql.c:158 find_database()
addons/res_config_mysql.c:329 realtime_mysql()
main/config.c:3257 ast_load_realtime_all_fields() (0x4fc94c+B4)
main/config.c:3291 ast_load_realtime_fields() (0x4fcbaa+32)
main/config.c:3340 ast_load_realtime() (0x4fccca+11E)
channels/chan_sip.c:5445 realtime_peer_by_name()
channels/chan_sip.c:5632 realtime_peer()
channels/chan_sip.c:5747 sip_find_peer_full()
channels/chan_sip.c:5786 sip_find_peer()
channels/chan_sip.c:30367 sip_devicestate()
main/devicestate.c:382 _ast_device_state()
main/devicestate.c:472 do_state_change()
main/devicestate.c:565 do_devstate_changes()
main/utils.c:1233 dummy_start()
:0 start_thread()
libc.so.6 clone() (0x7f355e172370+6D)
=== --- ---> Locked Here: res_config_mysql.c line 157 (find_database)
=== -------------------------------------------------------------------
===
=== Thread ID: 0x7f3558152700 LWP:62061 (do_monitor started at [29732] chan_sip.c restart_monitor())
=== ---> Lock #0 (chan_sip.c): MUTEX 29092 handle_request_do &netlock 0x7f3517ada420 (1)
main/backtrace.c:59 __ast_bt_get_addresses() (0x472b94+1D)
main/lock.c:258 __ast_pthread_mutex_lock() (0x5615bb+D8)
channels/chan_sip.c:29095 handle_request_do()
channels/chan_sip.c:29054 sipsock_read()
main/io.c:295 ast_io_wait() (0x557618+193)
channels/chan_sip.c:29699 do_monitor()
main/utils.c:1233 dummy_start()
:0 start_thread()
libc.so.6 clone() (0x7f355e172370+6D)
=== ---> Lock #1 (chan_sip.c): MUTEX 9328 sip_pvt_lock_full pvt 0x7f35300537b0 (1)
main/backtrace.c:59 __ast_bt_get_addresses() (0x472b94+1D)
main/lock.c:258 __ast_pthread_mutex_lock() (0x5615bb+D8)
main/astobj2.c:211 __ao2_lock() (0x466174+DE)
channels/chan_sip.c:9329 sip_pvt_lock_full()
channels/chan_sip.c:29108 handle_request_do()
channels/chan_sip.c:29054 sipsock_read()
main/io.c:295 ast_io_wait() (0x557618+193)
channels/chan_sip.c:29699 do_monitor()
main/utils.c:1233 dummy_start()
:0 start_thread()
libc.so.6 clone() (0x7f355e172370+6D)
=== ---> Lock #2 (res_config_mysql.c): MUTEX 157 find_database &cur->lock 0x3049f88 (1)
main/backtrace.c:59 __ast_bt_get_addresses() (0x472b94+1D)
main/lock.c:258 __ast_pthread_mutex_lock() (0x5615bb+D8)
addons/res_config_mysql.c:158 find_database()
addons/res_config_mysql.c:329 realtime_mysql()
main/config.c:3257 ast_load_realtime_all_fields() (0x4fc94c+B4)
main/config.c:3291 ast_load_realtime_fields() (0x4fcbaa+32)
main/config.c:3340 ast_load_realtime() (0x4fccca+11E)
channels/chan_sip.c:5445 realtime_peer_by_name()
channels/chan_sip.c:5632 realtime_peer()
channels/chan_sip.c:5747 sip_find_peer_full()
channels/chan_sip.c:5786 sip_find_peer()
channels/chan_sip.c:19075 check_peer_ok()
channels/chan_sip.c:19406 check_user_full()
channels/chan_sip.c:26349 handle_request_invite()
channels/chan_sip.c:28913 handle_incoming()
channels/chan_sip.c:29122 handle_request_do()
channels/chan_sip.c:29054 sipsock_read()
main/io.c:295 ast_io_wait() (0x557618+193)
channels/chan_sip.c:29699 do_monitor()
main/utils.c:1233 dummy_start()
:0 start_thread()
libc.so.6 clone() (0x7f355e172370+6D)
=== -------------------------------------------------------------------
===
=======================================================================
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list