[asterisk-bugs] [JIRA] (ASTERISK-27314) chan_sip: Crash/Deadlock with realtime peers (MySQL)

Fri Oct 20 11:21:22 CDT 2017

     [ https://issues.asterisk.org/jira/browse/ASTERISK-27314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joshua Colp updated ASTERISK-27314:
-----------------------------------

    Assignee: Cyril Ramière  (was: Unassigned)
      Status: Waiting for Feedback  (was: Triage)

This issue appears to be a result of the MySQL library itself, and not Asterisk:

{noformat}
Thread 27 (Thread 0x7f3558152700 (LWP 62061)):
#0  0x00007f3517b6ec7c in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#1  0x00007f3517b7ca8b in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#2  0x00007f3517b61f0d in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#3  0x00007f3517b49316 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#4  0x00007f3517b25588 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#5  0x00007f3517b25827 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#6  0x00007f3517b265ac in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#7  0x00007f3517b1af07 in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#8  0x00007f3517b2182b in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#9  0x00007f3517b14b26 in mysql_ping () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.20
No symbol table info available.
#10 0x00007f3518107bc1 in mysql_reconnect (conn=0x3049f80) at res_config_mysql.c:1374
        __PRETTY_FUNCTION__ = "mysql_reconnect"
#11 0x00007f35181018ff in realtime_mysql (database=0x7f355814f850 "asterisk", table=0x7f355814f950 "sippeersnew", rt_fields=0x7f3530058030) at res_config_mysql.c:348
        dbh = 0x3049f80
        result = 0x0
        row = 0xcdf350
        fields = 0x7f351830c620 <mysql_engine>
        numFields = 0
        i = 5226433
        sql = 0x7f353000e4e0
        buf = 0x7f353000e410
        stringp = 0x7f355814f810 "`372024X5177"
        chunk = 0x4fc188 <find_engine+452> "H211330H213]350dH3034%("
        op = 0x0
        escape = 0x7f3518108df8 ""
        field = 0x7f3530058030
        var = 0x0
        prev = 0x0
        __PRETTY_FUNCTION__ = "realtime_mysql"
#12 0x00000000004fca00 in ast_load_realtime_all_fields (family=0x7f35178b2247 "sippeers", fields=0x7f3530058030) at config.c:3257
{noformat}

The mysql_ping function is blocking and never returns, which then causes everything else wanting to use MySQL to block.

What MySQL client library are you using? Is it the latest?

> chan_sip: Crash/Deadlock with realtime peers (MySQL)
> ----------------------------------------------------
>
>                 Key: ASTERISK-27314
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-27314
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: General
>    Affects Versions: 14.5.0, 14.6.2, 15.0.0
>         Environment: Linux Ubuntu 16.04.3 LTS (4.4.0-96-generic)
>            Reporter: Cyril Ramière
>            Assignee: Cyril Ramière
>            Severity: Critical
>              Labels: crash, deadlock
>         Attachments: asterisk-2017-10-04T15-15-31+0200.coredumps.tar.gz, asterisk.zip, core-asterisk-running-2017-10-04T15-15-31+0200-brief.txt, core-asterisk-running-2017-10-04T15-15-31+0200-full.txt, core-asterisk-running-2017-10-04T15-15-31+0200-full.txt, core-asterisk-running-2017-10-04T15-15-31+0200-locks.txt, core-asterisk-running-2017-10-04T15-15-31+0200-thread1.txt, core-asterisk-running-2017-10-04T15-15-31+0200.zip, core_show_locks.txt
>
>
> Hi, 
> I think I found a problem with asterisk > 13.x (tested on 14.5.0 & 14.6.2 & 15.0.0) & MySQL realtime.
> When there are two (or more) incoming calls made almost simultaneously, or the server is "under load", the asterisk server stops responding to INVITEs almost instantaneously, locks one CPU to 100%, then acts strange (it is not fully locked but it doesn't accept new calls, while keeping active calls up)
> Found out that when asterisk is "stuck", doing "realtime mysql status" in the console causes a segfault or freeze the console input.
> After digging through the problem, it seems that the "sippeers => mysql,asterisk,sipUsers" line in extconfig.conf is causing all the troubles.
> Note : sql schema should be fine, I used the file path/to/src/asterisk-x.x.x/contrib/realtime/mysql/mysql_config.sql, connection to the server is fine too.
> Even with an empty table the problem happens.
> I made my tests with SIPP to reproduce the issue, here is the command (or you can just "load" the server with real calls):
> (limit=10 calls at rate 3calls/sec to exten 1000)
> sipp -sn uac -d 600000 -s "1000" 10.1.1.104 -l 10 -r 3 -mp 5606
> Configuration is : 
> Ubuntu 16.04.3 LTS (4.4.0-96-generic)
> - sip.conf : my trunk
> - extensions.conf : static dialplan for test + switch realtime (not used for the test)
> - extconfig.conf : use for MOH, extensions, peers (problem here)
> - res_mysql.conf : our SQL server
> The exact same configurations is fine on our older asterisks V13.9.1 (no problems).
> Here are the "core show locks" result, dumps attached to the ticket.
> core show locks
> [Edit by Rusty - Removed and attached to issue as core_show_locks.txt - Please attached all lengthy debug to the ticket]

--
This message was sent by Atlassian JIRA
(v6.2#6252)