[asterisk-bugs] [JIRA] (ASTERISK-27314) chan_sip: Crash/Deadlock with realtime peers (MySQL)

[Cyril Ramière (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-27314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=239026#comment-239026 ] 

Cyril Ramière commented on ASTERISK-27314:
------------------------------------------

Hello,

I've attached the configuration I used, this is a "minimum" configuration just to reproduce the problem. [asterisk.zip]

The problem seems linked to the latency/response time of the sql server.

I was able to run about 100 simultaneous calls with 5 calls/sec fine with a local sql server, but it locked up after a while (seems that just a little lag/delay can trigger the issue).
When RDS is used (remote SQL server with about 10-20ms connection delay) the problem happens almost instantaneously.

You can use the following command to add some latency to your network adapter, then the problem appears instantaneously on the second call.

{{tc qdisc add dev ethXX root netem delay 30ms}}



> chan_sip: Crash/Deadlock with realtime peers (MySQL)
> ----------------------------------------------------
>
>                 Key: ASTERISK-27314
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-27314
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: General
>    Affects Versions: 14.5.0, 14.6.2, 15.0.0
>         Environment: Linux Ubuntu 16.04.3 LTS (4.4.0-96-generic)
>            Reporter: Cyril Ramière
>            Assignee: Cyril Ramière
>            Severity: Critical
>              Labels: crash, deadlock
>         Attachments: asterisk-2017-10-04T15-15-31+0200.coredumps.tar.gz, asterisk.zip, core-asterisk-running-2017-10-04T15-15-31+0200-brief.txt, core-asterisk-running-2017-10-04T15-15-31+0200-full.txt, core-asterisk-running-2017-10-04T15-15-31+0200-full.txt, core-asterisk-running-2017-10-04T15-15-31+0200-locks.txt, core-asterisk-running-2017-10-04T15-15-31+0200-thread1.txt, core-asterisk-running-2017-10-04T15-15-31+0200.zip, core_show_locks.txt
>
>
> Hi, 
> I think I found a problem with asterisk > 13.x (tested on 14.5.0 & 14.6.2 & 15.0.0) & MySQL realtime.
> When there are two (or more) incoming calls made almost simultaneously, or the server is "under load", the asterisk server stops responding to INVITEs almost instantaneously, locks one CPU to 100%, then acts strange (it is not fully locked but it doesn't accept new calls, while keeping active calls up)
> Found out that when asterisk is "stuck", doing "realtime mysql status" in the console causes a segfault or freeze the console input.
> After digging through the problem, it seems that the "sippeers => mysql,asterisk,sipUsers" line in extconfig.conf is causing all the troubles.
> Note : sql schema should be fine, I used the file path/to/src/asterisk-x.x.x/contrib/realtime/mysql/mysql_config.sql, connection to the server is fine too.
> Even with an empty table the problem happens.
> I made my tests with SIPP to reproduce the issue, here is the command (or you can just "load" the server with real calls):
> (limit=10 calls at rate 3calls/sec to exten 1000)
> sipp -sn uac -d 600000 -s "1000" 10.1.1.104 -l 10 -r 3 -mp 5606
> Configuration is : 
> Ubuntu 16.04.3 LTS (4.4.0-96-generic)
> - sip.conf : my trunk
> - extensions.conf : static dialplan for test + switch realtime (not used for the test)
> - extconfig.conf : use for MOH, extensions, peers (problem here)
> - res_mysql.conf : our SQL server
> The exact same configurations is fine on our older asterisks V13.9.1 (no problems).
> Here are the "core show locks" result, dumps attached to the ticket.
> core show locks
> [Edit by Rusty - Removed and attached to issue as core_show_locks.txt - Please attached all lengthy debug to the ticket]



--
This message was sent by Atlassian JIRA
(v6.2#6252)