[asterisk-bugs] [JIRA] (ASTERISK-26551) Voicemail deadlock when under load with ODBC backend

[Joshua Colp (JIRA)]

     [ https://issues.asterisk.org/jira/browse/ASTERISK-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joshua Colp updated ASTERISK-26551:
-----------------------------------

    Assignee: David Moore  (was: Unassigned)
      Status: Waiting for Feedback  (was: Triage)

Which MySQL ODBC driver are you using? The deadlock itself appears to be as a result of the driver:

{code}
#0  0x00007fc5952a4add in read () at ../sysdeps/unix/syscall-template.S:81
#1  0x00007fc530946cc4 in vio_read_buff () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#2  0x00007fc53093680e in ?? () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#3  0x00007fc53093748d in my_net_read () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#4  0x00007fc5309308bc in cli_safe_read () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#5  0x00007fc530934243 in cli_advanced_command () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#6  0x00007fc53092c571 in mysql_ping () from /usr/lib/x86_64-linux-gnu/libmysqlclient.so.18
#7  0x00007fc530e61e24 in MySQLGetConnectAttr () from /usr/lib/x86_64-linux-gnu/odbc/libmyodbc.so
#8  0x00007fc530e4f8a3 in SQLGetConnectAttrImpl () from /usr/lib/x86_64-linux-gnu/odbc/libmyodbc.so
#9  0x00007fc53292fac8 in SQLGetConnectAttr () from /usr/lib/x86_64-linux-gnu/libodbc.so.2
#10 0x00007fc532b83c6d in connection_dead (connection=0x7fc50c149e18, class=0x2d38cf8) at res_odbc.c:778
{code}

We're doing a connection check to make sure it is still alive and it is hanging waiting to read data from the remote server.

> Voicemail deadlock when under load with ODBC backend
> ----------------------------------------------------
>
>                 Key: ASTERISK-26551
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26551
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Applications/app_voicemail, Applications/app_voicemail/ODBC
>    Affects Versions: 13.11.2
>         Environment: Up to date Debian 8
> chan_pjsip
> Cloned a production server and enabled debug then setup automated tests to keep the voicemail system under load in order to reproduce this bug.
>            Reporter: David Moore
>            Assignee: David Moore
>         Attachments: backtrace-threads10.txt, core-show-locks10.txt, dialplan.txt
>
>
> Voicemail-only server (answering incoming calls from peer with chan_pjsip)
> Bug can be reproduced by placing the voicemail system under load, I did this by leaving many simultaneous voicemails through automated means. After running for several hours the voicemail app locks up. The system will still accept incoming calls, and enter the Voicemail app, but no audio is returned and the call does not progress further.
> The system eventually recovers and resumes normal operation, this generally takes between 3 and 20 minutes.
> I took ten backtraces and ten copies of 'core show locks' during this deadlock on my debug server, I can upload all of them if necessary, I will begin by attaching the latest data set and will upload more upon request to avoid cluttering things up unnecessarily



--
This message was sent by Atlassian JIRA
(v6.2#6252)