[asterisk-bugs] [JIRA] (ASTERISK-26551) Voicemail deadlock when under load with ODBC backend

[David Moore (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-26551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=233550#comment-233550 ] 

David Moore edited comment on ASTERISK-26551 at 11/9/16 11:56 PM:
------------------------------------------------------------------

We were using max_connections of 20 for each instance of the connector, we bumped that to 200 and while I left the automated voicemails running the system deadlocked again. This lasted more than 30 minutes, when I got to the console it had 1867 active channels. The system placing the calls was doing no more than 40 simultaneously, all the channels stayed open on my debug server

I am assuming this is another ODBC blockage, I am uploading the same files as before, note that the backtrace is around 4MB

These errors were logged in connection with the deadlock, the system stopped responding a minute or two prior to the first:

[2016-11-09 23:12:36] WARNING[18754] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks.
[2016-11-09 23:15:16] WARNING[18754] taskprocessor.c: The 'subm:rtp_topic-000001e6' task processor queue reached 500 scheduled tasks.




was (Author: dmoore):
We were using max_connections of 20 for each instance of the connector, we bumped that to 200 and while I left the automated voicemails running the system deadlocked again. This lasted more than 30 minutes, when I got to the console it had 1867 active channels. The system placing the calls was doing no more than 40 simultaneously, all the channels stayed open on my debug server

I am assuming this is another ODBC blockage, I am uploading the same files as before, note that the backtrace is around 4MB

These errors were logged over the course of the deadlock:

[2016-11-08 22:28:14] WARNING[21402] taskprocessor.c: The 'subm:rtp_topic-000001e6' task processor queue reached 500 scheduled tasks.
[2016-11-09 00:20:31] WARNING[14030][C-00003df8] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks.
[2016-11-09 23:12:36] WARNING[18754] taskprocessor.c: The 'subm:manager_topic-00000007' task processor queue reached 3000 scheduled tasks.
[2016-11-09 23:15:16] WARNING[18754] taskprocessor.c: The 'subm:rtp_topic-000001e6' task processor queue reached 500 scheduled tasks.



> Voicemail deadlock when under load with ODBC backend
> ----------------------------------------------------
>
>                 Key: ASTERISK-26551
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-26551
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Applications/app_voicemail, Applications/app_voicemail/ODBC
>    Affects Versions: 13.11.2
>         Environment: Up to date Debian 8
> chan_pjsip
> Cloned a production server and enabled debug then setup automated tests to keep the voicemail system under load in order to reproduce this bug.
>            Reporter: David Moore
>            Assignee: Unassigned
>         Attachments: 2016-11-09-00_27_31-backtrace, 2016-11-09-00_27_31-channels, 2016-11-09-00_27_31-debug, 2016-11-09-00_27_31-locks, 2016-11-09-23_28_59-backtrace, 2016-11-09-23_28_59-channels, 2016-11-09-23_28_59-debug, 2016-11-09-23_28_59-locks, backtrace-threads10.txt, core-show-locks10.txt, dialplan.txt, odbc_show_all.txt
>
>
> Voicemail-only server (answering incoming calls from peer with chan_pjsip)
> Bug can be reproduced by placing the voicemail system under load, I did this by leaving many simultaneous voicemails through automated means. After running for several hours the voicemail app locks up. The system will still accept incoming calls, and enter the Voicemail app, but no audio is returned and the call does not progress further.
> The system eventually recovers and resumes normal operation, this generally takes between 3 and 20 minutes.
> I took ten backtraces and ten copies of 'core show locks' during this deadlock on my debug server, I can upload all of them if necessary, I will begin by attaching the latest data set and will upload more upon request to avoid cluttering things up unnecessarily



--
This message was sent by Atlassian JIRA
(v6.2#6252)