[asterisk-bugs] [JIRA] (ASTERISK-29474) core: Exceptionally long queue length with Local channels and ConfBridge

[Kevin Harwell (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-29474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=255370#comment-255370 ] 

Kevin Harwell commented on ASTERISK-29474:
------------------------------------------

In a typical deadlock you can usually find at least two threads being blocked, each waiting on the other.

What I usually do is take the address of the object attempting to be locked, and search elsewhere for it in the backtrace file. If you're lucky you find that same object being used in another thread that's blocked. If so, then I look at the stack trace for that thread, and the actual code and check to see if that object has indeed been locked someplace.

If the object address search does not turn up elsewhere then that probably means the object was locked in another thread (showing as blocked in the file), but locally in a function (meaning the object was not passed up/down the stack thus doesn't appear in the stack trace). Similar to the above though you can just look through the stack traces (most are usually the same), and the code trying to find where the object was locked.

Alternatively what you could be looking for if the above doesn't turn anything up is a thread that executed, locked the object, and then ended without unlocking it. This offending thread would of course not be in the backtrace, and the object it locked would be forever locked. Anything else attemping to access/lock the object would be blocked.

> core: Exceptionally long queue length with Local channels and ConfBridge
> ------------------------------------------------------------------------
>
>                 Key: ASTERISK-29474
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-29474
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Applications/app_confbridge
>    Affects Versions: 18.4.0
>            Reporter: N A
>            Assignee: Unassigned
>         Attachments: confbridge.conf, core-asterisk-running-2021-06-10T12-30-28-0400-brief.txt, core-asterisk-running-2021-06-10T12-30-28-0400-full.txt, core-asterisk-running-2021-06-10T12-30-28-0400-info.txt, core-asterisk-running-2021-06-10T12-30-28-0400-locks.txt, core-asterisk-running-2021-06-10T12-30-28-0400-thread1.txt, core-asterisk-running-2021-06-10T12-38-12-0400-brief.txt, core-asterisk-running-2021-06-10T12-38-12-0400-full.txt, core-asterisk-running-2021-06-10T12-38-12-0400-info.txt, core-asterisk-running-2021-06-10T12-38-12-0400-locks.txt, core-asterisk-running-2021-06-10T12-38-12-0400-thread1.txt, core-asterisk-running-2021-06-11T00-06-54+0000-brief.txt, core-asterisk-running-2021-06-11T00-06-54+0000-full.txt, core-asterisk-running-2021-06-11T00-06-54+0000-info.txt, core-asterisk-running-2021-06-11T00-06-54+0000-locks.txt, core-asterisk-running-2021-06-11T00-06-54+0000-thread1.txt, core-asterisk-running-2021-06-11T00-17-11+0000-brief.txt, core-asterisk-running-2021-06-11T00-17-11+0000-full.txt, core-asterisk-running-2021-06-11T00-17-11+0000-info.txt, core-asterisk-running-2021-06-11T00-17-11+0000-locks.txt, core-asterisk-running-2021-06-11T00-17-11+0000-thread1.txt, core-asterisk-running-2021-06-11T17-18-53+0000-full.txt, debug_log_123456.txt, debug_threads.txt, extensions.conf, locks.txt
>
>
> It's our friend "__ast_queue_frame: Exceptionally long queue length queuing to..." again.
> I'm mark this as app_confbridge related because 99% of the time, deadlocks on my systems are caused by ConfBridge and using other applications if possible does not cause this issue at all.
> This issue happens with ConfBridges in multiple different scenarios - below is one that replicates every single time:
> - bring another channel playing audio into a confbridge
> - bring another channel that records locally into the confbridge, but muted
> - have that original channel join that ConfBridge
> All is good. Now, as soon as, from another phone, I join that same confbridge, even if muted, this whole thing happens immediately until at least after all channels and the bridge are torn down. Usually Asterisk crashes fairly quickly when this kind of thing happens; in this specific scenario, it doesn't crash quickly but it does basically kill both channels effectively and these warnings continue until everything involved is torn down, though sometimes this persists 10-20 seconds after that. In certain other similar scenarios, Asterisk usually crashes within a minute as CPU usage climbs to 100%.
> This is just one scenario that leads to this deadlock, but it can happen in several different ways. Typically it happens when multiple parties are in this aforementioned bridge using ConfBridge. Replacing ConfBridge with ChanSpy in places where possible is enough to make the issue disappear, hence why this seems to be a bug with app_confbridge.
> Attached are output from astcoredumper and core show locks during the deadlock.
> In this particular deadlock, there seems to be some kind of infinite loop caused by infinite progress updates, although I only see the infinite progress stuff when Asterisk is compiled with DONT_OPTIMIZE and DEBUG_THREADS. Regardless, this deadlocks occurs.



--
This message was sent by Atlassian JIRA
(v6.2#6252)