[asterisk-bugs] [JIRA] (ASTERISK-25513) Crash: malloc failed with high load of subscriptions.

[Mark Michelson (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-25513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=228100#comment-228100 ] 

Mark Michelson commented on ASTERISK-25513:
-------------------------------------------

The most immediate thing that jumps out when looking at the attached backtrace is that it has 9066 threads in it. Doing a search for "worker_start" shows that 8990 of these threads are worker threads in threadpools. Of those, 138 of those are active, and 8852 of those are idle.

Next, the actual thread that crashed did so because an exception was thrown when attempting to allocate a block for a PJLib pool. Looking at the PJLib code, the specific exception is thrown when attempting to allocate results in a NULL return. The {{pool_policy_malloc}} block allocation policy can return NULL in two conditions:
# The pool factory's {{on_block_alloc}} function returns {{PJ_FALSE}}
# {{malloc}} returns NULL
The first condition is not possible because the caching pool used by us always returns {{PJ_TRUE}}. Therefore, the exception must have been thrown because {{malloc}} returned NULL.
In other words, the crash occurred because we could not successfully allocate memory.

In this particular case, John was performing a stress test of SIP subscriptions. In such a case, there likely was no buggy behavior that led to the number of threads created for the threadpool. Likely, a strong influx of traffic resulted in many threads being created to handle the SUBSCRIBE dialogs.

This doesn't mean that some sort of action could not be taken to help with this problem. Here are a few ideas:
# From a configuration point of view, a maximum threadpool size and a shorter idle timeout would result in fewer threads being allocated and faster reclamation of idle threads' memory. This is already made available in {{pjsip.conf}} as well as other configuration files that use threadpools and would require no code changes.
# Within the source code, we could change the algorithm for determining when to expand the threadpool. Currently, if all threads are active, and a new task is added, it results in a new batch of threads being created. It may be more prudent to expand the threadpool using some function based on the threadpool's current size and its growth delta. This would lead potentially to a growth rate that is close to logarithmic than linear in the presence of heavy spikes of traffic.
# Also within the code, we could potentially recover from the out of memory exception that PJLib throws. Since it uses an exception-throwing system, we could surround pool allocations in PJLib's try-catch macros and fail without crashing.

> Crash: malloc failed with high load of subscriptions.
> -----------------------------------------------------
>
>                 Key: ASTERISK-25513
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-25513
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: pjproject/pjsip, Resources/res_pjsip_pubsub
>    Affects Versions: 13.6.0
>         Environment: Asterisk GIT-13-9a021a4, pjproject 2.4, testsuite
>            Reporter: John Bigelow
>         Attachments: backtrace_14507.txt
>
>
> Asterisk crashed with a large load of subscriptions. Backtrace is attached. 



--
This message was sent by Atlassian JIRA
(v6.2#6252)