[asterisk-bugs] [JIRA] (ASTERISK-27121) res_pjsip_mwi: Memory leak on reload
Sergej (JIRA)
noreply at issues.asterisk.org
Thu Jul 13 01:40:57 CDT 2017
[ https://issues.asterisk.org/jira/browse/ASTERISK-27121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=237701#comment-237701 ]
Sergej commented on ASTERISK-27121:
-----------------------------------
Sure, here is the snippet output.
So unmodified version:
{noformat}
==== Leaked Object 0x21a81c8 history ====
[12352] stasis.c:354 stasis_topic_create: +1 100 at default - [**constructor**]
[12352] stasis.c:929 stasis_forward_all: +1 - [1]
[12352] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[12352] stasis.c:969 subscription_change_alloc: +1 - [3]
[12352] stasis.c:828 publish_msg: +1 - [4]
[12352] stasis.c:838 publish_msg: -1 - [5]
[12352] stasis.c:828 publish_msg: +1 - [4]
[12352] stasis.c:838 publish_msg: -1 - [5]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [4]
[12687] stasis.c:969 subscription_change_alloc: +1 - [5] <- important step (called from stasis_unsubscribe)
[12687] stasis.c:828 publish_msg: +1 - [6]
[12687] stasis.c:954 subscription_change_dtor: -1 - [7] <- destructor for subscription_change_alloc for subscription (not unsubscribe)
[12687] stasis.c:838 publish_msg: -1 - [6]
[12687] stasis.c:541 _dtor_topic: -1 - [5]
[12687] stasis.c:409 subscription_dtor: -1 - [4]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [3]
[12687] stasis.c:969 subscription_change_alloc: +1 - [4]
[12687] stasis.c:828 publish_msg: +1 - [5]
[12687] stasis.c:838 publish_msg: -1 - [6]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [5]
[12687] stasis.c:969 subscription_change_alloc: +1 - [6]
[12687] stasis.c:828 publish_msg: +1 - [7]
[12687] stasis.c:954 subscription_change_dtor: -1 - [8]
[12687] stasis.c:838 publish_msg: -1 - [7]
[12687] stasis.c:541 _dtor_topic: -1 - [6]
[12687] stasis.c:409 subscription_dtor: -1 - [5]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [4]
[12687] stasis.c:969 subscription_change_alloc: +1 - [5]
[12687] stasis.c:828 publish_msg: +1 - [6]
[12687] stasis.c:838 publish_msg: -1 - [7]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [6]
[12687] stasis.c:969 subscription_change_alloc: +1 - [7]
[12687] stasis.c:828 publish_msg: +1 - [8]
[12687] stasis.c:954 subscription_change_dtor: -1 - [9]
[12687] stasis.c:838 publish_msg: -1 - [8]
[12687] stasis.c:541 _dtor_topic: -1 - [7]
[12687] stasis.c:409 subscription_dtor: -1 - [6]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [5]
[12687] stasis.c:969 subscription_change_alloc: +1 - [6]
[12687] stasis.c:828 publish_msg: +1 - [7]
[12687] stasis.c:838 publish_msg: -1 - [8]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [7]
[12687] stasis.c:969 subscription_change_alloc: +1 - [8]
[12687] stasis.c:828 publish_msg: +1 - [9]
[12687] stasis.c:954 subscription_change_dtor: -1 - [10]
[12687] stasis.c:838 publish_msg: -1 - [9]
[12687] stasis.c:541 _dtor_topic: -1 - [8]
[12687] stasis.c:409 subscription_dtor: -1 - [7]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [6]
[12687] stasis.c:969 subscription_change_alloc: +1 - [7]
[12687] stasis.c:828 publish_msg: +1 - [8]
[12687] stasis.c:838 publish_msg: -1 - [9]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [8]
[12687] stasis.c:969 subscription_change_alloc: +1 - [9]
[12687] stasis.c:828 publish_msg: +1 - [10]
[12687] stasis.c:954 subscription_change_dtor: -1 - [11]
[12687] stasis.c:838 publish_msg: -1 - [10]
[12687] stasis.c:541 _dtor_topic: -1 - [9]
[12687] stasis.c:409 subscription_dtor: -1 - [8]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [7]
[12687] stasis.c:969 subscription_change_alloc: +1 - [8]
[12687] stasis.c:828 publish_msg: +1 - [9]
[12687] stasis.c:838 publish_msg: -1 - [10]
— reload —
[12687] stasis.c:541 stasis_unsubscribe: +1 - [9]
[12687] stasis.c:969 subscription_change_alloc: +1 - [10]
[12687] stasis.c:828 publish_msg: +1 - [11]
[12687] stasis.c:954 subscription_change_dtor: -1 - [12]
[12687] stasis.c:838 publish_msg: -1 - [11]
[12687] stasis.c:541 _dtor_topic: -1 - [10]
[12687] stasis.c:409 subscription_dtor: -1 - [9]
[12687] stasis.c:498 internal_stasis_subscribe: +1 - [8]
[12687] stasis.c:969 subscription_change_alloc: +1 - [9]
[12687] stasis.c:828 publish_msg: +1 - [10]
[12687] stasis.c:838 publish_msg: -1 - [11]
{noformat}
As you can see reference count keeps growing, due to the fact destructor for unsubscribe part of subscription_change_alloc is not happening.
Now, if we modify stasis_cache a bit:
{noformat}
@@ -803,6 +805,11 @@ static void caching_topic_exec(void *data, struct stasis_subscription *sub,
const struct ast_eid *msg_eid;
const char *msg_id;
+ msg_type = stasis_message_type(message);
+ if (msg_type == stasis_subscription_change_type()) {
+ return;
+ }
+
ast_assert(caching_topic != NULL);
ast_assert(caching_topic->topic != NULL);
ast_assert(caching_topic->cache != NULL);
{noformat}
Modified version:
{noformat}
==== Leaked Object 0x2197528 history ====
[18861] stasis.c:354 stasis_topic_create: +1 100 at default - [**constructor**]
[18861] stasis.c:932 stasis_forward_all: +1 - [1]
[18861] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[18861] stasis.c:973 subscription_change_alloc: +1 - [3]
[18861] stasis.c:831 publish_msg: +1 - [4]
[18861] stasis.c:841 publish_msg: -1 - [5]
[19003] stasis.c:958 subscription_change_dtor: -1 - [4] <- destructor for subscription_change_alloc (subscribe)
[18861] stasis.c:831 publish_msg: +1 - [3]
[18861] stasis.c:841 publish_msg: -1 - [4]
— reload —
[19015] stasis.c:541 stasis_unsubscribe: +1 - [3]
[19015] stasis.c:973 subscription_change_alloc: +1 - [4]
[19015] stasis.c:831 publish_msg: +1 - [5]
[19015] stasis.c:841 publish_msg: -1 - [6]
[19015] stasis.c:541 _dtor_topic: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4] <- destructor for subscription_change_alloc (unsubscribe)
[19015] stasis.c:409 subscription_dtor: -1 - [3]
[19015] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[19015] stasis.c:973 subscription_change_alloc: +1 - [3]
[19015] stasis.c:831 publish_msg: +1 - [4]
[19015] stasis.c:841 publish_msg: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
— reload —
[19015] stasis.c:541 stasis_unsubscribe: +1 - [3]
[19015] stasis.c:973 subscription_change_alloc: +1 - [4]
[19015] stasis.c:831 publish_msg: +1 - [5]
[19015] stasis.c:841 publish_msg: -1 - [6]
[19015] stasis.c:541 _dtor_topic: -1 - [5]
[19074] stasis.c:958 subscription_change_dtor: -1 - [4]
[19015] stasis.c:409 subscription_dtor: -1 - [3]
[19015] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[19015] stasis.c:973 subscription_change_alloc: +1 - [3]
[19015] stasis.c:831 publish_msg: +1 - [4]
[19015] stasis.c:841 publish_msg: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
— reload —
[19015] stasis.c:541 stasis_unsubscribe: +1 - [3]
[19015] stasis.c:973 subscription_change_alloc: +1 - [4]
[19015] stasis.c:831 publish_msg: +1 - [5]
[19015] stasis.c:841 publish_msg: -1 - [6]
[19015] stasis.c:541 _dtor_topic: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
[19002] stasis.c:409 subscription_dtor: -1 - [3]
[19015] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[19015] stasis.c:973 subscription_change_alloc: +1 - [3]
[19015] stasis.c:831 publish_msg: +1 - [4]
[19015] stasis.c:841 publish_msg: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
— reload —
[19015] stasis.c:541 stasis_unsubscribe: +1 - [3]
[19015] stasis.c:973 subscription_change_alloc: +1 - [4]
[19015] stasis.c:831 publish_msg: +1 - [5]
[19015] stasis.c:841 publish_msg: -1 - [6]
[19015] stasis.c:541 _dtor_topic: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
[19002] stasis.c:409 subscription_dtor: -1 - [3]
[19015] stasis.c:498 internal_stasis_subscribe: +1 - [2]
[19015] stasis.c:973 subscription_change_alloc: +1 - [3]
[19015] stasis.c:831 publish_msg: +1 - [4]
[19015] stasis.c:841 publish_msg: -1 - [5]
[19002] stasis.c:958 subscription_change_dtor: -1 - [4]
{noformat}
Some notes about configuration:
- Static pjsip.conf
- Have enough PJSIP endpoints with unsolicited MWI turned on (the more you have, it will increase due to number of MWIs)
> res_pjsip_mwi: Memory leak on reload
> ------------------------------------
>
> Key: ASTERISK-27121
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-27121
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Affects Versions: 13.17.0
> Reporter: Sergej
> Assignee: Benjamin Keith Ford
>
> There seems to be a memory leak around stasis_cache which can be easily replicated with reload of res_pjsip_mwi module. I have tracked it down with REF_DEBUG config option. This is on git version of Asterisk 13 branch (cad74cdd8f).
> My test case includes around 30 PJSIP endpoints with defined mailboxes option.
> The number of endpoints will affect the impact of memory leak due to number of active MWI subscriptions.
> Basically, here is the replicable scenario:
> Fetch RSS of Asterisk process
> Perform 200 reloads of res_pjsip_mwi
> Fetch RSS of Asterisk process again
> {noformat}
> ps -o rss -p $(pidof -s asterisk)
> RSS
> 69048
> for _ in $(seq 1 200); do asterisk -rx 'module reload res_pjsip_mwi.so' >/dev/null; done
> ps -o rss -p $(pidof -s asterisk)
> RSS
> 90460
> {noformat}
> By repeating the for loop, RSS of Asterisk process simply grows.
> The problem seems to be around subscribe and unsubscribe message that got cached in stasis_cache. Each subscribe message is cached. However, on unsubscribe message, subscribe message is replaced with unsubscribe message and is never freed as the id of new subscription messages changes.
> I cannot know for sure what is an ideal fix for this, but so far I was able to stop it by doing one of the following:
> 1) Do not cache stasis_subscription_change_type() messages in stasis_cache: caching_topic_exec. It seems app_voicemail is reading these messages from cache when starting polling thread, so not sure if this is ideal solution.
> 2) On unsubscribe message received in caching_topic_exec, make sure to remove subscription and cleanup the received message
> 3) Add another stasis_cache_clear_type() message in main/stasis.c for type stasis_subscription_change_type when doing unsubscribe
> When such fix is performed RSS does grow a bit, but then it slows down considerably.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list