[asterisk-bugs] [JIRA] (ASTERISK-29232) Memory Leak since 16.13.0
Sean Bright (JIRA)
noreply at issues.asterisk.org
Tue Nov 2 14:24:49 CDT 2021
[ https://issues.asterisk.org/jira/browse/ASTERISK-29232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=253288#comment-253288 ]
Sean Bright edited comment on ASTERISK-29232 at 11/2/21 2:24 PM:
-----------------------------------------------------------------
So for the last hour, the instance has had 0 calls, with around 14 or so devices registered, and here are the files with some significant allocation changes:
../src/pj/pool_policy_malloc.c +742592 bytes
datastore.c +40590 bytes
res_pjsip_pubsub.c +10720 bytes
sorcery.c +2191629 bytes
There are other ones as well, but those are the largest. The total change in 1 hour has been +3034055 bytes. (Roughly 3MB)
Sorcery appears to be the worst, so here's an output of SOME allocations for it:
{noformat}
11 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
72 bytes allocated by sorcery_object_type_alloc() line 728 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
312 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
66376 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
13 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
992 bytes allocated by __ast_sorcery_open() line 637 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
52 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
48 bytes allocated by sorcery_object_type_alloc() line 735 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
48 bytes allocated by sorcery_object_type_alloc() line 735 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by __ast_sorcery_wizard_register() line 451 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
24 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
14 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
40 bytes allocated by sorcery_object_type_alloc() line 708 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
66376 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
40 bytes allocated by sorcery_object_type_alloc() line 708 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
11 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
{noformat}
was (Author: lukeescude):
So for the last hour, the instance has had 0 calls, with around 14 or so devices registered, and here are the files with some significant allocation changes:
../src/pj/pool_policy_malloc.c +742592 bytes
datastore.c +40590 bytes
res_pjsip_pubsub.c +10720 bytes
sorcery.c +2191629 bytes
There are other ones as well, but those are the largest. The total change in 1 hour has been +3034055 bytes. (Roughly 3MB)
Sorcery appears to be the worst, so here's an output of SOME allocations for it:
11 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
72 bytes allocated by sorcery_object_type_alloc() line 728 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
312 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
66376 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
13 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
992 bytes allocated by __ast_sorcery_open() line 637 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
52 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
48 bytes allocated by sorcery_object_type_alloc() line 735 of sorcery.c
544 bytes allocated by sorcery_object_type_alloc() line 714 of sorcery.c
48 bytes allocated by sorcery_object_type_alloc() line 735 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
6 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by __ast_sorcery_wizard_register() line 451 of sorcery.c
72 bytes allocated by ast_sorcery_observer_add() line 2379 of sorcery.c
24 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
14 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
40 bytes allocated by sorcery_object_type_alloc() line 708 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
1248 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
66376 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
216 bytes allocated by ast_sorcery_object_fields_register() line 1165 of sorcery.c
272 bytes allocated by ast_sorcery_lockable_alloc() line 1712 of sorcery.c
4 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
304 bytes allocated by ast_sorcery_generic_alloc() line 1728 of sorcery.c
40 bytes allocated by sorcery_object_type_alloc() line 708 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
11 bytes allocated by ast_sorcery_alloc() line 1757 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
416 bytes allocated by sorcery_object_type_alloc() line 703 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
224 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
160 bytes allocated by sorcery_object_type_alloc() line 721 of sorcery.c
240 bytes allocated by __ast_sorcery_object_field_register() line 1205 of sorcery.c
> Memory Leak since 16.13.0
> -------------------------
>
> Key: ASTERISK-29232
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-29232
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Core/PBX
> Affects Versions: 16.15.0, 16.20.0
> Environment: CentOS 7 x64
> Reporter: Luke Escude
> Severity: Major
> Labels: fax
> Attachments: Apex-Analysis.xlsx, cw1-memchart.png, Jan6-1401.csv, nw1-memchart.png, PW3-Memchart.png
>
>
> So we have around 100 instances of Asterisk 16.13.0 that have been running for over 2 months, normal load (small businesses with less than 30 users each), without issue.
> We have another 350 instances of Asterisk 16.15.0 that we've started seeing a very linear increase in memory consumption over time. Specifically, we see higher-load instances (150+ users) last only a few days before hitting our artificial 3GB ceiling and getting restarted by the OOM killer.
> There are very few differences in our implementation of the 16.13 and 16.15 versions. All versions are set up as the following:
> - CentOS 7 64-bit
> - Voicemail over ODBC
> - unixODBC 2.3.1
> - MariaDB Connector (instead of the crappy mysql connector)
> - CDR over MySQL
> - SIP Trunks are registered every 2 minutes, qualified every 15 seconds.
> - User devices register every 10 minutes, qualified every 15 seconds.
> - User devices connect via TCP more often than UDP.
> - I have NO pjsip threadpool configuration options defined. I think the default is 50 threads?
> Here is what I am about to test within the next week:
> 1. unixODBC updated to 2.3.9
> 2. Longer SIP Trunk Registration period - Maybe PJSIP is working too hard?
> 3. Longer qualify timeout - Maybe PJSIP is working too hard?
> One of my first questions: Is it SAFE to compile asterisk with MALLOC_DEBUG and just leave it on permanently? I am scared to enable it, and suddenly have a bunch of users that are experiencing issues because I've enabled something that should only be enabled in Dev.
> Sorry for the length of the post, trying to cover as much ground as possible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list