[asterisk-bugs] [JIRA] (ASTERISK-25905) Memory leak during perf testing

[Robert McGilvray (JIRA)]

    [ https://issues.asterisk.org/jira/browse/ASTERISK-25905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=230301#comment-230301 ] 

Robert McGilvray commented on ASTERISK-25905:
---------------------------------------------

Hey Richard,

You're exactly right. I guess the behavior of CDR changed from 1.8, I don't think the old CDR logic created entries for every pairing and that's what's killing it. 

Anyway I replaced CDR with CEL and the issue has more or less gone away. I ran 1500 concurrency through the box overnight for a total of 1M calls. The RES size did grow to around 4G but this was with the extended cel_sqlite3_custom module. I'm running the test again with just cel_custom to see if the same thing occurs. 

Either way probably not a big deal, our weekly restart will clear up that small amount of extra memory. 

{noformat}
USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root     19138  0.0  0.0  47348  2796 pts/3    S+   10:27   0:00 rasterisk r
asterisk 29127 39.9 13.3 9120652 4360128 ?     Sl   Apr14 2899:04 /home/asterisk/asterisk/certified-13.1-cert4/sbin/asterisk -f -C /home/asterisk/asterisk/certified-13.1-cert4

ykt1cfbprd1*CLI> core show calls
2413 of 5000 max active calls (48.26% of capacity)
1064261 calls processed
{noformat}

Thanks for your help. 



> Memory leak during perf testing
> -------------------------------
>
>                 Key: ASTERISK-25905
>                 URL: https://issues.asterisk.org/jira/browse/ASTERISK-25905
>             Project: Asterisk
>          Issue Type: Bug
>      Security Level: None
>          Components: Applications/app_confbridge, pjproject/pjsip
>    Affects Versions: 13.8.0
>         Environment: Red Hat Enterprise Linux Server release 7.2 (Maipo)
> Linux ykt1cfbprd1 3.10.0-327.13.1.el7.x86_64
> certified/13.8-cert1-rc1
> pjproject-2.4.5
>            Reporter: Robert McGilvray
>            Assignee: Richard Mudgett
>            Severity: Minor
>         Attachments: loadtest.txt, memory-summary.txt
>
>
> ** I've been testing against the certified branch, last cloned yesterday with certified/13.8-cert1-rc1. It would not allow me to select that as a version however ** 
> While using sipp as a generator to load test Asterisk I've come across a memory leak that very quickly exhausts the host of resources. 
> The testing methodology is pretty simple: use sipp to launch 1500 concurrent calls to asterisk with a call rate of 25/sec. On the asterisk side use the RAND function to generate two numbers, one of which is the confbridge number and the other (either 0 or 1) is to determine whether to use the moderator profile or participant. The call is then dropped into a ConfBridge for 60s and Hungup. 
> After a few thousand completed calls the memory usage grows and eventually exhausts the host resources. I recompiled with MALLOC_DEBUG enabled, the output of memory show allocations is attached. It looks like the allocations are in stasis_channels, well after all channels have been disconnected. 
> {noformat}
> ykt1cfbprd1:/home/netops# ps -C asterisk u
> USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
> asterisk 20927  113  9.2 6292308 3029204 ?     Sl   16:32  24:47 /home/asterisk/asterisk-cert-13.8/sbin/asterisk -f -C /home/asterisk/asterisk-cert
> root     32052  0.0  0.0  47428  2840 pts/0    S+   16:43   0:00 rasterisk risk/asterisk-cert-13.8/sbin/asterisk -r
> {noformat}
> Please let me know if you need any further information.
> Thanks!!



--
This message was sent by Atlassian JIRA
(v6.2#6252)