[asterisk-bugs] [JIRA] (ASTERISK-29232) Memory Leak since 16.13.0
Luke Escude (JIRA)
noreply at issues.asterisk.org
Fri Nov 5 11:15:50 CDT 2021
[ https://issues.asterisk.org/jira/browse/ASTERISK-29232?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=256782#comment-256782 ]
Luke Escude commented on ASTERISK-29232:
----------------------------------------
Only 1 socket between Asterisk and the proxy (created by asterisk REGISTER to proxy). Then phone dialogs come in through it - So 100 devices could easily register through that 1 socket.
No DNS is used, we got rid of it forever ago in favor of raw IPs. Really cut down on basically all possible latency besides basic physics.
This is on my dev box, with only like 8 extensions on it:
{code}
pjsip dump endpt details
sip_endpoint.c Dumping endpoint 0x3521148:
cachpool Dumping caching pool:
cachpool Capacity=72192, max_capacity=1048576, used_cnt=183
cachpool Dumping all active pools:
cachpool pept0x35210a0: 26384464 of 26388096 (99%) used
cachpool tpmgr: 4824 of 5440 (88%) used
cachpool SIP: 240 of 1024 (23%) used
cachpool tsxlayer: 524600 of 525312 (99%) used
cachpool ua0x3500280: 529864 of 530432 (99%) used
cachpool tcptp: 1400 of 1536 (91%) used
cachpool glck0x39372c0: 584 of 1024 (57%) used
cachpool udp0x39388d0: 1320 of 1536 (85%) used
cachpool glck0x3938e10: 584 of 1024 (57%) used
cachpool rtd0x3939350: 66600 of 72096 (92%) used
cachpool resolver.c: 11696 of 16096 (72%) used
cachpool glck0x7fb4c4005860: 584 of 1024 (57%) used
cachpool evsub: 3696 of 5632 (65%) used
cachpool regc0x7fb4c4005b00: 5128 of 6144 (83%) used
cachpool regc0x7fb4c40036c0: 5128 of 6144 (83%) used
cachpool regc0x7fb4c4004de0: 4232 of 5120 (82%) used
cachpool regc0x7fb4c400b070: 4232 of 5120 (82%) used
cachpool auth_cli0x7fb4c4045a70: 192 of 1024 (18%) used
cachpool auth_cli0x7fb4c4046fb0: 192 of 1024 (18%) used
cachpool auth_cli0x7fb4c41e9c70: 184 of 1024 (17%) used
cachpool auth_cli0x7fb4c40f7c90: 184 of 1024 (17%) used
cachpool tcp: 69072 of 69120 (99%) used
cachpool glck0x7fb4c4223e90: 584 of 1024 (57%) used
cachpool rtd0x7fb4f018e730: 168 of 4096 (4%) used
cachpool tcp: 68720 of 69120 (99%) used
cachpool glck0x7fb52c09d620: 584 of 1024 (57%) used
cachpool rtd0x7fb4f0023a80: 168 of 4096 (4%) used
cachpool dlg0x7fb4c8000f90: 121928 of 131584 (92%) used
cachpool glck0x495b420: 584 of 1024 (57%) used
cachpool glck0x7fb4c4042650: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc1d8bb0: 121928 of 131584 (92%) used
cachpool glck0x7fb560014e80: 584 of 1024 (57%) used
cachpool glck0x7fb4c439b200: 584 of 1024 (57%) used
cachpool dlg0x7fb4c800ecd0: 121928 of 131584 (92%) used
cachpool glck0x7fb4c45d6ec0: 584 of 1024 (57%) used
cachpool glck0x7fb56000d2c0: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc3c0140: 121904 of 131584 (92%) used
cachpool glck0x7fb4c43bc1c0: 584 of 1024 (57%) used
cachpool glck0x7fb4c422b6a0: 584 of 1024 (57%) used
cachpool dlg0x7fb5540242e0: 121888 of 131584 (92%) used
cachpool glck0x7fb4e4004630: 584 of 1024 (57%) used
cachpool glck0x7fb52c09ecf0: 584 of 1024 (57%) used
cachpool dlg0x7fb4c416ba20: 121904 of 131584 (92%) used
cachpool glck0x7fb4c44eabe0: 584 of 1024 (57%) used
cachpool glck0x7fb4c467b300: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc0ceff0: 121952 of 131584 (92%) used
cachpool glck0x7fb4c427ac70: 584 of 1024 (57%) used
cachpool glck0x7fb4c44e6130: 584 of 1024 (57%) used
cachpool dlg0x7fb564402a60: 121952 of 131584 (92%) used
cachpool glck0x7fb4c422aa00: 584 of 1024 (57%) used
cachpool glck0x7fb54c069dd0: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc3135e0: 121936 of 131584 (92%) used
cachpool glck0x3b6ce70: 584 of 1024 (57%) used
cachpool glck0x7fb554000ff0: 584 of 1024 (57%) used
cachpool dlg0x7fb54c065920: 121952 of 131584 (92%) used
cachpool glck0x7fb5540343b0: 584 of 1024 (57%) used
cachpool glck0x7fb4c4116840: 584 of 1024 (57%) used
cachpool dlg0x7fb5540bf870: 121952 of 131584 (92%) used
cachpool glck0x7fb52c25af50: 584 of 1024 (57%) used
cachpool glck0x7fb4c419c330: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc057cc0: 121952 of 131584 (92%) used
cachpool glck0x7fb4c40fb410: 584 of 1024 (57%) used
cachpool glck0x7fb4c463a370: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0383b30: 121952 of 131584 (92%) used
cachpool glck0x7fb560007870: 584 of 1024 (57%) used
cachpool glck0x7fb4c40cb010: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc38c4b0: 121936 of 131584 (92%) used
cachpool glck0x7fb4dc3d7dd0: 584 of 1024 (57%) used
cachpool glck0x7fb4c4032c40: 584 of 1024 (57%) used
cachpool dlg0x7fb4f037d5c0: 121952 of 131584 (92%) used
cachpool glck0x7fb4c4506a20: 584 of 1024 (57%) used
cachpool glck0x7fb4c44ef680: 584 of 1024 (57%) used
cachpool dlg0x7fb4f048a410: 121952 of 131584 (92%) used
cachpool glck0x7fb4c42b7c20: 584 of 1024 (57%) used
cachpool glck0x7fb554078970: 584 of 1024 (57%) used
cachpool dlg0x7fb540058b80: 121952 of 131584 (92%) used
cachpool glck0x7fb4c44e5550: 584 of 1024 (57%) used
cachpool glck0x7fb4c4453bd0: 584 of 1024 (57%) used
cachpool dlg0x7fb52c0f2a90: 121952 of 131584 (92%) used
cachpool glck0x3fa6e40: 584 of 1024 (57%) used
cachpool glck0x7fb4c4254f30: 584 of 1024 (57%) used
cachpool dlg0x49c6c00: 121952 of 131584 (92%) used
cachpool glck0x7fb4c8004370: 584 of 1024 (57%) used
cachpool glck0x7fb4c417ebe0: 584 of 1024 (57%) used
cachpool dlg0x7fb5640764f0: 121952 of 131584 (92%) used
cachpool glck0x7fb4c43eac90: 584 of 1024 (57%) used
cachpool glck0x3face00: 584 of 1024 (57%) used
cachpool dlg0x7fb554031690: 121952 of 131584 (92%) used
cachpool glck0x7fb4c4348ab0: 584 of 1024 (57%) used
cachpool glck0x4938690: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0295550: 121936 of 131584 (92%) used
cachpool glck0x7fb4d400b460: 584 of 1024 (57%) used
cachpool glck0x7fb4f03cf540: 584 of 1024 (57%) used
cachpool dlg0x7fb554015460: 121952 of 131584 (92%) used
cachpool glck0x4a67690: 584 of 1024 (57%) used
cachpool glck0x7fb4f0381560: 584 of 1024 (57%) used
cachpool dlg0x7fb560006fd0: 121952 of 131584 (92%) used
cachpool glck0x7fb4c443f120: 584 of 1024 (57%) used
cachpool glck0x7fb4c4051760: 584 of 1024 (57%) used
cachpool dlg0x7fb4c40e4990: 121952 of 131584 (92%) used
cachpool glck0x7fb5643f27f0: 584 of 1024 (57%) used
cachpool glck0x7fb4c419e850: 584 of 1024 (57%) used
cachpool dlg0x7fb540084470: 208808 of 225792 (92%) used
cachpool glck0x7fb4c416aa60: 584 of 1024 (57%) used
cachpool glck0x7fb4c4344320: 584 of 1024 (57%) used
cachpool dlg0x7fb4c40d5660: 208808 of 225792 (92%) used
cachpool glck0x7fb4c4173e30: 584 of 1024 (57%) used
cachpool glck0x7fb4c474e1b0: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc00b510: 208808 of 225792 (92%) used
cachpool glck0x7fb5643dc040: 584 of 1024 (57%) used
cachpool glck0x7fb5641ca9c0: 584 of 1024 (57%) used
cachpool dlg0x7fb54c26d6a0: 121880 of 131584 (92%) used
cachpool glck0x495e9d0: 584 of 1024 (57%) used
cachpool glck0x7fb4c4376350: 584 of 1024 (57%) used
cachpool dlg0x7fb56425b920: 121880 of 131584 (92%) used
cachpool glck0x495fb80: 584 of 1024 (57%) used
cachpool glck0x7fb4c41721d0: 584 of 1024 (57%) used
cachpool dlg0x7fb4c42667f0: 121880 of 131584 (92%) used
cachpool glck0x7fb4c44a0c30: 584 of 1024 (57%) used
cachpool glck0x7fb4dc095010: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0270820: 121880 of 131584 (92%) used
cachpool glck0x7fb4c4200470: 584 of 1024 (57%) used
cachpool glck0x7fb4c414e790: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc0c61d0: 208648 of 225792 (92%) used
cachpool glck0x7fb4c4442630: 584 of 1024 (57%) used
cachpool glck0x7fb4c4016d90: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0287450: 208648 of 225792 (92%) used
cachpool glck0x7fb4c424d710: 584 of 1024 (57%) used
cachpool glck0x7fb4c4375ea0: 584 of 1024 (57%) used
cachpool dlg0x7fb4c4163fa0: 208648 of 225792 (92%) used
cachpool glck0x7fb4c40e5520: 584 of 1024 (57%) used
cachpool glck0x3fb2560: 584 of 1024 (57%) used
cachpool dlg0x7fb54c0b8e50: 208648 of 225792 (92%) used
cachpool glck0x7fb4c4413280: 584 of 1024 (57%) used
cachpool glck0x7fb4dc0caa40: 584 of 1024 (57%) used
cachpool dlg0x7fb4c4393520: 216840 of 235008 (92%) used
cachpool glck0x7fb4c402dd20: 584 of 1024 (57%) used
cachpool glck0x7fb4c40311e0: 584 of 1024 (57%) used
cachpool dlg0x7fb4c4396570: 217776 of 236032 (92%) used
cachpool glck0x7fb4c4194b30: 584 of 1024 (57%) used
cachpool glck0x7fb5643db860: 584 of 1024 (57%) used
cachpool dlg0x7fb554179100: 77176 of 83456 (92%) used
cachpool glck0x7fb4f037d320: 584 of 1024 (57%) used
cachpool glck0x7fb4c44e8c30: 584 of 1024 (57%) used
cachpool dlg0x7fb4f043d900: 17688 of 18944 (93%) used
cachpool glck0x7fb4c40ce350: 584 of 1024 (57%) used
cachpool glck0x7fb4c4362970: 584 of 1024 (57%) used
cachpool dlg0x7fb54c1bf880: 16744 of 17920 (93%) used
cachpool glck0x7fb4f0191250: 584 of 1024 (57%) used
cachpool glck0x7fb4c481a200: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0450cd0: 17696 of 18944 (93%) used
cachpool glck0x7fb4f007dc00: 584 of 1024 (57%) used
cachpool glck0x7fb4c41b6910: 584 of 1024 (57%) used
cachpool dlg0x7fb5540b25e0: 17688 of 18944 (93%) used
cachpool glck0x7fb4c40360f0: 584 of 1024 (57%) used
cachpool glck0x7fb4c405ef70: 584 of 1024 (57%) used
cachpool dlg0x7fb56407c9b0: 17688 of 18944 (93%) used
cachpool glck0x7fb4dc04a120: 584 of 1024 (57%) used
cachpool glck0x7fb4c4345e20: 584 of 1024 (57%) used
cachpool dlg0x7fb4f0458830: 17688 of 18944 (93%) used
cachpool glck0x7fb554059a10: 584 of 1024 (57%) used
cachpool glck0x7fb4c421db10: 584 of 1024 (57%) used
cachpool dlg0x7fb4c45ad270: 17672 of 18944 (93%) used
cachpool glck0x7fb4f04afff0: 584 of 1024 (57%) used
cachpool glck0x7fb5640c7530: 584 of 1024 (57%) used
cachpool dlg0x7fb4f03ccdd0: 17696 of 18944 (93%) used
cachpool glck0x4a124e0: 584 of 1024 (57%) used
cachpool glck0x7fb4c44f1ba0: 584 of 1024 (57%) used
cachpool dlg0x7fb56402d570: 17680 of 18944 (93%) used
cachpool glck0x7fb564189d90: 584 of 1024 (57%) used
cachpool glck0x7fb4f054aa80: 584 of 1024 (57%) used
cachpool dlg0x7fb4c44f9290: 17736 of 18944 (93%) used
cachpool glck0x7fb4f054f370: 584 of 1024 (57%) used
cachpool glck0x7fb55406f930: 584 of 1024 (57%) used
cachpool dlg0x7fb5540bc140: 17688 of 18944 (93%) used
cachpool glck0x7fb5540baa30: 584 of 1024 (57%) used
cachpool glck0x7fb4c44fc730: 584 of 1024 (57%) used
cachpool dlg0x7fb4c43d6a90: 17696 of 18944 (93%) used
cachpool glck0x7fb4c449e700: 584 of 1024 (57%) used
cachpool glck0x7fb4dc09c100: 584 of 1024 (57%) used
cachpool dlg0x7fb4dc317e50: 17680 of 18944 (93%) used
cachpool glck0x7fb4dc3c2ff0: 584 of 1024 (57%) used
cachpool glck0x7fb4dc22e630: 584 of 1024 (57%) used
cachpool Total 33487848 of 34028096 (98 %) used!
sip_endpoint.c Endpoint pool capacity=26388096, used_size=26384464
resolver.c Dumping resolver state:
resolver.c Name servers:
resolver.c NS 0: 208.67.222.222:53 (state=Active until -75763s, rtt=10000 ms)
resolver.c NS 1: 8.8.8.8:53 (state=Active until -75763s, rtt=10000 ms)
resolver.c NS 2: 1.1.1.1:53 (state=Active until -75763s, rtt=10000 ms)
resolver.c Nb. of cached responses: 0
resolver.c Nb. of pending queries: 0 (0)
resolver.c Nb. of pending query free nodes: 0
resolver.c Nb. of timer entries: 56
resolver.c Pool capacity: 16096, used size: 11696
sip_transport.c Dumping listeners:
sip_transport.c tcptp:5060 TCP:172.17.0.3:5060
sip_transport.c Dumping transports:
sip_transport.c udp0x39388d0 x-ast-txp:transport-udp(refcnt=3)
sip_transport.c tcpc0x7fb4c4382ca8 TCP to 207.246.88.196:4242 listener[tcptp:5060](refcnt=2)
sip_transport.c tcpc0x7fb4dc146478 TCP to 104.238.165.75:4242 listener[tcptp:5060](refcnt=2)
timer.c Dumping timer heap:
timer.c Cur size: 56 entries, max: 262142
timer.c Entries:
timer.c _id Id Elapsed Source
timer.c ----------------------------------
timer.c 30 0 21.296 res_pjsip_outbound_registration.c:683
timer.c 82 0 103.700 res_pjsip_outbound_registration.c:683
timer.c 25 0 23.500 res_pjsip_outbound_registration.c:683
timer.c 53 0 104.764 res_pjsip_outbound_registration.c:683
timer.c 20 2 347.630 ../src/pjsip-simple/evsub.c:524
timer.c 87 2 343.934 ../src/pjsip-simple/evsub.c:524
timer.c 27 2 378.627 ../src/pjsip-simple/evsub.c:524
timer.c 2 2 362.433 ../src/pjsip-simple/evsub.c:524
timer.c 83 2 158.243 ../src/pjsip-simple/evsub.c:524
timer.c 42 2 370.628 ../src/pjsip-simple/evsub.c:524
timer.c 48 2 348.935 ../src/pjsip-simple/evsub.c:524
timer.c 84 2 374.130 ../src/pjsip-simple/evsub.c:524
timer.c 58 2 351.630 ../src/pjsip-simple/evsub.c:524
timer.c 38 2 380.135 ../src/pjsip-simple/evsub.c:524
timer.c 26 2 391.961 ../src/pjsip-simple/evsub.c:524
timer.c 45 2 379.634 ../src/pjsip-simple/evsub.c:524
timer.c 55 2 365.627 ../src/pjsip-simple/evsub.c:524
timer.c 79 2 365.129 ../src/pjsip-simple/evsub.c:524
timer.c 47 2 169.745 ../src/pjsip-simple/evsub.c:524
timer.c 65 2 376.626 ../src/pjsip-simple/evsub.c:524
timer.c 11 2 505.205 ../src/pjsip-simple/evsub.c:524
timer.c 77 2 375.629 ../src/pjsip-simple/evsub.c:524
timer.c 35 2 383.129 ../src/pjsip-simple/evsub.c:524
timer.c 1 2 376.131 ../src/pjsip-simple/evsub.c:524
timer.c 37 2 381.130 ../src/pjsip-simple/evsub.c:524
timer.c 54 2 360.628 ../src/pjsip-simple/evsub.c:524
timer.c 40 2 377.129 ../src/pjsip-simple/evsub.c:524
timer.c 46 2 561.104 ../src/pjsip-simple/evsub.c:524
timer.c 18 2 506.957 ../src/pjsip-simple/evsub.c:524
timer.c 22 2 405.544 ../src/pjsip-simple/evsub.c:524
timer.c 34 2 392.524 ../src/pjsip-simple/evsub.c:524
timer.c 70 2 391.465 ../src/pjsip-simple/evsub.c:524
timer.c 80 2 504.920 ../src/pjsip-simple/evsub.c:524
timer.c 51 2 505.455 ../src/pjsip-simple/evsub.c:524
timer.c 62 2 382.129 ../src/pjsip-simple/evsub.c:524
timer.c 72 2 407.355 ../src/pjsip-simple/evsub.c:524
timer.c 75 2 389.356 ../src/pjsip-simple/evsub.c:524
timer.c 9 2 388.963 ../src/pjsip-simple/evsub.c:524
timer.c 29 2 170.240 ../src/pjsip-simple/evsub.c:524
timer.c 12 2 473.356 ../src/pjsip-simple/evsub.c:524
timer.c 71 2 539.931 ../src/pjsip-simple/evsub.c:524
timer.c 5 2 505.721 ../src/pjsip-simple/evsub.c:524
timer.c 21 2 506.611 ../src/pjsip-simple/evsub.c:524
timer.c 68 2 504.058 ../src/pjsip-simple/evsub.c:524
timer.c 8 2 505.999 ../src/pjsip-simple/evsub.c:524
timer.c 69 2 387.630 ../src/pjsip-simple/evsub.c:524
timer.c 67 2 508.207 ../src/pjsip-simple/evsub.c:524
timer.c 4 2 503.815 ../src/pjsip-simple/evsub.c:524
timer.c 3 2 506.251 ../src/pjsip-simple/evsub.c:524
timer.c 49 2 382.628 ../src/pjsip-simple/evsub.c:524
timer.c 33 2 395.960 ../src/pjsip-simple/evsub.c:524
timer.c 36 2 504.649 ../src/pjsip-simple/evsub.c:524
timer.c 31 2 400.526 ../src/pjsip-simple/evsub.c:524
timer.c 52 2 504.335 ../src/pjsip-simple/evsub.c:524
timer.c 56 2 389.533 ../src/pjsip-simple/evsub.c:524
timer.c 6 2 567.253 ../src/pjsip-simple/evsub.c:524
{code}
> Memory Leak since 16.13.0
> -------------------------
>
> Key: ASTERISK-29232
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-29232
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Core/PBX
> Affects Versions: 16.15.0, 16.20.0
> Environment: CentOS 7 x64
> Reporter: Luke Escude
> Assignee: Luke Escude
> Severity: Major
> Labels: fax
> Attachments: Analysis.xlsx, Apex-Analysis.xlsx, cw1-memchart.png, Jan6-1401.csv, nw1-memchart.png, PW3-Memchart.png
>
>
> So we have around 100 instances of Asterisk 16.13.0 that have been running for over 2 months, normal load (small businesses with less than 30 users each), without issue.
> We have another 350 instances of Asterisk 16.15.0 that we've started seeing a very linear increase in memory consumption over time. Specifically, we see higher-load instances (150+ users) last only a few days before hitting our artificial 3GB ceiling and getting restarted by the OOM killer.
> There are very few differences in our implementation of the 16.13 and 16.15 versions. All versions are set up as the following:
> - CentOS 7 64-bit
> - Voicemail over ODBC
> - unixODBC 2.3.1
> - MariaDB Connector (instead of the crappy mysql connector)
> - CDR over MySQL
> - SIP Trunks are registered every 2 minutes, qualified every 15 seconds.
> - User devices register every 10 minutes, qualified every 15 seconds.
> - User devices connect via TCP more often than UDP.
> - I have NO pjsip threadpool configuration options defined. I think the default is 50 threads?
> Here is what I am about to test within the next week:
> 1. unixODBC updated to 2.3.9
> 2. Longer SIP Trunk Registration period - Maybe PJSIP is working too hard?
> 3. Longer qualify timeout - Maybe PJSIP is working too hard?
> One of my first questions: Is it SAFE to compile asterisk with MALLOC_DEBUG and just leave it on permanently? I am scared to enable it, and suddenly have a bunch of users that are experiencing issues because I've enabled something that should only be enabled in Dev.
> Sorry for the length of the post, trying to cover as much ground as possible.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list