[asterisk-bugs] [JIRA] (ASTERISK-24837) chan_sip calls to Asterisk result in file descriptors growing exponentially while channels remain up
Corey Farrell (JIRA)
noreply at issues.asterisk.org
Fri Mar 20 00:25:34 CDT 2015
[ https://issues.asterisk.org/jira/browse/ASTERISK-24837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=225508#comment-225508 ]
Corey Farrell commented on ASTERISK-24837:
------------------------------------------
So it looks like {{core show fd}} doesn't match {{lsof|grep asterisk|grep FIFO}}. {{core show fd}} is saying that you have 4 open FD's per active channel (approximately). This would be correct, 2 for the alertpipe, 1 for RTP, 1 for RTCP. So about 800 FD's should be open when you have 200 active channels. The exact number may vary depending on how many channels have been hung up but not yet freed.
So about other parts of the system, can you attach your {{extconfig.conf}}? Can you try testing with valgrind? Be sure Asterisk was compiled with {{DONT_OPTIMIZE}} and without {{MALLOC_DEBUG}} menuselect Compiler Flags. Start Asterisk with {{valgrind --track-fds=yes --show-leak-kinds=all --log-file=valgrind.txt aserisk -c}}. This will run very slow, you should only run a limited number of calls. Assuming you can run it without crashing it should give a complete backtrace for any FD's that were left open. Once complete please attach {{valgrind.txt}} - do not post it as a comment.
>From a previous post:
{quote}
0 channels ----------> 32 handles
10 channels--------> 572
20 channels------->1512
30 channels------->2852
40 channels------->4952
{quote}
This tells we that the problem occurred as a result of 10 calls (there should not be 572 handles). So for this reason I think 10-20 calls for a test should be adequate.
> chan_sip calls to Asterisk result in file descriptors growing exponentially while channels remain up
> ----------------------------------------------------------------------------------------------------
>
> Key: ASTERISK-24837
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-24837
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Resources/res_rtp_asterisk
> Affects Versions: 13.2.0
> Environment: Linux 64
> Reporter: Private Name
> Assignee: Private Name
> Attachments: core_show_fd_120_channels.txt, sip.conf, trace.txt, trace.txt, trace.txt
>
>
> [Edit by Rusty - Please use the wiki formatting available to make reports easy to read - I'm going to clean this up for now.]
> When I originate several hundred calls using a call file, no dialplan, using app Echo() or app MusicOnHold, the calls connect, but when the other side starts to send media, after some 200+ calls I get several errors:
> {noformat}
> "ERROR[2433] res_rtp_asterisk.c: RTCP SR transmission error to 208.78.162.174:34443, rtcp halted Operation not permitted"
> {noformat}
> and the handle count explodes, measured at the end
> {noformat}
> lsof | grep asterisk | grep FIFO | wc -l
> 1025046
> {noformat}
> yes one million and change. The handle count never decreases as long as the channels are open. There are no active calls, only 500 channels.
> The call files are all identical:
> {noformat}
> Channel: SIP/0000000000 at demo
> CallerID: "0000000000" <>
> WaitTime: 45
> MaxRetries: 0
> RetryTime: 0
> Application: Echo
> Data:
> Archive: no
> {noformat}
> where demo is a simple peer like this
> {noformat}
> [demo]
> host=xxx.xxx.xxx.xx
> type=peer
> insecure=port,invite
> context=reject
> disallow=all
> allow=ulaw
> allow=g729
> session-timers=accept
> port=5060
> faxdetect=no
> transport=udp
> directmedia=yes
> {noformat}
> *Note 1:* the caller ID may vary call by call it makes no difference. The issue here is the very high handle count, which slows down and kills the machine, and the errors which show that many calls do not connect.
> *Note 2:* The calls do no across the internet, they go to a local box.
> I need to send as many calls in real life, with media, so this is a killer for my business model.
> If I set up debug=10 and verbose=20, I get thousands of lines identical like this ones
> {noformat}
> Feb 26 23:47:13] DEBUG[2433]: acl.c:963 ast_find_ourip: Attached to given IP address
> [Feb 26 23:47:13] DEBUG[5763]: res_rtp_asterisk.c:3958 ast_rtcp_read: Got RTCP report of 64 bytes
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:958 ast_find_ourip: Not an IPv4 nor IPv6 address, cannot get port.
> [Feb 26 23:47:13] DEBUG[5763]: netsock2.c:172 ast_sockaddr_split_hostport: Splitting 'dasaro' into...
> [Feb 26 23:47:13] DEBUG[5763]: netsock2.c:226 ast_sockaddr_split_hostport: ...host 'dasaro' and port ''.
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:958 ast_find_ourip: Not an IPv4 nor IPv6 address, cannot get port.
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:963 ast_find_ourip: Attached to given IP address
> {noformat}
> {noformat}
> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 1048576
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1048576
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {noformat}
> *Note 3:* Please note how the handles start to grow with each additional call
> {noformat}
> Count idle
> 46
> 1 call
> lsof | grep asterisk | grep FIFO | wc -l
> 104
> 2 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 168
> 3 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 240
> 4 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 320
> {noformat}
> As you can see, the handles do not grow linearly, but close to exponentially.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list