[asterisk-bugs] [JIRA] (ASTERISK-24837) chan_sip calls to Asterisk result in file descriptors growing exponentially while channels remain up
Private Name (JIRA)
noreply at issues.asterisk.org
Fri Mar 20 00:43:34 CDT 2015
[ https://issues.asterisk.org/jira/browse/ASTERISK-24837?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=225509#comment-225509 ]
Private Name edited comment on ASTERISK-24837 at 3/20/15 12:42 AM:
-------------------------------------------------------------------
You hit it right on the nail. I just went back and enabled FD leaks debug and connected 200 calls, with audio coming and returining with echo(). core show fd shows106 lines, but
lsof | grep asterisk | grep FIFO | wc -l
86832
note: my extconfig.conf is empty, and there is no dialplan, nada, zero, I reproduce the issue with no dialplan. I am posting here my script, in the hope that somebody with better training can crack this issue. This script will connect any number of calls, defined in variable K, with a delay of one second between calls.
----------------------
#!/bin/bash
random-string()
{
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w ${1:-32} | head -n 1
}
mdir=/var/spool/asterisk/outgoing
delay=1
K=200
y="0"
while [ $y -lt $K ]
do
y=$[$y+1]
#echo -e "$delay seconds $i\n"
#b=$(shuf /usr/src/src.txt | awk 'NR==1 {print}');b=${b:1:10}
b=s
printf -v x "Channel: SIP/%s at demo\n\
CallerID: \"%s\" <%s>\n\
WaitTime: 45\n\
MaxRetries: 0\n\
RetryTime: 10\n\
Application: Echo\n\
Data:\n\
Archive: no\n\n" $i $b $b
f=/tmp/$(random-string 16 ).call
echo -e "$y"
rm $f -f
echo -e "$x" > $f
touch -d "+ $delay seconds" $f
mv $f $mdir
delay=$(( $delay + 1))
done
---------------
on the far end, just play music on hold.
was (Author: falves11):
You hit it right in the nail. I just went back and enable FD leaks debug and connected 200 calls, witg audio coming and returinng with echo. core show fd shows106 lines, but
lsof | grep asterisk | grep FIFO | wc -l
86832
note: my extconfig.conf is empty, and there is no dialplan, nada, zero, I reproduce the issue with no dialplan. I am posting here my script, in the hope that somebody with better training can crack this issue. This script will connect any number of calls, defined in variable K, with a delay of one second between calls.
----------------------
#!/bin/bash
random-string()
{
cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w ${1:-32} | head -n 1
}
mdir=/var/spool/asterisk/outgoing
delay=1
K=200
y="0"
while [ $y -lt $K ]
do
y=$[$y+1]
#echo -e "$delay seconds $i\n"
#b=$(shuf /usr/src/src.txt | awk 'NR==1 {print}');b=${b:1:10}
b=s
printf -v x "Channel: SIP/%s at demo\n\
CallerID: \"%s\" <%s>\n\
WaitTime: 45\n\
MaxRetries: 0\n\
RetryTime: 10\n\
Application: Echo\n\
Data:\n\
Archive: no\n\n" $i $b $b
f=/tmp/$(random-string 16 ).call
echo -e "$y"
rm $f -f
echo -e "$x" > $f
touch -d "+ $delay seconds" $f
mv $f $mdir
delay=$(( $delay + 1))
done
---------------
on the far end, just play music on hold.
> chan_sip calls to Asterisk result in file descriptors growing exponentially while channels remain up
> ----------------------------------------------------------------------------------------------------
>
> Key: ASTERISK-24837
> URL: https://issues.asterisk.org/jira/browse/ASTERISK-24837
> Project: Asterisk
> Issue Type: Bug
> Security Level: None
> Components: Resources/res_rtp_asterisk
> Affects Versions: 13.2.0
> Environment: Linux 64
> Reporter: Private Name
> Assignee: Private Name
> Attachments: core_show_fd_120_channels.txt, sip.conf, trace.txt, trace.txt, trace.txt
>
>
> [Edit by Rusty - Please use the wiki formatting available to make reports easy to read - I'm going to clean this up for now.]
> When I originate several hundred calls using a call file, no dialplan, using app Echo() or app MusicOnHold, the calls connect, but when the other side starts to send media, after some 200+ calls I get several errors:
> {noformat}
> "ERROR[2433] res_rtp_asterisk.c: RTCP SR transmission error to 208.78.162.174:34443, rtcp halted Operation not permitted"
> {noformat}
> and the handle count explodes, measured at the end
> {noformat}
> lsof | grep asterisk | grep FIFO | wc -l
> 1025046
> {noformat}
> yes one million and change. The handle count never decreases as long as the channels are open. There are no active calls, only 500 channels.
> The call files are all identical:
> {noformat}
> Channel: SIP/0000000000 at demo
> CallerID: "0000000000" <>
> WaitTime: 45
> MaxRetries: 0
> RetryTime: 0
> Application: Echo
> Data:
> Archive: no
> {noformat}
> where demo is a simple peer like this
> {noformat}
> [demo]
> host=xxx.xxx.xxx.xx
> type=peer
> insecure=port,invite
> context=reject
> disallow=all
> allow=ulaw
> allow=g729
> session-timers=accept
> port=5060
> faxdetect=no
> transport=udp
> directmedia=yes
> {noformat}
> *Note 1:* the caller ID may vary call by call it makes no difference. The issue here is the very high handle count, which slows down and kills the machine, and the errors which show that many calls do not connect.
> *Note 2:* The calls do no across the internet, they go to a local box.
> I need to send as many calls in real life, with media, so this is a killer for my business model.
> If I set up debug=10 and verbose=20, I get thousands of lines identical like this ones
> {noformat}
> Feb 26 23:47:13] DEBUG[2433]: acl.c:963 ast_find_ourip: Attached to given IP address
> [Feb 26 23:47:13] DEBUG[5763]: res_rtp_asterisk.c:3958 ast_rtcp_read: Got RTCP report of 64 bytes
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:958 ast_find_ourip: Not an IPv4 nor IPv6 address, cannot get port.
> [Feb 26 23:47:13] DEBUG[5763]: netsock2.c:172 ast_sockaddr_split_hostport: Splitting 'dasaro' into...
> [Feb 26 23:47:13] DEBUG[5763]: netsock2.c:226 ast_sockaddr_split_hostport: ...host 'dasaro' and port ''.
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:958 ast_find_ourip: Not an IPv4 nor IPv6 address, cannot get port.
> [Feb 26 23:47:13] DEBUG[5763]: acl.c:963 ast_find_ourip: Attached to given IP address
> {noformat}
> {noformat}
> ulimit -a
> core file size (blocks, -c) unlimited
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 1048576
> max locked memory (kbytes, -l) unlimited
> max memory size (kbytes, -m) unlimited
> open files (-n) 1048576
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 8192
> cpu time (seconds, -t) unlimited
> max user processes (-u) unlimited
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
> {noformat}
> *Note 3:* Please note how the handles start to grow with each additional call
> {noformat}
> Count idle
> 46
> 1 call
> lsof | grep asterisk | grep FIFO | wc -l
> 104
> 2 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 168
> 3 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 240
> 4 calls
> lsof | grep asterisk | grep FIFO | wc -l
> 320
> {noformat}
> As you can see, the handles do not grow linearly, but close to exponentially.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
More information about the asterisk-bugs
mailing list