<p>Jenkins2 <strong>merged</strong> this change.</p><p><a href="https://gerrit.asterisk.org/8689">View Change</a></p><div style="white-space:pre-wrap">Approvals:
Corey Farrell: Looks good to me, but someone else must approve
Benjamin Keith Ford: Looks good to me, but someone else must approve
Joshua Colp: Looks good to me, approved
Jenkins2: Approved for Submit
</div><pre style="font-family: monospace,monospace; white-space: pre-wrap;">sipp, test_suite_utils: Default media port to an unused port<br><br>SIPp correctly chooses an available port for audio, but unfortunately it then<br>attempts to bind to the audio port + n for things like rtcp and video without<br>first checking if those other ports are unused:<br><br>https://github.com/SIPp/sipp/issues/276<br><br>This patch makes it so all ports needed by the scenario are available. It does<br>this by retrieving and checking for an unused port plus 'n' ports from the OS.<br>If the ports are available then the primary port is passed to the scenario using<br>the '-mp' option.<br><br>Change-Id: I3da461123afc30e1f5ca12e65d289eaa42d6de00<br>---<br>A lib/python/asterisk/self_test/test_utils_socket.py<br>M lib/python/asterisk/sipp.py<br>A lib/python/asterisk/utils_socket.py<br>3 files changed, 502 insertions(+), 0 deletions(-)<br><br></pre><pre style="font-family: monospace,monospace; white-space: pre-wrap;">diff --git a/lib/python/asterisk/self_test/test_utils_socket.py b/lib/python/asterisk/self_test/test_utils_socket.py<br>new file mode 100755<br>index 0000000..3a0c33d<br>--- /dev/null<br>+++ b/lib/python/asterisk/self_test/test_utils_socket.py<br>@@ -0,0 +1,175 @@<br>+#!/usr/bin/env python<br>+"""Utility module for socket handling<br>+<br>+This module provides tests for the utils_socket module.<br>+<br>+Copyright (C) 2018, Digium, Inc.<br>+Kevin Harwell <kharwell@digium.com><br>+<br>+This program is free software, distributed under the terms of<br>+the GNU General Public License Version 2.<br>+"""<br>+<br>+import logging<br>+import sys<br>+import unittest<br>+<br>+from socket import SOCK_STREAM, SOCK_DGRAM, AF_INET, AF_INET6<br>+<br>+sys.path.append('lib/python') # noqa<br>+from asterisk.utils_socket import Ports, PortError, get_available_port, MIN_PORT<br>+<br>+<br>+LOGGER = logging.getLogger(__name__)<br>+<br>+<br>+class PortTests(unittest.TestCase):<br>+ """Unit tests for port availability and reservations."""<br>+<br>+ def setUp(self):<br>+ self.ports = Ports()<br>+<br>+ def _on_family(self, host, socktype, cb):<br>+ host = host or ''<br>+<br>+ if '.' in host:<br>+ cb(host, socktype, AF_INET)<br>+ elif ':' in host:<br>+ cb(host, socktype, AF_INET6)<br>+ else:<br>+ for family in [AF_INET, AF_INET6]:<br>+ cb(host, socktype, family)<br>+<br>+ def _on_socktype(self, host, cb):<br>+ for socktype in [SOCK_STREAM, SOCK_DGRAM]:<br>+ self._on_family(host, socktype, cb)<br>+<br>+ def _on_host(self, cb):<br>+ self._on_socktype('', cb)<br>+<br>+ for host in ['', '0.0.0.0', '127.0.0.1', '::', '::1']:<br>+ self._on_socktype(host, cb)<br>+<br>+ def test_001_reserve(self):<br>+ """Test reserving ports for different families and types"""<br>+<br>+ p = [51234, 51235]<br>+<br>+ def reserve(h, s, f):<br>+ self.ports.reserve(p, s, f)<br>+ self._on_socktype(None, reserve)<br>+<br>+ def test(h, s, f):<br>+ self.assertEqual(self.ports.reserved_ports[s][f], p)<br>+ self._on_socktype(None, test)<br>+<br>+ def test_002_is_reserved(self):<br>+ """Test if port is and is not reserved for families and types"""<br>+<br>+ p = 51234<br>+<br>+ def reserve(h, s, f):<br>+ self.ports.reserve(p, s, f)<br>+ self._on_socktype(None, reserve)<br>+<br>+ self.assertTrue(self.ports.is_reserved(MIN_PORT - 1))<br>+<br>+ def test(h, s, f):<br>+ self.assertTrue(self.ports.is_reserved(p, s, f))<br>+ self._on_socktype(None, test)<br>+<br>+ def test2(h, s, f):<br>+ self.assertFalse(self.ports.is_reserved(71234, s, f))<br>+ self._on_socktype(None, test2)<br>+<br>+ def test_003_get_avail(self):<br>+ """Test retrieving a ports by families and types"""<br>+<br>+ p = [0, 0]<br>+<br>+ def test(h, s, f):<br>+ self.assertNotEqual(self.ports.get_avail(h, p, s, f), p)<br>+ self._on_host(test)<br>+<br>+ p = [51234, 51235]<br>+<br>+ def test2(h, s, f):<br>+ self.assertEqual(self.ports.get_avail(h, p, s, f), p)<br>+ self._on_host(test2)<br>+<br>+ def reserve(h, s, f):<br>+ self.ports.reserve(p, s, f)<br>+ self._on_socktype(None, reserve)<br>+<br>+ def test3(h, s, f):<br>+ self.assertRaises(PortError, self.ports.get_avail, h, p, s, f)<br>+ self._on_host(test3)<br>+<br>+ def test_004_get_range(self):<br>+ """Test retrieving a range of ports by families and types"""<br>+ p = 0<br>+<br>+ def test(h, s, f):<br>+ self.assertEqual(len(self.ports.get_range(h, p, s, f, 3)), 3)<br>+ self._on_host(test)<br>+<br>+ p = 50000<br>+<br>+ def test2(h, s, f):<br>+ self.assertEqual(self.ports.get_range(h, p, s, f, 3),<br>+ [50000, 50001, 50002])<br>+ self._on_host(test2)<br>+<br>+ def test3(h, s, f):<br>+ self.assertEqual(self.ports.get_range(h, p, s, f, -3),<br>+ [50000, 49999, 49998])<br>+ self._on_host(test3)<br>+<br>+ def test_005_get_range_and_reserve(self):<br>+ """Test retrieving and reserving a range of ports by<br>+ families and types"""<br>+ p = 50000<br>+<br>+ def get_range(h, s, f):<br>+ self.ports.get_range_and_reserve(h, p, s, f, 3)<br>+ self._on_socktype(None, get_range)<br>+<br>+ def test(h, s, f):<br>+ self.assertTrue(self.ports.is_reserved(50000, s, f))<br>+ self._on_host(test)<br>+<br>+ def test2(h, s, f):<br>+ self.assertTrue(self.ports.is_reserved(50001, s, f))<br>+ self._on_host(test2)<br>+<br>+ def test3(h, s, f):<br>+ self.assertTrue(self.ports.is_reserved(50002, s, f))<br>+ self._on_host(test3)<br>+<br>+ def test_006_get_available_port(self):<br>+ """Test retrieving an available port"""<br>+ p = 50000<br>+<br>+ def test(h, s, f):<br>+ self.assertEqual(get_available_port(h, s, f, 0, p), p)<br>+ # Limit to single host since using global object<br>+ self._on_socktype(None, test)<br>+<br>+ p = 50001<br>+<br>+ self.assertEqual(get_available_port(<br>+ config={'-i': '0.0.0.0'}, num=2, port=p), p)<br>+ self.assertEqual(get_available_port(<br>+ config={'-i': '0.0.0.0', '-t': ''}, num=2, port=p), p)<br>+ self.assertEqual(get_available_port(<br>+ config={'-i': '[::]'}, num=2, port=p), p)<br>+ self.assertEqual(get_available_port(<br>+ config={'-i': '[::]', '-t': ''}, num=2, port=p), p)<br>+<br>+<br>+if __name__ == "__main__":<br>+ """Run the unit tests"""<br>+<br>+ logging.basicConfig(stream=sys.stdout, level=logging.DEBUG,<br>+ format="%(module)s:%(lineno)d - %(message)s")<br>+ unittest.main()<br>diff --git a/lib/python/asterisk/sipp.py b/lib/python/asterisk/sipp.py<br>index 6da3108..b84965e 100644<br>--- a/lib/python/asterisk/sipp.py<br>+++ b/lib/python/asterisk/sipp.py<br>@@ -15,6 +15,7 @@<br> from abc import ABCMeta, abstractmethod<br> from twisted.internet import reactor, defer, protocol, error<br> from test_case import TestCase<br>+from utils_socket import get_available_port<br> <br> LOGGER = logging.getLogger(__name__)<br> <br>@@ -671,6 +672,19 @@<br> default_args['-oocsf'] = ('%s/sipp/%s' % (<br> self.test_dir, default_args['-oocsf']))<br> <br>+ if '-mp' not in default_args:<br>+ # Current SIPp correctly chooses an available port for audio, but<br>+ # unfortunately it then attempts to bind to the audio port + n for<br>+ # things like rtcp and video without first checking if those other<br>+ # ports are unused (https://github.com/SIPp/sipp/issues/276).<br>+ #<br>+ # So as a work around, if not given, we'll specify the media port<br>+ # ourselves, and make sure all associated ports are available.<br>+ #<br>+ # num = 4 = ports for audio rtp/rtcp and video rtp/rtcp<br>+ default_args['-mp'] = str(get_available_port(<br>+ config=default_args, num=4))<br>+<br> for (key, val) in default_args.items():<br> sipp_args.extend([key, val])<br> <br>diff --git a/lib/python/asterisk/utils_socket.py b/lib/python/asterisk/utils_socket.py<br>new file mode 100644<br>index 0000000..2a11c97<br>--- /dev/null<br>+++ b/lib/python/asterisk/utils_socket.py<br>@@ -0,0 +1,313 @@<br>+"""Utility module for socket handling<br>+<br>+This module provides classes and wrappers around the socket library.<br>+<br>+Copyright (C) 2018, Digium, Inc.<br>+Kevin Harwell <kharwell@digium.com><br>+<br>+This program is free software, distributed under the terms of<br>+the GNU General Public License Version 2.<br>+"""<br>+<br>+import logging<br>+<br>+from socket import *<br>+<br>+<br>+LOGGER = logging.getLogger(__name__)<br>+<br>+# Number of times to try and find an available (non-reserved and unused) port<br>+PORT_ATTEMPTS = 100<br>+<br>+# Don't allow any port to be retrieved and reserved below this value<br>+MIN_PORT = 10000<br>+<br>+<br>+def socket_type(socktype):<br>+ """Retrieve a string representation of the socket type."""<br>+ return {<br>+ SOCK_STREAM: 'TCP',<br>+ SOCK_DGRAM: 'UDP',<br>+ }.get(socktype, 'Unknown Type')<br>+<br>+<br>+def socket_family(family):<br>+ """Retrieve a string representation of the socket family."""<br>+ return {<br>+ AF_INET: 'IPv4',<br>+ AF_INET6: 'IPv6',<br>+ }.get(family, 'Unknown Family')<br>+<br>+<br>+def get_unused_os_port(host='', port=0, socktype=SOCK_STREAM, family=AF_INET):<br>+ """Retrieve an unused port from the OS.<br>+<br>+ Host can be an empty string, or a specific address. However, if an address<br>+ is specified the family must coincide with the address or an exception is<br>+ thrown. For example:<br>+<br>+ host='', family=<either AF_INET or AF_INET6><br>+ host='0.0.0.0', family=AF_INET<br>+ host='127.0.0.1', family=AF_INET<br>+ host='::', family=AF_INET6<br>+ host='::1', family=AF_INET6<br>+<br>+ If the given port is equal to zero then an available port is chosen by the<br>+ operating system and returned. If a number greater than zero is given for<br>+ the port, then this checks if that port is currently not in use (from the<br>+ operating system's perspective). If unused then it's returned. Zero is<br>+ returned if a port is unavailable.<br>+<br>+ Keyword Arguments:<br>+ host - The host address<br>+ port - Port number to check availability<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+<br>+ Return:<br>+ A usable port, or zero if a port is unavailable.<br>+ """<br>+<br>+ host = host.lstrip('[').rstrip(']')<br>+<br>+ res = 0<br>+ s = socket(family, socktype)<br>+ try:<br>+ s.bind((host, port))<br>+ res = s.getsockname()[1]<br>+ except error as e:<br>+ # errno = 98 is 'Port already in use'. However, if any error occurs<br>+ # just fail since we probably don't want to bind to it anyway.<br>+ LOGGER.debug("{0}/{1} port '{2}' is in use".format(<br>+ socket_type(socktype), socket_family(family), port))<br>+<br>+ s.close()<br>+ return res<br>+<br>+<br>+class PortError(ValueError):<br>+ """Error raised when the number of attempts to find an available<br>+ port is exceeded"""<br>+<br>+ def __init__(self, socktype, family, port=0, attempts=PORT_ATTEMPTS):<br>+ """Create a port error"""<br>+ if port:<br>+ msg = "{0}/{1} port '{2}' not available ".format(<br>+ socket_type(socktype), socket_family(family), port)<br>+ else:<br>+ msg = "Unable to get usable {0}/{1} port after {2} attempts".format(<br>+ socket_type(socktype), socket_family(family), attempts)<br>+ super(ValueError, self).__init__(msg)<br>+<br>+<br>+class Ports(object):<br>+ """An interface for port availability. Handles retrieving vacant ports,<br>+ and also guarantees returned ports are not used by others by adding it to,<br>+ and maintaining a container of reserved ports. The internal container is<br>+ a list of ports stored by socket types and family:<br>+<br>+ ports[socktype][family] = [<ports>]<br>+<br>+ The ports stored here represent ports that are considered unavailable and<br>+ should not be bound to.<br>+ """<br>+<br>+ def __init__(self):<br>+ """Create a reserved ports container."""<br>+ self.reserved_ports = {}<br>+<br>+ def reserve(self, ports, socktype=SOCK_STREAM, family=AF_INET):<br>+ """Mark the given ports as reserved. Meaning that even if the operating<br>+ system has these ports as free they'll be considered unusable if set as<br>+ reserved.<br>+<br>+ Keyword Arguments:<br>+ ports - A list of ports to reserve<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+ """<br>+<br>+ if isinstance(ports, int):<br>+ ports = [ports]<br>+ if not isinstance(ports, list):<br>+ ports = list(ports)<br>+<br>+ if socktype not in self.reserved_ports:<br>+ self.reserved_ports[socktype] = {}<br>+<br>+ if family not in self.reserved_ports[socktype]:<br>+ self.reserved_ports[socktype][family] = ports<br>+ else:<br>+ self.reserved_ports[socktype][family].extend(ports)<br>+<br>+ def is_reserved(self, port, socktype=SOCK_STREAM, family=AF_INET):<br>+ """Check to see if a given port is reserved.<br>+<br>+ Keyword Arguments:<br>+ port - The port to check<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+<br>+ Return:<br>+ True if the port is reserved, False otherwise<br>+ """<br>+<br>+ try:<br>+ if (port < MIN_PORT or<br>+ port in self.reserved_ports[socktype][family]):<br>+<br>+ LOGGER.debug("{0}/{1} port '{2}' is reserved".format(<br>+ socket_type(socktype), socket_family(family), port))<br>+ return True<br>+ except:<br>+ pass<br>+<br>+ return False<br>+<br>+ def get_avail(self, host='', ports=0, socktype=SOCK_STREAM, family=AF_INET,<br>+ attempts=PORT_ATTEMPTS):<br>+ """Retrieve available ports. Ports are considered available if they<br>+ have not been reserved and are currently not in use by something else.<br>+<br>+ If a given port is zero this function retrieves an available port from<br>+ the operating system. If non-zero it checks to make sure the port is<br>+ currently unused by the operating system.<br>+<br>+ Once retrieved or validated it makes sure the port has not already been<br>+ reserved. If it is reserved and port was initially zero it tries again<br>+ up to 'attempts' tries. Otherwise if it was non-zero an exception is<br>+ thrown.<br>+<br>+ Keyword Arguments:<br>+ host - host address<br>+ port - port number(s) to check availability<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+ attempts - The number of attempts to try and find a free port<br>+<br>+ Return:<br>+ Available ports as a list, otherwise a PortError exception is thrown<br>+ if none are available.<br>+ """<br>+<br>+ if isinstance(ports, int):<br>+ ports = [ports]<br>+ if not isinstance(ports, list):<br>+ ports = list(ports)<br>+<br>+ res = []<br>+ for port in ports:<br>+ for attempt in range(attempts):<br>+ p = get_unused_os_port(host, port, socktype, family)<br>+ if p != 0 and not self.is_reserved(p, socktype, family):<br>+ res.append(p)<br>+ break<br>+<br>+ if port:<br>+ raise PortError(socktype, family, port)<br>+ else:<br>+ raise PortError(socktype, family, attempts=attempts)<br>+<br>+ return res<br>+<br>+ def get_range(self, host='', port=0, socktype=SOCK_STREAM,<br>+ family=AF_INET, num=1, attempts=PORT_ATTEMPTS):<br>+ """Retrieve a range of ports starting at port.<br>+<br>+ Keyword Arguments:<br>+ host - host address<br>+ port - port number to get and/or check availability<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+ num - the number of ports to return including a given port<br>+ attempts - The number of attempts to try and find a free port<br>+<br>+ Return:<br>+ A list of available port(s). If none are available then a PortError<br>+ exception is thrown.<br>+ """<br>+<br>+ num = num or 1<br>+<br>+ for attempt in range(attempts):<br>+ step = 1 if num > 0 else -1<br>+ if port != 0:<br>+ return self.get_avail(<br>+ host, range(port, port + num, step),<br>+ socktype, family, attempts)<br>+<br>+ # Need a random port first<br>+ port = self.get_avail(host, 0, socktype, family)<br>+ ports = self.get_avail(<br>+ host, range(port[0] + step, port[0] + num, step),<br>+ socktype, family, attempts)<br>+ if ports:<br>+ return port + ports<br>+<br>+ raise PortError(socktype, family, attempts=attempts)<br>+<br>+ def get_range_and_reserve(self, host='', port=0, socktype=SOCK_STREAM,<br>+ family=AF_INET, num=0, attempts=PORT_ATTEMPTS):<br>+ """Retrieve available port and port +/- num and reserve them.<br>+<br>+ If any ports are returned by calling 'get_range' those ports are then<br>+ stored as 'reserved'.<br>+<br>+ Keyword Arguments:<br>+ host - host address<br>+ port - port number to check availability<br>+ socktype - socket types (SOCK_STREAM, SOCK_DGRAM, etc.)<br>+ family - protocol family (AF_INET, AF_INET6, etc.)<br>+ num - the number of ports to num +/- port<br>+ attempts - The number of attempts to try and find a free port<br>+<br>+ Return:<br>+ If available, return the port +/- num ports, otherwise return an<br>+ empty list.<br>+ """<br>+<br>+ res = self.get_range(host, port, socktype, family, num, attempts)<br>+ self.reserve(res, socktype, family)<br>+ return res<br>+<br>+<br>+PORTS = Ports()<br>+<br>+<br>+def get_available_port(host='', socktype=0, family=0, num=0,<br>+ port=0, config=None):<br>+ """Retrieve the primary available port, and reserve it and its offsets.<br>+<br>+ The majority of use cases probably involve the need to reserve multiple<br>+ ports (primary plus some offsets), but only retrieve the primary port.<br>+ This function does that.<br>+<br>+ This function purposely throws an exception if it cannot locate a<br>+ singular port.<br>+<br>+ Note:<br>+ This is a wrapper/convenience function. See the<br>+ Ports.get_range_and_reserve method for more functional details.<br>+<br>+ Result:<br>+ The singular primary available port.<br>+ """<br>+<br>+ if config:<br>+ host = host or config.get('-i')<br>+ socktype = SOCK_STREAM if '-t' in config else SOCK_DGRAM<br>+<br>+ # The family (without a host) and socktype should really always be<br>+ # specified when calling this function, but just in case use some defaults<br>+ if family < 1:<br>+ if host:<br>+ family = AF_INET6 if ':' in host else AF_INET<br>+ else:<br>+ family = AF_INET<br>+<br>+ if socktype < 1:<br>+ socktype = SOCK_DGRAM # Most tests are UDP based<br>+ available = PORTS.get_range_and_reserve(<br>+ host, port, socktype, family, num)<br>+ # If we don't have a valid socktype and family badness happens<br>+ return available[0]<br></pre><p>To view, visit <a href="https://gerrit.asterisk.org/8689">change 8689</a>. To unsubscribe, visit <a href="https://gerrit.asterisk.org/settings">settings</a>.</p><div itemscope itemtype="http://schema.org/EmailMessage"><div itemscope itemprop="action" itemtype="http://schema.org/ViewAction"><link itemprop="url" href="https://gerrit.asterisk.org/8689"/><meta itemprop="name" content="View Change"/></div></div>
<div style="display:none"> Gerrit-Project: testsuite </div>
<div style="display:none"> Gerrit-Branch: 13 </div>
<div style="display:none"> Gerrit-MessageType: merged </div>
<div style="display:none"> Gerrit-Change-Id: I3da461123afc30e1f5ca12e65d289eaa42d6de00 </div>
<div style="display:none"> Gerrit-Change-Number: 8689 </div>
<div style="display:none"> Gerrit-PatchSet: 7 </div>
<div style="display:none"> Gerrit-Owner: Kevin Harwell <kharwell@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Benjamin Keith Ford <bford@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Corey Farrell <git@cfware.com> </div>
<div style="display:none"> Gerrit-Reviewer: Jenkins2 </div>
<div style="display:none"> Gerrit-Reviewer: Joshua Colp <jcolp@digium.com> </div>
<div style="display:none"> Gerrit-Reviewer: Kevin Harwell <kharwell@digium.com> </div>