[Asterisk-code-review] testsuite: Allow copying of arbitrary files to asterisk instances (testsuite[19])
George Joseph
asteriskteam at digium.com
Wed Oct 6 08:56:29 CDT 2021
George Joseph has uploaded this change for review. ( https://gerrit.asterisk.org/c/testsuite/+/16587 )
Change subject: testsuite: Allow copying of arbitrary files to asterisk instances
......................................................................
testsuite: Allow copying of arbitrary files to asterisk instances
If you have a key or certificate file that has to go in
/var/lib/asterisk/keys, there was no way for a test to get
that file into that directory. Various work-arounds ensued.
* Added a replaceable parameter "<<instanceid>>" to config
file processing which would be replaced with the Asterisk
instance's id. I.E. 'ast1' would be '1'.
* Added automatic installation of files in a test's "files"
directory to its Asterisk instance's well-known directories.
To use, create directories named after entries in the
asterisk.conf "directories" category and place your files there.
It works much like the test's "configs" directory.
For example:
mytest/
configs/
ast1/
someconfig.conf
ast2/
someconfig.conf
files/
common/
astvarlibdir/
keys/
ca.crt
ast1/
astvarlibdir/
keys/
mykey1.pem
ast2/
astvarlibdir/
keys/
mykey2.pem
test-config.yaml
Since 'astvarlibdir' is usually defined in asterisk.conf as
'/var/lib/asterisk', this would copy 'ca.crt' to both Asterisk
instance's '/var/lib/asterisk/keys' directory, 'mykey1.pem' to
the first Asterisk instance's '/var/lib/asterisk/keys' directory
and 'mykey2.pem' to the second instance's '/var/lib/asterisk/keys'
directory.
Each of the instance's "someconfig.conf" could reference those
files like so:
cacert = <<astvarlibdir>>/keys/ca.crt.pem
keyfile = <<astvarlibdir>>/keys/mykey<<instanceid>>.pem
If you have files that can be shared among tests, you can define
'base-files-path' in each test's 'test-object-config' section which
would point to a directory structured like the 'files' directory
above.
For example
mytests/
myfiles/
common/
astvarlibdir/
keys/
ca.crt
ast1/
astvarlibdir/
keys/
mykey1.pem
ast2/
astvarlibdir/
keys/
mykey2.pem
mytest1/
...
mytest2/
...
Then in both test's test-config.yaml...
test-object-config:
base-files-path: 'tests/mytests/myfiles'
base-files-path is processed before the automatic files
processing of each test so individual tests can add its own files
or even override a common file if it chooses.
ASTERISK-29675
Change-Id: I69bb54fea459e83f65a474d3b74c40b28fe01b4c
---
M README.txt
M lib/python/asterisk/asterisk.py
M lib/python/asterisk/test_case.py
A lib/python/polyfill.py
M sample-yaml/test-config.yaml.sample
5 files changed, 221 insertions(+), 2 deletions(-)
git pull ssh://gerrit.asterisk.org:29418/testsuite refs/changes/87/16587/1
diff --git a/README.txt b/README.txt
index 459fbed..93888be 100644
--- a/README.txt
+++ b/README.txt
@@ -335,6 +335,41 @@
Since each Asterisk instance required difference SIP settings, each 'ast%d'
folder will have a different sip.conf file.
+You can also copy arbitrary files like sound, key and certificate files into
+any of the entries in the asterisk.conf "directories" category using its
+entry name name. All intermediate directories will be created if they
+don't already exist.
+
+For example, to have a certificate and keys copied to an Asterisk instance's
+'/var/lib/asterisk/keys' directory, you'd place it in...
+
+ basic-call/
+ files/
+ common/
+ astvarlibdir/
+ keys/
+ ca.crt
+ ast1/
+ astvarlibdir/
+ keys/
+ instance1-key.pem
+ ...
+ ast2/
+ astvarlibdir/
+ keys/
+ instance2-key.pem
+
+Since 'astvarlibdir' is defined in asterisk.conf as '/var/lib/asterisk',
+this would copy 'ca.crt' to both instance's '/var/lib/asterisk/keys/' directory,
+'instance1-key.pem' to instance 1's '/var/lib/asterisk/keys/' directory and
+'instance2-key.pem' to instance 2's '/var/lib/asterisk/keys/' directory.
+
+If you have files that can be shared among multiple tests, you can create a
+directory following the same structure as above in some parent directory and
+direct each test to include it with the 'base-files-path' parameter in
+its test-config.yaml file. See sample-yaml/test-config.yaml.sample for
+more info.
+
d) Test Execution
The "run-test" executable will be run by a top level application in the
diff --git a/lib/python/asterisk/asterisk.py b/lib/python/asterisk/asterisk.py
index 977dc6e..f1371db 100644
--- a/lib/python/asterisk/asterisk.py
+++ b/lib/python/asterisk/asterisk.py
@@ -16,6 +16,7 @@
import shutil
import logging
import fileinput
+import polyfill
from . import test_suite_utils
@@ -43,7 +44,6 @@
LOGGER = logging.getLogger(__name__)
-
class AsteriskRemoteProtocol(protocol.Protocol):
"""Class that acts as a remote protocol to Asterisk"""
@@ -367,6 +367,7 @@
self.original_astmoddir = ""
self.remote_config = remote_config
self.memcheck_delay_stop = 0
+ self.instance_id = 0
if test_config is not None and 'memcheck-delay-stop' in test_config:
self.memcheck_delay_stop = test_config['memcheck-delay-stop'] or 0
@@ -419,6 +420,7 @@
while True:
if not os.path.isdir("%s/ast%d" % (self.base, i)):
self.base = "%s/ast%d" % (self.base, i)
+ self.instance_id = i;
break
i += 1
@@ -673,6 +675,7 @@
for key in self.directories.keys():
value = value.replace("<<%s>>" % key,
"%s%s" % (self.base, self.directories[key]))
+ value = value.replace("<<instanceid>>", self.instance_id)
return value
# Quick little function for doing search and replace in a file used below.
@@ -787,6 +790,61 @@
except IOError:
LOGGER.warn("The destination is not writable '%s'" % target_path)
+ def install_files(self, source_path):
+ """Installs all files located in a directory to this
+ instance of Asterisk.
+
+ Keyword Arguments:
+ source_path This argument must be the path to the directory
+ containing the files to be installed into standard
+ locations.
+
+ Example: tests/my-cool-test/files/ast1
+ This directory must contain at least one sub-directory
+ matching the name of an entry from the asterisk.conf
+ "directories" category. For example "astvarlibdir".
+ Each entry in those directories will be copied to the
+ runtime equivalent for the test.
+
+ Example:
+ Given the following filesystem layout:
+ tests/my-cool-test/files/ast1
+ - astvarlibdir/
+ - sounds/
+ - recordings/
+ - myrecording.ulaw
+ - astkeydir/keys/
+ - mykey.pem
+
+ ...and given the following entries in asterisk.conf:
+ astvarlibdir => /var/lib/asterisk
+ astkeydir => /var/lib/asterisk
+
+ The myrecording.ulaw and mykey.pem files would be copied to
+ <testroot>/var/lib/asterisk/sounds/recordings/ and
+ <testroot>/var/lib/asterisk/keys/ respectively.
+ """
+
+ self._make_directory_structure()
+ if not os.path.exists(source_path):
+ return
+
+ for fname in os.listdir(source_path):
+ source = "%s/%s" % (source_path, fname)
+
+ if os.path.isdir(source):
+ # If the directory name isn't one of the
+ # well known directories, just skip it.
+ if not self.directories[fname]:
+ continue
+ # join() doesn't work if one of the paths is absolute
+ # so we have to skip the leading '/' in the directory
+ # entry. I.E /var/lib/asterisk becomes var/lib/asterisk
+ direntry = self.directories[fname]
+ dirpath = direntry[1:] if direntry[0] == '/' else direntry
+ dest = os.path.join(self.base, dirpath)
+ polyfill.copytree(source, dest, dirs_exist_ok=True)
+
def _overwrite_file(self, filename, values):
"""Overwrite a particular config file
diff --git a/lib/python/asterisk/test_case.py b/lib/python/asterisk/test_case.py
index 1c7edfa..c4cde27 100644
--- a/lib/python/asterisk/test_case.py
+++ b/lib/python/asterisk/test_case.py
@@ -122,6 +122,7 @@
self.ami = []
self.fastagi = []
self.base_config_path = None
+ self.base_files_path = None
self.reactor_timeout = 30
self.passed = None
self.fail_tokens = []
@@ -149,6 +150,8 @@
if test_config:
if 'config-path' in test_config:
self.base_config_path = test_config['config-path']
+ if 'base-files-path' in test_config:
+ self.base_files_path = test_config['base-files-path']
if 'reactor-timeout' in test_config:
self.reactor_timeout = test_config['reactor-timeout']
if 'memcheck-delay-stop' in test_config:
@@ -248,7 +251,8 @@
for i in range(count)]
return asterisks
- def create_asterisk(self, count=1, base_configs_path=None, test_config=None):
+ def create_asterisk(self, count=1, base_configs_path=None, test_config=None,
+ base_files_path=None):
"""Create n instances of Asterisk
Note: if the instances of Asterisk being created are remote, the
@@ -264,6 +268,11 @@
configuration can be overwritten by individual tests,
however.
test_config Test Configuration
+ base_files_path Provides common files for Asterisk instances
+ to use. This is useful for certain test types that use
+ the same files, like keys, all the time. This
+ configuration can be overwritten by individual tests,
+ however.
"""
for i, ast_config in enumerate(self.get_asterisk_hosts(count)):
local_num = ast_config.get('num')
@@ -300,6 +309,19 @@
(self.test_name, local_num),
self.test_config.get_deps())
+ # If a base files directory exists for this Asterisk instance
+ # has been provided, install it first
+ if base_files_path is None:
+ base_files_path = self.base_files_path
+ if base_files_path:
+ self.ast[i].install_files("%s/common" % (base_files_path))
+ self.ast[i].install_files("%s/ast%d" %
+ (base_files_path, local_num))
+ # Copy test specific config files
+ self.ast[i].install_files("%s/files/common" % (self.test_name))
+ self.ast[i].install_files("%s/files/ast%d" %
+ (self.test_name, local_num))
+
def create_ami_factory(self, count=1, username="user", secret="mysecret",
port=5038):
"""Create n instances of AMI. Each AMI instance will attempt to connect
diff --git a/lib/python/polyfill.py b/lib/python/polyfill.py
new file mode 100644
index 0000000..bd95d5f
--- /dev/null
+++ b/lib/python/polyfill.py
@@ -0,0 +1,85 @@
+import sys
+import os
+import shutil
+
+"""
+ These are polyfills for a few file/directory functions
+ that don't show up until Python 3.9.
+"""
+
+def makedirs(name, mode=0o777, exist_ok=False):
+ head, tail = os.path.split(name)
+ if not tail:
+ head, tail = os.path.split(head)
+ if head and tail and not os.path.exists(head):
+ try:
+ makedirs(head, exist_ok=exist_ok)
+ except FileExistsError:
+ # Defeats race condition when another thread created the path
+ pass
+ cdir = os.curdir
+ if isinstance(tail, bytes):
+ cdir = bytes(os.curdir, 'ASCII')
+ if tail == cdir: # xxx/newdir/. exists if xxx/newdir exists
+ return
+ try:
+ os.mkdir(name, mode)
+ except OSError:
+ if not exist_ok or not os.path.isdir(name):
+ raise
+
+def copytree(src, dst, symlinks=False, ignore=None, copy_function=shutil.copy2,
+ ignore_dangling_symlinks=False, dirs_exist_ok=False):
+ names = os.listdir(src)
+ if ignore is not None:
+ ignored_names = ignore(src, names)
+ else:
+ ignored_names = set()
+
+ makedirs(dst, exist_ok=dirs_exist_ok)
+ errors = []
+ for name in names:
+ if name in ignored_names:
+ continue
+ srcname = os.path.join(src, name)
+ dstname = os.path.join(dst, name)
+ try:
+ if os.path.islink(srcname):
+ linkto = os.readlink(srcname)
+ if symlinks:
+ # We can't just leave it to `copy_function` because legacy
+ # code with a custom `copy_function` may rely on copytree
+ # doing the right thing.
+ os.symlink(linkto, dstname)
+ shutil.copystat(srcname, dstname, follow_symlinks=not symlinks)
+ else:
+ # ignore dangling symlink if the flag is on
+ if not os.path.exists(linkto) and ignore_dangling_symlinks:
+ continue
+ # otherwise let the copy occurs. copy2 will raise an error
+ if os.path.isdir(srcname):
+ copytree(srcname, dstname, symlinks, ignore,
+ copy_function, dirs_exist_ok=dirs_exist_ok)
+ else:
+ copy_function(srcname, dstname)
+ elif os.path.isdir(srcname):
+ copytree(srcname, dstname, symlinks, ignore, copy_function, dirs_exist_ok=dirs_exist_ok)
+ else:
+ # Will raise a SpecialFileError for unsupported file types
+ copy_function(srcname, dstname)
+ # catch the Error from the recursive copytree so that we can
+ # continue with other files
+ except shutil.Error as err:
+ errors.extend(err.args[0])
+ except OSError as why:
+ errors.append((srcname, dstname, str(why)))
+ try:
+ shutil.copystat(src, dst)
+ except OSError as why:
+ # Copying file access times may fail on Windows
+ if getattr(why, 'winerror', None) is None:
+ errors.append((src, dst, str(why)))
+ if errors:
+ raise shutil.Error(errors)
+ return dst
+
diff --git a/sample-yaml/test-config.yaml.sample b/sample-yaml/test-config.yaml.sample
index 773b98f..37a2de9 100644
--- a/sample-yaml/test-config.yaml.sample
+++ b/sample-yaml/test-config.yaml.sample
@@ -204,6 +204,25 @@
# large sets of tests that cover the same functionality, and allows them
# to share config files.
config-path: 'tests/foo'
+
+ # A path to a base directory containing abritrary files to be copied
+ # into the Asterisk well-known directories for Asterisk instances
+ # in this test.
+ # The directory should be relative to the testsuite top-level directory.
+ # For example... If set to 'tests/mytests/myfiles' then files in
+ # 'tests/mytest/myfiles/common/astvarlibdir/keys' would be copied to all
+ # of this test's Asterisk instances's '/var/lib/asterisk/keys' directories.
+ # Files in 'tests/mytest/myfiles/ast1/astvarlibdir/keys' would be copied
+ # into the first Asterisk instance's '/var/lib/asterisk/keys' directory.
+ # Files in 'tests/mytest/myfiles/ast2/astvarlibdir/keys' would be copied
+ # into the second Asterisk instance's '/var/lib/asterisk/keys' directory.
+ # etc.
+ # This is useful for tests that can share files like keys and certificates.
+ # NOTE: Files in individual '<testdir>/files' directories are automatically
+ # copied so you don't need this parameter to share files among instances
+ # in the same test.
+ base-files-directory: 'tests/mytests/myfiles'
+
reactor-timeout: 30
spawn-after-hangup: True
# Indicate that the test is going to be restarting Asterisk so
--
To view, visit https://gerrit.asterisk.org/c/testsuite/+/16587
To unsubscribe, or for help writing mail filters, visit https://gerrit.asterisk.org/settings
Gerrit-Project: testsuite
Gerrit-Branch: 19
Gerrit-Change-Id: I69bb54fea459e83f65a474d3b74c40b28fe01b4c
Gerrit-Change-Number: 16587
Gerrit-PatchSet: 1
Gerrit-Owner: George Joseph <gjoseph at digium.com>
Gerrit-MessageType: newchange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-code-review/attachments/20211006/c740b223/attachment-0001.html>
More information about the asterisk-code-review
mailing list