[Asterisk-code-review] testsuite: Allow copying of arbitrary files to asterisk instances (testsuite[19])

George Joseph asteriskteam at digium.com
Wed Oct 6 08:56:29 CDT 2021


George Joseph has uploaded this change for review. ( https://gerrit.asterisk.org/c/testsuite/+/16587 )


Change subject: testsuite: Allow copying of arbitrary files to asterisk instances
......................................................................

testsuite: Allow copying of arbitrary files to asterisk instances

If you have a key or certificate file that has to go in
/var/lib/asterisk/keys, there was no way for a test to get
that file into that directory.  Various work-arounds ensued.

* Added a replaceable parameter "<<instanceid>>" to config
  file processing which would be replaced with the Asterisk
  instance's id.  I.E.  'ast1' would be '1'.

* Added automatic installation of files in a test's "files"
  directory to its Asterisk instance's well-known directories.
  To use, create directories named after entries in the
  asterisk.conf "directories" category and place your files there.
  It works much like the test's "configs" directory.

  For example:
    mytest/
      configs/
        ast1/
          someconfig.conf
        ast2/
          someconfig.conf
      files/
        common/
          astvarlibdir/
            keys/
              ca.crt
        ast1/
          astvarlibdir/
            keys/
              mykey1.pem
        ast2/
          astvarlibdir/
            keys/
              mykey2.pem
      test-config.yaml

  Since 'astvarlibdir' is usually defined in asterisk.conf as
  '/var/lib/asterisk', this would copy 'ca.crt' to both Asterisk
  instance's '/var/lib/asterisk/keys' directory, 'mykey1.pem' to
  the first Asterisk instance's '/var/lib/asterisk/keys' directory
  and 'mykey2.pem' to the second instance's '/var/lib/asterisk/keys'
  directory.

  Each of the instance's "someconfig.conf" could reference those
  files like so:

  cacert = <<astvarlibdir>>/keys/ca.crt.pem
  keyfile = <<astvarlibdir>>/keys/mykey<<instanceid>>.pem

  If you have files that can be shared among tests, you can define
  'base-files-path' in each test's 'test-object-config' section which
  would point to a directory structured like the 'files' directory
  above.

  For example
    mytests/
      myfiles/
        common/
          astvarlibdir/
            keys/
              ca.crt
        ast1/
          astvarlibdir/
            keys/
              mykey1.pem
        ast2/
          astvarlibdir/
            keys/
              mykey2.pem

      mytest1/
        ...
      mytest2/
        ...

    Then in both test's test-config.yaml...
    test-object-config:
        base-files-path: 'tests/mytests/myfiles'

  base-files-path is processed before the automatic files
  processing of each test so individual tests can add its own files
  or even override a common file if it chooses.

ASTERISK-29675

Change-Id: I69bb54fea459e83f65a474d3b74c40b28fe01b4c
---
M README.txt
M lib/python/asterisk/asterisk.py
M lib/python/asterisk/test_case.py
A lib/python/polyfill.py
M sample-yaml/test-config.yaml.sample
5 files changed, 221 insertions(+), 2 deletions(-)



  git pull ssh://gerrit.asterisk.org:29418/testsuite refs/changes/87/16587/1

diff --git a/README.txt b/README.txt
index 459fbed..93888be 100644
--- a/README.txt
+++ b/README.txt
@@ -335,6 +335,41 @@
 Since each Asterisk instance required difference SIP settings, each 'ast%d'
 folder will have a different sip.conf file.
 
+You can also copy arbitrary files like sound, key and certificate files into
+any of the entries in the asterisk.conf "directories" category using its
+entry name name.  All intermediate directories will be created if they
+don't already exist.
+
+For example, to have a certificate and keys copied to an Asterisk instance's
+'/var/lib/asterisk/keys' directory, you'd place it in...
+
+    basic-call/
+        files/
+            common/
+                astvarlibdir/
+                    keys/
+                        ca.crt
+            ast1/
+                astvarlibdir/
+                    keys/
+                        instance1-key.pem
+                ...
+            ast2/
+                astvarlibdir/
+                    keys/
+                        instance2-key.pem
+
+Since 'astvarlibdir' is defined in asterisk.conf as '/var/lib/asterisk',
+this would copy 'ca.crt' to both instance's '/var/lib/asterisk/keys/' directory,
+'instance1-key.pem' to instance 1's '/var/lib/asterisk/keys/' directory and
+'instance2-key.pem' to instance 2's '/var/lib/asterisk/keys/' directory.
+
+If you have files that can be shared among multiple tests, you can create a
+directory following the same structure as above in some parent directory and
+direct each test to include it with the 'base-files-path' parameter in
+its test-config.yaml file.  See sample-yaml/test-config.yaml.sample for
+more info.
+
 d) Test Execution
 
         The "run-test" executable will be run by a top level application in the
diff --git a/lib/python/asterisk/asterisk.py b/lib/python/asterisk/asterisk.py
index 977dc6e..f1371db 100644
--- a/lib/python/asterisk/asterisk.py
+++ b/lib/python/asterisk/asterisk.py
@@ -16,6 +16,7 @@
 import shutil
 import logging
 import fileinput
+import polyfill
 
 from . import test_suite_utils
 
@@ -43,7 +44,6 @@
 
 LOGGER = logging.getLogger(__name__)
 
-
 class AsteriskRemoteProtocol(protocol.Protocol):
     """Class that acts as a remote protocol to Asterisk"""
 
@@ -367,6 +367,7 @@
         self.original_astmoddir = ""
         self.remote_config = remote_config
         self.memcheck_delay_stop = 0
+        self.instance_id = 0
         if test_config is not None and 'memcheck-delay-stop' in test_config:
             self.memcheck_delay_stop = test_config['memcheck-delay-stop'] or 0
 
@@ -419,6 +420,7 @@
             while True:
                 if not os.path.isdir("%s/ast%d" % (self.base, i)):
                     self.base = "%s/ast%d" % (self.base, i)
+                    self.instance_id = i;
                     break
                 i += 1
 
@@ -673,6 +675,7 @@
         for key in self.directories.keys():
             value = value.replace("<<%s>>" % key,
                               "%s%s" % (self.base, self.directories[key]))
+            value = value.replace("<<instanceid>>", self.instance_id)
         return value
 
 # Quick little function for doing search and replace in a file used below.
@@ -787,6 +790,61 @@
         except IOError:
             LOGGER.warn("The destination is not writable '%s'" % target_path)
 
+    def install_files(self, source_path):
+        """Installs all files located in a directory to this
+        instance of Asterisk.
+
+        Keyword Arguments:
+        source_path This argument must be the path to the directory
+                 containing the files to be installed into standard
+                 locations.
+
+                 Example: tests/my-cool-test/files/ast1
+                 This directory must contain at least one sub-directory
+                 matching the name of an entry from the asterisk.conf
+                 "directories" category.  For example "astvarlibdir".
+                 Each entry in those directories will be copied to the
+                 runtime equivalent for the test.
+
+        Example:
+            Given the following filesystem layout:
+            tests/my-cool-test/files/ast1
+                                    - astvarlibdir/
+                                        - sounds/
+                                            - recordings/
+                                                - myrecording.ulaw
+                                    - astkeydir/keys/
+                                        - mykey.pem
+
+            ...and given the following entries in asterisk.conf:
+            astvarlibdir => /var/lib/asterisk
+            astkeydir => /var/lib/asterisk
+
+            The myrecording.ulaw and mykey.pem files would be copied to
+            <testroot>/var/lib/asterisk/sounds/recordings/ and
+            <testroot>/var/lib/asterisk/keys/ respectively.
+        """
+
+        self._make_directory_structure()
+        if not os.path.exists(source_path):
+            return
+
+        for fname in os.listdir(source_path):
+            source = "%s/%s" % (source_path, fname)
+
+            if os.path.isdir(source):
+                # If the directory name isn't one of the
+                # well known directories, just skip it.
+                if not self.directories[fname]:
+                    continue
+                # join() doesn't work if one of the paths is absolute
+                # so we have to skip the leading '/' in the directory
+                # entry.  I.E /var/lib/asterisk becomes var/lib/asterisk
+                direntry = self.directories[fname]
+                dirpath = direntry[1:] if direntry[0] == '/' else direntry
+                dest = os.path.join(self.base, dirpath)
+                polyfill.copytree(source, dest, dirs_exist_ok=True)
+
     def _overwrite_file(self, filename, values):
         """Overwrite a particular config file
 
diff --git a/lib/python/asterisk/test_case.py b/lib/python/asterisk/test_case.py
index 1c7edfa..c4cde27 100644
--- a/lib/python/asterisk/test_case.py
+++ b/lib/python/asterisk/test_case.py
@@ -122,6 +122,7 @@
         self.ami = []
         self.fastagi = []
         self.base_config_path = None
+        self.base_files_path = None
         self.reactor_timeout = 30
         self.passed = None
         self.fail_tokens = []
@@ -149,6 +150,8 @@
         if test_config:
             if 'config-path' in test_config:
                 self.base_config_path = test_config['config-path']
+            if 'base-files-path' in test_config:
+                self.base_files_path = test_config['base-files-path']
             if 'reactor-timeout' in test_config:
                 self.reactor_timeout = test_config['reactor-timeout']
             if 'memcheck-delay-stop' in test_config:
@@ -248,7 +251,8 @@
                          for i in range(count)]
         return asterisks
 
-    def create_asterisk(self, count=1, base_configs_path=None, test_config=None):
+    def create_asterisk(self, count=1, base_configs_path=None, test_config=None,
+                        base_files_path=None):
         """Create n instances of Asterisk
 
         Note: if the instances of Asterisk being created are remote, the
@@ -264,6 +268,11 @@
                           configuration can be overwritten by individual tests,
                           however.
         test_config       Test Configuration
+        base_files_path   Provides common files for Asterisk instances
+                          to use. This is useful for certain test types that use
+                          the same files, like keys, all the time. This
+                          configuration can be overwritten by individual tests,
+                          however.
         """
         for i, ast_config in enumerate(self.get_asterisk_hosts(count)):
             local_num = ast_config.get('num')
@@ -300,6 +309,19 @@
                                             (self.test_name, local_num),
                                             self.test_config.get_deps())
 
+                # If a base files directory exists for this Asterisk instance
+                # has been provided, install it first
+                if base_files_path is None:
+                    base_files_path = self.base_files_path
+                if base_files_path:
+                    self.ast[i].install_files("%s/common" % (base_files_path))
+                    self.ast[i].install_files("%s/ast%d" %
+                                            (base_files_path, local_num))
+                # Copy test specific config files
+                self.ast[i].install_files("%s/files/common" % (self.test_name))
+                self.ast[i].install_files("%s/files/ast%d" %
+                                            (self.test_name, local_num))
+
     def create_ami_factory(self, count=1, username="user", secret="mysecret",
                            port=5038):
         """Create n instances of AMI.  Each AMI instance will attempt to connect
diff --git a/lib/python/polyfill.py b/lib/python/polyfill.py
new file mode 100644
index 0000000..bd95d5f
--- /dev/null
+++ b/lib/python/polyfill.py
@@ -0,0 +1,85 @@
+import sys
+import os
+import shutil
+
+"""
+    These are polyfills for a few file/directory functions
+    that don't show up until Python 3.9.
+"""
+
+def makedirs(name, mode=0o777, exist_ok=False):
+    head, tail = os.path.split(name)
+    if not tail:
+        head, tail = os.path.split(head)
+    if head and tail and not os.path.exists(head):
+        try:
+            makedirs(head, exist_ok=exist_ok)
+        except FileExistsError:
+            # Defeats race condition when another thread created the path
+            pass
+        cdir = os.curdir
+        if isinstance(tail, bytes):
+            cdir = bytes(os.curdir, 'ASCII')
+        if tail == cdir:           # xxx/newdir/. exists if xxx/newdir exists
+            return
+    try:
+        os.mkdir(name, mode)
+    except OSError:
+        if not exist_ok or not os.path.isdir(name):
+            raise
+
+def copytree(src, dst, symlinks=False, ignore=None, copy_function=shutil.copy2,
+             ignore_dangling_symlinks=False, dirs_exist_ok=False):
+    names = os.listdir(src)
+    if ignore is not None:
+        ignored_names = ignore(src, names)
+    else:
+        ignored_names = set()
+
+    makedirs(dst, exist_ok=dirs_exist_ok)
+    errors = []
+    for name in names:
+        if name in ignored_names:
+            continue
+        srcname = os.path.join(src, name)
+        dstname = os.path.join(dst, name)
+        try:
+            if os.path.islink(srcname):
+                linkto = os.readlink(srcname)
+                if symlinks:
+                    # We can't just leave it to `copy_function` because legacy
+                    # code with a custom `copy_function` may rely on copytree
+                    # doing the right thing.
+                    os.symlink(linkto, dstname)
+                    shutil.copystat(srcname, dstname, follow_symlinks=not symlinks)
+                else:
+                    # ignore dangling symlink if the flag is on
+                    if not os.path.exists(linkto) and ignore_dangling_symlinks:
+                        continue
+                    # otherwise let the copy occurs. copy2 will raise an error
+                    if os.path.isdir(srcname):
+                        copytree(srcname, dstname, symlinks, ignore,
+                                 copy_function, dirs_exist_ok=dirs_exist_ok)
+                    else:
+                        copy_function(srcname, dstname)
+            elif os.path.isdir(srcname):
+                copytree(srcname, dstname, symlinks, ignore, copy_function, dirs_exist_ok=dirs_exist_ok)
+            else:
+                # Will raise a SpecialFileError for unsupported file types
+                copy_function(srcname, dstname)
+        # catch the Error from the recursive copytree so that we can
+        # continue with other files
+        except shutil.Error as err:
+            errors.extend(err.args[0])
+        except OSError as why:
+            errors.append((srcname, dstname, str(why)))
+    try:
+        shutil.copystat(src, dst)
+    except OSError as why:
+        # Copying file access times may fail on Windows
+        if getattr(why, 'winerror', None) is None:
+            errors.append((src, dst, str(why)))
+    if errors:
+        raise shutil.Error(errors)
+    return dst
+
diff --git a/sample-yaml/test-config.yaml.sample b/sample-yaml/test-config.yaml.sample
index 773b98f..37a2de9 100644
--- a/sample-yaml/test-config.yaml.sample
+++ b/sample-yaml/test-config.yaml.sample
@@ -204,6 +204,25 @@
     # large sets of tests that cover the same functionality, and allows them
     # to share config files.
     config-path: 'tests/foo'
+
+    # A path to a base directory containing abritrary files to be copied
+    # into the Asterisk well-known directories for Asterisk instances
+    # in this test.
+    # The directory should be relative to the testsuite top-level directory.
+    # For example... If set to 'tests/mytests/myfiles' then files in
+    # 'tests/mytest/myfiles/common/astvarlibdir/keys' would be copied to all
+    # of this test's Asterisk instances's '/var/lib/asterisk/keys' directories.
+    # Files in 'tests/mytest/myfiles/ast1/astvarlibdir/keys' would be copied
+    # into the first Asterisk instance's '/var/lib/asterisk/keys' directory.
+    # Files in 'tests/mytest/myfiles/ast2/astvarlibdir/keys' would be copied
+    # into the second Asterisk instance's '/var/lib/asterisk/keys' directory.
+    # etc.
+    # This is useful for tests that can share files like keys and certificates.
+    # NOTE: Files in individual '<testdir>/files' directories are automatically
+    # copied so you don't need this parameter to share files among instances
+    # in the same test.
+    base-files-directory: 'tests/mytests/myfiles'
+
     reactor-timeout: 30
     spawn-after-hangup: True
     # Indicate that the test is going to be restarting Asterisk so

-- 
To view, visit https://gerrit.asterisk.org/c/testsuite/+/16587
To unsubscribe, or for help writing mail filters, visit https://gerrit.asterisk.org/settings

Gerrit-Project: testsuite
Gerrit-Branch: 19
Gerrit-Change-Id: I69bb54fea459e83f65a474d3b74c40b28fe01b4c
Gerrit-Change-Number: 16587
Gerrit-PatchSet: 1
Gerrit-Owner: George Joseph <gjoseph at digium.com>
Gerrit-MessageType: newchange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-code-review/attachments/20211006/c740b223/attachment-0001.html>


More information about the asterisk-code-review mailing list