[Asterisk-code-review] Add the ability to execute the Test Suite against a remote i... (testsuite[master])

[Matt Jordan]

Hello Anonymous Coward #1000019,

I'd like you to reexamine a change.  Please visit

    https://gerrit.asterisk.org/1521

to look at the new patch set (#2).

Change subject: Add the ability to execute the Test Suite against a remote instance of Asterisk
......................................................................

Add the ability to execute the Test Suite against a remote instance of Asterisk

The normal operation of the Test Suite is for it to manage and control local
instances of Asterisk, sand-boxing as many instances as a test needs. This
works well when what is tested are ideal, logical blocks of functionality.
There are, however, a few drawbacks to this approach:
(1) The Test Suite runs on the same machine as the n instances of Asterisk that
    it ochestrates. While for some smaller tests this is fine, for some
    scenarios, this has drawbacks as it may affect the system under test.
(2) The Test Suite has to 'repeat itself' for large categories of tests. If
    what is being tested is a small piece of functionality, this repetition is
    merely a nuisance. For larger, complex systems, this is more problematic.

Fundamentally altering the nature of the Test Suite is clearly not a good idea:
we have hundreds of tests that assume they (a) control the configuration of the
system under test, and (b) that the system under test is on the same machine.
At the same time, the Test Suite has a lot of libraries and functionality that
make it suitable for testing more complex systems in which Asterisk is merely
a piece.

This patch makes that possible.

The Test Suite can now be placed into another 'mode' of operation. This mode
fundamentally changes how the management of Asterisk processes works, such that
the 'normal' tests in the Test Suite tree are non-operational. Instead, it
switches into a 'remote' mode, where it assumes that the Asterisk instances
under test exist on a remote system. The 'remote' mode is triggered by a
specific configuration in the global test-config.yaml file.

In remote mode, several changes occur:
(1) All CLI interaction with the remote Asterisk instances occurs over SSH.
    SSH credentials are provided for each remote Asterisk instance in the
    global test-config.yaml file.
(2) No configuration of the remote system is explicitly handled by the
    Asterisk class. Instead, it is assumed that another pluggable module
    would provide that work, or that the system has a static configuration
    that the test is aware of.
(3) We no longer attempt to start or stop Asterisk. Instead, we merely check
    that Asterisk is indeed fully booted. Tests will still receive callbacks
    when Asterisk has been 'started' and 'stopped', which merely reflect
    the beginning/ending of a test.
(4) AMI/AGI factories can now be pointed at a remote system. ARI always had
    this ability, given proper module configuration.
(5) Dependency testing is much more limited. Since the Asterisk system(s) are
    remote, and we may have many systems we are testing against, the following
    are not checked:
    - Asterisk versions
    - Asterisk build options
    - Asterisk module dependencies

Sample YAML demonstrating how to switch the Test Suite into remote mode has
been provided as well.

Change-Id: I59a7632e5a1cb38c46484c5035f36993139b7f3f
---
M lib/python/asterisk/asterisk.py
M lib/python/asterisk/test_case.py
M lib/python/asterisk/test_config.py
M sample-yaml/test-config.yaml.sample
M test-config.yaml
5 files changed, 354 insertions(+), 87 deletions(-)


  git pull ssh://gerrit.asterisk.org:29418/testsuite refs/changes/21/1521/2
-- 
To view, visit https://gerrit.asterisk.org/1521
To unsubscribe, visit https://gerrit.asterisk.org/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I59a7632e5a1cb38c46484c5035f36993139b7f3f
Gerrit-PatchSet: 2
Gerrit-Project: testsuite
Gerrit-Branch: master
Gerrit-Owner: Matt Jordan <mjordan at digium.com>
Gerrit-Reviewer: Anonymous Coward #1000019
Gerrit-Reviewer: Matt Jordan <mjordan at digium.com>