[Asterisk-code-review] testsuite: NOTIFY count tests unstable (testsuite[16])
Michael Bradeen
asteriskteam at digium.com
Mon Sep 12 13:38:39 CDT 2022
Michael Bradeen has uploaded this change for review. ( https://gerrit.asterisk.org/c/testsuite/+/19209 )
Change subject: testsuite: NOTIFY count tests unstable
......................................................................
testsuite: NOTIFY count tests unstable
scenario_iterator - new asterisk testsuite py module to allow
iteration of tests that require a series of sipp scenario(s)
+ ami event(s) pairs. Each iteration is started at the stop
of the previous to prevent collision.
mwi_aggregate and mailbox_count_changes - modified to use
this new method
ASTERISK-30214
Change-Id: Ic7f318bc953416ff8d8673ed4768fef264046fe3
---
A lib/python/asterisk/scenario_iterator.py
M tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
M tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
M tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
M tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
5 files changed, 277 insertions(+), 72 deletions(-)
git pull ssh://gerrit.asterisk.org:29418/testsuite refs/changes/09/19209/1
diff --git a/lib/python/asterisk/scenario_iterator.py b/lib/python/asterisk/scenario_iterator.py
new file mode 100644
index 0000000..6b8dbe6
--- /dev/null
+++ b/lib/python/asterisk/scenario_iterator.py
@@ -0,0 +1,175 @@
+'''
+Copyright (C) 2022, Sangoma Technologies Corp
+Mike Bradeen <mbradeen at sangoma.com>
+
+This program is free software, distributed under the terms of
+the GNU General Public License Version 2.
+
+The scenario_iterator is designed to let us start sipp, then perform an ami
+action as a cycle of start sipp->generate event(s)->sipp stops, start sipp...
+This is for tests that are performing the same test different ways or performing
+different iterations of the same test.
+
+An example would be to start a scenario that waits for a NOTIFY, then send the AMI
+to generate that NOTIFY, then starts a second scenario, send a second AMI, etc.
+
+There are two classes for this, the singleIterator and the multiIterator. The
+singleIterator is one-to-one, with each scenario start triggering the next ami and the
+scenario stop triggering the next scenario start. It is fed a pair of lists, one is a
+list of scenarios and the other is a list of AMI actions. After the final scenario,
+we add a 'done' indicator to the sipp side and a final event on the AMI side, example:
+
+scenarios = [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-4.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'done'}
+]
+
+actions = [
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'2', 'OldMessages':'0'},
+ {'Action': 'MWIDelete', 'Mailbox': 'alice'},
+ {'Action': 'UserEvent', 'UserEvent': 'testComplete'}
+]
+
+You then define a local instance and run it:
+
+def start_test(test_object, junk):
+ LOGGER.info("Starting mwi_check")
+ testrunner = singleIterator(test_object, scenarios, actions)
+ testrunner.run(junk)
+
+The multiIterator is a many-to-many mapper that allows multiple scenarios to be started, followed
+by multiple actions. The end of the last scenario in the list causes the next scenario to start.
+
+Multiple sequences can be tied to individual actions, or vice versa. In this example we start two
+scenarios but then only generate one corresponding AMI:
+
+scenarios = [
+ {'Name': 'mailbox_a', 'sequence': [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-1.xml', 'port': '5062', 'target': '127.0.0.1'}]},
+ {'Name': 'done'}
+]
+
+actions = [
+ {'Messages': [{'Action': 'MWIUpdate', 'Mailbox': 'mailbox_a', 'NewMessages':'2', 'OldMessages':'1'}]},
+ {'Messages': [{'Action': 'UserEvent', 'UserEvent': 'testComplete'}]}
+]
+
+In the above examples, we end the test with a testComplete event that we then use to trigger the
+test stop event in the corresponding yaml file, example:
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
+
+Also, this test REQUIRES the following be set in the sipp configuration section of your yaml file if
+you are using a sipp scenario (like a REGISTER) to kick off the sequnce:
+ stop-after-scenarios: False
+
+
+'''
+import sys
+import logging
+
+sys.path.append("lib/python")
+
+from twisted.internet import reactor
+from asterisk.sipp import SIPpScenario
+from asterisk.sipp import SIPpScenarioSequence
+
+LOGGER = logging.getLogger(__name__)
+
+
+class singleIterator(object):
+
+ def __init__(self, test_object, scenarios, actions):
+ self.test_object = test_object
+ self.scenarios = scenarios
+ self.actions = actions
+ self.iteration = 0
+
+ def __iterate(self):
+ scenario = self.scenarios[self.iteration]
+ message = self.actions[self.iteration]
+ self.iteration += 1
+ if scenario['Name'] != 'done':
+ self.__startScenario(scenario)
+ self.__sendMessage(message)
+
+ def __startScenario(self, scenario):
+ LOGGER.info("Starting sipp scenario %s" % scenario['Name'])
+ sipp_scenario = SIPpScenario(self.test_object.test_name,
+ {'scenario': scenario['Name'],
+ '-p': scenario['port']},
+ target=scenario['target'])
+ exiter = sipp_scenario.run(self.test_object)
+ exiter.addCallback(self.run)
+
+ def __sendMessage(self, message):
+ testami = self.test_object.ami[0]
+ LOGGER.info("Scheduling AMI %s" % message['Action'])
+ reactor.callLater(2, testami.sendMessage, message)
+
+ def run(self, junk):
+ self.__iterate()
+
+class multiIterator(object):
+
+ def __init__(self, test_object, scenariosequences, actions):
+ self.test_object = test_object
+ self.scenariosequences = scenariosequences
+ self.actions = actions
+ self.iteration = 0
+ self.sequencecounter = 1
+
+ def __iterate(self):
+ sippsequence = self.scenariosequences[self.iteration]
+ messagesequence = self.actions[self.iteration]
+ self.iteration += 1
+ if sippsequence['Name'] != 'done':
+ LOGGER.info("Starting sipp sequence %s" % sippsequence['Name'])
+ self.__startScenarios(sippsequence['sequence'])
+ self.__sendMessages(messagesequence['Messages'])
+
+ def __startScenarios(self, sippscenarios):
+
+ def intermediateHandler(junk):
+ LOGGER.debug("Series next")
+
+ sipp_sequence = SIPpScenarioSequence(self.test_object,
+ fail_on_any=True,
+ stop_on_done=False)
+ self.sequencecounter = 0
+ for scenario in sippscenarios:
+ LOGGER.info("Adding scenario %s to sequence" % scenario['Name'])
+ sipp_scenario = SIPpScenario(self.test_object.test_name,
+ {'scenario': scenario['Name'],
+ '-p': scenario['port']},
+ target=scenario['target'])
+ sipp_sequence.register_scenario(sipp_scenario)
+ self.sequencecounter +=1
+
+ sipp_sequence.register_scenario_stop_callback(self.run)
+ sipp_sequence.execute()
+
+ def __sendMessages(self, messages):
+ testami = self.test_object.ami[0]
+ delay = 2
+ for message in messages:
+ LOGGER.info("Scheduling AMI %s" % message['Action'])
+ reactor.callLater(delay, testami.sendMessage, message)
+ # spread out the messages
+ delay +=2
+
+ def run(self, junk):
+ if self.sequencecounter == 1:
+ self.__iterate()
+ else:
+ self.sequencecounter -=1
+
diff --git a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
index 9a5d5e3..45d1b8d 100644
--- a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
+++ b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
@@ -8,32 +8,43 @@
the GNU General Public License Version 2.
"""
+import mailbox
import sys
import logging
sys.path.append("lib/python")
-from twisted.internet import reactor
+from asterisk.scenario_iterator import multiIterator
LOGGER = logging.getLogger(__name__)
-mwis = [
- {'mailbox': 'mailbox_a', 'new': '2', 'old': '1'},
- {'mailbox': 'mailbox_b', 'new': '3', 'old': '3'},
+
+mailbox_a = [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-1.xml', 'port': '5062', 'target': '127.0.0.1'}
]
-def walk_states(test_object, accounts):
+mailbox_b = [
+ {'Name': 'alice-is-notified-2.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-2.xml', 'port': '5062', 'target': '127.0.0.1'}
+]
- testami = test_object.ami[0]
- statedelay = 2
- for mwi in mwis:
- LOGGER.info("Sending MWI update. new: %s, old %s" %
- (mwi['new'],
- mwi['old']))
- message = {
- 'Action': 'MWIUpdate',
- 'Mailbox': mwi['mailbox'],
- 'NewMessages': mwi['new'],
- 'OldMessages': mwi['old']
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
- statedelay += 1
+mwiscenarios = [
+ {'Name': 'mailbox_a', 'sequence': mailbox_a},
+ {'Name': 'mailbox_b', 'sequence': mailbox_b},
+ {'Name': 'done'}
+]
+
+mwis = [
+ {'Messages': [{'Action': 'MWIUpdate', 'Mailbox': 'mailbox_a', 'NewMessages':'2', 'OldMessages':'1'}]},
+ {'Messages': [{'Action': 'MWIUpdate', 'Mailbox': 'mailbox_b', 'NewMessages':'3', 'OldMessages':'3'}]},
+ {'Messages': [{'Action': 'UserEvent', 'UserEvent': 'testComplete'}]}
+]
+
+def start_test(test_object, junk):
+ LOGGER.info("Starting mwi_check")
+ testrunner = multiIterator(test_object, mwiscenarios, mwis)
+ testrunner.run(junk)
+
+def stop_test():
+ return
+
diff --git a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
index 48a0971..cabe5e2 100644
--- a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
+++ b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
@@ -1,5 +1,4 @@
testinfo:
- skip: 'Unstable - ASTERISK-30214'
summary: 'Ensures mailbox state is aggregated or not aggregated when appropriate'
description: |
"Alice and Bob both receive mailbox updates for mailbox_a and mailbox_b. However, Alice
@@ -30,13 +29,18 @@
test-object:
config-section: sipp-config
typename: 'sipp.SIPpTestCase'
+ modules:
+ -
+ config-section: 'ami-config'
+ typename: 'pluggable_modules.EventActionModule'
sipp-config:
connect-ami: 'True'
reactor-timeout: 30
fail-on-any: True
- start_callback_module: 'mwi_check'
- start_callback_method: 'walk_states'
+ stop-after-scenarios: False
+ stop_callback_module: 'mwi_check'
+ stop_callback_method: 'start_test'
test-iterations:
# We pass the initial registers the -aa flag then let them run for a second so they can get the
# 1-2 initial NOTIFY messages we don't care about. The MWI AMI updates are set to start after
@@ -46,12 +50,12 @@
'ordered-args': {'-aa'} }
- { 'key-args': {'scenario': 'bob-registers.xml', '-p': '5062'},
'ordered-args': {'-aa'} }
- # Combining this with the -m makes the scripts regex conditional on call_number
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-1.xml', '-p': '5061'} }
- - { 'key-args': {'scenario': 'bob-is-notified-1.xml', '-p': '5062'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-2.xml', '-p': '5061'} }
- - { 'key-args': {'scenario': 'bob-is-notified-2.xml', '-p': '5062'} }
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
diff --git a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
index 10a4e4f..28c4ded 100644
--- a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
+++ b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
@@ -6,36 +6,31 @@
sys.path.append("lib/python")
from twisted.internet import reactor
+from asterisk.scenario_iterator import singleIterator
LOGGER = logging.getLogger(__name__)
-mwis = [
- {'new': '2', 'old': '0'},
- {'new': '1', 'old': '1'},
- {'new': '0', 'old': '2'},
+mwiscenarios = [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-2.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-3.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-4.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'done'}
]
-def walk_states(test_object, junk):
+mwis = [
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'2', 'OldMessages':'0'},
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'1', 'OldMessages':'1'},
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'0', 'OldMessages':'2'},
+ {'Action': 'MWIDelete', 'Mailbox': 'alice'},
+ {'Action': 'UserEvent', 'UserEvent': 'testComplete'}
+]
- testami = test_object.ami[0]
- statedelay = 2
- for mwi in mwis:
- LOGGER.info("Sending MWI update. new: %s, old %s" %
- (mwi['new'],
- mwi['old']))
- message = {
- 'Action': 'MWIUpdate',
- 'Mailbox': 'alice',
- 'NewMessages': mwi['new'],
- 'OldMessages': mwi['old']
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
- statedelay += 1
+def start_test(test_object, junk):
+ LOGGER.info("Starting mwi_check")
+ testrunner = singleIterator(test_object, mwiscenarios, mwis)
+ testrunner.run(junk)
- # delete mailbox after walking states
- LOGGER.info("Deleting Mailbox")
- message = {
- 'Action': 'MWIDelete',
- 'Mailbox': 'alice',
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
+def stop_test():
+ return
+
diff --git a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
index 5cb1150..1bdeda7 100644
--- a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
+++ b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
@@ -26,13 +26,18 @@
test-object:
config-section: sipp-config
typename: 'sipp.SIPpTestCase'
+ modules:
+ -
+ config-section: 'ami-config'
+ typename: 'pluggable_modules.EventActionModule'
sipp-config:
connect-ami: 'True'
reactor-timeout: 30
fail-on-any: True
- start_callback_module: 'mwi_check'
- start_callback_method: 'walk_states'
+ stop-after-scenarios: False
+ stop_callback_module: 'mwi_check'
+ stop_callback_method: 'start_test'
test-iterations:
# We pass the initial registers the -aa flag then let them run for a second so they can get the
# 1-2 initial NOTIFY messages we don't care about. The MWI AMI updates are set to start after
@@ -40,16 +45,12 @@
scenarios:
- { 'key-args': {'scenario': 'alice-registers.xml', '-p': '5061'},
'ordered-args': {'-aa'} }
- # Combining this with the -m makes the scripts regex conditional on call_number
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-1.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-2.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-3.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-4.xml', '-p': '5061'} }
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
--
To view, visit https://gerrit.asterisk.org/c/testsuite/+/19209
To unsubscribe, or for help writing mail filters, visit https://gerrit.asterisk.org/settings
Gerrit-Project: testsuite
Gerrit-Branch: 16
Gerrit-Change-Id: Ic7f318bc953416ff8d8673ed4768fef264046fe3
Gerrit-Change-Number: 19209
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Bradeen <mbradeen at sangoma.com>
Gerrit-MessageType: newchange
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-code-review/attachments/20220912/9f1674f3/attachment-0001.html>
More information about the asterisk-code-review
mailing list