[Asterisk-code-review] testsuite: NOTIFY count tests unstable (testsuite[certified/18.9])
Friendly Automation
asteriskteam at digium.com
Thu Sep 15 13:38:21 CDT 2022
Friendly Automation has submitted this change. ( https://gerrit.asterisk.org/c/testsuite/+/19261 )
Change subject: testsuite: NOTIFY count tests unstable
......................................................................
testsuite: NOTIFY count tests unstable
scenario_iterator - new asterisk testsuite py module to allow
iteration of tests that require a series of sipp scenario(s)
+ ami event(s) pairs. Each iteration is started at the stop
of the previous to prevent collision.
mwi_aggregate and mailbox_count_changes - modified to use
this new method
ASTERISK-30214
Change-Id: Ic7f318bc953416ff8d8673ed4768fef264046fe3
---
A lib/python/asterisk/scenario_iterator.py
M tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
M tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
M tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
M tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
5 files changed, 329 insertions(+), 73 deletions(-)
Approvals:
George Joseph: Looks good to me, approved
Friendly Automation: Approved for Submit
diff --git a/lib/python/asterisk/scenario_iterator.py b/lib/python/asterisk/scenario_iterator.py
new file mode 100644
index 0000000..cff1af3
--- /dev/null
+++ b/lib/python/asterisk/scenario_iterator.py
@@ -0,0 +1,231 @@
+'''
+Copyright (C) 2022, Sangoma Technologies Corp
+Mike Bradeen <mbradeen at sangoma.com>
+
+This program is free software, distributed under the terms of
+the GNU General Public License Version 2.
+
+The scenario_iterator is designed to let us start sipp, then perform an ami
+action as a cycle of start sipp->generate event(s)->sipp stops, start sipp...
+This is for tests that are performing the same test different ways or performing
+different iterations of the same test.
+
+An example would be to start a scenario that waits for a NOTIFY, then send the AMI
+to generate that NOTIFY, then starts a second scenario, send a second AMI, etc.
+
+There are two classes for this, the singleIterator and the multiIterator. The
+singleIterator is one-to-one, with each scenario start triggering the next ami and the
+scenario stop triggering the next scenario start. It is fed a pair of lists, one is a
+list of scenarios and the other is a list of AMI actions. After the final scenario,
+we add a 'done' indicator to the sipp side and a final event on the AMI side, example:
+
+scenarios = [
+ {'Name': 'none'},
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-2.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'done'}
+]
+
+actions = [
+ {'Action': 'UserEvent', 'UserEvent': 'testStarted'}
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'2', 'OldMessages':'0'},
+ {'Action': 'MWIDelete', 'Mailbox': 'alice'},
+ {'Action': 'UserEvent', 'UserEvent': 'testComplete'}
+]
+
+You then define a local instance and run it:
+
+def start_test(test_object, junk):
+ testrunner = singleIterator(test_object, scenarios, actions)
+ testrunner.run(junk)
+
+The multiIterator is a many-to-many mapper that allows multiple scenarios to be started, followed
+by multiple actions. The end of the last scenario in the list causes the next scenario to start.
+
+Multiple sequences can be tied to individual actions, or vice versa. In this example we start two
+scenarios but then only generate one corresponding AMI:
+
+scenarios = [
+ {'Name': 'mailbox_a', 'sequence': [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-1.xml', 'port': '5062', 'target': '127.0.0.1'} ]},
+ {'Name': 'done'}
+]
+
+actions = [
+ {'Messages': [{'Action': 'MWIUpdate', 'Mailbox': 'mailbox_a', 'NewMessages':'2', 'OldMessages':'1'}]},
+ {'Messages': [{'Action': 'UserEvent', 'UserEvent': 'testComplete'}]}
+]
+
+In the above examples, we end the test with a testComplete event that we then use to trigger the
+test stop event in the corresponding yaml file, example:
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
+
+Also, this test REQUIRES the following be set in the sipp configuration section of your yaml file if
+you are using a sipp scenario (like a REGISTER) to kick off the sequnce:
+ stop-after-scenarios: False
+
+A Name of 'none' for a scenario(list) or an Action of 'none' skips that particular scenario or action,
+but not the other. This is mostly to allow one or more AMI events to be triggered before the
+corresponding sipp scenario is started but could also allow for intermediate sipp scenarios. Setting
+both to 'none' would be functionally the same as waiting for 1 second before going on to the next
+iteration.
+
+'''
+from asterisk.sipp import SIPpScenarioSequence
+from asterisk.sipp import SIPpScenario
+from twisted.internet import reactor
+import sys
+import logging
+
+sys.path.append("lib/python")
+
+LOGGER = logging.getLogger(__name__)
+
+sipp_terminator = {'Name': 'done'}
+empty_action = {'Action': 'none'}
+empty_action_list = {'Messages': [{'Action': 'none'}] }
+
+class singleIterator(object):
+
+ def __init__(self, test_object, scenarios, actions):
+ self.test_object = test_object
+ self.scenarios = scenarios
+ self.actions = actions
+ self.iteration = 0
+
+ def __iterate(self):
+ try:
+ scenario = self.scenarios[self.iteration]
+ except IndexError:
+ LOGGER.warning("End of scenario list without proper termination")
+ scenario = sipp_terminator
+ try:
+ message = self.actions[self.iteration]
+ except IndexError:
+ LOGGER.warning("End of action list without proper termination")
+ message = empty_action
+
+ self.iteration += 1
+ if scenario['Name'] == 'none':
+ # skip ahead to the next iteration but send the AMI
+ # action if set. Speed up the iteration.
+ self.__sendMessage(message, 0)
+ reactor.callLater(1, self.run)
+ elif scenario['Name'] != 'done':
+ # A scenaro was specified so run it then schedule the
+ # AMI event if there is one.
+ self.__startScenario(scenario)
+ self.__sendMessage(message)
+ else:
+ # At the final iteration, send any final AMI immediately
+ self.__sendMessage(message, 0)
+
+ def __startScenario(self, scenario):
+ LOGGER.info("Starting sipp scenario %s" % scenario['Name'])
+ sipp_scenario = SIPpScenario(self.test_object.test_name,
+ {'scenario': scenario['Name'],
+ '-p': scenario['port']},
+ target=scenario['target'])
+ exiter = sipp_scenario.run(self.test_object)
+ exiter.addCallback(self.run)
+
+ def __sendMessage(self, message, delay=2):
+ if message['Action'] != 'none':
+ testami = self.test_object.ami[0]
+ LOGGER.info("Scheduling AMI %s" % message['Action'])
+ reactor.callLater(delay, testami.sendMessage, message)
+
+ def run(self, junk=None):
+ self.__iterate()
+
+
+class multiIterator(object):
+
+ def __init__(self, test_object, scenariosequences, actions):
+ self.test_object = test_object
+ self.scenariosequences = scenariosequences
+ self.actions = actions
+ self.iteration = 0
+ self.sequencecounter = 1
+
+ def __iterate(self):
+ try:
+ sippsequence = self.scenariosequences[self.iteration]
+ except IndexError:
+ LOGGER.warning("End of scenarios list without proper termination")
+ sippsequence = sipp_terminator
+ try:
+ messagesequence = self.actions[self.iteration]
+ except IndexError:
+ LOGGER.warning("End of action list without proper termination")
+ messagesequence = empty_action_list
+
+ self.iteration += 1
+ if sippsequence['Name'] == 'none':
+ # skip ahead to the next iteration but send any AMI actions if
+ # set, one second apart. Run the next iteration based on how far
+ # out we've scheduled messages.
+ sequencedelay = self.__sendMessages(messagesequence['Messages'], 1)
+ reactor.callLater(sequencedelay, self.run)
+ elif sippsequence['Name'] != 'done':
+ # A scenaro sequence was specified so run it then schedule the
+ # AMI event(s) normally. Set the delay equal to how many scenarios
+ # were registered (as seconds) to try and scale against complexity
+ LOGGER.info("Starting sipp sequence %s" % sippsequence['Name'])
+ self.__startScenarios(sippsequence['sequence'])
+ self.__sendMessages(messagesequence['Messages'], self.sequencecounter)
+ else:
+ # At the final iteration, send any final AMI
+ sequencedelay = self.__sendMessages(messagesequence['Messages'], 0)
+
+ def __startScenarios(self, sippscenarios):
+
+ sipp_sequence = SIPpScenarioSequence(self.test_object,
+ fail_on_any=True,
+ stop_on_done=False)
+ self.sequencecounter = 0
+ for scenario in sippscenarios:
+ LOGGER.info("Adding scenario %s to sequence" % scenario['Name'])
+ sipp_scenario = SIPpScenario(self.test_object.test_name,
+ {'scenario': scenario['Name'],
+ '-p': scenario['port']},
+ target=scenario['target'])
+ sipp_sequence.register_scenario(sipp_scenario)
+ # keep track of how many scenarios we register so we don't
+ # iterate until we have stop_callback hits for all of them
+ self.sequencecounter += 1
+
+ sipp_sequence.register_scenario_stop_callback(self.run)
+ sipp_sequence.execute()
+
+ def __sendMessages(self, messages, delay = 2):
+ testami = self.test_object.ami[0]
+ messagedelay = delay
+ for message in messages:
+ if message['Action'] !='none':
+ LOGGER.info("Scheduling AMI %s" % message['Action'])
+ reactor.callLater(delay, testami.sendMessage, message)
+ # spread out the messages by 2 seconds
+ messagedelay += 2
+ # return last action's delay + 2, ie when to run the next
+ # command if you want it evenly distributed and after the
+ # last message scheduled here
+ return messagedelay
+
+ def run(self, junk=None):
+ # only run the next iteration once there is a call-back for
+ # each sipp scenario in the sequence. Otherwise, decrement
+ # until we get there
+ if self.sequencecounter == 1:
+ self.__iterate()
+ else:
+ self.sequencecounter -= 1
diff --git a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
index 9a5d5e3..9fadeb0 100644
--- a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
+++ b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/mwi_check.py
@@ -8,32 +8,39 @@
the GNU General Public License Version 2.
"""
+import mailbox
import sys
import logging
sys.path.append("lib/python")
-from twisted.internet import reactor
+from asterisk.scenario_iterator import multiIterator
LOGGER = logging.getLogger(__name__)
-mwis = [
- {'mailbox': 'mailbox_a', 'new': '2', 'old': '1'},
- {'mailbox': 'mailbox_b', 'new': '3', 'old': '3'},
+mwiscenarios = [
+ {'Name': 'mailbox_a', 'sequence': [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-1.xml', 'port': '5062', 'target': '127.0.0.1'} ]},
+ {'Name': 'mailbox_b', 'sequence': [
+ {'Name': 'alice-is-notified-2.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'bob-is-notified-2.xml', 'port': '5062', 'target': '127.0.0.1'} ]},
+ {'Name': 'done'}
]
-def walk_states(test_object, accounts):
+mwis = [
+ {'Messages': [
+ {'Action': 'MWIUpdate', 'Mailbox': 'mailbox_a', 'NewMessages':'2', 'OldMessages':'1'} ]},
+ {'Messages': [
+ {'Action': 'MWIUpdate', 'Mailbox': 'mailbox_b', 'NewMessages':'3', 'OldMessages':'3'} ]},
+ {'Messages': [
+ {'Action': 'UserEvent', 'UserEvent': 'testComplete'} ]}
+]
- testami = test_object.ami[0]
- statedelay = 2
- for mwi in mwis:
- LOGGER.info("Sending MWI update. new: %s, old %s" %
- (mwi['new'],
- mwi['old']))
- message = {
- 'Action': 'MWIUpdate',
- 'Mailbox': mwi['mailbox'],
- 'NewMessages': mwi['new'],
- 'OldMessages': mwi['old']
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
- statedelay += 1
+def start_test(test_object, junk):
+ LOGGER.info("Starting mwi_check")
+ testrunner = multiIterator(test_object, mwiscenarios, mwis)
+ testrunner.run(junk)
+
+def stop_test():
+ return
+
diff --git a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
index 48a0971..cabe5e2 100644
--- a/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
+++ b/tests/channels/pjsip/subscriptions/mwi/mwi_aggregate/test-config.yaml
@@ -1,5 +1,4 @@
testinfo:
- skip: 'Unstable - ASTERISK-30214'
summary: 'Ensures mailbox state is aggregated or not aggregated when appropriate'
description: |
"Alice and Bob both receive mailbox updates for mailbox_a and mailbox_b. However, Alice
@@ -30,13 +29,18 @@
test-object:
config-section: sipp-config
typename: 'sipp.SIPpTestCase'
+ modules:
+ -
+ config-section: 'ami-config'
+ typename: 'pluggable_modules.EventActionModule'
sipp-config:
connect-ami: 'True'
reactor-timeout: 30
fail-on-any: True
- start_callback_module: 'mwi_check'
- start_callback_method: 'walk_states'
+ stop-after-scenarios: False
+ stop_callback_module: 'mwi_check'
+ stop_callback_method: 'start_test'
test-iterations:
# We pass the initial registers the -aa flag then let them run for a second so they can get the
# 1-2 initial NOTIFY messages we don't care about. The MWI AMI updates are set to start after
@@ -46,12 +50,12 @@
'ordered-args': {'-aa'} }
- { 'key-args': {'scenario': 'bob-registers.xml', '-p': '5062'},
'ordered-args': {'-aa'} }
- # Combining this with the -m makes the scripts regex conditional on call_number
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-1.xml', '-p': '5061'} }
- - { 'key-args': {'scenario': 'bob-is-notified-1.xml', '-p': '5062'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-2.xml', '-p': '5061'} }
- - { 'key-args': {'scenario': 'bob-is-notified-2.xml', '-p': '5062'} }
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
diff --git a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
index f5378aa..28c4ded 100644
--- a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
+++ b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/mwi_check.py
@@ -6,36 +6,31 @@
sys.path.append("lib/python")
from twisted.internet import reactor
+from asterisk.scenario_iterator import singleIterator
LOGGER = logging.getLogger(__name__)
-mwis = [
- {'new': '2', 'old': '0'},
- {'new': '1', 'old': '1'},
- {'new': '0', 'old': '2'},
+mwiscenarios = [
+ {'Name': 'alice-is-notified-1.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-2.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-3.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'alice-is-notified-4.xml', 'port': '5061', 'target': '127.0.0.1'},
+ {'Name': 'done'}
]
-def walk_states(test_object, junk):
+mwis = [
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'2', 'OldMessages':'0'},
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'1', 'OldMessages':'1'},
+ {'Action': 'MWIUpdate', 'Mailbox': 'alice', 'NewMessages':'0', 'OldMessages':'2'},
+ {'Action': 'MWIDelete', 'Mailbox': 'alice'},
+ {'Action': 'UserEvent', 'UserEvent': 'testComplete'}
+]
- testami = test_object.ami[0]
- statedelay = 2
- for mwi in mwis:
- LOGGER.info("Sending MWI update. new: %s, old %s" %
- (mwi['new'],
- mwi['old']))
- message = {
- 'Action': 'MWIUpdate',
- 'Mailbox': 'alice',
- 'NewMessages': mwi['new'],
- 'OldMessages': mwi['old']
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
- statedelay += .25
+def start_test(test_object, junk):
+ LOGGER.info("Starting mwi_check")
+ testrunner = singleIterator(test_object, mwiscenarios, mwis)
+ testrunner.run(junk)
- # delete mailbox after walking states
- LOGGER.info("Deleting Mailbox")
- message = {
- 'Action': 'MWIDelete',
- 'Mailbox': 'alice',
- }
- reactor.callLater(statedelay, testami.sendMessage, message)
+def stop_test():
+ return
+
diff --git a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
index 5cb1150..14f9456 100644
--- a/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
+++ b/tests/channels/pjsip/subscriptions/mwi/unsolicited/mailbox_count_changes/test-config.yaml
@@ -1,5 +1,4 @@
testinfo:
- skip: 'See ASTERISK-30214'
summary: 'Ensures MWI bodies consist of accurate information'
description: |
"Unsolicited MWI notifications are sent to an endpoint as mailbox state updates. sipp
@@ -26,13 +25,18 @@
test-object:
config-section: sipp-config
typename: 'sipp.SIPpTestCase'
+ modules:
+ -
+ config-section: 'ami-config'
+ typename: 'pluggable_modules.EventActionModule'
sipp-config:
connect-ami: 'True'
reactor-timeout: 30
fail-on-any: True
- start_callback_module: 'mwi_check'
- start_callback_method: 'walk_states'
+ stop-after-scenarios: False
+ stop_callback_module: 'mwi_check'
+ stop_callback_method: 'start_test'
test-iterations:
# We pass the initial registers the -aa flag then let them run for a second so they can get the
# 1-2 initial NOTIFY messages we don't care about. The MWI AMI updates are set to start after
@@ -40,16 +44,12 @@
scenarios:
- { 'key-args': {'scenario': 'alice-registers.xml', '-p': '5061'},
'ordered-args': {'-aa'} }
- # Combining this with the -m makes the scripts regex conditional on call_number
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-1.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-2.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-3.xml', '-p': '5061'} }
- -
- scenarios:
- - { 'key-args': {'scenario': 'alice-is-notified-4.xml', '-p': '5061'} }
+ami-config:
+ -
+ ami-events:
+ conditions:
+ match:
+ Event: 'UserEvent'
+ UserEvent: 'testComplete'
+ count: 1
+ stop_test:
--
To view, visit https://gerrit.asterisk.org/c/testsuite/+/19261
To unsubscribe, or for help writing mail filters, visit https://gerrit.asterisk.org/settings
Gerrit-Project: testsuite
Gerrit-Branch: certified/18.9
Gerrit-Change-Id: Ic7f318bc953416ff8d8673ed4768fef264046fe3
Gerrit-Change-Number: 19261
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Bradeen <mbradeen at sangoma.com>
Gerrit-Reviewer: Friendly Automation
Gerrit-Reviewer: George Joseph <gjoseph at digium.com>
Gerrit-MessageType: merged
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-code-review/attachments/20220915/af150a72/attachment-0001.html>
More information about the asterisk-code-review
mailing list