[asterisk-dev] [Code Review]: Add test execution modes to the Asterisk Testsuite

Tue Sep 20 12:46:27 CDT 2011

> On Sept. 20, 2011, 12:27 p.m., Matthew Nicholson wrote:
> > /asterisk/trunk/README.txt, line 344
> > <https://reviewboard.asterisk.org/r/1447/diff/1/?file=20790#file20790line344>
> >
> >     Is there a way to use some sort of short hand for these extremely long condition names?
> 
> Paul Belanger wrote:
>     I would agree.  I like where we are going, however we are starting to make it complex for writing a test config file.  I would must rather see:
>     
>     reports:
>        Threads: yes
>        SIPDialogs: yes
>           Ignore: 'netconsole'
>           ...
>     
>     Then the test suite is responsible for building up the conditions.  At this point, I'm not sure why or how a Pre / Post check condition for Threads could be different or changed.

Nope.

We need the python module name (the thing after the first .) and the class name for the introspection to properly lookup the object to dynamically create.  The 'asterisk' is in there because our python system library path doesn't append that folder to it, and changing that would affect a lot of import statements throughout the code.

> On Sept. 20, 2011, 12:27 p.m., Matthew Nicholson wrote:
> > /asterisk/trunk/README.txt, line 355
> > <https://reviewboard.asterisk.org/r/1447/diff/1/?file=20790#file20790line355>
> >
> >     Some of these names are in camelCase while other names are all lowercase. We should make them all consistent. Unfortunately we started out with everything being lowercase (e.g. minversion), for longer field names that will be harder to read. We could also make them case insensitive, I don't know if that is an option though.

I agree, this is bad.  I went with camel case as that was in conjunction with the TestCase class, and (I think) is more in line with general python coding standards.  That isn't the case across the board, and a lot of our tests use K&R (or some derivative) C-style coding standards.

I'd be all for a consistent coding standard that specified this, and then do a refactoring to change them to all be the same.  There's plenty of tools that can do name changes very quickly and with little risk of error - if we decide on a consistent approach, I'll change them appropriately.

> On Sept. 20, 2011, 12:27 p.m., Matthew Nicholson wrote:
> > /asterisk/trunk/README.txt, lines 335-355
> > <https://reviewboard.asterisk.org/r/1447/diff/1/?file=20790#file20790line335>
> >
> >     Is there any reason that a precondition (or this specific precondition) would be used without the corresponding post condition? If not, it shouldn't be possible to specify the precondition separate from the postcondition. There should be some sort of pre and postcondition.

In this particular case, no, the pre-condition would most likely not be used without the corresponding post-condition - and in this particular case, you need the pre-condition for the post-condition to work.  That isn't true across the board for all of the test conditions, e.g., SipDialog - which has a different Pre and Post test condition but that are independent.

I've kept the relatedCondition in the post-test condition to not automatically define and create the condition it refers to as that would make the configuration of the pre-test condition difficult - for example, if all you needed was the following to define the ThreadPreTestCondition object, and you wanted to add ignoreThreads to the pre-condition, where would you?

name: 'asterisk.ThreadTestCondition.ThreadPostTestCondition'
type: 'Post'
ignoredThreads: # these only apply to the post condition
  - 'netconsole'
  - 'shaun_of_the_dead'
relatedCondition: 'asterisk.ThreadTestCondition.ThreadPreTestCondition'

In general, since there exists a global settings file that will handle the object creation - and individual tests need to only override those that they have specific settings for - I don't think the configuration setting burden is too onerous.

- mjordan

-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/1447/#review4398
-----------------------------------------------------------

On Sept. 19, 2011, 2:22 p.m., mjordan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/1447/
> -----------------------------------------------------------
> 
> (Updated Sept. 19, 2011, 2:22 p.m.)
> 
> 
> Review request for Asterisk Developers and Paul Belanger.
> 
> 
> Summary
> -------
> 
> This patch adds the ability for global 'modes' to be defined for the Asterisk test-suite.  Those settings in a mode can then be applied either at the base level in the runtests.py script, or by any test that inherits from TestClass.py.  As the global settings file makes use of the TestConfig class, all settings in the TestConfig class can theoretically be applied on a global scale - although since each test checks its dependencies independently, only the Test Conditions are currently useful.
> 
> One additional settings was added - exclude-tests.  This lets the global settings file explicitly call out tests (or subsets of tests) to not run without having to modify the tests.yaml file, and acts as a counterpart to the --test command line flag.
> 
> The only assumption made by this patch is that the global settings file is named 'test-config.yaml' and is located at the current run directory (the location of the runtests.py script).
> 
> 
> Diffs
> -----
> 
>   /asterisk/trunk/README.txt 2346 
>   /asterisk/trunk/lib/python/asterisk/TestCase.py 2346 
>   /asterisk/trunk/lib/python/asterisk/TestConfig.py 2346 
>   /asterisk/trunk/runtests.py 2346 
>   /asterisk/trunk/test-config.yaml PRE-CREATION 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_extensions/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_invalid_mailbox/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_invalid_password/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/authenticate_nominal/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_callback/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_delete/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_dialout/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_envelope/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_new_user/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_nominal/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/check_voicemail_reply/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/func_vmcount/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_external_notification/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_forwarding/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_forwarding_auto_urgent/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_nominal/test-config.yaml 2346 
>   /asterisk/trunk/tests/apps/voicemail/leave_voicemail_priority/test-config.yaml 2346 
> 
> Diff: https://reviewboard.asterisk.org/r/1447/diff
> 
> 
> Testing
> -------
> 
> Tested fast config mode, standard config mode, and pessimistic config mode.  Each worked as intended:
> * Fast config mode successfully excluded those tests with a reactor_timeout greater than one minute
> * Standard config mode applied no global settings
> * Pessimistic config mode applied the Test Conditions to all tests inheriting from TestCase.  Those tests that further defined test specific overrides applied those settings over the global conditions.
> 
> 
> Thanks,
> 
> mjordan
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20110920/f1c48e5d/attachment-0001.htm>