[asterisk-dev] [Code Review] Unit Test Framework

[Matthew Nicholson]

> On 2009-12-16 15:49:39, Matthew Nicholson wrote:
> > /trunk/include/asterisk/test.h, lines 105-109
> > <https://reviewboard.asterisk.org/r/447/diff/1/?file=7560#file7560line105>
> >
> >     Should there also be an AST_TEST_XFAIL result state for expected failures?
> >     
> >     And perhaps also an AST_TEST_ERROR state?  This would be used in the event that there was an error running a test that is unrelated to what the test is testing.  Things such as a memory allocation error.
> 
> David Vossel wrote:
>     If a test is expected to fail, would that not just be considered a pass?  Why would we need to differentiate the two?
> 
> Russell Bryant wrote:
>     Regarding an expected failure, can you provide a use case?
>     
>     Also, if a memory allocation error occurs, I would call that a test failure.  I would them expect the text that accompanies the error to indicate that the failure was due to memory allocation.  I don't think it needs to be any more complicated than that.

Expected failure would be used for known issues that we do not yet have a fix for, but we can reproduce with a test.  Until the fix is added, the test is expected to fail, but it is not a regression.  For example, say there was a rare SDP parsing bug that made 1 out of ever 6 million calls on a system fail.  We know how to reproduce it with a test, but the fix is complicated.  This test could be marked as expected to fail until the fix is added.

As for the error state, there is at least one other case I can think of other then memory allocation failures.  If a test needed to generate a sound file (or some other file), to then feed to the code it was testing, but for some reason it could not write to disk (disk full, permissions errors), the test would fail.  This type of failure is not really a failure of the code being tested, it is a failure of the code doing the testing.  I think distinguishing between these type of failures may be useful when we have 500 tests and 100 of them fail.


- Matthew


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/447/#review1311
-----------------------------------------------------------


On 2009-12-16 19:26:04, David Vossel wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/447/
> -----------------------------------------------------------
> 
> (Updated 2009-12-16 19:26:04)
> 
> 
> Review request for Asterisk Developers and Russell Bryant.
> 
> 
> Summary
> -------
> 
> The Unit Test Framework is a new API that manages registration and execution of unit tests in Asterisk with the purpose of verifying the operation of C functions.
> 
> The Framework consists of a single test manager accompanied by a list of registered test functions defined within the code.  A test is defined, registered, and unregistered from the framework using a set of macros which allow the test code to only be compiled within asterisk when the TEST_FRAMEWORK flag is enabled in menuselect.  This allows the test code to exist in the same file as the C functions it intends to verify.  Registered tests may be viewed and executed via a set of new CLI commands.  CLI commands are also present for generating and exporting test results into xml and txt formats.
> 
> For more information and use cases please refer to the documentation provided at the beginning of the test.h file.
> 
> 
> Diffs
> -----
> 
>   /trunk/include/asterisk/_private.h 235408 
>   /trunk/build_tools/cflags-devmode.xml 235408 
>   /trunk/include/asterisk/test.h PRE-CREATION 
>   /trunk/main/asterisk.c 235408 
>   /trunk/main/test.c PRE-CREATION 
>   /trunk/tests/test_heap.c 235408 
> 
> Diff: https://reviewboard.asterisk.org/r/447/diff
> 
> 
> Testing
> -------
> 
> test_heap.c has been modified to take advantage of the Test Framework.  I have executed and generated reports for these tests. 
> 
> 
> Thanks,
> 
> David
> 
>