[asterisk-dev] [Code Review] 3489: testsuite: Improve logging

[Matt Jordan]

> On May 8, 2014, 2:55 p.m., Mark Michelson wrote:
> > /asterisk/trunk/runtests.py, lines 138-139
> > <https://reviewboard.asterisk.org/r/3489/diff/1/?file=58017#file58017line138>
> >
> >     Assert that run_num > 0

It actually has to be here.

(1) i = 1 initially
(2) We then increment i for each run_%d directory we find
(3) The final value of run_num is (i - 1)

Since i is never decremented and is computed locally in this function, the lowest value run_num can have is 0.


- Matt


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviewboard.asterisk.org/r/3489/#review11858
-----------------------------------------------------------


On April 28, 2014, 3:18 p.m., Matt Jordan wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviewboard.asterisk.org/r/3489/
> -----------------------------------------------------------
> 
> (Updated April 28, 2014, 3:18 p.m.)
> 
> 
> Review request for Asterisk Developers.
> 
> 
> Repository: testsuite
> 
> 
> Description
> -------
> 
> This patch changes up the logging in the Asterisk Test Suite. It dramatically reduces the amount of logging done in the logs/ directory by default (although this can always be bumped up), and it redirects the vast majority of the test suite logs to the actual sandboxed test run directories themselves.
> 
> When we first added Python logging (it didn't always used to be there!) there were far fewer tests. Having all tests dump into the same log files actually felt like a benefit, since during test development (or when developing many tests) it made it easier to debug test failures. Over time, however, this approach has run into a number of problems:
> (1) Many of the tests are much 'chattier' than they used to be. Having twisted, starpy, websockets, requests, and other libraries dump information has greatly increased the number of log messages. Not to mention logging out every SIPp screen received...
> (2) There are a lot more tests than there were when this was added.
> (3) Tailing a log file is not always the best way to debug. Sometimes you have to search through an entire log file for one run. Finding the error becomes problematic when your editor of choice chokes on the size of the file.
> 
> Hence, this patch.
> 
> Logging now works in the following way:
> (1) The test_case.TestCase class now always sets up the logging. The logging set up done by test_runner was removed (as it only logged out a few messages before an instance of TestCase was created anyway).
> (2) A global logger can still be configured in logger.conf. It now only sets up a logger of INFO messages and higher. This allows a test executor to watch which tests are being run, without getting spammed. During test development, the log message level can be increased to DEBUG.
> (3) TestCase now places the Asterisk directories created by the test execution in a further subfolder, run_N (where N increases with each execution of that test). Where before you might have:
> 
> tests/my_test/ast1
> tests/my_test/ast2
> tests/my_test/ast3
> tests/my_test/ast4
> 
> You will now have:
> 
> tests/my_test/run_1/ast1
> tests/my_test/run_1/ast2
> tests/my_test/run_2/ast1
> tests/my_test/run_2/ast2
> 
> This lets you determine which Asterisk instances belong together, and also makes it possible for only the erroring test run to be archived when a test fails (as opposed to every Asterisk directory)
> (4) TestCase now creates a DEBUG and INFO log file(s) in the run directory. These contain the full Asterisk logs for the test run.
> 
> 
> Diffs
> -----
> 
>   /asterisk/trunk/runtests.py 5002 
>   /asterisk/trunk/logger.conf 5002 
>   /asterisk/trunk/lib/python/asterisk/test_runner.py 5002 
>   /asterisk/trunk/lib/python/asterisk/test_case.py 5002 
>   /asterisk/trunk/lib/python/asterisk/asterisk.py 5002 
> 
> Diff: https://reviewboard.asterisk.org/r/3489/diff/
> 
> 
> Testing
> -------
> 
> I've actually been running an instance of the test suite with this set up for several months, using it for development of several tests and generally tweaking it. I finally felt like it was "good enough".
> 
> Tested:
> * Failing tests archive appropriately
> * Crashing Asterisk gets archived
> * Global log file is still created with expected message levels
> * Test specific log files are created appropriately with expected message levels
> * Tested a pluggable module based test (pbx/dialplan) and a 'traditional' Python test (channels/SIP/sip_hold)
> 
> 
> Thanks,
> 
> Matt Jordan
> 
>

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.digium.com/pipermail/asterisk-dev/attachments/20140513/e8f90611/attachment.html>