<html>
<body>
<div style="font-family: Verdana, Arial, Helvetica, Sans-Serif;">
<table bgcolor="#f9f3c9" width="100%" cellpadding="8" style="border: 1px #c9c399 solid;">
<tr>
<td>
This is an automatically generated e-mail. To reply, visit:
<a href="https://reviewboard.asterisk.org/r/3489/">https://reviewboard.asterisk.org/r/3489/</a>
</td>
</tr>
</table>
<br />
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<p style="margin-top: 0;">On May 8th, 2014, 2:55 p.m. CDT, <b>Mark Michelson</b> wrote:</p>
<blockquote style="margin-left: 1em; border-left: 2px solid #d0d0d0; padding-left: 10px;">
<table width="100%" border="0" bgcolor="white" style="border: 1px solid #C0C0C0; border-collapse: collapse; margin: 2px padding: 2px;">
<thead>
<tr>
<th colspan="4" bgcolor="#F0F0F0" style="border-bottom: 1px solid #C0C0C0; font-size: 9pt; padding: 4px 8px; text-align: left;">
<a href="https://reviewboard.asterisk.org/r/3489/diff/1/?file=58017#file58017line138" style="color: black; font-weight: bold; text-decoration: underline;">/asterisk/trunk/runtests.py</a>
<span style="font-weight: normal;">
(Diff revision 1)
</span>
</th>
</tr>
</thead>
<tbody style="background-color: #e4d9cb; padding: 4px 8px; text-align: center;">
<tr>
<td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def __archive_core_dumps(self, core_dumps):</pre></td>
<td colspan="2"><pre style="font-size: 8pt; line-height: 140%; margin: 0; ">def _archive_core_dumps(self, core_dumps):</pre></td>
</tr>
</tbody>
<tbody>
<tr>
<th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">135</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">dest_dir</span> <span class="o">=</span> <span class="s">"./logs/</span><span class="si">%s</span><span class="s">/ast</span><span class="si">%d</span><span class="s">/var/log/asterisk"</span> <span class="o">%</span> <span class="p">(</span><span class="bp">self</span><span class="o">.</span><span class="n">test_name</span><span class="o">.</span><span class="n">lstrip</span><span class="p">(</span><span class="s">"tests/"</span><span class="p">),</span> <span class="n">i</span><span class="p">)</span></pre></td>
<th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">137</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">run_num</span> <span class="o">=</span> <span class="n">i</span> <span class="o">-</span> <span class="mi">1</span></pre></td>
</tr>
<tr>
<th bgcolor="#e9eaa8" style="border-right: 1px solid #C0C0C0;" align="right"><font size="2">136</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="sd">""" Only archive the logs if we havent archived it for this test run yet """</span></pre></td>
<th bgcolor="#e9eaa8" style="border-left: 1px solid #C0C0C0; border-right: 1px solid #C0C0C0;" align="right"><font size="2">138</font></th>
<td bgcolor="#fdfebc" width="50%"><pre style="font-size: 8pt; line-height: 140%; margin: 0; "> <span class="n">run_dir</span> <span class="o">=</span> <span class="n">os</span><span class="o">.</span><span class="n">path</span><span class="o">.</span><span class="n">join</span><span class="p">(</span><span class="n">test_run_dir</span><span class="p">,</span> <span class="s">'run_</span><span class="si">%d</span><span class="s">'</span> <span class="o">%</span> <span class="n">run_num</span><span class="p">)</span></pre></td>
</tr>
</tbody>
</table>
<pre style="white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">Assert that run_num > 0</pre>
</blockquote>
</blockquote>
<pre style="margin-left: 1em; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">It actually has to be here.
(1) i = 1 initially
(2) We then increment i for each run_%d directory we find
(3) The final value of run_num is (i - 1)
Since i is never decremented and is computed locally in this function, the lowest value run_num can have is 0.
</pre>
<br />
<p>- Matt</p>
<br />
<p>On April 28th, 2014, 3:18 p.m. CDT, Matt Jordan wrote:</p>
<table bgcolor="#fefadf" width="100%" cellspacing="0" cellpadding="8" style="background-image: url('https://reviewboard.asterisk.org/static/rb/images/review_request_box_top_bg.ab6f3b1072c9.png'); background-position: left top; background-repeat: repeat-x; border: 1px black solid;">
<tr>
<td>
<div>Review request for Asterisk Developers.</div>
<div>By Matt Jordan.</div>
<p style="color: grey;"><i>Updated April 28, 2014, 3:18 p.m.</i></p>
<div style="margin-top: 1.5em;">
<b style="color: #575012; font-size: 10pt;">Repository: </b>
testsuite
</div>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Description </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">This patch changes up the logging in the Asterisk Test Suite. It dramatically reduces the amount of logging done in the logs/ directory by default (although this can always be bumped up), and it redirects the vast majority of the test suite logs to the actual sandboxed test run directories themselves.
When we first added Python logging (it didn't always used to be there!) there were far fewer tests. Having all tests dump into the same log files actually felt like a benefit, since during test development (or when developing many tests) it made it easier to debug test failures. Over time, however, this approach has run into a number of problems:
(1) Many of the tests are much 'chattier' than they used to be. Having twisted, starpy, websockets, requests, and other libraries dump information has greatly increased the number of log messages. Not to mention logging out every SIPp screen received...
(2) There are a lot more tests than there were when this was added.
(3) Tailing a log file is not always the best way to debug. Sometimes you have to search through an entire log file for one run. Finding the error becomes problematic when your editor of choice chokes on the size of the file.
Hence, this patch.
Logging now works in the following way:
(1) The test_case.TestCase class now always sets up the logging. The logging set up done by test_runner was removed (as it only logged out a few messages before an instance of TestCase was created anyway).
(2) A global logger can still be configured in logger.conf. It now only sets up a logger of INFO messages and higher. This allows a test executor to watch which tests are being run, without getting spammed. During test development, the log message level can be increased to DEBUG.
(3) TestCase now places the Asterisk directories created by the test execution in a further subfolder, run_N (where N increases with each execution of that test). Where before you might have:
tests/my_test/ast1
tests/my_test/ast2
tests/my_test/ast3
tests/my_test/ast4
You will now have:
tests/my_test/run_1/ast1
tests/my_test/run_1/ast2
tests/my_test/run_2/ast1
tests/my_test/run_2/ast2
This lets you determine which Asterisk instances belong together, and also makes it possible for only the erroring test run to be archived when a test fails (as opposed to every Asterisk directory)
(4) TestCase now creates a DEBUG and INFO log file(s) in the run directory. These contain the full Asterisk logs for the test run.
</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Testing </h1>
<table width="100%" bgcolor="#ffffff" cellspacing="0" cellpadding="10" style="border: 1px solid #b8b5a0">
<tr>
<td>
<pre style="margin: 0; padding: 0; white-space: pre-wrap; white-space: -moz-pre-wrap; white-space: -pre-wrap; white-space: -o-pre-wrap; word-wrap: break-word;">I've actually been running an instance of the test suite with this set up for several months, using it for development of several tests and generally tweaking it. I finally felt like it was "good enough".
Tested:
* Failing tests archive appropriately
* Crashing Asterisk gets archived
* Global log file is still created with expected message levels
* Test specific log files are created appropriately with expected message levels
* Tested a pluggable module based test (pbx/dialplan) and a 'traditional' Python test (channels/SIP/sip_hold)</pre>
</td>
</tr>
</table>
<h1 style="color: #575012; font-size: 10pt; margin-top: 1.5em;">Diffs</b> </h1>
<ul style="margin-left: 3em; padding-left: 0;">
<li>/asterisk/trunk/runtests.py <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/logger.conf <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/test_runner.py <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/test_case.py <span style="color: grey">(5002)</span></li>
<li>/asterisk/trunk/lib/python/asterisk/asterisk.py <span style="color: grey">(5002)</span></li>
</ul>
<p><a href="https://reviewboard.asterisk.org/r/3489/diff/" style="margin-left: 3em;">View Diff</a></p>
</td>
</tr>
</table>
</div>
</body>
</html>