<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2172/18/9/_/styles/combined.css?spaceKey=AST&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/AST/Asterisk+Test+Suite+Documentation">Asterisk Test Suite Documentation</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~mjordan">Matt Jordan</a>
</h4>
<br/>
<h4>Changes (2)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
<tr><td class="diff-changed-lines" >h3. <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">Execute</span> <span class="diff-added-words"style="background-color: #dfd;">Executing</span> a set of tests by tag <br></td></tr>
<tr><td class="diff-unchanged" > <br>Many tests in the Asterisk Test Suite have tags that group them according to functionality that they test. The tags currently in use by all executable tests in the Test Suite can be determined using the _-L_ command line option: <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
<tr><td class="diff-changed-lines" >h3. <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">Execute</span> <span class="diff-added-words"style="background-color: #dfd;">Executing</span> a single test <br></td></tr>
<tr><td class="diff-unchanged" > <br>A single test can be executed using the _-t_ command line option. <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<div>
<ul>
<li><a href='#AsteriskTestSuiteDocumentation-AsteriskTesting'>Asterisk Testing</a></li>
<li><a href='#AsteriskTestSuiteDocumentation-TheAsteriskTestSuite'>The Asterisk Test Suite</a></li>
<ul>
<li><a href='#AsteriskTestSuiteDocumentation-TestExecution'>Test Execution</a></li>
<ul>
<li><a href='#AsteriskTestSuiteDocumentation-Executingalltests'>Executing all tests</a></li>
<li><a href='#AsteriskTestSuiteDocumentation-Executingpredefinedsetsoftests'>Executing pre-defined sets of tests</a></li>
<li><a href='#AsteriskTestSuiteDocumentation-Executingasetoftestsbytag'>Executing a set of tests by tag</a></li>
<li><a href='#AsteriskTestSuiteDocumentation-Executingasingletest'>Executing a single test</a></li>
</ul>
</ul>
<li><a href='#AsteriskTestSuiteDocumentation-AsteriskTestSuiteInformation'>Asterisk Test Suite Information</a></li>
<ul>
<li><a href='#AsteriskTestSuiteDocumentation-TestWriting'>Test Writing</a></li>
<li><a href='#AsteriskTestSuiteDocumentation-ReferenceInformation'>Reference Information</a></li>
</ul>
</ul></div>
<h1><a name="AsteriskTestSuiteDocumentation-AsteriskTesting"></a>Asterisk Testing</h1>
<p>Testing of Asterisk takes primarily three forms:</p>
<ul>
<li>Unit testing through the Asterisk Unit Test Framework</li>
<li>Functional testing through the Asterisk Test Suite</li>
<li>Formally defined system tests</li>
</ul>
<p>The Asterisk Unit Test framework exercises individual units within Asterisk in a 'white-box', 'bottom-up' approach, verifying that an individual unit maintains its interface contract with the rest of the codebase. The Asterisk Test Suite operates at the next higher level by exercising the functionality of entire modules within Asterisk and how they interact with the core of Asterisk. This is typically more of a 'black-box', 'top-down' approach, although since the typical writer of tests within the Asterisk Test Suite are developers, some knowledge of the modules under test typically occurs. Both of these tests are automated, and are executed on a frequent basis. Contrasting both of these approaches are System tests, which are typically manual and exercise the interaction of many modules in Asterisk together. System tests often occur in conjunction with initial releases of major branches and on a less frequent, periodic basis. The diagram below illustrates where each of these forms of testing fall within the larger context of a testing philosophy.</p>
<table width="100%">
<tr>
<td >
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 790px;"
usemap="#gliffy-map-19398799-1146"
src="/wiki/download/attachments/17793142/Test+Philosophy.png?version=2&modificationDate=1330526397876"
alt=""/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<p>Combined, the Asterisk Unit Test Framework and the Asterisk Test Suite allow the Asterisk development team to perform continuous integration, wherein changes are continually merged into the various supported branches and verified to not introduce major regressions. The unit tests and integration tests performed by these two frameworks guarantee that a basic level of functionality is not broken at any time by the introduction of a new feature or the fixing of a bug. Note that this does not mean that regressions do not occur between Asterisk versions, but that the risk associated with each check-in is reduced, and, as new tests are added, further minimized.</p>
<p>The Asterisk development team uses <a href="http://bamboo.asterisk.org/" class="external-link" rel="nofollow">Atlassian Bamboo</a> to perform the continuous integration tests.</p>
<h1><a name="AsteriskTestSuiteDocumentation-TheAsteriskTestSuite"></a>The Asterisk Test Suite</h1>
<p>The Asterisk Test Suite is an open source project that manages functional testing of Asterisk. It orchestrates instances of Asterisk along with various third party protocol injection applications, allowing a developer to write tests that exercise a wide variety of the functionality within Asterisk. While the Asterisk Test Suite is language agnostic, a set of libraries have been developed in Python and Lua to aid in test development. Currently, the Python libraries receive the majority of development attention, and most tests are written in Python.</p>
<h2><a name="AsteriskTestSuiteDocumentation-TestExecution"></a>Test Execution</h2>
<p>There are a number of ways the Asterisk Test Suite can be executed:</p>
<ul>
<li>Execute all tests</li>
<li>Execute a pre-defined set of tests</li>
<li>Execute a set of tests that share common tags</li>
<li>Execute a single test</li>
</ul>
<h3><a name="AsteriskTestSuiteDocumentation-Executingalltests"></a>Executing all tests</h3>
<p>Execute all tests in the Test Suite by running the run-tests.py script.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># ./runtests.py
Running tests for Asterisk SVN-trunk-r366462M
...
--> Running test 'tests/example' ...
Making sure Asterisk isn't running ...
Running ['tests/example/run-test'] ...</pre>
</div></div>
<div class='panelMacro'><table class='noteMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/warning.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Depending on permissions and the modules installed, you may need to run the runtests.py script with elevated permissions.</td></tr></table></div>
<h3><a name="AsteriskTestSuiteDocumentation-Executingpredefinedsetsoftests"></a>Executing pre-defined sets of tests</h3>
<p>Pre-defined sets of tests can be set up in the top level <em>test-config.yaml</em> configuration file. When the runtests.py script starts, it looks for the key in <em>global-settings/test-configuration</em>. The value specified for that key is used to look up test run specific settings. An example is shown below, where the <em>test-configuration</em> is set to <em>config-quick</em>.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># Global settings
global-settings:
# The active test configuration. The value must match a subsequent key
# in this file, which defines the global settings to apply to the test execution
# run.
test-configuration: config-quick
config-quick:</pre>
</div></div>
<p>A test configuration can exclude tests from a run by using the <em>exclude-tests</em> key. Each value under that key is a test that will <b>not</b> be run when the <em>runtests.py</em> script executes.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># Exclude some long-running tests
config-quick:
exclude-tests:
- 'authenticate_invalid_password'
- 'check_voicemail_callback'
- 'check_voicemail_delete'
- 'check_voicemail_dialout'
- 'check_voicemail_envelope'
- 'check_voicemail_new_user'
- 'check_voicemail_nominal'
- 'check_voicemail_reply'
- 'leave_voicemail_external_notification'
- 'leave_voicemail_nominal'
- 'gateway_g711_t38'
- 'gateway_mix1'
- 'gateway_mix2'
- 'gateway_mix3'
- 'gateway_mix4'
- 'gateway_native_t38'
- 'gateway_native_t38_ced'
- 'gateway_no_t38'
- 'gateway_t38_g711'
- 'gateway_timeout1'
- 'gateway_timeout2'
- 'gateway_timeout3'
- 'gateway_timeout4'
- 'gateway_timeout5'</pre>
</div></div>
<h3><a name="AsteriskTestSuiteDocumentation-Executingasetoftestsbytag"></a>Executing a set of tests by tag</h3>
<p>Many tests in the Asterisk Test Suite have tags that group them according to functionality that they test. The tags currently in use by all executable tests in the Test Suite can be determined using the <em>-L</em> command line option:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># ./runtests.py -L
Available test tags:
AGI AMI apps
bridge ccss CDR
chan_local chanspy confbridge
connected_line dial dialplan
dialplan_lua directory DTMF
fax fax_gateway fax_passthrough
features gosub GoSub
IAX incomplete macro
mixmonitor page parking
pickup queues redirecting
SIP subroutine transfer
voicemail</pre>
</div></div>
<p>All tests that have a tag can be executed using the <em>-g</em> command line option:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># ./runtests.py -g SIP</pre>
</div></div>
<p>Multiple tags can be specified as well, using multiple invocations of the <em>-g</em> command line option. A test must satisfy each tag specified in order to be executed.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># ./runtests.py -g SIP -g CDR</pre>
</div></div>
<h3><a name="AsteriskTestSuiteDocumentation-Executingasingletest"></a>Executing a single test</h3>
<p>A single test can be executed using the <em>-t</em> command line option.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false"># ./runtests.py -t tests/dialplan</pre>
</div></div>
<h1><a name="AsteriskTestSuiteDocumentation-AsteriskTestSuiteInformation"></a>Asterisk Test Suite Information</h1>
<h2><a name="AsteriskTestSuiteDocumentation-TestWriting"></a>Test Writing</h2>
<p><a href="/wiki/display/AST/Writing+a+Python+Test" title="Writing a Python Test">Writing a Python Test</a></p>
<h2><a name="AsteriskTestSuiteDocumentation-ReferenceInformation"></a>Reference Information</h2>
<p><a href="/wiki/display/AST/Python+Modules" title="Python Modules">Python Modules</a></p>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=AST">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/AST/Asterisk+Test+Suite+Documentation">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=17793142&revisedVersion=8&originalVersion=7">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/AST/Asterisk+Test+Suite+Documentation?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>