<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2171/18/9/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Testing+and+Asterisk+SCF">Testing and Asterisk SCF</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~beagles">Brent Eagles</a>
</h4>
<br/>
<h4>Changes (8)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{panel} <br> <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">From conception, testing been a major part of Asterisk SCF development. The library of tests is automatically run as part of the merge-triggered continuous integration process, and the use of standardized output via the boost::test framework facilitates analysis of tests results. Testing is approached at multiple levels, including low-level unit testing, higher level testing at the component level and full scale system testing with automated call simulation, online reporting and post-mortem analysis support. <br> <br></td></tr>
<tr><td class="diff-unchanged" >{panel:title=Testing and Asterisk SCF - Part of the Process|titleBGColor=#2022FF|bgColor=#7788FF} <br>* {color:white}Each component is expected to have a test suite.{color} <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">* {color:white}Tests are run automatically as part of the automated build/test process.{color} <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">* {color:white}The same tests are run automatically as part of the automated build/test process are easily run by an individual developer.{color} <br></td></tr>
<tr><td class="diff-unchanged" >* {color:white}Test-ability{color} {color:white}{*}is{*}{color} {color:white}a design consideration.{color} <br>* {color:white}Tests are included in code reviews.{color} <br>{panel} <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">_This is one of those slides that can almost be read with a bit of lingo to make things complete sentences_ <br></td></tr>
<tr><td class="diff-unchanged" > <br>{panel:title=Testing and Asterisk SCF - Test Types|titleBGColor=#2022FF|bgColor=#7788FF} <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > <br>{panel} <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">_ditto_ <br></td></tr>
<tr><td class="diff-unchanged" > <br>{panel:title=Testing and Asterisk SCF - Higher Level Testing|titleBGColor=#2022FF|bgColor=#7788FF} <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{panel} <br> <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Experience tells us that useful testing of distributed, highly concurrent systems with sufficient load levels is not easy. Configuration and setup is daunting enough, but the analysis of test results to detect and track down errors can be a nightmare. The Asterisk SCF System test framework provides customizable mechanisms for test configuration and setup and tools for managing and processing test results. This includes configuration of stress tests with useful load levels as well as collecting performance data for analysis. The test framework performs collection and analysis on *what actually happened* including wire traces and call graphs. While the test framework does not find the problem for you, it includes best-bank-for-the-buck tools to help you identify issues and track them down to their source. Another notable feature is the ability to produce documentation containing a description of the tests currently being performed, including details such as configuration and commands to be executed. Besides being useful collateral when it comes time to attempt reproducing a bug, it can be used to audit the system tests! <br> <br></td></tr>
<tr><td class="diff-unchanged" >{color:white} <br>{panel:title=Testing and Asterisk SCF - More About Plugins|titleBGColor=#2022FF|bgColor=#7788FF} <br></td></tr>
<tr><td class="diff-changed-lines" >* Plugin <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">architecture</span> <span class="diff-added-words"style="background-color: #dfd;">API,</span> with plugins for: <br></td></tr>
<tr><td class="diff-unchanged" >| configuration | component life-cycle control | <br>| fail-over simulation | external device simulation (SIP) | <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{color} <br> <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">The test framework provides a plugin API and includes some essential plugins. Interaction with external commands and processes are actually implemented as plugins so the API is under continuous use (eat your own dog food). Plugins to support new functionality including integrating additional external tools are straightforward to implement. <br> <br>A few of the current plugins deserve special mention. The PROTOS plugin incorporates the independently developed PROTOS test suite. Focusing on INVITE messages, PROTOS includes risk oriented tests aimed at exposing vulnerabilities and flaws in compliance. The SIPp pluging provides the ability to create arbitrary SIP messages which is so essential for system testing a communications system that includes a SIP session gateway. Last but certainly not least is the Wireshark plugin which is one of the tools for providing that much needed analysis of test results. <br> <br></td></tr>
<tr><td class="diff-unchanged" >{color:white} <br>{panel:title=Testing and Asterisk SCF - Output Examples|titleBGColor=#2022FF|bgColor=#7788FF} <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Yes, this is written in a very informal, conversational voice. It was easier getting the info out this way.</td></tr></table></div>
<h3><a name="TestingandAsteriskSCF-SlideSuggestions"></a>Slide Suggestions</h3>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li><font color="white">Testing is an integral part of development and the process.</font></li>
<li><font color="white">Builds and running of tests are automated.</font></li>
<li><font color="white">boost::test test framework.</font></li>
<li><font color="white">Different levels of testing.</font>
<ul>
<li><font color="white">Unit</font></li>
<li><font color="white">Component</font></li>
<li><font color="white">System</font></li>
</ul>
</li>
</ul>
</div></div>
<p>From conception, testing been a major part of Asterisk SCF development. The library of tests is automatically run as part of the merge-triggered continuous integration process, and the use of standardized output via the boost::test framework facilitates analysis of tests results. Testing is approached at multiple levels, including low-level unit testing, higher level testing at the component level and full scale system testing with automated call simulation, online reporting and post-mortem analysis support.</p>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF - Part of the Process</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li><font color="white">Each component is expected to have a test suite.</font></li>
<li><font color="white">The same tests are run automatically as part of the automated build/test process are easily run by an individual developer.</font></li>
<li><font color="white">Test-ability</font> <font color="white"><b>is</b></font> <font color="white">a design consideration.</font></li>
<li><font color="white">Tests are included in code reviews.</font></li>
</ul>
</div></div>
<p><em>This is one of those slides that can almost be read with a bit of lingo to make things complete sentences</em></p>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF - Test Types</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li><font color="white">Unit tests are typically in standalone executables and focus on small sections of high risk or fundamental code.</font></li>
<li><font color="white">Component tests load the component using a real-world configuration along with other supporting component and usually contain:</font>
<ul>
<li><font color="white">functional tests</font></li>
<li><font color="white">unit tests where mock objects would be required but are impractical</font></li>
<li><font color="white">not-quite-functional-or-unit tests that are small and focused but involve strings of components in contrived, artificial situations</font></li>
</ul>
</li>
<li><font color="white">Some components are impractical to test with unit and even component tests (e.g. SIP) This component is most vigorously tested at the system level which brings us to..</font></li>
</ul>
</div></div>
<p><em>ditto</em></p>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF - Higher Level Testing</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li><font color="white">Useful automated testing of a distributed, interactive, concurrent system != trivial.</font></li>
<li><font color="white">Enter the Asterisk SCF System Testing Framework</font>
<ul>
<li><font color="white">Heavy lifting for performance and stress testing as well as full system functional testing.</font></li>
<li><font color="white">Facilitating analysis is a primary design goal.</font></li>
<li><font color="white">Automatically generates detailed test specifications including commands and configuration. </font></li>
<li><font color="white">Implemented in Python and coordinates tests across multiple platforms. </font></li>
<li><font color="white">Can be run standalone in your own development environment. </font></li>
</ul>
</li>
</ul>
</div></div>
<p>Experience tells us that useful testing of distributed, highly concurrent systems with sufficient load levels is not easy. Configuration and setup is daunting enough, but the analysis of test results to detect and track down errors can be a nightmare. The Asterisk SCF System test framework provides customizable mechanisms for test configuration and setup and tools for managing and processing test results. This includes configuration of stress tests with useful load levels as well as collecting performance data for analysis. The test framework performs collection and analysis on <b>what actually happened</b> including wire traces and call graphs. While the test framework does not find the problem for you, it includes best-bank-for-the-buck tools to help you identify issues and track them down to their source. Another notable feature is the ability to produce documentation containing a description of the tests currently being performed, including details such as configuration and commands to be executed. Besides being useful collateral when it comes time to attempt reproducing a bug, it can be used to audit the system tests!</p>
<p><font color="white"><br/>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF - More About Plugins</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li>Plugin API, with plugins for:
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> configuration </td>
<td class='confluenceTd'> component life-cycle control </td>
</tr>
<tr>
<td class='confluenceTd'> fail-over simulation </td>
<td class='confluenceTd'> external device simulation (SIP) </td>
</tr>
<tr>
<td class='confluenceTd'> remote control </td>
<td class='confluenceTd'> system and post-mortem analysis </td>
</tr>
</tbody></table>
</div>
</li>
<li>Plugin spotlight
<ul>
<li>PROTOS - independently developed INVITE focused torture test</li>
<li>SIPp - automated, configurable simulation of call scenarios</li>
<li>Wireshark - on the wire logging of SIP, RTP, T38, etc. and generation of call graphs</li>
</ul>
</li>
</ul>
</div></div></font></p>
<p>The test framework provides a plugin API and includes some essential plugins. Interaction with external commands and processes are actually implemented as plugins so the API is under continuous use (eat your own dog food). Plugins to support new functionality including integrating additional external tools are straightforward to implement.</p>
<p>A few of the current plugins deserve special mention. The PROTOS plugin incorporates the independently developed PROTOS test suite. Focusing on INVITE messages, PROTOS includes risk oriented tests aimed at exposing vulnerabilities and flaws in compliance. The SIPp pluging provides the ability to create arbitrary SIP messages which is so essential for system testing a communications system that includes a SIP session gateway. Last but certainly not least is the Wireshark plugin which is one of the tools for providing that much needed analysis of test results.</p>
<p><font color="white"><br/>
<div class="panel" style="background-color: #7788FF;border-width: 1px;"><div class="panelHeader" style="border-bottom-width: 1px;background-color: #2022FF;"><b>Testing and Asterisk SCF - Output Examples</b></div><div class="panelContent" style="background-color: #7788FF;">
<ul>
<li><em>insert SIP call graph</em></li>
<li><em>screen shot of build/run artifact or something similar.. has to be cool!</em></li>
</ul>
</div></div></font></p>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=TOP">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Testing+and+Asterisk+SCF">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=19006242&revisedVersion=7&originalVersion=6">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Testing+and+Asterisk+SCF?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>