<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2166/4/9/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">Media Operations</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~kpfleming">Kevin P. Fleming</a>
</h4>
<div id="versionComment">
<b>Comment:</b>
first batch of edits related to CR-ASTSCF-138<br />
</div>
<br/>
<h4>Changes (21)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >h1. Introduction <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">In Asterisk SCF, a "media operation" refers to a process by which media is manipulated. Common media operations would be transcoding, jitter buffering, volume adjustment, pitch shifting, and potentially media injection. What follows is a discussion of how media operations fit into the architecture of Asterisk SCF, and how they can be used. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">In Asterisk SCF, a "media operation" refers to a service which manipulates media. Commonly used types of media operations include transcoding (codec/format conversion), transrating (sample rate conversion), jitter removal, volume adjustment, pitch shifting, tone detection and generation, and more. What follows is a discussion of how media operations fit into the architecture of Asterisk SCF, and how they can be used. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Without media operations, Asterisk SCF's media handling is straightforward, but not very effective. Each session is equipped with media stream sources and sinks. A stream source is where media comes from, and a stream sink is where media is sent. The bridging component can connect a source from one session to a sink of another session in order to connect media from the sessions together. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Without such media operations, Asterisk SCF's media handling would be straightforward, but not very effective. Each media stream on a session provides a {{source}} and a {{sink}}. A stream source provides a stream of media frames, and a stream sink accepts a stream of media frames. The {{source}} from one media stream can be connected to the {{sink}} of another media stream (assuming their formats are compatible) in order to connect the media flow between those streams; the bridging service does this for simple two-session bridges, for example. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-changed-lines" >h1. <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">Classification</span> <span class="diff-added-words"style="background-color: #dfd;">Categories</span> of media operations <br></td></tr>
<tr><td class="diff-unchanged" > <br>Media operations can be categorized as follows: <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">* Transforming: Transforming operations are those where the incoming and outgoing media types are different. Typically this will be the only function of the media operation. The change to the media may be a format change or a change to the parameters of the format. Examples of transforming media operations are transcoding and resampling. Usually transforming operations are not requested by choice, but rather they are required in order for media to be able to flow properly. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">* Transforming: Transforming operations modify incoming media frames into an different format before sending them onwards. The change to the media may be a format change or a change to the parameters of the format. Examples of transforming media operations are transcoding and transrating. Transforming operations are not usually put into a media stream path by choice (as a result of configuration, for example), but rather they are put there in order to allow two media streams of incompatible formats to be able to exchange media frames. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">* Adjusting. Adjusting media operations are those that make a change to the media without changing its compatibility in any way. Examples of adjusting media operations are jitter buffering and volume adjusting. Adjusting media operations are not required for media to pass, but they can be requested to enhance the user experience. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">* Adjusting: Adjusting operations are those that make a change to the media without changing its compatibility in any way. Examples of adjusting media operations are jitter buffering and volume adjusting. Adjusting media operations will frequently be put into a media stream path by choice (as a result of configuration, for example) in order to enhance the user experience, but may also be required in some situations to complete an otherwise *transforming* media stream path. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h1. Use of media operations <br> <br></td></tr>
<tr><td class="diff-changed-lines" >Media operations are valid to use anywhere that <span class="diff-added-words"style="background-color: #dfd;">stream-oriented</span> media exists. In practice, this means that there are two levels at which media operations may be used: <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-changed-lines" >* <span class="diff-added-words"style="background-color: #dfd;">Session:</span> Session <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">level. Session</span> level media operations will affect media to <span class="diff-changed-words"><span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">and/</span>or</span> from a specific session. This can be useful for operations that should not affect all parties in a bridged call. If a conference participant has hearing issues, then it may be good to place a volume adjustment media operation for media going to his session. <br></td></tr>
<tr><td class="diff-unchanged" > <br>* Bridge level. Bridge level media operations will affect media to and/or from all sessions involved in a particular bridge. This type will be more rare. <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >* Hooks. Session creation hooks and bridge creation hooks are ways to dynamically insert media operations for a session or bridge. <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">h1. Media operation slice <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">h1. Media Operation Slice definitions <br></td></tr>
<tr><td class="diff-unchanged" > <br>{code} <br></td></tr>
<tr><td class="diff-unchanged" >interface MediaOperation <br>{ <br></td></tr>
<tr><td class="diff-changed-lines" >AsteriskSCF::Media::V1::StreamSource* <span class="diff-changed-words">getSource<span class="diff-added-chars"style="background-color: #dfd;">()</span>;</span> <br></td></tr>
<tr><td class="diff-changed-lines" >AsteriskSCF::Media::V1::StreamSink* <span class="diff-changed-words">getSink<span class="diff-added-chars"style="background-color: #dfd;">()</span>;</span> <br></td></tr>
<tr><td class="diff-unchanged" > void destroy(); <br>}; <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{ <br> /** <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> * Create a new instance of a media operation. <br> * @param inSource optional source of media that will enter the operation <br> * @param inSink optional sink of media that the operation will send to <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> * Create an instance of a MediaOperation. <br> * @param source if provided, will be used only to determine the media format that will <br> * supplied to the desired MediaOperation <br> * @param sink if provided, will be used only to determine the media format that the <br> * desired MediaOperation must produce <br></td></tr>
<tr><td class="diff-unchanged" > */ <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">MediaOperation<span class="diff-added-chars"style="background-color: #dfd;">*</span></span> createMediaOperation( <br></td></tr>
<tr><td class="diff-unchanged" > AsteriskSCF::Media::V1::StreamSource* source, <br> AsteriskSCF::Media::V1::StreamSink* sink); <br>}; <br></td></tr>
<tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">When a session or bridge wishes to make use of a media operation, it can find the corresponding {{MediaOperationFactory}} using the service locator. From there, every instance of a {{MediaOperation}} can be requested as necessary. A {{MediaOperation}} consists of a media source and sink pairing. If more sources and sinks are required, then more {{MediaOperations}} can be requested from the {{MediaOperationFactory}}. The optional {{source}} and {{sink}} parameters may be provided to {{createMediaOperation}} in order to have the {{MediaOperationFactory}} attempt to create translation paths between formats if necessary/desired. The formats of media supported by a {{MediaOperation}} can be gleaned from getting the source and sink. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">When a media operation is needed to transform a media stream, an Asterisk SCF component can locate a suitable {{MediaOperationFactory}} using the Service Locator, and specifying the source and sink formats that the desired {{MediaOperation}} must support. If a suitable {{MediaOperationFactory}} is located, it can be used to construct {{MediaOperation}} instances, each of which can be used to handle a single media stream. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Some {{MediaOperationFactory}} services will only be able to instantiate a single type of {{MediaOperation}}, that support a single source format and a single sink format. Others will be able to construct a variety of {{MediaOperation}} instances. <br> <br>For {{MediaOperationFactory}} services that can provide a variety of {{MediaOperation}} instances, in order to describe to the {{MediaOperationFactory}} the *specific* source and sink formats that are needed, the {{createMediaOperation}} operation accepts a {{source}} parameter and a {{sink}} parameter. The {{MediaOperationFactory}} will learn from the source and sink what formats they support, and will then attempt to instantiate a {{MediaOperation}} that can transform the media from the source's format to the sink's format. <br> <br></td></tr>
<tr><td class="diff-unchanged" >h1. Dealing with formats <br> <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>This is a work in progress.</td></tr></table></div>
<h1><a name="MediaOperations-Introduction"></a>Introduction</h1>
<p>In Asterisk SCF, a "media operation" refers to a service which manipulates media. Commonly used types of media operations include transcoding (codec/format conversion), transrating (sample rate conversion), jitter removal, volume adjustment, pitch shifting, tone detection and generation, and more. What follows is a discussion of how media operations fit into the architecture of Asterisk SCF, and how they can be used.</p>
<p>Without such media operations, Asterisk SCF's media handling would be straightforward, but not very effective. Each media stream on a session provides a <tt>source</tt> and a <tt>sink</tt>. A stream source provides a stream of media frames, and a stream sink accepts a stream of media frames. The <tt>source</tt> from one media stream can be connected to the <tt>sink</tt> of another media stream (assuming their formats are compatible) in order to connect the media flow between those streams; the bridging service does this for simple two-session bridges, for example.</p>
<h1><a name="MediaOperations-Categoriesofmediaoperations"></a>Categories of media operations</h1>
<p>Media operations can be categorized as follows:</p>
<ul>
<li>Transforming: Transforming operations modify incoming media frames into an different format before sending them onwards. The change to the media may be a format change or a change to the parameters of the format. Examples of transforming media operations are transcoding and transrating. Transforming operations are not usually put into a media stream path by choice (as a result of configuration, for example), but rather they are put there in order to allow two media streams of incompatible formats to be able to exchange media frames.</li>
</ul>
<ul>
<li>Adjusting: Adjusting operations are those that make a change to the media without changing its compatibility in any way. Examples of adjusting media operations are jitter buffering and volume adjusting. Adjusting media operations will frequently be put into a media stream path by choice (as a result of configuration, for example) in order to enhance the user experience, but may also be required in some situations to complete an otherwise <b>transforming</b> media stream path.</li>
</ul>
<h1><a name="MediaOperations-Useofmediaoperations"></a>Use of media operations</h1>
<p>Media operations are valid to use anywhere that stream-oriented media exists. In practice, this means that there are two levels at which media operations may be used:</p>
<ul>
<li>Session: Session level media operations will affect media to or from a specific session. This can be useful for operations that should not affect all parties in a bridged call. If a conference participant has hearing issues, then it may be good to place a volume adjustment media operation for media going to his session.</li>
</ul>
<ul>
<li>Bridge level. Bridge level media operations will affect media to and/or from all sessions involved in a particular bridge. This type will be more rare.</li>
</ul>
<p>How does a session or bridge know to use media operations? There are two ways to use them:</p>
<ul>
<li>Configuration. Endpoints and bridges may be configured to have specific media operations used by default.</li>
<li>Hooks. Session creation hooks and bridge creation hooks are ways to dynamically insert media operations for a session or bridge.</li>
</ul>
<h1><a name="MediaOperations-MediaOperationSlicedefinitions"></a>Media Operation Slice definitions</h1>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">interface MediaOperation
{
AsteriskSCF::Media::V1::StreamSource* getSource();
AsteriskSCF::Media::V1::StreamSink* getSink();
void destroy();
};
interface MediaOperationFactory
{
/**
* Create an instance of a MediaOperation.
* @param source if provided, will be used only to determine the media format that will
* supplied to the desired MediaOperation
* @param sink if provided, will be used only to determine the media format that the
* desired MediaOperation must produce
*/
MediaOperation* createMediaOperation(
AsteriskSCF::Media::V1::StreamSource* source,
AsteriskSCF::Media::V1::StreamSink* sink);
};</pre>
</div></div>
<p>When a media operation is needed to transform a media stream, an Asterisk SCF component can locate a suitable <tt>MediaOperationFactory</tt> using the Service Locator, and specifying the source and sink formats that the desired <tt>MediaOperation</tt> must support. If a suitable <tt>MediaOperationFactory</tt> is located, it can be used to construct <tt>MediaOperation</tt> instances, each of which can be used to handle a single media stream.</p>
<p>Some <tt>MediaOperationFactory</tt> services will only be able to instantiate a single type of <tt>MediaOperation</tt>, that support a single source format and a single sink format. Others will be able to construct a variety of <tt>MediaOperation</tt> instances. </p>
<p>For <tt>MediaOperationFactory</tt> services that can provide a variety of <tt>MediaOperation</tt> instances, in order to describe to the <tt>MediaOperationFactory</tt> the <b>specific</b> source and sink formats that are needed, the <tt>createMediaOperation</tt> operation accepts a <tt>source</tt> parameter and a <tt>sink</tt> parameter. The <tt>MediaOperationFactory</tt> will learn from the source and sink what formats they support, and will then attempt to instantiate a <tt>MediaOperation</tt> that can transform the media from the source's format to the sink's format.</p>
<h1><a name="MediaOperations-Dealingwithformats"></a>Dealing with formats</h1>
<p>Most media operations will not be able to support any arbitrary media format. It is almost certain that there will be a need to transcode between formats when using a media operation. In order to be able to translate between formats, transcoding support is necessary. Transcoding is a bit tricky since it may take multiple translation steps to get from one format to another. Since each translation step would be a single media operation, it becomes difficult to try to request a single transcoding operation. Instead, what is more handy is to have a transcoder available. This service will maintain a graph of available formats and their ideal translation paths. Given any two formats, the transcoding service can then allocate media operations to satisfy the necessary translation.</p>
<p>The transcoder service can listen to service locator events to find when services register themselves with the service locator. When a service is determined to be a transforming media operation, the transcoding service can add the new operation to its translation graph.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">struct MediaOperationAttributes
{
/**
* The input format for a specific operation
*/
AsteriskSCF::Media::V1::Format inputFormat;
/**
* The output format for a specific operation
*/
AsteriskSCF::Meida::V1::Format outputFormat;
/**
* The cost of the operation.
* Lower cost indicates an "easier" translation,
* either because it is faster or uses fewer resources.
*/
int cost;
};
sequence<MediaOperationAttributes> MediaOperationAttributesSeq;
unsliceable class MediaOperationServiceLocatorParams extends AsteriskSCF::Core::Discovery::V1::ServiceLocatorParams
{
MediaOperationAttributeSeq attributes;
}</pre>
</div></div>
<p>When a <tt>MediaOperation</tt> comes into service, it will register a set of <tt>MediaOperationAttributes</tt> with the service locator. Each of these attributes will describe a pair of input and output formats and the cost of translating from the input to the output format. The transcoder can treat these pairs as new weighted directed edges to add to its graph of translations. As new edges are added, the transcoder can determine new paths and register these with the service locator. This way, when a component requests a <tt>MediaOperationFactory</tt> that will translate between two formats that require multiple steps, the transcoder can make this appear to be a single media operation.</p>
<p>In case that wasn't clear, imagine that a component registers itself with the service locator and tells the service locator that it can translate from format A to format B and from format B to format A. The transcoder is told of this new service and keeps a graph of these two possible translations. Later, another component tells the service locator that it can translate from format B to format C and from format C to format B. The transcoder is told of this new service and adds new edges to its graph. The transcoder now realizes that it has a path from A to C and from C to A. The transcoder in turn tells the service locator that he can handle these two new translations. This way, if a component at any point requests from the service locator a <tt>MediaOperationFactory</tt> to translate from A to C, the service locator will just point the component to the transcoder, who will have the appropriate means of creating the <tt>MediaOperation</tt> necessary.</p>
<h1><a name="MediaOperations-Example1"></a>Example 1</h1>
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>While the examples below help to illustrate how <tt>MediaOperations</tt> are established, it does not go into detail about how to properly chain <tt>MediaOperations</tt> together. That is for a future discussion.</td></tr></table></div>
<p>Let's examine a session level example. Let's say that a session gateway creates a session with an endpoint. We will be receiving G.711 ulaw audio from this endpoint. This endpoint has been configured to have a pitch shift media operation on its incoming media path. The pitch shifter requires 8 kHz signed linear audio as input and outputs the same (since it is an adjusting media operation).</p>
<p>The session gateway, upon setting up the G.711 ulaw stream source for the session, will inspect the configured media operations to employ. First, the pitch shifter is looked up. The pitch shifter is found using the service locator, and its <tt>MediaOperationFactory</tt> proxy is returned. From here, a new <tt>MediaOperation</tt> is allocated.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 453px;"
usemap="#GLIFFY_MAP_17203407_media_flow_mismatch"
src="/wiki/download/attachments/17203407/media_flow_mismatch.png?version=3&modificationDate=1312580451441"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow&#95;mismatch"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<p>There is a problem. The G.711 source cannot send audio directly to the 8 kHz signed linear sink of the pitch shifter. The session gateway calls into the service locator to find a <tt>MediaOperation</tt> that can translate between G.711 and 8 kHz signed linear audio. The service locator attempts to find a media operation with <tt>MediaOperationAttributes</tt> that satisfy this need. In this case, the transcoder's wizardry is not required because there is a single <tt>MediaOperation</tt> that can translate between the two formats, and the service locator knows about it directly.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 439px;"
usemap="#GLIFFY_MAP_17203407_media_flow"
src="/wiki/download/attachments/17203407/media_flow.png?version=3&modificationDate=1312580495332"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<h1><a name="MediaOperations-Example2"></a>Example 2</h1>
<p>Let's consider a case dealing with the bridge instead. In this scenario, two sessions wish to communicate. One session's source audio is G.722 and the sink provided by the other session is G.711 ulaw. The bridge, upon seeing that the two formats are incompatible, will request that the service locator find a <tt>MediaOperation</tt> with <tt>MediaOperationAtrributes</tt> that satisfy this translation. In this case, the transcoder has registered such a <tt>MediaOperation</tt> with the service locator. In reality, the transcoder has derived this <tt>MediaOperation</tt> from three separate operations. The picture below illustrates how the transcoder chains the operations together in order to create what appears to be a single <tt>MediaOperation</tt>.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1334px; height: 549px;"
usemap="#GLIFFY_MAP_17203407_bridge_media_flow"
src="/wiki/download/attachments/17203407/bridge_media_flow.png?version=4&modificationDate=1311371701128"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;bridge&#95;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=TOP">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=17203407&revisedVersion=27&originalVersion=26">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>