<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2166/3/7/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">Media Operations</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~mmichelson">Mark Michelson</a>
</h4>
<br/>
<h4>Changes (1)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >There is a problem. The G.711 source cannot send audio directly to the 8 kHz signed linear sink of the pitch shifter. The session gateway calls into the service locator to find a {{MediaOperation}} that can translate between G.711 and 8 kHz signed linear audio. The service locator attempts to find a media operation with {{MediaOperationAttributes}} that satisfy this need. In this case, the transcoder's wizardry is not required because there is a single {{MediaOperation}} that can translate between the two formats, and the service locator knows about it directly. <br> <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-changed-words">{gliffy:name=media_flow|align=left|size=L|version=<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">2</span><span class="diff-added-chars"style="background-color: #dfd;">3</span>}</span> <br></td></tr>
<tr><td class="diff-unchanged" > <br>h1. Example 2 <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>This is a work in progress.</td></tr></table></div>
<h1><a name="MediaOperations-Introduction"></a>Introduction</h1>
<p>In Asterisk SCF, a "media operation" refers to a process by which media is manipulated. Common media operations would be transcoding, jitter buffering, volume adjustment, pitch shifting, and potentially media injection. What follows is a discussion of how media operations fit into the architecture of Asterisk SCF, and how they can be used.</p>
<p>Without media operations, Asterisk SCF's media handling is straightforward, but not very effective. Each session is equipped with media stream sources and sinks. A stream source is where media comes from, and a stream sink is where media is sent. The bridging component can connect a source from one session to a sink of another session in order to connect media from the sessions together.</p>
<h1><a name="MediaOperations-Classificationofmediaoperations"></a>Classification of media operations</h1>
<p>Media operations can be categorized as follows:</p>
<ul>
<li>Transforming: Transforming operations are those where the incoming and outgoing media types are different. Typically this will be the only function of the media operation. The change to the media may be a format change or a change to the parameters of the format. Examples of transforming media operations are transcoding and resampling. Usually transforming operations are not requested by choice, but rather they are required in order for media to be able to flow properly.</li>
</ul>
<ul>
<li>Adjusting. Adjusting media operations are those that make a change to the media without changing its compatibility in any way. Examples of adjusting media operations are jitter buffering and volume adjusting. Adjusting media operations are not required for media to pass, but they can be requested to enhance the user experience.</li>
</ul>
<h1><a name="MediaOperations-Useofmediaoperations"></a>Use of media operations</h1>
<p>Media operations are valid to use anywhere that media exists. In practice, this means that there are two levels at which media operations may be used:</p>
<ul>
<li>Session level. Session level media operations will affect media to and/or from a specific session. This can be useful for operations that should not affect all parties in a bridged call. If a conference participant has hearing issues, then it may be good to place a volume adjustment media operation for media going to his session.</li>
</ul>
<ul>
<li>Bridge level. Bridge level media operations will affect media to and/or from all sessions involved in a particular bridge. This type will be more rare.</li>
</ul>
<p>How does a session or bridge know to use media operations? There are two ways to use them:</p>
<ul>
<li>Configuration. Endpoints and bridges may be configured to have specific media operations used by default.</li>
<li>Hooks. Session creation hooks and bridge creation hooks are ways to dynamically insert media operations for a session or bridge.</li>
</ul>
<h1><a name="MediaOperations-Mediaoperationslice"></a>Media operation slice</h1>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">interface MediaOperation
{
AsteriskSCF::Media::V1::StreamSource* getSource;
AsteriskSCF::Media::V1::StreamSink* getSink;
void destroy();
};
interface MediaOperationFactory
{
/**
* Create a new instance of a media operation.
* @param inSource optional source of media that will enter the operation
* @param inSink optional sink of media that the operation will send to
*/
MediaOperation createMediaOperation(
AsteriskSCF::Media::V1::StreamSource* source,
AsteriskSCF::Media::V1::StreamSink* sink);
};</pre>
</div></div>
<p>When a session or bridge wishes to make use of a media operation, it can find the corresponding <tt>MediaOperationFactory</tt> using the service locator. From there, every instance of a <tt>MediaOperation</tt> can be requested as necessary. A <tt>MediaOperation</tt> consists of a media source and sink pairing. If more sources and sinks are required, then more <tt>MediaOperations</tt> can be requested from the <tt>MediaOperationFactory</tt>. The optional <tt>source</tt> and <tt>sink</tt> parameters may be provided to <tt>createMediaOperation</tt> in order to have the <tt>MediaOperationFactory</tt> attempt to create translation paths between formats if necessary/desired. The formats of media supported by a <tt>MediaOperation</tt> can be gleaned from getting the source and sink.</p>
<h1><a name="MediaOperations-Dealingwithformats"></a>Dealing with formats</h1>
<p>Most media operations will not be able to support any arbitrary media format. It is almost certain that there will be a need to transcode between formats when using a media operation. In order to be able to translate between formats, transcoding support is necessary. Transcoding is a bit tricky since it may take multiple translation steps to get from one format to another. Since each translation step would be a single media operation, it becomes difficult to try to request a single transcoding operation. Instead, what is more handy is to have a transcoder available. This service will maintain a graph of available formats and their ideal translation paths. Given any two formats, the transcoding service can then allocate media operations to satisfy the necessary translation.</p>
<p>The transcoder service can listen to service locator events to find when services register themselves with the service locator. When a service is determined to be a transforming media operation, the transcoding service can add the new operation to its translation graph.</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">struct MediaOperationAttributes
{
/**
* The input format for a specific operation
*/
AsteriskSCF::Media::V1::Format inputFormat;
/**
* The output format for a specific operation
*/
AsteriskSCF::Meida::V1::Format outputFormat;
/**
* The cost of the operation.
* Lower cost indicates an "easier" translation,
* either because it is faster or uses fewer resources.
*/
int cost;
};
sequence<MediaOperationAttributes> MediaOperationAttributesSeq;
unsliceable class MediaOperationServiceLocatorParams extends AsteriskSCF::Core::Discovery::V1::ServiceLocatorParams
{
MediaOperationAttributeSeq attributes;
}</pre>
</div></div>
<p>When a <tt>MediaOperation</tt> comes into service, it will register a set of <tt>MediaOperationAttributes</tt> with the service locator. Each of these attributes will describe a pair of input and output formats and the cost of translating from the input to the output format. The transcoder can treat these pairs as new weighted directed edges to add to its graph of translations. As new edges are added, the transcoder can determine new paths and register these with the service locator. This way, when a component requests a <tt>MediaOperationFactory</tt> that will translate between two formats that require multiple steps, the transcoder can make this appear to be a single media operation.</p>
<p>In case that wasn't clear, imagine that a component registers itself with the service locator and tells the service locator that it can translate from format A to format B and from format B to format A. The transcoder is told of this new service and keeps a graph of these two possible translations. Later, another component tells the service locator that it can translate from format B to format C and from format C to format B. The transcoder is told of this new service and adds new edges to its graph. The transcoder now realizes that it has a path from A to C and from C to A. The transcoder in turn tells the service locator that he can handle these two new translations. This way, if a component at any point requests from the service locator a <tt>MediaOperationFactory</tt> to translate from A to C, the service locator will just point the component to the transcoder, who will have the appropriate means of creating the <tt>MediaOperation</tt> necessary.</p>
<h1><a name="MediaOperations-Example1"></a>Example 1</h1>
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>While the examples below help to illustrate how <tt>MediaOperations</tt> are established, it does not go into detail about how to properly chain <tt>MediaOperations</tt> together. That is for a future discussion.</td></tr></table></div>
<p>Let's examine a session level example. Let's say that a session gateway creates a session with an endpoint. We will be receiving G.711 ulaw audio from this endpoint. This endpoint has been configured to have a pitch shift media operation on its incoming media path. The pitch shifter requires 8 kHz signed linear audio as input and outputs the same (since it is an adjusting media operation).</p>
<p>The session gateway, upon setting up the G.711 ulaw stream source for the session, will inspect the configured media operations to employ. First, the pitch shifter is looked up. The pitch shifter is found using the service locator, and its <tt>MediaOperationFactory</tt> proxy is returned. From here, a new <tt>MediaOperation</tt> is allocated.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 453px;"
usemap="#GLIFFY_MAP_17203407_media_flow_mismatch"
src="/wiki/download/attachments/17203407/media_flow_mismatch.png?version=3&modificationDate=1312580451441"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow&#95;mismatch"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<p>There is a problem. The G.711 source cannot send audio directly to the 8 kHz signed linear sink of the pitch shifter. The session gateway calls into the service locator to find a <tt>MediaOperation</tt> that can translate between G.711 and 8 kHz signed linear audio. The service locator attempts to find a media operation with <tt>MediaOperationAttributes</tt> that satisfy this need. In this case, the transcoder's wizardry is not required because there is a single <tt>MediaOperation</tt> that can translate between the two formats, and the service locator knows about it directly.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 439px;"
usemap="#GLIFFY_MAP_17203407_media_flow"
src="/wiki/download/attachments/17203407/media_flow.png?version=3&modificationDate=1312580495332"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<h1><a name="MediaOperations-Example2"></a>Example 2</h1>
<p>Let's consider a case dealing with the bridge instead. In this scenario, two sessions wish to communicate. One session's source audio is G.722 and the sink provided by the other session is G.711 ulaw. The bridge, upon seeing that the two formats are incompatible, will request that the service locator find a <tt>MediaOperation</tt> with <tt>MediaOperationAtrributes</tt> that satisfy this translation. In this case, the transcoder has registered such a <tt>MediaOperation</tt> with the service locator. In reality, the transcoder has derived this <tt>MediaOperation</tt> from three separate operations. The picture below illustrates how the transcoder chains the operations together in order to create what appears to be a single <tt>MediaOperation</tt>.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1334px; height: 549px;"
usemap="#GLIFFY_MAP_17203407_bridge_media_flow"
src="/wiki/download/attachments/17203407/bridge_media_flow.png?version=4&modificationDate=1311371701128"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;bridge&#95;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=TOP">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=17203407&revisedVersion=26&originalVersion=25">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>