<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2166/3/7/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">Media Operations</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~mmichelson">Mark Michelson</a>
</h4>
<div id="versionComment">
<b>Comment:</b>
Add some more corrections based on review feedback.<br />
</div>
<br/>
<h4>Changes (21)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > AsteriskSCF::Media::V1::StreamSource* mediaSource; <br> AsteriskSCF::Media::V1::StreamSink* mediaSink; <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> AsteriskSCF::SessionCommunications::V1::TelephonyEventSource* telephonyEventSource; <br> AsteriskSCF::SessionCommunicatinos::V1::TelephonyEventSink* telephonyEventSink; <br></td></tr>
<tr><td class="diff-unchanged" > string id; <br>}; <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > /** <br> * Create a new instance of a media operation. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> * @param inSource optional source of media that will enter the operation <br> * @param inSink optional sink of media that the operation will send to <br></td></tr>
<tr><td class="diff-unchanged" > */ <br></td></tr>
<tr><td class="diff-changed-lines" >MediaOperation <span class="diff-changed-words">createInstance(<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">);</span></span> <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> AsteriskSCF::Media::V1::StreamSource* source, <br> AsteriskSCF::Media::V1::StreamSink* sink); <br></td></tr>
<tr><td class="diff-unchanged" > /** <br> * Destroy an instance of a media operation. <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{code} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">When a session or bridge wishes to make use of a media operation, it can find the corresponding {{MediaOperationManager}} using the service locator. From there, every instance of a {{MediaOperation}} can be requested as necessary. A {{MediaOperation}} consists of a media source and sink pairing, and a telephony event source and sink pairing. If more sources and sinks are required, then more {{MediaOperations}} can be requested from the {{MediaOperationManager}}. The telephony event source and sink are optional, and are provided in case the media operation wishes to react to telephony events. For instance, a volume control media operation may react to specific DTMF presses in order to increase or decrease the volume. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">When a session or bridge wishes to make use of a media operation, it can find the corresponding {{MediaOperationManager}} using the service locator. From there, every instance of a {{MediaOperation}} can be requested as necessary. A {{MediaOperation}} consists of a media source and sink pairing. If more sources and sinks are required, then more {{MediaOperations}} can be requested from the {{MediaOperationManager}}. The optional {{source}} and {{sink}} parameters may be provided to {{createInstance}} in order to have the MediaOperationManager attempt to create translation paths between formats if necessary/desired. The formats of media supported by a {{MediaOperation}} can be gleaned from the {{mediaSource}} and {{mediaSink}} parameters. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">The formats of media supported by a {{MediaOperation}} can be gleaned from the {{mediaSource}} and {{mediaSink}} parameters. <br> <br></td></tr>
<tr><td class="diff-unchanged" >h1. Dealing with formats <br> <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" > * <br> * @param inSource The initial source of media <br></td></tr>
<tr><td class="diff-changed-lines" >* @param inSink <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">The</span> <span class="diff-added-words"style="background-color: #dfd;">Optional</span> initial destination of <span class="diff-changed-words">media<span class="diff-added-chars"style="background-color: #dfd;">.</span></span> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> * @param outSource The new source of media for inSink <br> * @param outSink The new sink for media from inSource <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> * @return A media operation with source and sink set so that translation <br> * occurs as needed. <br></td></tr>
<tr><td class="diff-unchanged" > */ <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">void</span> <span class="diff-added-words"style="background-color: #dfd;">MediaOperation</span> createTranslationPath( <br></td></tr>
<tr><td class="diff-unchanged" > AsteriskSCF::Media::V1::StreamSource* inSource, <br></td></tr>
<tr><td class="diff-changed-lines" >AsteriskSCF::Media::V1::StreamSink* <span class="diff-changed-words">inSink<span class="diff-deleted-chars"style="color:#999;background-color:#fdd;text-decoration:line-through;">,</span><span class="diff-added-chars"style="background-color: #dfd;">)</span></span> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> out AsteriskSCF::Media::V1::StreamSource* outSource, <br></td></tr>
<tr><td class="diff-changed-lines" ><span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">out AsteriskSCF::Media::V1::StreamSink* outSink)</span> throws UnknownFormat, throws NoPathAvailable; <br></td></tr>
<tr><td class="diff-unchanged" > <br> /** <br> * Remove a translation path from the transcoder service. <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> * @param source The source returned as outSource from createTranslationPath <br> * @param sink The sink returned as outSink from createTranslationPath <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> * @param id the id of the MediaOperation returned from the call to createTranslationPath <br></td></tr>
<tr><td class="diff-unchanged" > */ <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;"> void destroyTranslationPath( <br> AsteriskSCF::Media::V1::StreamSource* source, <br> AsteriskSCF::Media::V1::StreamSink* sink) throws NonExistentPath; <br> <br> /** <br> * Adds a new transcoder to the service. <br> * The formats supported by the transcoder can be discerned <br> * through method calls on the transcoder <br> * parameter. <br> */ <br> void addTranscoder(MediaOperationManager* transcoder); <br> <br> /** <br> * Removes a transcoder from the service. <br> */ <br> void removeTranscoder(MediaOperationManager *transcoder); <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;"> void destroyTranslationPath(string id) throws NonExistentPath; <br></td></tr>
<tr><td class="diff-unchanged" >}; <br>{code} <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">With this, any transforming media operation will need to register itself with the transcoding service. This way, when transcoding is necessary, a component can ask the transcoder to set things up for them. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">The transcoder service can listen to service locator events to find when services register themselves with the service locator. When a service is determined to be a transforming media operation, the transcoding service can add the new operation to its translation graph. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h1. Example 1 <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >{gliffy:name=media_flow_mismatch|align=left|size=L|version=2} <br> <br></td></tr>
<tr><td class="diff-changed-lines" >There is a problem. The G.711 source cannot send audio directly to the 8 kHz signed linear sink of the jitter buffer. The session gateway calls on the <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">trnascoder</span> <span class="diff-added-words"style="background-color: #dfd;">transcoder</span> service to provide a path between G.711 ulaw and 8 kHz signed linear. The transcoder provides an appropriate source and a sink for translating the audio. The session gateway connects the sources and sinks as appropriate. <br></td></tr>
<tr><td class="diff-unchanged" > <br>{gliffy:name=media_flow|align=left|size=L|version=2} <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<div class='panelMacro'><table class='warningMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/forbidden.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>This is a work in progress.</td></tr></table></div>
<h1><a name="MediaOperations-Introduction"></a>Introduction</h1>
<p>In Asterisk SCF, a "media operation" refers to a process by which media is manipulated. Common media operations would be transcoding, jitter buffering, volume adjustment, pitch shifting, and potentially media injection. What follows is a discussion of how media operations fit into the architecture of Asterisk SCF, and how they can be used.</p>
<p>Without media operations, Asterisk SCF's media handling is straightforward, but not very effective. Each session is equipped with media stream sources and sinks. A stream source is where media comes from, and a stream sink is where media is sent. The bridging component can connect a source from one session to a sink of another session in order to connect media from the sessions together.</p>
<h1><a name="MediaOperations-Classificationofmediaoperations"></a>Classification of media operations</h1>
<p>Media operations can be categorized as follows:</p>
<ul>
<li>Transforming: Transforming operations are those where the incoming and outgoing media types are different. Typically this will be the only function of the media operation. The change to the media may be a format change or a change to the parameters of the format. Examples of transforming media operations are transcoding and resampling. Usually transforming operations are not requested by choice, but rather they are required in order for media to be able to flow properly.</li>
</ul>
<ul>
<li>Adjusting. Adjusting media operations are those that make a change to the media without changing its compatibility in any way. Examples of adjusting media operations are jitter buffering and volume adjusting. Adjusting media operations are not required for media to pass, but they can be requested to enhance the user experience.</li>
</ul>
<h1><a name="MediaOperations-Useofmediaoperations"></a>Use of media operations</h1>
<p>Media operations are valid to use anywhere that media exists. In practice, this means that there are two levels at which media operations may be used:</p>
<ul>
<li>Session level. Session level media operations will affect media to and/or from a specific session. This can be useful for operations that should not affect all parties in a bridged call. If a conference participant has hearing issues, then it may be good to place a volume adjustment media operation for media going to his session.</li>
</ul>
<ul>
<li>Bridge level. Bridge level media operations will affect media to and/or from all sessions involved in a particular bridge. This type will be more rare.</li>
</ul>
<p>How does a session or bridge know to use media operations? There are two ways to use them:</p>
<ul>
<li>Configuration. Endpoints and bridges may be configured to have specific media operations used by default.</li>
<li>Hooks. Session creation hooks and bridge creation hooks are ways to dynamically insert media operations for a session or bridge.</li>
</ul>
<h1><a name="MediaOperations-Mediaoperationslice"></a>Media operation slice</h1>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">class MediaOperation
{
AsteriskSCF::Media::V1::StreamSource* mediaSource;
AsteriskSCF::Media::V1::StreamSink* mediaSink;
string id;
};
interface MediaOperationManager
{
/**
* Create a new instance of a media operation.
* @param inSource optional source of media that will enter the operation
* @param inSink optional sink of media that the operation will send to
*/
MediaOperation createInstance(
AsteriskSCF::Media::V1::StreamSource* source,
AsteriskSCF::Media::V1::StreamSink* sink);
/**
* Destroy an instance of a media operation.
* The id is the id parameter of the MediaOperation to destroy.
*/
void destroyInstance(string id);
};</pre>
</div></div>
<p>When a session or bridge wishes to make use of a media operation, it can find the corresponding <tt>MediaOperationManager</tt> using the service locator. From there, every instance of a <tt>MediaOperation</tt> can be requested as necessary. A <tt>MediaOperation</tt> consists of a media source and sink pairing. If more sources and sinks are required, then more <tt>MediaOperations</tt> can be requested from the <tt>MediaOperationManager</tt>. The optional <tt>source</tt> and <tt>sink</tt> parameters may be provided to <tt>createInstance</tt> in order to have the MediaOperationManager attempt to create translation paths between formats if necessary/desired. The formats of media supported by a <tt>MediaOperation</tt> can be gleaned from the <tt>mediaSource</tt> and <tt>mediaSink</tt> parameters.</p>
<h1><a name="MediaOperations-Dealingwithformats"></a>Dealing with formats</h1>
<p>Most media operations will not be able to support any arbitrary media format. It is almost certain that there will be a need to transcode between formats when using a media operation. In order to be able to translate between formats, transcoding support is necessary. Transcoding is a bit tricky since it may take multiple translation steps to get from one format to another. Since each translation step would be a single media operation, it becomes difficult to try to request a single transcoding operation. Instead, what is more handy is to have a transcoding service available. This service will maintain a graph of available formats and their ideal translation paths. Given any two formats, the transcoding service can then allocate media operations to satisfy the necessary translation.</p>
<p>The transcoding service will have an interface like the following:</p>
<div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent">
<pre class="theme: Confluence; brush: java; gutter: false">/**
* Indicates the transcoding service was given an input source or sink with an unknown format
*/
exception UnknownFormat
{
};
/**
* Indicates the transcoding service is unable to translate between the formats given
*/
exception NoPathAvailable
{
};
/**
* Indicates the translation path trying to be removed cannot be found.
*/
exception NonexistentPath
{
};
interface TranscodingService
{
/**
* Sets up a translation path between a stream source and a stream sink.
*
* @param inSource The initial source of media
* @param inSink Optional initial destination of media.
* @return A media operation with source and sink set so that translation
* occurs as needed.
*/
MediaOperation createTranslationPath(
AsteriskSCF::Media::V1::StreamSource* inSource,
AsteriskSCF::Media::V1::StreamSink* inSink)
throws UnknownFormat, throws NoPathAvailable;
/**
* Remove a translation path from the transcoder service.
* @param id the id of the MediaOperation returned from the call to createTranslationPath
*/
void destroyTranslationPath(string id) throws NonExistentPath;
};</pre>
</div></div>
<p>The transcoder service can listen to service locator events to find when services register themselves with the service locator. When a service is determined to be a transforming media operation, the transcoding service can add the new operation to its translation graph.</p>
<h1><a name="MediaOperations-Example1"></a>Example 1</h1>
<p>Let's examine a session level example. Let's say that a session gateway creates a session with an endpoint. We will be receiving G.711 ulaw audio from this endpoint. This endpoint has been configured to have a jitter buffer media operation on its incoming media path. The jitter buffer requires 8 kHz signed linear audio as input and outputs the same (since it is an adjusting media operation).</p>
<p>The session gateway, upon setting up the G.711 ulaw stream source for the session, will inspect the configured media operations to employ. First, the jitter buffer is looked up. The jitter buffer is found using the service locator, and its <tt>MediaOperationManager</tt> proxy is returned. From here, a new <tt>MediaOperation</tt> is allocated.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 453px;"
usemap="#GLIFFY_MAP_17203407_media_flow_mismatch"
src="/wiki/download/attachments/17203407/media_flow_mismatch.png?version=2&modificationDate=1311370330871"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow&#95;mismatch"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<p>There is a problem. The G.711 source cannot send audio directly to the 8 kHz signed linear sink of the jitter buffer. The session gateway calls on the transcoder service to provide a path between G.711 ulaw and 8 kHz signed linear. The transcoder provides an appropriate source and a sink for translating the audio. The session gateway connects the sources and sinks as appropriate.</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1211px; height: 447px;"
usemap="#GLIFFY_MAP_17203407_media_flow"
src="/wiki/download/attachments/17203407/media_flow.png?version=2&modificationDate=1311370736049"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
<h1><a name="MediaOperations-Example2"></a>Example 2</h1>
<p>Let's consider a case dealing with the bridge instead. In this scenario, two sessions wish to communicate. One session's source audio is G.722 and the sink provided by the other session is G.711 ulaw. The bridge, upon seeing that the two formats are incompatible, will enlist the help of the transcoder. This time, the transcoder gets to be a bit more of a shining hero to the bridge because the transcoder is capable of chaining together several transforming media operations in order to make the translation path necessary. Let's have a look:</p>
<table width="100%">
<tr>
<td align="left">
<table>
<caption align="bottom">
</caption>
<tr>
<td>
<img style="border: none; width: 1334px; height: 549px;"
usemap="#GLIFFY_MAP_17203407_bridge_media_flow"
src="/wiki/download/attachments/17203407/bridge_media_flow.png?version=4&modificationDate=1311371701128"
alt="A&#32;Gliffy&#32;Diagram&#32;named&#58;&#32;bridge&#95;media&#95;flow"/>
</td>
</tr>
</table>
</td>
</tr>
</table>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=TOP">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=17203407&revisedVersion=23&originalVersion=22">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Media+Operations?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>