<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/2042/1/7/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Failover+and+Replication">Failover and Replication</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~kpfleming">Kevin P. Fleming</a>
</h4>
<div id="versionComment">
<b>Comment:</b>
copy editing; no functional changes<br />
</div>
<br/>
<h4>Changes (19)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-unchanged" >h3. Overview <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">Asterisk SCF is defined to support fault tolerance when deployed in conjunction with certain platform-specific enabling technologies. The primary mechanism for this is by supporting Active-Standby failover. In Active-Standby failover, a specific component has at least two instances running. One is in active mode and is processing requests in the Asterisk SCF system. All other instances are in standby mode in the event of failure or planned shutdown of the active component. Dynamic state replication allows the state of the active component to be reflected in its standby components in near real-time so that a standby is ready to be activated with a minimal amount of disruption of services to the rest of the Asterisk SCF system that rely on the component. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">Asterisk SCF is designed to support fault tolerance when deployed in conjunction with certain platform-specific enabling technologies. The primary mechanism to achieve this is Active-Standby failover. In Active-Standby failover, a specific component has at least two instances running. One is in active mode and is processing requests in the Asterisk SCF system; all other instances are in standby mode, ready to become active in the event of failure or planned shutdown of the active instance. Dynamic state replication allows the state of the active component to be reflected in the standby components in near real-time so that any standby instance is ready to be activated with a minimal amount of disruption of services to the rest of the Asterisk SCF system. <br></td></tr>
<tr><td class="diff-unchanged" > <br>{info} <br></td></tr>
<tr><td class="diff-unchanged" >Fallback is also expected to be feasible in the future. Other types of failover (ex: active-active) may also be supported in the future. <br></td></tr>
<tr><td class="diff-unchanged" >{info} <br> <br>h3. Replication Groups and the Replica Interface <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">We refer to a set of same-type components, where one is active and another one or more are in standby mode (to takeover in the event of failover), as a *replica group*. Each component that supports replication is required to implement the *Replica* interface in the AsteriskSCF.System.Component namespace, which provides *activate()* and *standby()* operations to control the component's roles in the replica group. During startup, every component that participates in a replica group will initialize itself to standby mode. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">We refer to a set of component instances, where one is active and the others are in standby mode, as a *replica group*. Each component that supports replication is required to implement the *Replica* interface in the AsteriskSCF::System::Component namespace, which provides *activate()* and *standby()* operations to control the instances' roles in the replica group. During startup, every instance that participates in a replica group will initialize itself to standby mode. <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">The determination of which component is active and which is standby is expected to be made by configuring technologies external to Asterisk SCF. Examples of such technologies would be Pacemaker or Windows Cluster Server. Asterisk SCF provides a python script which has access to the Replica interfaces of the Asterisk SCF components to provide a mechanism for these external technologies to activate specific component instances. This script is used both to activate a component in a replica group during startup as well as to activate a standby component when a failure is detected, or the previously active component needs to be shut down for maintenance. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">The determination of which instance is active and which is standby is expected to be made by configuring technologies external to Asterisk SCF. Commonly-deployed operating system platforms provide 'cluster service' packages; examples of such technologies would be Pacemaker or Windows Cluster Server. Asterisk SCF provides a script (written in Python) which can be used to invoke operations on the Replica interfaces of the Asterisk SCF component instances. This script can be used by these external technologies to activate specific component instances. It would be used both to activate an instance in a replica group during startup as well as to activate a standby instance when failover is required (planned or unplanned). <br></td></tr>
<tr><td class="diff-unchanged" > <br>| !ReplicaManagement.png|border=0! | <br></td></tr>
<tr><td class="diff-changed-lines" >| {center}{*}Figure 1. Active <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">Component</span> <span class="diff-added-words"style="background-color: #dfd;">Instance</span> Selection{*}{center} | <br></td></tr>
<tr><td class="diff-unchanged" > <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">As depicted by a switch in Figure 1, a key concept in the failover process is the notion of a movable IP address. The connection labled "Shared" in the diagram is actually moved from one node to another by the external failover system. To maintain communication with and control of every component in a replica group, we require another dedicated IP address for each node. The Component developer must ensure that the Replica interface is created on an [Object Adapter|http://www.zeroc.com/doc/Ice-3.4.1-IceTouch/manual/Adv_server.33.4.html] that can be configured to be on the dedicated IP address. In Figure 1, we call this the Backplane Adapter. The Backplane Adapter is for management interfaces and other control/coordination interfaces not related to the primary function that the component provides to the Asterisk SCF system. Conversely, the adapter associated with the movable IP address is referred to as the ServiceAdapter, as it provides access to the interfaces that the component publishes to make its functional services available to other Asterisk SCF components. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">As depicted by a switch in Figure 1, a key concept in the failover process is the notion of a movable IP address. The connection labled "Shared" in the diagram is actually moved from one node to another by the platform's cluster management system. To maintain communication with and control of every instance in a replica group, every node also has to provide at least one non-movable IP address. Asterisk SCF components will provide access to their Replica interfaces on [Object Adapters|http://www.zeroc.com/doc/Ice-3.4.1-IceTouch/manual/Adv_server.33.4.html]&nbsp;separate from those used to provide the component's services to the system. In Figure 1, we call this the Backplane Adapter. The Backplane Adapter is for management interfaces and other control/coordination interfaces not related to the primary function that the component provides to the Asterisk SCF system. Conversely, the adapter where the component provides its primary functions to the system, is referred to as the Service Adapter. When the component instances are configured, the Service Adapter would be configured to use the movable IP address, and the Backplane Adapter would be configured to use the non-movable IP address. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h3. State Replication <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">To support fast activation of standby components, Asterisk SCF uses a dynamic state replication approach that keeps standby components up-to-date with the internal state changes of the active component. This is documented on the page [State Replication|State Replication]. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">To support fast activation of standby component instances, Asterisk SCF uses a dynamic state replication approach that keeps them up-to-date with the internal state changes of the active instance. This is documented on the [State Replication|State Replication]&nbsp;page. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h3. Component Configuration <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">The component is configured in a deployment via an Ice configuration file which defines how to access the Service Locator, and a minimal set of other properties. The configuration details that pertain to the technical domain of the component are defined via pushed configuration data. <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">When an Asterisk SCF component instance is deployed, its Ice configuration file will tell it how to access the Service Locator, and a minimal set of other properties. The configuration details that pertain to the technical domain of the component (its primary function) are delivered to the component using its implementation of the ConfigurationService interface. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h5. Ice Properties <br> <br></td></tr>
<tr><td class="diff-deleted-lines" style="color:#999;background-color:#fdd;text-decoration:line-through;">To get a reference to the State Replicator that an active component sends updates to (and a standby component listens to) each component in a replica group should define an Ice property named: <br></td></tr>
<tr><td class="diff-added-lines" style="background-color: #dfd;">To get a reference to the State Replicator that an active instance sends updates to (and a standby instance listens to) each instance in a replica group should be configured with an Ice property named: <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br> <AdapterName>.StateReplicatorName <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-changed-lines" >To run a component that is designed to support replication outside of a replica group, the <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">component</span> <span class="diff-added-words"style="background-color: #dfd;">instance</span> should have this property set: <br></td></tr>
<tr><td class="diff-unchanged" >{noformat} <br> <AdapterName>.Standalone=true <br>{noformat} <br> <br></td></tr>
<tr><td class="diff-changed-lines" >When this property is set to true, the <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">component</span> <span class="diff-added-words"style="background-color: #dfd;">instance</span> will automatically set itself into an active state when started. If this property is false (or does not exist), the <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">component</span> <span class="diff-added-words"style="background-color: #dfd;">instance</span> will assume it is part of a replica group, and will startup in <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">a</span> standby mode. <br></td></tr>
<tr><td class="diff-unchanged" > <br>The Service Adapter's endpoints are defined using the standard Ice-prescribed _<AdapterName>.Endpoints_ property. The name of the Backplane Adapter will be derived from that of the Service Adapter by adding ".Backplane", so the endpoint properties will look like this: <br></td></tr>
<tr><td class="diff-snipped" >...<br></td></tr>
<tr><td class="diff-unchanged" >h5. Configuration Data <br> <br></td></tr>
<tr><td class="diff-unchanged" >Configuration data is pushed to a replica group using the approach defined on the page [Replication of Configuration Changes]. <br></td></tr>
<tr><td class="diff-unchanged" > <br>h4. Component Startup Sequence <br> <br></td></tr>
<tr><td class="diff-changed-lines" >The following sequence diagram describes the basic flow of a <span class="diff-changed-words">component<span class="diff-added-chars"style="background-color: #dfd;"> instance</span>'s</span> startup process. <br></td></tr>
<tr><td class="diff-unchanged" > <br>| !Component Startup for Replica Groups.png|border=1! | <br></td></tr>
<tr><td class="diff-changed-lines" >| {center}{*}Figure 2. Component Startup for Replica Group <span class="diff-deleted-words"style="color:#999;background-color:#fdd;text-decoration:line-through;">components.{*}{center}</span> <span class="diff-added-words"style="background-color: #dfd;">components.*{center}</span> | <br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<h3><a name="FailoverandReplication-Overview"></a>Overview</h3>
<p>Asterisk SCF is designed to support fault tolerance when deployed in conjunction with certain platform-specific enabling technologies. The primary mechanism to achieve this is Active-Standby failover. In Active-Standby failover, a specific component has at least two instances running. One is in active mode and is processing requests in the Asterisk SCF system; all other instances are in standby mode, ready to become active in the event of failure or planned shutdown of the active instance. Dynamic state replication allows the state of the active component to be reflected in the standby components in near real-time so that any standby instance is ready to be activated with a minimal amount of disruption of services to the rest of the Asterisk SCF system.</p>
<div class='panelMacro'><table class='infoMacro'><colgroup><col width='24'><col></colgroup><tr><td valign='top'><img src="/wiki/images/icons/emoticons/information.gif" width="16" height="16" align="absmiddle" alt="" border="0"></td><td>Fallback is also expected to be feasible in the future. Other types of failover (ex: active-active) may also be supported in the future.</td></tr></table></div>
<h3><a name="FailoverandReplication-ReplicationGroupsandtheReplicaInterface"></a>Replication Groups and the Replica Interface</h3>
<p>We refer to a set of component instances, where one is active and the others are in standby mode, as a <b>replica group</b>. Each component that supports replication is required to implement the <b>Replica</b> interface in the AsteriskSCF::System::Component namespace, which provides <b>activate()</b> and <b>standby()</b> operations to control the instances' roles in the replica group. During startup, every instance that participates in a replica group will initialize itself to standby mode.</p>
<p>The determination of which instance is active and which is standby is expected to be made by configuring technologies external to Asterisk SCF. Commonly-deployed operating system platforms provide 'cluster service' packages; examples of such technologies would be Pacemaker or Windows Cluster Server. Asterisk SCF provides a script (written in Python) which can be used to invoke operations on the Replica interfaces of the Asterisk SCF component instances. This script can be used by these external technologies to activate specific component instances. It would be used both to activate an instance in a replica group during startup as well as to activate a standby instance when failover is required (planned or unplanned).</p>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> <span class="image-wrap" style=""><img src="/wiki/download/attachments/13076639/ReplicaManagement.png?version=2&modificationDate=1305829967994" style="border: 0px solid black" /></span> </td>
</tr>
<tr>
<td class='confluenceTd'> <div class="" align="center"><b>Figure 1. Active Instance Selection</b></div>
</td>
</tr>
</tbody></table>
</div>
<p>As depicted by a switch in Figure 1, a key concept in the failover process is the notion of a movable IP address. The connection labled "Shared" in the diagram is actually moved from one node to another by the platform's cluster management system. To maintain communication with and control of every instance in a replica group, every node also has to provide at least one non-movable IP address. Asterisk SCF components will provide access to their Replica interfaces on <a href="http://www.zeroc.com/doc/Ice-3.4.1-IceTouch/manual/Adv_server.33.4.html" class="external-link" rel="nofollow">Object Adapters</a> separate from those used to provide the component's services to the system. In Figure 1, we call this the Backplane Adapter. The Backplane Adapter is for management interfaces and other control/coordination interfaces not related to the primary function that the component provides to the Asterisk SCF system. Conversely, the adapter where the component provides its primary functions to the system, is referred to as the Service Adapter. When the component instances are configured, the Service Adapter would be configured to use the movable IP address, and the Backplane Adapter would be configured to use the non-movable IP address.</p>
<h3><a name="FailoverandReplication-StateReplication"></a>State Replication</h3>
<p>To support fast activation of standby component instances, Asterisk SCF uses a dynamic state replication approach that keeps them up-to-date with the internal state changes of the active instance. This is documented on the <a href="/wiki/display/TOP/State+Replication" title="State Replication">State Replication</a> page.</p>
<h3><a name="FailoverandReplication-ComponentConfiguration"></a>Component Configuration</h3>
<p>When an Asterisk SCF component instance is deployed, its Ice configuration file will tell it how to access the Service Locator, and a minimal set of other properties. The configuration details that pertain to the technical domain of the component (its primary function) are delivered to the component using its implementation of the ConfigurationService interface.</p>
<h5><a name="FailoverandReplication-IceProperties"></a>Ice Properties</h5>
<p>To get a reference to the State Replicator that an active instance sends updates to (and a standby instance listens to) each instance in a replica group should be configured with an Ice property named:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> <AdapterName>.StateReplicatorName
</pre>
</div></div>
<p>To run a component that is designed to support replication outside of a replica group, the instance should have this property set:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> <AdapterName>.Standalone=true
</pre>
</div></div>
<p>When this property is set to true, the instance will automatically set itself into an active state when started. If this property is false (or does not exist), the instance will assume it is part of a replica group, and will startup in standby mode.</p>
<p>The Service Adapter's endpoints are defined using the standard Ice-prescribed <em><AdapterName>.Endpoints</em> property. The name of the Backplane Adapter will be derived from that of the Service Adapter by adding ".Backplane", so the endpoint properties will look like this:</p>
<div class="preformatted panel" style="border-width: 1px;"><div class="preformattedContent panelContent">
<pre> <AdapterName>.Endpoints=tcp -p 10040
<AdapterName>.Backplane.Endpoints=tcp -p 10041
</pre>
</div></div>
<h5><a name="FailoverandReplication-ConfigurationData"></a>Configuration Data</h5>
<p>Configuration data is pushed to a replica group using the approach defined on the page <a href="/wiki/display/TOP/Replication+of+Configuration+Changes" title="Replication of Configuration Changes">Replication of Configuration Changes</a>.</p>
<h4><a name="FailoverandReplication-ComponentStartupSequence"></a>Component Startup Sequence</h4>
<p>The following sequence diagram describes the basic flow of a component instance's startup process.</p>
<div class='table-wrap'>
<table class='confluenceTable'><tbody>
<tr>
<td class='confluenceTd'> <span class="image-wrap" style=""><img src="/wiki/download/attachments/13076639/Component+Startup+for+Replica+Groups.png?version=5&modificationDate=1305830809732" style="border: 1px solid black" /></span> </td>
</tr>
<tr>
<td class='confluenceTd'> <div class="" align="center"><b>Figure 2. Component Startup for Replica Group components.</b></div>
</td>
</tr>
</tbody></table>
</div>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;">
<a href="https://wiki.asterisk.org/wiki/users/viewnotifications.action" class="grey">Change Notification Preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Failover+and+Replication">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=13076639&revisedVersion=9&originalVersion=8">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Failover+and+Replication?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>