<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/en/2172/18/9/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Supporting+Failover+-+Understanding+What+Ice+is+Doing">Supporting Failover - Understanding What Ice is Doing</a></h2>
<h4>Page <b>edited</b> by <a href="https://wiki.asterisk.org/wiki/display/~beagles">Brent Eagles</a>
</h4>
<br/>
<h4>Changes (0)</h4>
<div id="page-diffs">
<table class="diff" cellpadding="0" cellspacing="0">
<tr><td class="diff-snipped" >...<br></td></tr>
</table>
</div> <h4>Full Content</h4>
<div class="notificationGreySide">
<h1><a name="SupportingFailover-UnderstandingWhatIceisDoing-Introduction"></a>Introduction</h1>
<p>The general process of failover is quite simple: a request is made on a Ice object that is implemented on a particular server, that server becames in-accessible or inoperative, a standby server is activated and the request is resent to be processed on then newly activated server. Simple. However, there are details that are important to consider when implementing servers, analyzing and predicting system behavior and avoiding issues.</p>
<ul>
<li>A request on an Ice object has specific stages from the moment is instigated on the client to the time the reply - if relevant - is received at the client. What these stages are depends on multiple factors, some of which will be elaborated on.</li>
<li>The Ice runtime does not share information with other Ice runtime instances. There is practically no way for an Ice environment to process a response to a request that was started in a different environment.</li>
<li>The Asterisk SCF API has been designed to carry extra information per-operation that allows nearly all operations to be classified as <b>idempotent</b>. This allows implementers to leverage features built into Ice to retry operations in the event of an Ice, non-user exception.</li>
<li>Ice does not provide a way to "heal" a chain of requests across multiple objects. If a client request on an object results in other requests and that first object fails-over, then the entire operation will be re-initiated once fail-over is completed and the operation is retried.</li>
</ul>
<p>This sister-document to <a href="/wiki/display/TOP/Supporting+Failover+-+Operation+Context+and+Idempotent+Operations" title="Supporting Failover - Operation Context and Idempotent Operations">Supporting Failover - Operation Context and Idempotent Operations</a> touches on some key details of Ice request processing that may influence how Operation Contexts are used in Asterisk SCF components.</p>
<h2><a name="SupportingFailover-UnderstandingWhatIceisDoing-TheClientSidetheDowncall"></a>The Client Side - the Downcall</h2>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-InvocationofanIceMediatedRequestonaProxy"></a>Invocation of an Ice Mediated Request on a Proxy</h3>
<p>The qualification "Ice Mediated Request" is significant as there are some methods on proxies that do not result in interaction with the Ice communicator. Only those requests that result in processing enabling remote procedure calls require any special consideration when fail-over occurs.</p>
<h4><a name="SupportingFailover-UnderstandingWhatIceisDoing-InvocationTypes"></a>Invocation Types </h4>
<p>All Ice mediated requests result in the construction of a <em>message</em> that is written to a network connection. The structure of the message is in accordance with the Ice Protocol. The behavior of the Ice runtime differs according to the <em>invocation type</em>. The descriptions are not exhaustive; only the details related to Asterisk SCF fail-over are described. </p>
<h5><a name="SupportingFailover-UnderstandingWhatIceisDoing-twoway"></a>twoway</h5>
<p>The <em>twoway</em> invocation type attempts to behave the same way as a standard local method invocation. The client perceives that the operation is processed synchronously and blocks until the method has completed and returned a result if there is one. The Ice runtime matches outgoing messages to incoming messages using identifiers. The identifier matches the sending of a message to the receipt of a reply. If a message is sent multiple times for one invocation, each send will have a different id. Request ids are relatively unique to a connection. If this occurs, the Ice runtime may receive multiple return messages. It will ignore all except the one that matches the last request sent. If a timeout is configured and exceeded and/or an error is detected during client side processing, a locally generated exception is thrown.</p>
<p>It is important to distinguish between the "client blocking" and what the network levels do in a twoway invocation. While a client may block while waiting for a reply, the connection that a request may have been sent on is used for any other request that may need to use it. A thread from the communicator's thread pool deals with processing replies and notifying blocked client threads.</p>
<h5><a name="SupportingFailover-UnderstandingWhatIceisDoing-%22other%22%3Aoneway%2Cbatchedrequestsetal."></a>"other" : oneway, batched requests et al. </h5>
<p>The details for processing oneway requests, batched requests, etc. differ in details significant enough to distinguish among them for perceived behavior. However, the traits that they have in common are the ones that are relevant to failover: </p>
<ul>
<li>The result of the request is not required or expected. Therefore, only requests that have no return type, out parameters and exception specifications may be invoked as one of these types.</li>
<li>Exceptions may be thrown upon invocation. Valid exceptions for these type of requests are restricted to local errors such as a failure to establish a connection</li>
<li>The invocation returns immediately. A successful return does not mean that the message for that request was successfully sent.</li>
<li>Ice uses a request id of -1 for all requests of this invocation type, indicating that there is no reply expected and no such matching can ever occur.</li>
</ul>
<h4><a name="SupportingFailover-UnderstandingWhatIceisDoing-AsynchronousMethodInvocation%28AMI%29"></a>Asynchronous Method Invocation (AMI)</h4>
<p>AMI is not an invocation type. AMI provides an alternate means of invoking a request and retrieving results. It is included here because it <b>is</b> significant to note that using AMI, while different from an invocation and result processing perspective, is much the same as a non-AMI invocation of the same type. The key difference is that the client does not block </p>
<h4><a name="SupportingFailover-UnderstandingWhatIceisDoing-IdempotentOperations"></a>Idempotent Operations</h4>
<p>Ice is designed to help developers to build well-behaved, correct distributed systems. The preservation of <em>at-most-once</em> semantics plays a key part. While Ice provides an automatic immediate retry of requests that fail due to certain conditions, it cannot retry requests that <b>may</b> have reached the server, e.g. the connection aborts while waiting for a request reply. Retrying at this point could violate <em>at-most-once</em> semantics. </p>
<p>Methods marked as idempotent in Slice are expected to behave as if the number and order of invocations has no effect on the consistency and correctness of a system. If a method is truly idempotent, it can be retried an arbitrary number of times safely, invoked before or after any other operation without adversely affecting correctness even when taking into consideration the distributed nature and lack of <em>end-to-end</em> ordering of operations. In other words, adhering to <em>at-most-once</em> guarantees is no longer required. Loosening this restriction allows the Ice to retry all idempotent methods when a local exception occurs. This facility is used throughout Asterisk SCF to facilitate transparent failover (see <a href="/wiki/display/TOP/Supporting+Failover+-+Operation+Context+and+Idempotent+Operations" title="Supporting Failover - Operation Context and Idempotent Operations">Supporting Failover - Operation Context and Idempotent Operations</a>). There is no visible impact on the client when the request is made, but the request processing code flow is altered slightly to allow automatic retries to occur for a more varied list of exception types.</p>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-RequestMessages"></a>Request Messages</h3>
<p>A request message is an encapsulated form of a request on an Ice object. It contains all of the information required for an object implementation to satisfy a request. Each time a downcall is processed, a new message with a new request id is created. If a client makes the "exactly the same" request (same object, same arguments, etc) it is a separate downcall and consequently a completely distinct message and request id. The Ice automatic retry mechanism results in the same behavior. At this level, Ice treats the message as a self-contained data element and much of what happens after the message is created is basic network operations.</p>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-AndDataPasses...NetworkCommunications"></a>And Data Passes... Network Communications</h3>
<p>Once Ice invokes network APIs to send message data to a recipient, it is the behavior of that particular networking "stack" that dictates what happens next. When debugging at this level, a good working knowledge of the the APIs (e.g. Win32 Asynchronous IO, POSIX sockets, OpenSSL, etc.) and protocols (e.g. TCP/IP, UDP/IP, TLS, etc.) is an asset. These are outside of the scope of this documented and are not described here.</p>
<p>While the details of the network communication are out of scope, some of the details of the events that may occur at this point are relevant. Besides exceptions that are created on the "server side", Ice treats errors and exceptions that occur locally differently depending on where in the processing of a message they occur. This is to preserve <em>at-most-once</em> call semantics and facilitates building systems that behave correctly when communication problems arise. If a local error occurs before a message is completely delivered to the network layer, it is expected to be incomplete and a request is may be retried without concern. However, if a message has been completely delivered to the network layer and an error is detected afterwards or if a timeout occurs, an ordinary message may not be safely retried unless it is a semantically idempotent operation.</p>
<h2><a name="SupportingFailover-UnderstandingWhatIceisDoing-ServerSide%3AtheUpcall"></a>Server Side: the Upcall</h2>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-RequestMessages"></a>Request Messages</h3>
<p>As complete requests are read from the network layers, the object id is used to locate a programmatic entity called a <em>servant</em> that is expected to provide an implementation that satisfies the request. If there is a problem reading the contents of the message the Ice runtime does <b>not</b> involve the servant but generates a relevant exception and enters the reply processing stages immediately. </p>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-DispatchingaRequest"></a>Dispatching a Request</h3>
<p>Ice employs a leader-follower style thread pool for allowing multiple requests to occur concurrently in a server. Once a valid servant has been selected, the current thread invokes a matching operation on the servant. There are two types of dispatch to consider: standard and <em>asynchronous method dispatch (AMD)</em>. In the standard upcall dispatch, the thread invoking the request blocks until the servant function returns with a result if necessary or an exception. Ice then takes the result of this call and moves on to the next stage. In AMD, a callback object is created and passed as an additional argument to a special form of the servant method. The thread invoking the request also blocks until this special form returns, but with the exception of "exceptions", does not move on to the next stage of returning a reply to the caller. Instead the callback object that was passed to the servant method must be used to return a result. A simplified way to look at it is that in standard dispatching, the thread for reading and dispatching a request is the same thread as the one that handles sending a reply to the caller. In AMD, the thread that reads and dispatches a request is not necessarily the same thread as the one that handles sending a reply to the caller. </p>
<p>It is at this point that it is important to point out that once the request is read from the network layer, more requests can be read and dispatched from that same connection or other connections. Ice does not serialize any of these operations by default. This has important implications with respect to thread safety in your servant implementations. Ice also imposes no order on locking or thread scheduling so there are no guarantees that a message read from the network will actually "run" before the next message that was read from the network layer. The thread-per-connection concurrency model <b>does</b> support this behavior but is not implemented by Ice.</p>
<h3><a name="SupportingFailover-UnderstandingWhatIceisDoing-TheServantCode"></a>The Servant Code</h3>
<p>As mentioned, Ice does not provide guarantees to prevent multiple threads from concurrently invoking a method on a servant. Servants must contain code to protect the integrity of reading and writing data that may be accessed concurrently. An Ice application <b>can</b> be configured to guarantee single threaded behavior on a single servant, but these configurations tend to be difficult to maintain and waste resources. The details of what happens in the servant is not particularly relevant here except that it is expected to return a result if required or to throw an exception.</p>
<h2><a name="SupportingFailover-UnderstandingWhatIceisDoing-ProcessingtheResult"></a>Processing the Result</h2>
<p>At this point it is useful to consider the standard non-AMD and AMD cases separately.</p>
<ul>
<li>Standard- the thread that read the message from the connection and dispatched the request now takes the result and creates a corresponding reply message that includes a status code indicating a successful result that may contain data for return values our out parameters or a code indicating an exception with the relevant data about the exception. User exceptions are a little different than non-user exceptions, but that it isn't particularly relevant at this point but <b>is important</b> later. This reply message contains the request Id of the originating request.</li>
</ul>
<ul>
<li>AMD- either the thread that read the message from the connection and dispatched the request <b>or</b> a different thread creates a reply message with a status code indicating a successful result or not along with any data to be returned as a result of the call. The reply message also contains the request id of the originating request.</li>
</ul>
<p>The thread responsible for creating the reply message and sending it to the network depends on the dispatch mechanism and the mechanism used to inform the runtime. Problems sending the reply may be logged, but are usually otherwise ignored. If the runtime cannot successfully communicate with the client, there is not much else to be done. The results of the operation are not rolled back either. This is one of the key points: as far as as the servant is concerned, once it provides a result it is done.</p>
<h2><a name="SupportingFailover-UnderstandingWhatIceisDoing-AndDataPasses...NetworkCommunications"></a>And Data Passes... Network Communications</h2>
<p>Sending a reply to the client is basically a "best-effort" mechanism. If a connection is lost, the reply has no real meaning since the id for the original request was connection specific and matching it with a request from another connection should never occur. There is not much the server can do to recover at this point.</p>
<h2><a name="SupportingFailover-UnderstandingWhatIceisDoing-TheClientSideReplyProcessing"></a>The Client Side - Reply Processing</h2>
<p>As already mentioned, only the reply that has a value matching a valid pending request is used when it arrives. Incoming replies that for previous attempts are simply discarded. As replies are read and matched up by an Ice Communicator thread, the discarding of the now-unnecessary replies is transparent to the caller.</p>
<h1><a name="SupportingFailover-UnderstandingWhatIceisDoing-WhyIsthisImportant%3F"></a>Why Is this Important?</h1>
<p>Understanding the interaction between the client code, the Ice communicator and the underlying network informs our expectations when devising failover strategies. How requests are dispatched to servants and how the server side handles replies informs how we implement our components, including how replication is implemented. It is also useful to think of the client and server sides as strongly decoupled. The decoupling is necessary to allow many desirable features to be easily supported. For example, indirect object identifiers, replicated servants, transparent multi-profile endpoint profiles, request routing, load balancing, typed notification services all benefit from a foundation that decouples the client and server sides. The trade-off is that more thought to what the middleware is doing is occasionally required when building systems that do non-trivial things. </p>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;" class="grey">
<a href="https://wiki.asterisk.org/wiki/users/removespacenotification.action?spaceKey=TOP">Stop watching space</a>
<span style="padding: 0px 5px;">|</span>
<a href="https://wiki.asterisk.org/wiki/users/editmyemailsettings.action">Change email notification preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Supporting+Failover+-+Understanding+What+Ice+is+Doing">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/pages/diffpagesbyversion.action?pageId=19009347&revisedVersion=11&originalVersion=10">View Changes</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Supporting+Failover+-+Understanding+What+Ice+is+Doing?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>