<html>
<head>
<base href="https://wiki.asterisk.org/wiki">
<link rel="stylesheet" href="/wiki/s/2030/1/7/_/styles/combined.css?spaceKey=TOP&forWysiwyg=true" type="text/css">
</head>
<body style="background: white;" bgcolor="white" class="email-body">
<div id="pageContent">
<div id="notificationFormat">
<div class="wiki-content">
<div class="email">
<h2><a href="https://wiki.asterisk.org/wiki/display/TOP/Ice+Scripting+Language+Mapping+Implementation+Overview">Ice Scripting Language Mapping Implementation Overview</a></h2>
<h4>Page <b>added</b> by <a href="https://wiki.asterisk.org/wiki/display/~beagles">Brent Eagles</a>
</h4>
<br/>
<div class="notificationGreySide">
<h1><a name="IceScriptingLanguageMappingImplementationOverview-IceScriptingLanguageMappingImplementationNotes"></a>Ice Scripting Language Mapping Implementation Notes</h1>
<p>This page describes some of the details of how the Ice for Ruby, Python and PHP language mappings are implemented. This is an "unofficial" description; if there the description herein does not agree with the source code, the source code should be considered the authoritative version. It is not intended to be exhaustive, but focusses on selected areas that may aide developers wishing to understand how the scripting language mappings work.</p>
<h2><a name="IceScriptingLanguageMappingImplementationOverview-Foundations"></a>Foundations</h2>
<p>This section describes the aspects of the Ice API and source code that form the foundations of scripting language mappings. These are:</p>
<ul>
<li>The Ice utility library</li>
<li>Slice parsing and code generation</li>
<li>Data marshalling, unmarshalling and request dispatch</li>
</ul>
<h3><a name="IceScriptingLanguageMappingImplementationOverview-TheIceutilitylibrary%28IceUtil%29"></a>The Ice utility library (IceUtil)</h3>
<p>The Ice product "went public" in February 2003. Before that it was being used as the foundation of a related project that required high performance and what could be called "considerable" scalability in both Java and C++. Key tenets of the Ice project have always been portability, consistent clean implementation, and adherance to best practices. At the time, Ice was crafted to compile and run on the following operating systems and development tools:</p>
<ul>
<li>AIX with VisualAge C++</li>
<li>FreeBSD with GNU C++(unofficial)</li>
<li>Linux with GNU C++ and Intel compiler</li>
<li>HP-UX with aCC</li>
<li>OSF/1 with Tru64 C++</li>
<li>SunOS GNU C++ and Sun Workshop C++</li>
<li>Windows 2000, XP and on with Visual Studio 6, 7.1 and newer as the became available.</li>
<li>MacOS X/Darwin with XCode C++</li>
</ul>
<p>In the early days of Ice, cross platform libraries for developing concurrent distributed systems in C++ were inadequate. Having considerable experience in implementing systems of this sort, ZeroC simply implemented their own. The IceUtil library contains portable support for:</p>
<ul>
<li>threads</li>
<li>thread-safe reference counted objects</li>
<li>UUID generation</li>
<li>thread synchronization objects</li>
<li>option parsing</li>
<li>timers</li>
<li>UTF-8 to UTF-16 to UTF-8 conversion</li>
</ul>
<p>The script language mappings use the IceUtil implementations wherever they are required. This allows the code across the particular implementations to be as similar as possible. In order to maintain license compatability and consistency across language mapping implementations, developers extending or modifying the language mappings are encouraged to do the same. This is also important if you are implementing a custom language mapping that you are considering contributing for inclusion with future Ice distributions.</p>
<p>This history lesson is also relevant if you find yourself asking "why didn't they just..." when looking at the C++ code or some of the generated code for the various languages. Some idioms that are common place today were not widely known when the code was originally written and may also not have been widely portable. Of course, it's also possible that the developers simply did not know that there was a "better way". Nobody's perfect!</p>
<h3><a name="IceScriptingLanguageMappingImplementationOverview-Sliceparsingandcodegeneration"></a>Slice parsing and code generation</h3>
<p>All Slice parsing, regardless of the language mapping, uses code contained in the Ice C++ libraries. As the Ice source distributions are organized by implementation language, you will find the code related to parsing Slice in the "cpp" subdirectory. The subdirectories of interest, relative to the cpp directory, are:</p>
<ul>
<li>include/Slice - the header files for the Slice parsing library</li>
<li>src/Slice - the core Slice parsing library</li>
<li>src/slice2py - the Slice to Python translator</li>
<li>src/slice2rb - the Slice to Ruby translator</li>
<li>src/slice2php - the Slice to PHP translator</li>
</ul>
<p>The src/Slice directory includes the implementation of the Slice parser, the implementations of the C++ objects that are created during parsing and some language specific support code. The other directories contain implementations of command line tools to generate native code for specific scripting languages.</p>
<p>The parser supports preprocessing through the libmcpp C-style preprocessor to perform file inclusion and compile time logic. The parser implementation uses bison and flex to generate the parsing and lexical analysis code. Modifying this code is generally unnecessary unless you wish to modify the Slice language, which is generally inadvisable. The Parser.cpp and Parser.h files contain the C++ classes that are instantiated and initialized during the parsing process. It is advisable to review this code to become familiar with what is available when modifying the generating code for the translators.</p>
<p>There are several C++ files of the form <a href="/wiki/pages/createpage.action?spaceKey=TOP&title=prefix&linkCreation=true&fromPageId=9568420" class="createlink">prefix</a>Util.<a href="/wiki/pages/createpage.action?spaceKey=TOP&title=cpp&linkCreation=true&fromPageId=9568420" class="createlink">h</a>, where the prefix is related to a language mapping. These files contain generator specific helper files. Some of these files are relatively small while others contain nearly the entire implementation of the generator. PythonUtil.cpp and RubyUtil.cpp are examples of the latter. In these cases the generator is basically in "library form" because these scripting language implementations support "on-the-fly" translation of Slice to their native language. Python and Ruby developers are not required to run the command line Slice generators, but can <b>load</b> the Slice directly into the Ice application with the Ice.loadSlice() API. The Ice.loadSlice() call performs all of the same operations running slice2py or slice2rb would, but immediately runs the generated code in the current scripting instance. This is very handy for prototyping and situations where a build system step is impractical.</p>
<p>The inclusion of language specific helpers in the Slice parser library itself is arguably inappropriate. It might be simply to reduce the build complexity for the languages that support dynamic loading of Slice. </p>
<p>The generators themselves are of varying degrees of complexity. In all cases, the generators invoke the parser on the Slice files indicated in the command line arguments, implement the file and directory management for producing the output files and write the generated code statements to the files.</p>
<p>The code generators work by traversing the collections of C++ objects that were creating during the parsing phase. The generator code is implemented as several specialized visitors that produce the generator output. The entire collection may be visited in multiple passes as needed.</p>
<p>For developers wishing to experiment, there is no reason a generator has to actually target an Ice language mapping or even a programming language. Other examples of Slice translators are:</p>
<ul>
<li>Slice to DDL for SQL databases</li>
<li>Type specific helper classes</li>
<li>generation of boiler plate code such as special servants and servant factories</li>
<li>test code generation</li>
</ul>
<p>Access to embedded metadata and comments further expand the possibilities.</p>
<h3><a name="IceScriptingLanguageMappingImplementationOverview-Datamarshalling%2Cunmarshallingandrequestdispatch."></a>Data marshalling, unmarshalling and request dispatch.</h3>
<p>Each of the script language mappings uses the C++ Ice implementation to marshal and unmarshal data and invoke requests. As Ice provides a documented API for dynamically creating and reading streams for arguments and responses as well as constructing requests (see Dynamic Ice in the Ice manual), using internal undocumented APIs is not necessary. However, implementing a mapping from scratch is not for the uninitiated and covers a very large range of topics related to Ice, object oriented middleware and extension APIs for the target scripting language.</p>
<p>The language mappings that come with ZeroC Ice are implemented in C++ and use the C++ version of the Ice runtime. The relevant languages and their main distributions had C based extension APIs, so basing the implementations on C++ was the natural choice. However, the required Ice APIs are also available in Java and C#. If an alternate language mapping is being implemented for a language that has extension APIs in Java or C#, the implementor can simply use the most appropriate Ice implementation. </p>
<p>This section is only relevant if the reader is attempting to modify a scripting mapping or implement their own. Simple translator extensions to produce utility libraries, documentation, etc. need not employ any of the mechanisms described below. It is recommended that a developer gain familiarity with developing Ice applications using the existing language mappings before proceeding.</p>
<p>The Ice manual has a complete description of <b>Dynamic Ice</b>. A strong understanding of the the <b>Ice Protocol</b>, <b>Ice Run Time in Detail</b> and <b>Object Life Cycle</b> is essential to understanding the existing language mappings or developing a custom language mapping. The key points that need to be fully understood are outlined below.</p>
<h4><a name="IceScriptingLanguageMappingImplementationOverview-DynamicIce"></a>Dynamic Ice</h4>
<p>Dynamic Ice is the Ice API for creating requests programmatically and preparing and reading argument <b>streams</b>. Streams are sequences of bytes that contain <b>encapsulations</b> of marshalled data. There are separate streams for input (unmarshalling) and output (marshalling) which are used for responses and in parameters respectively. A response may be received through a return value, exception or <b>out</b> parameter. Operations are invoked using the ice_invoke() family of member functions that are defined on the Ice proxy objects. Dynamic processing of requests on the server side is accomplished by creating a class deriving from Ice::Blobject and registering it as a servant on the appropriate object adapter. There is a simple demo application (Ice/invoke) and the Ice Glacier2 service uses dynamic Ice for forwarding requests. There is also the existing scripting language mappings as well.</p>
<h5><a name="IceScriptingLanguageMappingImplementationOverview-Encapsulations"></a>Encapsulations</h5>
<p>Understanding encapsulations and the Ice protocol is regrettably quite important. Slices and how they relate to hiearchies is essential. Ice also uses and object mapping and indirection scheme that preserves pointer semantics within an encapsulation. Once the necessity of methods such as readPendingObjects() and writePendingObjects() is understood and what type of activity is performed prior to and during such calls, the developer has probably grasped the essential points. Other details such as type name indexing through writeTypeId() and readTypeId() is also important, as well as the finer distinguishing points between objects and exceptions. Dealing with primitive types and even aggregates of primitive types is straightforward fortunately, however a complete language mapping will need to support all types. </p>
<p>A detail of the existing scripting language implementations that is worth mentioning is that once an object has been completely read from an encapsulation, the implementation does not hold on to a Ice specific C++ representation of that object. There may be a C/C++ level construct that represents the scripting language object (PyObject, zval, etc) but a representation that can be directly manipulated by the Ice runtime does not exist. If a scripting implementation needs to maintain some information about the original object for some reason, then it must carry this data attached to the language specific construct is some fashion. An example of this is the Ice::SlicedData object that may be created when reading objects that may have been sliced. Please note that this is a recent, Digium developed extension that will be included in a future version of Ice. For unsliceable classes to function properly, the Ice::SlicedData object must be associated with an scripting object as opaque data so that it may be referenced if that object is written as part of an Ice request later on.</p>
<h5><a name="IceScriptingLanguageMappingImplementationOverview-Moreonreadingobjects"></a>More on reading objects</h5>
<p>Ice provides a mechanism for passing objects by value. When an object is encountered when reading an encapsulation, it is necessary to construct an instance of some type to read the values of the member types. This is referred to as the Object Factory. In the non-scripting language mappings, default object factories are automatically generated so that classes that do not define methods can be used without further coding. The scripting language implementations don't have direct access to the object factory API and need to use an alternate method for creating objects. Ice allows for a <b>default</b> object factory to be registered which will be invoked whenever an object must be created when unmarshalling the objects from an encapsulation. A scripting implementation would register a default object factory and use the type id of the object to lookup the type information for the object being read. </p>
<p>As evident from the previous paragraphs, a scripting language implementation must now how to deal with types that may be read when dealing with an Ice request or response as well as constructing a request. The parsing and code generation is where an implementation would typically create a dictionary of types and requests containing the required information. </p>
<h3><a name="IceScriptingLanguageMappingImplementationOverview-Somelanguagespecificnotes%3A"></a>Some language specific notes:</h3>
<h4><a name="IceScriptingLanguageMappingImplementationOverview-IceforPython"></a>Ice for Python</h4>
<p>Python is currently the only <b>complete</b> implementation as it supports both servers and clients. Compared to C++, Java and C#, it is not <b>quite</b> complete in that it does not provide an implementation of IceBox etc. The IcePy implementation is a good place to go to see an example of both client side and server side implementations.</p>
<h4><a name="IceScriptingLanguageMappingImplementationOverview-IceforRuby"></a>Ice for Ruby</h4>
<p>Ruby currently only supports client side Ice. This is primarily due to issues with early implementations of Ruby threading. It is possible that the current Ruby runtimes and extension APIs could be used to quickly construct a server side implementation, particularily if the Python implementation can be used as a guide. There are some commented out sections in the Ruby implementation that were kept as possible future parts of a server side implementation.</p>
<h4><a name="IceScriptingLanguageMappingImplementationOverview-IceforPHP"></a>Ice for PHP</h4>
<p>PHP also only supports client side. The PHP implementation is a little less complete as there are no place holders for marshalling exceptions as exists in Ruby. In general, it is hard to conceive of why a PHP application might need a server side implemenation. However, it is possible that support for callback objects might be desirable for some in the future. Currently, there has not been a lot of interest in this area. Consequently not a lot of investigation has been done. PHP is also the only scripting language implementation that does not support loading Slice at runtime. </p>
<h3><a name="IceScriptingLanguageMappingImplementationOverview-TODOforscriptinglanguages"></a>TODO for scripting languages</h3>
<p>The most obvious standout for improving Ice's scripting language implementations is to add server side support to Ruby. Modifications to the mappings might also be proposed to make them more "idiomatic" to currently accepted best practices.</p>
</div>
<div id="commentsSection" class="wiki-content pageSection">
<div style="float: right;">
<a href="https://wiki.asterisk.org/wiki/users/viewnotifications.action" class="grey">Change Notification Preferences</a>
</div>
<a href="https://wiki.asterisk.org/wiki/display/TOP/Ice+Scripting+Language+Mapping+Implementation+Overview">View Online</a>
|
<a href="https://wiki.asterisk.org/wiki/display/TOP/Ice+Scripting+Language+Mapping+Implementation+Overview?showComments=true&showCommentArea=true#addcomment">Add Comment</a>
</div>
</div>
</div>
</div>
</div>
</body>
</html>