The Inferno network operating system provides several mechanisms to enable the development of programs with components distributed across multiple platforms. Inferno supports a uniform file interface to all resources (files, devices, and programs) and a powerful set of name space operations for creating the view of those resources required by an application. It also provides several communications mechanisms. These powerful capabilities support rapid development and delivery of dynamically distributed applications. This results in network applications that have the following attributes:

• Fault tolerance for application recovery

• Transparent load balancing of server processes
• Multi-server scalability

• Multi-client process sharing

• Server process location transparency

The Inferno network OS combined with the presented application distribution framework can offer these features without application interruption.

The dynamic distribution process is realized through the Inferno name space construction operations: mount and bind. A module is selected for distribution from the original application: MOD. It is assumed that MOD can be separated (i.e. MOD does not share data with other application modules). The application distribution process between a Device and a Server requires three simple steps:

1. Starting a server process to provide a channel interface that preserves the MOD API and reuses (executes as is) the original MOD module.
2. Mounting the server name space on the device (client) and binding the server channel to the appropriate location (the channel appears locally to the client).
3. Shadowing the original MOD module on the device using a client template that preserves the original MOD API and communicates via the local channel.

If MOD is dynamically loaded on demand by the original application and the communication interface between MOD and the rest of the application modules is stateless (e.g. a request based protocol like HTTP), then the three operations required for distribution can be executed while the application is executing. Dynamic distribution of the live application is possible in this case. In addition the distribution process does not require any change to the original application code. The following figure illustrates the distribution process.

The server executes the target MOD process as well as the server template process that invokes it (and preserves the MOD API.) To achieve this, the channel framework can execute on the server with the server template module and channel file name as its two arguments. The server is then ready to take requests.

Using the name space operations of mount and bind we can decide when the distribution is to occur and where the distributed processes are located. If the client application loads the target module dynamically, we can place the client template in the load location expected by the original application. This is done via a bind operation using the -b (before) option to shadow the real target module.

The template resides on the client or it can be obtained transparently from the server via name space configuration. Ideally we mount the server to some mount point on the client. Then bind the directory where client template resides to shadow the local module. Finally we bind the server channel directory to the local channel name space. At this point, the client application invokes template functions that are transparently called on the server.

The dynamic distribution process supports hot spare server replacement:

• A group of identical server processes is distributed on different machines.
• A name space manager (on the client) rebinds over the existing channel directory. The client application accesses a different server process each time.

• Switching server is transparent to the client application (under the above assumptions.)

Since the bind –b operation provides shadowing and does not remove the existing file we can bind over the channel file during an existing read/write sequence and not interfere with the current transfer. Subsequent requests will move to the new server without interruption. This is a very powerful mechanism for distribution and dynamic reconfiguration.

This strategy provides a mechanism to allow dynamic management of resources on the network. Included is the recovery of application services. The ability to dynamically bind a local name space creates a virtual view of all available processes in the network. As a particular resource becomes too busy the client can dynamically reconfigure its name space to point to the server that has a lighter load. This can occur without service interruption and even without user intervention. Additionally the recovery after failure is administrated in the same fashion. This process does not make any claims about the recovery of the state of the application on the server.

• Location Transparency

The distribution process is done without the application knowing where the server processes reside. This supports the separation of network and environment configurations from the application, and configuration decisions can be made dynamically.

• Multi-server scalability

An application can be broken up into several pieces and therefore distributed to multiple servers. In addition the server processes may also be distributed. The location of these modules at any point in time is transparent to the application.

• Load balancing

The applications can be distributed on the servers to optimize loading and can be moved dynamically as new resources become available. The load balancing operations can be managed by the client or centrally by a resource management server.

• Fault-tolerant Recovery

As resources fail clients may be dynamically reconfigured to new servers providing transparent service recovery. This mechanism does not attempt to maintain state for an application.

• Multi-client sharing

The distribution of modules allows the separation of services on many servers as well as the sharing of these services among many clients (and personalized services per client). A browser could have its communications module distributed to serve many clients, benefiting from the caching of web pages, larger pipe for downloads, etc. This could also be used to provide filtering or text-only services to thin clients, text to speech for wireless devices, or personalized services per authenticated user.

Distribution does not come for free. The application must be designed as independent (e.g. no memory sharing or call dependencies), therefore allowing movement to a separate process space. Typically it will have only a single entry point, however this is not mandatory. The client and server templates can be designed to accommodate multiple entry points although this would require a more sophisticated protocol. The means of communication is via byte arrays and the application developer would need to add dispatching information on to the byte array agreed to by both parties to allow the server template module to properly dispatch to the proper target calling module. Additionally this is designed as a transaction based protocol. Large transfers of information may exhaust the available buffer space on the client side, since the read is assumed to complete with one transfer. A generalized solution can be developed allowing block reads and writes.

The following describes an application distribution case study based on the new Inferno browser. The new browser architecture created by Doug Riecken’s team for Inferno release 2.0 is organized as a group of small load-able interacting modules (called software beads). Julie Carroll, John Keane and Alex Kononov designed a very well beaded browser software increasing the level of parallelism for optimal rendering, reducing the amount of memory requirement, and allowing experimentation with various distribution strategies. The communication module was selected for distribution. It has the right properties to be distributed and has the necessary functional attributes. The following figure illustrates the resulting browser architecture and its distribution.

Since the browser application is designed to execute on a small standalone device, the communication module is not originally ready for distribution. The module uses a function call interface to process communication request. However, module separation is possible without code change because the module does not depend on any other browser modules (except the marshalling and un-marshalling functions.) We use a shadowing technique that enables channel communication without changing the browser code. It also makes sense to distribute the communication module because:

1. It executes in parallel with user interface and rendering functions.
2. Communication may be more effective on the Server side (closer to network access).
3. Caching space is available on the Server side and may be shared by user groups.

A simple case of server process distribution can easily be implemented using the file2chan interface. In the following figure, a process named File2chan creates a channel file under the /chan directory called FileA using file2chan. A listing of the /chan directory on the server now shows the file /chan/FileA. A client process on the same system can open this file and perform read and write calls (just like any normal file) to communicate with the server process.

Since the FileA channel file is to the Inferno OS just like any other file it can be manipulate using the name space operators. To execute the same client process on a different machine than the server, we use mount on the client to present the server name space to the client as if it were local, and bind to map the server /chan directory to the client /chan. Now when the client process runs it performs the same operations as before except that it transparently accesses the server process on a different machine. We have simply created client and server processes that can be distributed across any machine and any network. The Inferno name space makes server access available to any client without requiring any specific network code in either process.

We have developed a channel framework using file2chan that allows the plug-in of a specified Limbo module that conforms to the target module protocol. We have applied this strategy to the new Inferno browser. The module that was chosen for distribution was the communication process. The distribution was accomplished without modifying any code in the current browser. The interfaces required to develop this and any new distributed modules are outlined below. Three software modules are used:

1. A generic channel framework on the server.
2. A client process template.
3. A server process template.

The client and server templates are necessary in this example because the communication module is not readily designed for inter process communication.

• Channel Framework

For each server module in the network a channel framework must be running. It provides the implementation of the channel file and the acceptance of requests and sending of replies to the client processes. The channel framework is capable of handling multiple requests from multiple clients. It takes two arguments:

1. The server template process to invoke upon receiving a request from a client
2. The name of the channel file used for client and server communications.

• Server Side Template Module

The server side template implements a standard interface that is known to the channel framework. The framework loads this template process. The server template is specific to an application providing the functions to unmarshal the request data from the channel (byte array) to the format required by the target process. It also marshals the reply from the target process before returning on the channel.

• Client Side Template Module

The client side template implements the same call/return interface that is used by the target side process. The client template will be loaded by the main client application in place of the real target. The client template opens the channel for read/write access, marshals the request (byte array), writes the request, reads the reply, unmarshals the data to the application specific format and return the reply.

The inferno operating system presents numerous facilities for distributing application modules throughout communication networks. The use of the Inferno file2chan interface provides a simple mechanism to present a software module as a file interface and access it remotely via Inferno name space. This approach enables the design of applications for seamless distribution throughout communication networks. The properties of shadowing (bind –b) allows the transparent movement of processes giving way to load balancing, fault tolerance, use of multiple servers and access of server processes by multiple clients. The function of transparency applies to clients, servers, multi-hopping of resources, and re-distribution of already distributed services. The framework presented here offers a mechanism (with certain restrictions) to distribute a stand-alone application on networked servers without changing it. The framework allows seamless transition between a fully resident process and one that is dynamically distributed. It should be noted that when the process is not distributed it maintains its original calling interface avoiding any communications overhead. The choice of modules for distribution remains a careful design decision. However, the ability to rapidly experiment with various design partitions for an application is remarkable and unique to the Inferno network operating system. Dynamic partitioning could enable a service application to adapt to network resource loads and varying client configuration and demands.