Figure 1: Although the Remote Procedure Call system somewhat simplified the job of a client-server system developer, it still requires running separate interface files through a preprocessor and filling in the resulting stub files.
$Id: jedipaper.html,v 1.103 1997/11/21 11:20:34 adam Exp $
$Date: 1997/11/21 11:20:34 $
A snapshot of this paper is also available in Word and Postscript format.
The Java Environment for Distributed Invocation (JEDI) is efficient, dynamic, and easier to use than alternative communication systems for distributed Java objects. Existing techniques for invoking methods on remote objects in client-server systems provide insufficient support to the developers who use them. Current state-of-the-art mechanisms for remote method calls on Java objects, such as RMI, require users to perform a complicated series of steps. Furthermore, the compiled static interfaces these systems use limit their functionality. This paper presents the design and implementation of JEDI's simpler alternative approach utilizing dynamic proxies while eschewing compiled static interfaces. A means of integrating JEDI with a publicly available CORBA ORB is discussed next, followed by the tests used to ensure the robustness of the JEDI system. Comparing this system's performance with that of other communication facilities such as UDP, TCP, RMI, and the Infospheres Infrastructure demonstrates the efficiency of JEDI. A calendar-scheduling application illustrates the flexibility and usability tradeoffs of employing JEDI in distributed client-server applications.
Java programs can use the Internet [Gosling et al., 1996] for distributed computations in many different ways. One such technique involves message passing [Hoare, 1978] between objects on different machines, as exemplified by Caltech's Infospheres [Chandy et al., 1996], IBM's Aglets [Lange and Oshima, 1997], and the iBus multicast system [Maffeis, 1997]. Another technique involves accessing remote objects through a request broker active on a remote machine using CORBA [OMG, 1995] or DCOM [Chappell, 1996]. Some systems communicate with remote objects through a gateway to a Web server using HTTP and CGI [ Berners-Lee et al., 1992]. Method calls on remote objects may be made using Open Network Computing Remote Procedure Calls [Srinivasan, 1995] or Java's Remote Method Invocation [Java RMI, 1997]. With each of these techniques, the programmer must deal with creating extra interfaces (often in a different language) and must do other setup work to handle low-level communication details.
This paper explores remote method calling facilities that automatically handle some of the more cumbersome communication and synchronization responsibilities [Peterson and Davie, 1996]. Many existing systems, including Java's RMI, require a programmer to run interface code through a pre-processor to create stub and skeleton objects. We have developed an alternative system for remote method calling, offering the programmer complete control over communication while simplifying the model of distributed computing. Since our system uses Java's serialization capabilities, a programmer can automatically send any object to a remote machine.
First, we will discuss and evaluate several existing systems, including RPC, RMI, CORBA and IIOP, DCOM, and Infospheres. These systems motivate the design and implementation of JEDI, a system that allows dynamic method invocation (the ability to call any method of a remote object at run-time without relying on statically compiled interfaces) and requires fewer development steps than other existing systems. Then we describe how a developer would use JEDI. Next, we discuss several experiments in client-server computing to determine the flexibility, scalability, and ease-of-use of the JEDI system. These experiments include studying the integration of JEDI with CORBA, testing the performance and reliability of JEDI, and using JEDI to develop an application from scratch. We conclude with a short summary of our JEDI findings.
Until recently, Java lacked a native client-server method invocation paradigm [Resnick, 1996], but several supplementary systems are available to provide this functionality. In this section, we explore and compare some of the current communication mechanisms in client-server systems: RPC, RMI, CORBA and IIOP, DCOM, and Infospheres.
For years, programmers have used ONC's Remote Procedure Call (RPC) system [Srinivasan, 1995] to automate some of the communication tasks between client programs and their servers. Although RPC was one of the first systems to simplify the development of distributed applications over the use of plain socket connections, RPC does not handle remote procedure calls automatically. The programmer must first design the interfaces (step 1 in figure 1) on both the client side and server side so they will connect properly when the distributed system is invoked.
Next, the user creates a .x file (step 2) that specifies the designed interface. This file is then run through rpcgen (step 3) to construct client and server stubs. The programmer then fills in the client and server stubs produced by rpcgen with code that implements the desired behaviors (step 4). This code is then compiled for execution (step 5). Some consider this system to be more straightforward than standalone sockets or message passing, because the client can invoke operations on the server by using what looks like a local procedure call. Making network communications into procedure calls meshes well with the top-down design techniques of procedural programming. However, the RPC system requires that the programmer tediously set up all the remote methods in the interface description file, run the preprocessor, and deal with the resulting stub files.
Figure 1: Although the Remote Procedure Call system somewhat simplified the job of a
client-server system developer, it still requires running separate interface files through a
preprocessor and filling in the resulting stub files.
The Remote Method Invocation (RMI) system furnished by Java 1.1 allows RPC-like access to remote objects [Java RMI, 1997]. RMI includes support for data serialization, remote class loading, and socket manipulation.
To use RMI, an application developer creates a Java interface for the object to be accessed remotely (step 1 of figure 2), and then writes a server class to implement this interface (step 2). The Java compiler compiles the server code (step 3), and the RMI preprocessor rmic uses the resulting class to create the server skeleton and client-side stub (step 4). The client may be written and compiled any time after the server interface is written (steps 5 and 6). Before the client can access a remote object on a server (step 7), the RMI registry must be run (step 8), the server object must be created in a Java VM (step 9), and the server must register itself in the RMI registry (step 10).
Figure 2: RMI improves the RPC-like access for remote Java objects, but adds several extra
tasks for the developer.
We believe that the extra preprocessing steps introduce unnecessary complexity. They add several additional object files to the compilation process, and restrict the methods that can be run on remote objects to those that are described in a static interface. If a server object is updated to include new functionality and a new interface, the client will be unable to use the new interface without resorting to Java's somewhat awkward reflection capabilities.
RMI is difficult for developers to use, because it does not allow client programs to use new functionality in server programs without first recompiling (and redistributing) the client programs or using Java's reflection to get around this static limitation of RMI. Another cumbersome feature of RMI is its requirement that programmers must use a separate stub compiler to generate code for the server and client-side stubs; this feature is also part of the CORBA system, described in the next section. RMI can use only a few wire protocols (currently TCP/IP and HTTP), but some applications would benefit from the use of custom transport protocols available through a generic message infrastructure. The ACE framework [Schmidt, 1994][Jain and Schmidt, 1997] and the iBus project [Maffeis, 1997] both provide a layered component-based Java communication system that allows plug-in custom transport protocols to provide different quality of service facilities to applications.
ObjectSpace Voyager [ObjectSpace, 1997] provides remote method invocation facilities much like RMI's, but makes the development process much simpler and provides additional features. Developers run an existing class through the Voyager preprocessor to create a stub class with all the methods the original class had. This saves them the work of writing an remote interface file and changing their code to implement the interface. Although Voyager allows dynamic method calls, it requires developers to specify methods with the unintelligible method signature syntax used by the Java virtual machine.
The Object Management Group's (OMG) Common Object Request Broker Architecture (CORBA) allows the development of distributed applications with component objects [Vinoski, 1997]. CORBA's language-independence allows objects written in different languages to communicate with one another. All object interaction is routed through intermediary Object Request Brokers (ORBs) which communicate through the industry-standard IIOP protocol (see figure 3). CORBA uses client and server stubs created from an interface definition written in the ISO Interface Definition Language (IDL).
Figure 3: Through the CORBA bus, client objects send requests for method invocations to
the remote ORB, which routes the request through the object adapter and server skeleton to
the server object.
To create a Java-based client-server application in CORBA, the programmer first writes a IDL file defining the signatures of the methods that need to be called remotely. The IDL file is then run through a Java preprocessor which creates an interface for the server object and a stub class that will forward method calls to that server object. Then the programmer writes the client program and an object that implements the server's interface. The client and server can then be compiled and run.
CORBA has support for clients to discover and use interfaces dynamically through its Dynamic Invocation Interface (DII) [OMG, 1995]. When using DII, the client creates its method calls at run-time, rather than calling methods in the stub. A CORBA specification for the Dynamic Skeleton Interface (DSI), allows server objects to update their interfaces at run-time [OMG, 1995]. Any method invoked through the DSI is passed through a single upcall method (written by the programmer) that is responsible for checking the method name and forwarding it to the correct implementation method.
We looked at two particular ORB implementations. Xerox PARC's Inter-Language Unification (ILU) system [Janssen and Spreitzer, 1997] interoperates with other CORBA ORBs using the Internet Inter-ORB Protocol (IIOP). Although ILU does not implement many features of the commercial ORBs [Eng, 1997], it provides DII and is freely available. As described in our section "Experiments in Client-Server Computing", we have worked on an ILU interface that allows CORBA objects to call methods on JEDI objects.
VisiBroker, Visigenic's implementation of CORBA, has similar features to ILU but includes more complete functionality. Caffeine, a part of VisiBroker, includes a compiler that generates IDL code from a Java interface, making the CORBA development process in Java much like that of RMI.
Microsoft has recently developed a Java interface to their Distributed Component Object Model (DCOM) [Chappell, 1996]. DCOM is another system that allows RPC-like calls on remote objects; it uses a DCE-like IDL language to define interfaces. Note that Microsoft's IDL (MIDL) is compliant with neither CORBA IDL nor DCE IDL.
Like CORBA, RMI, and RPC, DCOM requires compiling interfaces written in its IDL into stub objects. However, DCOM has the added complexity of requiring that a type library for the object be created as well. The server object must then be written to implement the defined interface. The client code is fairly straightforward, but DCOM objects can only be accessed through an interface, not directly. Also, both the client and the server must register the DCOM object with the operating system before the client may access it. To use a reference to a local DCOM object in Java, a program must first cast the object to its corresponding interface before using the class. Although several ports are planned for the future, DCOM is presently available only on Windows 95 or NT systems.
DCOM Automation allows clients to make dynamic method calls. By exposing one or more dispinterfaces (a set of methods that can be called dynamically), an object can make methods available to clients that were not compiled with DCOM interface stubs. The client packages up the arguments to the call in a variant data type and combines this with an integer ID denoting which method to call. These parameters are passed to the Invoke method of the DCOM interface IDispatch. When the Invoke call reaches the server, the destination object must check the ID in order to discover which method is being called before unpacking the variant parameters and implementing the call.
Comparing each of these distributed object communication mechanisms [Orfali and Harkey, 1997], we note that RMI, IIOP, and DCOM all offer somewhat seamless Java integration, typed parameter support, and reasonable performance. However, all three approaches suffer from high setup costs due to programming complexity, lack of configuration ease, evolving wire-level security, and limited dynamic discovery and dynamic dispatch when compared with systems such as NeXT's Portable Distributed Objects [Prabhakar, 1995]. Furthermore, although IIOP was designed to scale to accommodate communication among many objects, neither DCOM nor RMI presently seems suitable for communication among more than a handful of objects.
The Infospheres Infrastructure [Chandy et al., 1996] offers a solution to the scaling problem by providing mailboxes that can send and receive typed messages. With these mechanisms, developers can set up sessions of persistent communicating objects [Chandy and Rifkin, 1997]. JEDI was originally constructed as an invocation layer built on top of the Infospheres message-passing communication layer. As the package evolved, the Infospheres plumbing was replaced by a more performant communication layer using UDP.
Work is proceeding on the design and implementation of the second generation of the Infospheres Infrastructure, which integrates JEDI's invocation facilities. With this design, we hope to make JEDI even easier to use and provide features like security, authentication, and an even more flexible and performant communication layer.
Originally, JEDI was designed to make access to remote objects completely transparent to the programmer, handling all of the networking and synchronization details. As the package developed, we realized that complete transparency is not always desirable in distributed systems. Several unique characteristics of distributed systems, including uncertain delays and distributed failure, must be considered when designing such a system [Waldo et al., 1994]. Unfortunately, when the distributed nature of a system is not hidden, programmers must often deal with low-level coding issues and with complicated development tools such as RMI. JEDI's focus therefore shifted to making robust distributed systems easier to implement.
We envision a distributed computing model with billions of objects scattered over the globe, interacting with each other via the Internet [Chandy et al, 1996]. Because objects in one Java VM usually communicate with method calls, we believe that a communications system based on remote method calls is conceptually more simple than a message-passing system. Although message-passing is a more general communication framework, developers are more comfortable reasoning about and using method calls to communicate between objects. Unfortunately, many existing remote method call systems are quite complex and have many steps to learn and repeat. JEDI's primary goal is to simplify or eliminate as many of these steps as possible.
JEDI provides a simple and flexible dynamic invocation service. Conventional RPCs, for the most part, are based on static interface definition files. RPC requires a static, compile-time interface, and standard CORBA requires the same. Through the Dynamic Invocation Interface facilities, CORBA clients can discover resources dynamically. However, DII is somewhat difficult to use --- many steps are required to construct a dynamic request object [Orfali and Harkey, 1997]. If the client must query the server for the interface of the method it wants to call, performance may decrease dramatically. DCOM Automation is likewise complicated and difficult to use. With JEDI, calling a method dynamically is a simple process: a client must bind to the object it is calling, and then it can invoke any method by name. Querying the interface of a remote object is as simple as calling getClass() on the object (using JEDI) and then finding its public interface using Java's reflection capabilities.
As the number of objects running on the Internet increases, truly dynamic RPC interfaces will be a necessity, because it will become impossible to take tightly-integrated object systems off-line so that they can be recompiled to produce new static interfaces. With massive distribution, knowledge of precompiled stubs of every object in the network universe just is not practical, because massive distribution requires dynamic typing and construction of messages at run-time. Just as latency and partial failure are inherent aspects of distributed computing, coping with dynamically changing and unknown interfaces is an inherent aspect of massively distributed computing. To that end, JEDI is a remote method call system that can dynamically call any method on any Java object at run-time.
We now describe the design of JEDI, and its use in distributed Java systems.
The simplicity of using the JEDI system is illustrated in figure 4. Instead of involving a stub compiler, JEDI provides a software library for making remote method calls; as a result, any method can be called remotely at run-time. The development process is identical to writing any non-distributed Java program: classes are written, compiled and run. Any object can be called from another machine simply by giving it a name. A client can bind to a remote object by creating a proxy with the remote machine, port, and object name. The client can then call any method of the object with one simple line of source code.
Figure 4: The JEDI package simplifies a programmer's task considerably. The user needs
only to implement, compile, and start up the client and server programs. Servers name their
objects in a registry service so clients can call methods on them.
Unlike most other RPC schemes, no source code changes must be made to an object before its methods can be called globally. This means that methods of any object, including the core library objects like String, can be called remotely--even if the source code for the object's class is not available. Thus, "legacy" Java objects not designed for distribution can nevertheless be integrated into distributed systems without writing the wrapper classes necessary with other RPC systems.
Since JEDI does not require a precompiler, the program does not need to know the signature of any method it will call until the call is actually made. An interesting consequence of this is that JEDI does not require the shutting down of a distributed object to update its interface for new remote calls. Instead, that object can be replaced on-the-fly by an update version providing expanded features and clients will be able to access the additional functionality immediately. By using a dynamic system like JEDI for remote method calls, distributed systems can scale up to more objects.
Because JEDI is a library-based system, it fits more naturally into the usual program-development cycle than precompiler-based RPC systems. In general it is easier to learn to use a software library than to use a new command-line tool. Also, it is easier to understand what is happening inside a software library than to understand the black-box code generated by a precompiler--especially if the library's source is available.
A potential disadvantage of a dynamic scheme like JEDI is that there is no static type-checking. A method call can fail at run-time if the programmer makes a mistake and misspells the method or passes the wrong parameters. However, in a distributed computing system the programmer must be aware that any method may fail because of a failure in the network or in the remote machine. No distributed system can mask the failure of an arbitrary method call. Furthermore, a system like RMI that provides interfaces to remote objects still cannot ensure that the programmer does not try to make an illegal method call -- it just reports the error as an illegal cast rather than a nonexistent method. Thus, the dynamic typing system of JEDI is not a significant potential point of failure for the application programmer.
Java's serialization mechanisms simplify programming for the JEDI system. Any object that implements java.io.Serializable may be passed to a method of a remote object:
public class MyClass implements java.io.Serializable {
// class definition here
}
The JEDI system permits the creation of distributed systems using the intuitive remote method call paradigm without the complexity of many similar schemes. JEDI provides simple but dynamic remote method calls, giving programmers the ability to make run-time modifications.
| Consider a simple remote method call using JEDI, as illustrated in figure 5. Before a method call can be made, the object on which it acts must be registered under a name on the server. This is accomplished by getting the local repository (use Repository.local()) and calling the bind method with the new name and the object to be registered. Then any client can call a method on that object using JEDI. | Repository methods
LocalRepository methods
|
| The client begins this process by creating a Proxy object with the network address (such as "www.caltech.edu") and the name of the object. Once a proxy has been created, any method can be called on a remote object by calling function() on the proxy. There are several versions of function(); the most general accepts a method name and a vector of arguments to that method and returns an arbitrary object. Underneath it all, the JEDI system will send the method call information to the remote machine, which will find the object associated with the proxy. It will then look up the method with the correct name, invoke the method, and pass the return value back to the client. If any exception is thrown in the method, or the method or object cannot be found, or there is a communication error, an exception will be thrown from function(). An example demonstrates these concepts. | Proxy methods
|
This simple JEDI server allows remote clients to call any method on a String object.
import info.jedi.*;
public class ServerTest
{
public static void main(String args[])
{
String string = "Hello";
Repository.local().bind(string,"HelloString");
}
}
To expose an object to remote clients, a user needs to create the object and bind it to a name in the repository. In this example, we have created the string "Hello" and bound it to the name "HelloString" in the local Repository. Under the hood, our call to Repository.local() initialized the JEDI system to listen for incoming remote method calls on the default port. Our server is now ready to accept requests from the following simple client:
import info.jedi.*;
public class ClientTest
{
public static void main(String args[])
throws Exception
{
Proxy proxy = new Proxy("harmonica.cs.caltech.edu","HelloString");
System.out.println("The length of the remote string is "
+ proxy.function("length"));
}
}
This client creates a proxy object to represent the string on the remote machine. In our example, the server is running on harmonica.cs.caltech.edu, so this is the network address we pass to the proxy constructor. Since we registered the object under "HelloString," this is the object name we pass to the proxy constructor. Once the proxy has been created, we can call any function on the remote object. In this case, we call the length() method of the String. We use a convenient version of Proxy.function() that does not take a vector of arguments, because the String.length() method has no arguments. Other versions of Proxy.function() are provided for calling methods with different numbers of arguments. The length() method will return an int, which will be wrapped in an Integer object, passed back over the network to the client, and finally returned by proxy.function(). It will be converted into a string by the concatenation operator, and the output of the client will be:
The length of the remote string is 5
Figure 5: JEDI allows a client to set up a Proxy object through which it makes its remote
method calls. When calling a method through the proxy, the method name and arguments are
sent through JEDI's communication layer to the server, which finds the object in its object
repository and finds the proper method using Java reflection. The method is then called on
the object on the server machine, with the return result shipped over the JEDI wire back to
the client program. New objects on the server can be plugged in on-the-fly, so dynamic
methods can be invoked at run-time.
JEDI includes a simple but powerful general messaging infrastructure. This infrastructure is designed to be both efficient and modular, and is implemented in several layers with a protocol stack architecture similar to that of iBus [Maffeis, 1997]. One layer can use the services of another; thus the remote method call facility uses the reliability layer to make robust remote method calls over an unreliable network connections. Because the layers are loosely connected, a separate messaging service layer could make use of the reliability layer to provide robust message passing, as is done with the ACE system [Schmidt, 1994].
Figure 6: The JEDI MailDaemon sits at a socket, waiting for UDP packets. Upon receipt of a
UDP packet, it routes the packet accordingly to the PingHandler, the RMCHandler, the
ReliableHandler, or some other packet handler. The BSD Daemon is copyright 1988 by
Marshall Kirk McKusick.
| The center of the JEDI system is the MailDaemon class, illustrated in figure 6. Any program needing to communicate uses a MailDaemon to forward incoming UDP packets to the appropriate packet handler. When a packet is received, it is converted into an InputPacket, which associates a DataInputStream for reading the contents of the packet. Then the first byte is examined; this byte indicates which of up to 256 well-known packet handlers will process the packet. Each packet handler must conform to interface PacketHandler, which defines a method that takes an InputPacket and returns true if processing of that packet is complete. If processing is not complete when the packet handler returns, the next byte is assumed to be the next handler. This provides the layering mechanism in JEDI: a packet can first be processed by a reliability layer and then passed on to a higher-level layer that uses data from the packet for computation. The slots that are not yet used are filled with DefaultPacketHandlers, which simply ignore any packets they are passed. | MailDaemon methods
InputPacket fields
|
| A ResourceHandler is created for the MailDaemon, and sets up threads of type ReceiveThread to handle incoming packets. If there is any possibility of a thread suspending during the handling of a packet, it should call ResourceHandler.threadBusy() followed by ResourceHandler.threadIdle() when its work is done. These method ensure there are enough threads to handle incoming packets, and create another ReceiveThread if there are not. In this way many simultaneous packets can be serviced, up to the thread limit of the virtual machine. For efficiency, the ResourceHandler also keeps track of a pool of 64KB packets (which are expensive to create for each incoming packet). | ResourceHandler methods
ReceiveThread methods
|
| OutputPacket is a convenience class for creating, filling, and sending a JEDI packet. It creates a DataOutputStream that fills up a ByteArrayOutputStream. It also provides methods for sending the packet to another host and for resending it if it gets lost on the way. | OutputPacket fields
|
| As a test for the system, the simplest handler provided is the PingHandler. This class simply sends a packet back to the original host. The PingHandler class has a main() method so a program can test the latency of its network connection through JEDI. It also provides a tool to compare the overhead of the JEDI messaging structure with that of a simple UDP ping. | PingHandler methods
|
| The reliability layer is implemented through ReliableHandler. Clients call ReliableHandler.addReliability(p) to make packet p reliable. ReliableHandler extends Thread, and the run() method will periodically resend each reliable packet until it times out or is acknowledged by the remote host. No packet ordering is necessary in a remote method call system, so all available packets will be sent immediately without waiting for acknowledgments. When a reliable packet is received, the ReliableHandler is called. It will check to make sure the packet has not been duplicated, and then send an acknowledge packet so that the sending host knows that the packet has been received. If the remote host cannot be contacted within a specified period, a TimeoutException will be thrown to the caller. | ReliableHandler methods
|
Figure 7: The JEDI package automatically handles the seven steps of a remote method call.
| The JEDI remote method call facility is implemented through several classes. The Proxy object asks the RMCHandler object associated with the local MailDaemon to send the method call out over the network (step 1 in figure 7). The RMCHandler object creates an RMCCall object to represent the method call (step 2). The RMCCall object uses the reliability layer to send a reliable packet to the remote machine (step 3), which is intercepted by the RMCHandler object there (step 4). This handler will create an equivalent RMCCall object on that end of the network, look up the object's name in the local Repository object (step 5), and dispatch the method call using a method from class Remote (step 6). Remote.staticCallFunction() takes an object, a method name, and a vector of arguments, and uses Java's reflection facility to find and call the appropriate method (step 7). The RMCCall object then sends the return value back to the original client, where it is decoded and passed on to the user. | RMCHandler methods
RMCCall methods
Remote methods
|
To determine the flexibility, reliability, scalability, and ease-of-use of the JEDI system, four key experiments were performed. We investigated integrating JEDI with CORBA ORBs, and designed a comprehensive test suite to demonstrate the reliability of the JEDI system. Later, we tested and compared the performance of JEDI with several other systems and compared the implementation of a simple scheduling application using JEDI, RMI, and Infospheres.
We experimented with creating a CORBA object that would allow remote invocation of JEDI objects. The testbed chosen was ILU because it is free and openly available from Xerox. To make a JEDI object accessible to CORBA ORBs using IIOP, we created the following CORBA IDL file to expose an interface to the Remote class:
module jedi {
interface ILURMCCall {
exception JediException{};
typedef sequence < any > Arguments;
any RemoteCallFunction (in string object_name, in string function_name,
inout Arguments arguments) raises (JediException);
};
};
Through this interface, CORBA objects can access JEDI objects through a method similar to the DII interface, as illustrated in figure 8. A CORBA call comes into the ILU system (step 1), where it is decoded and mapped to the ILURMCCall object (step 2+3). ILU then calls the implementation, ILURMCCallImpl (step 4), which accesses the JEDI registry to find the requested object (step 5). We then invoke a function in the JEDI Remote library to call the requested method (step 6). Remote then calls the method and passes back any result or exception information that was generated (step 7). This information is then returned to the original caller through ILU.
Figure 8: The JEDI system can be integrated with a CORBA-compliant object request broker
such as ILU.
To ensure a high-confidence, robust system, we devised a comprehensive test suite for JED to test all portions of the info.jedi.net and info.jedi packages. The complete source code for the test suite is available with the JEDI distribution at our web site in the info.jedi.testsuite package. In addition to testing JEDI, this code provides many examples of the different ways to use JEDI.
The simplest part of the test suite calls remote methods with different signatures and return types. We call remote methods with no parameters, and with String, int, and boolean argument types. A call to a method with a serializable user-defined tree-like structured data type parameter checked for the proper use of Java serialization. Finally, we call a static method and a method with one argument of each of the above types.
To ensure that performance scales up with the number of method calls, our test suite sends a user-defined number of method calls, reporting the time per message after each 10% of the messages have been sent. This part of the test suite has been tested with 1,000,000 method calls with no observed performance degradation.
To test multiple concurrent JEDI calls, we create 100 threads, each of which called a remote semaphore method. This synchronized method implements a 100-thread barrier. None of the method calls may return until all of them have entered the barrier method.
Next, to make sure that proxies can be sent to remote methods as parameters and used successfully, our test client calls a method on the server, receiving a proxy from the return value of the method. This proxy is then used to call a server method that is passed a count of 100 and another proxy to the client. The server method recursively calls the same method on the client with a count of 99 and proxy to the server. This process continues until recursive method calls had been made 100 levels deep between client and server, after which they all return.
Finally, we test the error-handling capabilities of JEDI. This includes catching exceptions thrown by remote methods and ensuring that they print out stack traces with methods from both the local and remote machines, catching "not serializable" exceptions for parameters and return values that do not implement Serializable, catching time-out exceptions when a remote host does not respond within a specified time period, and catching exceptions where the specified remote object or method does not exist.
The successful completion of our test suite gives us confidence that our JEDI infrastructure is reliable. Its reliability has been further demonstrated as we have begun to build the next generation of Caltech's Infospheres Infrastructure on this solid JEDI foundation.
We tested the JEDI package on Sun's JDK 1.1 virtual machine, running on Solaris 2.5 on two SparcSation LXs with 16M of RAM and 10 Mbit Ethernet connections. Although performance numbers can vary from one machine to another, we expect that the relative performance of JEDI to the other Java-based systems will remain approximately the same.
We repeated tests several times and reported the best times achieved for each technology, to filter out the random effects of other users and programs on the two testing machines. As shown in figure 9, each test sent data back and forth between the machines 100 times, and smaller numbers are better. Notice that both ping and call times are reported for JEDI; the call time includes all of the overhead incurred by passing method names and parameters, looking up the right method, and maintaining call reliability. The ping times given measure the time required to send an unreliable JEDI packet to a server and back.
Figure 9: The performance of JEDI pings and actual JEDI remote method calls, in
milliseconds, compares well with the performances of pings using UDP, RMI, TCP/IP, and
the Infospheres info.net. Jonathan: Make this table more up-to-date. Fix the
accompanying text below as well. Compare iBus too? Put in two charts, one of which
contains setups and the other of which does not? Tests on a 100Mb system?
Although we considered using TCP/IP to send method call data, we were concerned that its scalability would be restricted by the limited number of sockets the operating system is able to create. Our early (incorrect) tests also implied that the Java VM implementation of TCP/IP sockets was very slow, since the default Java TCP/IP stream does not do any data buffering. While it is still relatively expensive to create a TCP/IP stream for each method call, acceptable performance may be attainable by reusing a single stream for multiple method calls. We hope to provide the option of using this transport in future versions of JEDI.
Because of the Infospheres work being done here at Caltech, we initially used the info.net library of the Infospheres Infrastructure [Chandy et al., 1998] as our communications protocol. However, this system provides rich features that slow its performance, such as ensured ordered messages. Performance tuning has not yet begun in earnest for the info.net library, so using it incurs a considerable performance penalty (100 back-and-forth messages took 14.9 seconds). As a result, we elected to build a more simple subsystem for JEDI communication.
This new JEDI messaging system is quite efficient. Sending 100 short UDP ping messages took 0.467 seconds on the machines we tested. Using the underlying JEDI infrastructure directly, a ping took 0.692 seconds, mainly because several convenience objects are created for each ping. On a slow, interpreted system, 2.25 ms per ping (225 ms over 100 pings) represents fairly low overhead.
JEDI's remote method call facility also compares well with RMI in the performance domain. Testing simple functions that return a string, we found that RMI made 100 remote method calls in 2.714 seconds, while JEDI was able to accomplish the same task in only 2.272 seconds. This is remarkable, considering that RMI uses hard-coded method names and signatures that are fixed at compile time, whereas JEDI can call any method on any object at run-time using reflection. However, JEDI still lacks some functionality provided by RMI, including support for method calls involving more than 64K of data passed as parameters.
Research has shown the importance of measuring not only two-way ping latency in distributed object oriented systems, but also throughput and latency for sending different kinds of data structures [e.g., Schmidt and Gokhale, 1996]. We tested JEDI and RMI by passing a 35 Kb, richly typed, tree-like data structure in a remote method call. In this case RMI sent the structure in 7.3 seconds, while JEDI took 11.4 seconds. Since this is significantly longer than the time required for communication, we theorize that the delay is mostly due to Java's serialization. Tests of serialization confirm that just serializing the data structure into a byte array takes over 5 seconds. Because the communication layer of JEDI sends all of its data in one packet, while the TCP/IP implementation in RMI sends data as it is produced, we believe that RMI is able to begin decoding the serialized data on the remote machine while it is still being encoded on the originating machine. This accounts for RMI's performance advantage in sending large, structured data. We hope to add the capability to send structured data more efficiently in a future release of JEDI.
During our performance experiments, we did not compare our JEDI mechanisms with CORBA invocations, because significant research is being conducted to make CORBA more performant and scaleable over high-speed networks [e.g., Schmidt and Gokhale, 1997], resulting in several techniques for optimizing IIOP performance [Schmidt and Gokhale, 1998]. We note that using the dynamic capabilities of CORBA comparable to those provided by JEDI can result in a performance degradation of more than an order of magnitude in some ORB implementations [Orfali et al., 1996]. JEDI's niche is in low-end distributed system development as an efficient, easier-to-use alternative to RMI in Java programs. An example of this use is the calendar application we describe next.
Figure 10: Jonathan, Scott, Mani, and Adam are in group 1; Scott, James, and Mani are
in group 2; and Mani, Jonathan, Adam, and James are in group 3. Each person has a single
calendar application that handles the scheduling of the social calendar for that person; for
example, Mani's calendar application handles the scheduling of his meetings with groups 1,
2, and 3. When a group leader decides to hold a meeting, he queries the other group members
in a peer-to-peer session [Chandy and Rifkin, 1997] to determine an appropriate meeting
time. The calendar application then locks in the appropriate slots for the group meeting in
each of the respective group members' schedules.
As illustrated in figure 10, a calendar scheduling application [Chandy et al., 1996] is an example of distributed resource management [Ramamoorthi et al., 1997]. For comparison of the application of different techniques for distributed program development, we implemented this calendar application using Java with simple local method calls, after which we distributed the program using RMI, Infospheres, and JEDI. We note that coordinating the scheduling activities of multiple distributed calendar programs for each port of the system was aided by a responsibility-driven design [Lea, 1997].
Because of the request-response nature of the scheduling algorithm we used, this application maps naturally to remote method call semantics. As a result, using the Infospheres Infrastructure's message-passing system required more work than the remote method call systems. For example, locking the calendar objects for each member of a group requires the following code:
for (i=0; i < numMembers; i++) {
sendBox.bind (new Place(memberAddresses[i]));
sendBox.send (new CalendarMessage(REQUEST_LOCK));
CalendarMessage response = (CalendarMessage) receiveBox.receive();
}
On the receiving end, a thread must be specifically set up to wait for incoming messages at a mailbox, attempt to lock the calendar, and send a response back to the process that requested the lock. If the locking operation could block, the user must create another thread to handle other incoming messages while the thread is blocking.
The advantage of a RPC-based system like JEDI or RMI for this application is that many aspects are handled manually. For example, the user does not need to write the CalendarMessage class (although in the case of RMI, the user will have to write a new remote interface definition file instead!). Also, the run-time library handles creating enough threads to service incoming requests (in case any method calls block). In addition, the RPC-like syntax is more convenient for the programmer because it requires fewer lines of code and matches the familiar method call paradigm. Other systems, particularly ones that do not require a synchronous response to every network message, are better suited to the message-passing scheme such as the Infospheres info.net package.
Using RMI made the code highly readable (since remote calls look just like local ones), but RMI's multiple implementation steps were time-consuming, because an extra interface had to be developed for every object that was accessed remotely. For example, we developed a Semaphore object to guard access to each user's calendar; to allow remote users to lock and unlock the semaphore, we needed to create the following interface:
package jedi.calendar.remote;
import java.rmi.*;
public interface SemaphoreInterface extends Remote
{
public void lock() throws InterruptedException, RemoteException;
public void unlock() throws RemoteException;
}
This additional interface was not a useful part of our overall design, as the Semaphore class definition is a complete specification of the interface to our Semaphore object. Although coding such a simple interface is not difficult, it would be time consuming and error prone to develop an interface for every class in a large collection of distributed objects. In a world with billions of different interacting objects, maintaining a separate remote interface for each one is not a scaleable solution.
Using JEDI did not result in code as pretty as RMI, because remote method calls are made through the generic library facility, rather than to a remote interface with a stub hidden behind it. However, developing the JEDI version was much quicker, simpler, and easier than the RMI version, because no separate interface files had to be layered on top of the existing objects, no preprocessor had to be run, and we did not need to keep track of stub and skeleton class files.
Locking the semaphore for each group member's calendar provides an example illustrating how RMI calls and JEDI calls are made. In RMI, this process looks like a simple procedure call due to the stub/skeleton system and the remote interface:
for (i = 0; i < numMembers; i++) {
MemberInterface member = (MemberInterface)members.elementAt(i);
SemaphoreInterface semaphore = member.semaphore()
semaphore.lock();
}
With JEDI, the call is conceptually similar, but syntactically more complex because there is no magic preprocessor to create a Java object with the correct interface:
for(i = 0; i < numMembers; i++) {
Proxy member = (Proxy) members.elementAt(i);
Proxy semaphore = (Proxy) member.function("semaphore");
semaphore.function("lock");
}
At the same time, the Semaphore class being accessed through JEDI was not modified in any way from a local Semaphore class. This demonstrates that JEDI can call methods on objects even when the source code cannot be changed. A protocol such as RMI that depends on changing the source code to implement a remote interface can never be used with libraries that are not designed with distributed computing in mind. In contrast, JEDI allows objects of any class to be fully network-capable.
One weakness in our design became apparent during this implementation: two Java Virtual Machines (VMs) cannot share a JEDI port. In our testing if we wanted two calendars to reside on one machine, we had to set up the calendar application using a different port for each member, rather than looking up the member's name in some sort of machine-global index. This demonstrates that if two JEDI objects are on different Java VMs in one machine, any process that needs to connect to them must keep track of their respective ports. Since RMI depends on a separate registry process running on each machine, we were able to look up calendars in the RMI system by name, rather than by port. In the future, we may add a machine-global directory service so that more than one JEDI VM can be run on one machine without forcing developers to deal with port numbers.
We compare the advantages and disadvantages of using JEDI with those of using RMI or CORBA.
| Feature | JEDI | RMI | CORBA |
|---|---|---|---|
| Ease of use | relatively easy | more difficult | more difficult |
| Dynamic invocation | yes | no | yes |
| Pass object by value | yes | yes | no (an RFP exists but is not implemented) |
| Pass object by reference | yes | yes | yes |
| Steps involved | few | many | many |
| Inter-language | possible through a CORBA interface or bridge | no | yes |
| Dynamic discovery | yes | limited to Remote interfaces | yes |
| Forces interface creation | no | yes | yes |
| Java integration | yes | requires rmic preprocessor | requires Java to IDL preprocessor |
| Security | only provided through native Java security mechanisms | provided through a special RMI security manager | provided through CORBA services |
| Transaction capabilities | no | no | through CORBA services |
| Round-trip ping performance | good | good | best* |
| Allows remote calls to Java libraries not designed for remote access? | yes | no | no |
* The comparison of CORBA to RMI is from [Orfali et al., 1997]. On a particular platform and implementation a round trip ping requires 3.3 ms under CORBA's IIOP and 5.5 ms under RMI.
From this table, we observe that JEDI is most suitable for projects requiring a Java-based RPC system with conceptual simplicity, ease of use, acceptable performance, and dynamic invocation capabilities. Although JEDI is presently less suitable for applications that require inter-language communication or advanced security and transaction capabilities, these features may be added to the JEDI system in the future.
The JEDI package can be extended in many compelling ways; some of the planned future extensions include:
The JEDI system gives a developer flexibility with its dynamic dispatch of remote method calls and the potential for dynamic discovery of remote object methods through reflection. The JEDI approach is scaleable, in that its communication layer provides efficient communication among many Java objects over the Internet. Ease-of-use of the JEDI package was demonstrated with the rapid conversion of a calendar scheduling application from a single machine application to a robust client-server system. Many possibilities exist for extending the JEDI package to provide a rich but simple and dynamic RPC-like mechanism for Java programmers.
This work was supported under the Caltech Infospheres Project. The Infospheres Project is sponsored by the Air Force Office of Scientific Research under grant AFOSR F49620-94-1-0244, by the CISE directorate of the National Science Foundation under Problem Solving Environments grant CCR-9527130, by the NSF Center for Research on Parallel Computation under Cooperative Agreement Number CCR-9120008, and by Parasoft and Novell. The BSD Daemon is copyright 1988 by Marshall Kirk McKusick. He has given the daemon a temporary leave from his BSD duties to study this paper on networking in Java. We would also like to thank K. Mani Chandy, Mark Baker, Ron Resnick, Joseph Kiniry, and the anonymous referees for their helpful suggestions to improve this paper. The JEDI packages are available for download from the JEDI homepage, at http://www.ugcs.caltech.edu/~jedi/.