REST for Java developers, Part 3: NetKernel

NetKernel is a URI-based microkernel environment that combines the best concepts from REST, Unix, and service-oriented architecture to build extremely productive, scalable, and maintainable systems. NetKernel not only makes it easy to build RESTful systems in Java and other languages that run on the JVM; it also changes the way you think about tying components together.

So far this series has given you an introduction to REST and a quick walk-through of Restlet, an API that makes it easier to build and consume RESTful interfaces in Java. You have seen that REST is only partially about URLs. At a deeper level, it is about logically connected components, late-binding resolution of representation, and linkage between related information resources. With all of this, you ought to feel that you know enough to successfully combine REST and Java; in fact, you do. But the story doesn’t end there.

In this article, you will learn about NetKernel, a next-generation software environment that mixes what people like about REST, Unix pipes and filters, and service-oriented architecture (SOA). Not only can NetKernel make you more productive, but the abstractions also allow you to fully use modern, multicore, multi-CPU systems with little to no effort. At a minimum, if you understand NetKernel, you will understand REST at a deeper level.

Clashing abstractions and excessive coupling

The software industry has made great progress over the years, but there’s still work to do. The clash of abstractions and excessive coupling between components remain some of the biggest problems needing to be solved. Countless efforts have been made in the object-oriented world to address the issue with interfaces, the Law of Demeter, separation of concerns, dependency injection, design patterns, remoting abstractions, interface definition languages, SOAs, and so on. At the end of the day, however, objects end up feeling like the wrong level of abstraction to bridge the worlds of software components, data, services, and concepts. They are fragile and break easily in the face of inevitable change. Modern languages like Ruby, Groovy, Scala, and Clojure can help, but at some point it feels like any language or object binding is too specific for all needs.

It is certainly possible to build good, running systems with the object-oriented mindset, but as requirements and technologies change, these systems become crufty and hard to maintain. Organizations end up committing to a combination of technologies and sticking with them. They settle on a particular language, a particular data model, a particular database schema, and a particular platform. Sure, you can change any one or more of these commitments over time, but it’s still painful to do. This strategy of picking a small set of technologies can help minimize variance, and maintenance and training costs, but it also often prevents you from using the right tool for the job.

Many people are starting to realize that REST can fit into this space to provide an information-based solution to a wide variety of integration needs. Information references can be converted into specific forms on demand. With tools like the Restlet API, you have the flexibility of logical bindings but the convenience of local object references. The next article in this series will tie together many of these ideas into a larger vision; in this article I’ll focus on the benefits that you might accrue from pushing the ideas of REST into the inner environment, not just the boundaries between systems.

Object-oriented abstractions

Today, we are used to dealing with URLs for documents. http://javaworld.com, for example, is a logical name that resolves to the JavaWorld homepage. You do not (necessarily) know anything about what’s involved with producing the page, nor do you care what technologies are used; you simply want to consume the content on the page. By default, when you input the URL to your browser, you get a nicely formatted HTML page back. As long as that contract is maintained, the folks behind the scenes at JavaWorld are free to change their underlying technology without breaking the site for you.

The industry is becoming comfortable with the same idea for information in general. The URL http://someserver.com/customer/012345 might produce an XML document such as:

<customer id="012345">
      <name>Sue Jones</name>
      <address>
        <street>12345 Main St.</street>
        <city>Cleveland</city>
        <state>OH</state>
        <zipcode>44114</zipcode>
      </address>
    </customer>

As long as you get back what you expect from such a server, the information producers are free to support alternate representations (JSON, for example) for other clients, change their underlying technology, and so on.

In the object-oriented world, we approximate this behavior by using interfaces. But without some extraordinary gymnastics, we lack the freedom to return completely arbitrary object representations. Anything that satisfies the interface language binding can stand in, but it is difficult to substitute arbitrary resources (scripts, an alternate language implementation, raw data, and so on). We also can’t easily identify the results of issuing arbitrary requests of our interface implementations. Because we can’t identify a particular invocation explicitly, we can’t determine whether a request has been processed already and whether it needs to be done again. Imagine a complicated XSL Transformation (XSLT) being applied to a document, or issuing a database query for a particular customer, or invoking a SOAP Web service. Sure, our implementations can cache at each level, but that becomes a nontrivial amount of work, and we lack the visibility behind the scenes to cache what is most useful to cache. There is no opportunity to optimize across these caches based on how the system is being used.

Objects do have their place in our software strategies, but there is room to consider other ways of building modern systems.

Other approaches to scaling software

NetKernel solves several of these problems. In many ways, it came to the solutions indirectly. The project, which began at HP Labs in the UK, originally aimed to help solve the impending mismatch between software and hardware. Shared-state object-based systems and other conventional techniques do not scale to take advantage of extra CPUs and cores. The NetKernel work was eventually spun off into a company called 1060 Research, Ltd. NetKernel is available through a dual (open source and commercial) license.

Others have tried to solve the hardware/software mismatch in various ways. Ericsson’s Erlang is a language and runtime environment that forces developers to write functional software that scales. The ejabberd XMPP server from ProcessOne is an example of a modern, scalable system built in this manner. Clojure, Haskell, Scala, and F# are other increasingly popular languages that try to solve this (and other) problems with cool, modern features.

Another approach to taking better advantage of spare computational horsepower is to create an infrastructure that virtualizes the request and schedules it elsewhere. This tactic is cutely known as cloud computing these days, but the ideas have been around for decades as old-school time-sharing systems, SETI@Home, the Great Internet Mersenne Prime Search (GIMPS), distributed.net, Parabon Computation, Popular Power, grid computing, and so on.

Although each of these approaches works, and can work well, you often have to commit to the vision prematurely. To embrace Erlang, you must embrace its syntax and idiosyncrasies. There are certainly ways to integrate, say, Erlang and Java, but the bindings are brittle and often point-to-point. The cloud approach is also a useful and usable way of building flexible, scalable systems, but the infrastructure can easily overshadow smaller uses. It is the kind of thing you want to use if it is a good fit, but that may not always be obvious right away. Deciding after the fact could be a costly and expensive change.

Neither the language approach nor the cloud approach solves the caching issues, because they don’t let you to identify uniquely both behavior and the result of calling that behavior. Some modern languages support a technique called memoization, but it is usually specific to a particular calculation, not a general strategy.

URLs for arbitrary behavior

NetKernel combines many of these ideas by supporting a URI-based, functional model that might resolve locally or might involve distributed requests across multiple servers completely transparently to the client. Internally it uses a microkernel and runs on the JVM so it is easy to build solutions using languages that are already comfortable to you. You can adopt new languages as needed (as long as they run on the JVM) and take full advantage of any extra cores or CPUs your hardware has available. This all sounds too good to be true, but it really isn’t.

Modern object-oriented-based systems might abstract over a persistence layer by using an interface like this one:

public interface PersistenceLayer {
    Customer getCustomer(String id);
    List<Customer> getCustomers(String status);
    boolean storeCustomer(Customer a);
}

You could then imagine specific implementations for storing to flat files, an RDBMS via JDBC, an RDBMS via Hibernate, a remote fetch via a SOAP service — anything really.

In NetKernel, you give this concept of a persistence layer a logical name and bind it to a specific instance. Internally, it uses an active URI model to address functionality, so you might call it active:fetch-customers, active:store-customer, and so on. This is how you’d refer to the functionality to fetch and update customer records. The active URI scheme takes key-value pairs as a series of +key@value arguments.

For reasons that are probably not yet clear, you want the referenced values to be URIs as well if possible. Retrieval of customer accounts might be active:fetch-customer+id@cust:012345 or something similar. NetKernel would interpret this URI based on a certain module configuration and map the request to the behavior to perform the request. Just as with a URL and a Web site, you have no idea what kind of technology is used to retrieve the information. As a client who needs the information, you probably do not care. Documentation will tell you what form it will come back in. As long as it stays in that form, the module definition can change and your code will not break. This is the benefit of late-binding, logically connected components to help improve maintainability. You are treating the result of fetching a customer record as an information resource. Weakly typed and dynamic languages help get you started in this direction; this approach is just the next step.

NetKernel allows many ways to invoke this kind of behavior. For this article, we’ll use BeanShell for basic scripting:

main() {
    req = context.createSubRequest("active:fetch-customer");
    req.addArgument("id", "cust:012345");
    resp = context.issueSubRequest(req);
    context.createResponseFrom(resp);
}

Behind the scenes, NetKernel flattens this request to the full active:fetch-customer+id@cust:012345 URI. This represents a stateless, RESTful request; all of the information necessary to satisfy the request is passed in as part of the request. It identifies both the handler (active:fetch-customer) and the full state (+id@cust:012345). It treats the invocation of this behavior with this parameter as an information resource. This is an example of bringing REST inside. The behavior is invoked through a logical name, just as you might invoke a REST service at http://someserver/customer/012345.

The request is scheduled asynchronously on one of the microkernel threads. If you are running on a multicore, multi-CPU box, this will automatically scale up to take advantage of those resources without you (or the module implementor) having to think much about it. This is possible because what comes back is an immutable view of the result set, not unlike an HTML page or some JSON that might be returned as the result of issuing an HTTP request.

With a loosely typed language like BeanShell (or Groovy, also supported by NetKernel), you often do not know or care in what form information is returned. If you know that it came back as some form of XML by default, you might add the line:

resp.setMimeType("text/xml");

Using NetKernel: A short demo

It’s time to give NetKernel a try, using a sample application. Download the application source and install it according to the directions in the included README.txt file. There’s an XML skew to the configuration files and these examples, but NetKernel has fundamentally nothing to do with XML per se. It would be just as easy to write custom modules to deal with your domain objects if you wanted to.

The sample module is configured via the module.xml file to accept requests that match the following patterns (as well as others):

<export>
    <uri>
      <match>active:fetch-customer.*</match>
      <match>ffcpl:/customer.*</match>
      ...
    </uri>
  </export>

These are exported by the module definition, so any module that imports this one can issue requests for the behavior I describe below. Other modules are configured to convert external requests such as http://someserver/customer into an internal request of: ffcpl:/customer. This is just an internal request that gets remapped to the active:fetch-customers scheme like this:

<rewrite>
    <rule>
      <match>ffcpl:/customer</match>
      <to>active:fetch-customers</to>
    </rule>
  </rewrite>

The question remains, What actually executes the request? What handler do you rewrite the active URI to? Eventually, you imagine you will use a database, Web service, or other RESTful system to pull the information in but, when you get started, you may not know this. For the time being, you will simply dump some XML representations of customers into a directory and serve them up by munging them together. The initial rewrite rule will point to a BeanShell script to take care of this:

<rewrite>
    <match>active:fetch-customers</match>
    <to>active:beanshell+operator@ffcpl:/scripts/defaultcustomers.bsh</to>
  </rewrite>

This tells NetKernel to convert any request for active:fetch-customers into a request to execute the specified BeanShell. Notice how the pattern continues. You are identifying the functionality (active:beanshell, the logical name for interpreting BeanShell scripts) and the state needed to satisfy the request (a BeanShell script found at ffcpl:/scripts/defaultcustomers.bsh). Another module is responsible for handling this request. You can use it from your customer module because the NetKernel script module is included in your module definition:

<import>
    <uri>urn:org:ten60:netkernel:ext:script</uri>
  </import>

You could go look at that module’s definition to find out how the script is actually executed, but for now, you do not care. You simply want it to happen. The default customer-handling BeanShell looks like Listing 1:

Listing 1. Default customer-handling script

import org.ten60.netkernel.layer1.nkf.*;
import org.ten60.netkernel.layer1.nkf.impl.*;
import org.ten60.netkernel.xml.representation.*;
import org.ten60.netkernel.xml.xda.*;

import com.ten60.netkernel.urii.aspect.*;

main()
{
  // Fetch the list of XML files in the data directory
  req = context.createSubRequest("active:mls");
  req.addArgument("operator", "ffcpl:/etc/MLS-Config.xml");
  req.setAspectClass(IAspectXDA.class);

  mls = context.issueSubRequestForAspect(req);

  // Iterate over the results
  xdaROItor = mls.getXDA().readOnlyIterator("/mls/dir/res");

  sb = new StringBuffer("<customers>");

  while(xdaROItor.hasNext()) {
    xdaROItor.next();
    current = xdaROItor.getText(".", true);

    // Fetch the content of the appropriate file
    cust = context.sourceAspect("ffcpl:/data/" + current, IAspectString.class);

    // Append to the results
    sb.append(cust.getString());
  }

  sb.append("</customers>");

  response = context.createResponseFrom(new StringAspect(sb.toString()));
  response.setExpired();
  response.setMimeType("text/xml");

  context.setResponse(response);
}

Listing 1 is busy, but reasonably straightforward to follow. First, a call is issued to the active:mls handler, which will fetch all of the files in a module that follow a certain pattern. The etc/MLS-Config.xml file from the module you installed shows that any XML files in the data directory should be returned:

<mls>
    <module-uri>urn:net:bosatsu:jw-rest</module-uri>
    <filter>ffcpl:/data/.*xml</filter>
  </mls>

Note: This capability of fetching module-specific files can be reused and cached, just like anything else in NetKernel!

A listing of the sample XML files are returned and iterated over. Each document is fetched as a String:

cust = context.sourceAspect("ffcpl:/data/" + current, IAspectString.class);
  sb.append(cust.getString());

Then each document is appended to a StringBuffer to return the results. This one call to sourceAspect underscores NetKernel ‘s RESTful nature quite nicely. You have a logical request (ffcpl:/data/somefile.xml), a verb (SOURCE, equivalent to REST’s GET) and content negotiation. Here, NetKernel is asked to convert the file on disk into a String representation known as a StringAspect. It would be trivial to convert the request to return a byte array, a JDOM Document, a DOM instance, or any other form you might want in a different context. An Aspect in NetKernel is an immutable view of a particular representation of a resource. Because it is immutable, the whole request can be cached until one of the input resources changes. If you do not care what form comes back, you can call the source method instead, which takes only the URI for the information resource you want.

Once the StringBuffer has all of the documents, the customer element is terminated, converted to a String and wrapped in a StringAspect. Based on existing mappings, if NetKernel is running and you installed the example module, you should get a list of customers back in plain XML when you hit http://localhost:8080/customer. Remember the HTTP request is rewritten to an active:fetch-customers request, which gets rewritten to an execution of ffcpl:/scripts/defaultcustomers.bsh.

Layering applications

You usually want a nicely formatted list of data, so now you will convert the XML that comes back from active:fetch-customers into some nice HTML. NetKernel has plenty of tools to help you do this. XRL is a recursive template mechanism not unlike Cocoon and similar tools. To keep you from getting confused, the module is configured to map external requests for http://localhost:8080/xrlcustomers to a BeanShell that will do the XSLT conversion of the data that is returned from the logically named customer persistence layer. Normally, you would use the same URL and content negotiation to decide whether to return XML or HTML. The module exports ffcpl:/xrlcustomers.* and then configures this behavior with a rewrite rule:

<rewrite>
    <rule>
      <match>(ffcpl:/xrlcustomers.*)</match>
      <to>active:mapper+operator@ffcpl:/links.xml+operand@$1</to>
    </rule>
  </rewrite>

This works because the sample module imports the XRL handling behavior:

<import>
    <uri>urn:org:ten60:netkernel:ext:xrl</uri>
  </import>

The XRL handler expects a file of links that customizes one or more rewrite rules. (It would get really old having to specify them all by hand!)

<links basepath="ffcpl:/xrlcustomers/">
    <link>
      <name>index</name>
      <ext>/index.html</ext>
      <int>active:xrl-html+template@ffcpl:/content/customers.html+content@xrl:customers</int>
      <args>links</args>
    </link>
    <link>
      <name>customers</name>
      <int>active:beanshell+operator@ffcpl:/scripts/customers.bsh</int>
    </link>
  </links>

A complete XRL tutorial is beyond this article’s scope, but what is happening is that a request for ffcpl:/xrlcustomers/index.html is going to be rewritten to invoke the templating of the ffcpl:/content/customers.html file with whatever comes back from issuing the xrl:customers link. This currently points to the execution of ffcpl:/scripts/customers.bsh. The HTML template looks like Listing 2:

Listing 2. HTML template

<html xmlns:xrl="http://1060.org/xrl">
  <!-- Style cribbed from http://www.somacon.com/p338.php -->
  <head>
    <style type="text/css">
    table.customer {
      border: 6px inset #8B8378;
      -moz-border-radius: 6px;
    }
    table.customer td {
    border: 1px solid black;
      padding: 0.2em 2ex 0.2em 2ex;
      color: black;
    }
    table.customer tr.d0 td {
      background-color: #FCF6CF;
    }
    table.customer tr.d1 td {
      background-color: #FEFEF2;
    }
    </style>
  </head>

  <body>
    <h1>Customers</h1>
    <xrl:include href="xrl:content"/>
  </body>
</html>

The <xrl:include> tag will be replaced by the body of whatever is passed in through the content parameter. In this way, the HTML does not have to change if the implementation to generate what goes in it does.

Java developers who look at all these rewrites are surely thinking this looks obnoxious, inefficient, or insane. The hardest part of transitioning from an object-oriented to a resource-oriented perspective is to let go of your preconceived notions about how things should work. The steps I’ve shown you so far link a few things together logically. It is easy to change where the content comes from for the templating by changing a rewrite rule. Very large refactorings can be integrated into existing systems by making a change this small. To understand the power and beauty of REST on the inside, you must try to go with the flow!

The ffcpl:/scripts/customers.bsh script is quite straightforward: Go get the customer information and then style it:

main()
{
  req = context.createSubRequest("active:fetch-customers");
  customers = context.issueSubRequest(req);

  req = context.createSubRequest("active:xslt");
  req.addArgument("operator", "ffcpl:/scripts/style.xsl");
  req.addArgument("operand", customers);
  output = context.issueSubRequest(req);

  response = context.createResponseFrom(output);
  context.setResponse(response);
}

By now, you will not be surprised to see that applying XSLT to XML content looks like anything else in NetKernel. Under the hood, the module that handles this stuff uses Saxon, but again you do not really care. In Listing 3, we convert any <customers> elements into a table and populate it with rows filled from every matching <customer> element.

Listing 3. Transforming XML with XSLT

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet xmlns:xsl="http://www.w3.org/1999/XSL/Transform" version="1.0">
  <xsl:template match="customers">
    <table class="customer legacyTable">
      <tr class="header">
        <th>ID</th>
        <th>Name</th>
        <th>Street</th>
        <th>City</th>
        <th>State</th>
        <th>Zip</th>
      </tr>
      <xsl:apply-templates/>
    </table>
  </xsl:template>

  <xsl:template match="customer">
    <tr>
      <xsl:if test="position() mod 2 != 0">
        <xsl:attribute  name="class">d0</xsl:attribute>
      </xsl:if>
      <xsl:if test="position() mod 2 != 1">
        <xsl:attribute  name="class">d1</xsl:attribute>
      </xsl:if>
      <td><xsl:value-of select="@id"/></td>
      <td><xsl:value-of select="name"/></td>
      <td><xsl:value-of select="address/street"/></td>
      <td><xsl:value-of select="address/city"/></td>
      <td><xsl:value-of select="address/state"/></td>
      <td><xsl:value-of select="address/zipcode"/></td>
    </tr>
  </xsl:template>
</xsl:stylesheet>

Pointing your browser at http://localhost:8080/xrlcustomer/index.html should yield the content shown in Figure 1.

Ruby on Rails and Grails scaffolding can produce this much content by running a few scripts. Once you get some more NetKernel under your belt, however, you will be able write efficient, flexible, cached scaffolding too that should perform extremely well. NetKernel is not necessarily a replacement for these MVC frameworks (although it could be). Instead, you can easily imagine using a NetKernel backend with Rails via Active Resource. If you like Ruby, NetKernel supports it as one of its implementation languages. Groovy is supported nicely too.

Migration

The power of what you have seen so far is efficiency, productivity, maintainability, and the ability to use the right tool for the right job. The XSLT style sheet concisely expresses an efficient conversion of XML to XSLT in a language designed for that purpose. Trying to handle the conversion in Java would be ugly, error-prone, inefficient, and hard to maintain. NetKernel makes it extremely easy to move between the right tools. It also makes it possible to change technology choices without having a huge impact on dependent code. Again, interfaces kind of allow you to do this in languages like Java, but logical connections are much more flexible.

With working code to display customer information in place, it is now trivial to migrate the active:fetch-customers handler from the file-based prototype to an actual database-backed engine. This example uses HSQLDB, but it is easy to support other RDBMSs by modifying the driver configuration in etc/ConfigRDBMS.xml. First, seed the database by pointing your browser to: http://localhost:8080/jw-rest/db-init. This gets rewritten to a DPML script (another NetKernel language) that will populate the relational database. If you modify the example to use something other than HSQLDB, you will likely have to modify the SQL syntax as well. With the database initialized, it is now easy to come up with a new BeanShell that issues a SQL database query and then styles the results as XML:

main()
{
  req = context.createSubRequest("active:sqlQuery");
  req.addArgument("operand", "ffcpl:/scripts/customers.sql");
  output = context.issueSubRequest(req);

  req = context.createSubRequest("active:xslt");
  req.addArgument("operator", "ffcpl:/scripts/results.xsl");
  req.addArgument("operand", output);
  output = context.issueSubRequest(req);

  response = context.createResponseFrom(output);
  response.setMimeType("text/xml");
  context.setResponse(response);
}

You will continue to notice the same patterns, the same flexibility, and the freedom to choose the right language for the job. Investigating the implementation details is left as an exercise for the reader. Consult the NetKernel documentation (http://localhost:1060) for details. In order to get this code to run instead of the file-based version above, you simply change the rewrite rule in module.xml for active:fetch-customers to:

<rewrite>
  <match>active:fetch-customers</match>
  <to>active:beanshell+operator@ffcpl:/scripts/dbcustomers.bsh</to>
</rewrite>

This kind of change requires a NetKernel bounce (there are other ways to handle it, but this is the easiest way for now). CTRL-C in the terminal window where you launched NetKernel and restart it. Now, if you hit http://localhost:8080/customer, you should see some different data styled like the files were above.

One of the main benefits of logically connecting your components is that clients do not break when implementations change. If you point your browser back to: http://localhost:8080/xrlcustomers/index.html, you should see the new data styled via the old process. This is a powerful strategy for migration.

In conclusion

The benefits of REST for integrating systems are becoming quite well understood. The idea of taking the ideas inside your software architectures is quite revolutionary. NetKernel isn’t necessary for building RESTful systems; it just makes it very easy to do. The beauty of an environment like this is that the same benefits apply inside as outside. The potential to cache, pick implementation languages, change implementation, and so on are all powerful and they’re generally unavailable as easily in any other system. Orchestration across heterogeneous back-end systems is trivial. Changing a locally resolved call to a remote, distributed call can be done on the fly. You can leverage cloud assets if needed without having to design your system around the concept. You are also free to take advantage of modern languages such as Clojure, Groovy, and Scala; domain-specific languages such as XSLT; and legacy code that has been written in Java for years.

Dealing with your object representations is also a perfectly reasonable thing to do within NetKernel. It is not that objects are bad or useless; they remain good implementation tools. They simply do not represent a level of abstraction that works well for unifying everything that goes into modern systems. There have been valiant attempts to do so, but so far none have really taken off. The resource-oriented style embodied by NetKernel could be just such a successful attempt. Not only can you easily end up reusing most of what you have committed to in the past (Java, Hibernate, SOAP, JDBC, and so on), but you can start to build new types of systems as you go; it is not an all-or-nothing proposition.

Even if you cannot use NetKernel where you work, downloading it and playing with it will help you understand REST at a deeper level. You must be warned, however: once you start building systems the NetKernel way, your thinking about software is likely to change forever.

The next article in this series will tie up many of these ideas into a larger vision for building modern, information-driven architectures.

Brian Sletten is President of Bosatsu Consulting, Inc. a services company focused on using Web and semantic-oriented technologies to solve architectural and data-integration problems not handled by conventional tools and techniques. He has a background as a system architect, developer, mentor, and trainer, with experience in the online game, defense, finance, and commercial domains. Brian has a B.S. in computer science from the College of William and Mary and lives in Fairfax, VA.