The case for Java modularity

The lack of an effective modular programming model has frustrated Java developers since the Java platform’s infancy. Until recently no single solution had stuck. In this article, Jeff Hanson explains why modularity is a next evolutionary step for the Java platform. He then surveys the two JSRs that propose to bring modularity to Java 7 — JSR 291: Dynamic Component Support for Java SE and JSR 277: Java Module System. Both would bring the programming model popularized by OSGi to the Java platform, but they differ — sometimes greatly — in execution and philosophy.

The Java developer community has been seeking a modular component framework and model since Java first appeared in the mid 1990s. The JavaBeans framework was billed as a programming and deployment mechanism that would compete with ActiveX as a component technology and that could be consumed in component-based development environments, much like Visual Basic. JavaBeans, we were told, would be easily packaged using the JAR format and distributed and deployed wherever we wanted, fulfilling Sun’s write once, run anywhere (WORA) promise.

As most of us know, it didn’t work out that way. Many attempts have been made to solve the problem of modularity since that time (the list includes EJBs, EAR, WAR, Java applets, and the Java Network Launching Protocol) but each of them has soon encountered it own distribution and deployment woes.

This article discusses the latest attempts at making Java deployment artifacts and mechanisms modular and efficient, namely JSR 277: Java Module System and JSR 291: Dynamic Component Support for Java SE. I’ll discuss the background of modular programming on the Java platform and why it is so important to Java developers today. I’ll also provide an overview of both of the emerging approaches to modularity in Java 7, with some perspective on their inception and history to date in the Java Community Process. Finally, I’ll walk through the high-level concepts of a modular programming system — such as module definition, distribution format, and versioning — and explain how JSR 277 and JSR 291 would handle them differently.

The long and winding road

Shortly after the JavaBeans framework was introduced, it became apparent that tools vendors were not joining the modularity quest fast enough, and that the AWT framework was not robust enough for Java to flourish on the desktop. Even when a particular vendor exploited AWT, JavaBeans, and JAR files in a way that showed promise, JAR and JRE dependency hell soon reared their ugly heads, bringing many of the horrors associated with DLL hell to the world of Java.

Then the quest for modularity was diverted to distributed systems and Web application environments, and this is where things got really interesting. The idea was to provide a framework for building distributed systems (or the components of such a system) according to specifications, with standard interfaces. Such systems could then be deployed to any Java-based environment.

This Java-and-WORA-based utopia would consist of systems built using best-of-breed components wired together to form complex and robust distributed applications. The Java 2 Platform, Enterprise Edition (J2EE) was formulated to meet these goals, along with the EJB, EAR, and WAR specifications and deployment mechanisms. Now a set of resources, classes, configuration files and the like could be implemented, packaged, and deployed to any standard J2EE application server. An EJB, EAR, or WAR could use any component or library available on the standard classpath. The quest was over — or so it seemed.

Once components like EJBs, EARs, and WARs started making their way into real-world enterprise systems, a frustrating situation surfaced. Even though the ancillary frameworks and libraries needed by the components were in place at runtime, different versions of these frameworks and libraries (JAR files) were inevitably encountered. To deal with these versioning issues, components started to indicate the correct version of the ancillary JAR files in their deployment file. That worked fine until the application server or container included its own version of the files, because the server or container usually placed its JAR files closer to the start of the classpath search tree. All kinds of version-related problems began to arise as a result.

Compounding this confusion was possibility that a new version of a library or framework could be deployed to a system that was already in production. Most class loaders kept class definitions resident in memory as long as the JVM was running, and weren’t able to load new versions of the classes. Therefore, to deploy a new version of a framework or library, the server’s or container’s JVM needed to be stopped and restarted. This resulted in downtime-related errors, frustrations, and unhappy users.

To solve this problem, server and container providers came up with hot deployment: specialized classloading mechanisms and plug-in frameworks that would allow the deployment of updated frameworks and libraries at runtime. Hot deployment is no trivial task, and even the most sophisticated of solutions are often fragile and bug-prone. Other problems surface when the classloading mechanism is tweaked, such as multiple versions of the same class existing in the same JVM at the same time, increased perm space, and so on.

Obviously, a standard, complete solution is needed to address the long-standing problem space of modularity.

Introducing the Java modularity contenders

OSGi and the Eclipse plug-in frameworks are two of the more recent attempts to solve the problem of efficient module mechanisms in Java. Both of these technologies, however, have been created outside of the standard Java Specification Request (JSR) realm. JSR 277 (the Java Module System) and JSR 291 (Dynamic Component Support for Java SE) were introduced to bring Java modularity to the Java platform proper. Both are vying for inclusion in Java 7.

JSR 277 approaches the modularity problem from the perspective of the Java Runtime Environment (JRE). It defines a solution that provides Java API enhancements and specifies a distribution format, a versioning scheme, a module repository, and a set of support tools. JSR 277 is currently slated to be deployed with version 7 of the JDK, sometime in early 2009.

JSR 291 attempts to exploit the popularity of the OSGi system of modularity, which includes a service layer, a strict component lifecycle, a mechanism for exposing services from modules, and so on (see the Resources section to learn more about OSGi). JSR 291 aims to formally bring OSGi into the Java programming language. The dynamic component model underlying OSGi and JSR 291 supports the assembly of applications from components and implementation detail hiding between components, as well as the management of those components’ lifecycle.

Similar concepts underlie JSR 277 and JSR 291, but there are significant differences between the two initiatives. For the remainder of this article, I’ll focus on the high-level requirements essential to a modular programming system. You’ll have the opportunity to see how each JSR does or does not implement these requirements.

Module definition

Each specification for Java modularity has a slightly different view of what a module actually is and how it should be defined.

JSR 291

JSR 291 defines a unit of modularization and execution as a bundle. A bundle is a JAR file comprised of Java classes, resources, and manifest headers. Among other things, a bundle can specify its own version, the packages it imports and exports, the exports it requires from other bundles, its target execution environments, and more.

JSR 277

Straightforwardly enough, JSR 277 defines a unit of modularization as a module. A module is metadata that is compiled and packaged together with classes and other resources as a deployment artifact known as a Java Module (JAM). A JAM file is a JAR file that contains a file named METADATA.module, which contains information about the module, including its name, version, and dependency imports, along with the classes and resources that the module exports.

Module identification

Identifying a module is fundamental to gaining access to it and its ancillary packages or features. Mechanisms for the identification of components in the Java programming space have always attempted to address problems such as namespace collision, and Java modularity is no exception.

JSR 291

A JSR 291 bundle is identified by an identifier, a location, or a symbolic name. An identifier is a number assigned to the bundle by the container when a bundle is installed. A bundle location is a name assigned to the bundle when it is installed that defines the location of the bundle; this would typically be a URL for the JAR file. A symbolic name is a fully qualified name assigned to the bundle by the developer that adheres to the naming constraints defined in the Java Language Specification.

JSR 277

A JSR 277 module is identified by a fully qualified name assigned to the module by the developer. This name must adhere to the naming constraints defined in the Java Language Specification.

Runtime loading and reloading

The loading and reloading or deployment and redeployment of a software component is an important concern, especially in enterprise systems where the uptime of a host container can be extremely precious. The Java platform has attempted to address this issue since it entered the enterprise space.

The Java classloader framework has continually dealt with restrictive issues centered on the loading and reloading of classes. In particular, Java classloading is a fragile balancing act, with developers needing to consider flexibility (loading new versions of a given class dynamically), class-definition footprint (perm space), and couplings between a class and its consumers.

JSR 291

In JSR 291, when an updated bundle is redeployed, an explicit call must be made to the PackageAdmin.refreshPackages() method to force the container to reconnect existing bundle clients with the newly updated bundle.

JSR 277

In JSR 277, when a module definition is updated and redeployed, the module is reloaded when the Repository.reload() call is invoked. However, support for this method on the part of the repository vendor is optional.

Distribution format

Java applications and components to be distributed in a number of typical ways:

• In a JAR file, or in a file whose format derives from JAR (WAR, EAR, etc.)
• As an applet
• As a JNLP component

Each distribution format has its idiosyncrasies. In particular, building the runtime required to support each format has proved to be a tricky task in itself. The Java modularity specifications seek to address this problem from the outset.

JSR 291

In JSR 291, a bundle JAR file is the only artifact used for distribution and deployment. A bundle is a standard JAR file containing resources, classfiles, a manifest with bundle information, and other ancillary data, such as documentation, icons, and HTML files.

JSR 277

In JSR 277, a JAM file is the artifact used for distribution and deployment. The JAM file format is based on the JAR file format and contains classfiles, resources, and a METADATA.module file under the MODULE-INF path. The metadata includes information about the module, such as its dependencies or a resource export list. Listing 1 is an example of a METADATA.module file.

Listing 1. METADATA.module

METADATA.module:
(
   name=com.example.mymodule
   extensible-metadata=[@Version("2.0")]
   imports=
      [ImportModule(com.example.utils, @VersionConstraint("2.0+")),
       ImportModule(com.example.io, @VersionConstraint("3.0+"))]
   class-exports=[com.example.foo.Class1,
                  com.example.foo.Interface2]
   members=[com.example.foo.Class1,
            com.example.foo.Interface2,
            com.example.foo.Class3]
)

To provide a mechanism for encapsulating multiple packages in a single artifact, JSR 277 introduces the concept of a super package. A super package is a collection of packages declared along with the exported classes and interfaces for each package. Listing 2 demonstrates how to declare a super package in Java code.

Listing 2. Declaring a super package

Superpackage com.example.myservice
{
   // the packages that are part of this superpackage
   member package example.foo.myapp, example.foo.myapp.processing;

   // the list of exported classes and interfaces
   export example.foo.myapp.Main, example.foo.myapp.Helper;
}

A super package is compiled to bytecode. Access control is enforced by the compiler and the runtime JVM, just as it is for as classfiles. Details about a super package can be discovered using reflective API extensions that are slated to be added to the java.lang.Class class when JSR 277 is formally released.

Versioning

The Java classloading mechanism has been a thorn in the side of developers who need to deliver updated versions of their components, primarily because of the tight couplings between a class and its consumer. When a new class is loaded by the Java classloading framework, it is simply regarded by the host classloader as an entirely new class, and old versions of the class are retained due to associations with components holding existing references to the old versions. This creates a fragile environment for managing class versions. Java modularity seeks to solve this issue and make versioning a graceful process.

JSR 291

JSR 291 allows you to declare a version or a range of versions to be used as constraints for an import definition or an export definition. The format of a version in JSR 291 is defined as major.minor.micro-qualifier — 1.6.1-b32 would be an example.

JSR 277

JSR 277 establishes a versioning scheme that is defined in the metadata of the module definition. This version information is used to define the version of the module itself as well as any constraints on its module dependencies. Dependencies can be constrained by a version, a version range, or both. The format of a version in JSR 277 is defined as major.minor.micro.update-qualifier — 1.6.1.2-b32 would be an example. (This four-digit versioning scheme has been surprisingly controversial.)

Lifecycle

As modules are updated and reloaded by a host container, they must be gracefully stopped, terminated, and flushed from memory. Java modularity seeks to provide a lifecycle structure and provide information and functionality to keep a system robust and error free.

JSR 291

JSR 291 defines a strict lifecycle, with the following states:

• Installed: The bundle has been installed successfully.
• Resolved: All classes exposed by the bundle are available and the bundle is ready to be started.
• Starting: The bundle is being started.
• Active: The bundle has been started and is ready for use by a bundle client.
• Stopping: The bundle is being stopped, and all resources related to the bundle will be deactivated, unregistered, and so on.
• Stopped: The bundle has been stopped.
• Uninstalled: The bundle has been uninstalled. Resources related to the bundle are made unavailable by the container framework.
• Updated: The bundle has been modified.

A bundle starts its active life when its bundle activator is called. A bundle activator is an implementation of the BundleActivator interface, containing start and stop methods, and is identified by the Bundle-Activator manifest header.

JSR 277

The JSR 277 lifecycle is more loosely defined and is controlled by the containing repository. A module can be set to one of the following states by a repository:

• New: The module is constructed but not initialized.
• Preparing: The module is being prepared.
• Validating: The module has been prepared and is being validated.
• Resolved: The module has been prepared and validated, but not has not yet been initialized.
• Ready: The module has been initialized and is ready to be used.
• Error: The module is in an error state.

Execution context

A component or module typically executes in context to its host environment. For example, the resources you can access from a module depend on whether that module is executed as a standalone desktop application or as a component in a Web application, where filesystem access is more restricted.

JSR 291

In JSR 291, a bundle recognizes its execution context via a BundleContext object that is passed to the bundle by the containing framework. A JSR 291 framework passes a BundleContext object to a bundle’s BundleActivator implementation during activation. The BundleContext object contains information about the framework and the JSR 291 service registry.

JSR 277

A module in JSR 277 is executed in the context of the JVM, just as a regular Java class is. Executable modules (that is, modules with a main class) are instantiated by an application launcher using the module’s definition, which is retrieved from the module repository. Modules without a main class are loaded by a module classloader.

Inter-component relationships

The relationships between classes and modules in Java programming is a fundamental concern when considering things like method signatures, ancillary classes, package dependencies, and the like.

JSR 291

In JSR 291 the process of connecting (that is, wiring together) importer bundles and exporter bundles is referred to as resolving. During this wiring process, constraints are checked and enforced. This process is performed before the JVM loads or executes a bundle.

A wire is a link between an exporter bundle and an importer bundle. Bundles interconnected by wires form a graph. A bundle can only be resolved if all mandatory imports are wired and all required bundles are available and their exports wired.

JSR 291’s service layer allows bundles to register service objects with the service registry. The service layer provides a mechanism for bundles to publish, find, and bind to each other’s services. Bundles use services registered by other bundles by obtaining a service reference from the bundle and then passing this reference to the BundleContext object. The BundleContext then returns a service object to the calling bundle, which in turn becomes dependent upon the service object’s lifecycle until the service object is released.

JSR 277

In JSR 277, a module’s import declarations define that module’s dependencies upon other modules. Imports are specified metadata of the module definition.

Linking a module together with its imports is known as resolving. A resolved module is a module instance that is interconnected with its imported modules. A mechanism known as an import policy is provided to allow each module to interconnect its imported modules. This mechanism relies on the module providing an instance of the ImportPolicy class to the module system.

Registries and repositories

Registration and discovery of components is a fundamental requirement of any system offering dynamic component technologies. It is of vital importance that consumers be able to find the proper component or module — and the correct version of that component or module — for a given task. Registries and repositories are typical solutions for this issue.

JSR 291

JSR 291 provides a service layer that exposes a service registry. In the service layer, bundles register service objects with that service registry. Other bundles can then obtain references to the service objects. Bundles can use a service reference to get the properties and call methods on the service.

JSR 277

JSR 277 defines the concept of a repository for storing, discovering, and retrieving module definitions in the module system. Just like the classloading system, a module system has one or more repositories. A bootstrap repository exposes the core platform modules and is the only mandatory repository in the module system; other possible repositories include a global repository, an application repository, or a URL repository.

As with classloading, repositories use a delegation model to locate module definitions. Each repository has a parent repository. When a repository is asked to find a module definition, it will delegate the request to its parent repository first and then handle the request itself if the parent can not.

Security

Access privileges and code authenticity are issues that Java has diligently tried to address from its inception, and these problems only get trickier in modular systems.

JSR 291

The security layer of JSR 291 is optional and based on the Java 2 security architecture. Code and JAR files can be authenticated by location and by signer. Admin permissions can be delegated to the framework deployer in order to restrict permissions to only those necessary for an individual bundle. Bundles can specify permissions for actions that extension or fragment bundles can perform, such as the permission to expose packages to another bundle, the permission to require packages from another bundle, or the permission to attach to another bundle. Bundles can only export and import packages for which they have the necessary permissions.

A resource in a bundle can be accessed via admin methods on the bundle. The caller of these methods must have admin permission privileges. Other actions that require admin permissions include calling methods to start or stop the bundle, loading classes that use the bundle, and retrieving the bundle context.

JSR 277

JSR 277 modules operate within the standard Java security model. Signing and signature verification for JAM files work just as they do for the JAR file format.

A new class, ModuleSystemPermission, is provided to be used by security policy providers in order to check invocation on some module system APIs. Some of these checks include installing new module definitions, uninstalling module definitions, creating a new repository, and shutting down a repository.

Migration from JAR files

The JAR file format has been the packaging and deployment standard for Java from the early days of the platform. Therefore, it is important for new packaging and deployment technologies to address the issue of migrating from JAR formats to whatever new format is introduced. JSR 291 and JSR 277 recognize this constraint and address it in different ways.

JSR 291

A JSR 291 bundle is based on the JAR file format with additional information, such as an enhanced manifest file and optional documentation. In a JSR 291 framework, a bundle is deployed as a standard JAR file with specific additions made to the JAR file manifest, including the bundle name, the bundle version, exported packages, and so on. A bundle JAR file can contain other JAR files.

JSR 277

The JAM format of JSR 277 is based on the JAR file format, with the addition of a METADATA.module file placed under the MODULE-INF path in the JAR file.

JAM files can be compressed using a hyper-compression scheme known as Pack200. This compression technology was defined by JSR 200; it reduces the size of JAR files by optimizing storage of specific features of Java classfiles. For instance, it removes redundant class attributes and merges constant pool data.

Localization

Because Java aspires to a worldwide reach, it has always offered facilities for localization, offering an instance of a component or module for consumption with the necessary manipulation of text and other resources to fit a given locale and language.

JSR 291

In JSR 291, the Bundle-Localization manifest header contains information about the location in the bundle where localization files can be found. The default value is OSGI-INF/l10n/bundle. Localization files use a naming scheme somewhat similar to the standard resource bundle localization scheme used for Java SE. Specifically, localization files share a common base name followed by an underscore and one or more suffixes (denoting language, country, and variant) as defined in java.util.Locale. The localization file must end with the .properties suffix. For instance, an English localization file would be OSGI-INF/l10n/bundle_en.properties, while the Swedish version of this file would be named OSGI-INF/l10n/bundle_sv.properties.

JSR 277

No explicit mechanisms for localization are defined in JSR 277 outside of what is already established by the Java SE standard.

Java modularity: Where do we go from here?

The need for a Java modularity standard is apparent. However, the future is still very much in question. JSR 277 is still in draft review, and very little activity is emerging publicly from the expert group. Based on recent discussion, it seems the JSR 277 expert group is determined to make JSR 277 interoperable with JSR 291. In contrast, JSR 291 is in the final release stage, and has gone through several significant revisions; moreover, it is based on OSGi, a platform already in wide use.

In conclusion

The need for a modular component system for the Java platform has been apparent for almost as long as Java has been known to the mainstream software development community. JSR 277 and JSR 291 seek to fulfill many of the same requirements of a modular system of components.

JSR 277 approaches the problem from the perspective of the JRE by defining Java API enhancements, a distribution format, a versioning scheme, a module repository, and a set of support tools.

JSR 291 exploits the popularity of the OSGi system of modularity with a module system that includes a module assembly framework, a service layer, a strict component lifecycle, and a mechanism for exposing services from modules.

The different approaches taken by each JSR expose and address some of the deficiencies found in each, but there’s also a high degree of overlap that could prove confusing. Many have proposed that the ideal solution lies in the two working well with each other. Modules written to one specification ought to work in the other, and each project should support Java-language enhancements proposed by the other.

See the Resources section to learn more about the topics discussed in this article.