IBM promotes Spark, Docker, Node, and Go goodies

By itself, IBM’s recently unveiled DeveloperWorks Open portal is mostly an aggregator for projects IBM has already open-sourced and is trying to highlight. But that doesn’t mean some of the projects aren’t worth extra attention. Here are five of the most useful in categories as diverse as Node.js development, the Spark data-processing framework, the Go language, and Docker containers.

Spark Kernel (Spark/Scala/IPython)

IBM’s full-tilt interest in the Spark data processing framework for Hadoop (and other technologies) has been brewing for some time. This includes more than contributions to Spark, but also new tooling around Spark to make it easier to learn and adopt. Case in point: Spark Kernel, which lets remote applications operate with Spark in real time and can replace the existing Spark Shell.

The Shell is commonly employed for interactive work with Spark, but it can’t easily be made into a service that other applications can access. Also, IBM claims many of the other solutions are too slow, cumbersome, or limited. Spark Kernel takes on the IPython message protocol, originally used to create interactive data-visualization “notebooks” for scientific computing applications, but here it’s a way to send code back to a Spark cluster and retrieve the results. Plus, IPython notebooks can connect with Spark Kernel, a boon to existing users interested in plugging into what Spark and its back ends have to offer.

Node Application Metrics (C++/Node.js)

One long-standing complaint about Node.js applications is that they’ve been opaque — that debugging Node apps can be a painful experience unless you’re using high-end tools or a specific platform. Node Application Metrics is designed to provide “a foundational infrastructure for collecting resource and performance monitoring data for Node.js-based applications,” for insight into most any package or module running on Node.

Data can be harvested from within Node apps by way of C or JavaScript APIs, then piped into analytics or performance monitoring tools. IBM mentioned StatsD and Graphite, as well as Elasticsearch and Kibana, but the possibilities don’t stop there.

Clampify (Go/Docker)

“Envy is as good a reason as any to invent something,” says the page for Clampify. In this case, it was envy over OpenStack’s Neutron networking system. Was it possible to take that component, with all its network-isolation and network-routing functionality, and make it work with Docker containers in a Swarm cluster instead of OpenStack? Short answer: Yes.

Among the possibilities proposed: Containers running different workloads for the same user can be isolated in their own networks by workload. The project is in its early stages, but intriguing ideas are developing around it. For example, using Docker’s “libnetwork” interface for “finer-grained control over how virtual interfaces are wired into the Docker container,” so that Clampify works with “other clustering systems such as Apache Mesos,” or setting up multitenant clustering with Docker Swarm using Clampify.

Node-red (Node.js/JavaScript)

Visual programming tools are typically used for designing workflows or information flows, such as database diagrams. Node-red was devised to quickly snap together code for programming Internet of things devices, à la Lego Mindstorms. In its current state, it’s less useful for creating production code and more suited to quick prototyping that can be refined later on.

i18n4go (Go)

Here’s a minor but useful Go project for an increasingly common software headache. Go has support for i18n (internationalization) tools, but existing Go programs can be tedious to internationalize. i18n4go provides a workflow for the process in one application — extracting strings from Go applications, merging them into a single package (to eliminate redundancies), preparing translations, and making the underlying code ready to accept them.