Java image scaling performance

I’ve been homebrewing a slideshow application – more on that later – and learning lots about Java’s AWT graphics libraries. This is a post on a rookie error I made with image scaling.

TL;DR – The real performance hit is in creating image buffers and moving data between them. The actual calculations are almost trivial in comparison. So, creating one image buffer and then just drawing everything into it will be fastest.

OK, some code. First some common setup code:

		BufferedImage img = ... // Get an image from somewhere
		BufferedImage after = new BufferedImage(w, h, BufferedImage.TYPE_INT_ARGB);
		Graphics g = after.getGraphics();
		
		// This just defines the transform, it doesn't apply it
                AffineTransform at = new AffineTransform();
		at.setToTranslation(translateX, translateY);
		at.scale(1/ratio, 1/ratio);
		AffineTransformOp scaleOp = 
		   new AffineTransformOp(at, AffineTransformOp.TYPE_NEAREST_NEIGHBOR);

This is slow:

		scaleOp.filter(img, after);

Why? Because it creates a whole new BufferedImage, then copies it into after.

This is about 100 times faster:

		g.drawImage(img, (int) translateX, (int) translateY, (int) scaledWidth, (int) scaledHeight, Color.black, this);

Because the scary affine transform is gone, right? Wrong. This is fast too, about 50 times faster than the first version:

		after = scaleOp.filter(img, null);

And this is just as fast as drawImage():

		((Graphics2D) g).drawRenderedImage(img,at);

How would you choose between them? Well, if you want rotations you’d need to use the one of affine transform versions. Snippet 3 may involve you in color model shenanigans you’d rather avoid (see http://stackoverflow.com/questions/1841629/affinetransformop-speed-memory-question), so my conclusion is: if you don’t need rotations, use drawImage(), otherwise use drawRenderedImage(). However, I’m sure there are a thousand other ways to do this, and I’m nowhere near having a handle on best practice.

The app-pocalypse ourobouros

The New York Times reports on Google et al.’s crusade to save us from app fragmentation.

Lol.

The writer of the NY Times piece (and/or his sub-editors) has achieved the superhuman feat of not drawing any conclusions from this evocative material, but the signposts are all there:

“It is not just a matter of consumer convenience. For Google and Facebook, and any company that has built its business on the web, it is a matter of controlling the next entryway to the Internet — the mobile device.”

“But as people spend more time on their mobile devices and in their apps, their Internet has taken a step backward, becoming more isolated, more disorganized and ultimately harder to use — more like the web before search engines.

“How remarkable it is that we are back in 1997!” said Roger McNamee, co-founder of Elevation Partners, an investment firm in Menlo Park, Calif.”

“Take Google, which makes money helping people search the web. When people search in apps, it is mostly left out. And while the company has a fast-growing business selling apps through devices that use its Android operating system, that pales in comparison to its business selling search advertising.”

““Once we’re all using the same plumbing, everyone can go and build businesses and interesting experiences on top of that,” said Eddie O’Neil, a Facebook product manager working on the company’s program, App Links.”

The ironies are many, exquisite, and multi-layered. The complete and transparent prioritization of commercial value over social value is, well, honest. Let’s make some noise, then sell earplugs. One company’s sub-standard product is another company’s value-add opportunity. Well, actually, it’s the same company’s value-add opportunity. That what makes it so fun, amirite?

One of the more remarkable ironies is this: Apple, having created the app-pocalypse and crippled web interoperability with the whole Flash fiasco, remains “cool”; Google, having enthusiastically out-app-ed Apple and upped the ante by turning their web browser, of all things, into an app platform, remains “good” (or at least “not evil”); and Microsoft, having produced the most web-capable platform of the three (as I have remarked before), and been vilified by the tech press for its efforts, remains “evil”.

mod_cfml wrangling

In a previous post I talked about how Railo’s default install with mod_cfml can cause problems when you have a lot of virtual hosts. This post will deal with some details of various configurations to deal with those problems. I’ll assume an Apache + Linux install for these config examples.

Config file locations

Tomcat config is in /opt/railo/tomcat/conf
Tomcat context files are in /opt/railo/tomcat/conf/Catalina/<hostname>/ROOT.xml, where “hostname” is the hostname from the Tomcat host configuration. Mod_cfml uses the hostname from the URL for this purpose, but you can define it to be anything.
Apache config is in /etc/apache2

1. Explicitly define Tomcat hosts and aliases

The first thing you can do is to explicitly handle most or all of your Tomcat context creation, rather than let mod_cfml do it for you. Because of the way mod_cfml works, it stays completely out of the loop and adds no overhead for explicitly defined contexts.

For each distinct Railo application, add a <Host> element to Tomcat’s server.xml, like so:

<Host name="lotsahosts.me" appBase="webapps">
   <Alias>myalias1.lotsahosts.me<Alias>
   <Alias>myalias2.lotsahosts.me<Alias>
   <Alias>myalias3.lotsahosts.me<Alias>
</Host>

Important: this configuration is in addition to your virtual host setup in your web server.

Then create the context file, which will be called /opt/railo/tomcat/conf/Catalina/lotsahosts.me/ROOT.xml, and should contain:

<?xml version='1.0' encoding='utf-8'?>
<Context docBase="/var/www/myapps/lotsahosts_webroot">
  <WatchedResource>WEB-INF/web.xml</WatchedResource>
</Context>

TBH I don’t think the WatchedResource element is needed. It’s just come along for the cut-and-paste ride since forever.

To reiterate, you’ll need Host element and a context file for every distinct Railo application. How do you know when you need a new Host element, as opposed to just adding an Alias? It comes down to what your ColdFusion code is expecting. If it’s OK with handling multiple hostnames, then go ahead and use an alias. Otherwise, add a Host element and context file.

2. Disable the mod_cfml Tomcat valve

After the above config change, mod_cfml will stay out of the picture until your web server sends through a host header that you haven’t explicitly handled. At that point, mod_cfml springs into action and creates the context for you.

This can be a reasonable way to operate if you frequently need to provision new virtual hosts that are all just aliases into the one web app. You can let mod_cfml dynamically create the contexts, but keep the total context count down by periodically sweeping them up into your static configuration (i.e. add an <Alias> element to server xml and then just delete the context folder from conf/Catalina). However, if your new virtual hosts are not just aliases, your context count will unavoidably increase, and you’ll run into mod_cfml’s startup overhead.

So, to disable the mod_cfml Tomcat valve, just comment out these lines in Tomcat’s server.xml:

	<Valve className="mod_cfml.core"
		loggingEnabled="true"
		waitForContext="60"
		maxContexts="200"
		timeBetweenContexts="1"
		/>	

Once you’ve done that (and restarted Railo), any host header that you haven’t explicitly handled will result in a 404 error.

3. Remove the web server’s mod_cfml component

If you’ve read the mod_cfml documentation (which I’d recommend), you’ll know that mod_cfml is actually a matched pair of components, one on the web server side and one on the Tomcat side. The web server component works differently depending on which web server you run. On Apache, it’s a very lightweight component that runs on top of the usual mod_proxy or mod_jk setup and adds some headers to help the Tomcat valve know how to set the docBase for new contexts.

I’m not sure why you’d need or want to remove the web server component, as the startup and memory overheads are all on the Tomcat side. But, for completeness, here’s how to do it for Apache 2.2: simply remove the PerlRequire, PerlHeaderParserHandler, and PerlSetVar directives from /etc/apache2/apache2.conf. Note that on older Apaches those directives might be in httpd.conf.

On IIS, mod_cfml uses the Boncode connector. If you want to remove that, you’ll have to replace it with another connector, but I’m no IIS guru, so I’ll leave it at that.

Railo and mod_cfml

The way Railo works with Tomcat is quite different to Adobe ColdFusion (ACF), and the differences can be pretty important if you have a lot of virtual hosts.

I don’t want to go into the gory details about how Tomcat works, but a few background points:

  • Tomcat has a virtual host concept analogous to web server virtual hosts, where there’s a default host and then specific host configurations tied to defined hostnames.
  • Tomcat’s virtual host configuration is completely independent of the front end web server (e.g. Apache or IIS)
  • Within each virtual host is at least one “context”, which essentially maps to a classloader and a set of resources (e.g. a directory on the filesystem).

All of this is quite different to the way webservers work, so Railo and ACF both try to hide it from you, in different ways.

ACF sets up Tomcat with just the default virtual host and a single default context. The default host will handle any request not bound to some other host, so with no further configuration every hostname ends up in the default Tomcat host and context. Simple and easy. In effect, your entire ColdFusion environment, multiple hostnames, multiple applications, the whole shebang, lives inside a single Tomcat context.

This is decidedly not idiomatic from a Java servlet container point of view, where each application gets at least its own context. Among other things, this allows the application to define its own classpath, thus isolating applications from issues like JAR conflicts with other applications. So Railo takes this path – each ColdFusion application maps to a Tomcat application, with its own context. That means Tomcat must be configured for each and every hostname and application folder.

But wait – ColdFusion is supposed to hide all unpleasant details, isn’t it? CF developers shouldn’t need to know or care about Java-specific stuff like servlet container configuration, right? So Railo introduces another piece, mod_cfml. The Railo installer configures this by default, and what it does is watch for unrecognised hostnames and create new Tomcat contexts for them on the fly. Pretty neat trick really, and it makes Railo just as seamless and noob-friendly as ACF. Until…

Until you migrate an environment with hundreds of virtual hosts from ACF to Railo. At which point, several things might start to cause problems:

  1. Context memory overhead: it may be only 1-2MB per context, but that’s enough that your site that hummed along nicely in a 512MB JVM is now completely unresponsive with memory stress.
  2. Context startup time: on my anemic dev box, it takes mod_cfml 30 seconds to create a new context, and over a minute to validate an existing context after a server restart. Even a fairly beefy staging box only brings that down to 10 seconds per context. Multiply that by 500, and you’ve got a problem.
  3. Context creation throttling: because of these overheads, mod_cfml throttles context creation to avoid becoming a DoS vector. By default, you get one context creation per 30 seconds up to a maximum of 200 contexts per 24 hour period. You’ll simply get a 503 error on every virtual host after the first 200.
  4. Context restart: this deserves a separate dot point – mod_cfml “creates” every context the first time it is hit after a restart, even if it already exists. That means that even if you have fewer than 200 virtual hosts, you can hit the limit simply by restarting the Railo service. Sites that worked before the restart suddenly become unresponsive after the restart.

What to do? There are, as always, options:

You can beef up your environment. Make sure you have enough memory, enough CPU, adjust the mod_cfml throttling settings, and preload all the contexts by hitting them once after each server restart (so your users don’t get the delay). Of course you’re now busy fiddling with servlet container configuration, so mod_cfml isn’t earning its keep from a simplification point of view.

You can just ditch mod_cfml and use mod_proxy or whatever-it-is-on-IIS, and then configure Tomcat manually. This lets you group virtual hosts into shared contexts (as per ACF), thus avoiding the issues that come with uncontrolled proliferation of contexts. To be honest, when I installed Railo I took one look at mod_cfml and said “No thanks” – and that was before I knew about the gotchas listed above. I reckon if you can configure a web server, you can configure a servlet container, so mod_cfml just isn’t solving any problem that I have.

Or, you can manually configure Tomcat to recognise all your existing virtual hosts, but keep mod_cfml around to pick up any new hostnames. This can be handy if you add hostnames (say, one hostname per client) on a daily basis. The new ones will all get a context each, but to control this you can periodically add the new hostnames to your shared context and delete the standalone contexts.

Leave a comment if you want me to post a detailed how-to for any of these options.

Edit: more detail on configuration tweaks

Woocommerce – testing if a product is in a descendent category

To test whether a WordPress post is within a particular category or any of its subcategories, there’s a handy snippet in the WordPress codex here: http://codex.wordpress.org/Function_Reference/in_category#Testing_if_a_post_is_in_a_descendant_category

For WordPress noobs like me (and what environment has more noobs than WordPress?), though, it’s not obvious how to make that work for Woocommerce product categories. There are two key things you need to know:

  1. Woocommerce product categories do not use the normal category system. They use a custom taxonomy called ‘product_cat’
  2. The in_category function used in that snippet only works with the normal categories. For custom taxonomies, you have to use has_term() instead.

The revised snippet looks like this:

if ( ! function_exists( 'post_is_in_descendant_product_cat' ) ) {
	function post_is_in_descendant_product_cat( $cats, $_post = null ) {
		foreach ( (array) $cats as $cat ) {
			// get_term_children() accepts integer ID only
			$descendants = get_term_children( (int) $cat, 'product_cat' );
			if ( $descendants && has_term($descendants, 'product_cat', $_post) )
				return true;
		}
		return false;
	}
}

jQuery .validate – what does optional really mean?

When you look customising validation rules and/or handlers for the jQuery validate plugin, you run across a lot of code that looks like this:

return this.optional( element ) || !/Invalid|NaN/.test( new Date( value ).toString() );

So obviously, that means “Don’t bother checking this field if it’s optional”, right? Wrong!

There are two important things you need to know about jQuery validate’s optional() function:

  1. It is NOT telling you whether the field is optional. It is just telling you whether the field is empty.
  2. It returns false if the field is NOT empty, and it returns the string “dependency-mismatch” if the field IS empty.

If you’re quietly thinking “WTF?” at this point, you’re not alone. The semantics do make a bent kind of sense once you know what’s going on, though. Obviously there’s no point checking any validation but “required” on an empty field. So the code above is saying “If this field is OK, it will be either because its empty but optional, or because this rule passes”. Note that “If” – it’s not saying the field IS optional, it’s saying it will have to be optional to be OK. The actual optional check is left up to the “required” validation handler.

You can stop now and just remember the two points above. But if you’re keen, let me bend your mind a little further. If you dive into the plugin source code, you see the implementation of optional looks like this:

optional: function( element ) {
			var val = this.elementValue( element );
			return !$.validator.methods.required.call( this, val, element ) && "dependency-mismatch";
		}

Surely the first part of line 3 is testing whether the field is required? Actually, no it’s not. It’s just calling the same function that the real “required” rule would call – in other words (leaving aside complications with dependencies), testing whether the field is empty.

Summary: in this code, “optional” means “empty”, “required” may mean “empty” or “required” depending on the context, and “dependency-mismatch” means “true”. In the immortal words of Phil Karlton

Structural linking with Jersey + Moxy – an AspectJ solution

The requirement

In a REST resource representation, structural links can be embedded in the resource to allow the client to navigate to subresources. This is in contrast to so-called transitional links, which represent state transitions.

    <parent id="100">
        <!-- I am a transitional link -->
        <link href="http://myserver/parents/100" rel="delete">  

        <!-- We are child resources with structural links to our full representation -->
        <child id="1" href="http://myserver/children/1"/>       
        <child id="2" href="http://myserver/children/2"/>
    </parent>

The JAX-RS project has a good discussion of the differences, as well as some explanation of why transitional links are so much easier to support than structural links.

I’ve been struggling to find a good way to implement structural links, especially as I want to add a couple more constraints:

  1. The domain model must not be touched. Even when you do have access to the source code, I simply don’t think it’s kosher to pollute a domain model with cross-cutting concerns like serialization to a particular format.
  2. I won’t put up with death-by-boilerplate, such as creating a DTO for each domain model class.

I posted this problem as a Stack Overflow question along with some candidate approaches. This blog post is about the first candidate, the AspectJ solution.

The solution sketch

This solution uses Jersey’s very handy declarative linking capability. You can add a field to a model class and Jersey will populate it for you. Even more impressively, you can simply point Jersey at a method in your REST API and Jersey will derive your link from the @Path annotations on that method. Too easy. Just too bad this violates my #1 constraint, that the model not be touched.

Enter AspectJ. We can use an intertype declaration to add the field, and everything just works.

The downside of this solution is that the structural link configuration is not truly externalized (with respect to the model). Yes, we’ve avoided altering the model source code – which, IMHO, is no small win, as preserving the expressive power of the model source code is a pretty high priority for me. However, at the byte code level, we have most definitely altered the model, at least within this compilation unit. The new fields will be visible to any reflection-driven framework and may pop up in other representations. If we can live with that, this is a pretty clean solution.

Example code

This code is also available on BitBucket.

Model classes

package testing;

import java.util.ArrayList;
import java.util.List;

public class Planet {

    private int id = 1;
    private String name = "test";
    private double radius = 3.0;

    private List<Moon> moons = new ArrayList<Moon>(0);
    
    public void addMoon(Moon moon) {
    	moons.add(moon);
    }
}
package testing;

public class Moon {
	
	private String name;
	
	// No-arg constructor is a requirement of JAXB
	public Moon() {
	}
	
	public Moon(String name) {
		this.name = name;
	}
	
	public String getName() {
		return name;
	}

}

Resource class

For the proof-of-concept, we’re just serving an freshly created instance of the model.

package testing;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.PathParam;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("/services")
@Produces({MediaType.APPLICATION_XML,MediaType.APPLICATION_JSON})
public class Services {
	
	private Planet initPlanet() {
		Planet p = new Planet();
		p.addMoon(new Moon("moon1"));
		p.addMoon(new Moon("moon2"));
		return p;
	}

	@GET
	public Planet planet () {
		return initPlanet();
	}
	
	@GET @Path("/moons/{moonid}")
	public Moon moon (@PathParam("moonid") String name) {
		return new Moon(name);
	}
	
}

Jersey configuration

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" id="WebApp_ID" version="3.0">
  <display-name>test</display-name>
  
  <servlet>
    <servlet-name>Jersey REST Service</servlet-name>
    <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
	<init-param>
	    <param-name>javax.ws.rs.Application</param-name>
	    <param-value>testing.MyApplication</param-value>
	</init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  
  <servlet-mapping>
    <servlet-name>Jersey REST Service</servlet-name>
    <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>
  
</web-app>

Application class

This is the class pointed to by the web.xml, and it really just exists so we can register the declarative linking feature.

package testing;

import org.glassfish.jersey.linking.DeclarativeLinkingFeature;
import org.glassfish.jersey.server.ResourceConfig;

public class MyApplication extends ResourceConfig {
	
    public MyApplication() {
        packages("testing");
        register(DeclarativeLinkingFeature.class);
    }

}

Moxy configuration

Moxy’s ability to configure JAXB mappings entirely from an external file is a cornerstone of my approach to serving up unmodified domain objects.

jaxb.properties

This is how we let Jersey know we want to use Moxy as the JAXB provider.

javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory

The moxy mapping file, planets.oxm.xml

Note that the href elements mentioned in the mapping file do not exist in the model source code.

<?xml version="1.0"?>
<xml-bindings
    xmlns="http://www.eclipse.org/eclipselink/xsds/persistence/oxm"
    package-name="testing"
    xml-mapping-metadata-complete="true"
    xml-accessor-type="NONE">
    <java-types>
        <java-type name="Planet">
            <xml-root-element/>
            <java-attributes>
	           	<xml-attribute java-attribute="href"/>
                <xml-element java-attribute="name"/>
                <xml-element java-attribute="radius"/>
 				<xml-element java-attribute="moons" name="moon">
 					<xml-element-wrapper name="moons"/>
 				</xml-element> 	
            </java-attributes>
        </java-type>
        <java-type name="Moon">
            <xml-root-element/>
        	<java-attributes>
	           	<xml-attribute java-attribute="href"/>
        		<xml-element java-attribute="name"/>
        	</java-attributes>
        </java-type>
    </java-types>
</xml-bindings>

Moxy context resolver

This is how we supply Moxy with the correct mapping file. The @Provider annotation lets Jersey automatically find our context resolver.

package testing;

import java.util.HashMap;
import java.util.Map;

import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.Provider;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import org.eclipse.persistence.jaxb.JAXBContextProperties;

@Provider
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public class MyMoxyContextResolver implements ContextResolver<JAXBContext> {
    private JAXBContext context = null;
 
    public JAXBContext getContext(Class<?> type) {
	System.out.println("Invoking MyMoxyContextResolver.getContext");
    	if (context == null || context != null) {
	        try {
	        	System.out.println("MyMoxyContextResolver - Creating new JAXBContext");
	            Map<String, Object> properties = new HashMap<String, Object>(1);
	            properties.put(JAXBContextProperties.OXM_METADATA_SOURCE, "testing/planets.oxm.xml");
	            context = JAXBContext.newInstance(new Class[] {Planet.class}, properties);
	            
	        } catch(JAXBException e) {
	            throw new RuntimeException(e);
	        }
    	}
       return context;
    }

}

Custom MessageBodyWriter

As per this StackOverflow question, we need to add a custom MessageBodyWriter to convince Jersey to marshall a completely unannotated model. Again, Jersey picks this up automatically.

package testing;

import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
import javax.ws.rs.ext.Providers;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;
import javax.xml.bind.Marshaller;


@Provider
@Produces("application/xml")
public class MyMessageBodyWriter implements MessageBodyWriter<Object> {

	@Context
	protected Providers providers;

	@Override
    public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return true;
    }
 
    @Override
    public long getSize(Object les, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return -1;
    }
 
    @Override
    public void writeTo(Object les, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException, WebApplicationException {
        try {
        	ContextResolver<JAXBContext> resolver = providers.getContextResolver(JAXBContext.class, mediaType);
        	JAXBContext context = resolver.getContext(type);
        	
        	Marshaller m = context.createMarshaller();
			m.marshal(les, entityStream);

        } catch (JAXBException e) {
            e.printStackTrace();
        }
    }
    
}

The aspect

This is what makes it all work. As this project is laid out all in one package, all you need to do is compile with the AspectJ compiler and this aspect will be woven into the model classes. In a more realistic project, you’d have the model classes on the inpath and this aspect on the aspectpath.

package testing;

import org.glassfish.jersey.linking.InjectLink;
import org.glassfish.jersey.linking.Binding;

public aspect HrefInjector {
	
	private String Planet.href;
	declare @field : * Planet.href : @InjectLink(
										resource=Services.class, 
										style=InjectLink.Style.ABSOLUTE
									) ;

	private String Moon.href;
	declare @field : * Moon.href : @InjectLink(
										resource=Services.class,
										method="moon",
										bindings={@Binding(
												name="moonid", value="${instance.name}"
												)},
										style=InjectLink.Style.ABSOLUTE
									) ;

}

Project layout

Screenshot from 2014-06-27 16:40:30

(click to enlarge)

Dependencies

As they appear in the gradle build file:

dependencies {
   compile 'org.glassfish.jersey.containers:jersey-container-servlet:2.9'
   compile 'org.glassfish.jersey.media:jersey-media-moxy:2.9'
   compile 'org.glassfish.jersey.ext:jersey-declarative-linking:2.9'
   
   providedCompile 'javax.servlet:servlet-api:2.5'
}

As always, you can run

./gradlew dependencies

to see the full list of transitive dependencies.

Sample output

XML

	<planet href="http://localhost:8080/reststructlinks/rest/services">
		<name>test</name>
		<radius>3.0</radius>
		<moons>
			<moon href="http://localhost:8080/reststructlinks/rest/services/moons/moon1">
				<name>moon1</name>
			</moon>
			<moon href="http://localhost:8080/reststructlinks/rest/services/moons/moon2">
				<name>moon2</name>
			</moon>
		</moons>
	</planet>

JSON

{
  "href": "http://localhost:8080/reststructlinks/rest/services",
  "name": "test",
  "radius": 3,
  "moons": {
    "moon": [
      {
        "href": "http://localhost:8080/reststructlinks/rest/services/moons/moon1",
        "name": "moon1"
      },
      {
        "href": "http://localhost:8080/reststructlinks/rest/services/moons/moon2",
        "name": "moon2"
      }
    ]
  }
}

Conclusion

A few concluding comments:

  1. There are only two source files concerned with structural linking – the mapping file, and the aspect. The rest of the system operates exactly as per usual. This satisifies the requirement for structural linking to be separated out as a crosscutting concern.
  2. It’s entirely appropriate that the mapping file is involved, as it means you can control when and where you want the structural links by modifying the mapping file.
  3. The aspect requires no special configuration, as it is using bog-standard built-in Jersey functionality.
  4. Jersey’s built-in UriBuilder approach to transitional linking co-exists happily with this aspect-driven approach.
  5. Of course, some people/projects are allergic to AspectJ for whatever reason, so it would still be good to see another candidate solution emerge.

Hope this helps!

Unconfuse yourself about REST, HATEOAS, and API design

I set out to create a simple REST API about a week ago. Having got a basic proof of concept working, I thought I’d look up the accepted wisdom on API design. Well! After a couple of days hacking through the seething jungle of opinion, holy wars and plain misinformation, I now realize it might be a little too soon for “accepted”. However, there were three things I found after much labour that really helped clarify my thinking, so I thought I’d call those out here.

Roy Fielding’s semi-rant on the hypermedia constraint

http://roy.gbiv.com/untangled/2008/rest-apis-must-be-hypertext-driven

This article opened my eyes to the larger issues that REST was originally designed to address. It also is a handy test – if, like me, you find that much of the terminology in this article is unfamiliar, that’s a good sign that you haven’t really engaged with the work that’s been done on those issues. Knowing what you don’t know is always a good thing.

There’s a lot of nonsense not particularly useful stuff out there in the blogosphere about API discoverability, with the proponents struggling to come up with a really convincing benefit, and the opponents holding up the chimera of completely automated discoverability by a completely generic automaton as a kind of reductio ad absurdum. Fielding makes it clear that, while the automaton might be nice if you can afford to build it, a discoverable API is just as important for that most ubiquitous of generic user agents, the next generation of application programmers.

“REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution.” – Roy Fielding, comment #8

What’s a media type and how do you create one?

Two things become clear in the comment stream from the above article (and other sources):

  1. Fielding sees “out-of-band” information (that is, the stuff that the parties in a client-server interaction have to know that is not contained within the interaction itself) as a threat to API longevity
  2. More importantly, and apparently not widely understood, Fielding doesn’t see the media type definition as being out-of-band. In fact it’s the media type definition that saves you from needing out-of-band information.

So what’s a media type and how do you create one? This is apparently widely misunderstood (including by me), and it’s surprisingly hard to find a simple answer.

In fact, there is a very simple answer. A media type is an information standard. You create one by writing it down. You can share it globally by registering it with IANA. The MIME types with which we are all familiar are simply pointers to this registry. Or, you can share it locally by just telling your mates where the standards document is. Here’s an incredibly simple example.

http://www.iana.org/assignments/media-types/media-types.xhtml
http://www.iana.org/assignments/media-types/application/activemessage

Again, the waters are muddied by people leaping immediately to the most difficult case, that of automatic discovery by generic automatons. If your “user agents” are developers, all you need in your media type documentation is human-readable text.

How to design a mashup

You can’t, of course. That’s the point of a mashup. You just “expose” some information and the genius of the crowd will then mash it together with other things in ways that you couldn’t possibly imagine, right? Hands-up everyone who thought that designing REST APIs is about exposing information (I have both hands up at this point). Unfortunately, this is utter nonsense, and even more unfortunately it doesn’t take much thinking to see that it’s nonsense. Who really believes that Google didn’t know people would be embedding maps with pins on them?

Fortunately, the talented guys at Jayway wrote a very nice article called Why hypermedia APIs? which reminded me that designing API’s is very closely related to designing applications (remember what the “A” and the “P” in API stand for?).

http://www.jayway.com/2012/10/06/why-hypermedia-apis/

Designing applications starting with use cases is something I know how to do. Of course it’s blindingly obvious that API’s must also be designed with use cases in mind, but I had allowed myself to be almost terminally distracted by the mashup rhetoric along with with an abundance of bad examples.

Conclusion

Those were my three biggest pain points, which I’ll just restate here for easy access:

  1. What on earth is this HATEOAS stuff all about and what does it mean for my design?
  2. What’s a media type (really)
  3. How do I design something that will be used in “limitless and unimaginable ways”?

The resources I’ve pointed to in this post were invaluable in unconfusing me, so I hope they’ll help you too.

Using Moxy with Jersey for XML Part 3

See part 1 for my initial confused ramblings, and part 2 for a workaround. This third and final part will hopefully bring some clarity to both the initial problem and the solution.

Firstly, a brief statement of how I went astray:

The use case

The requirement is to create a REST API for manipulating an existing domain model. While I do have access to the domain model source code, I’m reluctant to alter it for this specific scenario, so anything requiring annotating the domain model is out. Similarly, while it’s possible to create an intermediate model (e.g. a set of DTO’s) and annotate that, that approach has always seemed to me like excessive double handling. To my mind, XML is supposed to be the dumb DTO format. I know there are use cases which require all these layers, but I also know that most of the time YAGNI.

The solution sketch

  1. JAX-RS seems to be a nicely concise, annotation-driven way to create REST services on a servlet container
  2. Jersey is a JAX-RS implementation with some traction, and some useful integrations with various containers and format providers
  3. By using JAXB with Jersey, we remove the need to write the XML marshalling/unmarshalling layer
  4. JAXB by default is completely annotation-driven, but Moxy is a JAXB implementation that can be configured via an external mapping document. Happily, Jersey ships with Moxy integration.
  5. Finally, Moxy gives us JSON capability for free.

So that was the plan. Implement that stack of frameworks, create a mapping file, and serve up both XML and JSON based on the one unannotated domain model. There are lots of examples around to help get started:

plus many more linked from the above. There’s no example I could find that reproduces all the elements of my solution sketch – Jersey + JAXB + Moxy + external mapping file + unannotated file – but surely there’s enough to get started.

What the documentation doesn’t tell you

Spot the flaw in my thinking:

  1. Jersey integrates with Moxy out of the box – TRUE
  2. Moxy supports the use of an external mapping file – TRUE
  3. When you use Moxy with the external mapping, you don’t need any annotations on the model – TRUE
  4. The Jersey docs even show you how to configure Moxy with a mapping file – TRUE
  5. Therefore, you can use Jersey + Moxy without annotations on the model – FALSE

Stumbled at the last hurdle. To add to the bafflement, this all works flawlessly for JSON marshalling, but fails for XML. I’ll go into more detail below as to why this is so, but for now, here’s what you need to know:

Jersey’s Moxy integration does not expose all of Moxy’s functionality for the XML case. There is still some of the standard annotation-driven JAXB code in the mix, and to get past that your model must be annotated at least with @XmlRootElement

Once you’ve satisified Jersey’s default JAXB provider by adding that one annotation, Jersey will happily hand you over to Moxy and all the external mapping goodness works just fine.

Is there a better way?

But wait, I started off by saying I didn’t want to annotate my model. If you really don’t want to or can’t annotate the model, there is another way. You need to register your own provider. This example gives the basic idea, although I needed to tweak it a bit to get it to work (more on that below). This is in fact what Jersey’s Moxy code does for the JSON case, which is why that works and the XML case doesn’t.

Example code

Most of this example code is identical to all the other examples out there, but I’ll collect it here as the full description of what finally worked for me. This is using Jersey 2.9 and EclipseLink Moxy 2.5 on Java 7. This code can also be found on bitbucket.

web.xml

<?xml version="1.0" encoding="UTF-8"?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://java.sun.com/xml/ns/javaee" xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_3_0.xsd" id="WebApp_ID" version="3.0">
  <display-name>test</display-name>
  
  <servlet>
    <servlet-name>Jersey REST Service</servlet-name>
    <servlet-class>org.glassfish.jersey.servlet.ServletContainer</servlet-class>
    <init-param>
      <param-name>jersey.config.server.provider.packages</param-name>
      <param-value>testing</param-value>
    </init-param>
    <load-on-startup>1</load-on-startup>
  </servlet>
  
  <servlet-mapping>
    <servlet-name>Jersey REST Service</servlet-name>
    <url-pattern>/rest/*</url-pattern>
  </servlet-mapping>
  
</web-app>

JAX-RS service definition

Without the custom provider described further down, this code works for the JSON case (/json) and the two endpoints that use the annotated model (/json2 and /xml2), but fails for the unannotated XML case (/xml).

package testing;

import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;

@Path("/services")
public class Services {

	@GET @Path("/json")
	@Produces(MediaType.APPLICATION_JSON)
	public Planets serviceListJSON () {
		return new Planets();
	}
	
	@GET @Path("/xml")
	@Produces(MediaType.APPLICATION_XML)
	public Planets  serviceListXML () {
		return new Planets();
	}
	
	@GET @Path("/json2")
	@Produces(MediaType.APPLICATION_JSON)
	public PlanetsAnnotated serviceListJSON2 () {
		return new PlanetsAnnotated();
	}
	
	@GET @Path("/xml2")
	@Produces(MediaType.APPLICATION_XML)
	public PlanetsAnnotated  serviceListXML2 () {
		return new PlanetsAnnotated();
	}
	
}

Model classes

package testing;

public class Planets {

    private int id = 1;
    private String name = "test";
    private double radius = 3.0;


}
package testing;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class PlanetsAnnotated {

    private int id = 1;
    private String name = "test";
    private double radius = 3.0;


}

Mapping file

<?xml version="1.0"?>
<xml-bindings
    xmlns="http://www.eclipse.org/eclipselink/xsds/persistence/oxm"
    package-name="testing"
    xml-mapping-metadata-complete="true"
    xml-accessor-type="NONE">
    <java-types>
        <java-type name="Planets">
            <xml-root-element/>
            <java-attributes>
                <xml-element java-attribute="id"/>
                <xml-element java-attribute="name"/>
                <xml-element java-attribute="radius"/>
            </java-attributes>
        </java-type>
         <java-type name="PlanetsAnnotated">
            <xml-root-element/>
            <java-attributes>
                <xml-element java-attribute="id"/>
                <xml-element java-attribute="name"/>
                <xml-element java-attribute="radius"/>
            </java-attributes>
        </java-type>
    </java-types>
</xml-bindings>

jaxb.properties

javax.xml.bind.context.factory=org.eclipse.persistence.jaxb.JAXBContextFactory

Context resolver

This is one of several ways to give Moxy the mapping file.

package testing;

import java.util.HashMap;
import java.util.Map;

import javax.ws.rs.Produces;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.Provider;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

import org.eclipse.persistence.jaxb.JAXBContextProperties;

@Provider
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public class MyMoxyContextResolver implements ContextResolver<JAXBContext> {
    private JAXBContext context = null;
 
    public JAXBContext getContext(Class<?> type) {
	System.out.println("Invoking MyMoxyContextResolver.getContext");
    	if (context == null) {
	        try {
	            Map<String, Object> properties = new HashMap<String, Object>(1);
	            properties.put(JAXBContextProperties.OXM_METADATA_SOURCE, "testing/planets.oxm.xml");
	            context = JAXBContext.newInstance(new Class[] {Planets.class}, properties);
	        } catch(JAXBException e) {
	            throw new RuntimeException(e);
	        }
    	}
       return context;
    }

}

Custom provider

This is the part that makes the XML case finally work without annotations. For a full solution this would implement MessageBodyReader as well.

package testing;

import java.io.IOException;
import java.io.OutputStream;
import java.lang.annotation.Annotation;
import java.lang.reflect.Type;

import javax.ws.rs.Produces;
import javax.ws.rs.WebApplicationException;
import javax.ws.rs.core.Context;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.MultivaluedMap;
import javax.ws.rs.ext.ContextResolver;
import javax.ws.rs.ext.MessageBodyWriter;
import javax.ws.rs.ext.Provider;
import javax.ws.rs.ext.Providers;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBException;

@Provider
@Produces("application/xml")
public class MyMessageBodyWriter implements MessageBodyWriter<Object> {

	@Context
	protected Providers providers;

	@Override
    public boolean isWriteable(Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return true;
    }
 
    @Override
    public long getSize(Object les, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType) {
        return -1;
    }
 
    @Override
    public void writeTo(Object les, Class<?> type, Type genericType, Annotation[] annotations, MediaType mediaType, MultivaluedMap<String, Object> httpHeaders, OutputStream entityStream) throws IOException, WebApplicationException {
        try {
        	ContextResolver<JAXBContext> resolver = providers.getContextResolver(JAXBContext.class, mediaType);
        	JAXBContext context = resolver.getContext(type);
        	
            context.createMarshaller().marshal(les, entityStream);
        } catch (JAXBException e) {
            e.printStackTrace();
        }
    }
    
}

Source code layout

This is my example structure for a gradle project.

src/
   main/
      java/
           jaxb.properties
           MyMessageBodyWriter.java
           MyMoxyContextResolver.java
           Planets.java
           planets.oxm.xml
           PlanetsAnnotated.java
           Services.java
   webapp/
      WEB-INF/
           web.xml 

Help!

This is a super-simple proof-of-concept. No doubt it has all sorts of sub-optimal choices, and for all I know breaks all sorts of other use cases while solving mine. However, as far as I can tell there is no other example out there for this specific scenario. It would be great if this could be evolved into something more robust, so please let me know if you have any suggestions.

The gory details

This is a sketch of what’s happening in the innards of Jersey when using JAXB to marshal the results of a service call. I’m including this as background to any future discussion on what a supported solution might look like. For example, maybe MoxyXmlFeature should register a provider that relaxes the requirement for @XmlRootAnnotation.

Jersey

At startup time:

  • Jersey registers built-in providers
  • Features (discovered on the classpath) register providers and context resolvers
  • The jaxb.properties file defines the JAXBContextFactory to be used

At request processing time:

  • The annotated JAX-RS service method returns a Java object
  • Jersey’s interceptor fires (triggered by the @Produces annotation)
  • Jersey searches its list of providers based on the class of the returned Java object and on the media type required
  • The chosen provider searches for a context resolver that is configured to provide a context for the Java class
  • The chosen context resolver obtains and configures a JAXBContext and returns it to the provider
  • The JAXBContext returns a marshaller
  • The provider invokes the marshaller, which emits the marshalled representation of the Java object

Each provider has a list of rules defining what it will handle. For example, there’s a built-in XmlRootElementProvider, which will only handle classes annotated with @XmlRootElement for media type application/xml. If no matching provider can be found, an exception is thrown.

So, on to Moxy. The jersey-media-moxy module, which ships with Jersey and provides Moxy integration, registers two features – MoxyJsonFeature and MoxyXmlFeature.

MoxyJsonFeature registers a provider, ConfigurableMoxyJsonProvider, which will handle marshalling any object where the media type is application/json. You can then use either the MoxyJsonConfig method or register your own ContextResolver to configure a mapping file if so desired.

MoxyXmlFeature, on the other hand, does not register a provider. For annotated models, one of Jersey’s built-in providers will work just fine, but for non-annotated models that means that no provider will be found and an exception will be thrown. MoxyXmlFeature does register a context resolver, thus providing a mechanism for configuring a mapping file, but without any annotations to get you past the Jersey provider that context resolver will never be invoked. Hence the need to create our own provider.

Using Moxy with Jersey for XML Part 2

In part 1 I described the ways in which the documented approach doesn’t work. This post is about my first workaround, but for a comprehensive debrief, see part 3.

The approach that works is:

  1. Build with the jersey-media-moxy module – so
  2. Put the jaxb.properties file in the right place – so
  3. Create a mapping file – so
  4. Create a ContextResolver – example. This will be used for JSON but not for XML.
  5. Create a MessageBodyWriter – example. This will be used for XML.
  6. Most importantly, write and let me know where on earth I went wrong to need to jump through all these hoops 🙂
  7. Check out this little demo project to see how it all fits together.