Using Moxy with Jersey for XML

This post is about my initial attempts to get Jersey and Moxy working the way I want. See part 3 for the full analysis of why I was having trouble and my eventual solution.

Jersey is an implementation of JAX-RS, the Java standard for REST web services. To this you can add JAXB, which allows your REST services to use declarative metadata to map domain objects to a wire format, typically either JSON or XML.

The downside of the JAXB reference implementation is that the metadata is expressed as annotations on the domain model. Personally, I think that this kind of mapping is definitely a cross-cutting concern and is to be kept out of the domain model source code at all costs. Moxy comes to the rescue! You can plug Moxy in as your JAXB provider, replacing the reference implementation entirely, and Moxy supports externalizing the mapping metadata to an XML file. So, how to get it working?

The Jersey documentation is very simple and to the point. To use Moxy for JSON binding, just place the appropriate jar on the classpath – see instructions. You can then use a custom JAXB context resolver to set Moxy up with a mapping file, and away you go.

The documentation for using Moxy for XML binding is even simpler. The simplest way is:

… use the standard JAXB mechanisms to define the JAXBContextFactory from which a JAXBContext instance would be obtained (for more on this topic, read JavaDoc on JAXBContext)…

That boils down to putting a jaxb.properties file alongside your model, as documented here.

The alternative mechanism is to manually configure Jersey, registering and configuring the MoxyXmlFeature.

However, neither mechanism actually works. It appears that, while Jerseys Moxy JSON feature registers a MessageBodyWriter to handle JSON writes, the XML feature does not do anything of the sort, instead falling back onto Jersey’s default XML MessageBodyWriter. Unfortunately, the default XML writer appears to completely ignore any registered ContextResolvers you’ve registered. Thus, while you can marshall a completely unannotated domain object to JSON, if you try to marshall the same object to XML you’ll get an exception.

See https://bitbucket.org/jmetcher/resttest for a small project showing Jersey setup strictly according to the documentation. Check out the master branch for the basic setup.

The solution is to register your own custom MessageBodyWriter, as described here. Sadly, as far as I can tell this is 100% missing from the documentation. Check out the MBW branch of the above project to see what it looks like with the recommended MessageBodyWriter approach. Unfortunately, this doesn’t work either. It appears that, contrary to about a dozen examples dotted around the web, injection of the ContextResolver does not work, as you’ll see by the NPE if you run that code.

Finally, have a look at the MBW_Better branch of the project. As this one actually works, I’ll pull it out into a separate post.

Simple way to marshall POJOs using JAXB

JAXB is a specification for marshalling and unmarshalling java objects to and from XML. The normal way (i.e. most commonly documented) to use it is to annotate your Java object. The annotations give you a lot of control, but even if you’re happy with the default settings, you still need to at least provide the @XMLRootElement annotation to make your class visible to JAXB, like so:

package jaxbtest;

import javax.xml.bind.annotation.XmlRootElement;

@XmlRootElement
public class ServiceDTO {
	public MyDomainClass mdc;
	
    public ServiceDTO() {}
    
    public ServiceDTO(MyDomainClass mdc) {
    	this.mdc = mdc;
    }
}

OK, this isn’t a real DTO. The next step would be to copy all the required data from the domain class into fields of the DTO. But it will serve our purpose for the moment.

To marshal ServiceDTO to XML, we can then do this:

    	MyDomainClass mdc = new MyDomainClass();

    	JAXBContext jc = JAXBContext.newInstance(ServiceDTO.class);
    	Marshaller marshaller = jc.createMarshaller();
    	marshaller.marshal(new ServiceDTO(mdc), System.out);

What if you can’t or don’t want to add JAXB annotations to your domain? See this very helpful post by the very helpful Blaise Doughan. You can still get fine-grained control over marshalling and unmarshalling by using Moxy to external JAXB configuration.

There is another way, though. In the java doc for JAXBContext, we find this intriguing statement:

Not only the new context will recognize all the classes specified, but it will also recognize any classes that are directly/indirectly referenced statically from the specified classes

That means that, given the above code, we can also marshal the domain object directly, like this:

marshaller.marshal(mdc, System.out);

As long as your class is referenced by an annotated class, it’s fair game for the marshaller.

I haven’t seen this technique mentioned at all in my research, so I presume there are some fundamental limitations to it. If you know of any, please let me know in the comments.

Eclipse WTP + Gradle

This post is a collection of handy guides for setting up Eclipse and Gradle for servlet development.

Why do this? Eclipse is a nice development environment, and Eclipse WTP with its servlet container integration provides a slick code/compile/deploy/debug cycle. But Eclipse is not a build system. In particular, if you want transitive dependency management, you’ll need something like Gradle or Maven. The downside is that you then have two completely separate configurations to manage. However, both Gradle and Maven supply utilities to assist in keeping the two in sync.

So, the guides:

http://www.vogella.com/tutorials/EclipseWTP/article.html
This is a great guide for setting up and testing an Eclipse WTP environment with embedded Tomcat

You may then run into this bug in Eclipse:
http://stackoverflow.com/questions/14791843/eclipse-add-tomcat-7-blank-server-name

and, on Ubuntu, this bug:
http://stackoverflow.com/questions/13423593/eclipse-4-2-juno-cannot-create-a-server-using-the-selected-type-in-tomcat-7

Gradle uses a different default project layout to the default Eclipse dynamic web project layout. Both are configurable, but the path of least resistance is to set up your project the Gradle way, like this:
https://weblogs.java.net/blog/manningpubs/archive/2013/03/18/building-java-web-application-gradle

and then import that configuration into eclipse:
http://www.gradle.org/docs/current/userguide/eclipse_plugin.html
and
http://www.gradle.org/docs/current/dsl/org.gradle.plugins.ide.eclipse.model.EclipseWtp.html

At this point, you know enough to create a single project that you can fire up on Tomcat within Eclipse, or Jetty via Gradle. Any container-specific configuration will be a problem, of course, and unfortunately the Jetty Eclipse adapter is no longer maintained. There is a Tomcat plugin for Gradle, but I haven’t yet tried it.

Hibernate, Javassist proxies and equals()

Hibernate uses Javassist proxies to help with lazy loading. If you’ve got no idea what that means, don’t worry – to use Hibernate, you’re not supposed to need to know anything about it. However, it is another one of those clever “transparent” framework programming techniques that can really give you some headaches in debugging if you don’t know it’s there (see http://lagod.id.au/blog/?p=266 for a couple more). There are four scenarios where the transparency gets a bit murky and you need to a know a bit more of what’s under the hood to avoid going crazy.

The first two, along with a good description of what Hibernate is doing with the proxies and why, can be found here: http://blog.xebia.com/2008/03/08/advanced-hibernate-proxy-pitfalls/. The third is any kind of reflective programming – see http://blog.jeremymartin.name/2008/02/java-reflection-hibernate-proxy-objects.html for a nice description of a typical issue and a good workaround. [Edit March 2014: Also applies to any framework that uses reflection, such as Apache BeanUtils/PropertyUtils].

The fourth, which is the one I really want to talk about, is the fact that the proxy actually is not strictly equal to the real object (in the sense of ==). Sounds obvious really. Not usually a problem, as you usually just have a reference to the proxy. However, there is a particular circumstance where you can end up with a reference to both the proxy and the real object. You will almost certainly believe them to be the identical object. There will be nothing about their behavior that will indicate to you that they are not, except for the fact that when you compare the with == you’ll get a false result. In fact, because the default implementation of equals() uses ==, the same applies to comparing them with equals().

You don’t need to do anything too wacky to end up in this very odd state. Consider this object:

class  MyClass {

Item head;

List items = new ArrayList(0);

// ... plus getters and setters
}

mapped to a database structure that looks like (pseudocode):

Table MyClass
Integer myClassID
Integer FK_headID

Table Item
Integer itemID
Integer FK_myClassID

So it’s a list of items, with an extra reference to one of the items broken out into a separate field for easy access. A bit denormalised, but if it’s a big list and it’s expensive to calculate “head”, it’s the kind of thing we might do without thinking too much about it. The important thing is that “head” is also in the list. Hibernate supports this mapping quite happily. You can load, manipulate and save this object graph all day long. However, this:

foreach (Item i: myClass.getItems()) {
  System.out.println(i.equals(myClass.getHead()));
}

will print false for every value in the list, even when you’ve confirmed that for one of them i.getItemId() == head.getItemId().

So what’s the trick? Using a standard many-to-one mapping for head, and a standard one-to-many mapping for the item list, myClass.getHead() returns a Javassist proxy object, and iterating through the list returns the unproxied object. If you look at it in a debugger, you can see that the real object backing the proxy is in fact the identical object that you get from the list – this is not a case where you’ve inadvertently mappedd one database record into two different in-memory objects. However, realobject != proxy (and vice versa).

The fix is easy. Although lazy loading is the default for Hibernate, it’s almost never a significant benefit for single objects. Simply set lazy=”false” in your mapping file for the many-to-one association. You can leave lazy loading enabled for the list, as it works by a different mechanism. If you really do need lazy loading for the single object and don’t mind a slightly more complex build configuration, there’s lazy=”no-proxy” (documented here: http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/performance.html#performance-fetching-proxies). And if you don’t mind Hibernate-specific code in your domain model (something I avoid like the plague, personally), you can also explicitly “unwrap” the proxy as described here: http://stackoverflow.com/questions/16383742/hibernate-javassistlazyinitializer-problems-with-validation.

So there you have it. A lesson in leaky abstraction, and, for a refreshing change, a head-scratching object equality problem that didn’t, for once, turn out to be some semi-intractable classloader wackiness.

ColdFusion 9 behaviour with java.lang.Error

It seems that a java.lang.Error can cause problems for a CF9 server.

The test case looks like this:



Couple interesting things happen when I run this:

Firstly, I get no error page. No response body at all, just a response header with a status of “500 blah”. Fair enough, java.lang.Error is supposed to mean a serious unrecoverable error so when receiving it ColdFusion is quite right in just stopping. File that under “things I should have known but for some reason never came across”.

Secondly, and more worryingly, if I refresh the page a couple of times I pretty quickly run into this:

Server Error
The server encountered an internal error and was unable to complete your request.

Application server is busy. Either there are too many concurrent requests or the server still is starting up.

with the accompanying entry in the Apache error log

returning error page for Jrun too busy or out of memory

In fact I might get that “server busy” error for any other page on the server, not just a refresh of my test page. After a while it “fixes” itself, as in no more server busy errors. Until the next time I run my test page.

What does this all mean? My guess would be that Jrun quite reasonably decides to nuke the thread that has reported the serious and unrecoverable error, but then quite unreasonably sends the next request into the post-nuke radioactive wasteland of that thread. The thread, in a fit of post-holocaust dudgeon, refuses to respond at all, and Apache can do nothing but convey the bad news to the long-suffering user. Once the thread has expired out of the thread pool the problem vanishes.

It’d be interesting to know if this happens in CF10. In the meantime, I’m off to work out why my java code is throwing a java.lang.Error in the first place.

Environment stuff: CF 9.02 running whatever version of Apache2 is current for Ubuntu 12.04; CF9.01 on ditto ditto Apache2 ditto Ubuntu 9.04

More fun with the (ColdFusion) truth

In my last post I discovered that ColdFusion

true

is the string

"true"

. You can compare it to a java boolean true using “is”, but not using .equals(). Fair enough, one is a string and the other isn’t. “is” is ColdFusion magic, and .equals() is just poor dumb java. And if you’re wondering, “eq” is just as magic as “is”.

Does that mean all ColdFusion booleans are actually strings. No! The result of a boolean expression in ColdFusion is actually a java boolean.


a = true; // this is a string
b = "true"; // this is the same string - literally the same in-memory object
c = a is true; // this is a boolean
d = b is true; // this is literally the same boolean

a is c; // this is true
a.equals(c); // this is false
// etc.

So it turns out that

true

is not, as I thought, a built-in boolean constant, it’s actually a special syntactical case for creating a string – that can then be used in ColdFusion boolean expressions. This is reminding me of my perl days (that’s a good thing – I loved perl).

If you never had to interoperate with Java none of this would matter. The booleans and strings intermix seamlessly within ColdFusion. If you do, it’s important to know that ColdFusion actually does not have any built-in boolean constants that Java will recognise.


myJavaFunc(true);  // this won't work

myJavaFunc(true is true); // Somewhat disturbingly, this will

myJavaFunc(javacast("boolean", true)); // so will this

myJavaFunc(CreateObject("java","java.lang.Boolean").init("true")); // and so will this

I’d tend to go with the third form using javacast. Use the second form only if you want to mess with your junior programmers heads 🙂 Reminds me of those crazy funsters who put “and 1=1” at the end of all their SQL queries.

ColdFusion true is not a Java boolean

None of these ColdFusion constructs strictly equate to a true-valued Java boolean (or Boolean):

true
"true"
1
YES

But this does:

JavaCast("java.lang.Boolean", true);

Makes me wonder what ColdFusion’s literal true (no quotes) actually is. Probably just a string.

Edit prompted by Todd:
Definitely just a string. And definitely not a boolean. Also not particularly suprising but something I’d never thought about is the fact that ColdFusion “is” is not the same as Java “equals”.


a = true;
b = CreateObject("java","java.lang.Boolean").init("true");
c = "true";



	a = true;
b = CreateObject("java","java.lang.Boolean").init("true");
c = "true";



b.equals(a) #b.equals(a)#
a.equals(b) #a.equals(b)#
b.equals(JavaCast("boolean",a)) #b.equals(JavaCast("boolean",a))#
b is a #b is a#
a is b #a is b#

a.equals(c) #a.equals(c)#
c.equals(a) #c.equals(a)#
a is c #a is c#
c is a #c is a#

b.equals(c) #b.equals(c)#
c.equals(b) #c.equals(b)#
b is c #b is c#
c is b #c is b#

gives this:

a = true;
b = CreateObject("java","java.lang.Boolean").init("true");
c = "true";

java.lang.String

b.equals(a) NO
a.equals(b) NO
b.equals(JavaCast("boolean",a)) YES
b is a YES
a is b YES

a.equals(c) YES
c.equals(a) YES
a is c YES
c is a YES

b.equals(c) NO
c.equals(b) NO
b is c YES
c is b YES

Unit testing persistence with template methods

Now that we have techniques for making sure our tests hit the database, what are we going to test? Load by ID, load by name, load all, save, delete – sound like a reasonable starting point? I also throw in tests for what I call “finders”, which are utility classes for loading objects by anything other than primary or natural keys. Essentially these are DAOs, but I call them finders because I use the same idea for non-persistent objects as well. I must emphasise again – I’m only trying to exercise my persistence layer here. Domain-specific behaviour is tested in a different test suite.

One more concept before we get down to tin tacks. All my domain objects are accessed via a “manager” classes, which have methods like – you get guessed it – load by ID, load by name, save, delete, etc. The key to this approach to test generation is there’s a high level interface for managers, an interface for finders, and another one for domain objects. The specifics aren’t as important as the fact these interaces exist.

OK, with all of that, the test method for loading a generic ObjectType by ID looks like this:

public abstract class BasePersistenceTests extends BaseContextTests {
	@Test
	public void testGetByID() {
		
		ObjectType domainObject2 = getChildObjectManager().get(domainObject.getId());
		
		if (hasTextIDName()) {
			assertEquals(testObjectName, getTextIDValue(domainObject2));
		}
		assertEquals(domainObject.getId(), domainObject2.getId());

	}
}

As you can see, this is a method on an abstract base class. It’s the same for every test class for every domain object. Obviously there’s a bit more going on here, so let’s go through that.

getChildObjectManager() is the only piece that is different for each domain object under test. It specifies the the manager for this object, and once that’s provided our base classes can use our standard interfaces for everything they need to do.

domainObject was set up in our pre-test transaction that we talked about previously. Still within this same BasePersistenceTests class, we have:

	protected ObjectType domainObject;
	protected String testObjectName = "testObj";

	protected void doSetupBeforeTransaction() {
		domainObject = createChildObject();
	}
	protected ObjectType createChildObject() {
		return getChildObjectManager().create(testObjectName);
	}

doSetupBeforeTransaction() is invoked by our pre-test transaction setup code. Once again, identical for every domain object test class. In true template method style, we could override either or both of these methods for more exotic cases – for example, when we need to supply more inputs to the create() method. Although it’s amazing for how many domain objects you can create an instance with sensible defaults just by specifying a name.

hasTextIDName() and getTextIDValue(): I won’t go into the gory details. Usually hasTextIDName() just returns true (you can override it to return false), and getTextIDValue() uses reflection to look for a special annotation and get the value of the annotated field.

getID() is defined by the base domain object class – pretty obvious what this does.

So, our minimal test class looks like this:

public class myObjectPersistenceTests extends
		myPackagePersistenceTests {

	@Override
	protected DomainObjectManager getChildObjectManager() {
		return myServiceSingleton.getMyObjectManager();
	}

}

That’s the whole thing – only the imports left out for brevity. I get 11 standard persistence tests – the getByID I showed you above, plus a few more I didn’t show – for half-a-dozen lines of code.

This simplest case is for an aggregate root. Things get slightly more complex for child objects within an aggregate, and more complex again for association objects. I top out at about a dozen overridden primitive operations, for about 60 lines of code in the test class, but get a couple of extra tests for testing things like cascades.

Now, obviously, if you’d read this far, I haven’t left you with any runnable code. For you to run my code you’d need to have your domain model set up just like mine, with all the same base classes and interfaces, and that’s not very likely, is it? At some point I might publish my base classes and interfaces, and you can port your code base over to my system ;). In the meantime, the take-home message is:

  • You can and should test your persistence layer/mappings, as well as domain object behaviour
  • Persistence layer testing has overheads and fragilities not needed by behaviour testing, so do it in a separate test suite
  • A decent set of base domain classes makes it possible to write generic base test classes
  • Use of the template method pattern means you only write the standard stuff once – and there’s a lot of standard stuff
  • All of which makes life much less boring 🙂

Previous posts:
Spring/Hibernate unit testing part 2
Spring/Hibernate unit testing

Spring/Hibernate unit testing part 2

The basic Spring transactional testing approach is great for testing domain logic. If you want to test your persistence layer, though, it falls short in some important ways. Most importantly, with the usual approach you will never actually load an object from the database. All the objects you create will hang around in Hibernate’s session cache, and will always be fetched from there. Even objects you created in a @Before method will be within the one Hibernate session.

As an example of why this matters, you can misspell your primary key name in the hibernate mapping file, and these tests will never tell you. Don’t ask me how I know 🙂

The solution is to create some test data before the main test transaction kicks in. That has some significant disadvantages, so I only do it for test classes that are specifically testing the persistence layer. What are the disadvantages?

  • If something goes wrong with your setup or teardown code you can be left with a dirty database and you’ll have to clean up before any further tests will work. In a CI environment that can be a real pain, so ideally you’d be regenerating your test database after each run of the persistence tests.
  • Because you need three transactions per test, it’s significantly slower
  • You have to write the transactional boilerplate for setup and teardown

So, it’s worth doing, but it by no means replaces the basic single-transaction approach. Most of your tests should still be using that approach.

In use, the persistence test class looks something like this:

@ContextConfiguration(locations={"/myApplicationContext.xml"})
public class MyPersistenceTests extends  AbstractTransactionalJUnit4SpringContextTests   {
        @Resource protected SessionFactory sf;
	@Resource protected org.springframework.orm.hibernate3.HibernateTransactionManager txManager;

	@BeforeTransaction
	public void setupBeforeTransaction() {
		DefaultTransactionDefinition def = new DefaultTransactionDefinition();
		def.setName("SomeTxName");
		def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);

		TransactionStatus status = txManager.getTransaction(def);
		try {
			// Your setup code here
			txManager.commit(status);
		}
		catch (Exception e) {
			txManager.rollback(status);
			throw new Error (e);
		}
	}
	
	@AfterTransaction
	public void teardownAfterTransaction() {
		DefaultTransactionDefinition def = new DefaultTransactionDefinition();
		def.setName("SomeTxName");
		def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);

		TransactionStatus status = txManager.getTransaction(def);
		try {
			// your teardown code here
			txManager.commit(status);
		}
		catch (Exception e) {
			txManager.rollback(status);
			throw new Error (e);
		}

               // Your test methods here
	}

We’re still extending the same Spring base class as before. We’ve just injected our transaction manager and added a couple of methods. The @BeforeTransaction and @AfterTransaction annotations are provided by Spring for exactly this use case. Spring will still wrap our test methods in a transaction automatically, and still roll it back after each method invocation.

So now in our test methods we can load any of the objects we created in our “before transaction” code and be sure we are hitting the database.

After writing 20 or 30 of these persistence test classes you start to notice that they all look very similar. Create an object. Make sure you can load it. Tear it down. This is even more pronounced if you use a base domain object class as I do. In my next post I’ll talk about some techniques for writing abstract test classes that do most of the repetitive stuff.

Spring/Hibernate unit testing

Spring provides some nice tools to assist in unit testing with Hibernate, which are well covered in Spring’s excellent documentation. I want to recap some basics here, though. Later on I want to talk about persistence layer test generation, and I’ll need to refer back to these notes.

One of Spring’s convenient tools is a base test class that does a number of nifty things:

  • Sets up your application context, including Hibernate SessionFactory if you have one.
  • Uses that context to do DI into your test classes, which is a nice way to access your singletons within your tests.
  • Wraps each test method in a transaction which gets rolled back at the end of the method. Complete test isolation with zero work on your part!

In use, it looks something like this:

// This tells spring where to find my config.  Note that this could be a special test config rather than my production config
@ContextConfiguration(locations={"/myApplicationContext.xml"})
public class MyContextTests extends  AbstractTransactionalJUnit4SpringContextTests   {
	
        // Standard Spring DI
	@Resource protected SessionFactory sf;
	
       // a basic unit test (utility methods omitted).  Note that I'm creating new objects and flushing them to the database,
       // but not bothering to clean up in any way.
	@Test
	public void testCategoryManager_createDuplicateInDifferentService() {		
		Category category = createCategory("PCTEST");
		assertEquals("PCTEST", category.getName());
		sf.getCurrentSession().flush();	
		createCategoryFromOtherService("PCTEST");
	}	
}

This is a godsend for most test scenarios. It falls down in some ways, though, and I’ll talk about that in my next post.