A tale of utilization vs performance

I once worked on a SaaS product that’s typical of a lot of B2B and niche B2C products.  It had a userbase in the thousands or tens of thousands, and provided business functions that involved heavyweight, complex workflows.  Not many users, lots of compute.  I would say that this is the norm for line of business applications, outside of the industry rockstars that are in the news all the time.

We didn’t need docker.  Docker’s claim to fame is that it’s lightweight.  Once you deploy a multi-gigabyte JVM app that can keep lots of cores busy, it really doesn’t matter if you’ve deployed it onto a VM or a container.  The host overhead is in the noise.   In a containers-on-VMs scenario, no VM is ever going to run more than one instance of the container, so the container layer is just management and skillset overhead.

We didn’t need autoscaling.  This application could keep a server farm busy, but it really only needed the one server farm.  It was easily sharded, if we did hit those limits.  It wasn’t bursty, and it used enough queuing that it could handily absorb the bursts that did happen.

We didn’t need Kubernetes because we didn’t need docker or autoscaling.  Again, unnecessary overhead in skillset and tooling.

We didn’t need these things because we weren’t twitter.  We weren’t even one micro-twitter (if that’s a unit of scale).  We might have got to one million users total, eventually (although I don’t think they ever did). We weren’t ever going to get to one million users a month, let alone a day.

We did have performance problems.  Of course we did.  They were caused by bad SQL and bad algorithms.  We know those were the causes, because we could see the issues staring us in the face.  Every time we did a PoC optimisation exercise, we could easily find improvements of multiple orders of magnitude.  But we never committed to regression testing those and getting them through to production.

Instead, we addressed our performance problems by spending so much on infrastructure that we could have hired two or three more developers.  You have to buy a lot of infrastructure to make your application 1000 times faster.  We settled for a bit less than that.

But it didn’t matter how fast or slow our application was, because we fixated on utilization.  The faster we made our application, the worse our utilization looked.  To a server admin, an average CPU utilization of 20% looks like a healthy server.  To an accountant, it looks like an 80% cost saving waiting to be had.

So we took our slow, unoptimized application, and moved it to docker and Kubernetes.  We didn’t get our extra developers, so we never got to optimize at all.  We took a big hit in training and migration, so productivity dipped.  Reliability got worse for a while, because we made some mistakes in ops.  And our application still had performance problems, because any one request by any one user was still running massively underperforming algorithms and overwhelming the database with unoptimisable queries and deadlocks.

However, our utilization figures were immaculate.  As for the performance issues: when I left, they were talking about putting those bad algorithms into lambdas.


The app-pocalypse ourobouros

The New York Times reports on Google et al.’s crusade to save us from app fragmentation.


The writer of the NY Times piece (and/or his sub-editors) has achieved the superhuman feat of not drawing any conclusions from this evocative material, but the signposts are all there:

“It is not just a matter of consumer convenience. For Google and Facebook, and any company that has built its business on the web, it is a matter of controlling the next entryway to the Internet — the mobile device.”

“But as people spend more time on their mobile devices and in their apps, their Internet has taken a step backward, becoming more isolated, more disorganized and ultimately harder to use — more like the web before search engines.

“How remarkable it is that we are back in 1997!” said Roger McNamee, co-founder of Elevation Partners, an investment firm in Menlo Park, Calif.”

“Take Google, which makes money helping people search the web. When people search in apps, it is mostly left out. And while the company has a fast-growing business selling apps through devices that use its Android operating system, that pales in comparison to its business selling search advertising.”

““Once we’re all using the same plumbing, everyone can go and build businesses and interesting experiences on top of that,” said Eddie O’Neil, a Facebook product manager working on the company’s program, App Links.”

The ironies are many, exquisite, and multi-layered. The complete and transparent prioritization of commercial value over social value is, well, honest. Let’s make some noise, then sell earplugs. One company’s sub-standard product is another company’s value-add opportunity. Well, actually, it’s the same company’s value-add opportunity. That what makes it so fun, amirite?

One of the more remarkable ironies is this: Apple, having created the app-pocalypse and crippled web interoperability with the whole Flash fiasco, remains “cool”; Google, having enthusiastically out-app-ed Apple and upped the ante by turning their web browser, of all things, into an app platform, remains “good” (or at least “not evil”); and Microsoft, having produced the most web-capable platform of the three (as I have remarked before), and been vilified by the tech press for its efforts, remains “evil”.

Unconfuse yourself about REST, HATEOAS, and API design

I set out to create a simple REST API about a week ago. Having got a basic proof of concept working, I thought I’d look up the accepted wisdom on API design. Well! After a couple of days hacking through the seething jungle of opinion, holy wars and plain misinformation, I now realize it might be a little too soon for “accepted”. However, there were three things I found after much labour that really helped clarify my thinking, so I thought I’d call those out here.

Roy Fielding’s semi-rant on the hypermedia constraint


This article opened my eyes to the larger issues that REST was originally designed to address. It also is a handy test – if, like me, you find that much of the terminology in this article is unfamiliar, that’s a good sign that you haven’t really engaged with the work that’s been done on those issues. Knowing what you don’t know is always a good thing.

There’s a lot of nonsense not particularly useful stuff out there in the blogosphere about API discoverability, with the proponents struggling to come up with a really convincing benefit, and the opponents holding up the chimera of completely automated discoverability by a completely generic automaton as a kind of reductio ad absurdum. Fielding makes it clear that, while the automaton might be nice if you can afford to build it, a discoverable API is just as important for that most ubiquitous of generic user agents, the next generation of application programmers.

“REST is software design on the scale of decades: every detail is intended to promote software longevity and independent evolution.” – Roy Fielding, comment #8

What’s a media type and how do you create one?

Two things become clear in the comment stream from the above article (and other sources):

  1. Fielding sees “out-of-band” information (that is, the stuff that the parties in a client-server interaction have to know that is not contained within the interaction itself) as a threat to API longevity
  2. More importantly, and apparently not widely understood, Fielding doesn’t see the media type definition as being out-of-band. In fact it’s the media type definition that saves you from needing out-of-band information.

So what’s a media type and how do you create one? This is apparently widely misunderstood (including by me), and it’s surprisingly hard to find a simple answer.

In fact, there is a very simple answer. A media type is an information standard. You create one by writing it down. You can share it globally by registering it with IANA. The MIME types with which we are all familiar are simply pointers to this registry. Or, you can share it locally by just telling your mates where the standards document is. Here’s an incredibly simple example.


Again, the waters are muddied by people leaping immediately to the most difficult case, that of automatic discovery by generic automatons. If your “user agents” are developers, all you need in your media type documentation is human-readable text.

How to design a mashup

You can’t, of course. That’s the point of a mashup. You just “expose” some information and the genius of the crowd will then mash it together with other things in ways that you couldn’t possibly imagine, right? Hands-up everyone who thought that designing REST APIs is about exposing information (I have both hands up at this point). Unfortunately, this is utter nonsense, and even more unfortunately it doesn’t take much thinking to see that it’s nonsense. Who really believes that Google didn’t know people would be embedding maps with pins on them?

Fortunately, the talented guys at Jayway wrote a very nice article called Why hypermedia APIs? which reminded me that designing API’s is very closely related to designing applications (remember what the “A” and the “P” in API stand for?).


Designing applications starting with use cases is something I know how to do. Of course it’s blindingly obvious that API’s must also be designed with use cases in mind, but I had allowed myself to be almost terminally distracted by the mashup rhetoric along with with an abundance of bad examples.


Those were my three biggest pain points, which I’ll just restate here for easy access:

  1. What on earth is this HATEOAS stuff all about and what does it mean for my design?
  2. What’s a media type (really)
  3. How do I design something that will be used in “limitless and unimaginable ways”?

The resources I’ve pointed to in this post were invaluable in unconfusing me, so I hope they’ll help you too.

Eclipse WTP + Gradle

This post is a collection of handy guides for setting up Eclipse and Gradle for servlet development.

Why do this? Eclipse is a nice development environment, and Eclipse WTP with its servlet container integration provides a slick code/compile/deploy/debug cycle. But Eclipse is not a build system. In particular, if you want transitive dependency management, you’ll need something like Gradle or Maven. The downside is that you then have two completely separate configurations to manage. However, both Gradle and Maven supply utilities to assist in keeping the two in sync.

So, the guides:

This is a great guide for setting up and testing an Eclipse WTP environment with embedded Tomcat

You may then run into this bug in Eclipse:

and, on Ubuntu, this bug:

Gradle uses a different default project layout to the default Eclipse dynamic web project layout. Both are configurable, but the path of least resistance is to set up your project the Gradle way, like this:

and then import that configuration into eclipse:

At this point, you know enough to create a single project that you can fire up on Tomcat within Eclipse, or Jetty via Gradle. Any container-specific configuration will be a problem, of course, and unfortunately the Jetty Eclipse adapter is no longer maintained. There is a Tomcat plugin for Gradle, but I haven’t yet tried it.

Converting Selenium IDE to WebDriver

I’m in the middle of converting a bunch of Selenium IDE tests into Java WebDriver tests. Selenium IDE can export its scripts in Java WebDriver format, but there are a bunch of issues that you need to resolve manually – UNLESS you have the following regex-fu.

First, while still in selenese (the IDE native format), make sure all DOM references have an explicit selector. Usually it’s id, but you could use “name”. Selenese will helpfully try both if you leave it out, but WebDriver will assume nothing, so you must be explicit. Most places I use id-ed elements it’s to store something, so I run this regex over my whole test script directory. There are bunch of tabs and linefeeds in there, so the best bet is to copy and paste a representative bit of code out of the selenese file into your find/replace script:





Next, we need to deal with the fact that IDE will do variable interpolation in strings, and WebDriver will not. The exporter doesn’t deal with this at all, so we need to turn things like “Hello ${username},” into “Hello” + username + “,”. This regex over the exported WebDriver scripts will do the trick.




" + \1 + "

Finally, the exporter will throw a huge number of try/catch blocks into the output so you can accumulate error messages if you want. If, like me, you just want the script to fail when an assertion fails, do a couple more passes over your WebDriver code to replace both of these expressions with empty strings:

\t\ttry \{\R
\t\t\} catch \(Error e\) \{\R\t\t  verificationErrors\.append\(e\.toString\(\)\);\R\t\t\}\R

Finally, put this function in your WebDriver test class. The IDE exporter will generate code that references it, but you must define it.

	private boolean isElementPresent(By by){
	    return driver.findElements(by).size() != 0;

That does 90% of the work in a few seconds. After that, some test cases run straight away, others I still have to clean up, usually because I have some implicit id reference somewhere. Note: I run these regexes in Eclipse, other regex engines might require some tweaking to the patterns.

Flash updater bloatware spam officially not a bug

I got so sick of removing Google Toolbar from friend’s computers every couple of months, that I raised a bug with Adobe: https://bugbase.adobe.com/index.cfm?event=bug&id=3541523. You can read about it there, but the basic premise is that bundling bloatware with security updates represents a bug in the security process, in that it promotes suboptimal behavior.

Anyway, today I got auto-notified that the bug is “withdrawn”. No reason given, no indication of who did it. So I guess that’s official then – it’s not a bug. Installing third-party add-ons with security updates is now industry best practice.

You know, I was pretty peeved when Apple painted a target on Flash’s back a few years back. Web interoperability went to hell in a handbasket almost overnight, and Apple did it to us all for no reason other than self-interest. However, since then, Adobe has been competing with Oracle to undermine trust in the technologies underlying that interoperability. Apple might have done the right thing even if for the wrong reasons. I can only hope that the winner out of all this is W3C, not the App Store.

Surface RT

A year after joining the smart phone revolution, I’m taking it to the next level with the purchase of my first tablet. Through a not particularly scientific process, I chose a Microsoft Surface RT. Here’s why.

Firstly, I’ll talk about what I’m not expecting from a tablet.

I’m not expecting a mobile workstation. My mobile workstation is a Sandy Bridge laptop with 16GB of RAM and 2TB of storage, running half a dozen virtual machines under VMWare. We’re a long way out from having tablets that can replace that.

I’m not really expecting a mobile comms hub either. I already have all the communications I need on my phone. Because it’s a Windows Phone, I can at least read (if not easily write) Word, Excel and Powerpoint files on it.

I don’t need a mobile entertainment hub. Tiny movies aren’t my thing (yes, I still class tablet screens as tiny for movie purposes) and my phone, again, handles all my music and podcasts. As for games – seriously, the games on PC are so much better.

Finally, I don’t need a gazillion apps. I’ve spend the last few years watching friends and colleagues lovingly show off their latest app downloads on their i-devices, and frankly I haven’t seen anything that’s as exciting as email+web (and let’s face it, for anyone who has a life email+web are only exciting up to a point).

So what do I want?

I want to be able to type on the plane. I want to be able to read on the plane, things that are too bulky to carry or too dull to waste paper on (like technical journals). I want Word, Powerpoint and Excel, because those are important for my job. I want a quick web browser for when my PC is off and it’s not worth booting it up. Oh, and I want battery life. Anything less than a full day isn’t going to cut it (so hyperbooks and Surface Pro need not apply).

So, to the Surface RT. I by no means did an exhaustive comparison, but here’s what I thought I was getting in the surface. Solid hardware design and implementation, an excellent keyboard, good battery life. Excellent UI – better IMHO than anything Apple has done in years. The right mix of software. Good price.

So far the Surface ticks all these boxes.

As for the FUD about desktop mode – frankly, I love desktop mode. 90% of the time I’m in Metro (or whatever it’s called now) mode, like any other tablet but better. For Office apps I’m in keyboard-centric (but still touch-friendly) mode. Plus I have access to a command prompt, I can map network drives, use standard RDP – all stuff that makes my geek self feel right at home and yet doesn’t impinge at all on the tablet UI. And no, I don’t care that only Microsoft can write for desktop mode – see comment about mobile workstations above.

Windows Phone first impressions

As per my previous post, I went and snapped up a Nokia Lumia 800 on a postpaid deal. Money doesn’t have its usual meaning in phone plans (does anyone actually pay $500 for $500 worth of calls? in which case, why say those calls are worth $500?) but as far as I can work out the phone ends up costing me about $120 over a SIM-only plan. So, with Jake’s warning ringing in my ears – Jake, tune in in a year’s time to hear me admit you were right – I’ve finally joined the smart-phone revolution with a soon-to-be-superseded version of the least popular platform. Crazy? Read on to find out…

First, I’ll revisit my criteria, in order of importance:

  • Battery life of at least one full day, preferably two
  • Sync to multiple accounts
  • UI that doesn’t get in my way

Battery life
Tick. With light use I get over two days. With heavy mobile data and GPS use I went all of a long day and still had something in the tank.
A word of warning: as shipped the battery life is miserable, but with the automatically applied firmware update it miraculously almost triples. So although you can happily use this phone without ever syncing to a PC – don’t, because you need the PC for the update.

Sync to multiple accounts
Big tick. I like to keep personal email/calendar/tasks separate to work. It took me about five minutes to set this phone up to sync with my work Exchange account, my personal GMail account, Facebook and Windows Live. I get a unified contact list, calendar and task list, colour coded to the source. I can show or hide individual data sources so I can turn off my work calendar when I’m on holidays.

Contact data is merged but still linked back to the original source on a field-by-field level. For example, I’ll get a person’s Facebook photo and status, their phone number from Exchange and their email address from Google all seamlessly displayed on the one contact “card”. When editing a record which is linked to more than one source, I can choose which source I’m editing and only those fields appear on the edit form. When adding, I can choose which service I’m adding the record to, and I can link records from different services to merge them into the one contact.

None of this is flashy or obtrusive – the fact that I’m dealing with multiple data sources is there when I need to know about it, and invisible when I don’t. In short, it works (to my mind) exactly how it should.

As an example of how powerful this is: as part of setting up this phone I had to choose a back-end service for my calendar and tasks. I’ve been using a Palm Zire, so this stuff wasn’t previously in the cloud. The Google task list sync requires some setup on the Google site as far as I know, but I only spent 30 seconds thinking about it because a solution that worked immediately was using the Windows Live task list. So now I’m using Google for mail and calendar and Windows Live for tasks. It turns out I’m also using Windows Live SkyDrive for document storage. But on the phone, all of that is completely transparent – the backend cloud service has become a commodity.

There are a couple of other nice side effects of all this. Firstly, the UI is consistent across all services – it’s the same mail app, task app etc. no matter what back-end you’re dealing with. In fact I don’t even know what the online Windows Live task UI looks like. Secondly, the cloud just stays out of the way. I don’t consider myself a Windows Live user at all – I’m a Google guy for cloud and Facebook for social – so if Windows Live’s aspirations to compete with Google+ and Facebook were even slightly obtrusive that would be a deal breaker. But no, Live just acts as a data store without a hint of an upsell or unwanted notification stream.

Phew, sync is obviously a biggie.

User interface

I actually don’t have much to say about this, which is exactly how it should be. The UI should just stay out of my way. It does. It should let me find what I need quickly. It does. The live tile thing is kind of nifty. To my eye the start screen is a little easier to use than the rows of icons on the iPhone. The whole thing has the obviousness and slickness that impressed everybody about the iPhone when it was launched, but more pared-down, more obvious, and even slicker.

I actually think PCs should work like this. Microsoft might be onto something with Metro.

Other stuff

There’s lots of stuff in phone reviews that I couldn’t care less about. So here are those things:

Camera – it works. Compared to a phone camera 5 years ago it’s great. Compared to my DSLR it’s garbage.
Apps – yes, there are some. Don’t ask me what. I’m in front of a computer pretty much everyday. The apps there are much better.
GPS – actually, the GPS is pretty cool. The Nokia Drive app poos all over the Bing version, and I’d even pick it over a dedicated TomTom.
Games – gimme a break.
Music player – it works. Honestly, this is something I care about, but it’s pretty hard to get it wrong.
PC software – almost irrelevant with everything cloud-enabled. Zune looks OK. It could hardly be worse than iTunes.
Screen – actually, the screen looks fantastic to me. Ok, so I’m old and my eyes aren’t the best, but to be honest I can’t imagine why anyone would see the WP7 resolution limitation as a problem.

A different take on the Windows Phone 8 announcement

I must be the only person left on the planet who doesn’t care about compute power on my phone.

I’m still using an old Nokia “dumb” flip phone, for two reasons: a) it works and b) the battery lasts 3-5 days. If you think reason a) is a bit trite, let me tell you about the two different acquaintances who managed to delete the phone app from their android devices…
However, for a variety of reasons, I’m thinking maybe it’s time to join the smart phone revolution.

This isn’t a smartphone comparison, but let me just say a couple of things. iOS devices are just sufficiently off-key in a Windows environment, particularly a non-Exchange Outlook environment, to rule them out. The sync setup just gets too screwy too easily. And heaven help you if a “helpful” friend enables iCloud (don’t ask me how I know). On the Android side, two different family members (no, not the ones who deleted their phone apps) have had their handsets just steadily degrade in the most uncanny imitation of Windows 95 bit-rot. Plus, for me, the Android UI just doesn’t quite gel.

So what about Windows Phone? The UI looks good, the sync situation (according to the docs) is at least sane, but best of all, these guys have a sensible power draw. Single core, limited resolution, 3G only – it all adds up to just possibly minimum bearable battery life.

Then, yesterday – the WP8 announcement. Windows phones are about to turn into multi-core, high-res, 4G power hogs. Sure, process improvements will claw back some of the losses, but man, what I wouldn’t give for an extra day of battery life instead. Don’t get me wrong, I like compute power – that’s why I have a giant desktop rig and a fire-breathing Clevo laptop. But on my phone, I want to be able to make phone calls, and preferably for more than just a few hours.

So I’m going to hoof it down to the phone shop and snap up a WP7 Lumia while I still can. And keep the old flip-phone handy just in case.