A tale of utilization vs performance

I once worked on a SaaS product that’s typical of a lot of B2B and niche B2C products.  It had a userbase in the thousands or tens of thousands, and provided business functions that involved heavyweight, complex workflows.  Not many users, lots of compute.  I would say that this is the norm for line of business applications, outside of the industry rockstars that are in the news all the time.

We didn’t need docker.  Docker’s claim to fame is that it’s lightweight.  Once you deploy a multi-gigabyte JVM app that can keep lots of cores busy, it really doesn’t matter if you’ve deployed it onto a VM or a container.  The host overhead is in the noise.   In a containers-on-VMs scenario, no VM is ever going to run more than one instance of the container, so the container layer is just management and skillset overhead.

We didn’t need autoscaling.  This application could keep a server farm busy, but it really only needed the one server farm.  It was easily sharded, if we did hit those limits.  It wasn’t bursty, and it used enough queuing that it could handily absorb the bursts that did happen.

We didn’t need Kubernetes because we didn’t need docker or autoscaling.  Again, unnecessary overhead in skillset and tooling.

We didn’t need these things because we weren’t twitter.  We weren’t even one micro-twitter (if that’s a unit of scale).  We might have got to one million users total, eventually (although I don’t think they ever did). We weren’t ever going to get to one million users a month, let alone a day.

We did have performance problems.  Of course we did.  They were caused by bad SQL and bad algorithms.  We know those were the causes, because we could see the issues staring us in the face.  Every time we did a PoC optimisation exercise, we could easily find improvements of multiple orders of magnitude.  But we never committed to regression testing those and getting them through to production.

Instead, we addressed our performance problems by spending so much on infrastructure that we could have hired two or three more developers.  You have to buy a lot of infrastructure to make your application 1000 times faster.  We settled for a bit less than that.

But it didn’t matter how fast or slow our application was, because we fixated on utilization.  The faster we made our application, the worse our utilization looked.  To a server admin, an average CPU utilization of 20% looks like a healthy server.  To an accountant, it looks like an 80% cost saving waiting to be had.

So we took our slow, unoptimized application, and moved it to docker and Kubernetes.  We didn’t get our extra developers, so we never got to optimize at all.  We took a big hit in training and migration, so productivity dipped.  Reliability got worse for a while, because we made some mistakes in ops.  And our application still had performance problems, because any one request by any one user was still running massively underperforming algorithms and overwhelming the database with unoptimisable queries and deadlocks.

However, our utilization figures were immaculate.  As for the performance issues: when I left, they were talking about putting those bad algorithms into lambdas.

 

Eclipse WTP + Gradle

This post is a collection of handy guides for setting up Eclipse and Gradle for servlet development.

Why do this? Eclipse is a nice development environment, and Eclipse WTP with its servlet container integration provides a slick code/compile/deploy/debug cycle. But Eclipse is not a build system. In particular, if you want transitive dependency management, you’ll need something like Gradle or Maven. The downside is that you then have two completely separate configurations to manage. However, both Gradle and Maven supply utilities to assist in keeping the two in sync.

So, the guides:

http://www.vogella.com/tutorials/EclipseWTP/article.html
This is a great guide for setting up and testing an Eclipse WTP environment with embedded Tomcat

You may then run into this bug in Eclipse:
http://stackoverflow.com/questions/14791843/eclipse-add-tomcat-7-blank-server-name

and, on Ubuntu, this bug:
http://stackoverflow.com/questions/13423593/eclipse-4-2-juno-cannot-create-a-server-using-the-selected-type-in-tomcat-7

Gradle uses a different default project layout to the default Eclipse dynamic web project layout. Both are configurable, but the path of least resistance is to set up your project the Gradle way, like this:
https://weblogs.java.net/blog/manningpubs/archive/2013/03/18/building-java-web-application-gradle

and then import that configuration into eclipse:
http://www.gradle.org/docs/current/userguide/eclipse_plugin.html
and
http://www.gradle.org/docs/current/dsl/org.gradle.plugins.ide.eclipse.model.EclipseWtp.html

At this point, you know enough to create a single project that you can fire up on Tomcat within Eclipse, or Jetty via Gradle. Any container-specific configuration will be a problem, of course, and unfortunately the Jetty Eclipse adapter is no longer maintained. There is a Tomcat plugin for Gradle, but I haven’t yet tried it.

Vagrant and Ansible on Windows hosts

Vagrant runs nicely on Windows. Ansible does not. It can, at a pinch, with Cygwin etc – but that setup doesn’t work with the Vagrant Ansible provisioner.

There is a solution: use Vagrant’s shell provisioner to fire up Ansible on the guest, something like this. It adds a bit of overhead having to install another copy of Ansible on every guest, but this approach does work unchanged on both Linux and Windows hosts. We can still keep the config scripts centralized on the host, and access them via VirtualBox shared folders. Or can we?

TL;DR:

  • All the VirtualBox shared folders options have issues when the host is Windows and the guest is Linux.
  • The safest workaround is to copy your whole config scripts directory to a local directory on the guest, and run Ansible entirely on guest.

If you do keep your config scripts on the host, you’ll eventually run into the dreaded “atomic move” issue. That is, Unix file systems support an atomic move operation, and Windows file systems do not. See the archetypal gedit vs virtualbox pass-the-bug fest for a typical manifestation.

When using vboxfs synced folders, host file system semantics remain in place. That means even though you’re running Ansible on the Linux guest, your Windows host is still going to get you. You can provoke this just by using the Ansible template module, even when only the source file is on the host filesystem. To be defensive, you’d have to assume any command or program on the Linux side could have problems with vboxfs folders until proven otherwise. Oracle appears to be in no hurry to fix this or even admit that it needs fixing. Do you really want to spend your life tripping over all of the problem cases?

Ok, no problem, vagrant also supports rsync and smb for synced folders. Rsync would have to work, because the files end up on the guest where Windows can’t get at us anymore, and SMB must be easy because it’s Windows’ native protocol, right? Actually, not so simple.

Rsync requires an rsync.exe on the path of the host. You can get that by installing Cygwin. That might be acceptable for your own dev machine, but if you are using Vagrant to distribute demo environments, expecting all users to turn their machines into a bizarre Unix/Windows hybrid is probably not realistic. And there’s no installer-free Cygwin anymore (or any other Windows rsync port, for that matter), so the rsync route ends up being just too invasive.

SMB must just work out of the box, though, right? Unless you’re on Windows 8, wrong. Vagrant’s smb support needs Powershell 3.0. Powershell 3.0 needs .Net 4.0. On Windows 7 that’s two more installers, and even then it won’t work unless your version of Vagrant is very recent.

So what’s the solution? Obviously, get a Mac or a dedicated Linux dev box. But what about those of us working at home on a personal project, with only one machine that has to be a Windows machine because all our users run Windows and because dogfooding, and because running Windows as a guest isn’t great for anything graphics intensive? In that case – get *everything* over onto the guest as quickly and simply as possible, and then stay entirely on the guest. Vboxfs works fine for simple copying, or if your config is in a public repo maybe just pull it down straight onto the guest.