Performance Benchmarks Of Application-Servers

I think that many of the performance benchmarks of ruby application servers (AS) give the wrong impression. I will show here why.

The performance of the ruby application servers (AS) is usually characterized by requests per second (RPS). But this metric hides some important information.

Consider this comprehensive benchmark for example.

As you can see there are big differences in RPS for different setups. This is correct, but there is a little not so obvious detail one might overlook.

You might think:

Wow, server X is so much faster than server Y. It handles 6 times the RPS. So we should switch from Y to X in order to also be able to handle 6 times the RPS.

The first part is correct, but the second part is wrong.

You calculate RPS by dividing 1 second by the time T one request takes.

RPS = 1000ms / T

This yields the following relationship between T and RPS (The x-axis representing T and the y-axis representing RPS):

As you can see, on the left side the curve quickly goes nearly vertical. But the farther you move to the right, the more similar the curve gets to a horizontal line.

What this means is: If you have a very small T, then small changes in T have a very big impact on the value of RPS, while when you have a big T, small changes to T have close to no effect on the value of RPS.

In the diagram it is visible that at x-values (T) roughly below 10ms the curve goes nearly vertical, while above 10ms it goes nearly horizontal. When looking at the benchmarks of ruby AS, they usually only benchmark a simple “hello world” action, which is really fast (a few ms tops). So the competition between the frameworks and AS takes place below 10ms. If you then calculate the RPS for those average request times you get huge values and huge differences between the AS. But if you look at the differences between the actual request times (T), its only a few milliseconds or even less.

Now, lets say your application code takes 400ms, then these few milliseconds totally don’t matter. Its not like because server X is 10 times as fast as server Y in the benchmarks it is reasonable to switch to server X and expect your requests to be 10 times faster than before. The majority of the time is still spent in your application code which still takes 400ms after switching to the faster server.

This means:

Don’t choose your AS solely based on the numbers presented in those benchmarks. Don’t switch AS for performance reasons if your application code takes more than, say, 30ms. Only think about that switch if your application code takes less than those 30ms, and only think about it if you really need that tiny bit of performance improvement. In the “normal” cases, where your application code takes more than 30ms, you should instead focus on these criterias for chosing your AS or application framework:

• stability (is it production ready? does it crash? is it actively maintained?)
• security
• maintainability (is it easy to deploy/start/stop/monitor?)
• event driven vs multiple threads vs multiple processes
• …

If you have performance problems, they are most likely located in your application code and not caused by the overhead of the AS or the application framework.