sniping.org

• 2007-03-07: Added results for testing with Mongrel and the ActiveRecord session store using MyISAM tables.

As many of you know I’m a rather big fan of Ruby on Rails, the web application development framework. I adore Ruby as a language for all sorts of tasks, and I’ve found I’m able to get a great deal of work done very quickly with Rails and its approach to web application design. I’m not alone in theUpdated se beliefs, too, as the Rails community grew rapidly, and the software has many fans.

One of its “Achilles’ heels,” however, is performance. The Rails framework is very large, Ruby’s performance tends to lag behind that of other common web languages (at least, in Ruby 1.8; 2.0 is going to bring us an entirely new interpreter with significant performance boosts), and the recommendations from the community for significantly boosting performance tend to largely sound similar to “cache it!” It’s all a trade-off, just like everything in technology and life, however — what costs you incur in actual request-per-second performance tend to easily be made up for in improved developer productivity and happiness. The value for not having to bang your head against a wall dealing with JSP/Servlets or PHP and its abundance of code-in-view SQL is significant.

But I was curious just what kind of performance we were talking about. In considering how our team at work manages and deploys Rails apps to our production web environment — right now we’re using Apache and mod_fastcgi because that’s what was available when we built the web hosting environment’s Rails bits — I’ve been looking at couple of options:

• Apache with mod_fcgid — This seems to be better supported than mod_fastcgi
• Mongrel — The pure-Ruby web server that’s all the rage in the Rails community

With Mongrel, we’d certainly be fronting it with Apache and either using mod_proxy and farming out the requests through the Pen TCP load-balancer, or using mod_rewrite and a fancy randomized map and skipping the Pen middle-man. So I ran some benchmarks.

Note: These benchmarks are unscientific. I hate publishing benchmarks to the web because I absolutely hate the way people tear them apart. I’m not trying to conclusively prove anything here, so please don’t take me to the mat because you disagree with my findings and want to show off your “Pedant of the Year” award; these are the tests I ran and I’m telling you the results I got. If, however, you find flaw with the way I ran the tests that might change the standings significantly, or if you have other ideas that might be of use to me in my evaluations for our environment, then by all means comment away.

The Applications

I wrote two “apps” for this. One just spit out the word “Hello.” using render :text and the latter spit out the current date using Time.now.to_s. Quick, easy, and theoretically should be able to scale as high as the server on which they’re running can go. I call them “Hello” and “DateTime,” respectively, in the test results below. I also ran tests that produced roughly, though not identically, the same output using PHP and ASP.NET. The only real difference between the versions is that the Rails and PHP versions produced text/plain output and the ASP.NET version produced full HTML.

The Environment

I tested with ApacheBench 2.0.40 running on a Fedora Core 5 (PPC) box on the same physical subnet. The command line was always:

ab -n 1000 -c 25 $TARGET_URL

The tests were run against one of two machines: a Linux box or a Windows box (only for the IIS/ASP.NET test). The specs:

Linux Box

• Dual 2.4GHz Intel Xeon CPUs with 512KB L2 cache each and Hyperthreading enabled
• 2048MB ECC DDR-SDRAM
• Gentoo Linux
• Apache 2.0.58
• Mongrel 1.0.1
• Pen 0.12.1
• mod_fcgid 1.10
• PHP (mod_php) 5.1.6

Windows Box

• Dual 2.8GHz Intel Xeon CPUs with 512KB L2 cache each and Hyperthreading enabled
• 4096MB ECC DDR-SDRAM
• Windows Server 2003 Standard Edition
• Internet Information Services 6.0
• ASP.NET 2.0.50727

The Results

In the interest of brevity, I’m only including the requests per second results from ApacheBench here. That, and I didn’t keep notes on all the other stats that ab gives me. If there’s demand I’d be happy to re-run the tests and give full dumps of the result output.

Hello

Setup	Run 1	Run 2	Run 3	Run 4	Run 5
Apache + mod_fcgid	39.91	95.56	125.39	175.24	173.69
Apache + 5 Mongrels (mod_rewrite w/random map)	123.85	128.17	127.77	128.30	125.40
IIS + ASP.NET	1746.30	1849.70	1920.48	1901.87	1938.77
Apache + mod_php	971.30	1838.64	1919.39	1955.21	1915.31

DateTime

Setup	Run 1	Run 2	Run 3	Run 4	Run 5
Apache + mod_fcgid	37.81	176.79	177.08	173.60	180.98
Apache + 5 Mongrels (mod_rewrite w/random map)	125.27	128.43	125.59	125.65	124.93
IIS + ASP.NET	1208.21	1686.03	1810.25	1781.81	1697.07
Apache + mod_php	1610.16	1594.38	1653.21	1513.73	1663.34

Hello + DateTime

Interested in doing some further delving into Rails tweaking, I combined the Hello and DateTime tests into one action in the controller that produced render :text => "Hello. The current date is: #{Time.now.to_s}" so I could more easily test things in one run. Yes, I know it’s not really the same as having both tests run. I don’t care. These results aren’t necessarily comparable to the ones from the individual runs. Run your own benchmarks if you don’t like mine.

Batch #	Setup	Run 1	Run 2	Run 3	Run 4	Run 5
1	Apache + mod_fcgid	40.76	95.76	142.96	181.50	176.36
2	Apache + Pen + 5 Mongrels	105.96	120.05	123.26	118.93	118.37
3	Apache + Pen + 10 Mongrels	119.63	118.97	121.77	127.45	114.86
4	As #2 w/ActiveRecord session store	45.37	43.77	42.54	31.30	31.45
5	As #2 w/AR session store (MyISAM tables)	132.25	130.23	131.46	131.41	128.78
6	As #2 w/memory session store	176.80	183.57	184.54	178.50	183.13
7	As #2 w/memcache session store	122.37	122.66	121.64	125.01	120.98
8	As #1 w/memory session store	37.31	193.54	176.41	234.15	182.06
9	As #1 w/memcache session store	27.96	182.69	166.86	113.83	183.98

What Did You Learn?

The conclusions that I’m drawing from this “research” are probably pretty obvious: ASP.NET and PHP outperform Rails by a factor of at least ten in the first two tests. That’s insane. I understand that you can probably improve performance significantly by caching, caching, and more caching, but… Jesus. Ten times faster? I also know that PHP really isn’t a fair comparison since it’s a language and not a framework, but ASP.NET certainly falls into the framework camp in my opinion and it still whomps Rails’ butt.

The second set of tests — with the combined Hello and DateTime action — is a bit more interesting. ActiveRecord as a session store flat-out stinks unless you’re using the MyISAM engine, which doesn’t support transactions, for your session store table. This is fine, though, since you don’t need transactions for that sort of thing. I could probably also tweak memcached to improve its performance, too. I was pretty impressed by the memory session store’s performance, and when combined with the faster — though arguably less manageable in a large-scale, multi-app and multi-user environment like our web hosting environment — mod_fcgid, I was able to push the requests per second above 200. Little victories, I suppose.

Still, I suspect that managing hordes of Mongrels (that’s just fun to say) with mongrel_cluster will be a much nicer environment, especially when we need to run each app as its own service account for security purposes. I think we could find a good way to allocate one Pen per application, farmed out to n Mongrels, and communicating with Apache via mod_proxy.

Final Thoughts

Rails rocks. I still love it to death, and I think it’s a fantastic framework built on a fantastic language that brings a lot of new people into the web application fold. I will, however, be taking another look at ASP.NET 2.0. I’m thinking it might be fun to see how well that fits together with db4o, the object database for .NET and Java for producing an app I’m currently working on. I’m not as afraid of Windows as a server platform as I used to be.

Anyway, there are my benchmark numbers and my short thoughts on them. So go ahead, Internet. I’ve put potentially flawed benchmarks up on the web. Flame on.