Is Scala the New Ruby?

You’ve probably always have associated Ruby on Rails with Twitter, but Scala has recently been getting the attention after Twitter’s back-end code are slowly being migrated from Ruby on Rails to Scala:

“For Payne, who oversees Twitter’s API development, your ticket to the web promised land is Scala, the next-generation language developed by Martin Odersky at Switzerland’s École Polytechnique Fédérale de Lausanne.

Sometime in 2007, Twitter came to the realization that its Ruby-based message queuing system couldn’t keep up with all that Web2.0rhea. “It grew with us for a number of months, and then, at a certain point, it just hit a wall,” Payne said. “And that wall was not so much it code but in limitations in Ruby virtual machine. There’s a lot of things that Ruby is great at, but long running processes? Particularly memory intensive ones? Not so much.”

By mid-2008, one of these Ruby message queues completely crashed and developers needed two and a half hours to shove the dropped Tweets back through the system. When your game is micro-blogging, that’s a lifetime. Then, in his spare time, one developer ported the code to Scala. According to Payne, the Scala queue could process the same message backlog in 20 seconds.”

If you’re a Ruby lover, you’d be asking the same question in your head – why would you want to use Scala (especially for Twitter-like applications)?:

“Bill Venners: I’m curious, and the Ruby folks will want it spelled out: Can you elaborate on what you felt the Ruby language lacked in the area of reliable, high performance code?

Steve Jenson:One of the things that I’ve found throughout my career is the need to have long-lived processes. And Ruby, like many scripting languages, has trouble being an environment for long lived processes. But the JVM is very good at that, because it’s been optimized for that over the last ten years. So Scala provides a basis for writing long-lived servers, and that’s primarily what we use it for at Twitter right now. Another thing we really like about Scala is static typing that’s not painful. Sometimes it would be really nice in Ruby to say things like, here’s an optional type annotation. This is the type we really expect to see here. And we find that really useful in Scala, to be able to specify the type information.

Robey Pointer: Also, Ruby doesn’t really have good thread support yet. It’s getting better, but when we were writing these servers, green threads were the only thing available. Green threads don’t use the actual operating system’s kernel threads. They sort of emulate threads by periodically stopping what they are doing and checking whether another “thread” wants to run. So Ruby is emulating threads within a single core or a processor. We wanted to run on multi-core servers that don’t have an infinite amount of memory. And if you don’t have good threading support, you really need multiple processes. And because Ruby’s garbage collector is not quite as good as Java’s, each process uses up a lot of memory. We can’t really run very many Ruby daemon processes on a single machine without consuming large amounts of memory. Whereas with running things on the JVM we can run many threads in the same heap, and let that one process take all the machine’s memory for its playground.”

Curious now? If you want to learn more about Scala try this site and of course play around with Lift, the web framework for Scala. Here’s a great .pdf file by Debasish Gosh and Steve Vinoski on Scala and Lift

Oh yes, there will still be those who will defend Ruby for Twitter:

“I would be curious to hear what arguments, if any, Twitter had against JRuby, or other message queues. The public arguments I’ve seen, and the decision making process I’m inferring from it, seem like an incredibly inept one. This belies something I’ve heard about those who deal with the Twitter people: their scaling problems come not so much from Ruby but from bad design decisions.

Overall, I think they’re missing the point of Ruby. Ruby shines as an automation and “glue code” language, providing you with a swiss army knife that lets you easily integrate many software components writen in whatever language you want into a single, cohesive application. Message queues are commoditized in the software world, and clients exist for virtually all of them in the Ruby world. AMQP, XMPP, Stomp, you name it and chances are Ruby speaks it. The EngineYard folks are doing wonderful things with Ruby and XMPP/AMQP in their Vertebra and Nanite projects. The Twitter folks are… off reinventing message queues in Ruby, then blaming Ruby when their implementation turns out to be slow.”

Posted by Paul “The Pageman” Pajo