What do you get when you mix a mobile video game with battling monsters, massively multiplayer capabilities (including player-versus-player), real-time combat and chat with beautifully colored artwork?
Answer: BattleCamp by PennyPop.
This mobile game combines some truly excellent artwork and addictive gameplay with some elements that you’ve likely seen in multiple “MMO” (“Massively Multiplayer Online”) video games like World of Warcraft™ or Guild Wars 2™.
In BattleCamp, you start with a small contingent of your own monsters that you use to battle other monsters in a never ending quest to collect as many as you can. As you progress, you have the option of leveling-up your monsters, gaining more power as you go.
However, the most interesting thing about BattleCamp to readers is the technology that runs the game behind the scenes. As fast and responsive as BattleCamp is, it may surprise some to learn that all its server-side actions run in Ruby!
And we’re not talking about some “fancy” Ruby interpreter here either. No custom patches, no special concurrency hacks or tweaks - just good ol’ MRI.
PennyPop operates two inter-dependent application clusters for BattleCamp. The first is a standard Ruby on Rails application that acts as an API and data transaction layer. The second cluster is what PennyPop calls their “Virtual World” - a large collection of virtual machines running their EventMachine-powered application, written in Ruby, to which clients (players’ mobile devices) make a persistent TCP connection.
There are two basic types of actions in the game. First, players can manage inventory, perform in-app purchases of items to gain certain conveniences (though nothing game-breaking), manage their quests, and so on.
Secondly, players can speak with others and of course, perhaps most importantly, battle NPCs (non-player characters) and other players.
The first set of actions can be considered “latency-tolerant” - in other words, one or two seconds isn’t a big deal, which makes them perfectly suited to having the client (mobile application) submit an API request.
However, the second set of actions - “real-time” actions - aren’t at all latency-tolerant. When fighting in the game, the actions you submit to, and receive from the server must feel very fluid and responsive, or the game loses its appeal. This is why PennyPop utilizes a second cluster for EventMachine-based actions.
When a user first logs in to the game, they are initially talking to the API running on Engine Yard. That API then accesses information it has stored in memcached and Amazon’s DynamoDB to figure out which machine and port to send that user to for connecting to one of PennyPop’s “Virtual World” servers. The client then forms a persistent TCP connection to the assigned server and sends events, to which the server responds as needed.
Think of it like this: when you check into a hotel, the front desk tells you which room to go into. PennyPop’s infrastructure is much the same. When users first connect to the game, the client is told by the API which server in their Virtual World cluster to form a persistent connection with. This is how PennyPop is able to build a responsive MMO back-end with Ruby that processes roughly 40,000 RPM according to New Relic.
When actions take place in the game, such as a battle being won or an item being awarded, the Virtual World server on which the player is playing sends an API request to the Rails application stating that the player performed such-and-such an action (e.g. “Player X collected reward Y”). That Rails application then takes care of storing this information inside DynamoDB and sends a response back to the Virtual World application.
An Innovative Data Storage Practice PennyPop’s API doesn’t make use of ActiveRecord or anything quite like it; instead, PennyPop opted to create their own data storage layer to “talk to” DynamoDB. However, DynamoDB has an interesting limitation in the amount of information it can store in a specific key. As PennyPop can frequently exceed that with their data model, they’ve developed an interesting approach to storing values that would normally exceed that size limitation.
When a record is retrieved, as with most other key/value stores, the key is simply referenced. However, the value of that key is a zipped (compressed) payload of base64 encoded data. When the BattleCamp API needs to read or manipulate this data, it retrieves it from DynamoDB (utilizing secondary indexes if needed since the data is stored in such a way that doesn’t lend itself well to map/reduce), decompresses it, and decodes the base64 payload. Now PennyPop has a viable representation of their model (user, character, etc.). At this point they can then manipulate the data (update an attribute, add an item, etc.) and base64 encode it again, compress it again, and store the compressed payload in DynamoDB.
This involves a significant amount of CPU overhead each time data needs to be read or written. This means that PennyPop’s infrastructure costs are likely to be a little higher, as they need more CPU resources and virtual machines to handle load. When asked about this, Charles Ju, Founder of PennyPop, told us that they are focused on ensuring a better experience for their gamers and delivering updates and new features quickly, even if that requires paying for additional virtual machines.
“Engine Yard makes the headaches that come with maintaining infrastructure go away. You have a full team for database management, a fully staffed, 24/7 support team that is second to none, and [they are] constantly improving the technology. It would be insane for me to try to replicate that myself. If someone is building a real business, or embarking on a serious endeavor, they should get all the help they can, especially when it’s affordable. And Engine Yard, to a real business, is very affordable. If you’re using Rails, it’s foolish not to use Engine Yard.” - Charles Ju Founder, PennyPop
Industry: Mobile Gaming
Location: California, USA
Use Case: Rapid application deployment and scaling
Results:
A Ruby backend capable of processing approximately 40,000 requests per minute, and a business model capable of meeting increasing customer demand while maintaining a small, tight team.
