At some point, it was mentioned on the
official boards that if you wanted to see that we were serious, you should check out the website. As it turns out, we are serious, and so is the website, but that probably isn't apparent at first glance. So I'm going to talk about what's going on behind the scenes and the decisions that led to the current webserver setup.
Goals:
- Scale very well
- Handle burst traffic without crawling
- Reliable
- Automatic optimization
- Provide the power to do whatever we feel like (hosting a full item database, etc.)
- Provide high bandwidth for members to collaborate on projects with large chunks of data like videos
The first thing I did was to imagine what the server would look like if it was the monster it would need to be if everyone playing
SW:TOR decided they wanted to visit the website a hundred times per day. This is intentional overkill, but it allows me to properly scale back to a basic configuration that could very easily scale up to that size if it is ever needed. The monster needs a real
content delivery network, a massive and redundant
relational database, a large cluster of webservers,
squids, load balancers, and a bunch of ancillary servers to handle backups, media files, compression, image scaling, mail, etc. I also had to take into consideration what technologies I consider mandatory for the site, which really boils down to just
PHP.
Distilling the monster down to something that can be deployed for a reasonable cost led to the following decisions. There would be three domains. Two of the domains would handle static content and could evolve into the content delivery network. The remaining domain would handle the heavy lifting -- the php. There would be a database server and a file server. The file server would be accessible from the outside for uploading and downloading big files without going through the website. That made a minimum of five logical servers, and the next thing that had to be considered was what technologies I wanted to use.
There were three choices for the primary webserver:
Apache HTTP Server,
nginx (pron. Engine-X), and
lighttpd (pron. Lighty). I was never really happy with Lighty and the whole 'memory issue', so that left Apache and nginx. I don't like using Fast-CGI for PHP (which is another discussion altogether), so that left Apache for the heavy lifting. That's not a bad thing as Apache is highly configurable and robust with a long history. Choosing Apache also gives access to mod_<whatever> for any issues that might crop up. The problem with Apache is a scaling issue--it is a resource hog when it is trying to handle multiple requests and during bursts of traffic it can get mired down quickly. A reverse proxy goes a long way to lightening the load, so that was the next decision to make. Adding in the reverse proxy increases the number of logical servers to six.
The reverse proxies I considered were Squid,
Varnish and nginx. I benchmarked all three in a test environment, but that was really for future reference--I already knew I was going to choose nginx for the basic config because the primary cache was going to be disk based and not memory based for cost reasons. Varnish and/or Squid have their place in the monster, but for the baby, nginx was an easy choice. The benchmarks used (ApacheBench and Load Impact) showed that nginx outperformed Squid and Varnish in the basic configuration... Which was a bit of a surprise, but certainly a pleasant one (even if the benchmarks are completely synthetic).
The choice of nginx for the reverse proxy also made it the server of choice for the domains that were going to serve static content. lighttpd would also perform well for this task, but limiting the technologies in use keeps maintenance woes to a minimum.
The database server?
MySQL. An easy choice for various reasons, but especially in terms of scalability and what I personally have the most experience with. It can also be easily switched over to
Amazon RDS which can meet the needs of the monster in a Multi-AZ deployment. And speaking of Amazon...
The choice for the file server is
Amazon S3. This decision is all about cost, scalability and accessibility. An S3 bucket can be mounted as a local file system on almost any platform allowing ease of use for members to collaborate on videos. More significantly, it can be mounted on the webservers and used with
Amazon CloudFront if I want to eventually use it as part of a CDN. It certainly isn't fast when used as a local file system, but it is fast enough for the intended usage, especially if it is fronted by nginx servers caching the content.
Next: Deciding what handles what, configuration and general optimization.