GameZone/Performance on multiple servers

CS290F Fall 2006 - UCSB Computer Science - Thorsten von Eicken

Up Up and Away!!

• After hours of hacking with the configuration scripts, httpd, mongrel and mysql settings, we ramped from a single machine doing all tasks to separate web, app and db servers. Soon we had more than one. With the bottleneck on mysql removed, it was a smooth sail upwards. We had a little over 15M of database, ( pretty less by any standard as we had 10000+ records ) so, the db handled it effortlessly, without a lot of CPU and HD intensive activity. Most of the time, it was not even appearing on the top of 'top'. As instructed, we tweaked our ' --rate' to go from 0.5 all the way to 50 in gradual increments, and we were able to see the point. We did see some saturation around 40 to 50 sessions per second. Here are the performance graphs.

Where is that bottleneck ?

Now to go after the bottleneck. Each of these app servers seem to be doing their job pretty well and we can see some very good scaling. However, the real peak of when the website would saturate still eludes us. There could be several explainations for this.

1. Low number of records in DB: This is making life too easy for mysql. It is probably serving from memory most of the time. So, it is not the bottle neck, for now :)
2. Are we throwing in more and more app servers before the current one is not fully loaded? Probably not, since, we see the performance of each of them saturate. The apps seem to be really needed to get the scaling that we seek.
3. How about the Web Server then? Not at all.. He is the one having it the easiest so far! We have never seen the httpd sweat it out, as all he is doing is offloading to the mongrels

Here is what we found when we plotted the reply rate per mongrel server (4 X Number of App Servers as there are 4 mongrel instances running on each app server) at maximum load ( --rate = 50)

As we can see, each mongrel is nearly responsible for the same number of replies. Also, adding new mongrels has had linear scaling so far. Except is the AllinOne case, where the system itself was overloaded (cpu) and hence, the reply rate per mongrel was lower than the other cases. This leads us to believe that the DB is where we need to set our eyes on. We need to increase the size of the DB and monitor it closely find the position where it gets overwhelmed, at which point, we expect to see some degradation is the "reply rate per mongrel". So, our next step is to increase to 100K records and perform the same tests as above.

Scaling to several App servers

Here are the results of scaling. These graphs are the final, most optimized, results and show the scaling after caching of fragments using memcached.

As the graphs indicate, gamezone can support 50 users per second.