Project4
CS290F Fall 2006 - UCSB Computer Science - Thorsten von Eicken
Contents |
Scaling Hot Spots On a Single EC2 Server
Overview
The goal of this project was to optimize the performance of our application on a single EC2 server. After running httperf benchmark to test the performance of our web site, we were able to get considerable impovements in both throughput and response time. In Project 3, we identified several optimizations that can be made to our database. By eliminating several unnecessary MySQL queries and adding an index to our restaurant table, we were able to drastically improve the performance of our application on a single server. We pushed the envelope even further by implementing caching to improve performance. In the end, all three critical paths through our site saw around 2X improvements in performance.
Database Optimizations
Our database contains data from YahooLocal and FaceBook Web Services. The primary dataset in our production database is 27,105 entries of restaurants and the database is 11 MB.
DB optimization 1
Added "review_count" and "total_rating" fields in the Restaurants table so that whenever a user writes a new review and gives a new rating for a specific restaurant, the "review_count" field gets incremented and the new rating value gets added to the current "total_rating" value of that restaurant. With those two fields, each restaurant can figure out its average rating by simply dividing the "total_rating" value by the "review_count" value. This approach prevents the application from searching the entire "reviews" table to extract reviews that are linked to this restaurant; thus eliminating unnecessary search queries to the database.
Atomic Transaction
The atomic transaction described above is handled by performing a row-level locking on data via. When an object is retrieved, we call "SELECT FOR UPDATE" on the object so that no other processors can access or modify it until the update operation is completed.
DB optimization 2
Added index of cuisines to the database. This is the only column that is worth indexing, as our MySQL analysis showed in project three. This is due to the fact that this is the only SELECT statement where the where clause is exact. (E.g. SELECT * FROM restaurants WHERE (cuisine = 'Italian') ).
DB optimization 3
As the user navigates through our site, his facebook record is retrieved from our database. As our original analysis showed, we make 3-4 queries to get the facebook data out of our database for each critical path. Our original perception was that this can be optimized to make a single query when the user signs in by storing the data in the session. However, since the session itself is stored in the database, we must still make a db query to get the facebook data. One slight advantage is that the session table is much smaller than the facebook table for each user.
Db optimization 4
After running httperf on path 2 of our site we noticed that the performance is drastically reduced with each run of the benchmark. Futher analysis showed that this is due to the fact that all of the reviews were getting posted on the same restaurant page. The reviews table size was 1704 after all of the runs completed. Since we did not implement pagination for the reviews, the restaurant page was loading all of the 1704 entries from the database! This killed our performance. By adding pagination to the reviews, we were able to remove this bottleneck.
RoR Cache Implement
Caching mechanism is implemented to the service so that it caches most of the html pages into its memory space, such as user_info, reviews, and restaurant search results. Since it allows the server to serve the requests without frequently accessing its database, it is expected to bring a great deal of performance improvement. However, whenever a user submits a new review or edit an existing review, it deletes all the cached data that contains any information related to this newly submitted or updated review, thus it always guarantees up-to-date pages to the users.
See the link ETC2 for more detail on the implementation on distributed servers
HTTPERF Load Generator
The data was collected using the following httperf parameters:
httperf --hog --server=domu-12-31-33-00-03-da.usma1.compute.amazonaws.com --wsesslog=100,0,<script file> --session-cookie --print-reply --rate $rate where $rate is monotonically incremented from 1 to 10 by .5.
To run the benchmarks, we used a bash script to drive the httperf benchmarking tool and wrote a perl script to parse the output. This allowed us to collect the data at a fraction of the time.
All benchmarks ran for more that 30 seconds.
Futher analysis of the httperf output showed that our bottleneck was the database and the disk performance. The CPU utilization remained below 50% for all of the runs. However, the disk performance was maxed out at 270 Kb/S. Since we are running on a single machine, network performance was not a factor.
See the link ETC3 for more detail on the HTTPERF script
Performance Results
Explanation of Results
The data above was collected after all of the optimizations were made. We also had some intermediate results, showing that the main gain in peformance came from caching the html pages and adding an index to the database. With every path, we see about 2x performance gain in reponse time and throughput. The peak CPU utilization for single server was around 13%(user+system) and the peak disk throughput was 214 KB/s. The CPU utilization is misleading since we had the httperf benchmark running on the same EC2 instance. Top output shows that httperf can take up to 30-40% of CPU alone, making a total of 50% CPU utilization for the load generator and the application. Writing a simple microbenchmark to test the maximum allowable CPU utilization on one EC2 instance, we saw that the CPU maxes out at about 45%. The is a clear indicator that the CPU was actually the bottleneck in our set-up. Another microbenchmark was written to test the maximum performance of the disk. We used iostat (iostat -k 1 100) to test the maximum read and writes. The write peak performance was 60Mbits/sec and the read was 50Mbits/sec, which was well above the disk utilization for our runs. Therefore, the data indicates that as we scale our application to more instances for part B, we will be able to improve performance with the additional CPU power.
Challenges
The most challenging part of this project was veryfying the validity of the data. We had to redo all of the runs numerous times due to miss configuration. The Apache, Mongrel and MySQL logs were very useful at helping us explain anomalies in the data. We also manually visited all of the URIs in the httperf configuration files to double check they were correct. Futhermore, the scripts we wrote to drive the httperf benchmark tool and parse its output were extremely useful for gathering all of the data efficiently.
