I should point out to start with - we didn't write the original site - we've just undertaken various bits of maintenance on the code base for the last 2-3 years. The site is built on Propel, PHP5 and a PostgreSQL database.
Despite the much advertised recession, this year, they had so many orders that their back end order management system became unusable - it appeared to exhibit O(N^2) type performance, so once there were more than around 60 orders outstanding it became unusable (page load time would be 2-3 minutes or more).
So first off - we had to fix the backend order processing code - so they could quickly deal with the backlog of orders. This was easily accomplished by introducing paging of the orders (show 30 at once, rather than displaying all) and giving them a "fulfill all" "print all" buttons to improve their user experience and save them from having to perform as much navigation.
Next, is the more interesting part - fixing the performance issues with the public facing code. Firstly, we undertook some crude performance measurements of the site, and found that it averaged out at around 8-10 requests/second - depending on the page being viewed. This is without any caching of data or other improvements on our behalf, but was with use of an PHP op-code cache on the server (we'd introduced this previously in an attempt at improving performance before!).
Next, we inspected the code base and identified three areas where we could make easy improvements - and after introducing each, we repeated our performance testing on a local server. Each time we saw a performance improvement, and the end result is that the live site has a performance of around 130-180 requests/second.
- Firstly we introduced an autoloader, which dramatically reduced the number of files being opened on a single web request, and saw performance improve by more than 100%
- Next, we introduced caching within the Propel peer classes to avoid database hits where possible - this was using the Zend Framework's Zend_Cache_Frontend_Class component. Through appropriate use of tagging, we can ensure that stale data isn't served, and also ensure that cache misses are limited.
- Finally, we introduced Page level caching for customers without orders in their shopping basket - this gave another impressive speed bump, and obviously allows the codebase to avoid loading any of the Propel logic - and totally avoid hitting the database. Due to how their site is constructed, this caching can only be undertaken if the visitor has an empty shopping cart - if they don't, then we fall back to the other improvements.
As a closing note, obviously they could have upgraded the server hardware and/or scaled out in a traditional manner. However, the site tends to be relatively quiet for a large part of the year (a steady stream of customers - not all at once). Throwing hardware at the problem wouldn't have solved the underlying problem, and considering the performance improvements we made, it's easy to assume that they'd need to have more than quadrupled their hosting costs in order to achieve the same result as we had with 2-3 days of development and testing. As it stands now, their site should continue to run on a single CPU dedicated server - the same one they've been using for a number of years - which is far more 'green' than having new additional hardware and hosting costs.
Technologies Used
- Memcached
- Zend Framework (Zend_Cache, Zend_Cache_Page, Zend_Cache_Class)
- __autoload
Pretty pictures
We deployed our updates on Friday 9th of July 2009; you'll note the immediate drop-off in resource usage on that day.
CPU Usage (Weekly)
The hourly spikes are caused by a cron job; ignoring these you can see the load has decreased a lot.
System Load
