Scenario

One server (8 cpu cores, 32gb of memory, RAID5 disks) running WordPress (PHP 5.3.x and MySQL 5.1.x). WordPress site(s) would often grind to a halt as the server struggled to respond to requests quickly enough.

Problems

Lack of available I/O capacity, identified by :

1. Numerous processes often in a ‘waiting for I/O’ state (as shown by ‘vmstat’)
2. MySQL becoming backed up with queries and frequently overwhelmed (running out of connections; large number of queries waiting in the process list). Problem especially apparent when a new post was made to the site – as this causes a flushing of the MySQL query cache, any on disk cache’s from WordPress and any related caches (e.g. the cached copy of the front page would need regenerating).
3. Performance graphs from ‘munin’ plot I/O operations along with other virtual memory subsystem events.

Solutions

1. Profile WordPress code to identify expensive queries, add additional caching where possible. Using e.g. the debug-objects plugin to provide timings and help identify the origin of some queries.
2. Caching of pages using e.g. WPSuperCache or W3 Total Cache was already in place but not sufficient
3. Use memcached, rather than disk as an object cache for WordPress
4. Use the BufferedLog directive within Apache to try and bulk up I/O operations
5. Install Varnish as a front end cache, using a malloc memory backend to reduce disk I/O (no need to read file(s) from disk)

Of the above, use of Varnish has had the biggest impact on disk I/O – resulting in a significant decrease. It also led to a significant decrease in the number of Apache processes in use (each of which is relatively memory hungry) – this is possible as a large part of the site is static content (images/stylesheets/javascript) and unchanging.

Varnish configuration

1. Use a malloc backend, 10GiB in size. (See /etc/default/varnish on a Debian based system). Configure to use “-s malloc,10G” as a startup parameter. Before using the ‘malloc’ backend (and using the default file backend) there was no improvement in I/O and performance problems persisted.
2. Ignore cookies for all javascript, images and stylesheets to improve the cache hit rate
3. Ignore query parameters on all images, javascript and stylesheets – again to improve caching
4. Force a future expiry date on static assets to encourage browser side caching of files
5. Ignore all cookies for HTML/PHP content, unless they contain “wordpress_logged_in” – which is a suitable indication that the end user is authenticated.
6. Add a backend health probe – so Varnish can display a suitable error page to end users – rather than showing either a Apache/Wordpress error message or timing out.

Pretty Pictures

Here are some graphs taken from the server, which give some idea of the impact Varnish has had.

Firstly, we have the number of accesses registered by Apache – before and after Varnish was introduced. Because Varnish sits infront of Apache, when it is introduced the workload of Apache drops.

Graph showing the number of Apache Accesses over a week long period

Number of Apache Processes

This next graph shows the hit rate logged by Varnish – i.e. the number of hits per second it deals with. As varnish is made ‘live’ it’s hit-rate increases while Apache’s decreases.

Number of HTTP requests received by Varnish

Before the 07th, Varnish was only accessible for testing, on/after the 7th the site’s DNS entries were changed to route traffic through Varnish. Further configuration changes were made to improve the cache hit rate.

Finally, we have the I/O graph – note how initially varnish doesn’t help with the I/O load on the server (if anything it makes it worse between the 7th and 10th).

On the 10th Varnish was reconfigured to use the malloc backend – at which point the I/O load drops down and appears to remain more consistent.

I’ve also been “volunteered” into presenting a similar talk to the Wolverhampton branch of BCS sometime in the next month or two.

The talk covered a little bit of history (i.e.. what was happening 5-10 years ago) and where we are today. It then shows how various technologies can be used to create an optimised website or mobile application, gave some examples of what we [Pale Purple] have created and issues we’ve encountered.

Thanks to BCS for arranging a great meeting which was well attended.

We were there on Friday – and I saw the talks on :

• “The Journey towards Continuous Integration” (which has prompted me to upgrade our internal Jenkins based CI infrastructure and introduce Sonar for long term statistics capture)
• “Security Audits as an Integral part of PHP application development” (the presenter unfortunately wasn’t aware of any tools which could be easily integrated with a build environment like Jenkins, so the only approach seems to still be manual review or penetration testing on deliverables).
• “Profiling PHP Applications” (which was good, but could have been improved if the presenter had a “real” application to demo with which had performance problems).
• “Try { getting people to come to a talk about exceptions}” – which was an interesting look on exception and error reporting within software.

The conference was (as with previous years) well attended and contained a number of useful nuggets of information which made attending worthwhile.

As with previous years, videos of all the talks should be online – so catch-up should be possible.

Why does it matter? Well, although it’s tempting to think everyone must have pretty quick >4mbit ADSL, it’s not always the case -

1. Mobile users may be on a GPRS (2G) or 3G connection – which is normally far less than 1Mbit/s (3g minimum should be 200Kbit). At this speed our average page would take about 40 seconds to load.
2. Some people (often in rural areas) will be limited to around around 512Kb or 1Mbit – due to the long distances between themselves and the nearest telephone exchange (so around about 10 seconds to load).
3. If you’re lucky enough to have around a 10Mbit ADSL connection, then the download would take around about 1 second.

We seem to be seeing mobile users accounting for around about 10-20% of the traffic to websites at the moment – and this figure is likely to grow as tablet computing becomes more popular. Having a mobile theme on your website is likely to help with this – so resources are loaded via AJAX (see e.g jQueryMobile)

From a developers point of view, there are a number of things we can do to help reduce the ‘weight’ of a web page – for example :

1. Enable compression on textual data being sent back by the web server using deflate or gzip (trading off some CPU time against network bandwidth) (Example below)
2. Resize images so they are not being resized in a stylesheet / by the browser – when using PHP, something like the phpimageresize plugin could be used (Example below)
3. Ensure static assets have appropriate / correct expiry times/headers to encourage client side caching (or caching by upstream proxy servers) (see mod_expires) (Example below)
4. Try and use ‘common’ URLs for JS libraries (e.g. Google’s API hosting) – as there’s a reasonable chance the user will already have the script cached in their browser before visiting your site.
5. Compact or minimise stylesheets and javascript resources – removing unnecessary spaces and comments (see e.g. the YUI compressor )
6. Delay loading/fetching of Javascript (see e.g. the contentLoader script from jsclasses.org) until after the initial page has loaded.
7. Merge multiple resources together (the Google mod_pagespeed Apache plugin can do this) – so rather than your browser making multiple requests to the server for multiple javascript files, it sees only one.
8. Appropriate use of AJAX to load content (saving the user from having to download unchanged HTML between ‘pages’).

Server side data compression

If you’re using Apache, enabling the ‘deflate’ module and then having something like the following in a .htaccess file should work well -

AddOutputFilterByType DEFLATE text/css application/x-javascript text/x-component text/html

PHP can also perform the compression for you, but handling the compression through the web server will give better results (more stuff will be compressed).

Image Resizing

If a graphic designer has uploaded an image to the website, and it then gets resized through CSS, your browser is technically downloading more data than it needs to. Instances of this can be easily discovered using the Google PageSpeed tool.

Dynamic resizing of images can be done using the phpImageResize tool (and I’m sure there are many other alternatives) as follows – on the assumption we want to only show a 20px x 20px image:

<img src="<?php echo resize('images/whatever.jpg', array('h' => 20, 'w' => 20, 'scale' => true)); ?>">


For one customer we found correctly resizing the images reduced page ‘weight’ from around about 5mb originally to 1mb (it is a very image heavy news site/blog). Such a drastic reduction will make the site feel ‘snappier’ and more responsive to all users, save on server resources (bandwidth) and allow the server to handle more traffic at once. This should lead to more page impressions and therefore greater advertising revenue.

Expiry Times

Adding something like the following to a ‘.htaccess’ file should work well, assuming Apache as your web server:

ExpiresByType text/css A7200
ExpiresByType application/x-javascript A7200

Which tells the browser to cache the javascript and CSS files for 2 hours after access. Often the expiry value can be far higher – often days, or even years.

It’s also normally useful to turn off eTag support at this point (“FileEtag None” in .htaccess) – to try and stop browsers trying to validate whether their cache is up to date on each request.

Common URLs for JS Libraries

For example, rather than hosting jQuery from your own server (which is probably identical to jQuery on many other servers) you could use e.g.

<script src=”https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js”></script>

Which is both minified and there’s a reasonable chance the end user may already have it cached due to other websites using it. See the Google code site for more information.

The circuit breaker design pattern is a fairly simple, and handy approach to dealing with remote services which may be offline. To explain the pattern, here’s a semi-true story -

So, imagine the front page of a website includes some output from talking to twitter. However, one day twitter is offline – and all visitors to the site start to experience a 10-15 second delay in page load time. You investigate the problem and discover it’s due to Twitter being offline – in that all requests to it are timing out. What would be nice at this point in time is if your code could have known in advance that Twitter was offline and skipped making a request on it – therefore saving the end users from experiencing a page load delay.

In order to implement the above, we need some sort of shared state between requests – within a PHP context, we could use something like memcache or APC (both are fast and pretty simple to use).

So, before your code checks a remote service, it checks the circuit breaker’s status. If the circuit breaker says “ok” then your code talks to the remote service and notifies the circuit breaker of the outcome (success or failure). If more than a set number of failures occur within a set time period then the circuit breaker changes state and your code can therefore avoid experiencing whatever timeouts may have otherwise occurred.

Pseudo code to illustrate the above could look like :

<?php
$cb = new CircuitBreaker('talking_to_twitter');
if($cb->isOk()) {
    $url = 'http://search.twitter.com/search.json?q=cake';
    $json = @file_get_contents($url);
    if(false === $json) {
        // immediate problem with twitter -
        // tell the circuit breaker.
        $cb->fail();
    }
    else {
        // let's try and decode the data - check it's valid.
        $data = @json_decode($json);
        if(false === $data || !isset($data['completed_in'])) {
            $cb->fail();
        }
        else {
            // everything looks good... tell the circuit breaker
            // and print something trivial out.
            $cb->success();
            echo $data['results'][0]['text'];
        }
   }
}
else {
   // circuit breaker thinks twitter is ill, so don't bother trying
   echo "Sorry, twitter is down   ";
}

Unfortunately there don’t seem to be many PHP CircuitBreakers around – the best examples I can find (outside of what I think is proprietary code belonging to a customer) is this post (which uses a database) and this proposal for one to go into the Zend Framework.

A good circuit breaker would need to :

• Support multiple instances (e.g. one for Twitter, one for talking to Google or whatever)
• Support variable timeouts and thresholds
• Support different storage ‘backends’ (e.g. memcache, APC, MySQL (perhaps) or fileSystem) – at which point it might be best for it to just use Zend Cache.

Your primary role will involve building and testing PHP based web applications.

1. PHP – you should know about design patterns (MVC, Factory, CircuitBreaker and so on), be able to talk to a database and be aware of security considerations. You should have knowledge/experience with PHP Frameworks (e.g. Zend).
2. CSS
3. Javascript – AJAX, jQuery …
4. Unit testing – PHPUnit, Selenium and Jenkins.

If anyone is interested, please contact david at palepurple.co.uk with a CV and salary expectations.

Please no agencies.

In this case saved in a file called /usr/local/bin/load_check,

#!/bin/bash

if [ -z $1 ]; then
    echo "Incorrect usage .... " > /dev/stderr
    exit 1
fi

LOADLIMIT=$1

load_avg=$(uptime | awk -F 'load average:' '{print $2}' | cut -d, -f1)
if [[ $load_avg < $LOADLIMIT ]]; then
    exit 0
fi
exit 1

And usage would look like :

/usr/local/bin/load_check 3 && run/whatever/command

It’s now possible to modify non-essential cron jobs (for example /etc/cron.d/munin) so that they do not run if the system is deemed too busy – so changing :

*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi

to

*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/local/bin/load_check 4 && /usr/bin/munin-cron; fi

Will result in munin-cron not running if the load is over 4. Rinse and repeat for other cron jobs which aren’t critical.

As a whole, the students were better this year – their CVs and covering letters had fewer obvious mistakes and appeared better prepared. The majority of students were also smartly dressed which gave a good impression.

However, many students undersell themselves – CVs were often missing reference to work they’d done outside their degree scheme – or the extent of the reference was “Perl” or “Python”. Yet, in one example a student had written a Python/MySQL GUI application and others had experimented with JQuery, CSS3 or HTML5.

Most students expressed a deep interest or passion in a specific field – yet they would often lack supporting “evidence” of self directed research. Being able to mention a mailing list / google group / relevant website / conference or hot topic within that field would lend credibility to the claim – employers want employees who are genuinely motivated and interested.

Finally – it was common to see students saying something like “It will help me in my degree to work for you as I’ll learn to do X and Y”. Unsurprisingly an employer is not likely to be interested in what the student will get from the employment period – they are interested in what benefits the student will bring to them and their team.

See also - http://www.palepurple.co.uk/interviewing-students-some-findings

1. The Pomodoro Technique for time keeping
2. Cloud hosting (managed vs unmanaged with Microsoft’s Azure+ and Orchestra.io as examples). This included live demos and a good coverage of the various options available (and why you might pick (for example) EC2 over Orchestra or vice versa).
3. Solr – document indexing (which we are already using, but there was some useful content within the marathon 2 hour talk/workshop)
4. Unit testing as an afterthought – Marco Tabini (phparch.com) spoke about his belief that when writing unit tests, code coverage on it’s own is not enough when it comes testing through unit tests – like phpunit – as it’s also necessary to cover how components interact with each other, and therefore effectively do testing of an entire application and not just isolated components (something we’d already discovered).
5. Doctrine2 – an overview of why you should be using an ORM and not lower level SQL everywhere – followed by a quick overview of why Doctrine2 is better/faster/superior to Doctrine1.
6. PHP on the CLI – a few useful titbits where hidden within this talk covering using PHP for command line scripts (options parsing via the getopt library being the most useful).