Why does it matter? Well, although it’s tempting to think everyone must have pretty quick >4mbit ADSL, it’s not always the case -

1. Mobile users may be on a GPRS (2G) or 3G connection – which is normally far less than 1Mbit/s (3g minimum should be 200Kbit). At this speed our average page would take about 40 seconds to load.
2. Some people (often in rural areas) will be limited to around around 512Kb or 1Mbit – due to the long distances between themselves and the nearest telephone exchange (so around about 10 seconds to load).
3. If you’re lucky enough to have around a 10Mbit ADSL connection, then the download would take around about 1 second.

We seem to be seeing mobile users accounting for around about 10-20% of the traffic to websites at the moment – and this figure is likely to grow as tablet computing becomes more popular. Having a mobile theme on your website is likely to help with this – so resources are loaded via AJAX (see e.g jQueryMobile)

From a developers point of view, there are a number of things we can do to help reduce the ‘weight’ of a web page – for example :

1. Enable compression on textual data being sent back by the web server using deflate or gzip (trading off some CPU time against network bandwidth) (Example below)
2. Resize images so they are not being resized in a stylesheet / by the browser – when using PHP, something like the phpimageresize plugin could be used (Example below)
3. Ensure static assets have appropriate / correct expiry times/headers to encourage client side caching (or caching by upstream proxy servers) (see mod_expires) (Example below)
4. Try and use ‘common’ URLs for JS libraries (e.g. Google’s API hosting) – as there’s a reasonable chance the user will already have the script cached in their browser before visiting your site.
5. Compact or minimise stylesheets and javascript resources – removing unnecessary spaces and comments (see e.g. the YUI compressor )
6. Delay loading/fetching of Javascript (see e.g. the contentLoader script from jsclasses.org) until after the initial page has loaded.
7. Merge multiple resources together (the Google mod_pagespeed Apache plugin can do this) – so rather than your browser making multiple requests to the server for multiple javascript files, it sees only one.
8. Appropriate use of AJAX to load content (saving the user from having to download unchanged HTML between ‘pages’).

Server side data compression

If you’re using Apache, enabling the ‘deflate’ module and then having something like the following in a .htaccess file should work well -

AddOutputFilterByType DEFLATE text/css application/x-javascript text/x-component text/html

PHP can also perform the compression for you, but handling the compression through the web server will give better results (more stuff will be compressed).

Image Resizing

If a graphic designer has uploaded an image to the website, and it then gets resized through CSS, your browser is technically downloading more data than it needs to. Instances of this can be easily discovered using the Google PageSpeed tool.

Dynamic resizing of images can be done using the phpImageResize tool (and I’m sure there are many other alternatives) as follows – on the assumption we want to only show a 20px x 20px image:

<img src="<?php echo resize('images/whatever.jpg', array('h' => 20, 'w' => 20, 'scale' => true)); ?>">


For one customer we found correctly resizing the images reduced page ‘weight’ from around about 5mb originally to 1mb (it is a very image heavy news site/blog). Such a drastic reduction will make the site feel ‘snappier’ and more responsive to all users, save on server resources (bandwidth) and allow the server to handle more traffic at once. This should lead to more page impressions and therefore greater advertising revenue.

Expiry Times

Adding something like the following to a ‘.htaccess’ file should work well, assuming Apache as your web server:

ExpiresByType text/css A7200
ExpiresByType application/x-javascript A7200

Which tells the browser to cache the javascript and CSS files for 2 hours after access. Often the expiry value can be far higher – often days, or even years.

It’s also normally useful to turn off eTag support at this point (“FileEtag None” in .htaccess) – to try and stop browsers trying to validate whether their cache is up to date on each request.

Common URLs for JS Libraries

For example, rather than hosting jQuery from your own server (which is probably identical to jQuery on many other servers) you could use e.g.

<script src=”https://ajax.googleapis.com/ajax/libs/jquery/1.7.1/jquery.min.js”></script>

Which is both minified and there’s a reasonable chance the end user may already have it cached due to other websites using it. See the Google code site for more information.

The circuit breaker design pattern is a fairly simple, and handy approach to dealing with remote services which may be offline. To explain the pattern, here’s a semi-true story -

So, imagine the front page of a website includes some output from talking to twitter. However, one day twitter is offline – and all visitors to the site start to experience a 10-15 second delay in page load time. You investigate the problem and discover it’s due to Twitter being offline – in that all requests to it are timing out. What would be nice at this point in time is if your code could have known in advance that Twitter was offline and skipped making a request on it – therefore saving the end users from experiencing a page load delay.

In order to implement the above, we need some sort of shared state between requests – within a PHP context, we could use something like memcache or APC (both are fast and pretty simple to use).

So, before your code checks a remote service, it checks the circuit breaker’s status. If the circuit breaker says “ok” then your code talks to the remote service and notifies the circuit breaker of the outcome (success or failure). If more than a set number of failures occur within a set time period then the circuit breaker changes state and your code can therefore avoid experiencing whatever timeouts may have otherwise occurred.

Pseudo code to illustrate the above could look like :

<?php
$cb = new CircuitBreaker('talking_to_twitter');
if($cb->isOk()) {
    $url = 'http://search.twitter.com/search.json?q=cake';
    $json = @file_get_contents($url);
    if(false === $json) {
        // immediate problem with twitter -
        // tell the circuit breaker.
        $cb->fail();
    }
    else {
        // let's try and decode the data - check it's valid.
        $data = @json_decode($json);
        if(false === $data || !isset($data['completed_in'])) {
            $cb->fail();
        }
        else {
            // everything looks good... tell the circuit breaker
            // and print something trivial out.
            $cb->success();
            echo $data['results'][0]['text'];
        }
   }
}
else {
   // circuit breaker thinks twitter is ill, so don't bother trying
   echo "Sorry, twitter is down   ";
}

Unfortunately there don’t seem to be many PHP CircuitBreakers around – the best examples I can find (outside of what I think is proprietary code belonging to a customer) is this post (which uses a database) and this proposal for one to go into the Zend Framework.

A good circuit breaker would need to :

• Support multiple instances (e.g. one for Twitter, one for talking to Google or whatever)
• Support variable timeouts and thresholds
• Support different storage ‘backends’ (e.g. memcache, APC, MySQL (perhaps) or fileSystem) – at which point it might be best for it to just use Zend Cache.

Your primary role will involve building and testing PHP based web applications.

1. PHP – you should know about design patterns (MVC, Factory, CircuitBreaker and so on), be able to talk to a database and be aware of security considerations. You should have knowledge/experience with PHP Frameworks (e.g. Zend).
2. CSS
3. Javascript – AJAX, jQuery …
4. Unit testing – PHPUnit, Selenium and Jenkins.

If anyone is interested, please contact david at palepurple.co.uk with a CV and salary expectations.

Please no agencies.

In this case saved in a file called /usr/local/bin/load_check,

#!/bin/bash

if [ -z $1 ]; then
    echo "Incorrect usage .... " > /dev/stderr
    exit 1
fi

LOADLIMIT=$1

load_avg=$(uptime | awk -F 'load average:' '{print $2}' | cut -d, -f1)
if [[ $load_avg < $LOADLIMIT ]]; then
    exit 0
fi
exit 1

And usage would look like :

/usr/local/bin/load_check 3 && run/whatever/command

It’s now possible to modify non-essential cron jobs (for example /etc/cron.d/munin) so that they do not run if the system is deemed too busy – so changing :

*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/bin/munin-cron; fi

to

*/5 * * * *     munin if [ -x /usr/bin/munin-cron ]; then /usr/local/bin/load_check 4 && /usr/bin/munin-cron; fi

Will result in munin-cron not running if the load is over 4. Rinse and repeat for other cron jobs which aren’t critical.

As a whole, the students were better this year – their CVs and covering letters had fewer obvious mistakes and appeared better prepared. The majority of students were also smartly dressed which gave a good impression.

However, many students undersell themselves – CVs were often missing reference to work they’d done outside their degree scheme – or the extent of the reference was “Perl” or “Python”. Yet, in one example a student had written a Python/MySQL GUI application and others had experimented with JQuery, CSS3 or HTML5.

Most students expressed a deep interest or passion in a specific field – yet they would often lack supporting “evidence” of self directed research. Being able to mention a mailing list / google group / relevant website / conference or hot topic within that field would lend credibility to the claim – employers want employees who are genuinely motivated and interested.

Finally – it was common to see students saying something like “It will help me in my degree to work for you as I’ll learn to do X and Y”. Unsurprisingly an employer is not likely to be interested in what the student will get from the employment period – they are interested in what benefits the student will bring to them and their team.

See also - http://www.palepurple.co.uk/interviewing-students-some-findings

1. The Pomodoro Technique for time keeping
2. Cloud hosting (managed vs unmanaged with Microsoft’s Azure+ and Orchestra.io as examples). This included live demos and a good coverage of the various options available (and why you might pick (for example) EC2 over Orchestra or vice versa).
3. Solr – document indexing (which we are already using, but there was some useful content within the marathon 2 hour talk/workshop)
4. Unit testing as an afterthought – Marco Tabini (phparch.com) spoke about his belief that when writing unit tests, code coverage on it’s own is not enough when it comes testing through unit tests – like phpunit – as it’s also necessary to cover how components interact with each other, and therefore effectively do testing of an entire application and not just isolated components (something we’d already discovered).
5. Doctrine2 – an overview of why you should be using an ORM and not lower level SQL everywhere – followed by a quick overview of why Doctrine2 is better/faster/superior to Doctrine1.
6. PHP on the CLI – a few useful titbits where hidden within this talk covering using PHP for command line scripts (options parsing via the getopt library being the most useful).

Munin compromises two parts – firstly ‘munin-node’ which runs as a daemon, and is responsible for running various plugin (statistics retrieval) scripts and storing the output values.  Typically plugins exist for all common services (MySQL, Apache, memcached, CPU usage etc). Secondly, a cron job processes the stored data to make pretty graphs which are accessible to the end user through a web browser. For example :

Munin graph showing MySQL Queries over time

Munin graph showing MySQL queries and throughput

In the case of the above, it’s easy to see that on around the start of Week 21 something significantly changed the amount of ‘work’ MySQL was doing – in this case, we’d been doing some performance optimisations on WordPress and there was a significant improvement recorded. Without a long term plot of traffic, it would have been difficult for us to quantify the improvement.

In another scenario, we increased the amount of memory used by memcached – expecting to see a reduction in MySQL traffic (as more data ought to be cached in memory). Did it help? Not in this instance.

The original poster provided a questionnaire – however, this didn’t really allow for much in the way of a detailed answer, so here’s our 2 pence. As a caveat: there are obviously other causes…. :

Lack of Awareness

Some developers just aren’t aware of, or don’t care about, security – as a sweeping generalisation they probably won’t count themselves as a professional programmer. For example: someone who was originally a web designer and learns enough PHP to process a form and slowly moves onto bigger things. Such people are likely to be copying and pasting code which has been butchered over time to add in new fields and functionality. Their understanding of the code and language is often minimal (which is possible due to PHP’s low barrier to entry).

Limited Knowledge

More professional/full time developers will be aware of (for example) SQL injections – but, in some cases, only because a user had typed in “O’Reilly” or > in a form and it resulted in an error occurring – and an error report being raised to them. The developer is likely to have solved the problem by littering the code-base with calls to functions like addslashes() or htmlentities() – however, this can result in the data being encoded/escaped multiple times (so you might see “it\\\\\\\\\\\\\\’s” or &ampgt;).

Over-reliance on trust

Other times developers may believe they can trust the end user – phases like the following are likely heard in this scenario :

• “They’re employees – it’s a restricted system” (e.g. accessible only on the internal network)
• “These are non-technical users”
• “You have to be logged in before you can view/edit any data …”

Unfortunately a reliance on trust will probably lead to authentication bypass, cross site scripting and possibly data disclosure – for example, not ensuring a user is logged in before allowing for the deletion/modification of data, or not checking that someone is authorised to view an invoice before showing it.

Poor application design

In other circumstances, a developer will just have a horrible ‘foundation’ to work with – whether they wrote or inherited a code-base with fundamental design issues which make it hard to develop in a secure manner going forward. As an illustration, something as simple as the following PHP code where the original author assumed the caller would always escape/sanitise/validate the inputs before using them -

function add_user($sanitised_login, $sanitised_name, $sanitised_password) {
    // probably missing length and other validity checks here.
    mysql_query("INSERT INTO users (login, name, password)
                 VALUES ('$sanitised_login', '$sanitised_name', '$sanitised_password')");
}

And at some point, the function is called and the inputs will not have been appropriately validated or sanitised – like :

add_user($_POST['x'], .....).

This is clearly a case of bad design – possibly coupled with commercial pressures which stop the developer from going back and refactoring it when the problems become apparent.

Commercial Pressures

It’s rare to find a customer who states security on a requirements specification. Likewise, being given the resources to perform a pro-active audit of a code-base is unlikely to gain approval from many managers when the time spent on an audit could be used to add additional functionality – which may result in a greater number of sales. Most organisations are focussed on short-term results, so refactoring a code-base, or introducing a testing suite which could take weeks or months to perform, is likely to be difficult to introduce – even with well documented long term benefits.

We frequently help sites to become PCI DSS compliant (which generally involves a combination of systems administration and code review/modification) – in this instance there has been a clear commercial pressure to be secure – but certainly the code review/patching part is at the wrong stage of the development cycle.

The less obvious vulnerabilities

There are security vulnerabilities which are harder to spot, such as session fixation, race conditions and symlink attacks.

Some of these are dealt with through following best practice and developer experience (for example: knowing they should not write out to a hard coded file path like /tmp/tmpfile.txt and should use tempnam()/tmpfile() instead).

Others may be through use of a framework (where the framework calls session_regenerate_id() when someone authenticates, avoiding session fixation) or perhaps it passes view content through the HtmlPurifier library before it’s rendered to reduce the chance of Cross Site Scripting. Alternatively, perhaps it uses a database abstraction layer like Doctrine or Propel which makes it effectively immune to SQL injection.

Suffice to say, in our PHP Security training course we cover various design specific items which should help the above.

Everyone makes mistakes

We recently reviewed a PHP app written in the Yii framework (which seems to be quite a good framework). However, the developers had introduced an XSS vulnerability in a custom 404 error page they’d added – an unescaped error message was being output to the page (which contained part of the URL).

The Yii framework obviously contains functionality to protect an end user from such issues – for example using CHtml::encode($error) would have stopped http://whatever/blah/<img src="http://lolwut.com/layout/lolwut.jpg" alt="" /> from working as well.

Which leads into ….

Detection and Risk-reduction

So, although use of a framework will give a project a good foundation and hopefully reduce the scope for vulnerabilities, like everything, it is not a magic bullet and will not necessarily make the software immune to vulnerabilities. A “determined” developer can still introduce vulnerabilities. And obviously people sometimes make mistakes.

We didn’t realise one of our customers had an XSS vulnerability in code we’d written for them until we pointed an automated tool at the search form, and it’s for exactly this reason that we include practicals covering the use of a number of scanning tools within our PHP security training course.

1. The previous website was based on Drupal, but an outdated / now unsupported version, which would at some point probably give us maintenance problems.
2. By moving to WordPress we move to something we have more day-to-day experience with
3. The new theme has HTML5 elements within it (header, footer, article and so on) which should aid navigation and indexing.
4. The website now supports mobile devices (Android, iPhone) (screenshots below)
5. The previous site tried to display blog posts and tweets on the front page, which we thought was probably confusing – so now tweets are always in the right sidebar, and the blog now has its own distinct home.
6. The migration has resulted in us creating a new theme, which we hope is simpler to navigate and less cluttered

The content of the site hasn’t changed all that much – one or two pages have been dropped which weren’t really necessary, and it’s prodded us into updating some of the content (e.g. examples of our work and so on).

We should have the appropriate redirect rules in to stop us creating any 404 pages or harming our SEO positions.

We’ve been hosting a couple of Xerte based sites (for Techdis and Warwick University) for a couple of years now. Being an open source project – we initially added support to the Techdis version so that it automatically cleans itself monthly and so on (therefore creating an ideal sandpit).

In the past we’d encountered problems with XOT which seemed to only affect our hosting environment – and we’d spend a few hours tracking them down, patch the code and then commit our fixes to our local subversion repository and post a patch on the mailing list. A couple of weeks ago we noticed toolkits 1.7 had been released, and asked one customer if they’d like to be upgraded. Not surprisingly, they did.

So we set up a demo site for the 1.7 code base, using a copy of a customer’s live database – hoping it would suffice to illustrate the new code work(s) before upgrading their live version. Unfortunately it didn’t run – the database schema had significantly changed between versions – and there remains no obvious guide/help as to how to upgrade it.

Next up, we tried installing from scratch – but then found the various bugs we’d squashed out before had reappeared (or hadn’t been fixed in upstream).

At this point it became obvious that we needed to push our changes upstream to save ourselves future work.

So, over the last couple of days, we’ve been fairly busy, and thankfully the Xerte project have given us commit access – so hopefully our changes won’t be lost. This is just a short summary of a few changes we’ve made to the Subversion trunk of toolkits:

The existing codebase is vulnerable to SQL injection, and needs PHP’s magic_quotes functionality enabled. For various reasons (beyond the scope of this short blog post) this isn’t ideal.

Toolkit’s often issues database queries like the below :

include $site->php_library_path . "database_library.php";
$mysql_id=database_connect("index.php database connect success",
                                              "index.php database connect fail");
$query_response = mysql_query("SELECT x From y WHERE z = '{$_POST['template_id']}');
if($query_response === false) {
     receive_message($_SESSION['login_ldap'], "admin",
                                "critical", "something is wrong", "something is wrong");
} else {
    $row = mysql_fetch_array($query_response);
    echo $row['blahblah'];
}

Some problems with which are :

1. Insecure – relying on magic_quotes to protect against SQL injection
2. It’s not escaping the output
3. Verbose – it’s easy enough to shrink the above into something more concise (you’ll see effectively the same 8-10 lines pasted throughout the code base with minor changes)
4. Poor error handling – what if we fail to connect to the database? What if the query string is invalid or no records are returned? Will we as developers have any way of finding out what error occurred yesterday? Or will it just silently fail for Miss O’Reilly?
5. We’re obviously tied to always using MySQL

The code base also ran with PHP’s error_reporting effectively turned off – so undefined variable warnings and so on were not being reported

So, some fixes we’ve made are for instance :

1. Add a db_query($sql, $params) function which performs a crude emulation of prepared statements – so providing some protection from SQL Injection. e.g.$rows = db_query("select * from foo where id = ?", array($id));
2. Create convenience method(s) for recording debug messages (_debug($string)), so if something breaks some sort of audit trail exists.
3. Through the config.php file, support a ‘development’ flag, which if enabled will set the error_reporting and _debug() function to work/do stuff
4. Create a db_query_one($sql, $params) function – this returns a single row or null – ideal for use if you are only expecting one row back – for example, retrieval by primary key.

So the above code ends up being shrunk to :

require_once("config.php");
$row = db_query_one("SELECT x From y WHERE z = ?", array($_POST['template_id']));
if($row === null) {
     receive_message($_SESSION['login_id'], "admin",
                                "critical", "something is wrong", "something is wrong");
} else {
    echo $row['x'];
}

config.php always loads the database_library.php file, and starts a PHP session – before these were handled in a slightly haphazard manner – and in reality as it’s not possible for the someone to use the app and not require the functionality from either.

Other fixes/changes made include :

1. Stopping some obvious Javascript errors from occurring with newer browsers – such as Chrome.
2. Fixing some Javascript related bugs – for example, popups being created but having empty messages
3. Merging in 5-6 patches others had contributed to the project from the Google Code bug tracker
4. Addition of some documentation covering the setup of LDAP for the application’s authentication
5. Addition of a simple logging function (_debug($string)) which should help others when installing the application in the future).
6. Re-indentation of some of the code base – so it more closely follows more popular PHP coding styles (e.g. 4 space indentation and so on).
7. Tracking down of why XOT wasn’t correctly creating a new Learning Object – this appears to have been related to a copy function which did a depth first copy (and then returned in the wrong place – so new templates were always left blank if .svn directories were present as the contents of the .svn directory would be copied, but not the “real” files.). We fixed this through refactoring to use a ‘cp -r’ (copy_r) style function (more readable, ignores .svn directories and so on).
8. Simplification of input validation and error reporting in the login script

There is plenty of work remaining to be done – for instance :

1. Removal of all ‘manual’ MySQL queries to be replaced with db_query*(), and therefore remove the chance of SQL injection.
2. Removal of the legacy Javascript, and changing to use something like jQuery which should provide for greater cross browser compatibility
3. Return of data to Javascript (from PHP) to be via the JSON format, rather than inline (and often) unstructured strings.
4. Adding some sort of upgrade capability (either documentation or hopefully an automated script which just ‘does it’

Still, at least a start has been made.