Following this report there was a reasonable amount of press interest, it also formed the basis of my Innovation Launchpad presentations which took me to the final stage (and Downing Street) as one of ten. Now the Cabinet Office has published the 2010/2011 edition of the report it gives me the opportunity to revisit my initial findings and see whether or not if, after a year of lobbying, the UK Government's new ICT strategy does in fact get value for money.

This report holds particular significance as in the last year the Government's IT procurement strategy has changed greatly. With new emphasis on engaging with SMEs (25% of all Central Government contracts) and finding cost savings, particularly in ICT procurement, it will be interesting to see if there are any signs of this strategy leading to a reduction in expenditure in web hosting costs for departmental websites.

Selected sites data

In the previous report I took a sample of data from 10 websites. I chose websites that reflected the trend across all 45 sites in the report. Since my first report was published Eatwell.gov.uk has now been taken down. I will be using the same 9 websites in this report excluding Eatwell to continue to allow a comparison year-on-year. Once again in some cases the data for all websites listed in the 2010 / 2011 document has been used.

Site specific cost per request

The stand out results in terms of dramatic change from the above comparison are Businesslink, Communities, NHS and DFT. Businesslink with almost the same traffic levels as last year has an increased hosting expenditure by 20%, resulting in a higher cost per 1,000 page views. A redesign or new build would be expected to explain a rise like this however their was no expenditure last year for either of these services.

Interestingly NHS.uk's hosting expenditure has also risen, this time by 43.6% from last year (£2,610,000 to £3,750,000) but it has seen a 24% fall in annual page requests. In a cloud or utility based hosting environment it would be expected that a fall in visitor numbers would see a fall in hosting expenditure. This unexplained increase in hosting expenditure is a cause for concern, especially considering the fall in page requests to the site.

The stand out success story is dft.gov.uk. There was a redesign last year and more importantly, dft was migrated to an open source application, namely WordPress. The hosting expenditure for 1,000 page views for dft.gov.uk has decreased by 51% (£46 to £22.34) but the annual page views have increased strongly by 140% (17,497,004 to 42,084,781). Generally you would expect hosting infrastructure to rise in line with page views adding costs, so clearly the migration to WordPress has resulted in much lower cost for 1,000 page views.

This could be explained by the technology requirements for hosting WordPress. Being written in PHP and MySQL means the technologies are commonly available, with no license fees, and can be hosted on a free to use and distribute Linux & Apache based system. When comparing this to Java or ASP based websites the hosting costs are considerably lower.

The annual hosting expenditure for dft.gov.uk has remained very similar (£810,000 - £940,304) even though the cost per 1,000 page views has decreased. With a move to WordPress you should expect to see costs reduce, however with an almost constant hosting expenditure annually, it would seem as though DFT are on a fixed hosting contract of some sort. If this is correct, they are not fully capitalising on the potential savings available to them. Yes, WordPress brings cost savings on a usage basis, but it is also very cheap to host compared to other apps. The annual hosting expenditure should reflect this instead of remaining almost constant when compared to last year. Yes DFT are getting value for money on last year by serving more visitors for less, but the total hosting expenditure should reflect this more granularly.

Migrating DFT to WordPress was a move I suggested in August last year! 'Migrating DFT to WordPress'.

Comparing this years and last years sample data above it is clear to see that Businesslink's and NHS's hosting expenditure has risen, despite cost concerns being raised throughout last year, and that DFT.gov.uk's cost per 1,000 visitors has dramatically reduced. There is also some consistency with Communities and HMRC in particular requiring very similar hosting expenditure year on year, (Communities £372,000 / £323,783 & HMRC £1,071,000 / £1,159,738).

I think that overall it is concerning that the majority of the sample sites have seen increased costs proportionally. Although the Government IT strategy is relatively new, clearly there has been no immediate effect if, instead of keeping costs consistent at a minimum, costs have actually risen in the majority of cases.

The greatest concern to me is the Number10.gov.uk expenditure. This site was promising, using an open source framework, however the cost per 1,000 visitors has actually increased relatively significantly (£2 to £6.87 - 243%). Yes, the total hosting & infrastructure expenditure has reduced, however the cost of actual page views, and therefore processing / request time, has risen sharply. Clearly the site is not being hosted on a utility based system allowing Number10 to benefit fully from running on an application such as WordPress. As a flagship next generation site I believe this rise of expenditure should be addressed immediately. Hopefully this year the G-Cloud project, and our involvement, will begin to solve these problems.

Average costs

The following figures are a comparison of the average cost for 1,000 and for a single page view across all the sample sites that received data from each report. Using the average across the sample gives a clear indication whether or not the Government's strategy to reduce IT expenditure is, overall, working to reduce costs.

The outcome is positive with a 15% reduction in the cost per 1,000 page requests across all website data provided in these reports. This is clearly a win for Central Government, particularly with the new IT procurement strategy I mentioned above.

Conclusion

I would like to think we can look forward to another set of positive results based on the analysis next year, especially with Chris Chant and his team's G-Cloud efforts.

I think the attitude towards development and web hosting described in James Stewart's blog on the Government Digital Service site is greatly encouraging, and suggests that the attitude in Central Government towards SIs and relationships with Incumbent Suppliers truly is changing. James writes..

Cloud hosting options are continuing to develop rapidly, and so are the requirements for the GOV.UK platform. As parts of our platform stabilise we may well find that we want a hybrid hosting environment, where some aspects live somewhere like AWS and others are hosted on other cloud platforms or even on traditional dedicated hardware. We’re primarily focused on working out what will provide value now rather than being too preoccupied with potential future scenarios, but we have been careful to follow our own rules and not tie ourselves too tightly to any one hosting platform.

I hope that next year I can write again about an improvement in the results. I would like to see firstly, a greater adoption of open source applications for website redesigns, especially following the success of dft.gov.uk. Secondly, more engagement with SMEs who will often prove their ability to produce savings when dealing with large private entities. Finally, the rapid adoption of the G-Cloud programme, pan-Government, and success stories based on the G-Cloud suppliers service offerings.

SocialBizzle came to us with a problem. Their current hosting was slow, and they were concerned about downtime. Reputation is everything, so downtime is a disaster for a new business! CatN ticked all the boxes as a fast, stable hosting company, and our vCluster product was perfect for their needs. With a simple payment structure which allowed SocialBizzle to scale, and a platform designed specifically for hosting PHP application like WordPress, vCluster was the perfect solution.

"We were looking for hosting that was reliable and affordable but at the same time would cater to us as we grew. CatN delivered each of these objectives and also provided excellent support."

Ali Gallop, Founder - SocialBizzle.com

To help the guys get moving quickly, the CatN engineer team migrated the SocialBizzle.com website to the new vCluster free of charge. We made sure that all the site functionality remained intact, and even helped SocialBizzle update their DNS records. This created a great development environment for them to complete their site before going live.

Just before the launch of SocialBizzle.com, they contacted CatN again with some performance concerns relating to their website. We quoted for the work and put a CatN WordPress expert to work improving the website load time. After a couple of hours work the average page load time went down from 7.5 seconds to just under 2 seconds. Even on the homepage which contains third party hosted HD video content which cannot be cached, the page load time improved from 13 seconds to 4.5 seconds.

Moving forwards with the SocialBizzle team, we are currently discussing a partnership to allow us to offer their Facebook training services at a discounted rate to our CatN clients. We want to create an eco-system of companies who can add real value to our clients, and whose services will help our clients businesses and websites grow.

However there are easier ways to build links, some of which I mentioned at the end of an earlier post (Getting your site indexed), but these links are not very powerful and are only really good for getting your site indexed.

There are many ways you can build links to your site, some of which I have listed below.

The good Ol' fashioned link request emails

This tactic has been around for a long time, is probably one of the most used, and can provide the best return, you can potentially gain a lot of quality links by simply asking Webmasters to link to your site. This tactic works much better with more established sites with some authority rather then new sites.

The only problem with this is that it can be very time consuming and with more web savvy site owners today they are sometimes reluctant to place a link on their site, as this would negatively affect their Google page rank.

A good trick when using this tactic is to provide something in return, for example I check the Webmasters site for crawl, code and SEO errors and inform them on these before asking for a link, I then keep a conversation going with them until eventually I ask for a link.

This works well, but you have to target related websites, if you have a site about Barbie dolls its very unlikely that a Car Hire site is going to link back to you no matter how much you help them. Another way you can use this tactic is by simply having meaningful conversations with other Webmasters, build a relationship and then ask for a link.

Another way you can use this tactic is by simply having meaningful conversations with other Webmasters, build a relationship and then ask for a link.

Sending out 1000 emails with the same email template will not work anymore, this will do more damage than good because if you manage to get 5 links from doing this you will have missed out on 995 websites which will no longer trust you and just see your emails as spam.

Give Webmasters an award!

Webmasters love to be made to feel special, so if for example you have a gardening blog why not do a ‘The Best of Gardening’ list, and name your top five favorite gardening sites, these can even be your competitors!

To everyone you mention in the list you can send them and email saying something like ‘Congratulations you are featured in our ‘The best of list’ here is your badge’ You will then send them a badge which they can place on their site.

A lot of Webmasters will place this badge on their site because it makes them feel special, obviously this badge will have a link to your site embedded.

However if you do plan on using this technique then you should not over use it, do not spam random webmasters and always provide value.

This method really relies on you having at least a mid popular site with traffic, no one is going to put a badge on their site which they got from a PR0, new and low quality site.

Give your customers a discount

If you run a membership site, then you can offer your customers a small discount on future bills if they place a link on their site pointing to yours.

This can be any % and the take up you will have on this will be brilliant, people love saving money especially when they don't have to do much! How easy is it to simply place a link on their site!

However this can be seen as ‘paying for links’ you should take caution when using this method. You will usually find this tactic used when submitting your website to a business directory where they will either offer you a paid listing but if you give them a reciprocal link in return, your listing is free.

A good way to get around the dangers of essentially ‘buying links’ is by advising people to use the ‘nofollow’ tag, even though these links will not help your search engine positions, they can be a great way of getting some cheap advertising, without paying commission on each new member they sign up like an affiliate program. If you do go down this route then it would be better to ask your customers to use a banner instead of just a simple link.

Have some seriously amazing content

This tactic is used a lot in the SEO world, where I do believe that having some brilliant content on your site can gain you a lot of links. You will need to build links in other ways first in order for that content to be seen, otherwise your site will not be in the SERPs in the first place.

This is unless of course you have a large social following then you can just share it and you’ll hopefully gain some links (if your content is good enough!).

To build some great content you need to be creative, research and have perfect grammar/spelling.

Great content doesn’t need to always come in the form of text on a page, recently infographics have been a popular linking strategy which can pull in hundreds of links if the infographic is up to scratch of course.

Lists are also great for link bait, people love reading lists like the ‘top 10 action films’ or ‘the top 100 blogs online’ for example.

Build a WordPress plugin

This tactic can be seriously amazing and can build you thousands of links.

For example our survey and quiz tool plugin has been downloaded 45,890 times and each time someone uses it the plugin places a link to our site (this is a very clear link with no form of deception).

When using this strategy it is very important to NOT hide or cloak the link back to your site, simply put something like ‘Powered by: [Your Link]’ under the widget. Otherwise this could be frowned on and is border lining black hat!

Of course this is not limited to a WordPress plugin you can create a plugin for any CMS, or even create a website script which places a link. For example the people that have developed the Arcade Script place a powered by back link in the footer of anyone using the script, currently they have built around 9k links by using this method.

Article Marketing

Article marketing is a traditional way of building links to your site used mainly by Internet marketers.

There is some controversy over whether this method still works or not after various Google updates including 'Panda' but the fact remains (which has been proven) if you post your articles in the right places, these links do in fact work well.

By ‘The Right Places’ I mean not in places that have been heavily spammed, for example there is an article directory called Go Articles which auto approves articles, this means they do not have guidelines or a human editor, evidently their article directory has a load of spam and is really poor quality.

Saying this I did test out ‘Go Articles’ where the link wasn't very good on their site, my article did get republished by around 4 other sites! Ok these sites were not the best and didn’t give me a huge boost in the SERPs (Search Engine Results Pages) but this is the reason people like article marketing not because the link they get from the article directory but all the other links they easily get from other websites republishing their articles.

Three of the best article directories are:

As well as using article directories, you can use hosted blogs and web 2.0 sites to build links, this is the same principle as article directories, create great content that will get seen by the community and hopefully linked to which will make the link to your site even more powerful. Some of the best hosted blog platforms are:

Some of the best hosted blog platforms are:

Website	Google Page Rank	Alexa
WordPress	9	18
Blogspot	9	7
Squidoo	7	210
Hub Pages	6	297
Weebly	8	366
Yola	7	3,560
Live Journal	8	102
Blogsome	6	4,341

Other link building tactics that don’t work as well but are easy

Website Directories

These I believe are a thing of the past, they used to work well but not so much now. There have been a lot of new web directories pop up but most are low quality and spammy. Of course you can still find a good directory like DMOZ, but these are rare and can take a long time to get approved. The Yahoo directory used to be quite powerful as well until it was discontinued.

Forum links

Forum links are weak, you shouldn’t spend much time if any trying to get them, at best you will receive traffic from the forum but the links will not really help improve your rankings.

However if you are promoting a specific product or service then a targeted forum could be a good place to go. The trick is to try and make yourself an authority on your subject, for example if we were to use forums we would go to a hosting forum and help people out, we would include our link in the signature section and hopefully we would gain new visitors and subscribers.

Blog comments

There is nothing wrong with blog comments, as long as they are done in a non spammy way, they won’t really help you increase your search engine rankings that much but they can boost your PR nicely.

The idea with blog comments is to not be spammy and actually contribute to the forum instead of just commenting for a link, also please don’t use your keyword as your name, this never works and the comment will not be approved.

Reciprocal link exchanges

Reciprocal link exchanges have been around since the beginning of time (internet time), all it involves is swapping links with another webmaster. Yes these did used to work well back in the year 2000, but now they hold little value.

However if you are linking out to another website, then it is worth asking for a link back, it doesn't take long and will do some good. Just don't spend too much time searching for people to do reciprocal link exchanges with.

Link building tactics Google will kill you for!

Unfortunately in the world of SEO there are a lot of bad people who like to cheat and make the Internet a giant ball of spam. But lucky for us (not so much for them) Google (the people that basically rule the Internet) are cracking down on these spammers, I am going to list some of the things you shouldn’t do when link building.

• Do not believe the hype of people selling 1,000+ links for £10, they will not work, they will not do anything for your rankings other then damage them!


• Do not buy links, and if you are determined to buy links do not use a market place, contact Webmasters directly.


• Do not participate in any link schemes, link farms or anything that is automated. It will not work


• Do not buy 100 domain names, build low quality sites and link them together, its a waste of money.


• Do not participate in 3-way linking Google is not stupid.


• Do not use fiverr.com gigs to build links they are only $5 for a reason. You will see people offering 10k link packages or 250+ edu links these are ‘unethical’ links and will most likely harm your site!


• Do not link to bad neighbourhoods (spam sites/adult sites) unless of course you use the 'nofollow' tag.

Link building can be very time consuming but you should never go over to the dark side (spam and going against Google’s guidelines) otherwise known as 'Black Hat SEO'. When using these tactics the results are never permanent and sooner or later you will get caught out and penalised by the search engines. I have done lots of research on this which included a huge number of tests using a lot of dodgy tactics, yes you can see results for a while but not for long.

Avoiding the Google Dance when link building

If you haven’t heard of the Google Dance before then you are lucky, a lot of new sites get caught in this and it can last for a very long time.

The Google Dance should not be confused with the ‘Google Sandbox’ a majority of new sites are affected by the Google Sandbox which basically consists of:

Your site ranking highly for the first few weeks and then it gets dropped from the search engine results completely only to re-appear deep in the results (page 5 or more). It will the fluctuate for a few weeks before it settles into its final position, this is just Google figuring out where your site should be placed in the SERPs.

The Google Dance however is very similar but it can last a lot longer, sometimes a year or more. You will know after the first month whether you have been affected by this or not simply by tracking your rankings. If you see your site on the first page for your targeted keyword for a few days and then it drops way down in the rankings for a week or so and then back up to first page and so on then you have more then likely been affected.

You can track 10 of your keywords free using: SERPFox.

The Google Dance usually happens when you have done one of either of these things:

How can you overcome the Google Dance?

Overcoming the Google Dance can and most probably will take a bit of time (around two months) it would be much better to avoid it in the first place.

However if you have been affected then there are several steps you can take:

1. You should continue to build back links to your site, but no more ‘rubbish’ links, try to gain authority links with some nice PR.

2. You need to check your site for crawl errors you can use Screaming Frog if you do have crawl errors you should fix these ASAP.

3. Check your robots.txt file, and if using WordPress download, install and configure the All in One SEO Pack.

4. Remove all duplicate content from your site, if you have two pages with the same content remove one and use Copyscape to check the internet for any duplicates of your websites content.

If Copyscape does find duplicate content elsewhere on the internet, then you should either remove it off your website or if you have written the content yourself which means you are the copyright owner you should email the webmaster that took your work and ask for its removal. If the website owner does not remove your content send him a cease and desist letter, a template can be found here: Download cease and desist letter template

Unfortunately this is about all you can do to overcome the Google Dance, I would say the most important thing is to build good back links, and removal of duplicate content.

Final words on link building

There are many more ways in which you can link build, you just need to be creative, persuasive and professional.

While you are on your search for some great link building advice you should know that you will never need to buy anything to have an effective link building campaign, all the information you need can be found online and in this post. There are many marketers online that will try and sell you their 'methods' of gaining links, usually these will be of no use and will only repeat things you have already read here.

In our next SEO post we will be talking about how to optimise your meta description and title for the best click through rates and how these affect your search engine rankings.

It seems this little under advertised feature of the WordPress.org plugin directory is really easy to do and makes your page look that much better. I recently added a banner to Simply Poll plugin page and it looks really swish now.

Here's the 3 steps to doing so:

1. You need to create a PNG image with the dimensions 772 pixels wide by 250 pixels heigh. Note: make sure you don't put anything critical towards the bottom left of the image as this is where the name of the plugin appears.

2. Create a new folder within the root of your plugin's subversion directory called `assets` and save the image in that folder with the file name `banner-772x250.png`

3. svn stat the SVN folder and then commit and you'll have yourself a banner on the plugin page.

I'm interested to see if this increases the number of downloads for a plugin as it does make the page standout more so than other plugin pages.

Now we move onto rankings and how to move up in the SERPs (Search engine rankings position). Any good SEO campaign will begin with an On-Page SEO strategy, only when you're confident that this is in good shape should you consider moving onto Off-Page SEO.

Before you start your On-Page SEO the first thing you should know is why you are doing it and how do Google and other search engines use it.

The reason behind On-Page SEO is so that Google knows what your web page is about. If your page is empty or is solely a flash site which Google cannot read then how is Google supposed to know what your site is about? This is why many of the suggestions below include adding keywords to certain areas of your website and in specific ways.

There are many different factors to consider when completing your On-Page SEO. A lot of the time people can be all over the place when completing On-Page SEO which means some things get overlooked. Follow our On-Page SEO guide to have some seriously great On-Page SEO for your site.

Finding Your Keywords

To create a successful SEO campaign you will need a bunch of keywords you want to target.

Keywords can be long like 'how to complete On-Page SEO' these are known as longtail keywords or they can be shorter phrases like 'on-page SEO'. The benefit of using longtail keywords is that usually they will be easier to rank for.

How to put keywords into your posts and pages properly

A lot of people get hooked on just finding one keyword with low competition and high traffic, this is where some SEOs are going wrong. Google will not determine what your site or page is about by just taking the one most used phrase on a page, however if it does they will not reward you as much as you could be rewarded.

When doing keyword research the best thing to do is to grab around 5 keywords for any page, these will consist of one main keyword and 4 other keywords known as LSIs (Latent Semantic Indexing). To do this you can use the Google keyword tool and look at their suggestions. Here is how you would find the keywords:

1. Go to the Google Keyword tool and put in a broad term for your page, in our example we used 'Widgets'.



2. Underneath your words search volumes you will see the section titled 'Keyword Ideas' these are words or phrases that Google relates to the term you put into the search box.



3. You then select one of the terms to be your main keyword, this will usually be the one with low or medium competition and a high number of searches. A high number of searches can be anything from 1,500 onwards.

4. Once you have selected a main keyword you will then browse the list of keyword ideas and choose 4 other keywords which you will use in your page, these should be of low to medium competition but the search volume is not that important.

5. Once you have selected 5 keywords you should then follow these rules:
   
   The main keyword should have a keyword density of around 3% - 4% and no more.
   The other keywords should maintain a density of around 1% - 2%

When doing keyword research it is always best to use 'Exact Match' feature within the Google Keyword Tool. You select this by checking the 'Exact Match Box' located to the left of the Keyword Tool.

Using this method will give your on page SEO a nice boost compared to your competitors. As Google will have more of an understanding of what you page is about. This is more commonly known as using LSI keywords.

What are LSI (Latent Semantic Indexing) Keywords?

LSI keywords help Google and other search engines determine what your website is about. LSI keywords are basically keywords that are related to your main keyword for any given page.

The main reason LSI keywords are important is because some words have different meanings, for example if you are targeting the keyword 'Red Mice' then Google may think it is about Red Mice as in the animal, but if you use other keywords that are related to that keyword then Google will better understand what type of mice you mean.

In the example of 'Red Mice' some of the related keywords are:
Wireless computer mouse
Computer mouse
Optical mouse

If you include these search terms in your page then the search engines will not get confused and will start showing your website for the proper search terms. It is also said to help your rankings for your main keywords.

One thing you should keep in mind is that LSI keywords do not usually contain your main keyword, because this could be seen as keyword stuffing.

Site Architecture

This is a very important factor when completing On-Page SEO asmnot only does it allow you to place keywords into the URL but it also makes your URLs more user friendly. Please see the post 'Site Architecture' for a more in-depth analysis of this.

Page Titles

Page titles do play an important task in the On-Page factors, I have personally seen blog posts rank well solely on their URL and Page title. The page title should contain at least your main keyword and if possible one of your secondary keywords for the target page.

Keyword density

Keyword density is the percentage of how many times your keywords are mentioned in any page compared to the rest of the text.

Your keyword density for your main keyword should be around 3% - 4%, and your LSI keywords should have a density of around 1% - 2%. This is not always possible and before you stuff your page with keywords you should think about the visitors to your site, it will be much better to have a page that reads well then a page which is stuffed with keywords. You should always try not to go over a 4% keyword density otherwise you could get penalised by Google for 'Keyword Stuffing'.

Content is King

Great content will always dominate the search engines, it is always best to have a minimum of 100 words on a page, I would usually aim for at least 500 words.

Meta Tags

Not many websites use the meta keywords tag anymore, in fact it is good practise not to. Google does not pay any attention to this tag and it will not increase your position in the SERPs. All it does do is allow your competition to see what keywords you are targeting, which could cause a lot of problems!

Including keywords in your meta description tag does not help your rankings in the search engines but it does improve your click through rate. It is very good practice to manually fill in your meta description tags where possible. This will not only help you improve your click through rates but also make your websites result in the listings look more appealing. The meta description should be no longer then 156 characters, this is because Google and other search engines will cut off your description at this point.

If you have a blog with a lot of content on then going through and adding meta descriptions to your backlog of content can take a long time. If you are using WordPress then I recommend installing the All In One SEO Pack Plugin. This plugin will allow you to auto generate the meta description tag for all of your posts.

Header Tags

Using your keywords in <h1>,<h2> and <h3> header tags are a good way to improve your On-Page SEO. A general rule is to not use more then one <h1> tag that has your keyword in on any given page.

On-page Site Performance and Technical Issues

Site Speed

One of Googles ranking factors is site speed. Though this is only a minor factor, as Tesco say 'every little helps'.

You can increase your page speed by either condensing some of your code, or if you are using WordPresss then you could Install a Cache Plugin like WP Super Cache which will increase your load speed slightly.

Avoid 404 Pages

Google hates pages that are unreachable, if you have any 404 errors then you should fix these as soon as you can. I have seen a lot of instances where a webmaster just forgot about an internal link which points to a page which no longer exists. You should use a tool like Xenu's Link Sleuth to regularly check your website for 404 errors.

Use 301 redirects not 302 redirects

If it is possible for you to use a 301 permanent redirect, rather then a 302 temporary redirect then you should always use these.

Make use of the Rel Canonical tag

Use the rel canonical tag in any places that you may suffer from duplicate content. This will be on a page like your blog where you have a snippet of you blog post on the home page.

If you are unaware of how to use the Rel Canonical tag then you should visit the Google support page here:
Rel Canonical Tag Explanation.

Use the 'No Follow' tag on outgoing links

If you link to an outside website sometimes it might be better to use the no follow tag to preserve your page rank, this is usually used for sites you do not officially vouch for. If however you feel the site is worth the link then do not add the no follow and share the love.

Conclusion

The most important thing you must remember when completing your On-Page SEO is that your website was built for real life visitors not search engines, so your content, url's and titles must always read well and make sense.

In our next SEO post we will talk about Off-Page SEO, including how to setup social media, increase pagerank and build links to your site.

We have now exceeded our quota of beta users and therefore disabled the ability to create new vCluster packages in the run up to the launch of version 2.0. This allows us to continue collating the feedback generated and focus on implementing the new suggestions and improve the features available in the new version.

Some of the feature requests have made it on to our Roadmap and will be available with the launch of version 2.0, with any suggestions that do not make it into the initial release added to our backlog for possible implementation in the future.

Back in August 2011 we provided a few teasers of the new control panel in its development stages, alongside allowing visitors to our stall at WordCamp 2011 weekend access to the control panel. This provided us with some initial customer feedback, which proved useful in finalising the design and allowing us to focus on features that improve the hosting product itself.

There have now been over 30 significant changes improvements made since version 1.0 and more additions on the roadmap that are under development.

If you have any questions about vCluster or any of our hosting products, please feel free to contact us with your queries and we will be happy to help.

On Friday our NFS server that powers vCluster malfunctioned causing all our web facing servers to lock up, waiting for disk IO. This caused downtime that began at approximately 16:10, Friday afternoon. This downtime went on intermittently throughout the weekend with services not reliably available until Monday lunchtime.

Of course this is unacceptable so let me begin by apologising to all our customers and vCluster users who were affected by this down time. We have worked tirelessly throughout the weekend to restore normal service but it is clear this is just the beginning.

To restore your faith in our service it is important to understand what went wrong from a technical perspective and I hope that Mark’s post will give those who are technically inclined some comfort that we have identified and fixed the root cause. We also want to assure you from a strategic perspective by showing what we have put in place to allay any concerns you might have in this area.

There are two key points I wish to make. Firstly the culture of our business is one of agile working methods, secondly we are working on very big, industry disrupting challenges. Typically with the types of challenges we are facing, companies will plan endlessly to try and minimise any interruption or disruption to service before they start.

We have taken the opposite approach to risk management. Our planning cycles are short and we iterate quickly as we move forward. We believe this is the right approach to achieve our goal of disrupting the category we work in with new ideas, rather than just incrementally improving on the work of others.

For all the benefits of agile working it does increase short term risks of failure. We have put our product in “Beta” for this reason while we work on large platform development projects and ideas. When it is mature to a commercial grade we will celebrate and remove this label but until then we also look to communicate openly and swiftly what is going on if we encounter problems.

It is important to us that our customers know we are working hard not only at issue resolutions but also at continuous development behind the scenes. I can assure you this is happening 24/7 when problems do occur. We care deeply about restoring normal service as swiftly as possible if any issue arises. Everyone in CatN is dedicated to, and passionate about, creating something truly industry changing from the resilience of our network to the innovation in our Control Panel.

We have received criticism as well as some much appreciated support and understanding from our customers as a result of this outage. I want you all to know we know we seriously fell below our own standards and what you should be able to expect from us but that we have learned a lot and will improve again as a result.

One of the greatest positives to come out of this downtime is a plan for a new series of technical blogs explaining what is going on behind the scenes at CatN, including a detailed plan of how we will defend against outages like this in the future. We are also hiring more engineers as our current huge client growth has impacted the resource available to develop new systems. We are recruiting to support this growth and have made two new appointments already. Our two new System Administrators will be focusing on evaluating redundant storage solutions for the vCluster stack.

Incidentally, if you would like to be involved with CatN and our rapid growth, the System Administrator job description is on LinkedIn: http://www.linkedin.com/jobs?viewJob=&jobId=2112024&trk=jobs_biz_nprem

I hope that steps like this will restore some of your confidence in us, and show you that we really are iteratively improving every level of our platform behind the scenes.

We remain at your service and grateful that you judge us on how we respond to problems.

At approximately 16:10UK on Friday (28/10/2011), our monitoring system picked up an outage on the vCluster platform and our engineering team began to investigate the problem.

A short while later the issue was narrowed down to a malfunctioning NFS layer causing the Edge web servers to lock up waiting for NFS IO. Further investigation showed that this behaviour was specific to one user - "apache-c1edge". This user is critical to the operation of the Edge web servers, which in turn are critical for the fast delivery of static content and proxying to the PHP application servers.

Tracing the NFS lockup

Using the strace tool we were able to work out where the IO was hanging:

open("/usr/lib/locale/locale-archive", O_RDONLY|O_LARGEFILE) = 3
fstat64(3, {st_mode=S_IFREG|0644, st_size=56458352, ...}) = 0
mmap2(NULL, 2097152, PROT_READ, MAP_PRIVATE, 3, 0) = 0xb7d4c000
mmap2(NULL, 212992, PROT_READ, MAP_PRIVATE, 3, 0x1391) = 0xb7d18000
mmap2(NULL, 24576, PROT_READ, MAP_PRIVATE, 3, 0x13ca) = 0xb7d12000
mmap2(NULL, 4096, PROT_READ, MAP_PRIVATE, 3, 0x13f4) = 0xb7d11000
close(3)                                = 0
ioctl(1, SNDCTL_TMR_TIMEBASE or TCGETS, {B38400 opost isig icanon echo ...}) = 0
ioctl(1, TIOCGWINSZ, {ws_row=48, ws_col=161, ws_xpixel=966, ws_ypixel=672}) = 0
stat64("/sites/client-example",

From this we could see that the Edge server was hanging on an NFS access() request. This problem would not be fixed until Sunday night (see later).

Filesystem consistency check

During the process of unmounting/remounting the storage, we found that the filesystem was overdue a consistency check.

This is because we have been concentrating on building our new vCluster platform with 2N redundancy rather than upgrading the existing system. Only having a single NFS server meant that we were unable to take the storage down to perform a file system check.

We decided to run a full filesystem check to rule out errors due to undetected corruption.

This process took around 3 hours to complete, during which time we continued to investigate the NFS IO lockup using a temporary storage mount. Upon completion the NFS hang remained. Several minor issues were fixed by the filesystem check but there were no lost inodes (files).

Edge server upgrades

To rule out a kernel bug we upgraded the kernels on the affected Edge servers. As soon as these servers were upgraded, the system came back online.

Again, though we didn't have knowledge of a specific bug fix in that kernel, it certainly seemed to fix the problem. With the cluster working, our engineers continued to monitor the system closely.

We were unable to find a specific bug fix though, so we remained skeptical the issue was resolved.

What happened on Sunday afternoon?

Sure enough, on Sunday afternoon the situation reappeared. Our monitoring system picked up the first alert at 15:13UK and shortly after our engineers confirmed the same problem was back.

At this point we considered several possibilities:

• A network/hardware issue
• An issue with LDAP lookups
• A bug in the NFS server kernel or NFS utils

One engineer was despatched to work on each query.

A network MTU issue

The first problem to be found was a network MTU issue between the NFS and LDAP servers. Our setup uses a mixture of bonding and vlans, and the ldap vlan on the nfs server had inherited an incorrect MTU of 9000 from the underlying device.

The MTU was set back to the normal 1500 bytes setting and this appeared to get the cluster working again. Unfortunately the cluster soon ground back to a halt.

NIC Errors

Further investigation found DMA errors on one of the NFS server network interfaces. Immediately we assessed our options - we could either replace the card or migrate to a new, known good server. Unfortunately the NIC was embedded on the server mainboard so to replace it would essentially mean pulling a board from our spares, which would be a lengthy process.

Fortunately we had a recently installed NFS server at hand on our development platform, running on our latest, most powerful hardware and Centos 6. This build was ultimately destined for the next platform release but it had been running long enough to show stability.

We migrated vCluster storage to this new machine.

rpc.mountd --managed-gids issues

Part of the CatN security model requires us to use the "--managed-gids" option to rpc.mountd. The reason for this is that the NFS protocol only allows for auxiliary membership of up to 16 groups, with the NFS client putting this data in the NFS ACCESS request.

Turning on this option enables the server to perform it's own lookup to LDAP, fixing up the problem.

Unfortunately it turns out that there is a subtle bug in the upstream nfs-utils package in Centos 6 which prevents managed GIDs from working in scenarios with >100 group memberships involved. Our development system had not yet reached this level of test accounts and the issue had therefore not shown up.

Fortunately a patch was readily available so our engineers built a patched nfs-utils package based on the default 1.2.2 version. Here is the commit in the linux-nfs tree:

http://git.linux-nfs.org/?p=steved/nfs-utils.git;a=commitdiff;h=5604b35a61e22930873ffc4e9971002f578e7978

And here is the post that led to finding this fix:

Final checks

After a brief period online we took the cluster back offline for final checks. During this time we performed a full set of tests to ensure that all layers of the vCluster platform were performing correctly.

Monday Morning

On Sunday night we patched NFS utils to version 1.2.2, and although it got the vCluster platform running we were still getting some warnings in the log. On Monday morning at 4am we went down again, we believe this was the same issue.

In response we upgraded NFS utils to version 1.2.3 plus the managed gids patch and this made all warnings go away in the log and NFS was immediately working again. This patch has since survived an intense backup load and has been stable for more than 24 hours, producing no warning messages in the log.

The follow up

• The response team will regroup after 48 hours of stability to review what happened, and see how our response could be improved in future.
• We will schedule another maintenance window to allow us some time to tidy up properly; for example our custom RPM is manually installed at the moment, but we need to put it into a proper yum repository and configure the system to track our custom package instead.
• We will continue to monitor the NFS very closely and be ready to react should any further issues crop up.

CatN was invited to become a foundation delivery partner and early adopter of the programme. Following various scoping sessions in the new year we will be offering our services in a cloud catalogue, allowing Government to procure on-demand computing services. This will cut out the long 10 year plus contracts that projects are tied in on, and massively reduce expenditure, as well as agility in procuring services. Of course all the benefits that cloud computing offers such as elasticity and resilience will also play a huge factor in improving the way Government procures IT, and also how much it costs them!

Chris Chant, Executive Director in the Cabinet Office, spoke about G-Cloud and what he is trying to achieve at the Institute for Government in London.

I will very shortly be publishing a report comparing last years website metrics for department websites with the report released earlier this year to see if the new Government ICT strategy has had any affect yet in reducing expenditure.

The best place to read the Government's plans for cloud services within their overall ICT strategy is on the Cabinet Office website, http://www.cabinetoffice.gov.uk/resource-library/uk-government-ict-strategy-resources

The new road map took a little inspiration from the beta.bbc.co.uk home page, and from our own whiteboards covering the walls of the CatN HQ! You can find the road map at http://catn.com/roadmap. Remember to check back as we will be updating the white boards as features are completed and suggested.

If you wish to suggest anything that you can't see listed, please head over to our Get Satisfaction page. We're really keen to gain as much feedback as possible as we finalise vCluster. Here can you submit product ideas and any bugs you may have found and the most popular ideas will be added to our development sprint.

Of course, if you fancy testing vCluster for yourself, submit your interest in our paid beta program from the link in our header.