Migrating the Department for Transport website to WordPress
By Joe Gardiner Monday, 23rd August 2010
From the conclusions made in the article UK government's spending on hosting and development, I decided to migrate the Government site DFT (Department for Transport) to a php application, specifically WordPress, as this is a CMS we have much experience developing and hosting.
Summary
The key points from the rest of this report follow:
Migration - We migrated dft.gov.uk from it's current management system to WordPress to illustrate the ease of the process and that the site can be moved to an open source, free management system for very little cost. We had no access to any site files but used a small script to crawl the sites content and images.
Hosting - The copied site is now hosted on our php platform vCluster and was developed on the same platform it is now hosted on.
Expenditure - The migration and hosting process resulted in costs to us of £5000. This small investment could save up to £64,800 per month, £777,600 per annum, and this is for a single Government site.
You can view the result of the migration and our sample site running on WordPress here: DFT sample site.
My aim is to illustrate the ease of the process and to show the reduction in expenditure that can be achieved by using an open source hosting platform and a low cost language to write and maintain, like php. I chose DFT as it had one of the worst value for money "cost per request" rates in my Government value for money report.
| Government Site | Annual hosting and infrastructure spend | Annual page requests | Cost per request (£'s) |
|---|---|---|---|
| dft.gov.uk | £810,000 | 17497004 | 0.046 |
To put the cost per request for DFT into context, the next highest cost per request is communities.gov.uk:
| Government Site | Annual hosting and infrastructure spend | Annual page requests | Cost per request (£'s) |
|---|---|---|---|
| communities.gov.uk | £372,000 | 29767248 | 0.012 |
As you can see, DFT is almost four times as expensive per request than communities.
The site dft.gov.uk can be considered a "static" web site. This means that content is not dynamically generated, and that there are no complex application on the dft.gov.uk. the DFT site is purely informational and contains static content for this purpose.
Migration process
By Iain Cambridge
Data Collection
As I didn’t have access to a copy of the raw database to gather the data, I crawled the site using a simple PHP spider. With use of Curl and regular expressions the data was collected from the DFT server and prepared for insertion into a new wordpress install.
Data sorting
I sorted and kept the category structure of the original site with the ease using wp_insert_category() function which allowed me to create a category while defining the category’s description, name and parent category ID.
During this process I encountered a bug in the function which prevented wp_get_categories() from creating children categories from the newest parent category inserted. To counteract this problem at the end of the importing I created a redundant category in the wp-admin category management system and deleted it once all categories were created. The insertions of posts was made extremely easy with the use of wp_insert_post().
Theme
I started with recreating the top navigation menu which I achieved by selecting only the parent categories. The navigation colour change for the currently selected category was achieved by using grandchild categories with the use of post_is_in_descendant_category() which I found on the wordpress codex.
To recreate the footer I created four dynamic widget areas, created link categories, and inserted them into the footer. I used three link widgets as well as text widgets into which I inserted html. On the front page I created a sidebar widget for navigation and used two text widgets with two link widgets to recreate the look of the DfT site. The Link Categories was used to recreate the “Current Issues”,”DfT Spotlights” and “Useful Links” navigation.
To recreate the "what’s new" section I created a page with the nice name of “whatsnew” then created a theme file called page-whatsnew.php and copied the basic wordpress home.php functionality into it.
Plugins
There were certain theming and page functions which required plugins, some of these I was able to download from wordpress.org. However I did have to write plugins for certain features, this was made extremely easy with WordPress’s plugin API.
One of these plugins was created to extend the category data that could be held. The DfT site had “excerpt” like data for the categories as well as a description of the category. I found that most categories also had a related links in the side bar along with a widget showing the most visited posts in that category and children categories. As well categories excerpts the category extension included a widget to attach a link to a category to a categories data as related links.
A second plugin created an A to Z section to replicate the section in the original DfT site. I edited a plugin called Breadcrumb NavXT to give the option of displaying the users current page in the breadcrumb navigation.
The unedited plugins I used straight of the shelf was PS Auto Sitemap to create the site map and WP-Paginate to give a better pagination display for the what’s new section. Finally I installed and configured SuperCache to improve site performance.
The sample site is now being hosted on a Fubra development vCluster, on the CatN platform. We migrated 4 levels deep giving a good overview of the sites categories without scraping the entire sites data which would be very time consuming for a sample case study. After extensive testing of the site, we have achieved full functionality in WordPress (4 levels deep), and have achieved excellent site performance with no extra configuration of the CatN platform required.
You can view the result of the migration and our sample site running on WordPress here: DFT sample site.
How much?
It is difficult to give a precise figure for the required expenditure for hosting dft.gov.uk on a CatN vCluster, as we have no access to site analytics. However, I have compared the percentage of global page views with other sites hosted by CatN on the Alexa Web Information site, to give a suggestion of the price range DFT may fit into, and I have page impressions and total site visitors data from the COI Report on Progress 2009/10.
Note: There are some questions over the accuracy of Alexa traffic data, particularly with sites heavily dependent on social media that will receive spikes for certain articles effecting similar site rankings. I am not fully dependent on Alexa statistics, but am just using them as a guide to fit dft.gov.uk into a realistic price range as an indicator of the cost of hosting on the CatN platform.
The sites I am comparing are not heavily dependant on social media, and are not tiny sites whose scale is difficult for Alexa to manage. All of these sites receive consistent traffic at the levels expressed in the following graphs, and do not use the Alexa toolbar to manipulate Alexa rankings.
Have a look at my blog post Alexa Accuracy, or an Alexa blog about site rankings: What's going on with my Alexa rank?Alexa and Analytics
Adding Site 2 as a comparison to dft.gov.uk reveals that it is gaining a higher pageviews rank in Alexa, in fact at some points on the graph, four times as many. As I have access to analytic data for Site 2 a more accurate comparison can be made
| Website | Page impressions (June 2010) | Site visits (June 2010) |
|---|---|---|
| dft.gov.uk | 1,458,084 | 426,748 |
| Site 2 | 8,755,142 | 1,755,767 |
The analytic data agrees with Alexa's ranking, showing Site 2 to be approximately four times the size of dft.gov.uk in terms of traffic and global reach. Although it would be possible to scale down Site 2's hosting expenditure for an indication of the required spend for DFT on our platform, comparison with similarly sized sites will increase the accuracy of the estimate.
Site 3 and Site 4 are given a similar rank by Alexa to DFT, so are worth considering in more detail. The similarities in rank may be reflected in traffic patterns, allowing a clearer indication of the required expenditure for hosting DFT on a vCluster.
| Website | Page impressions (June 2010) | Site visits (June 2010) |
|---|---|---|
| dft.gov.uk | 1,458,084 | 426,748 |
| Site 3 | 2,106,436 | 361,207 |
| Site 4 | 1,153,383 | 268,780 |
The analytic data shows that dft.gov.uk can be considered a similar size in terms of traffic to Site 3 and larger than Site 4, so a fair estimate should be similar to Site 3 and approximately a quarter the size of Site 2, a pessimistic estimation would be around half the size of Site 2.
COI pricing comparison
From the COI Reporting on Progress 2009/10 I have the following site data for dft.gov.uk:
| Website | Page impressions (June 2010) | Site visits (June 2010) | Hosting expenditure June 2010 (£'s) |
|---|---|---|---|
| dft.gov.uk | 1,458,084 | 426,748 | 67,500 |
From the traffic data of Site 2, 3 and 4 in my previous comparison, I can compare hosting expenditure across the sample with the spend disclosed by the COI report for an indication of the expenditure required for hosting DFT on a vCluster.
| Website | Page impressions (June 2010) | Site visits (June 2010) | Hosting expenditure June 2010 (£'s) |
|---|---|---|---|
| dft.gov.uk | 1,458,084 | 426,748 | 67,500 |
| Site 2 | 8,755,142 | 1,755,767 | 2,295 |
| Site 3 | 2,106,436 | 361,207 | 307 |
| Site 4 | 1,153,383 | 268,780 | 240 |
In comparison to sites of a similar size in terms of traffic, particularly Site 3 and Site 4, DFT seems to have an enormously inflated monthly hosting expenditure. I have visualised this below with a following graph displaying page visits.
Hosting expenditure (June 2010) for sample sites
Site visits (June 2010) for sample sites
There is no clear correlation between site visitors and hosting expenditure when comparing any of the sample sites to dft.gov.uk. Site 2 with 3.5 times more visitors has a hosting expenditure almost 30 times smaller per month. This is an annual difference of £782,460, provided DFT is ported to a php CMS such as WordPress. Presuming that dft.gov.uk did achieve the same traffic levels as Site 2 and was spending approximately £3,000 per month using a vCluster, DFT could expect to make savings of up to 96.6%.
Estimate
It is difficult to give a precise estimate, there are clear indication from my analysis of the price bracket DFT should fit into if hosted on a CatN vCluster:
- Using the sample comparison sites as a pricing guide
- Taking into account the requirements of the dft.gov.uk website's static content
- Inputting traffic data from the COI report to the vCluster Platform Hour Calculator
Please note that the estimate is precisely that, and does not include any development costs required for migrating dft.gov.uk to an open source php CMS.
Comparison guide
| dft.gov.uk | From | To |
|---|---|---|
| Price range from comparison sites: | £240 | £2,295 |
| Platform Hour setting | Estimated monthly hits | Platform Hours used | Platform Hour charge |
|---|---|---|---|
| Static | 245,807,280 | 3,392 | £1,696 |
Taking the Platform Hour charge of £1,696 and adding the package charge for vCluster Ultimate of £100, all that remains is Data Transfer charges if usage exceeds allowance.
The average page size from 10 sample pages of dft.gov.uk is 287kb. Multiplying average page size with monthly page views allows a data transfer estimate to be made.
| Monthly page views | Average page size | Monthly Data Transfer estimate | Data Transfer charges |
|---|---|---|---|
| 1,458,083 | 287kb | 418GB | None |
As the vCluster Ultimate package allows 500GB of data transfer before usage charges are incurred, from the above estimate dft.gov.uk would incur no data transfer charges as 418GB are being used. Therefore the only usage charges from this estimate are for Platform Hours. The total charge based on these figures would be £1,796 (Platform Hours + vCluster Ultimate subscription).
| Subscription charge | Platform Hour overage | Data transfer overage | Total |
|---|---|---|---|
| £100 | £1,696 | None | £1,796 |
The estimate of £1,796 fits within the comparison sites range of £240 to £2,295, and is similar to the average of the range. As the top end of the range, Site 2, has traffic levels 6 times higher than dft.gov.uk and is charged £2,295, I would feel confident that DFT would be very unlikely to be charged more than £4,000 per month for being hosted on a platform such as the CatN vCluster. In fact £4,000 is a pessimistic estimate based on the analysis in this report.
Even if dft.gov.uk did achieve monthly bills near £4,000, it would still be receiving a saving of 94% per month from the current £67,500 expenditure, and this is based on the worst case scenario for DFT hosting expenditure.
Can the government afford to ignore the potential savings of moving to an open source, cloud hosting platform? In a period of savings and public service cuts, surely web hosting expenditure should be reviewed when such large savings seem possible.
You can view the result of the migration and our sample site running on WordPress here: DFT sample site. Don't believe it's WordPress? Add "wp-admin" to the site url for the WordPress login screen.
Posted in Government Hosting |
3 Comments » 
Hi Joe
Im Jason & I work for a large UK government website which is a big sister site to the one discussed in your blog. We are currently looking to prototype some new developments & site improvements and were very interested in your findings as we feel to build a WordPress “clone” of our existing site to help us openly discuss & implement change is a safe offline option which could (functionality permitting) eventually be a contender to replace our incumbent site structure.
From the approach you have taken and the steps therein to a resourcing and ultimately a detailed costing point of view (including hosting), we would like to talk to you about your experiences and how we can possibly do business. Could you please e-mail me at the above address? (Apologies for being slightly obscure but at this very early stage, we would prefer to keep things on a more confidential level)
Many thanks & regards
Jason Phillips
Hello, i know this is not exactly your topic “Migrating the Department for Transport website to WordPress CatN, but i have a blog using the wordpress platform as well and i’m having issues with my comments displaying. is there a setting i am forgetting? maybe you could help me out? thank you.
Hello,
You may not have the comments tage in your index.php file.
You need to add just before the end of the WordPress loop.
Thanks,
Joe