NFS Server Outages
By Mark Sutton Wednesday, 2nd November 2011
What happened?
At approximately 16:10UK on Friday (28/10/2011), our monitoring system picked up an outage on the vCluster platform and our engineering team began to investigate the problem.
A short while later the issue was narrowed down to a malfunctioning NFS layer causing the Edge web servers to lock up waiting for NFS IO. Further investigation showed that this behaviour was specific to one user – “apache-c1edge”. This user is critical to the operation of the Edge web servers, which in turn are critical for the fast delivery of static content and proxying to the PHP application servers.
Tracing the NFS lockup
Using the strace tool we were able to work out where the IO was hanging:
Posted in Service Announcements, vCluster | 3 Comments »
WordPress VIP hosting for the rest of us…
By Mark Sutton Tuesday, 9th February 2010
As you may have read, TechCrunch, a blog focused on profiling and reviewing internet products, was hacked a couple of weeks ago.
They were using Rackspace’s hosting service, but if you run a traceroute on the TechCrunch domain you’ll see that it’s now pointing to wordpress.com. Even more of a giveaway, they have changed the hosting tag in the footer of their site, so it now says, “Powered by WordPress.com VIP“.
Posted in Hosting, Wordpress | No Comments »
Speculations on the Facebook PHP Rumours…
By Mark Sutton Monday, 1st February 2010
Rumours abound this weekend, and as usual in such matters, details are thin on the ground. Did that meeting really happen? And who is this lonesome coder supposedly re-writing PHP for the last two years? Is it a new runtime or a compiler? Someone knows, and for sure something is afoot, but what is it?
Posted in PHP | No Comments »
Securing mod_php – without the vasectomy
By Mark Sutton Thursday, 28th January 2010
In it’s most typical setup, Apache 2/mod_php is not secure enough for shared virtual hosting.
With the typical mod_php setup, a rogue webmaster or bug in a single PHP script could compromise every script and configuration file for every site hosted on the server, no matter how security conscious the developer.
Posted in Security | 6 Comments »
OpenVZ forced umount of lustre mount problem
By Mark Sutton Tuesday, 31st March 2009
Recently we managed to find an answer to a quite worrying lustre problem that has been bugging us for some time. Every now and then on servers running OpenVZ containers that make use of lustre filesystem we would see a log entry in /var/log/messages saying:
kernel: Lustre: setting import lustre-server-MDT0000_UUID INACTIVE by administrator request
followed by a number of broken mounts/fs errors inside containers running on the server that the log entry appeared. In effect, all the containers making extensive use of the same lustre server would stop working properly (for example, apache serving sites from lustre mounts would start spawning processes all of which would be unsuccessfully trying to read data from the mounts).
Posted in Technology, vCluster | 2 Comments »