OpenVZ forced umount of lustre mount problem

By Mark Sutton Tuesday, 31st March 2009

Recently we managed to find an answer to a quite worrying lustre problem that has been bugging us for some time. Every now and then on servers running OpenVZ containers that make use of lustre filesystem we would see a log entry in /var/log/messages saying:

kernel: Lustre: setting import lustre-server-MDT0000_UUID INACTIVE by administrator request

followed by a number of broken mounts/fs errors inside containers running on the server that the log entry appeared. In effect, all the containers making extensive use of the same lustre server would stop working properly (for example, apache serving sites from lustre mounts would start spawning processes all of which would be unsuccessfully trying to read data from the mounts). read more...

Posted in Technology, vCluster | 2 Comments »

Adventures with Lustre

By Mark Sutton Thursday, 26th March 2009

For the last few months we've been busy integrating, testing and tuning Lustre for use on our hosting platform. I thought I'd share some notes... Lustre is most widely used in HPC settings and there seem to be relatively few operations using it in conjunction with web servers and virtualisation. Reading through the wiki and mailling lists it soon becomes clear that Lustre has not been designed with small files and high metadata request rates in mind. read more...

Posted in Technology, vCluster | 3 Comments »

Preparing Procurve Switches for Production

By Mark Sutton Monday, 23rd March 2009

We recently took delivery of some new Procurve 5406zl switches for our Cloud platform. As with any new device being attached to the network there are quite a few tasks that must be done before it can be plugged into the production network. Naturally we document this internally but this time the process was quite interesting so I thought I'd blog it as well. read more...

Posted in Technology, vCluster | No Comments »

.htaccess revisited

By Mark Sutton Thursday, 19th March 2009

About a year ago Dawid posted about the performance of .htaccess files. We decided to revisit these tests to compare the performance of .htaccess files on local disk and network filesystems. The network filesystem we used was Lustre, chosen partly because we are doing a lot of testing with Lustre at the moment but also because of its' known issues with metatdata and small file performance. The test setup is a Dual Xeon 3.2GHz server with 4GB of RAM and a gigabit network connection to the testing and storage vlans. We installed the latest version of our Apache 2.2.11 build with mpm-worker and two vhosts. One vhost has a DocumentRoot on local disk, the other has a DocumentRoot on Lustre storage. read more...

Posted in Technology, vCluster | 1 Comment »

Getting to know Lustre

By Mark Sutton Tuesday, 17th March 2009

Lustre is a complex cluster filesystem aimed at super computing clusters, offering scalability to many Gigabytes per second and Petabytes of raw storage. We've been testing it for some time as a base filesystem for virtualisation and I wanted to share some notes. read more...