About a year ago Dawid posted about the performance of .htaccess files. We decided to revisit these tests to compare the performance of .htaccess files on local disk and network filesystems. The network filesystem we used was Lustre, chosen partly because we are doing a lot of testing with Lustre at the moment but also because of its’ known issues with metatdata and small file performance.

The test setup is a Dual Xeon 3.2GHz server with 4GB of RAM and a gigabit network connection to the testing and storage vlans. We installed the latest version of our Apache 2.2.11 build with mpm-worker and two vhosts. One vhost has a DocumentRoot on local disk, the other has a DocumentRoot on Lustre storage.

We used apachebench (ab) to take the performance measurements as follows:

ab -n 5000 -c10 http://10.10.0.41/testfile.html

This test runs 5000 requests with a concurrency of 10 users. For each test run we noted down the mean total request time, the number of requests per second and the average network bandwidth used in kbit/s.

Test Variations

We tested the following combinations:

  1. Local disk, AllowOverride set to none for /

  2. Local disk, AllowOverride set to all for /, no .htaccess files in the test tree

  3. Local disk, AllowOverride set to all for /, with empty .htaccess files in the test tree

  4. Lustre filesystem, AllowOverride set to none for /

  5. Lustre filesystem, AllowOverride set to all for /, no .htaccess files in the test tree

  6. Lustre filesystem, AllowOverride set to all for /, with empty .htaccess files in the test tree

  7. Lustre filesystem, AllowOverride set to all for /, disabled for subdirs of /, no .htaccess files in the test tree

  8. Lustre filesystem, AllowOverride set to all for /, disabled for subdirs of /, with empty .htaccess files in the test tree

Test tree

The test tree was created on both local disk and local filesystems as follows:

mkdir -p /path/to/docroot/1/2/3/4/5/6/7/8/9/10

We put a dummy .html file in each directory and added/removed .htaccess files as necessary to complete the experiment.

For each test run we took measurements at request depths of 1,2,5 and 10 to determine performance as the amount of directories being scanned for .htaccess file increase. For example to test a request depth of 5 against the Lustre filesystem we used:

ab -n 5000 -c10 http://lustre.testrig.local/1/2/3/4/5/testfile.html

Results

1. Local disk, AllowOverride set to none for /

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 3 3471 881
2 3 3391 861
5 3 3407 865
10 3 3335 846

2. Local disk, AllowOverride set to all for /, no .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 3 3841 975
2 3 3202 813
5 3 3111 790
10 3 2814 714

3. Local disk, AllowOverride set to all for /, with empty .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 3 3404 864
2 3 2988 758
5 4 2736 695
10 4 2419 614

4. Lustre filesystem, AllowOverride set to none for /

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 4 2523 643
2 4 2681 683
5 4 2614 666
10 4 2360 601

5. Lustre filesystem, AllowOverride set to all for /, no .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 5 2078 529
2 6 1741 444
5 8 1255 320
10 12 839 213

6. Lustre filesystem, AllowOverride set to all for /, with empty .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 9 1042 265
2 14 719 183
5 21 477 121
10 36 275 70

7. Lustre filesystem, AllowOverride set to all for /, disabled for subdirs of /, no .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 5 1968 501
2 5 1961 500
5 5 1933 492
10 5 1878 478

8. Lustre filesystem, AllowOverride set to all for /, disabled for subdirs of /, with empty .htaccess files in the test tree

Request depth Mean response time (ms) Requests per second Network Rate (kb/s)
1 9 1060 270
2 9 1045 266
5 10 1035 263
10 10 1010 257

Observations

Local disk/Lustre Comparisons

  • The local disk tests verify Dawid’s previous observation that enabling .htaccess makes a resaonably small, but measurable difference to performance.
  • In general, using network storage (Lustre, probably NFS, Samba etc) for DocumentRoot storage is slower than local disk.
  • Enabling .htaccess on a network filesystem reduces performance even further, with an amplification effect as the request depth increases. At a request depth of 10, local disk was more than TEN times faster than Lustre!

Lustre-specific observations

  • With DocumentRoot on Lustre and AllowOverride All, an empty .htaccess file gives approximately TWICE the performance than no .htaccess file at all.

UPDATE: After adding some timing debug code to Apache we deduced that we must have got the results the wrong way round. The measurements were taken again to be certain…

  • Enabling overrides recursively from the top of the DocumentRoot tree seriously kills the performance of the server.
  • Preventing recursive .htaccess file lookups limits the performance hit.

Preventing recursive overrides

Preventing recursive overrides is quite straightforward. Simply add something like the following to your <VirtualHost> configuration:

# Allow overrides in the document root
<Directory /path/to/docroot>
AllowOverride All
</Directory>

# Disable overrides below the document root
<Directory /path/to/docroot/*>
AllowOverride None
</Directory>

Improving performance with Lustre

As a final exercise we delved into the Apache source code to see what was actually going on inside Apache.

We’ve spotted a potential way of bringing .htaccess performance on Lustre up to par with local disk. In fact our idea might even improve performance for local disk as well…

Apache 2 has an internal cache for .htaccess files, but this is local to the request being handled and dies with the request. As far as we can tell this cache is designed is to speed up complicated .htaccess configurations where loops and subrequests cause the same .htaccess files to be accessed multiple times within a single request.

While we have no doubt that this improves performance in complicated configurations, we wondered if creating a server-wide caching mechanism for .htaccess files would help on busy servers, especially those with large document trees on network storage.

We’re busily coding something up and will post again on this topic in the very near future. Until then, happy Apache tweaking!

Mark Sutton Chief Technology Officer

As CTO Mark defines the technical strategy for CatN, designs and builds our key IT assets and engages with the IT community to communicate the CatN vision. You can find Mark on Google+