Home > Uncategorized > The Right Way to Optimize Apache’s .htaccess Files

The Right Way to Optimize Apache’s .htaccess Files

June 30th, 2010

While researching Magento performance optimizations you have probably already read about how to optimize Apache’s configuration by moving your configuration directives into the configuration files and out of the .htaccess files. Of course you need root to do this, but assuming you can, there are still very very wrong ways to do this that will result in no real performance gains, leave gaping security holes, and consume more time than necessary. Read on, a handy script for solving these problems and before and after performance benchmarks to prove the gains are included.

Mistake #1

If you move your .htaccess contents into the apache config files that is great, but if you forget to disable .htaccess files using AllowOverride None then you’ve entirely defeated the purpose! The reason this technique can result in performance gains is that it allows the server to ignore the .htaccess files which means fewer directory traversals and fewer file reads with each request. One page load probably results in 20-100 HTTP requests to your Apache server. If you have deeply nested files (all of those cached product images and skin images come to mind) then this results in a lot of disk access. So, you must disable the .htaccess files completely with AllowOverride None! Which brings me to the second major mistake that people can make…

Mistake #2

Now that you’ve disabled .htaccess, guess what? Those .htaccess files had a purpose.. For example, to deny access to your app/etc/local.xml config file!! Great way to expose your database password, let’s just hope the user is a localhost-only user and you don’t re-use passwords… Also, in downloader/.htaccess for example, compression is disabled so that the Magento Connect console can circumvent Apache’s buffering to display the console output as it occurs. There are others as well. Point is, now you have to move all of these .htaccess files into your apache config. Ugh.. Which brings me to my last point…

Mistake #3

It doesn’t have to be difficult! I’ve written a bash script (buildhtaccess.sh, see below) which provides you an easy way to benefit from this tweak. In essence the following script will find all of your .htaccess files and compile them into one file which can then be included in your Apache config with only a few lines. If you modify an .htaccess file or a new one is installed by an extension, re-run the script and reload the apache config.

Tutorial

First, download and run the script to generate your .htaccess-combined file.

$ cd <webroot>
$ wget http://gist.github.com/raw/459311/buildhtaccess.sh
$ bash buildhtaccess.sh

Next, edit your apache config file. Find the <VirtualHost> directive and at the end of it add the following:

  # Include combined Magento .htaccess files
  <Directory /your_webroot_here>
    AllowOverride None
  </Directory>
  Include /your_webroot_here/.htaccess-combined

Lastly, run a graceful reload of the Apache config. A full restart is not necessary. Now you’re done.

Code

And here is the code for generating the .htaccess-combined file:

Benchmarks

For my benchmark I ran “ab” (apache bench) since it is quick and easy but admittedly not the best for benchmarking overall site performance. In this case apache bench is going to under state the value of this tweak since one page load in a browser will result in repeated hits on the webserver, all of which will be affected by this optimization. So, I benchmarked on /errors/503.php so that the database is not factored but PHP code is still run. Specifically, I used $ ab -n 1000 -c 20 .../errors/503.php.
Before: ~3000 requests per second (3018 max)
After: ~4600 requests per second (5030 max)

That’s a pretty nice improvement. Unfortunately it is only substantial with fast requests such as the error pages and static files but it should definitely reduce disk contention on a busy server and increase scalability, but probably not do a whole lot for user-experience on a low-contention server.

I’d love to see your results, please post them in the comments!

Uncategorized , ,

  • http://www.sonassihosting.com Ben Lessani

    Hi Colin,

    An interesting post, and the concept is sound. We've done a lot of testing around this and although there are performance increases for static content – Magento does rely heavily on the delivery of dynamic content. The majority of which can be held in memory via an OpCode cache (using SHM on Eacc/APC) or by using Memcache as a ./var storage means.

    The problem being, if it is a sole server, responsible for both static and dynamic content, the limitation of the dynamic content will far exceed that of the static content by a factor of around 100 – making static file delivery optimisation somewhat moot. User CPU usage will far exceed any IOWait incurred.

    There are also the downsides of losing the flexibility of .htaccess on-the-fly configuration, giving the same effective functionality (or lack of) Nginx. Which would be a better replacement for static file delivery, and would almost double the throughput of Apache for static content. Given your example requires root/privileged access, one would assume you could also install new applications.

    However, in a clustered/split server environment, performance tuning static content servers yeilds excellent results and serves a great performance boost (once dynamic content has been isolated from static content). We operate a similar level of functionality on our clustered service.

  • http://colin.mollenhour.com Colin Mollenhour

    Agreed on this optimization not being beneficial for sluggish dynamic requests. I also later came to the conclusion that for Magento putting nginx in front of apache to serve static content directly was a better solution. But, for the people that want an Apache-only solution this is probably the next best thing and for the pedantic crowd this sure beats doing the same thing manually (which has been suggested too many times).

  • Peter Svegrup

    Hi Colin,

    If the server setup is both static and dynamic being delivered from the same instance, do you still recommend putting ngnix in front of apache or even swapping out apache with nginx entirely? I have read so many conflicting recommendations on the subject that it would be great to know what is the best practice and what possibly changes the recommendation?

    Thanks in advance

    Peter

  • http://colin.mollenhour.com Colin Mollenhour

    If you’re using a CDN for static content it should hardly matter at all. Even if you aren’t using a CDN it won’t matter much, but I still like to use nginx for static files so that all apache processes are PHP processes. This makes the logs and monitoring more useful as well. Setup your nginx config to only have access to /media, /js and /skin (e.g. using a location { root } config for each) and you have a very simple, secure setup.

  • surfer

    Thank you very much! I see a slight improvement of 5-10% in transactions, so it's worth implementing this.
    I've used a small sitemap with links to product pages.

  • Doug Lukas

    With regard to RAM use reduction; would installing this on my apache server that is running multiple instances of Drupal CMS sites be beneficial?

  • Homer Dong

    Although this approach is widely aware. Serving an easy to use script does really benefit the community extensively. Much appreciated.