Mistake #1

If you move your .htaccess contents into the apache config files that is great, but if you forget to disable .htaccess files using AllowOverride None then you’ve entirely defeated the purpose! The reason this technique can result in performance gains is that it allows the server to ignore the .htaccess files which means fewer directory traversals and fewer file reads with each request. One page load probably results in 20-100 HTTP requests to your Apache server. If you have deeply nested files (all of those cached product images and skin images come to mind) then this results in a lot of disk access. So, you must disable the .htaccess files completely with AllowOverride None! Which brings me to the second major mistake that people can make…

Mistake #2

Now that you’ve disabled .htaccess, guess what? Those .htaccess files had a purpose.. For example, to deny access to your app/etc/local.xml config file!! Great way to expose your database password, let’s just hope the user is a localhost-only user and you don’t re-use passwords… Also, in downloader/.htaccess for example, compression is disabled so that the Magento Connect console can circumvent Apache’s buffering to display the console output as it occurs. There are others as well. Point is, now you have to move all of these .htaccess files into your apache config. Ugh.. Which brings me to my last point…

Mistake #3

It doesn’t have to be difficult! I’ve written a bash script (buildhtaccess.sh, see below) which provides you an easy way to benefit from this tweak. In essence the following script will find all of your .htaccess files and compile them into one file which can then be included in your Apache config with only a few lines. If you modify an .htaccess file or a new one is installed by an extension, re-run the script and reload the apache config.

Tutorial

First, download and run the script to generate your .htaccess-combined file.

$ cd <webroot>
$ wget http://gist.github.com/raw/459311/buildhtaccess.sh
$ bash buildhtaccess.sh

Next, edit your apache config file. Find the <VirtualHost> directive and at the end of it add the following:

  # Include combined Magento .htaccess files
  <Directory /your_webroot_here>
    AllowOverride None
  </Directory>
  Include /your_webroot_here/.htaccess-combined

Lastly, run a graceful reload of the Apache config. A full restart is not necessary. Now you’re done.

Code

And here is the code for generating the .htaccess-combined file:

Benchmarks

For my benchmark I ran “ab” (apache bench) since it is quick and easy but admittedly not the best for benchmarking overall site performance. In this case apache bench is going to under state the value of this tweak since one page load in a browser will result in repeated hits on the webserver, all of which will be affected by this optimization. So, I benchmarked on /errors/503.php so that the database is not factored but PHP code is still run. Specifically, I used $ ab -n 1000 -c 20 .../errors/503.php.

Before: ~3000 requests per second (3018 max)

After: ~4600 requests per second (5030 max)

That’s a pretty nice improvement. Unfortunately it is only substantial with fast requests such as the error pages and static files but it should definitely reduce disk contention on a busy server and increase scalability, but probably not do a whole lot for user-experience on a low-contention server.

I’d love to see your results, please post them in the comments!

Additionally, Module Manager now has it’s own home on Google Code so check it out! (pun intended)

Here is an example of the new nested modules feature:

# My template files
skin               skin/frontend/my/default
design             app/design/frontend/my/default

# Import Colin_HotDealz module
@import            modules/Colin_HotDealz

Update 1/30/2010:
Added nested modules, renamed .modman file to modman, released under Apache License 2.0, and hosted on Google Code.

This project was inspired by my recent Magento development work so I’m not sure how much application it has outside of that, but if you’ve ever wanted to use svn:externals with individual files (you can’t btw) then this is the next best thing (and therefore the best thing). Perhaps some other VCS supports this functionality natively , if you know of it please clue me in. I’ve used git and bazaar but still like the simplicity of subversion for small, non-distributed projects.

Magento’s source code is split among several different groups of files: code, design, locale, skin, libs, etc (as in config), etc.. (as in etcetera)
You really cannot avoid ever having to mix a file in here or there which means svn:externals won’t cut it. Besides, svn:externals won’t let you make one commit from all of your externals so to commit your changes you have to do it once for each folder which makes your commit history nearly useless. Enter “modman”, my Magento module manager. While it could be used for deployment it is primarily geared towards developers. Here is how it works:

Create a branch and put your source code in it. Now create a “.modman” file within the root of the branch. In this file you define (much like svn:externals) a path within your repository and the path that it should map to. Here is an example (with a fictitious module named “HotDealz”:

code   app/code/local/Colin/HotDealz
template   app/design/frontend/default/default/template/hotdealz
hotdealz.xml   app/design/frontend/default/default/layout/hotdealz.xml
images   skin/frontend/default/default/images/hotdealz
HotDealz.csv   app/locale/en_US/Colin_HotDealz.csz

As you can see, very similar to svn:externals only with individual file support which is handy for things like locale files that can’t go in subdirectories.

How It Works

Usage is similar to regular command line usage of subversion. The “modman” script provides a wrapper around svn which checks the code out in a .modman directory at the root of your project then reads the .modman file and creates symbolic links as necessary so that your source code is actually all contained within one directory and your webserver accesses the files where they are supposed to be. No svn:externals definitions are used. Obviously this requires a few things:

1. bash
2. a webserver that is configured to FollowSymLinks

If you do web development on Windows, you may be thinking, “What about Windows support!?”. Well, you have a few options:

• Do your development inside a VirtualBox instance of Linux and share the drive to Windows for editing on your IDE (recommended).
• Try to configure cygwin to work (untested, not recommended).

I chose the former long before I wrote this script since the servers for my projects are all Linux-based and doing development on WAMP when your project runs on L*MP is silly. You can actually still use TortoiseSVN by simply opening the .modman directory from explorer so really after the initial checkout you do not need to continue to use the script except for when adding new definitions to the .modman file.

Sample Usage

Here are some typical commands used with modman:

$ modman --help
$ modman init
$ modman hotdealz checkout "svn url here"
$ modman hotdealz status (can be run from any directory within the project root)
$ modman hotdealz commit
$ modman hotdealz update
$ modman hotdealz add Colin_HotDealz.xml app/etc/modules/HotDealz.xml
etc...

Most commonly used svn commands are passed through to svn for convenience but for advanced work you can just cd to the .modman directory or use TortoiseSVN or your IDE’s integrated subversion tools (I use Netbeans).

Documentation

----------------------
Magento Module Manager
----------------------
Usage: modman <module> <action> [svn args]
Supported global actions (no module specified):
  init           initialize the current directory as the modman root (no module name)
  update-all     update all modules that are currently checked out

Supported actions: (all svn commands support additional arguments)
  checkout       checkout a new modman compatible module
  export         export a modman compatible module (does real copy instead of symlinks)
  update [...]   update module
  add <svn_path> <real_path>   add a file/dir to the .modman file and working copy
  delete <svn_path>   remove a file/dir from the .modman file and working copy
  status         show status of entire module
  diff [...]     run svn diff on module
  commit [...]   commit changes to module
  info           show the module's working copy path and run svn info on working copy
  list           list the definitions in the .modman file

The repository should contain a file called ".modman" which defines which files
go where relative to the directory where modman was initialized.

If additional arguments are given to commands such as commit, paths
should be relative to the svn root.

---- Start example .modman file ----
# Comments are supported, begin a line with a hash
code                   app/code/local/My/Module/
design                 app/design/frontend/default/default/mymodule/
locale/My_Module.xml   app/locale/en_US/My_Module.xml
My_Module.xml          app/etc/modules/My_Module.xml
---- End example .modman file ----

Author:
Colin Mollenhour

http://colin.mollenhour.com/

colin@mollenhour.com

View Source

Download

Leave a comment if you use it!

It turns out that when layered navigation indexes are refreshed the website_id is 0 which causes the filtered results to be empty which makes layered navigation disappear. The fix is to simply place an if statement around the filter dependent on the website id no being 0. The Magento Connect module has been updated with this fix (0.3.1).

Ok, I think this will finally be the last update to this module… I’ve learned a lot about Magento since the first version of this module, and think I finally did it right this time. This update adds support for advanced search (thanks saho for the bug report) and also every other aspect of the store I could find.

I ended up doing a search like so:

app/code/core/Mage $ grep -rl addVisibleIn.\*FilterToCollection .
./Checkout/Block/Cart/Crosssell.php
./Tag/Block/Product/Result.php
./Tag/Block/Customer/View.php
./Adminhtml/Block/Sales/Order/Create/Search/Grid.php
./Adminhtml/Model/Sales/Order/Random.php
./Bundle/Block/Catalog/Product/List/Partof.php
./Bundle/Model/Observer.php
./Reports/Block/Product/Abstract.php
./Wishlist/Block/Share/Email/Items.php
./Wishlist/Block/Share/Wishlist.php
./Wishlist/Block/Customer/Wishlist.php
./Wishlist/Block/Customer/Sidebar.php
./Wishlist/Model/Mysql4/Wishlist.php
./CatalogSearch/Model/Layer.php
./CatalogSearch/Model/Advanced.php
./Rss/Block/Catalog/New.php
./Rss/Block/Catalog/Special.php
./CatalogIndex/Model/Indexer.php
./Sales/Model/Order.php
./Catalog/Block/Seo/Sitemap/Product.php
./Catalog/Block/Product/Compare/List.php
./Catalog/Block/Product/New.php
./Catalog/Block/Product/List/Related.php
./Catalog/Block/Product/List/Crosssell.php
./Catalog/Block/Product/List/Upsell.php
./Catalog/Block/Product/Bestsellers.php
./Catalog/Model/Layer.php
./Catalog/Model/Product/Visibility.php
./Catalog/Helper/Product/Compare.php

and determined (obviously) that it would be best to simply override the addVisibleIn*FilterToCollection functions rather than to try to override all of the above. I actually discovered the following code which turned out to not be what I wanted, but I think that is due to it not being updated when Varien added the stock_status table:

Mage::getSingleton('cataloginventory/stock')->addInStockFilterToCollection($collection);

However, I liked the idea of having such a function available that actually works as intended (the above does not work on configurable products) so I did just that.

First, the function that can be used on any product collection to filter out of stock products:

app/code/local/Lucky/InStockOnly/Model/Stock.php

<?php

class Lucky_InStockOnly_Model_Stock {

  /**
   * Add a statusInStock requirement for visibility
   */
  public function addInStockFilterToCollection($collection)
  {
    if($websiteId = Mage::app()->getWebsite()->getWebsiteId()) {
      $collection->joinField(
        'stock_status',
        'cataloginventory/stock_status',
        'stock_status',
        'product_id=entity_id', array(
          'stock_status' => Mage_CatalogInventory_Model_Stock_Status::STATUS_IN_STOCK,
          'website_id' => $websiteId
        )
      );
    }
  }

}

<?php

class Lucky_InStockOnly_Model_Visibility extends Mage_Catalog_Model_Product_Visibility {

  public function addVisibleInCatalogFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInCatalogFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

  public function addVisibleInSearchFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInSearchFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

  public function addVisibleInSiteFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInSiteFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

}

<?xml version="1.0"?>
<config>

<global>
  <models>

    <catalog>
      <rewrite>
        <product_visibility>Lucky_InStockOnly_Model_Visibility</product_visibility>
      </rewrite>
    </catalog>
    <cataloginventory>
      <rewrite>
        <instockonly>Lucky_InStockOnly_Model_Stock</instockonly>
      </rewrite>
    </cataloginventory>

  </models>
</global>

</config>

<config>
  <modules>
    <Lucky_InStockOnly>
      <active>true</active>
      <codePool>local</codePool>
    </Lucky_InStockOnly>
  </modules>
</config>