This module adds a column to the orders grid which displays either the product name and ordered quantities or the product SKU and ordered quantities which is configurable under Sales > Orders Grid Products Column. The one not selected to be displayed will be added to the title attribute so it can be seen with a mouse hover. The column is filterable by either the product name or SKU depending on the configuration.

Enjoy!

1. Merge the two (or more) module .xml files manually into one .xml file with the proper XML element order.
2. Name the .xml files (or symlinks a.la. modman) such that they sort alphabetically in the proper order.

The element order within a single file will be stable (SimpleXML), and Magento loads the files alphabetically except that those beginning with Mage_ come first. Problem solved.

Example:

Before:
  app/etc/modules/My_Module.xml
  app/etc/modules/Their_Module.xml
After:
  app/etc/modules/50_Their_Module.xml
  app/etc/modules/60_My_Module.xml

Now “My_Module” is loaded after “Their_Module” so I can override their config and/or layout without modifying their files and without My_Module depending on Their_Module!

The purpose of this post is not to provide a bunch of cache backend benchmarks, but rather to simply introduce my benchmark code in the hopes that others conduct their own tests and hopefully publish their findings. A link to this post is appreciated. Also, if there are any criticisms of the benchmark I’d love to see a pull request. :)

The benchmark suite is fully-featured:

• Repeatable tests. Dataset is written to static files so the exact same test can be repeated, even with entirely different backends.
• Test datasets can easily be zipped up and copied to different environments or shared for others to use.
• Can relatively easily test multiple pre-generated datasets to compare different scenarios on the same hardware.
• Uses true multi-process benchmarking, each process with a different set of random operations.
• Flexible dataset generation via options to init command. Cache record data size, number of tags, expiration, popularity and volatility are all randomized.

Currently the benchmarks are run via the command line so testing the APC backend or any others that only work via a cgi or apache module environment will not work. This could be remedied easily enough with the use of CuRL and some php copy/paste if you had the desire to test on your actual web server.

Here is an example run using the Redis backend using my dev environment, a Lubuntu VirtualBox guest:

Cache Backend: Zend_Cache_Backend_Redis
Loading 'default' test data...
Loaded 10000 cache records in 29.1080 seconds. Data size is 5008.9K
Analyzing current cache contents...
Counted 10023 cache IDs and 2005 cache tags in 0.2062 seconds
Benchmarking getIdsMatchingTags...
Average: 0.00036 seconds (36.82 ids per tag)
Benchmarking 4 concurrent clients, each with 100000 operations...
4 concurrent clients completed in 62 seconds

         |   reads|  writes|  cleans
------------------------------------
Client  1| 1811.83|  184.66|    6.81
Client  2| 1799.84|  165.29|    6.91
Client  3| 1818.90|  165.17|    6.79
Client  0| 1790.91|  153.56|    7.40
------------------------------------
ops/sec  | 7221.48|  668.68|   27.91

The important numbers to look at are the summed ops/sec. Given the three variables: dataset, hardware and backend, it is easy to change just one of these without affecting the others so this benchmark can be used to test any one of the three variables reliably. The three metrics observed are reads, writes and cleans. The first two are pretty self-explanatory. The third is a clean operation on a single tag using Zend_Cache::CLEANING_MODE_MATCHING_ANY_TAG which is the only mode Magento ever uses other than Zend_Cache::CLEANING_MODE_ALL for manual cache refreshes. Individual read/write operations are very fast so given the large number of operations in a test I did not feel the need to examine min, max, average, or standard deviations.

The test uses (hopefully) sane defaults for dataset generation parameters, but there is plenty of flexibility. I advise you to examine your production environment (number of cache keys, number of cache tags, number of concurrent clients) to tweak the test to more closely match your own environment. Here is the output of the --help cli parameter:

$ php shell/cache-benchmark.php --help
This script will either initialize a new benchmark dataset or run a benchmark.

Usage:  php -f shell/cache-benchmark.php [command] [options]

Commands:
  init [options]        Initialize a new dataset.
  load --name <string>  Load an existing dataset.
  clean                 Flush the cache backend.
  tags                  Benchmark getIdsMatchingTags method.
  ops [options]         Execute a pre-generated set of operations on the existing cache.

'init' options:
  --name <string>       A unique name for this dataset (default to "default")
  --keys <num>          Number of cache keys (default to 10000)
  --tags <num>          Number of cache tags (default to 2000)
  --min-tags <num>      The min number of tags to use for each record (default 0)
  --max-tags <num>      The max number of tags to use for each record (default 15)
  --min-rec-size <num>  The smallest size for a record (default 1)
  --max-rec-size <num>  The largest size for a record (default 1024)
  --clients <num>       The number of clients for multi-threaded testing (defaults to 4)
  --seed <num>          The random number generator seed (default random)

'ops' options:
  --name <string>       The dataset to use (from the --name option from init command)
  --client <num>        Client number (0-n where n is --clients option from init command)
  -q|--quiet            Be less verbose.

To handle multi-process benchmarking the test is actually launched from a shell script which backgrounds each client and sums the results using awk so unless you are doing single-process benchmarks you never need to invoke the ‘ops’ command yourself.

Give me the code already!

The code is hosted at github.com/colinmollenhour/magento-cache-benchmark. If you use modman you can install it like so:

modman clone cachebench git://github.com/colinmollenhour/magento-cache-benchmark.git

Or, you may also download it directly and just extract cache-benchmark.php to the “shell” folder in your Magento installation.

Run a test!

Assuming you’ve cloned/downloaded the code already, here is how you run your first test:

php shell/cache-benchmark.php init
bash var/cachebench/default/run.sh

Could it get any easier?

PS. I included a “Null” backend which is just a black hole for the purpose of getting a general idea of your PHP overhead.

EDIT: Updated to use multi-line continuations for readability.

# Substitute for svn:externals
@shell \
  SRC=https://example.com/source/magento/common; \
  DEST=modules; \
  if [ -d $DEST/.svn ]; then \
    svn update $DEST; \
  else \
    svn checkout $SRC $DEST; \
  fi

# Import subversion-based modules
@import    modules/Foo
@import    modules/Bar

Now, when I clone the module (git is supported natively as of 1.1.0) or any time I run an update, deploy or repair, the subversion modules will be checked out or updated as well. So, if you ever need to mix and match SCM’s, go right ahead. I still recommend the use of git submodules for all-git projects and svn:externals for all-subversion projects, of course.

I’ll consider adding a builtin function that tidies up the above and handles things like switching urls..

Mistake #1

If you move your .htaccess contents into the apache config files that is great, but if you forget to disable .htaccess files using AllowOverride None then you’ve entirely defeated the purpose! The reason this technique can result in performance gains is that it allows the server to ignore the .htaccess files which means fewer directory traversals and fewer file reads with each request. One page load probably results in 20-100 HTTP requests to your Apache server. If you have deeply nested files (all of those cached product images and skin images come to mind) then this results in a lot of disk access. So, you must disable the .htaccess files completely with AllowOverride None! Which brings me to the second major mistake that people can make…

Mistake #2

Now that you’ve disabled .htaccess, guess what? Those .htaccess files had a purpose.. For example, to deny access to your app/etc/local.xml config file!! Great way to expose your database password, let’s just hope the user is a localhost-only user and you don’t re-use passwords… Also, in downloader/.htaccess for example, compression is disabled so that the Magento Connect console can circumvent Apache’s buffering to display the console output as it occurs. There are others as well. Point is, now you have to move all of these .htaccess files into your apache config. Ugh.. Which brings me to my last point…

Mistake #3

It doesn’t have to be difficult! I’ve written a bash script (buildhtaccess.sh, see below) which provides you an easy way to benefit from this tweak. In essence the following script will find all of your .htaccess files and compile them into one file which can then be included in your Apache config with only a few lines. If you modify an .htaccess file or a new one is installed by an extension, re-run the script and reload the apache config.

Tutorial

First, download and run the script to generate your .htaccess-combined file.

$ cd <webroot>
$ wget http://gist.github.com/raw/459311/buildhtaccess.sh
$ bash buildhtaccess.sh

Next, edit your apache config file. Find the <VirtualHost> directive and at the end of it add the following:

  # Include combined Magento .htaccess files
  <Directory /your_webroot_here>
    AllowOverride None
  </Directory>
  Include /your_webroot_here/.htaccess-combined

Lastly, run a graceful reload of the Apache config. A full restart is not necessary. Now you’re done.

Code

And here is the code for generating the .htaccess-combined file:

Benchmarks

For my benchmark I ran “ab” (apache bench) since it is quick and easy but admittedly not the best for benchmarking overall site performance. In this case apache bench is going to under state the value of this tweak since one page load in a browser will result in repeated hits on the webserver, all of which will be affected by this optimization. So, I benchmarked on /errors/503.php so that the database is not factored but PHP code is still run. Specifically, I used $ ab -n 1000 -c 20 .../errors/503.php.

Before: ~3000 requests per second (3018 max)

After: ~4600 requests per second (5030 max)

That’s a pretty nice improvement. Unfortunately it is only substantial with fast requests such as the error pages and static files but it should definitely reduce disk contention on a busy server and increase scalability, but probably not do a whole lot for user-experience on a low-contention server.

I’d love to see your results, please post them in the comments!

Additionally, Module Manager now has it’s own home on Google Code so check it out! (pun intended)

Here is an example of the new nested modules feature:

# My template files
skin               skin/frontend/my/default
design             app/design/frontend/my/default

# Import Colin_HotDealz module
@import            modules/Colin_HotDealz

Update 1/30/2010:
Added nested modules, renamed .modman file to modman, released under Apache License 2.0, and hosted on Google Code.

This project was inspired by my recent Magento development work so I’m not sure how much application it has outside of that, but if you’ve ever wanted to use svn:externals with individual files (you can’t btw) then this is the next best thing (and therefore the best thing). Perhaps some other VCS supports this functionality natively , if you know of it please clue me in. I’ve used git and bazaar but still like the simplicity of subversion for small, non-distributed projects.

Magento’s source code is split among several different groups of files: code, design, locale, skin, libs, etc (as in config), etc.. (as in etcetera)
You really cannot avoid ever having to mix a file in here or there which means svn:externals won’t cut it. Besides, svn:externals won’t let you make one commit from all of your externals so to commit your changes you have to do it once for each folder which makes your commit history nearly useless. Enter “modman”, my Magento module manager. While it could be used for deployment it is primarily geared towards developers. Here is how it works:

Create a branch and put your source code in it. Now create a “.modman” file within the root of the branch. In this file you define (much like svn:externals) a path within your repository and the path that it should map to. Here is an example (with a fictitious module named “HotDealz”:

code   app/code/local/Colin/HotDealz 
template   app/design/frontend/default/default/template/hotdealz
hotdealz.xml   app/design/frontend/default/default/layout/hotdealz.xml
images   skin/frontend/default/default/images/hotdealz
HotDealz.csv   app/locale/en_US/Colin_HotDealz.csz

As you can see, very similar to svn:externals only with individual file support which is handy for things like locale files that can’t go in subdirectories.

How It Works

Usage is similar to regular command line usage of subversion. The “modman” script provides a wrapper around svn which checks the code out in a .modman directory at the root of your project then reads the .modman file and creates symbolic links as necessary so that your source code is actually all contained within one directory and your webserver accesses the files where they are supposed to be. No svn:externals definitions are used. Obviously this requires a few things:

1. bash
2. a webserver that is configured to FollowSymLinks

If you do web development on Windows, you may be thinking, “What about Windows support!?”. Well, you have a few options:

• Do your development inside a VirtualBox instance of Linux and share the drive to Windows for editing on your IDE (recommended).
• Try to configure cygwin to work (untested, not recommended).

I chose the former long before I wrote this script since the servers for my projects are all Linux-based and doing development on WAMP when your project runs on L*MP is silly. You can actually still use TortoiseSVN by simply opening the .modman directory from explorer so really after the initial checkout you do not need to continue to use the script except for when adding new definitions to the .modman file.

Sample Usage

Here are some typical commands used with modman:

$ modman --help
$ modman init
$ modman hotdealz checkout "svn url here"
$ modman hotdealz status (can be run from any directory within the project root)
$ modman hotdealz commit
$ modman hotdealz update
$ modman hotdealz add Colin_HotDealz.xml app/etc/modules/HotDealz.xml
etc...

Most commonly used svn commands are passed through to svn for convenience but for advanced work you can just cd to the .modman directory or use TortoiseSVN or your IDE’s integrated subversion tools (I use Netbeans).

Documentation

----------------------
Magento Module Manager
----------------------
Usage: modman <module> <action> [svn args]
Supported global actions (no module specified):
  init           initialize the current directory as the modman root (no module name)
  update-all     update all modules that are currently checked out

Supported actions: (all svn commands support additional arguments)
  checkout       checkout a new modman compatible module
  export         export a modman compatible module (does real copy instead of symlinks)
  update [...]   update module
  add <svn_path> <real_path>   add a file/dir to the .modman file and working copy
  delete <svn_path>   remove a file/dir from the .modman file and working copy
  status         show status of entire module
  diff [...]     run svn diff on module
  commit [...]   commit changes to module
  info           show the module's working copy path and run svn info on working copy
  list           list the definitions in the .modman file

The repository should contain a file called ".modman" which defines which files
go where relative to the directory where modman was initialized.

If additional arguments are given to commands such as commit, paths
should be relative to the svn root.

---- Start example .modman file ----
# Comments are supported, begin a line with a hash
code                   app/code/local/My/Module/
design                 app/design/frontend/default/default/mymodule/
locale/My_Module.xml   app/locale/en_US/My_Module.xml
My_Module.xml          app/etc/modules/My_Module.xml
---- End example .modman file ----

Author:
Colin Mollenhour

http://colin.mollenhour.com/

colin@mollenhour.com

View Source

Download

Leave a comment if you use it!

It turns out that when layered navigation indexes are refreshed the website_id is 0 which causes the filtered results to be empty which makes layered navigation disappear. The fix is to simply place an if statement around the filter dependent on the website id no being 0. The Magento Connect module has been updated with this fix (0.3.1).

Ok, I think this will finally be the last update to this module… I’ve learned a lot about Magento since the first version of this module, and think I finally did it right this time. This update adds support for advanced search (thanks saho for the bug report) and also every other aspect of the store I could find.

I ended up doing a search like so:

app/code/core/Mage $ grep -rl addVisibleIn.\*FilterToCollection .
./Checkout/Block/Cart/Crosssell.php
./Tag/Block/Product/Result.php
./Tag/Block/Customer/View.php
./Adminhtml/Block/Sales/Order/Create/Search/Grid.php
./Adminhtml/Model/Sales/Order/Random.php
./Bundle/Block/Catalog/Product/List/Partof.php
./Bundle/Model/Observer.php
./Reports/Block/Product/Abstract.php
./Wishlist/Block/Share/Email/Items.php
./Wishlist/Block/Share/Wishlist.php
./Wishlist/Block/Customer/Wishlist.php
./Wishlist/Block/Customer/Sidebar.php
./Wishlist/Model/Mysql4/Wishlist.php
./CatalogSearch/Model/Layer.php
./CatalogSearch/Model/Advanced.php
./Rss/Block/Catalog/New.php
./Rss/Block/Catalog/Special.php
./CatalogIndex/Model/Indexer.php
./Sales/Model/Order.php
./Catalog/Block/Seo/Sitemap/Product.php
./Catalog/Block/Product/Compare/List.php
./Catalog/Block/Product/New.php
./Catalog/Block/Product/List/Related.php
./Catalog/Block/Product/List/Crosssell.php
./Catalog/Block/Product/List/Upsell.php
./Catalog/Block/Product/Bestsellers.php
./Catalog/Model/Layer.php
./Catalog/Model/Product/Visibility.php
./Catalog/Helper/Product/Compare.php

and determined (obviously) that it would be best to simply override the addVisibleIn*FilterToCollection functions rather than to try to override all of the above. I actually discovered the following code which turned out to not be what I wanted, but I think that is due to it not being updated when Varien added the stock_status table:

Mage::getSingleton('cataloginventory/stock')->addInStockFilterToCollection($collection);

However, I liked the idea of having such a function available that actually works as intended (the above does not work on configurable products) so I did just that.

First, the function that can be used on any product collection to filter out of stock products:

app/code/local/Lucky/InStockOnly/Model/Stock.php

<?php

class Lucky_InStockOnly_Model_Stock {

  /**
   * Add a statusInStock requirement for visibility
   */
  public function addInStockFilterToCollection($collection)
  {
    if($websiteId = Mage::app()->getWebsite()->getWebsiteId()) {
      $collection->joinField(
        'stock_status',
        'cataloginventory/stock_status',
        'stock_status',
        'product_id=entity_id', array(
          'stock_status' => Mage_CatalogInventory_Model_Stock_Status::STATUS_IN_STOCK,
          'website_id' => $websiteId
        )
      );
    }
  }

}

<?php

class Lucky_InStockOnly_Model_Visibility extends Mage_Catalog_Model_Product_Visibility {

  public function addVisibleInCatalogFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInCatalogFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

  public function addVisibleInSearchFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInSearchFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

  public function addVisibleInSiteFilterToCollection(Mage_Eav_Model_Entity_Collection_Abstract $collection)
  {
    parent::addVisibleInSiteFilterToCollection($collection);
    Mage::getSingleton('cataloginventory/instockonly')->addInStockFilterToCollection($collection);
    return $this;
  }

}

<?xml version="1.0"?>
<config>

<global>
  <models>

    <catalog>
      <rewrite>
        <product_visibility>Lucky_InStockOnly_Model_Visibility</product_visibility>
      </rewrite>
    </catalog>
    <cataloginventory>
      <rewrite>
        <instockonly>Lucky_InStockOnly_Model_Stock</instockonly>
      </rewrite>
    </cataloginventory>

  </models>
</global>

</config>

<config>
  <modules>
    <Lucky_InStockOnly>
      <active>true</active>
      <codePool>local</codePool>
    </Lucky_InStockOnly>
  </modules>
</config>